Cabin control method, device, equipment and medium

Abstract

The invention discloses a cockpit control method. The method comprises the following steps: acquiring target video data; performing feature extraction on the target video data to obtain optical flow information and RGB pixel information of the target video data; inputting the optical flow information and the RGB pixel information into a pre-trained prediction model, and predicting a shooting view angle of target video data or / and a wind speed of a space where a target object in the target video data is located; controlling the cabin based on the shooting view angle of the target video data or / and the wind speed of the space where the target object in the target video data is located; according to the invention, through the multi-frame short video data, the model can better identify the change of the continuous information of the picture, and the identification rate is high; by extracting the optical flow information, the model can pay attention to the motion condition in a picture, and a gain effect on prediction is achieved; and meanwhile, RGB information is fused with multi-frame optical flow information, so that the motion characteristics of picture contents can be increased, the two characteristics are mutually gained, and even if an abnormal condition is encountered, better prediction can be carried out.

Description

Technical Field

[0001] The present application relates to the technical field of cockpit control, and in particular to a cockpit control method, device, equipment and medium. Background Art

[0002] Vehicles have become an indispensable tool in people's lives, and people use vehicles more and more frequently. With the development of vehicle intelligence and networking, the smart cockpit system needs to provide a control platform for human-computer interaction and in-vehicle infotainment services. Smart cockpits based on videos to achieve vehicle-human interaction are becoming more and more popular with users. When predicting the wind speed of the environment in which the object is located in the video, a single picture can be used for prediction based on deep learning, but there are significant limitations in using a single picture for wind speed prediction, mainly manifested in the lack of information dimension: a single picture can only capture static information at a specific moment and in a local area, and lacks dynamic data and spatial overview of wind speed changes over time. This makes it difficult to extract effective features directly related to wind speed from the image, increasing the complexity of model design and the difficulty of training. In addition, a single picture is easily affected by external environmental factors such as light and weather, resulting in poor robustness and generalization of the model for wind speed prediction, and prone to overfitting problems, resulting in poor wind speed control in the smart cockpit and poor passenger experience. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present application provides a cockpit control method, device, equipment and medium to solve at least one defect in the prior art.

[0004] To achieve the above objectives and other objectives, the present application provides a cockpit control method, which is applied to a smart cockpit. The cockpit control method includes:

[0005] Get target video data;

[0006] Performing feature extraction on the target video data to obtain optical flow information and RGB pixel information of the target video data;

[0007] The optical flow information and the RGB pixel information are input into a pre-trained prediction model, and the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data are predicted by the prediction model; wherein the sample prediction model is obtained by training the initial prediction model with the optical flow information and RGB pixel information of the video data as input and the shooting angle of the video data or / and the wind speed of the space where the target object is located in the sample video data as output;

[0008] The cockpit is controlled based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

[0009] In one embodiment of the present application, the prediction model includes: a first feature extraction layer, a second feature extraction layer, a feature fusion layer, a first classification layer, and a second classification layer. The step of predicting the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data by using the prediction model includes:

[0010] Extracting optical flow features of the target video data based on the optical flow information through the first feature extraction layer;

[0011] Extracting RGB features of the target video data based on the RGB pixel information through the second feature extraction layer;

[0012] The feature fusion layer is used to perform feature fusion on the optical flow feature and the RGB feature to obtain a fused feature.

[0013] The first classification layer is used to predict the shooting angle of the target video data based on the fusion feature, and the second classification layer is used to predict the wind speed of the space where the target object in the target video data is located based on the fusion feature.

[0014] In one embodiment of the present application, the step of obtaining target video data includes:

[0015] Get the original video data;

[0016] Performing frame extraction processing on the original video data to obtain a video frame sequence;

[0017] The video frame sequence is segmented to obtain a plurality of video frame subsequences as target video data.

[0018] In one embodiment of the present application, the first feature extraction layer includes a first feature extraction unit and a second feature extraction unit, and the step of extracting the optical flow features of the target video data based on the optical flow information through the first feature extraction layer includes:

[0019] Extracting a spatial feature vector of the optical flow information by the first feature extraction unit;

[0020] Based on the spatial feature vector of the optical flow information, the spatiotemporal feature vector of the optical flow information is extracted by the second feature extraction unit as an optical flow feature.

[0021] In one embodiment of the present application, the second feature extraction layer includes a third feature extraction unit and a fourth feature extraction unit, and the step of extracting the RGB features of the target video data based on the RGB pixel information through the second feature extraction layer includes:

[0022] Extracting a spatial feature vector of the RGB pixel information by the third feature extraction unit;

[0023] Based on the spatial feature vector of the RGB pixel information, the spatiotemporal feature vector of the RGB pixel information is extracted by the fourth feature extraction unit as an RGB feature.

[0024] In one embodiment of the present application, the first feature extraction unit and the third feature extraction unit are deep residual networks Resnet18, and the second feature extraction unit and the fourth feature extraction unit are bidirectional long short-term memory artificial neural networks.

[0025] In one embodiment of the present application, the step of using the feature fusion layer to perform feature fusion on the optical flow feature and the RGB feature includes:

[0026] Performing weighted summation on the optical flow feature and the RGB feature to obtain a fusion feature;

[0027] fusion_featrue=w0*rgb_feature+w1*flow_feature

[0028] Among them, fusion_featrue represents fusion features, rgb_feature represents RGB features, flow_feature represents optical flow features, w0, w1 represent weights, w0+w1=1.

[0029] In order to achieve the above-mentioned object and other related objects, the present application provides a cockpit control device, the cockpit control device comprising:

[0030] A data acquisition module, used for acquiring target video data;

[0031] A feature extraction module is used to extract features from the target video data to obtain optical flow information and RGB pixel information of the target video data;

[0032] A prediction module, used for inputting the optical flow information and the RGB pixel information into a pre-trained prediction model, and predicting the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data through the prediction model; wherein the prediction model is obtained by training the initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the angle of the sample shooting video data or / and the wind speed of the space where the target object is located in the sample video data as output;

[0033] The cockpit control module is used to control the cockpit based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

[0034] To achieve the above objectives and other related objectives, the present application provides a cockpit control device, including:

[0035] one or more processors; and

[0036] The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the memory implements the cockpit control method.

[0037] To achieve the above objectives and other related objectives, the present application provides one or more machine-readable media on which instructions are stored, which, when executed by one or more processors, enable the processors to execute the cockpit control method.

[0038] Beneficial effects of this application:

[0039] A cockpit control method of the present application includes: obtaining target video data; extracting features from the target video data to obtain optical flow information and RGB pixel information of the target video data; inputting the optical flow information and the RGB pixel information into a pre-trained prediction model, and predicting the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data through the prediction model; wherein the prediction model is obtained by training the initial prediction model with the optical flow information and RGB pixel information of the sample video data as input, and the angle of the sample video data or / and the wind speed of the space where the target object is located in the sample video data as output; and controlling the cockpit based on the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data. The present application can make the model better recognize the changes in the continuous information of the picture through multiple frames of short video data, and the recognition rate is high; by extracting the optical flow information, the model can pay attention to the motion in the picture, which has a gain effect on the prediction; at the same time, the RGB information fuses multiple frames of optical flow information, which can increase the motion characteristics of the picture content, and the two characteristics gain each other, so that even in the event of abnormal situations, it can be better predicted.

[0040] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0042] Figure 1 A schematic diagram of an implementation environment of a cockpit control method according to an embodiment of the present application;

[0043] Figure 2 This is a principle block diagram of a cockpit control method according to an embodiment of the present application;

[0044] Figure 3 A flowchart of a prediction process according to an embodiment of the present application;

[0045] Figure 4 is a block diagram of a cockpit control device according to an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the memory of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0049] Although the terms "first", "second", "A", and "B", etc. may be used herein to describe various elements, these elements should not be limited by these terms and are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the following technology. The term "and / or" includes a combination of multiple related items or any of the multiple related items.

[0050] As used herein, unless the context indicates otherwise, the singular form is intended to include the plural form, and it will be understood that the term "comprising" means the presence of stated features, quantities, steps, operations, elements, or combinations thereof, but does not exclude the presence or addition of one or more other features, quantities, steps, operations, elements, components, or combinations thereof.

[0051] Before the detailed description, it is intended to clarify that the division of components in this specification is divided only by the main function of each component. That is, two or more components to be described below can be combined into one component, or can be divided into two or more components according to more detailed functions. In addition to the main function of the component, each component to be described below can also perform some or all of the functions of other components, and some of the main functions of each component can be exclusively performed by other components.

[0052] Figure 1 Schematic diagram of the implementation environment of the cockpit control method of an embodiment of the present application. Figure 1 , the implementation environment includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 communicate with each other through a wireless network. The terminal device 110 can obtain target video data from the server 120; extract features of the target video data to obtain optical flow information and RGB pixel information of the target video data; input the optical flow information and the RGB pixel information into a pre-trained prediction model, and predict the shooting angle of the target video data or / and the wind speed of the space where the target object in the target video data is located through the prediction model; wherein the prediction model is obtained by training the initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the angle of the sample shooting video data or / and the wind speed of the space where the target object in the sample video data is located as output; the cockpit is controlled based on the shooting angle of the target video data or / and the wind speed of the space where the target object in the target video data is located. This application uses multi-frame short video data to enable the model to better recognize changes in continuous information in the picture, with a higher recognition rate; by extracting optical flow information, the model can pay attention to the motion in the picture, which has a gain effect on prediction; at the same time, RGB information is fused with multi-frame optical flow information to increase the motion characteristics of the picture content. The two features gain each other, and even abnormal situations can be better predicted.

[0053] It should be understood that Figure 1 The number of terminal devices 110 and servers 120 in the embodiment is only for illustration. Any number of terminal devices 110 and servers 120 may be provided according to actual needs.

[0054] Among them, the terminal device 110 corresponds to the client, which can be any electronic device with a user input interface, including but not limited to smart phones, tablets, laptops, computers, car computers, etc., among which the user input interface includes but is not limited to touch screens, keyboards, physical buttons, audio pickup devices, etc.

[0055] Among them, server 120 corresponds to the service end, which can be a server that provides various services. It can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and basic cloud computing services such as big data and artificial intelligence platforms. This is not restricted here.

[0056] The terminal device 110 can communicate with the server 120 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), and this is not restricted here.

[0057] In order to implement the cockpit control method disclosed in the present application, the prediction model needs to be trained in advance. The model training process includes:

[0058] S1. Obtain a historical data sample set and divide it into a training sample set and a test sample set;

[0059] First, a large amount of video data needs to be collected for the entertainment scenes in the cabin. Since the input of this application is short video data, the long video data needs to be extracted and divided into short videos that meet the requirements. Furthermore, in order to ensure that the saved short videos do not jump in time and space, and to ensure that the saved short videos are not too similar to cause data redundancy, 5 frames of data will be extracted for every second during data segmentation. Data that is too long will also cause pressure on subsequent calculations, so it is saved as 15 frames of data for 3 seconds.

[0060] Since this application is based on supervised data for training, after the data segmentation is completed, the data needs to be labeled with classification labels, and the shooting angle of the video data and / or the wind speed of the space where the target object is located in the video data are marked. When using the annotation software to annotate the video data, you first need to load the image directory, and then load the label directory. In the middle of the software is the visualization of the short video data, which will be played in a loop. The corresponding number needs to be entered in the input box below. Here, the wind speed is divided into three levels: breeze, strong wind, and no wind, corresponding to the three numbers 0, 1, and 2 respectively. The shooting angle is divided into: first-person perspective and non-first-person perspective, corresponding to the two numbers 0 and 1. For example, the short video listed has the annotation content: 2-1, which represents no wind-non-first-person perspective.

[0061] After the above short video data is labeled, the label file corresponding to the short video file name will be saved in the label folder. Then the video data is processed, and the short video data and label file are merged into a file that can be used for subsequent model training. Each line is the short video file name and its corresponding label file, which can then be used for subsequent model training.

[0062] S2. Use the training sample set to train the preset deep learning model;

[0063] Specifically, during training, 70% of the training samples in the training sample set can be used to train the deep learning model, and 30% of the training samples can be used to test the deep learning model.

[0064] S3. Use the test sample set to test the trained deep learning model. When the test result meets the preset conditions, the neural network model is determined to be the trained prediction model.

[0065] In the training phase, the category of each sample is known after data annotation, so the result obtained by the model needs to be calculated with the known label. The cross entropy is used here. The calculated loss is used to reversely update the gradients of all layers of the model, and then update the layer weights. The weights are saved after the model is stable. In the inference phase, the model parameters are not changed, and the true label of the sample is also unknown. The original label of the input short video sample obtained by the model prediction may be in the following form: wind speed prediction result [0.3, 0.2, 0.5], view prediction result [0.4, 0.6]. The index id with the largest value is 2 and 1 (index number starts from 0), so the true label corresponding to the index is no wind and non-first view.

[0066] At this point, the training of the prediction model is completed.

[0067] See also Figure 2 , Figure 2 This is a flow chart of a cockpit control method according to an embodiment of the present application. This method can be used to control a smart cockpit. Figure 2 As shown, the cockpit control method at least includes steps S210 to S240:

[0068] Step S210, obtaining target video data;

[0069] The target video data is the video data to be displayed by the terminal device. The video data can be obtained by the terminal device from the cloud server. Then, the video data obtained from the cloud server is processed to obtain the target video data.

[0070] In an embodiment of the present application, the step of obtaining target video data includes:

[0071] Acquire original video data; perform frame extraction processing on the original video data to obtain a video frame sequence; perform data segmentation on the video frame sequence to obtain multiple video frame subsequences as target video data.

[0072] It should be noted that the original video data can be obtained by the terminal device from the cloud server. Since the input of this application is short video data, it is necessary to first extract frames from the long video data to obtain a video frame sequence. The video frame sequence is then segmented and saved as multiple short videos that meet the requirements. Furthermore, in order to ensure that the saved short videos do not jump in time and space, and to ensure that the saved short videos do not have too many similarities that cause data redundancy, 5 frames of data are extracted for every second during data segmentation. Data that is too long will also cause pressure on subsequent calculations, so it is saved as 15 frames of data for 3 seconds.

[0073] Step S220, extracting features from the target video data to obtain optical flow information and RGB pixel information of the target video data;

[0074] To extract the optical flow information, we need to use opencv to calculate the dense optical flow between two frames. After the image is grayed, the optical flow is calculated for two consecutive frames in a loop. However, the number of consecutive combinations of 15 frames is 14. In order to ensure that the dimension of the calculated frame does not change, the optical flow of the first group is initialized to a matrix of all 0s. In this way, the dimension does not change, which is [B, 15, C, H, W]. B is the batch size, which represents how much data is sent to the model at a time. 15 is a fixed value, which represents how many frames each data has. C is the number of channels. H and W are the height and width of the short video. Among them, C = 2 is the motion vector of the current pixel x, y calculated by the optical flow.

[0075] For the extraction of RGB pixel information, use the cv2.VideoCapture function to read the short video data and initialize a tensor matrix with a dimension of [15, C, H, W]. In order to save the original pixel data read later, loop the read video data, read each frame of data, and use the cv2.cvtColor function to convert the channel from BGR to RGB for each frame of data, and then store it in the tensor matrix initialized before. There are 15 frames in total. Here, the value of C=3 represents the normalized original RGB pixels of the image. For example, the original RGB value is [230, 100, 20]. After normalization, it is divided by 255 to get [0.0784, 0.0824, 0.0980]. Then the average value [0.485, 0.456, 0.406] is subtracted in RGB order, and then divided by the standard deviation [0.229, 0.224, 0.225] to get [-1.7754, -1.6681, -1.3687]. The purpose of this step is to make all the values ​​have 0 mean and unit variance after processing, which can provide faster convergence and more stable training.

[0076] Step S230, input the optical flow information and the RGB pixel information into a pre-trained prediction model, and predict the shooting angle of the target video data and / or the wind speed of the space where the target object is located in the target video data through the prediction model; wherein the prediction model is obtained by training an initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the angle of view of the sample shooting video data and / or the wind speed of the space where the target object is located in the sample video data as output.

[0077] In one embodiment of the present application, the prediction model includes: a first feature extraction layer, a second feature extraction layer, a feature fusion layer, a first classification layer, and a second classification layer. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a prediction process according to an embodiment of the present application. Figure 3 In the step of predicting the shooting angle of the target video data or / and the wind speed of the space where the target object in the target video data is located by using the prediction model comprises:

[0078] Step S310, extracting optical flow features of the target video data based on the optical flow information through the first feature extraction layer;

[0079] In an embodiment of the present application, the first feature extraction layer includes a first feature extraction unit and a second feature extraction unit, and the step of extracting the optical flow features of the target video data based on the optical flow information through the first feature extraction layer includes:

[0080] The spatial feature vector of the optical flow information is extracted by the first feature extraction unit; based on the spatial feature vector of the optical flow information, the spatiotemporal feature vector of the optical flow information is extracted by the second feature extraction unit as an optical flow feature.

[0081] It should be noted that the first feature extraction unit is the deep residual network Resnet18, and the second feature extraction unit is the bidirectional long short-term memory artificial neural network LSTM (Long Short-Term Memory).

[0082] Resnet18 is a deep residual network, which consists of 18 layers of convolutional neural networks, including multiple residual blocks, each of which consists of two 3x3 convolutional layers and 1x1 convolutional layers. In this application, the feature extraction capability of Resnet18 is mainly used. The model will cyclically extract internal hidden features from multi-frame dense optical flow features. This part of the features includes the motion state of pixel content represented by the optical flow extracted from multi-frame dense optical flow features, the motion connection features between pixels, etc., that is, the spatial feature vector of optical flow information. Then, a bidirectional LSTM is used to associate the feature information of these multiple frames in the time dimension, and then learn the long-term dependencies in the data, that is, the spatiotemporal feature vector of optical flow information - optical flow features.

[0083] Step S320, extracting RGB features of the target video data based on the RGB pixel information through the second feature extraction layer;

[0084] In an embodiment of the present application, the second feature extraction layer includes a third feature extraction unit and a fourth feature extraction unit, and the step of extracting the RGB features of the target video data based on the RGB pixel information through the second feature extraction layer includes:

[0085] The spatial feature vector of the RGB pixel information is extracted by the third feature extraction unit; based on the spatial feature vector of the RGB pixel information, the spatiotemporal feature vector of the RGB pixel information is extracted by the fourth feature extraction unit as the RGB feature.

[0086] It should be noted that the third feature extraction unit is the deep residual network Resnet18, and the fourth feature extraction unit is the bidirectional long short-term memory artificial neural network LSTM (Long Short-Term Memory).

[0087] In this application, the resnet18 feature extraction capability is mainly used. The model will cyclically extract internal hidden features from multiple frames of RGB raw data. This part of the features includes the feature information of the RGB raw data itself, including: boundaries, corners, contours, color features, etc., that is, the spatial feature vector of RGB pixel information. Then, a bidirectional LSTM is used to associate the feature information of these multiple frames in the time dimension, and then learn the long-term dependency in the data, that is, the spatiotemporal feature vector of RGB pixel information - RGB features.

[0088] Step S330: Using the feature fusion layer to fuse the optical flow feature and the RGB feature to obtain a fused feature.

[0089] In an embodiment of the present application, the step of using the feature fusion layer to perform feature fusion on the optical flow feature and the RGB feature includes:

[0090] Performing weighted summation on the optical flow feature and the RGB feature to obtain a fusion feature;

[0091] fusion_featrue=w0*rgb_feature+w1*flow_feature

[0092] Among them, fusion_featrue represents fusion features, rgb_feature represents RGB features, flow_feature represents optical flow features, w0, w1 represent weights, w0+w1=1.

[0093] Step S340, using the first classification layer to predict the shooting angle of the target video data based on the fusion feature, and using the second classification layer to predict the wind speed of the space where the target object is located in the target video data based on the fusion feature.

[0094] It should be noted that wind speed prediction and viewing angle prediction are both separate classification layers, and the classification layer can be a fully connected layer, in which the dimension dim of the feature input to the classification layer is the same as the dimension dim of the fused feature. The fully connected layer predicts the shooting viewing angle of the target video data and the wind speed of the space where the target object is located in the target video data by fusing features.

[0095] It should be noted that the target object in the target video data may be any object in the video data, such as a tree, a cloud, a person, a car, and the like.

[0096] Step S240: Controlling the cockpit based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

[0097] When the shooting angle of the target video data or / and the wind speed of the space where the target object in the target video data is located are obtained based on the prediction model, the cockpit can be controlled to allow the user to have an immersive experience. For example, when the wind speed in the video being watched is predicted to be a breeze, the wind output by the vehicle's air supply system can be controlled to be a breeze, and when the wind speed in the video being watched is predicted to be a strong wind, the wind output by the vehicle's air supply system can be controlled to be a strong wind. When the angle of view of the target video data being played is predicted to be the first angle of view, the seat movement can be controlled, such as forward, backward, or left, etc.

[0098] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Figure 4 is a block diagram of a cockpit control device according to an embodiment of the present application. The device can be applied to Figure 1 The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0100] like Figure 4 As shown, a cockpit control device comprises:

[0101] A data acquisition module 410 is used to acquire target video data;

[0102] A feature extraction module 420 is used to extract features from the target video data to obtain optical flow information and RGB pixel information of the target video data;

[0103] A prediction module 430 is used to input the optical flow information and the RGB pixel information into a pre-trained prediction model, and predict the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data through the prediction model; wherein the prediction model is obtained by training an initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the angle of the sample shooting video data or / and the wind speed of the space where the target object is located in the sample video data as output;

[0104] The cockpit control module is used to control the cockpit based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

[0105] In an embodiment of the present application, the prediction model includes:

[0106] A first feature extraction layer, used to extract optical flow features of the target video data based on the optical flow information;

[0107] A second feature extraction layer, used to extract RGB features of the target video data based on the RGB pixel information;

[0108] The feature fusion layer is used to fuse the optical flow feature and the RGB feature to obtain a fused feature.

[0109] A first classification layer, used to predict the shooting angle of the target video data based on the fusion feature;

[0110] The second classification layer is used to predict the wind speed of the space where the target object in the target video data is located based on the fusion feature.

[0111] In an embodiment of the present application, the data acquisition module is used to acquire original video data; perform frame extraction processing on the original video data to obtain a video frame sequence; and perform data segmentation on the video frame sequence to obtain multiple video frame subsequences as target video data.

[0112] In an embodiment of the present application, the first feature extraction layer includes:

[0113] A first feature extraction unit, used to extract a spatial feature vector of the optical flow information;

[0114] The second feature extraction unit is used to extract the spatiotemporal feature vector of the optical flow information as the optical flow feature based on the spatial feature vector of the optical flow information.

[0115] In an embodiment of the present application, the second feature extraction layer includes:

[0116] A third feature extraction unit, used to extract a spatial feature vector of the RGB pixel information;

[0117] The fourth feature extraction unit is used to extract the spatiotemporal feature vector of the RGB pixel information as the RGB feature based on the spatial feature vector of the RGB pixel information.

[0118] In an embodiment of the present application, the first feature extraction unit and the third feature extraction unit are deep residual networks Resnet18, and the second feature extraction unit and the fourth feature extraction unit are bidirectional long short-term memory artificial neural networks.

[0119] In the embodiment of the present application, the feature fusion layer performs weighted summation on the optical flow feature and the RGB feature to obtain a fusion feature;

[0120] fusion_featrue=w0*rgb_feature+w1*flow_feature

[0121] Among them, fusion_featrue represents fusion features, rgb_feature represents RGB features, flow_feature represents optical flow features, w0, w1 represent weights, w0+w1=1.

[0122] It should be noted that the cockpit control device provided in the above embodiment and the cockpit control method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the cockpit control device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0123] An embodiment of the present application also provides a device, including: one or more processors; and a memory for storing one or more programs, when the one or more programs are executed by one or more processors, the memory implements the cockpit control method in the above embodiment.

[0124] The embodiments of the present application further provide one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the cockpit control method in the above embodiments.

[0125] Figure 5 The schematic diagram of the structure of a computer system suitable for implementing the memory of the embodiment of the present application is shown. It should be noted that: Figure 5 The computer system of the memory shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0126] like Figure 5 As shown, the computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM), such as the method in the above embodiment. In the RAM, various programs and data required for system operation are also stored. The CPU, ROM and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0127] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage part as needed.

[0128] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the cockpit control method implemented above. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the system of the present application are executed.

[0129] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0130] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0132] Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the cockpit control method as described above. The computer-readable storage medium may be included in the memory described in the above embodiment, or may exist independently without being assembled into the memory.

[0133] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the cockpit control method provided in each of the above embodiments.

[0134] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A cockpit control method, characterized in that: The cockpit control method comprises: Get target video data; Performing feature extraction on the target video data to obtain optical flow information and RGB pixel information of the target video data; The optical flow information and the RGB pixel information are input into a pre-trained prediction model, and the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data are predicted by the prediction model; wherein the prediction model is obtained by training an initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the shooting angle of the sample video data or / and the wind speed of the space where the target object is located in the sample video data as output; The cockpit is controlled based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

2. The cockpit control method according to claim 1, characterized in that: The prediction model includes: a first feature extraction layer, a second feature extraction layer, a feature fusion layer, a first classification layer, and a second classification layer. The step of predicting the shooting angle of the target video data or / and the wind speed of the space where the target object in the target video data is located by using the prediction model includes: Extracting optical flow features of the target video data based on the optical flow information through the first feature extraction layer; Extracting RGB features of the target video data based on the RGB pixel information through the second feature extraction layer; Using the feature fusion layer to fuse the optical flow feature and the RGB feature to obtain a fused feature; The first classification layer is used to predict the shooting angle of the target video data based on the fusion feature, and the second classification layer is used to predict the wind speed of the space where the target object in the target video data is located based on the fusion feature.

3. The cockpit control method according to claim 1, characterized in that: The step of obtaining target video data comprises: Get the original video data; Performing frame extraction processing on the original video data to obtain a video frame sequence; The video frame sequence is segmented to obtain a plurality of video frame subsequences as target video data.

4. The cockpit control method according to claim 1, characterized in that: The first feature extraction layer includes a first feature extraction unit and a second feature extraction unit, and the step of extracting the optical flow features of the target video data based on the optical flow information through the first feature extraction layer includes: Extracting a spatial feature vector of the optical flow information by the first feature extraction unit; Based on the spatial feature vector of the optical flow information, the spatiotemporal feature vector of the optical flow information is extracted by the second feature extraction unit as an optical flow feature.

5. The cockpit control method according to claim 1 or 4, characterized in that: The second feature extraction layer includes a third feature extraction unit and a fourth feature extraction unit, and the step of extracting the RGB features of the target video data based on the RGB pixel information through the second feature extraction layer includes: Extracting a spatial feature vector of the RGB pixel information by the third feature extraction unit; Based on the spatial feature vector of the RGB pixel information, the spatiotemporal feature vector of the RGB pixel information is extracted by the fourth feature extraction unit as an RGB feature.

6. The cockpit control method according to claim 5, characterized in that: The first feature extraction unit and the third feature extraction unit are deep residual networks Resnet18, and the second feature extraction unit and the fourth feature extraction unit are bidirectional long short-term memory artificial neural networks.

7. The cockpit control method according to claim 1, characterized in that: The step of using the feature fusion layer to perform feature fusion on the optical flow feature and the RGB feature comprises: Performing weighted summation on the optical flow feature and the RGB feature to obtain a fusion feature; fusion_featrue=w0*rgb_feature+w1*flow_feature Among them, fusion_featrue represents fusion features, rgb_feature represents RGB features, flow_feature represents optical flow features, w0, w1 represent weights, w0+w1=1.

8. A cockpit control device, characterized in that: The cockpit control device comprises: A data acquisition module, used for acquiring target video data; A feature extraction module is used to extract features from the target video data to obtain optical flow information and RGB pixel information of the target video data; A prediction module, used for inputting the optical flow information and the RGB pixel information into a pre-trained prediction model, and predicting the shooting angle of the target video data or / and the wind speed of the space where the target object is located in the target video data through the prediction model; wherein the prediction model is obtained by training an initial prediction model with the optical flow information and RGB pixel information of the sample video data as input and the shooting angle of the sample video data or / and the wind speed of the space where the target object is located in the sample video data as output; The cockpit control module is used to control the cockpit based on the shooting angle of the target video data and / or the wind speed of the space where the target object in the target video data is located.

9. A cockpit control device, characterized in that: include: one or more processors; and A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the memory implements the cockpit control method according to any one of claims 1 to 7.

10. A machine-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, enable the processors to execute the cockpit control method as described in any one of claims 1-7.

Images