Acquisition method, device and equipment of motion direction and computer readable storage medium

By combining the motion prediction model and optical flow calculation method, the fusion weight fusion motion prediction model and optical flow calculation results are used to solve the problem of low efficiency in the prior art motion state acquisition, and more efficient and accurate motion direction acquisition is achieved.

CN119006532APending Publication Date: 2024-11-22CHONGQING JINKANG NEW ENERGY VEHICLE CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410963413.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-22

Smart Images

  • Figure CN119006532A_ABST
    Figure CN119006532A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a motion direction acquisition method and device, equipment and a storage medium, and the method comprises the steps: inputting target video data into a motion prediction model for prediction, and obtaining a target detection frame, a target object type and a first motion direction; performing optical flow calculation according to the target detection frame to obtain a second motion direction; and fusing the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category. By means of the technical scheme, the problem that in the prior art, due to the fact that the calculated amount is large, the motion state obtaining efficiency is low can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and computer-readable storage medium for acquiring a motion direction. Background Art

[0002] At present, when identifying the direction of movement of an object, the following methods are used: derive the spatial, temporal and conformal constraints of a stable video based on the subjective visual perception of human beings; use the optical flow method to obtain the motion trajectory of feature points in the video; establish a sparse linear equation system based on the above constraints and the motion trajectory of feature points, and solve them to obtain the grid points of each frame of the stable video; distort the image according to the position of the grid points before and after the stabilization of each video frame to obtain a stable video image. The above method of obtaining the motion state simply uses the calculation of the dense optical flow field to obtain the motion trajectory, but this method requires dense optical flow calculation of the offset of each pixel point, which will result in a significantly larger amount of calculation.

[0003] It can be seen that the motion state acquisition method in the prior art has the problem of high computational complexity leading to low motion state acquisition efficiency. Summary of the invention

[0004] In view of the above problems, the present application provides a method, device, equipment and computer-readable storage medium for obtaining a motion direction, which are used to solve the problem in the prior art that a large amount of calculation leads to low efficiency in obtaining the motion state.

[0005] According to a first aspect of an embodiment of the present application, a method for obtaining a motion direction is provided, the method comprising: inputting target video data into a motion prediction model for prediction to obtain a target detection frame, a target object category and a first motion direction; performing optical flow calculation based on the target detection frame to obtain a second motion direction; and fusing the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0006] In an optional manner, the first motion direction and the second motion direction are fused based on a fusion weight to obtain the target motion direction corresponding to the target object category, further including: normalizing the first motion direction and the second motion direction respectively to obtain a first motion probability vector and a second motion probability vector; summing the first motion probability vector and the second motion probability vector based on the fusion weight to obtain a motion direction probability vector; and obtaining the target motion direction corresponding to the target object category according to the maximum probability index in the motion direction probability vector.

[0007] In an optional manner, before inputting the target video data into the motion prediction model for prediction to obtain the target detection frame, the target object category and the first motion direction, it further includes: extracting frames from the acquired training video data to obtain multiple training images; annotating the multiple training images respectively to obtain the training detection frame, the training object category and the motion direction of the training object; encoding each of the annotated training images to obtain multiple first training image vectors; performing feature extraction on each of the first training image vectors to obtain multiple training image feature vectors; and performing model training based on each of the training image feature vectors, the training detection frame, the training object category and the motion direction of the training object to obtain the motion prediction model.

[0008] In an optional manner, encoding each of the labeled training images to obtain a plurality of first training image vectors further includes: dividing each of the training images into a plurality of sub-images; linearly encoding each of the sub-images to obtain a plurality of first sub-image vectors; encoding the position of each sub-image to obtain a plurality of position vectors of the sub-images; fusing each of the first sub-image vectors with the position vector corresponding to the first sub-image vector to obtain a plurality of second sub-image vectors; wherein each of the first training image vectors includes the plurality of second sub-image vectors.

[0009] In an optional manner, the motion prediction model includes multiple feature extraction encoders, each of which includes: a self-attention layer, a feedforward network layer and a normalization layer; the feature extraction of each of the first training image vectors to obtain multiple training image feature vectors further includes: inputting each of the first training image vectors into multiple feature extraction encoders respectively; using the self-attention layer to perform feature extraction on the first training image vector to obtain a second training image vector; using the feedforward network layer to perform feature extraction on the second training image vector to obtain a third training image vector; using the normalization layer to perform layer normalization on the third training image vector to obtain a fourth training image vector; fusing the fourth training image vectors output by each of the feature extraction encoders to obtain each of the training image feature vectors.

[0010] In an optional manner, the motion prediction model includes: a detection frame linear layer, a category linear layer and a motion direction linear layer; the model training is performed based on the training image feature vector, the training detection frame, the training object category and the motion direction of the training object to obtain the motion prediction model, further including: optimizing the parameters of the detection frame linear layer based on the training image feature vector and the training detection frame; optimizing the parameters of the category linear layer based on the training image feature vector and the training object category; optimizing the parameters of the motion direction linear layer based on the training image feature vector and the motion direction of the training object.

[0011] In an optional manner, the separately labeling of the multiple training images to obtain training detection frames, training object categories, and the movement direction of the training objects further includes: placing each of the multiple training images in the same interface window as the training video data in sequence according to the video playback order; and labeling the current training image with the training detection frame, training object category, and the movement direction of the training object based on the target object and the actual movement direction of the target object in the training video data being played.

[0012] According to a second aspect of an embodiment of the present application, a device for acquiring a motion direction is provided, the device comprising: a model prediction module, used to input target video data into a motion prediction model for prediction to obtain a target detection frame, a target object category and a first motion direction; an optical flow calculation module, used to perform optical flow calculation based on the target detection frame to obtain a second motion direction; and a fusion module, used to fuse the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a controller; a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the controller implements the following operations of the method for obtaining the motion direction: inputting the target video data into a motion prediction model for prediction to obtain a target detection frame, a target object category and a first motion direction; performing optical flow calculation based on the target detection frame to obtain a second motion direction; and fusing the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0014] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on a motion direction acquisition device / electronic device, the motion direction acquisition device / electronic device performs the following operations of the motion direction acquisition method: inputting target video data into a motion prediction model for prediction to obtain a target detection frame, a target object category and a first motion direction; performing optical flow calculation based on the target detection frame to obtain a second motion direction; and fusing the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0015] The embodiment of the present application inputs the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction, so that the target detection frame captured from the image can be subjected to optical flow calculation again. There is no need to perform optical flow calculation on the entire image, which can improve the computational efficiency of the optical flow calculation. Moreover, the pixel points in the target detection frame are already moving objects, and the second motion direction obtained by the optical flow calculation is more accurate. By setting the fusion weight to determine the credibility of the first motion direction obtained by the prediction model and the second motion direction obtained by the optical flow calculation, the first motion direction and the second motion direction are fused according to the fusion weight to obtain the target motion direction, so that the target motion direction of the target object obtained is more accurate.

[0016] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0018] Figure 1 A flow chart of a first embodiment of a method for acquiring a motion direction provided by the present application is shown;

[0019] Figure 2 A flow chart of a second embodiment of a method for acquiring a motion direction provided by the present application is shown;

[0020] Figure 3 A flowchart of a third embodiment of a method for acquiring a motion direction provided by the present application is shown;

[0021] Figure 4 A flowchart of a fourth embodiment of a method for acquiring a motion direction provided by the present application is shown;

[0022] Figure 5 A flowchart of a fifth embodiment of a method for acquiring a motion direction provided by the present application is shown;

[0023] Figure 6 A schematic structural diagram of an embodiment of a motion direction acquisition device provided by the present application is shown;

[0024] Figure 7 A schematic structural diagram of an embodiment of an electronic device provided by the present application is shown. DETAILED DESCRIPTION

[0025] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0028] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0029] Figure 1 The flowchart of the first embodiment of a method for obtaining a motion direction provided by the present application is shown, and the method is executed by an electronic device. Figure 1 As shown, the method comprises the following steps:

[0030] Step S110: inputting the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction;

[0031] Among them, after the target video data is input into the motion prediction model, multiple images are extracted, and the target detection frame in each image can be predicted, as well as the target object in the target detection frame. For example, the target detection frame may contain other objects such as people and cars, and the movement direction of each target object is obtained. For example, the first movement direction may be left, right, up, and down, etc.

[0032] Step S120: performing optical flow calculation according to the target detection frame to obtain a second motion direction;

[0033] Among them, by performing optical flow calculation on the target detection frame captured from the image, there is no need to perform optical flow calculation on the entire image, which can improve the computational efficiency of the optical flow calculation. Moreover, the pixels in the target detection frame are already moving objects, and the second motion direction obtained through the optical flow calculation is more accurate.

[0034] Step S130: fusing the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0035] Among them, there is a certain difference between the first motion direction obtained by the motion prediction model and the second motion direction obtained by the optical flow calculation. The credibility of the first motion direction obtained by the prediction model and the second motion direction obtained by the optical flow calculation is determined by setting the fusion weight, and the first motion direction and the second motion direction are fused according to the fusion weight to obtain the target motion direction, so that the target motion direction of the target object is more accurate.

[0036] By inputting the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction, so that the target detection frame captured from the image can be subjected to optical flow calculation, there is no need to perform optical flow calculation on the entire image, which can improve the computational efficiency of the optical flow calculation, and the pixel points in the target detection frame are already moving objects, and the second motion direction obtained by the optical flow calculation is more accurate; by setting the fusion weight to determine the credibility of the first motion direction obtained by the prediction model and the second motion direction obtained by the optical flow calculation, the first motion direction and the second motion direction are fused according to the fusion weight to obtain the target motion direction, so that the target motion direction of the target object obtained is more accurate.

[0037] Figure 2 The flowchart of the second embodiment of a method for obtaining a motion direction provided by the present application is shown, and the method is executed by an electronic device. Figure 2 As shown, the first motion direction and the second motion direction are fused based on the fusion weight to obtain the target motion direction corresponding to the target object category, further comprising the following steps:

[0038] Step S210: Normalizing the first motion direction and the second motion direction respectively to obtain a first motion probability vector and a second motion probability vector;

[0039] Among them, the first motion direction obtained by the motion prediction model is a numerical vector, and the second motion direction obtained by the optical flow calculation is a numerical vector; normalizing the first motion direction can obtain a first motion probability vector, and normalizing the second motion direction can obtain a second motion probability vector, that is, performing normalization processing (for example, using a softmax function) on the first motion direction and the second motion direction respectively can obtain the first motion probability vector and the second motion probability vector, and by obtaining the first motion probability vector and the second motion probability vector, the target motion direction can be calculated based on the fusion weight.

[0040] Step S220: summing the first motion probability vector and the second motion probability vector based on the fusion weight to obtain a motion direction probability vector;

[0041] The fusion weight may be calculated according to specific circumstances, or may be set based on experience. For example, the weight of the first motion probability vector may be set to 0.8, and the weight of the second motion probability vector may be set to 0.2.

[0042] Movement direction probability vector = 0.8*first movement probability vector + 0.2*second movement probability vector

[0043] Step S230: obtaining the target motion direction corresponding to the target object category according to the maximum probability index in the motion direction probability vector.

[0044] Among them, when the maximum probability in the motion direction probability vector represents the motion direction of the target object. For example, the first motion direction is predicted by the motion prediction model as [1 -1-2 2], and the first motion direction [1 -1-2 2] is subjected to a softmax operation to obtain a first motion probability vector [0.25 0.03 0.02 0.7], which respectively represent the probabilities of up, down, left, and right, that is, the probability of the upward direction is 0.25, the probability of the downward direction is 0.03, the probability of the left direction is 0.02, and the probability of the right direction is 0.7. The second motion direction is obtained by optical flow calculation as [4 2 0 3], and the second motion direction [4 2 0 3] is subjected to a softmax operation to obtain a second motion probability vector [0.65 0.08 0.02 0.24], which respectively represent the probabilities of up, down, left, and right, that is, the probability of the upward direction is 0.65, the probability of the downward direction is 0.08, the probability of the left direction is 0.02, and the probability of the right direction is 0.24.

[0045] Movement direction probability vector = 0.8*[0.25 0.03 0.02 0.7]+0.2*[0.65 0.08 0.02 0.24]

[0046] =[0.2 0.024 0.016 0.56]+[0.13 0.016 0.004 0.048]

[0047] =[0.33 0.04 0.02 0.06]

[0048] That is, in the motion direction probability vector, the probability of the upward direction is 0.33, the probability of the downward direction is 0.04, the probability of the left direction is 0.02, and the probability of the right direction is 0.06. From the above probability distribution, it can be seen that the maximum probability of the downward direction is 0.6, indicating that the target motion direction of the target object is downward.

[0049] Figure 3 The flowchart of the third embodiment of a method for obtaining a motion direction provided by the present application is shown, and the method is executed by an electronic device. Figure 3 As shown, before inputting the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction, the method further includes:

[0050] Step S310: extracting frames from the acquired training video data to obtain a plurality of training images;

[0051] Among them, the training video data can be film materials, including various movies, TV series, and landscape scenes, to judge the motion state of specific target objects in these materials relative to the picture. In the data preprocessing stage, in view of the subsequent use of deep learning supervised training strategies, the training video data is framed to obtain multiple training images, so that a single training image can be put into the motion prediction model for training.

[0052] Step S320: annotating the plurality of training images respectively to obtain training detection frames, training object categories and movement directions of the training objects;

[0053] Specifically, step S320 further includes:

[0054] Step S321: placing each of the plurality of training images in the same interface window with the training video data in sequence according to the video playback order;

[0055] For example, in the same interface window, the current training image can be placed above the same interface window, and the training video data can be placed below the same interface window. The placement positions of the current training image and the training video data are not limited here, and can be any position in the same interface window. The training images are placed in the same interface window in sequence according to the order of video playback, and the movement direction of the current training image can be accurately obtained.

[0056] Step S322: annotating the current training image with the training detection frame, the training object category and the movement direction of the training object according to the target object and the actual movement direction of the target object in the played training video data.

[0057] Among them, placing the played training video data and the current training image in the same interface window can facilitate the user to obtain the movement direction of the target object in the training image according to the played training video data, and can further improve the accuracy and efficiency of annotation.

[0058] Step S330: Encode each of the labeled training images to obtain a plurality of first training image vectors;

[0059] Wherein, each labeled training image is encoded to obtain a plurality of first training image vectors, so that the motion prediction model can recognize the training image and perform corresponding calculations.

[0060] Step S340: extracting features from each of the first training image vectors to obtain a plurality of training image feature vectors;

[0061] By performing feature extraction on each of the first training image vectors, the motion prediction model can be trained, and the training image feature vector of each first training image can be extracted, thereby further reducing the amount of calculation.

[0062] Step S350: performing model training based on each of the training image feature vectors, the training detection box, the training object category and the motion direction of the training object to obtain the motion prediction model.

[0063] Among them, model training is performed through the training image feature vector, the annotated training detection frame, the training object category and the movement direction of the training object, so that the motion model can learn that the input training image feature vector corresponds to the training detection frame, the training object category and the movement direction of the training object, and subsequently the target video data is input to the motion prediction model, and the motion prediction model can output the corresponding target detection frame, target object category and first movement direction.

[0064] Figure 4FIG. 4 is a flowchart of a fourth embodiment of a method for obtaining a motion direction provided by the present application, the method being executed by an electronic device. Figure 4 As shown, encoding each of the labeled training images to obtain a plurality of first training image vectors further includes:

[0065] Step S410: dividing each of the training images into a plurality of sub-images;

[0066] Each of the training images will be divided into multiple non-overlapping sub-images, which are called image patches. A 224x224 pixel image can be divided into 16 14x14 pixel patches. The specific size of the sub-image can be set according to the size of the actual target object. The size of each sub-image can also be different. The size and number of each training sub-image are not limited here.

[0067] Step S420: linearly encode each of the sub-images to obtain a plurality of first sub-image vectors;

[0068] Each sub-image is converted into a first sub-image vector so that recognition calculation can be performed on each sub-image. Specifically, each sub-image is converted into a vector of fixed dimension, namely, the first sub-image vector, through a fully connected layer.

[0069] Step S430: obtaining position vectors of a plurality of sub-images according to the position code of each sub-image;

[0070] The position encoding of each sub-image obtains position vectors of multiple sub-images, so that the position of each sub-image can be obtained, so that the model can learn the position information of the sequence between each sub-image.

[0071] Step S440: fusing each of the first sub-image vectors and the position vector corresponding to the first sub-image vector to obtain a plurality of second sub-image vectors;

[0072] Each of the first training image vectors includes the plurality of second sub-image vectors. Each first sub-image vector and the position vector corresponding to the first sub-image vector are added to obtain the plurality of second sub-image vectors, so as to represent the original shallow features and positions of each sub-image in the first training image vector.

[0073] Figure 5 FIG. 5 is a flowchart of a fifth embodiment of a method for obtaining a motion direction provided by the present application, the method being executed by an electronic device. Figure 5As shown, the motion prediction model includes multiple feature extraction encoders, each of which includes: a self-attention layer, a feedforward network layer and a normalization layer; the feature extraction of each of the first training image vectors to obtain multiple training image feature vectors further includes:

[0074] Step S510: input each of the first training image vectors into a plurality of feature extraction encoders respectively;

[0075] Wherein, the multiple motion prediction models include multiple feature extraction codes. Since the initial value of each feature encoder is different, each of the first training image vectors is input into multiple feature extraction encoders respectively, and the extracted features may be different.

[0076] Step S520: extracting features from the first training image vector using a self-attention layer to obtain a second training image vector;

[0077] Specifically, the self-attention layer calculates its Q, K, and V vectors, which is usually achieved by multiplying the input vector with three different weight matrices, which can be expressed as follows:

[0078] Q=X·W Q

[0079] K=X·W K

[0080] V=X·W V

[0081] Wherein, X represents the first training image vector;

[0082] W Q represents the learning weight of the Q vector;

[0083] W K represents the learning weight of K vector;

[0084] W V Represents the learned weight of the V vector.

[0085] Calculate the attention score of each element in the first training image vector and all other elements. The attention score can be calculated according to the Q vector and the K vector, and the self-attention score is converted into a probability distribution through a softmax operation, so that the attention contribution of each element to all other elements is non-negative, and the sum is 1; the self-attention score (Attention(Q,K,V)) can be obtained by the following calculation formula:

[0086]

[0087] Where, Q represents the Q vector;

[0088] K T represents the transpose of the K vector;

[0089] d k Represents the dimension of the K vector;

[0090] The calculated attention score and V vector are weightedly summed to obtain the final output vector, i.e., the second training image vector.

[0091] Step S530: extracting features from the second training image vector using a feedforward network layer to obtain a third training image vector;

[0092] Among them, the second training image vector obtained is further transformed nonlinearly using a feedforward network layer to continue to extract the contour of the target object in the image, such as the contours, points and other features of people and cars, based on the learned global feature information, that is, the second training image vector.

[0093] Step S540: using a normalization layer to perform layer normalization on the third training image vector to obtain a fourth training image vector;

[0094] Among them, the normalization layer does not depend on the batch size of the processed images and can effectively process sequence data. Layer normalization is performed on the output of the self-attention layer and the feedforward network layer to stabilize the training process and reduce the sensitivity to the initialization weights.

[0095] Step S550: fusing the fourth training image vectors output by each of the feature extraction encoders to obtain each of the training image feature vectors.

[0096] Among them, since the features extracted by each feature extraction encoder may be different, fusing the fourth training image vector output by each feature extraction encoder can make the extracted features more comprehensive and the final training image feature vector more accurate, thereby making the motion direction finally obtained by the prediction model more accurate.

[0097] In another embodiment of the present application, the motion prediction model includes: a detection frame linear layer, a category linear layer and a motion direction linear layer; the model training is performed based on the training image feature vector, the training detection frame, the training object category and the motion direction of the training object to obtain the motion prediction model, further including:

[0098] Optimizing the parameters of the linear layer of the detection frame based on the training image feature vector and the training detection frame;

[0099] Among them, by inputting the training image feature vector into the detection frame linear layer, the detection frame linear layer can learn according to the labeled training detection frame and training image feature vector, and optimize the parameters of the detection frame linear layer. After the model is trained, the image feature vector obtained from the target image is input into the detection frame line performance to predict the center coordinates of the target detection frame and the width and height of the target detection frame.

[0100] Optimizing the parameters of the category linear layer based on the training image feature vector and the training object category;

[0101] Among them, by inputting the training image feature vector into the category linear layer, the category linear layer can optimize the parameters of the category linear layer according to the labeled training object and the training image feature vector. After the model is trained, the image feature vector obtained from the target image is input into the category linear layer to predict the category of the target object, such as people and cars.

[0102] Optimizing parameters of the motion direction linear layer based on the training image feature vector and the motion direction of the training object.

[0103] Among them, by inputting the training image feature vector into the motion direction linear layer, the motion direction linear layer is made to learn according to the marked motion direction of the training object, and the parameters of the motion direction linear layer are optimized. After the model is trained, the image feature vector obtained from the target image is input into the motion direction linear layer to predict whether the motion direction of the target object is up, down, left or right.

[0104] It should be noted that after obtaining the movement direction of the target object, a dropblock2D layer will be used to delete the random feature information in the feature according to the global feature information to improve the robustness of the model. Then, after a linear transformation, the feature dimension is reduced to the number of direction categories, which can be 4 dimensions. Then, a softmax operation is performed on this vector to obtain the probability information. The index with the largest probability is selected as the target movement direction category label.

[0105] Figure 6 The schematic diagram of the structure of an embodiment of a motion direction acquisition device provided by the present application is shown. Figure 6 As shown, the device 600 includes: a model prediction module 610, an optical flow calculation module 620 and a fusion module 630.

[0106] The model prediction module 610 is used to input the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction;

[0107] An optical flow calculation module 620, configured to perform optical flow calculation according to the target detection frame to obtain a second motion direction;

[0108] The fusion module 630 is used to fuse the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

[0109] It should be noted that the device for obtaining the motion direction provided in the above embodiment and the method for obtaining the motion direction provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0110] Figure 7 A schematic diagram of the structure of an embodiment of the electronic device provided by the present application is shown, which shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0111] See also Figure 7 As shown, the electronic device includes: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned method for obtaining the motion direction is executed.

[0112] Please continue reading Figure 7 As shown, the computer system 700 of the electronic device includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method in the above embodiment. In RAM 703, various programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other through bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0113] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0114] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0115] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the method for obtaining the motion direction as described above when the computer program is executed by a processor. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0116] Another aspect of the present application also provides a computer program product or a computer program, which includes at least one executable instruction. When the executable instruction is executed on a motion direction acquisition device / electronic device, the motion direction acquisition device / electronic device executes the motion direction acquisition method as described above.

[0117] The executable instructions may be specifically used to enable the motion direction acquisition device / electronic device to perform the following operations:

[0118] Inputting the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction;

[0119] Performing optical flow calculation according to the target detection frame to obtain a second motion direction;

[0120] The first motion direction and the second motion direction are fused based on the fusion weight to obtain a target motion direction corresponding to the target object category.

[0121] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0122] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0124] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0125] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0126] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for obtaining a motion direction, It is characterized in that The method comprises: Inputting the target video data into the motion prediction model to predict the target detection frame, the target object category and the first motion direction; Performing optical flow calculation according to the target detection frame to obtain a second motion direction; The first motion direction and the second motion direction are fused based on the fusion weight to obtain a target motion direction corresponding to the target object category.

2. A method for obtaining a motion direction according to claim 1, It is characterized in that The fusing the first motion direction and the second motion direction based on the fusion weight to obtain the target motion direction corresponding to the target object category further includes: Normalizing the first motion direction and the second motion direction respectively to obtain a first motion probability vector and a second motion probability vector; Summing the first motion probability vector and the second motion probability vector based on the fusion weight to obtain a motion direction probability vector; The target motion direction corresponding to the target object category is obtained according to the maximum probability index in the motion direction probability vector.

3. The method according to claim 2, It is characterized in that Before inputting the target video data into the motion prediction model to predict and obtain the target detection frame, the target object category and the first motion direction, the method further includes: Extract frames from the acquired training video data to obtain multiple training images; Annotating the plurality of training images respectively to obtain training detection frames, training object categories and movement directions of the training objects; Encoding each of the labeled training images to obtain a plurality of first training image vectors; Performing feature extraction on each of the first training image vectors to obtain a plurality of training image feature vectors; Model training is performed based on each of the training image feature vectors, the training detection frame, the training object category and the motion direction of the training object to obtain the motion prediction model.

4. The method according to claim 3, It is characterized in that The step of encoding each of the labeled training images to obtain a plurality of first training image vectors further comprises: Splitting each of the training images into a plurality of sub-images; Linearly encode each of the sub-images to obtain a plurality of first sub-image vectors; Obtaining position vectors of a plurality of sub-images according to the position coding of each sub-image; Each of the first sub-image vectors and the position vector corresponding to the first sub-image vector are fused to obtain a plurality of second sub-image vectors; wherein each of the first training image vectors includes the plurality of second sub-image vectors.

5. The method according to claim 3, It is characterized in that The motion prediction model includes a plurality of feature extraction encoders, each of which includes: a self-attention layer, a feedforward network layer, and a normalization layer; the feature extraction of each of the first training image vectors to obtain a plurality of training image feature vectors further includes: Input each of the first training image vectors into a plurality of feature extraction encoders respectively; Using a self-attention layer to extract features from the first training image vector to obtain a second training image vector; Using a feed-forward network layer to perform feature extraction on the second training image vector to obtain a third training image vector; Using a normalization layer to perform layer normalization on the third training image vector to obtain a fourth training image vector; The fourth training image vectors output by each of the feature extraction encoders are fused to obtain each of the training image feature vectors.

6. The method according to claim 3, It is characterized in that The motion prediction model comprises: a detection frame linear layer, a category linear layer and a motion direction linear layer; the model training is performed based on the training image feature vector, the training detection frame, the training object category and the motion direction of the training object to obtain the motion prediction model, further comprising: Optimizing the parameters of the linear layer of the detection frame based on the training image feature vector and the training detection frame; Optimizing the parameters of the category linear layer based on the training image feature vector and the training object category; Optimizing parameters of the motion direction linear layer based on the training image feature vector and the motion direction of the training object.

7. The method according to claim 6, It is characterized in that The step of respectively labeling the plurality of training images to obtain training detection frames, training object categories, and movement directions of the training objects further includes: Placing each of the plurality of training images in the same interface window with the training video data in sequence according to the video playback order; According to the target object and the actual movement direction of the target object in the played training video data, the training detection frame, the training object category and the movement direction of the training object are annotated on the current training image.

8. A device for obtaining a movement direction, It is characterized in that The device comprises: A model prediction module, used for inputting the target video data into the motion prediction model to predict and obtain a target detection frame, a target object category and a first motion direction; An optical flow calculation module, used to perform optical flow calculation according to the target detection frame to obtain a second motion direction; A fusion module is used to fuse the first motion direction and the second motion direction based on a fusion weight to obtain a target motion direction corresponding to the target object category.

9. An electronic device, It is characterized in that include: Controller; A memory is used to store one or more programs. When the one or more programs are executed by the controller, the controller implements the operation of the method for obtaining the motion direction described in any one of claims 1-7.

10. A computer-readable storage medium, It is characterized in that The storage medium stores at least one executable instruction. When the executable instruction is executed on the motion direction acquisition device / electronic device, the motion direction acquisition device / electronic device performs the operation of the motion direction acquisition method as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Motion direction prediction method and device, electronic equipment and storage medium

    CN109697387A

  • Lightweight deep neural network rotating target detection method and system

    CN111242122A

  • Fine-grained industrial motion mode classification method, storage medium, equipment and device

    CN112489092A

  • Vehicle recognition method and device, computer equipment and storage medium

    CN113763717A

  • Motion direction prediction method, electronic equipment and storage medium

    CN116883453A