Method and device for extracting motion features from video frames

Abstract

The present invention proposes a method and device for extracting motion features from video frames, wherein the method includes: obtaining a feature map of each frame in the video data, and determining, for any target frame, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to a set spatial coordinate of a moving object displayed in the target frame and a second feature corresponding to a set-sized neighborhood of the moving object displayed in the next frame, thereby determining, based on the first feature and the second feature, an aggregate feature of the neighborhood in the next frame, and determining the motion features in the target frame based on the aggregate feature. Since the aggregate feature is determined based on the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, compared to the prior art that only captures the correlation between corresponding coordinate points in adjacent frames, the present invention expands the acquisition area and improves the accuracy of the motion features of the video frame.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for extracting motion features from video frames. Background Art

[0002] With the increasing demand for video capture, a vast amount of video data has been accumulated across a wide range of scenarios. Consequently, video understanding—the automated identification and analysis of video content—is essential. Action recognition is a core area of ​​video understanding, used to identify actions in videos, typically those of people, though it can also be the actions of objects other than the human body.

[0003] Related technologies often assume that the actions between consecutive video frames are well aligned spatially. Feature aggregation is performed based on actions corresponding to the same position in adjacent frames, and action features are then collected based on the aggregated features. However, this assumption of good spatial alignment between consecutive video frames is only applicable to a limited range of situations. For situations with large object motion speeds and large deformations, the collection of video action features may be inaccurate. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first purpose of the present invention is to propose a method for extracting motion features from video frames, so as to determine the aggregated features of the neighborhood in the next frame based on a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size of the neighborhood of the moving object displayed in the next frame, determine the motion features in the target frame based on the aggregated features, and improve the accuracy of the motion features of the video frames.

[0006] The second object of the present invention is to provide a device for extracting motion features from video frames.

[0007] A third objective of the present invention is to provide an electronic device.

[0008] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium.

[0009] A fifth object of the present invention is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, a first embodiment of the present invention provides a method for extracting motion features from a video frame, comprising:

[0011] Obtain feature maps of each frame in the video data;

[0012] For any target frame, determine, based on a feature map of the target frame and a feature map of a next frame, a first feature corresponding to a set spatial coordinate of a moving object displayed in the target frame and a second feature corresponding to a neighborhood of a set size of the moving object displayed in the next frame; the first feature is used to indicate the moving object displayed at the corresponding set spatial coordinate in the target frame; the second feature is used to indicate the moving object displayed in the neighborhood of the same set spatial coordinate in the next frame;

[0013] Determining, based on the first feature and the second feature, an aggregate feature of the neighborhood in a next frame; wherein the aggregate feature is used to indicate a correlation between a moving object displayed by a set spatial coordinate corresponding to the first feature and a moving object displayed in a neighborhood corresponding to the second feature;

[0014] The first feature and the aggregated feature are fused to obtain the action feature in the target frame.

[0015] Optionally, as a first possible implementation manner of the first aspect, determining, for the arbitrary target frame, based on the feature map of the target frame and the feature map of the next frame, a second feature corresponding to a neighborhood of a set size showing the moving object in the next frame includes:

[0016] For any target frame, determining a motion saliency map based on the feature map of the target frame and the feature map of the next frame;

[0017] For the motion saliency map, using a coordinate offset estimator to predict the offset of features of each spatial coordinate within the neighborhood of the set size in the next frame, so as to determine a plurality of sampling coordinates within the neighborhood according to the offset;

[0018] Using bilinear interpolation method, determine the sampling features corresponding to each sampling coordinate;

[0019] According to each of the sampling features, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame is determined.

[0020] Optionally, as a second possible implementation manner of the first aspect, determining, for any target frame, a motion saliency map based on a feature map of the target frame and a feature map of the next frame includes:

[0021] For any target frame, determining an inter-frame feature difference based on a feature map of the target frame and a feature map of the next frame;

[0022] Normalizing the inter-frame feature differences using an activation function to obtain a corresponding attention map; wherein the attention map is used to indicate the spatial position of a moving object that moves between two frames;

[0023] The attention map is multiplied with the feature map of the next frame to obtain a motion saliency map.

[0024] Optionally, as a third possible implementation manner of the first aspect, determining the aggregate feature of the neighborhood in the next frame based on the first feature and the second feature includes:

[0025] Performing a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix; wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in a neighborhood corresponding to the first feature and the second feature;

[0026] The similarity matrix is ​​used as a weight to perform a matrix multiplication operation with the transposed matrix of the second feature to obtain the aggregated features of the neighborhood in the next frame.

[0027] Optionally, as a fourth possible implementation manner of the first aspect, the method further includes:

[0028] For any target frame, a two-dimensional convolutional neural network or a visual feature extraction Transformer network is used to determine the feature map of the target frame and the feature map of the next frame.

[0029] Optionally, as a fifth possible implementation manner of the first aspect, determining the feature map of the target frame and the feature map of the next frame using a two-dimensional convolutional neural network and / or a visual feature extraction Transformer network includes:

[0030] Determine a feature map of the target frame and the next frame using a convolution kernel in a basic residual module of a two-dimensional convolutional neural network; or

[0031] The MSA module with a regular window configuration in the Transformer network is used to determine feature maps for the target frame and the next frame.

[0032] The method for extracting motion features from video frames provided by an embodiment of the present invention obtains a feature map of each frame in the video data, and thereby determines, for any target frame, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size of the neighborhood of the moving object displayed in the next frame, thereby determining the aggregated features of the neighborhood in the next frame based on the first and second features, and determining the motion features in the target frame based on the aggregated features. Since the aggregated features are determined based on the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, compared to the prior art that only captures the correlation between corresponding coordinate points in adjacent frames, the present invention expands the acquisition area and improves the accuracy of the motion features of the video frame.

[0033] To achieve the above-mentioned purpose, a second embodiment of the present invention provides a motion feature extraction device for a video frame, comprising:

[0034] An acquisition module, used to obtain feature maps of each frame in the video data;

[0035] A first determination module is configured to determine, for any target frame, based on a feature map of the target frame and a feature map of a next frame, a first feature corresponding to a set spatial coordinate of a moving object displayed in the target frame and a second feature corresponding to a neighborhood of a set size of the moving object displayed in the next frame; the first feature is configured to indicate the moving object displayed at the corresponding set spatial coordinate in the target frame; and the second feature is configured to indicate the moving object displayed in the neighborhood of the same set spatial coordinate in the next frame;

[0036] a second determining module, configured to determine, based on the first feature and the second feature, an aggregate feature of the neighborhood in a next frame; wherein the aggregate feature is used to indicate a correlation between a moving object represented by a set spatial coordinate corresponding to the first feature and a moving object represented in a neighborhood corresponding to the second feature;

[0037] A fusion module is used to fuse the first feature and the aggregated feature to obtain the action feature in the target frame.

[0038] Optionally, as a first possible implementation manner of the second aspect, the first determining module includes:

[0039] a first determining unit, configured to determine, for the arbitrary target frame, a motion saliency map based on a feature map of the target frame and a feature map of the next frame;

[0040] a prediction unit configured to predict, based on the motion saliency map, an offset of features of each spatial coordinate within the neighborhood of the set size in the next frame using a coordinate offset estimator, so as to determine a plurality of sampling coordinates within the neighborhood according to the offset;

[0041] A second determining unit is configured to determine a sampling feature corresponding to each sampling coordinate by using a bilinear interpolation method;

[0042] The third determining unit is configured to determine, based on each of the sampling features, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame.

[0043] Optionally, as a second possible implementation manner of the second aspect, the first determining unit is further configured to:

[0044] For any target frame, determining an inter-frame feature difference based on a feature map of the target frame and a feature map of the next frame;

[0045] Normalizing the inter-frame feature differences using an activation function to obtain a corresponding attention map; wherein the attention map is used to indicate the spatial position of a moving object that moves between two frames;

[0046] The attention map is multiplied with the feature map of the next frame to obtain a motion saliency map.

[0047] Optionally, as a third possible implementation manner of the second aspect, the second determining module includes:

[0048] a first processing unit, configured to perform a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix; wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in a neighborhood corresponding to the first feature and the second feature;

[0049] The second processing unit is configured to perform a matrix multiplication operation on the transposed matrix of the second feature by using the similarity matrix as a weight to obtain an aggregated feature of the neighborhood in a next frame.

[0050] Optionally, as a fourth possible implementation manner of the second aspect, the apparatus further includes:

[0051] The third determination module is used to determine the feature map of the target frame and the feature map of the next frame using a two-dimensional convolutional neural network or a visual feature extraction Transformer network for any target frame.

[0052] Optionally, as a fifth possible implementation manner of the second aspect, the third determining module is further configured to:

[0053] Determine a feature map of the target frame and the next frame using a convolution kernel in a basic residual module of a two-dimensional convolutional neural network; or

[0054] The MSA module with a regular window configuration in the Transformer network is used to determine feature maps for the target frame and the next frame.

[0055] The motion feature extraction device for video frames provided by an embodiment of the present invention obtains a feature map of each frame in the video data, and thereby determines, for any target frame, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size of the neighborhood of the moving object displayed in the next frame, thereby determining the aggregated features of the neighborhood in the next frame based on the first and second features, and determining the motion features in the target frame based on the aggregated features. Since the aggregated features are determined based on the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, compared to the prior art that only captures the correlation between corresponding coordinate points in adjacent frames, the present invention expands the acquisition area and improves the accuracy of the motion features of the video frame.

[0056] To achieve the above-mentioned object, a third embodiment of the present invention provides an electronic device, comprising:

[0057] at least one processor; and

[0058] a memory communicatively connected to the at least one processor; wherein,

[0059] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0060] In order to achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0061] In order to achieve the above-mentioned purpose, a fifth embodiment of the present invention proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0062] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0064] Figure 1 A schematic flow chart of a method for extracting motion features from a video frame provided by an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the principles of a time-domain convolution solution, a time-domain self-attention solution, and the technical solution provided in an embodiment of the present invention;

[0066] Figure 3 A schematic flow chart of another method for extracting motion features from video frames provided by an embodiment of the present invention;

[0067] Figure 4 A schematic diagram of a process for determining a motion saliency map provided by an embodiment of the present invention;

[0068] Figure 5 A schematic flow chart of another method for extracting motion features from video frames provided by an embodiment of the present invention;

[0069] Figure 6 A schematic diagram illustrating the principle of a method for extracting motion features from a video frame provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram illustrating the principle of another method for extracting motion features from video frames provided by an embodiment of the present invention;

[0071] Figure 8 A schematic flow chart of another method for extracting motion features from video frames provided by an embodiment of the present invention;

[0072] Figure 9 A schematic diagram of an application of a method for extracting motion features from video frames provided by an embodiment of the present invention;

[0073] Figure 10 A schematic structural diagram of a device for extracting motion features from video frames provided by an embodiment of the present invention;

[0074] Figure 11 A schematic structural diagram of another device for extracting motion features from video frames provided by an embodiment of the present invention; and

[0075] Figure 12 A schematic structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0077] The following describes a method, apparatus, electronic device, storage medium, and computer program product for extracting motion features from video frames according to embodiments of the present invention with reference to the accompanying drawings.

[0078] Figure 1 A flowchart of a method for extracting motion features from video frames provided by an embodiment of the present invention.

[0079] In related technologies, with the continuous development of deep learning, the following two schemes are mainly used to extract motion features from video frames: one is the time domain convolution scheme, which uses spatiotemporal three-dimensional convolution or decomposed spatiotemporal three-dimensional convolution (spatial convolution + temporal convolution) to capture the motion, and the other is the time domain self-attention scheme, which uses the self-attention mechanism in the time domain for feature extraction. However, both schemes are based on a common assumption - the actions in space can be well aligned between consecutive frames, so that features can be aggregated based on the actions corresponding to the same position between adjacent frames, and then the action features are collected based on the aggregated features. However, this default situation where the actions between consecutive frames of the video can be well aligned in space has a small scope of application, and may cause inaccurate collection of video motion features when the object's motion speed and action deformation are relatively large.

[0080] To address this problem, an embodiment of the present invention provides a method for extracting motion features from video frames, so as to determine the aggregated features of the neighborhood in the next frame based on a first feature corresponding to a set spatial coordinate of a moving object in a target frame and a second feature corresponding to a set size of a neighborhood of the moving object in the next frame, and determine the motion features in the target frame based on the aggregated features, thereby improving the accuracy of the motion features of the video frames. Figure 1 As shown, the motion feature extraction method of the video frame includes the following steps:

[0081] Step 101: Obtain a feature map of each frame in the video data.

[0082] It should be noted that the method for extracting motion features from video frames provided in the embodiments of the present invention can be performed by a device for extracting motion features from video frames. The device for extracting motion features from video frames can be an electronic device or can be configured in an electronic device. The electronic device can be any stationary or mobile computing device capable of data processing, such as a mobile computing device such as a laptop computer, a smartphone, or a wearable device, or a stationary computing device such as a desktop computer, or a server, or other types of computing devices, etc., and this is not limited in the embodiments of the present invention.

[0083] It is understandable that a video can be considered to be composed of a series of video frame sequences. In this embodiment, the motion feature extraction device for video frames can obtain the feature map of each frame in the video data and perform subsequent steps. As a possible implementation method, the motion feature extraction device for video frames can use a neural network to process the video data to obtain the feature map of each frame in the video data. It should be noted that the motion feature extraction device for video frames in this embodiment can obtain the feature map of each frame in the video data through various open, legal and compliant methods.

[0084] In one possible implementation of this embodiment, the video frame motion feature extraction device can obtain feature maps for each frame in the video data through online or offline acquisition. For example, after authorization, the video frame motion feature extraction device can collect feature maps for each frame in the video data in real time at a historical moment, or after authorization, it can obtain feature maps for each frame in the video data through offline manual methods, etc., and this embodiment does not impose any restrictions on this.

[0085] In another possible implementation of this embodiment, the video frame motion feature extraction device may obtain the feature map of each frame in the video data through network transmission or physical copying. For example, after authorization, the video frame motion feature extraction device may obtain the feature map of each frame in the video data from another device through network transmission or physical copying.

[0086] Step 102, for any target frame, determine, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size neighborhood of the moving object displayed in the next frame.

[0087] In this embodiment, the first feature can be used to indicate the moving object displayed by the corresponding set spatial coordinates in the target frame, and the second feature is used to indicate the moving object displayed in the neighborhood of the same set spatial coordinates in the next frame, so that for any target frame, the first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and the second feature corresponding to the neighborhood of the set size of the moving object displayed in the next frame can be determined based on the acquired feature map of the target frame and the feature map of the next frame. Among them, the set spatial coordinates can be represented by (x, y), and the neighborhood of the set size can be represented by a k×k grid. It should be noted that this embodiment does not limit the specific values ​​of x, y in the set spatial coordinates (x, y) and k in the set size k×k. Optionally, they can be set according to manual experience, or they can be dynamically adjusted according to actual application requirements. This is not limited in this embodiment.

[0088] In one possible implementation of this embodiment, the first feature may be a query feature, i.e., a feature of the item to be queried, which can be understood as the feature corresponding to the set spatial coordinates of the moving object displayed in the target frame. The second feature may be a key feature and a value feature, i.e., the feature of each item in the query set, which can be understood as the feature corresponding to each spatial coordinate within a neighborhood of a set size of the moving object displayed in the next frame. The key feature and the value feature differ only in the notation. Optionally, in response to the acquired feature map of the video frame being a three-dimensional feature map, the three-dimensional feature map can be first converted into a two-dimensional sequence. Then, for any target frame, the first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame, i.e., the query feature, and the second feature corresponding to the neighborhood of the set size of the moving object displayed in the next frame, i.e., the key feature and the value feature, are determined. For example, assume that the feature map of each frame in the video data is a three-dimensional feature map F, and the dimensions of the three-dimensional feature map F are C×L×H×W, where C, H×W, and L represent the channel size, two-dimensional feature size, and time length of the feature, respectively. Therefore, the three-dimensional feature map F can be converted into a two-dimensional sequence Then, for any target frame, assuming it is the t-th frame, determine the features of the set spatial coordinates (x, y) in the t-th frame, that is, the query feature, recorded as Q t ∈R C , and the features of the neighborhood of the set size around the set spatial coordinate (x, y) in the t+1 frame, that is, the key feature and the value feature, recorded as K t+1 ∈R C×{k×k} and V t+1 ∈R C×{k×k} .

[0089] Step 103: Determine the aggregated features of the neighborhood in the next frame based on the first feature and the second feature.

[0090] In this embodiment, the aggregate feature is used to indicate the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature. Since the first feature can be used to indicate the moving object displayed by the corresponding set spatial coordinates in the target frame, and the second feature is used to indicate the moving object displayed in the neighborhood corresponding to the same set spatial coordinates in the next frame, the aggregate feature of the neighborhood in the next frame can be determined based on the first feature and the second feature. As a possible implementation method, the similarity between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature can be measured based on the first feature and the second feature to obtain the corresponding similarity matrix, and then the similarity matrix and the second feature are fused in a channel-by-channel manner to obtain the aggregate feature of the neighborhood in the next frame.

[0091] Step 104: Fusing the first feature and the aggregated feature to obtain the action feature in the target frame.

[0092] In this embodiment, since the aggregated feature is used to indicate the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, the first feature corresponding to the set spatial coordinates displaying the moving object in the target frame is enhanced by the aggregated feature to obtain the motion feature in the target frame. In this way, the features of the same spatial coordinates in the time domain are captured while taking into account the correlation between adjacent frames, making the motion feature in the target frame ultimately obtained more accurate.

[0093] In a possible implementation of this embodiment, the first feature and the fusion feature can be added together to obtain the action feature in the target frame. For example, assuming that the first feature is recorded as Q t , the aggregated feature is recorded as A t+1 , so the action feature Y in the target frame t It can be calculated using the following formula:

[0094] Y t =Q t +A t+1

[0095] It can be understood that the present technical solution will explore the correlation between the features of any set spatial coordinate position in any target frame and the features of each spatial coordinate position in the neighborhood of a set size around the position in the next frame, so that the feature map of the target frame will be enhanced by the feature map of the next frame in this way. Therefore, the present technical solution will operate between each pair of adjacent frames. It should be noted that for the last frame in the video data, since there is no next frame for correlation processing, it can enhance itself by calculating the correlation between the features of any set spatial coordinate position in the frame and the features of each spatial coordinate position in the neighborhood of a set size around the position. This ensures that the present technical solution can maintain a fixed time domain length in the time domain.

[0096] The method for extracting motion features from video frames provided by an embodiment of the present invention obtains a feature map of each frame in the video data, and thereby determines, for any target frame, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size of the neighborhood of the moving object displayed in the next frame, thereby determining the aggregated features of the neighborhood in the next frame based on the first and second features, and determining the motion features in the target frame based on the aggregated features. Since the aggregated features are determined based on the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, compared to the prior art that only captures the correlation between corresponding coordinate points in adjacent frames, the present invention expands the acquisition area and improves the accuracy of the motion features of the video frame.

[0097] From the above analysis, it can be seen that there are three different schemes for feature extraction of video frames, namely, the time domain convolution scheme, the time domain self-attention scheme and the technical scheme of this invention. In order to clearly illustrate the differences between the above three schemes, the present invention also provides a schematic diagram of the principles of the above three schemes. Figure 2 A schematic diagram of the principle of a time domain convolution solution, a time domain self-attention solution and the technical solution provided by an embodiment of the present invention. Figure 2 (a) is the time domain convolution scheme, Figure 2 (b) is the temporal self-attention scheme, Figure 2 (c) is the technical solution of this invention.

[0098] like Figure 2As shown in (a), in the time domain convolution scheme, multiple features at the same spatial coordinate position of multiple consecutive video frames are convolved according to the time sequence of each video frame. When the amplitude of motion deformation in the video frame is too large, different moving objects will be represented at the same spatial coordinate position between consecutive frames. For example, the spatial coordinate position in the t-1th frame and the t+1th frame represents the sports field, while the spatial coordinate position in the tth frame represents the pole vaulter. Therefore, when using time domain convolution to perform time domain feature aggregation, that is, when the same spatial position is used to aggregate features in the time domain, the video motion features will be lost.

[0099] like Figure 2 As shown in (b), in the temporal self-attention scheme, the feature of any set spatial coordinate position of the target frame, i.e., the t-th frame, is used as the query feature, the feature of the same spatial coordinate position of the previous frame, i.e., the t-1th frame, is used as the key feature, and the feature of the same spatial coordinate position of the next frame, i.e., the t+1th frame, is used as the value feature for temporal feature aggregation. Similarly, when the amplitude of motion deformation in the video frame is too large, different moving objects are represented at the same spatial coordinate position between consecutive frames, resulting in the loss of video motion features.

[0100] like Figure 2 As shown in (c), in this technical solution, the feature of any set spatial coordinate position of the target frame, i.e., the t-th frame, is used as the query feature, and the features of each spatial coordinate position within a set size of the same spatial coordinate position of the next frame, i.e., the t+1-th frame, are used as the key feature and value feature to perform time domain feature aggregation, thereby expanding the aggregation of features from the aggregation of the same spatial coordinate position to the aggregation of each spatial coordinate position within the field. On the one hand, the receptive field of the time domain aggregation is expanded, and on the other hand, the spatial coordinate positions within the field are explicitly utilized to align the overall spatial motion. Therefore, when the action undergoes a large deformation, it can ensure that the time domain feature aggregation between consecutive frames occurs at the corresponding action area position, thereby enhancing the effect of time domain feature aggregation and solving the problem of loss of video action features caused by large motion or deformation.

[0101] From the above analysis, it can be seen that in an embodiment of the present invention, for any target frame, the second feature corresponding to the set size of the neighborhood showing the moving object in the next frame can be determined based on the feature map of the target frame and the feature map of the next frame. In order to clearly explain how the present invention determines the second feature corresponding to the set size of the neighborhood showing the moving object in the next frame according to the feature map of the target frame and the feature map of the next frame for any target frame, the present invention also provides a method for extracting motion features of video frames.

[0102] Figure 3A flowchart of another method for extracting motion features from video frames provided by an embodiment of the present invention.

[0103] like Figure 3 As shown, the method for extracting motion features of a video frame may include the following steps:

[0104] Step 301: Obtain a feature map of each frame in the video data.

[0105] It should be noted that the execution process of this step can refer to the execution process of step 101 in the previous embodiment, and the principles are the same, so it will not be repeated here.

[0106] Step 302 : For any target frame, determine a first feature and a motion saliency map corresponding to set spatial coordinates of a moving object in the target frame based on the feature map of the target frame and the feature map of the next frame.

[0107] In this embodiment, the first feature can be used to indicate the moving object displayed by the corresponding set spatial coordinates in the target frame. For any target frame, the first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame can be determined based on the acquired feature map of the target frame.

[0108] In this embodiment, since the feature map of any target frame and the feature map of the next frame can be obtained, the motion saliency map (MSM) can be determined based on the feature map of the target frame and the feature map of the next frame, which can be denoted as f m .

[0109] Step 303 : for the motion saliency map, a coordinate offset estimator is used to predict the offset of features of each spatial coordinate in a neighborhood of a set size in the next frame, so as to determine a plurality of sampling coordinates in the neighborhood according to the offset.

[0110] In this embodiment, for the motion saliency map f m , a coordinate offset estimator can be used to predict the offset of the features of each spatial coordinate in a neighborhood of a set size in the next frame, thereby determining multiple sampling coordinates in the neighborhood based on the offset value. The coordinate offset estimator can be implemented by a two-dimensional convolution, and the number of output channels of the convolution can be 2k 2 Optionally, the offset of the features of each spatial coordinate in a neighborhood of a set size in the next frame can be recorded as (Δa, Δb). That is, (Δa, Δb) is the coordinate displacement corresponding to the spatial coordinate point p = (a, b) in a k×k grid in a neighborhood of a set size centered on the set spatial coordinate (x, y) of the target frame, so that the corresponding sampling coordinate in the neighborhood can be expressed as p' = (a+Δa, b+Δb). Thus, multiple sampling coordinates in the neighborhood can be determined.

[0111] Step 304: Use bilinear interpolation to determine the sampling features corresponding to each sampling coordinate.

[0112] In this embodiment, after determining multiple sampling coordinates in the neighborhood, bilinear interpolation can be used to determine the sampling features corresponding to each sampling coordinate. Optionally, for any sampling coordinate p', the following calculation formula can be used to determine the sampling feature K' corresponding to the sampling coordinate p' t+1 (p′):

[0113]

[0114] Among them, p' is used to represent the sampling coordinates, that is, the differentiable spatial position (including the spatial position of the offset), p is used to represent all integer spatial positions in the neighborhood (that is, the original regular position), K t+1 (p) is used to represent the feature corresponding to position p in the regular k×k neighborhood grid, and G is the kernel of bilinear interpolation.

[0115] Step 305 : Determine, based on each sampling feature, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame.

[0116] In this embodiment, after obtaining the sampling features corresponding to each sampling coordinate, the second feature corresponding to the neighborhood of the set size of the set spatial coordinate in the next frame can be determined according to each sampling feature. 2 The sampling features corresponding to each sampling coordinate in the neighborhood of the next frame are used as the second features corresponding to the neighborhood of the set size of the set spatial coordinate in the next frame, that is, the key feature and the value feature, which can be recorded as K t+1 ∈R C×{k×k} and V t+1 ∈R C×{k×k} .

[0117] Step 306: Determine the aggregated features of the neighborhood in the next frame based on the first feature and the second feature.

[0118] Step 307: Fuse the first feature and the aggregated feature to obtain the action feature in the target frame.

[0119] It should be noted that the execution process of steps 306-307 can refer to the execution process of steps 103-104 in the previous embodiment, and the principles are the same, so they will not be repeated here.

[0120] The method for extracting motion features from video frames provided by an embodiment of the present invention determines, for any target frame, a first feature and a motion saliency map corresponding to set spatial coordinates of a moving object displayed in the target frame based on a feature map of the target frame and a feature map of the next frame. Based on the motion saliency map, a coordinate offset estimator is used to predict the offset of the features of each spatial coordinate within a neighborhood of a set size in the next frame, so as to determine multiple sampling coordinates within the neighborhood based on the offset. After determining the sampling features corresponding to each sampling coordinate using a bilinear interpolation method, a second feature corresponding to a neighborhood of a set size of the set spatial coordinate in the next frame is determined based on each sampling feature. Thus, by performing coordinate offset prediction of the neighborhood based on the motion saliency feature map, the problem of ignoring the geometric deformation caused by the motion of the moving object due to direct calculation within the neighborhood of a set size is avoided.

[0121] To make it clear Figure 3 In the embodiment shown, in step 302, for any target frame, a motion saliency map is determined based on the feature map of the target frame and the feature map of the next frame. This embodiment provides Figure 4 The flowchart of determining the motion saliency map is shown in FIG. Figure 4 As shown, determining the motion saliency map may include the following steps:

[0122] Step 401 : for any target frame, determine the feature difference between frames based on the feature map of the target frame and the feature map of the next frame.

[0123] Here, for any target frame, the feature difference between frames can be determined based on the feature map of the target frame and the feature map of the next frame. Optionally, the feature map of the target frame can be expressed as f of the tth frame. t , the feature map of the next frame can be expressed as f of the t+1th frame t+1 , so the inter-frame feature difference Δf between the feature map of the target frame and the feature map of the next frame can be calculated using the following formula:

[0124] Δf=f t+1 -f t

[0125] In step 402, an activation function is used to normalize the feature differences between frames to obtain a corresponding attention map.

[0126] Here, after determining the inter-frame feature difference, the inter-frame feature difference can be normalized using an activation function to obtain a corresponding attention map. The attention map is used to indicate the spatial position of a moving object that moves between two frames. Optionally, the inter-frame feature difference can be represented as Δf, and the activation function can be a sigmoid function, so that the attention map can be represented as sigmoid(Δf).

[0127] Step 403: multiply the attention map and the feature map of the next frame to obtain a motion saliency map.

[0128] Here, since the attention map is used to indicate the spatial position of the moving object with motion between two frames, that is, the attention map dynamically points out the spatial position with relatively large motion in the next frame, the motion saliency map is obtained by multiplying the attention map by the feature map of the next frame. Optionally, the attention map can be expressed as sigmoid(Δf), and the feature map of the next frame can be expressed as f of the t+1th frame. t+1 , the motion saliency map can represent f m , so f can be calculated using the following formula m :

[0129] f m =sigmoid(Δf)×f t+1

[0130] In summary, for any target frame, we can determine the inter-frame feature difference based on the feature map of the target frame and the feature map of the next frame, and then use an activation function to normalize the inter-frame feature difference to obtain the corresponding attention map. Then, we multiply the attention map with the feature map of the next frame to obtain the motion saliency map. Thus, for any target frame, we can determine the motion saliency map based on the feature map of the target frame and the feature map of the next frame.

[0131] From the above analysis, it can be seen that in an embodiment of the present invention, the aggregation features of the neighborhood in the next frame can be determined based on the first feature and the second feature. In order to clearly explain how the present invention determines the aggregation features of the neighborhood in the next frame based on the first feature and the second feature, the present invention also provides a method for extracting motion features of video frames.

[0132] Figure 5 A flowchart of another method for extracting motion features from video frames provided by an embodiment of the present invention.

[0133] like Figure 5 As shown, the method for extracting motion features of a video frame may include the following steps:

[0134] Step 501: Obtain a feature map of each frame in the video data.

[0135] Step 502, for any target frame, determine, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size neighborhood of the moving object displayed in the next frame.

[0136] It should be noted that the execution process of steps 501-502 can refer to the execution process of steps 101-102 in the above embodiment, and the principles are the same, so they will not be repeated here.

[0137] Step 503: Perform a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix.

[0138] In this embodiment, the first feature and the second feature can be matrix multiplied to obtain a similarity matrix. The similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in the neighborhood corresponding to the first feature and the second feature. Optionally, the first feature can be a query feature, denoted by Q t ∈R C , the second feature can be key feature and value feature, denoted as K t+1 ∈R C×{k×k} and V t+1 ∈R C×{k×k} , the similarity matrix can be recorded as W cor Among them, the key feature and the value feature are just different in the notation. Therefore, the query feature and the key feature can be matrix multiplied to obtain the similarity matrix W cor , the specific calculation formula is as follows:

[0139]

[0140] in, Represents matrix multiplication operation, that is, dot product operation. The similarity matrix W cor It measures the similarity between the spatial coordinate features in the neighborhood of the query feature and the key feature.

[0141] Step 504 : Using the similarity matrix as a weight, perform a matrix multiplication operation with the transposed matrix of the second feature to obtain the aggregated features of the neighborhood in the next frame.

[0142] In this embodiment, the similarity matrix can be used as a weight to perform matrix multiplication with the transposed matrix of the second feature to obtain the aggregated features of the neighborhood in the next frame. Optionally, the second feature can be a key feature and a value feature, denoted as K t+1 ∈R C×{k×k} and V t+1 ∈R C×{k×k} , the similarity matrix can be recorded as W cor , the aggregated feature can be recorded as A t+1 Among them, the key feature and the value feature are just different in the token. cor As a weight, perform matrix multiplication with the transposed matrix of the value feature to obtain the aggregated feature A of the neighborhood in the next frame t+1, the specific calculation formula is as follows:

[0143]

[0144] in, represents matrix multiplication operation, i.e. dot product operation, [·] T Here, the similarity matrix is ​​used as a weight to aggregate the spatial coordinate features in the neighborhood of adjacent frames and enhance the query features.

[0145] Step 505: Fuse the first feature and the aggregated feature to obtain the action feature in the target frame.

[0146] It should be noted that the execution process of this step can refer to the execution process of step 104 in the above embodiment, and the principles are the same, so it will not be repeated here.

[0147] The method for extracting motion features from video frames provided by an embodiment of the present invention performs a matrix multiplication operation on a first feature and a second feature to obtain a similarity matrix, wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in the neighborhood corresponding to the first and second features. The similarity matrix is ​​then used as a weight and matrix multiplied with the transposed matrix of the second feature to obtain the aggregated features of the neighborhood in the next frame. This allows the obtained aggregate matrix to be expanded from the aggregation of the same spatial coordinate position to the aggregation of each spatial coordinate position within the domain, explicitly utilizing each spatial coordinate position within the domain to align the overall spatial motion, thereby enhancing the effect of temporal feature aggregation.

[0148] In order to explain the above embodiment more clearly, an example is given below.

[0149] Figure 6 The schematic diagram of the principle of a method for extracting motion features from a video frame provided by an embodiment of the present invention. Figure 6 (a) is the SIFA (Stand-alone Inter-Frame Attention) operator proposed by the motion feature extraction method of the video frame provided by the present invention, Figure 6 (b) is the joint spatiotemporal self-attention operator, Figure 6 (c) is the decoupled spatiotemporal self-attention operator.

[0150] like Figure 6 As shown in the time aggregation between adjacent frames in the upper left corner, the motion feature extraction method of the video frame can extract motion features for consecutive video frames from frame 1 to frame t+1, and the feature map of each target frame will be enhanced by the feature map of the next frame.

[0151] like Figure 6As shown in the upper right corner of the action feature of the t-th frame, the query feature corresponding to the set spatial coordinates of the t-th frame and the key feature corresponding to the neighborhood of the set size of the t+1-th frame can be matrix multiplied to obtain the similarity matrix W cor , and then the similarity matrix W cor Perform matrix multiplication with the value feature corresponding to the neighborhood of the set size of the t+1th frame to obtain the aggregate feature A t+1 , and aggregate feature A t+1 Add it to the query feature corresponding to the set spatial coordinates of the t-th frame to enhance the query feature.

[0152] like Figure 6 As shown in (a), in the SIFA operator proposed by the motion feature extraction method for video frames provided by the present invention, the fourth matrix in the second row of the feature map of frame t can be used as the set spatial coordinates, and the square area enclosed by the first to third rows and the third to fifth columns in the feature map of frame t+1 can be used as the neighborhood of a set size in frame t+1. Since the relevant calculations are only performed within the neighborhood of the next frame, the computational complexity is low. Furthermore, since the fusion features are determined through a larger neighborhood rather than simply mining the temporal changes of the same spatial location, the information interaction between frames is enriched.

[0153] like Figure 6 As shown in (b), in the joint spatiotemporal self-attention operator, the fourth matrix in the second row of the feature map of frame t is also used as the set spatial coordinate, i.e., the target matrix. When calculating the features of this matrix, it is necessary to calculate the features of the feature map of frame t-1, i.e., the features of all matrices in the feature map of frame t-1, and the features of the feature map of frame t+1, i.e., the features of all matrices in the feature map of frame t+1. The features of the target matrix are determined based on the features of each matrix in frame t-1 and frame t+1, which results in a large amount of computation.

[0154] like Figure 6 As shown in (c), in the decoupled spatiotemporal self-attention operator, the fourth matrix in the second row of the feature map of frame t is also used as the set spatial coordinate, i.e., the target matrix. When calculating the features of this matrix, not only the features of the feature map of frame t-1 and the features of the feature map of frame t+1 must be calculated, but also the features of the fourth matrix in the second row of the feature map of frame t-1 and the fourth matrix in the second row of the feature map of frame t+1 must be calculated to determine the features of the target matrix, which requires more computation.

[0155] Figure 7 Schematic diagram of another method for extracting motion features from video frames provided by an embodiment of the present invention. As shown in FIG7 , first, according to the feature map f of the tth frame,t , determine the query feature corresponding to the set spatial coordinates in the feature map of the tth frame, and according to the feature map f of the tth frame t and the feature map f of the t+1th frame t+1 , determine the inter-frame feature difference Δf, and then use the sigmoid function to normalize the inter-frame feature difference Δf and compare it with the feature map f of the t+1 frame t+1 Multiply them together to get the motion saliency map f m , and then according to the motion saliency map f m , the coordinate offset estimator is used to predict the offset of the features of each spatial coordinate in the neighborhood of the set size in the next frame to obtain multiple sampling coordinates in the neighborhood, and the bilinear interpolation method is used to determine the sampling features corresponding to each sampling coordinate to determine the key features and value features corresponding to the neighborhood of the set size in the t+1 frame, thereby performing a matrix multiplication operation on the query features corresponding to the set spatial coordinates of the t frame and the key features corresponding to the neighborhood of the set size in the t+1 frame to obtain the similarity matrix W cor , and then the similarity matrix W cor Perform matrix multiplication with the value feature corresponding to the neighborhood of the set size of the t+1th frame to obtain the aggregate feature A t+1 , and aggregate feature A t+1 Add it to the query feature corresponding to the set spatial coordinates of the t-th frame to enhance the query feature.

[0156] It can be understood that the motion feature extraction method of the video frame provided by the present invention can be applied to the two-dimensional convolutional neural network and the visual Transformer network. In order to clearly illustrate how the present invention applies the motion feature extraction method of the video frame provided by the present invention to the two-dimensional convolutional neural network and the visual Transformer network, the present invention also provides a motion feature extraction method for the video frame.

[0157] Figure 8 A flowchart of another method for extracting motion features from video frames provided by an embodiment of the present invention.

[0158] like Figure 8 As shown, the method for extracting motion features of a video frame may include the following steps:

[0159] Step 801: Obtain a feature map of each frame in the video data.

[0160] It should be noted that the execution process of this step can refer to the execution process of step 101 in the above embodiment. The principles are the same and will not be repeated here.

[0161] Step 802: For any target frame, a two-dimensional convolutional neural network or a visual feature extraction Transformer network is used to determine the feature map of the target frame and the feature map of the next frame.

[0162] In this embodiment, a two-dimensional convolutional neural network or a visual feature extraction transformer network can be used to determine the feature map of the target frame and the feature map of the next frame for any target frame. Specifically, the convolution kernel in the basic residual module of the two-dimensional convolutional neural network can be used to determine the feature map of the target frame and the next frame, or the MSA module with a regular window configuration in the transformer network can be used to determine the feature map of the target frame and the next frame.

[0163] Step 803, for any target frame, determine, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size neighborhood of the moving object displayed in the next frame.

[0164] Step 804: Determine the aggregated features of the neighborhood in the next frame based on the first feature and the second feature.

[0165] Step 805: Fuse the first feature and the aggregated feature to obtain the action feature in the target frame.

[0166] It should be noted that the execution process of steps 803-805 can refer to the execution process of steps 102-104 in the above embodiment, and the principles are the same, so they will not be repeated here.

[0167] The method for extracting motion features from video frames provided by the present invention improves video feature learning by using a two-dimensional convolutional neural network or a visual feature extraction Transformer network to determine the feature map of the target frame and the feature map of the next frame for any target frame.

[0168] In order to explain the above embodiment more clearly, an example is given below.

[0169] Figure 9 This is a schematic diagram of an application of a method for extracting motion features from a video frame provided by an embodiment of the present invention. Figure 9 (a) is to apply the motion feature extraction method of the video frame provided by the present invention to a two-dimensional convolutional neural network, Figure 9 (b) is to apply the motion feature extraction method of the video frame provided by the present invention to the visual Transformer network.

[0170] like Figure 9As shown in (a), since most video network structures are formed by decoupling the spatiotemporal 3D convolution to form 2D spatial convolution and temporal 1D convolution, and the temporal 1D convolution is usually embedded after the 2D spatial convolution for temporal modeling, the SIFA-Block module proposed by the motion feature extraction method of the video frame provided by the present invention can be embedded after the 3x3 convolution in the ResNet basic residual module. Among them, the SIFA-Block module is used to perform Figure 7 The process is shown in Figure 2. It can be understood that since the SIFA-Block module is only embedded in the last three stages of ResNet, only a small amount of computation is added. In addition, the global pooling operation is applied to the output features to obtain frame-level features for the final video action classification optimization.

[0171] The SIFA-Block module proposed by the method for extracting motion features from video frames provided by the present invention can also be embedded into the Swin-Transformer basic network to construct a SIFA-Transformer for video feature modeling. In particular, for two consecutive Swin-Transformer basic modules, such as Figure 9 As shown in (b), the SIFA-Block module can be placed only after the MSA module with a regular window configuration (W-MSA). Here, the model can reshape the patch sequence output by the W-MSA module into the dimensions C×L×H×W as the input of the SIFA-Block module. Furthermore, for the final reshaped output, the model can use global pooling to obtain frame-level features for feature learning.

[0172] In order to implement the above embodiment, the present invention further proposes a device for extracting motion features from video frames.

[0173] Figure 10 A schematic structural diagram of a device for extracting motion features from video frames provided by an embodiment of the present invention.

[0174] like Figure 10 As shown, the motion feature extraction device for the video frame includes: an acquisition module 11, a first determination module 12, a second determination module 13 and a fusion module 14.

[0175] An acquisition module 11 is used to acquire a feature map of each frame in the video data;

[0176] A first determining module 12 is configured to determine, for any target frame, based on a feature map of the target frame and a feature map of a next frame, a first feature corresponding to a set spatial coordinate at which a moving object is displayed in the target frame and a second feature corresponding to a neighborhood of a set size at which the moving object is displayed in the next frame; the first feature is configured to indicate the moving object displayed at the corresponding set spatial coordinate in the target frame; and the second feature is configured to indicate the moving object displayed in the neighborhood of the same set spatial coordinate in the next frame;

[0177] a second determining module 13 configured to determine, based on the first feature and the second feature, an aggregate feature of the neighborhood in a next frame; wherein the aggregate feature is configured to indicate a correlation between a moving object represented by a set spatial coordinate corresponding to the first feature and a moving object represented in a neighborhood corresponding to the second feature;

[0178] The fusion module 14 is configured to fuse the first feature and the aggregated feature to obtain the action feature in the target frame.

[0179] Furthermore, in a possible implementation of the embodiment of the present invention, the first determining module 12 includes:

[0180] The first determining unit 1201 is configured to determine, for the arbitrary target frame, a motion saliency map based on the feature map of the target frame and the feature map of the next frame;

[0181] A prediction unit 1202 is configured to predict, based on the motion saliency map, an offset of features of each spatial coordinate within the neighborhood of the set size in the next frame using a coordinate offset estimator, so as to determine a plurality of sampling coordinates within the neighborhood based on the offset;

[0182] The second determining unit 1203 is configured to determine the sampling feature corresponding to each sampling coordinate by using a bilinear interpolation method;

[0183] The third determining unit 1204 is configured to determine, based on the sampling features, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame.

[0184] Furthermore, in a possible implementation of the embodiment of the present invention, the first determining unit 1201 is further configured to:

[0185] For any target frame, determining an inter-frame feature difference based on a feature map of the target frame and a feature map of the next frame;

[0186] Normalizing the inter-frame feature differences using an activation function to obtain a corresponding attention map; wherein the attention map is used to indicate the spatial position of a moving object that moves between two frames;

[0187] The attention map is multiplied with the feature map of the next frame to obtain a motion saliency map.

[0188] Furthermore, in a possible implementation of the embodiment of the present invention, the second determining module 13 includes:

[0189] a first processing unit, configured to perform a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix; wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in a neighborhood corresponding to the first feature and the second feature;

[0190] The second processing unit is configured to perform a matrix multiplication operation on the transposed matrix of the second feature by using the similarity matrix as a weight to obtain an aggregated feature of the neighborhood in a next frame.

[0191] It should be noted that the aforementioned explanation of the embodiment of the method for extracting motion features from video frames is also applicable to the device for extracting motion features from video frames of this embodiment, and will not be repeated here.

[0192] Based on the above embodiment, the embodiment of the present invention further provides a possible implementation of a device for extracting motion features from a video frame. Figure 11 This is a structural diagram of another device for extracting motion features from video frames provided by an embodiment of the present invention. Based on the previous embodiment, the device for extracting motion features from video frames further includes: a third determining module 15.

[0193] The third determination module 15 is used to determine the feature map of the target frame and the feature map of the next frame by using a two-dimensional convolutional neural network or a visual feature extraction Transformer network for any target frame.

[0194] Furthermore, in a possible implementation of the embodiment of the present invention, the third determining module 15 is further configured to:

[0195] Determine a feature map of the target frame and the next frame using a convolution kernel in a basic residual module of a two-dimensional convolutional neural network; or

[0196] The MSA module with a regular window configuration in the Transformer network is used to determine feature maps for the target frame and the next frame.

[0197] The motion feature extraction device for video frames provided by an embodiment of the present invention obtains a feature map of each frame in the video data, and thereby determines, for any target frame, based on the feature map of the target frame and the feature map of the next frame, a first feature corresponding to the set spatial coordinates of the moving object displayed in the target frame and a second feature corresponding to the set size of the neighborhood of the moving object displayed in the next frame, thereby determining the aggregated features of the neighborhood in the next frame based on the first and second features, and determining the motion features in the target frame based on the aggregated features. Since the aggregated features are determined based on the correlation between the moving object displayed by the set spatial coordinates corresponding to the first feature and the moving object displayed in the neighborhood corresponding to the second feature, compared to the prior art that only captures the correlation between corresponding coordinate points in adjacent frames, the present invention expands the acquisition area and improves the accuracy of the motion features of the video frame.

[0198] In order to implement the above embodiments, the present invention also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the motion feature extraction method of the video frame proposed in any of the aforementioned embodiments of the present invention.

[0199] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention can implement the present invention Figure 1-11 The process of the embodiment shown is as follows: Figure 12 As shown, the electronic device may include: a shell 1, a processor 2, a memory 3, a circuit board 4 and a power supply circuit 5, wherein the circuit board 4 is placed inside the space enclosed by the shell 1, and the processor 2 and the memory 3 are arranged on the circuit board 4; the power supply circuit 5 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 3 is used to store executable program code; the processor 2 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 3, so as to execute the motion feature extraction method of the video frame described in any of the aforementioned embodiments.

[0200] For details on the specific execution process of the above steps by the processor 2 and the steps further executed by the processor 2 by running the executable program code, please refer to the present invention. Figure 1-11 The description of the illustrated embodiment will not be repeated here.

[0201] In order to implement the above embodiments, the present invention further proposes a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the motion feature extraction method of the video frame proposed in any of the above embodiments of the present invention.

[0202] In order to implement the above embodiments, the present invention further proposes a computer program product, including a computer program, which, when executed by a processor, implements the method for extracting motion features from video frames proposed in any of the above embodiments of the present invention.

[0203] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0204] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0205] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the invention belong.

[0206] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0207] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the present invention: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0208] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0209] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0210] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for extracting motion features from video frames, characterized in that: The following steps are involved: Obtain feature maps of each frame in the video data; For any target frame, determine, based on a feature map of the target frame and a feature map of a next frame, a first feature corresponding to a set spatial coordinate of a moving object displayed in the target frame and a second feature corresponding to a neighborhood of a set size of the moving object displayed in the next frame; wherein the first feature is used to indicate the moving object displayed at the corresponding set spatial coordinate in the target frame; and the second feature is used to indicate the moving object displayed in the neighborhood of the same set spatial coordinate in the next frame; Determining, based on the first feature and the second feature, an aggregate feature of the neighborhood in a next frame; wherein the aggregate feature is used to indicate a correlation between a moving object displayed by a set spatial coordinate corresponding to the first feature and a moving object displayed in a neighborhood corresponding to the second feature; fusing the first feature and the aggregated feature to obtain an action feature in the target frame; The determining, for the arbitrary target frame, based on the feature map of the target frame and the feature map of the next frame, a second feature corresponding to a neighborhood of a set size showing the moving object in the next frame includes: For any target frame, determining a motion saliency map based on the feature map of the target frame and the feature map of the next frame; For the motion saliency map, using a coordinate offset estimator to predict the offset of features of each spatial coordinate within a neighborhood of a set size in the next frame, so as to determine a plurality of sampling coordinates within the neighborhood according to the offset; Using bilinear interpolation method, determine the sampling features corresponding to each sampling coordinate; According to each of the sampling features, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame is determined.

2. The method according to claim 1, characterized in that The step of determining a motion saliency map for any target frame according to the feature map of the target frame and the feature map of the next frame includes: For any target frame, determining an inter-frame feature difference based on a feature map of the target frame and a feature map of the next frame; Normalizing the inter-frame feature differences using an activation function to obtain a corresponding attention map; wherein the attention map is used to indicate the spatial position of a moving object that moves between two frames; The attention map is multiplied with the feature map of the next frame to obtain a motion saliency map.

3. The method according to any one of claims 1-2, characterized in that The determining, based on the first feature and the second feature, an aggregate feature of the neighborhood in the next frame includes: Performing a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix; wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in a neighborhood corresponding to the first feature and the second feature; The similarity matrix is ​​used as a weight to perform a matrix multiplication operation with the transposed matrix of the second feature to obtain the aggregated features of the neighborhood in the next frame.

4. The method according to any one of claims 1 to 2, characterized in that The method further comprises: For any target frame, a two-dimensional convolutional neural network or a visual feature extraction Transformer network is used to determine the feature map of the target frame and the feature map of the next frame.

5. The method according to claim 4, characterized in that The method of using a two-dimensional convolutional neural network or a visual feature extraction Transformer network to determine the feature map of the target frame and the feature map of the next frame includes: Determine a feature map of the target frame and the next frame using a convolution kernel in a basic residual module of a two-dimensional convolutional neural network; or The MSA module with a regular window configuration in the Transformer network is used to determine feature maps for the target frame and the next frame.

6. A motion feature extraction device for a video frame, characterized in that: include: An acquisition module, used to obtain feature maps of each frame in the video data; A first determination module is configured to determine, for any target frame, based on a feature map of the target frame and a feature map of a next frame, a first feature corresponding to a set spatial coordinate of a moving object displayed in the target frame and a second feature corresponding to a neighborhood of a set size of the moving object displayed in the next frame; the first feature is configured to indicate the moving object displayed at the corresponding set spatial coordinate in the target frame; and the second feature is configured to indicate the moving object displayed in the neighborhood of the same set spatial coordinate in the next frame; a second determining module, configured to determine, based on the first feature and the second feature, an aggregate feature of the neighborhood in a next frame; wherein the aggregate feature is used to indicate a correlation between a moving object represented by a set spatial coordinate corresponding to the first feature and a moving object represented in a neighborhood corresponding to the second feature; a fusion module, configured to fuse the first feature and the aggregated feature to obtain an action feature in the target frame; The first determining module includes: a first determining unit, configured to determine, for the arbitrary target frame, a motion saliency map based on a feature map of the target frame and a feature map of the next frame; a prediction unit configured to predict, based on the motion saliency map, an offset of features of each spatial coordinate within the neighborhood of the set size in the next frame using a coordinate offset estimator, so as to determine a plurality of sampling coordinates within the neighborhood according to the offset; A second determining unit is configured to determine a sampling feature corresponding to each sampling coordinate by using a bilinear interpolation method; The third determining unit is configured to determine, based on each of the sampling features, a second feature corresponding to a neighborhood of a set size of a set spatial coordinate in a next frame.

7. The device according to claim 6, characterized in that The first determining unit is further configured to: For any target frame, determining an inter-frame feature difference based on a feature map of the target frame and a feature map of the next frame; Normalizing the inter-frame feature differences using an activation function to obtain a corresponding attention map; wherein the attention map is used to indicate the spatial position of a moving object that moves between two frames; The attention map is multiplied with the feature map of the next frame to obtain a motion saliency map.

8. The device according to any one of claims 6 to 7, characterized in that: The second determining module includes: a first processing unit, configured to perform a matrix multiplication operation on the first feature and the second feature to obtain a similarity matrix; wherein the similarity matrix is ​​used to represent the similarity between each spatial coordinate feature in a neighborhood corresponding to the first feature and the second feature; The second processing unit is configured to perform a matrix multiplication operation on the transposed matrix of the second feature by using the similarity matrix as a weight to obtain an aggregated feature of the neighborhood in a next frame.

9. The device according to any one of claims 6-7, characterized in that The device further comprises: The third determination module is used to determine the feature map of the target frame and the feature map of the next frame using a two-dimensional convolutional neural network or a visual feature extraction Transformer network for any target frame.

10. The device according to claim 9, characterized in that The third determining module is further configured to: Determine a feature map of the target frame and the next frame using a convolution kernel in a basic residual module of a two-dimensional convolutional neural network; or The MSA module with a regular window configuration in the Transformer network is used to determine feature maps for the target frame and the next frame.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images