Multi-modal bird action recognition method and device based on action speed characteristics

Abstract

The invention discloses a multi-mode bird action recognition method and device based on action speed characteristics. The method comprises the following steps: calculating an optical flow motion vector between two adjacent frames of images of an input video; dividing the optical flow motion vector values according to intervals to obtain a plurality of speed categories; converting each fast and slow category into a binary vector to obtain motion fast and slow features; fusing the motion speed feature and the vision-text feature to jointly serve as a semantic query part of the model; training a model: inputting the semantic query part and the video feature vector into a decoder, guiding the decoder by using the semantic query part, so that the decoder can focus on information of the semantic query part to decode to obtain an action recognition result, calculating an average precision mean value of the action recognition result, and outputting the action recognition result; the average precision mean value is used as an index for evaluating the training effect. According to the method, the time dynamic information in bird actions can be better captured, and the method has higher recognition accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a multimodal bird motion recognition method and device based on motion speed characteristics. Background Art

[0002] Bird motion recognition is an important task in computer vision and is widely used in ecological monitoring, intelligent security, and animal behavior research. Traditional bird motion recognition methods mainly rely on object recognition technology in images, but these methods often have difficulty accurately capturing and understanding the temporal dynamic characteristics of bird motion. Especially in scenes where the speed of motion changes is large, existing methods have difficulty in effectively extracting motion speed and frequency information, which affects the recognition effect. Therefore, how to combine motion information and visual features to improve the accuracy and robustness of bird motion recognition is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal bird motion recognition method and device based on motion speed features. By introducing optical flow analysis technology and TimeSformer architecture, the speed features constructed by the optical flow method are integrated into the semantic query module of the model. The present invention can more accurately capture the speed changes of bird motions, thereby improving the recognition performance of bird motions.

[0004] To achieve the above object, the present invention provides a multimodal bird motion recognition method based on motion speed characteristics, comprising:

[0005] Step 1, calculating the optical flow motion vector between two adjacent frames of the input video;

[0006] Step 2, dividing the optical flow motion vector value in step 1 into intervals to obtain multiple fast and slow categories;

[0007] Step 3, convert each speed category in step 2 into a binary vector to obtain the speed feature of the action;

[0008] Step 4, combining the action speed feature of step 3 with the visual-text feature, and using them together as the semantic query part of the model; wherein the visual-text feature is extracted from the input video by the video-text encoder according to the preset action label;

[0009] Step 5, training model: input the semantic query part and the video feature vector of step 4 into the decoder. The decoder uses the semantic query part to guide the decoder so that the decoder can focus on decoding the information of the semantic query part to obtain the action recognition result, and calculate the average precision mean of the action recognition result. The average precision mean is used as an indicator to evaluate the training effect. Among them, the video feature vector is obtained by randomly sampling the input video by the video encoder.

[0010] Preferably, step 1 specifically includes:

[0011] Step 1.1, assume that the brightness of the image changes from I1(x, y) at time t1 to I2(x, y) at time t2, and the displacement of each pixel in the image along the x and y directions is u and v. For each pixel point, use the optical flow equation (1) to obtain a two-dimensional optical flow vector field. Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point:

[0012]

[0013] in, and is the spatial gradient of the image, is the time gradient;

[0014] Step 1.2, calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

[0015] Preferably, step 3 specifically includes:

[0016] After the interval is divided, the fast and slow category set C = {c1, c2, ...c i ,,...,c n}, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector v = [v1, V2, ..., v j , ..., v n ],in Only the element at the index position representing the category is 1, and the elements at other positions are all 0.

[0017] Preferably, step 4 specifically includes:

[0018] Step 4.1, concatenate the vector v composed of the action speed features and the visual-text feature vector obtained by the video-text encoder to obtain a vector with dimension d z The vector z;

[0019] Step 4.2, through a linear transformation q = Wz + b, the concatenated vector z is mapped to a dimension d q The semantic query vector q is ensured to have the dimension of the semantic query vector q aligned with the semantic query of the model.

[0020] The present invention also provides a multi-modal bird motion recognition device based on motion speed characteristics, comprising:

[0021] An optical flow motion vector calculation unit, which is used to calculate the optical flow motion vector between two adjacent frames of an input video;

[0022] A category division unit, which is used to divide the optical flow motion vector value of the optical flow motion vector calculation unit according to intervals to obtain a plurality of fast and slow categories;

[0023] A vector conversion unit, which is used to convert each speed category of the category division unit into a binary vector to obtain a speed feature of the action;

[0024] A feature fusion unit is used to fuse the action speed feature of the vector conversion unit with the visual-text feature, and use them together as the semantic query part of the model; wherein the visual-text feature is extracted from the input video by the video-text encoder according to the preset action label;

[0025] A decoding calculation unit is used for training the model: the semantic query part of the feature fusion unit and the video feature vector are input into the decoder, and the decoder uses the semantic query part to guide the decoder so that the decoder can focus on decoding the information of the semantic query part to obtain the action recognition result, and the average precision mean of the action recognition result is calculated. The average precision mean is used as an indicator for evaluating the training effect; wherein, the video feature vector is obtained by randomly sampling the input video through the video encoder.

[0026] Preferably, the optical flow motion vector calculation unit specifically includes:

[0027] The optical flow vector calculation module is used to set the brightness of the image from I1(x, y) at time t1 to I2(x, y) at time t2, and the displacement of each pixel in the image along the x direction and y direction is u and v. For each pixel point, a two-dimensional optical flow vector field is obtained using the optical flow equation (1). Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point:

[0028]

[0029] in, and is the spatial gradient of the image, is the time gradient;

[0030] The optical flow motion vector module is used to calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

[0031] Preferably, the vector conversion unit specifically includes:

[0032] After the interval is divided, the fast and slow category set C = {c1, c2, ...c i , ..., c n}, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector v = [v1, V2, ..., v j , ..., vn], where Only the element at the index position representing the category is 1, and the elements at other positions are all 0.

[0033] Preferably, the feature fusion unit specifically includes:

[0034] The vector concatenation module is used to concatenate the vector v composed of the action speed features and the visual-text feature vector obtained by the video-text encoder to obtain a vector with dimension d. z The vector z;

[0035] The dimension alignment module is used to map the concatenated vector z to a dimension of d through a linear transformation q = Wz + b q The semantic query vector q is ensured to have the dimension of the semantic query vector q aligned with the semantic query of the model.

[0036] The present invention has the following advantages due to the adoption of the above technical solution:

[0037] By combining optical flow analysis, multimodal query and TimeSformer architecture, the present invention can effectively extract the speed characteristics of bird movements and integrate them into the action recognition model. Compared with traditional bird action recognition methods, the present invention can better capture the temporal dynamic information in bird movements, especially when the action frequency changes greatly, and has higher recognition accuracy and robustness. In addition, the fused multimodal features can enhance the semantic understanding ability of the model and further improve the recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 4 is a model structure diagram of a multimodal bird motion recognition method based on motion speed features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0040] In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the scope of protection of the present invention.

[0041] like Figure 1 As shown, the multimodal bird action recognition method based on action speed characteristics provided by the embodiment of the present invention includes:

[0042] Step 1: Calculate the optical flow motion vector between two adjacent frames of the input video.

[0043] In one embodiment, step 1 specifically includes:

[0044] Step 1.1, assuming that the brightness of the image changes with time, and from the image I1(x, y) at time t1 to I2(x, y) at time t2, the displacement of each pixel in the image along the x and y directions of the image is u and v. For each pixel point, the optical flow equation (1) is used to obtain a two-dimensional optical flow vector field. Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point:

[0045]

[0046] in, and is the spatial gradient of the image, is the time gradient.

[0047] Through step 1.1, a two-dimensional optical flow vector field can be obtained. Each vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point. This vector is also called a pixel optical flow vector.

[0048] Step 1.2, calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

[0049] In addition to the above steps 1.1 and 1.2, step 1 can also be implemented by existing methods such as the Lucas-Kanade method and the Horn-Schunck method.

[0050] Step 2: Divide the optical flow motion vector values ​​of step 1 into intervals to obtain multiple fast and slow categories. The principle of "dividing the optical flow motion vector values ​​of step 1 into intervals" can be: [0-10], [10-20], [20-30], [30-40], [40-50], [50-100], [100-180], [180+], which are divided into 8 categories in total. Other methods can also be used for division according to actual conditions.

[0051] Step 3: Convert each speed category in step 2 into a binary vector to obtain the speed feature of the action, which is used as the feature representation of the speed of the action.

[0052] In one embodiment, step 3 specifically includes:

[0053] After the interval is divided, the fast and slow category set C = {c1, c2, ...c i ,,...,c n}, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector v = [v1, V2, ..., v j , ..., v n ],in Only the element representing the index position of the category is 1, and the elements at other positions are all 0. For example, if you encode three categories, dog, cat, and bird, then the categories and corresponding vectors are dog [1, 0, 0], cat [0, 1, 0], and bird [0, 0, 1].

[0054] One-Hot encoding is used to simply and intuitively convert categories into feature vectors.

[0055] Of course, step 3 can also be obtained by other methods, such as label encoding, which directly maps each category to an integer (such as dog=1, cat=2, bird=3).

[0056] Step 4: The action speed feature in step 3 is combined with the visual-text feature to serve as the semantic query part of the model. The visual-text feature is generated by the video-text encoder (e.g. Figure 1The "CLIP" shown in the figure; the "video frame" and "action label" arrows point to the "visual-text encoder", where the arrow means "put in") are extracted from the input video according to the preset action labels. This further enhances the video features extracted by TimeSformer to better complete the task of bird action recognition. In view of the wide range of bird action frequency variations, this step proposes the idea of ​​incorporating action speed features into the bird action recognition model so that the action frequency information can be better utilized during recognition.

[0057] In one embodiment, step 4 specifically includes:

[0058] Step 4.1, concatenate the vector v composed of the action speed features and the visual-text feature vector obtained by the video-text encoder to obtain a vector with dimension d z The vector z.

[0059] Step 4.2, through a linear transformation q = Wz + b, the concatenated vector z is mapped to a dimension d q The semantic query vector q is ensured to have the dimension of the semantic query vector q aligned with the semantic query of the model.

[0060] Step 5: Train the model using public datasets such as the Animal Kingdom dataset: The model includes a video encoder that uses the TimeSformer backbone network, which is a spatiotemporal variant of the Transformer architecture, adding temporal self-attention modules and spatial self-attention modules to the original framework. The TimeSformer backbone network is used to randomly sample the input video, for example, 12 frames of images are extracted from each video, to obtain a video feature vector, which contains the spatiotemporal context information of the video.

[0061] When training the model, the semantic query part of step 4 and the video feature vector are input into the decoder. The decoder uses the semantic query part to guide the decoder so that the decoder can focus on the information of the semantic query part for decoding, convert the vector into the probabilities of different action categories and multi-label classification, and then output the preset number (for example, 5) of action categories with the largest probability and their probabilities, which are the action recognition results. The mean average precision (MAP) of the action recognition results is calculated, and the mean average precision is used as an indicator to evaluate the training effect.

[0062] The embodiment of the present invention further provides a multimodal bird action recognition device based on action speed features, which includes an optical flow motion vector calculation unit, a category division unit, a vector conversion unit, a feature fusion unit and a decoding calculation unit, wherein:

[0063] The optical flow motion vector calculation unit is used to calculate the optical flow motion vector between two adjacent frames of the input video.

[0064] The category division unit is used to divide the optical flow motion vector value of the optical flow motion vector calculation unit into intervals to obtain a plurality of fast and slow categories.

[0065] The vector conversion unit is used to convert each speed category of the category division unit into a binary vector to obtain the speed feature of the action.

[0066] The feature fusion unit is used to fuse the action speed feature of the vector conversion unit with the visual-text feature, and together they serve as the semantic query part of the model; wherein the visual-text feature is extracted from the input video by the video-text encoder according to the preset action label.

[0067] The decoding calculation unit is used for training the model: the semantic query part of the feature fusion unit and the video feature vector are input into the decoder, and the decoder uses the semantic query part to guide the decoder so that the decoder can focus on decoding the information of the semantic query part to obtain the action recognition result, and calculate the average precision mean of the action recognition result. The average precision mean is used as an indicator for evaluating the training effect; wherein, the video feature vector is obtained by randomly sampling the input video through the video encoder.

[0068] In one embodiment, the optical flow motion vector calculation unit specifically includes an optical flow vector calculation module and an optical flow motion vector module, wherein:

[0069] The optical flow vector calculation module is used to set the brightness of the image from I1(x, y) at time t1 to I2(x, y) at time t2, and the displacement of each pixel in the image along the x and y directions is u and v. For each pixel point, the optical flow equation (1) is used to obtain a two-dimensional optical flow vector field. Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point.

[0070] The optical flow motion vector module is used to calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

[0071] In one embodiment, the vector conversion unit specifically includes: after the interval is divided, a fast and slow category set C = {c1, c2, ...c i ,,...,c n}, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector v = [v1, V2, ..., v j , ..., v n ],in Only the element at the index position representing the category is 1, and the elements at other positions are all 0.

[0072] In one embodiment, the feature fusion unit specifically includes a vector concatenation module and a dimension alignment module, wherein:

[0073] The vector concatenation module is used to concatenate the vector v composed of the action speed features and the visual-text feature vector obtained by the video-text encoder to obtain a vector with a dimension of d. z The vector z.

[0074] The dimension alignment module is used to map the concatenated vector z to a dimension of d through a linear transformation q = Wz + b q The semantic query vector q of is ensured to align the dimension of the semantic query vector q with the semantic query part of the model.

[0075] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal bird motion recognition method based on motion speed features, characterized in that: include: Step 1, calculating the optical flow motion vector between two adjacent frames of the input video; Step 2, dividing the optical flow motion vector value in step 1 into intervals to obtain multiple fast and slow categories; Step 3, convert each speed category in step 2 into a binary vector to obtain the speed feature of the action; Step 4, combining the action speed feature of step 3 with the visual-text feature, and using them together as the semantic query part of the model; wherein the visual-text feature is extracted from the input video by the video-text encoder according to the preset action label; Step 5, training model: input the semantic query part and the video feature vector of step 4 into the decoder. The decoder uses the semantic query part to guide the decoder so that the decoder can focus on decoding the information of the semantic query part to obtain the action recognition result, and calculate the average precision mean of the action recognition result. The average precision mean is used as an indicator to evaluate the training effect. Among them, the video feature vector is obtained by randomly sampling the input video by the video encoder.

2. The multimodal bird motion recognition method based on motion speed characteristics as claimed in claim 1, characterized in that: Step 1 specifically includes: Step 1.1, assume that the brightness of the image changes from I1(x, y) at time t1 to I2(x, y) at time t2, and the displacement of each pixel in the image along the x and y directions is u and v. For each pixel point, use the optical flow equation (1) to obtain a two-dimensional optical flow vector field. Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point: in, and is the spatial gradient of the image, is the time gradient; Step 1.2, calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

3. The multimodal bird motion recognition method based on motion speed characteristics as claimed in claim 1, characterized in that: Step 3 specifically includes: After the interval is divided, the fast and slow category set C = {c1, c2, ... c i ,,…,c n }, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector of length n consisting of action speed features ν = [ν1,v2,…,υ j ,…,v n ],in Only the element at the index position representing the category is 1, and the elements at other positions are all 0.

4. The multimodal bird motion recognition method based on motion speed characteristics according to any one of claims 1 to 3, characterized in that: Step 4 specifically includes: Step 4.1, concatenate the vector ν composed of the action speed feature and the visual-text feature vector obtained by the video-text encoder to obtain a dimension d z The vector z; Step 4.2, through a linear transformation q = Wz + b, the concatenated vector z is mapped to a dimension d q The semantic query vector q is ensured to have the dimension of the semantic query vector q aligned with the semantic query of the model.

5. A multimodal bird motion recognition device based on motion speed characteristics, characterized in that: include: An optical flow motion vector calculation unit, which is used to calculate the optical flow motion vector between two adjacent frames of an input video; A category division unit, which is used to divide the optical flow motion vector value of the optical flow motion vector calculation unit according to intervals to obtain a plurality of fast and slow categories; A vector conversion unit, which is used to convert each speed category of the category division unit into a binary vector to obtain a speed feature of the action; A feature fusion unit is used to fuse the action speed feature of the vector conversion unit with the visual-text feature, and use them together as the semantic query part of the model; wherein the visual-text feature is extracted from the input video by the video-text encoder according to the preset action label; A decoding calculation unit is used for training the model: the semantic query part of the feature fusion unit and the video feature vector are input into the decoder, and the decoder uses the semantic query part to guide the decoder so that the decoder can focus on decoding the information of the semantic query part to obtain the action recognition result, and the average precision mean of the action recognition result is calculated. The average precision mean is used as an indicator for evaluating the training effect; wherein, the video feature vector is obtained by randomly sampling the input video through the video encoder.

6. The multi-modal bird motion recognition device based on motion speed characteristics as claimed in claim 5, characterized in that: The optical flow motion vector calculation unit specifically includes: The optical flow vector calculation module is used to set the brightness of the image from I1(x, y) at time t1 to I2(x, y) at time t2, and the displacement of each pixel in the image along the x and y directions is u and v. For each pixel point, a two-dimensional optical flow vector field is obtained using the optical flow equation (1). Each pixel optical flow vector in the optical flow vector field represents the movement direction and speed of the corresponding pixel point: in, and is the spatial gradient of the image, is the time gradient; The optical flow motion vector module is used to calculate the average value of the optical flow vectors of all pixels in each image as the optical flow motion vector of the frame image.

7. The multi-modal bird motion recognition device based on motion speed characteristics as claimed in claim 5, characterized in that: The vector conversion unit specifically includes: After the interval is divided, the fast and slow category set C = {c1, c2, ... c i ,,…,c n }, where n is the total number of fast and slow categories, and each fast and slow category c i The corresponding One-Hot encoding is set to a vector of length n consisting of action speed features ν = [ν1, ν2, ..., υ j ,…,v n ],in Only the element at the index position representing the category is 1, and the elements at other positions are all 0.

8. The multimodal bird motion recognition device based on motion speed characteristics according to any one of claims 5 to 7, characterized in that: The feature fusion unit specifically includes: The vector concatenation module is used to concatenate the vector ν composed of the action speed feature and the visual-text feature vector obtained by the video-text encoder to obtain a vector with a dimension of d. z The vector z; The dimension alignment module is used to map the concatenated vector z to a dimension of d through a linear transformation q = Wz + b q The semantic query vector q is ensured to have the dimension of the semantic query vector q aligned with the semantic query of the model.

Images