A dual-stream global-local action recognition method, system, device and storage medium based on video input

Abstract

A dual-stream global-local action recognition method, system, device and storage medium based on video input, the method comprising: collecting video input of a single identified object behavior, extracting the identified object feature key points using an identified object key point recognition method, and selecting the key points for frame-by-frame cropping in the identified object behavior video to obtain local images of multiple corresponding areas; performing data preprocessing on multiple local video inputs and global video inputs consisting of the identified object local images; using local videos and original video input networks respectively to train a local feature extraction module and a global feature extraction module; adding a local feature enhancement module and a result fusion structure for collaborative training to obtain a global-local action recognition model; performing action recognition; the system, device and medium are used to implement a dual-stream global-local action recognition method with multiple local extractions; the present invention is simple to operate and can improve the effect of the action recognition prediction results obtained by the overall dual-stream global-local action recognition method.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing and understanding, and specifically relates to a dual-stream global-local action recognition method, system, device and storage medium based on video input. Background Art

[0002] Human action recognition technology primarily encompasses three aspects: human target recognition, human tracking, and action recognition. Action recognition is a higher-level computer vision component based on the first two. Developing a robust action recognition algorithm has significant theoretical significance and broad application prospects, including intelligent video surveillance and video retrieval. To reduce the interference of redundant background information and learn about human dynamics in videos, many methods fuse information from multiple modalities for recognition. Additionally, some methods combine global and local information within the video to better accomplish the recognition task. Global-local methods involve capturing local parts of the human body within the video for action recognition. StNet concatenates N consecutive frames of images along the RGB channel dimension to form a global representation of the video, called a hypergraph. Local spatiotemporal features are extracted from the hypergraph, and then combined to extract global spatiotemporal features in the temporal dimension. Attention-based methods use an attention mechanism to emphasize local information within the video as a branch, fusing it with the global network through a softmax layer to integrate global and local features.

[0003] Patent application CN113761992A discloses a video action recognition method, including: acquiring a video; inputting the video into the hidden layer of a neural network model for processing to obtain the recognized object in the video and the motion corresponding to the recognized object, wherein the hidden layer of the neural network model includes multiple processing units; and based on the recognized object in the video and the motion corresponding to the recognized object, outputting the action recognition result of the video; wherein, in at least one processing unit, sequentially extracting spatial features and temporal features in the video as the input of at least one processing unit, combining the spatial features and temporal features in the video, and performing point-by-point convolution operations to output the spatial semantic information and temporal semantic information of the video. According to the technical solution provided by the application, the video processing process has a stronger ability to encode spatiotemporal relationships, and can extract more meaningful features with a smaller number of parameters, so that more useful information can be learned from the data set using a more compact structure. In the process of implementing video processing, only a single processor is required to process the amount of video that previously required multiple processors to process.

[0004] The above-mentioned existing behavior recognition methods involve processing a single video input and obtaining prediction results from the processed temporal and spatial features, without deeply exploring the more detailed spatial feature information in the video image. In addition, some methods use multimodal fusion methods to characterize global features using a two-stream or even multi-stream model with the same structure in parallel, ignoring a large amount of fine-grained local information. The method using local interception emphasizes identifying the local part of the object to obtain dynamic information, while ignoring global information and some useful fine-grained information while identifying the local part. Global-local methods based on image stitching and attention mechanisms emphasize local information, but the local fine-grained information is lost in the preprocessing process before feature extraction. Summary of the Invention

[0005] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a dual-stream global-local action recognition method, system, device and storage medium based on video input, which can recognize the global information of the object action in the learning video while paying attention to the local fine-grained feature information, and can obtain the local detail information of the recognized object from the local video input obtained by the local interception operation, and combine the features of different local areas to obtain more video dynamic local fine-grained information to obtain better global features.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A dual-stream global-local action recognition method based on video input specifically comprises the following steps:

[0008] Step 1: Collect video input of a single object being recognized, extract key points of the object in the video using a recognition object key point recognition method, select the extracted key points, and crop them according to the key point positions to obtain multiple local images of the object; perform data preprocessing on the multiple local video inputs and the global video input composed of the multiple local images of the object being recognized;

[0009] Step 2: A dual-stream global-local action recognition network is used to extract action features from the global and local videos; that is, a local feature extraction network and a global feature extraction network are used to perform feature extraction operations on the input local video and original video data respectively;

[0010] Step 3: Add a local feature enhancement module to process the intermediate output feature map containing local information in the local network to obtain an attention-guided mask, and use the attention-guided mask to enhance the intermediate results of the global network in the spatial and temporal dimensions;

[0011] Step 4: Co-train the dual-stream network and the enhancement module to obtain a global-local action recognition model for action recognition.

[0012] The specific method of step 1 is as follows:

[0013] 1.1) Using a keypoint recognition network to identify multiple keypoints of a single identified object from the captured input video, a local image centered around the keypoint is cropped based on the identified keypoints.

[0014] 1.2) The key point data of the identified object is set to J i =(x i ,y i ),i=1...clip size , J i Indicates the position of the key point of the identified object in an image, and sets the local image centered on the key point as a box with a side length of len Square crop box , get the clip of each frame size To identify the key points of the object J i The shape at the center is a box len ×box len A local image of the key point center:

[0015] crop box =(x i -box len / 2,y i -box len / 2),(x i +box len / 2,y i +box len / 2)

[0016] From each frame, we can get G local images I local :

[0017]

[0018] Among them, the crop function represents the part where the crop_box is located from the input image, the function P(·) represents the data preprocessing operation, and the local video input composed of the local image is local The formal definition is as follows:

[0019]

[0020] Input

[0021] The specific method of step 2 is as follows:

[0022] 2.1) The global feature extraction network extracts the action features in the global video and obtains the prediction results;

[0023] 2.2) Local feature extraction network extracts action features in local videos;

[0024] 2.2.1) Segment the video sequence based on the G key points of different parts of the identified object in the video frame of the local video obtained by step 2 preprocessing;

[0025] 2.2.2) Using the result of step 2.2.1) as input data, perform local network feature extraction: Split the input data into G groups of local data, perform network feature extraction on each of them, and then merge them; that is, treat the input data as a combination of the G groups of local data, and the local module can process the G groups of local data separately;

[0026] 2.2.3) After the network feature extraction in step 2.2.2) is completed, the data is divided into G groups, and the prediction results are obtained separately and averaged to obtain the final prediction result.

[0027] The specific method of step 3 is as follows:

[0028] 3.1) Obtain local average features; the intermediate feature map of the output of the global network is The intermediate feature map of the output of the local feature extraction network is L2 out ,

[0029] First, the output feature map of the local feature extraction network is processed to obtain the local average features and the features are divided into groups in the filter dimension;

[0030] Then, respectively The feature maps are averaged to obtain the local average feature map:

[0031]

[0032] 3.2) Time series alignment; in the process of feature extraction, after downsampling, feature L2 out Compressed to T2 in the time dimension T, feature Full L1out It is compressed to T1 in the time dimension, and the frame_size dimension is copied in the time dimension to obtain Make the temporal dimension of the generated attention-guided mask consistent with the target features of the guided full-frame flow Figure 1 To align the local clip_size average features, perform time series alignment;

[0033] 3.3) Building Attention-Guided Masks: Using L2 out_3 Building spatiotemporal attention-guiding masks,

[0034] First, create aFull_C1_out Empty mask with the same time and space dimensions empty , According to the clipping frame when the image data in the local video sequence is obtained, the local average feature is calculated to restore the shape in the guidance mask trans ∈R h1×w1 ;

[0035] Then, scale the shape of the local average feature from 16×16 to h1×w1;

[0036] Finally, in the spatial dimension, the G local feature maps are integrated into the mask of a frame according to the relative position of the local features in this frame. In the temporal dimension, according to the relative position of the local features that change in different frames, an attention-guided mask with the same shape as the target full-frame flow feature map is established frame by frame. attention , guide mask Mask attention It represents the fusion of the local network's understanding of G groups of local spatial information in the time dimension.

[0037] The specific method of step 4 is as follows:

[0038] After obtaining the attention guidance mask, the mask is multiplied element-wise with the feature map output by the global network Conv1 to obtain the guided feature

[0039]

[0040] After adding the local feature enhancement module, the global feature extraction module and the local feature extraction module are trained collaboratively to obtain the final multi-local extraction dual-stream global-local action recognition network.

[0041] A dual-stream global-local action recognition system based on video input, comprising a recognition object key point feature extraction module, a data preprocessing module and a recognition module;

[0042] The recognition object key point feature extraction module uses the recognition object key point recognition method to extract the recognition object key points in the video and select the extracted recognition object key points, and cuts the recognition object parts according to the key point positions to obtain multiple recognition object parts as input to the local feature extraction module;

[0043] The data preprocessing module uses the cropping and flipping methods commonly used in the field to preprocess the input data globally and locally.

[0044] The recognition module includes a local feature extraction module, a global feature extraction module and a local feature enhancement module; the local feature extraction module is constructed using the local bottleneck structure proposed in the present invention as a basic module, and simultaneously processes multiple local inputs obtained by the data preprocessing module, and finally summarizes and obtains the recognition result; the global feature extraction module processes the global features to obtain the recognition result; the local feature enhancement module is trained collaboratively with the global feature extraction module and the local feature extraction module to obtain a global-local action recognition model; action recognition is performed using the above method.

[0045] A dual-stream global-local action recognition device based on video input, comprising:

[0046] memory for storing computer programs;

[0047] A processor is used to implement the dual-stream global-local action recognition method based on video multi-local extraction described in steps 1 to 4 when executing the computer program.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the dual-stream global-local action recognition method with multi-local extraction described in steps 1 to 4.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] First, the present invention, combined with a local cropping operation, can automatically locate the positions of multiple selected key parts of the identified object in each frame of the input video, eliminating the need for additional manual labeling of key parts of the identified object and reducing the input data requirements, requiring only the original video input. The localized video image of the identified object obtained using the local cropping operation can better represent the local details of the identified object than the video image fed into the global network.

[0051] Second, in the dual-stream network of the present invention, the local feature extraction network can simultaneously process multiple processed input video images of the local part of the recognition object, reducing the time and resource consumption of feature extraction in video recognition. The local video image obtained by the local interception operation is predicted after feature extraction, benefiting from more local detailed visual information, and can obtain better prediction results;

[0052] Third, the local feature enhancement module incorporated into the global-local dual-stream network of the present invention aggregates the features of multiple key parts of the identified object in the local feature extraction module in the spatiotemporal domain to produce an attention-guiding mask. This module enhances the capabilities of the global feature extraction module, thereby improving the performance of the action recognition prediction results obtained by the overall dual-stream global-local action recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the dual-stream global-local action recognition method based on multi-local extraction of video according to an embodiment of the present invention.

[0054] Figure 2 This is a comparison of the global feature extraction network and local feature extraction network models described in an embodiment of the present invention.

[0055] Figure 3 The bottleneck structure is used to compare the global feature extraction network and the local feature extraction network model described in the embodiment of the present invention. Figure 3 (a) is the global ResNeXt bottleneck structure, Figure 3 (b) is the local bottleneck structure.

[0056] Figure 4 This is a flow chart of the dual-stream global-local action recognition method based on multi-local extraction of video according to an embodiment of the present invention.

[0057] Figure 5 Schematic diagram of the structure of the system according to an embodiment of the present invention.

[0058] Figure 6 This is a schematic diagram of the structure of the storage medium described in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0060] Example:

[0061] like Figure 1 As shown, taking human motion behavior as an example, this embodiment provides a dual-stream global-local motion recognition method based on video input, including the following steps:

[0062] S1. Collecting a video input of a single human action, extracting and selecting human key points from the video using a human key point recognition method, and cropping the extracted human key points according to the key point positions to obtain multiple human partial images; performing data preprocessing on the multiple local video inputs and the global video input composed of the multiple human partial images;

[0063] More specifically:

[0064] From the captured video of a single human action, select frame_size frames as the recognition target. Using the HRNet network as a keypoint identifier, we extract seven key points of the person in the video: the nose tip, left and right shoulders, left and right elbows, and the tips of the middle fingers of both hands. The cropping box (box_len) is set to the average length of 1.5 times the distance from the nose tip to the left shoulder across all frames of the video. Centered on the keypoint, this box is processed to produce a square image with a side length of box_len. To ensure that the local feature extraction module requires eight partial images as input, the facial image centered on the nose tip is copied. This ultimately yields the positional information for the eight required partial images.

[0065] The key point recognition network is used to identify multiple key points of the human body from the video of a single human action obtained by the acquisition input. Based on the identified key points of the human body, the local image centered on the key points is cropped. The human key point data is set as J i =(x i ,y i ),i=1...clip size .J i Indicates the position of the key points of the human body in an image. Set the local image centered at the key point as a box with a side length of len Square crop box , get the clip of each frame size Key points of the human body i A local image centered at the keypoint of shape box_len×box_len.

[0066] crop box =(x i -box len / 2,y i -box len / 2),(x i +box len / 2,y i +box len / 2)

[0067] From each frame, G local images I_local can be obtained:

[0068] I_local=P(crop(I k ,crop_box)),I local ∈R 3×128×128

[0069] The crop function crops the portion of the input image where crop_box is located, and the function P(·) represents the data preprocessing operation, specifically adjusting the shape and size of the input image to 3×box_len×box_len. The local video input Input_local composed of the local image can be formally defined as follows:

[0070]

[0071] Input

[0072] like Figure 4 As shown in , taking a frame image with a shape of 512×512 in the input video as an example, the average distance from the tip of the nose to the left shoulder of a person in the video is 107 pixels. We cut out a 160×160 part from the image to represent the facial video information of the person, and the local information corresponding to the other 7 predefined human body points is the same. Due to the need to control the number of parameters of the model, the input of the ResNeXt network is converted into a set of pictures with a shape of 128×128 after data preprocessing. A simple calculation shows that the global feature extraction network is first-class, and the local information of the frame can only be retained by 25% after data preprocessing. The local feature extraction network can learn image data containing 80% of local detail information. From Figure 4 The difference in the amount of information contained in the full-frame input and the local information input can also be seen intuitively.

[0073] S2. Using a dual-stream global-local action recognition network to extract action features from global and local videos; that is, using a local feature extraction network and a global feature extraction network to perform feature extraction operations on the input local video and original video data;

[0074] More specifically:

[0075] The global feature extraction network uses the ResNeXt network to extract the structure of the global video, including a convolutional layer and four bottleneck blocks containing residual structures, such as Figure 3 As shown in .

[0076] The input used by the local feature extraction network comes from the original video data. Each set of data is a video sequence segmented from the video frame with the human body key points as the center. The local feature extraction network structure design is as follows Figure 3 As shown in , the Conv1 layer of ResNeXt is also set to a group convolution layer with a group number of G, and the 1×1 convolution and 3×3 convolution in the bottleneck structure are also set to a group convolution layer with a group number of G. Figure 3As can be seen from the figure, this design ensures that the model can learn different groups of local features separately during the learning process. The result of Layer4 is passed through the AdaptiveAvgPool3d layer to obtain a 2048-dimensional prediction vector, which represents the recognition results of clip_size local parts. It is split into clip_size×(2048 / clip_size) and then sent to the fc layer to obtain the prediction results of each part. Finally, the clip_size group results are averaged. The specific parameter information of the global feature extraction network and the local feature extraction network is as follows: Figure 2 As shown in .

[0077] S3. Add the local feature enhancement module and the result fusion structure for collaborative training to obtain a global-local action recognition model, and use this model for action recognition.

[0078] More specifically:

[0079] The output of the global network Conv1 is enhanced in both spatial and temporal dimensions using the feature maps of the local network's intermediate output containing local information. The local feature enhancement module, as shown in the figure, consists of three parts: 1. Obtaining local average features, 2. Time series alignment, and 3. Creating an attention-guiding mask.

[0080] The features of the Conv1 output of the global network are:

[0081]

[0082] The characteristics of layer 2 output of the ResNeXt-Local network are:

[0083]

[0084] First, the features are divided into clip_size groups in the filter dimension, and then 512 / clip_size feature maps are averaged to obtain the local average feature map Then time series alignment is required. Due to downsampling during feature extraction, the feature L2_out is compressed from frame_size to The feature Full_L1_out is compressed in the time dimension to In order to make the temporal dimension of the generated attention-guided mask consistent with the target features of the guided full-frame flow, Figure 1 Therefore, copy the frame_size dimension in the time dimension and get Finally, using L2 out_3 Building spatiotemporal attention-guiding masks.

[0085] There are three steps to build a spatiotemporal attention mask. First, build an empty mask with the same temporal and spatial dimensions as Full_C1_out. The shape in the guide mask is restored by calculating the local average feature based on the clipping frame when the local image data is acquired. trans ∈R h1×w1 Then, the shape of the local average feature is scaled from 16×16 to h1×w1. Finally, in the spatial dimension, the G local feature maps are integrated into the mask of one frame according to the relative position of the local features in this frame. In the temporal dimension, according to the relative position of the local features that changes in different frames, an attention-guided mask with the same shape as the target full-frame flow feature map is established frame by frame. attention . Guide Mask attention It represents the fusion of the local network's understanding of G groups of local spatial information in the time dimension.

[0086] S4. After obtaining the attention guidance mask, perform element-wise multiplication of the mask with the feature map output by the global network Conv2 to obtain the guided feature Att_Full_C1_out.

[0087]

[0088] After adding the local feature enhancement module, the global feature extraction module and the local feature extraction module are trained collaboratively to obtain the final multi-local extraction dual-stream global-local action recognition network.

[0089] In particular, the network structure in this embodiment can also be implemented using other network structures that can achieve the same technical effect, such as Figure 2 、 Figure 3 The global feature extraction network ResNeXt in can be any form of feature extraction network, and the local feature extraction network ResNeXt-Local can be any form of feature extraction network that performs split-feature extraction-merge operations on input features.

[0090] like Figure 5 As shown, a dual-stream global-local action recognition system based on video input includes a human key point feature extraction module, a data preprocessing module and a recognition module;

[0091] The human body key point feature extraction module uses a human body key point recognition method to extract and select the extracted human body key points in the video, and cuts out multiple human body parts according to the key point positions as input to the local feature extraction module;

[0092] The data preprocessing module uses the cropping and flipping methods commonly used in the field to preprocess the input data globally and locally;

[0093] The recognition module includes a local feature extraction module, a global feature extraction module and a local feature enhancement module. The local feature extraction module is constructed using the local bottleneck structure proposed in the present invention as a basic module, and simultaneously processes multiple local inputs obtained by the data preprocessing module, and finally summarizes and obtains the recognition result; the global feature extraction module processes the global features to obtain the recognition result; the local feature enhancement module is trained collaboratively with the global feature extraction module and the local feature extraction module to obtain a global-local action recognition model; and action recognition is performed using the above method.

[0094] It should be noted here that the above system is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. This system is a first-person perspective action recognition method applied to the above embodiment.

[0095] A dual-stream global-local action recognition device based on video input includes: a memory for storing a computer program;

[0096] A processor is configured to implement the dual-stream global-local action recognition method based on video multi-local extraction as described in any one of claims 1 to 5 when executing the computer program.

[0097] like Figure 6 As shown, a storage medium stores a program. When the program is executed by a processor, the first-person perspective action recognition method of the above embodiment is implemented, specifically:

[0098] The human body key points in the video are extracted and selected using the human body key point recognition method, and multiple human body parts are obtained according to the key point positions as the input of the local feature extraction module;

[0099] Use the cropping and flipping methods commonly used in the field to preprocess the input globally and locally;

[0100] The local feature extraction module is constructed using the local bottleneck structure proposed in the present invention as a basic module, and simultaneously processes multiple local inputs obtained by the data preprocessing module, and finally summarizes and obtains the recognition result; the global feature extraction module processes the global features to obtain the recognition result; the local feature enhancement module and the global feature extraction module and the local feature extraction module are trained collaboratively to obtain a global-local action recognition model; action recognition is performed using the above method.

[0101] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0102] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A dual-stream global-local action recognition method based on video input, characterized by: The specific steps include: Step 1: Collect video input of a single object being recognized, extract key points of the object in the video using a recognition object key point recognition method, select the extracted key points, and crop them according to the key point positions to obtain multiple local images of the object; perform data preprocessing on the multiple local video inputs and the global video input composed of the multiple local images of the object being recognized; Step 2: A dual-stream global-local action recognition network is used to extract action features from the global and local videos; that is, a local feature extraction network and a global feature extraction network are used to perform feature extraction operations on the input local video and original video data respectively; Step 3: Add a local feature enhancement module to process the intermediate output feature map containing local information in the local network to obtain the attention-guided mask, and use the attention-guided mask to enhance the intermediate results of the global network in the spatial and temporal dimensions; the attention-guided mask established frame by frame attention The shape of the target full-frame flow feature map is the same, and G local images I can be obtained in each frame local , the obtained attention guide mask Mask attention It represents the fusion of the local network's understanding of the local spatial information of group G in the time dimension, and multiplies the mask element-wise with the feature map output by the global network Conv1 to obtain the guided feature Step 4: Co-train the dual-stream network and the enhancement module to obtain a global-local action recognition model for action recognition.

2. The dual-stream global-local action recognition method based on video input according to claim 1, characterized in that The specific method of step 1 is as follows: 1.1) Using a keypoint recognition network to identify multiple keypoints of a single identified object from the captured input video, a local image centered around the keypoint is cropped based on the identified keypoints. 1.2) The key point data of the identified object is set to J i =(x i ,y i ),i=1...clip_size,J i Indicates the position of the key point of the object to be identified in an image, sets the local image centered on the key point to a square crop_box with a side length of box_len, and obtains clip_size key points J of each frame to identify the object i The local image of the key point center with a shape of box_len×box_len as the center is crop_box=(x i -box_len / 2,y i -box_len / 2),(x i +box_len / 2,y i +box_len / 2) From each frame, we can get G local images I local : I local =P(crop(I k ,crop_box)),I local ∈R 3×box_len×box_len The crop function represents cropping the portion of the input image where the crop_box is located, the function P(·) represents the data preprocessing operation, and the local video input Input_local composed of the local image can be formally defined as follows: Input_local∈R frame_size,(3×G),box_len,box_len .

3. The dual-stream global-local action recognition method based on video input according to claim 1, characterized in that The specific method of step 2 is as follows: 2.1) The global feature extraction network extracts the action features in the global video and obtains the prediction results; 2.2) Local feature extraction network extracts action features in local videos; 2.2.1) Segment the video sequence based on the G key points of different parts of the identified object in the video frame of the local video obtained by step 2 preprocessing; 2.2.2) Using the result of step 2.2.1) as input data, perform local network feature extraction: Split the input data into G groups of local data, perform network feature extraction on each of them, and then merge them; that is, treat the input data as a combination of the G groups of local data, and the local module can process the G groups of local data separately; 2.2.3) After the network feature extraction in step 2.2.2) is completed, the data is divided into G groups, and the prediction results are obtained separately and averaged to obtain the final prediction result.

4. The dual-stream global-local action recognition method based on video input according to claim 2, characterized in that The specific method of step 3 is as follows: 3.1) Obtain local average features; the intermediate feature map of the global network output is Full C1_out , The intermediate feature map of the output of the local feature extraction network is L2 out , First, the output feature map of the local feature extraction network is processed to obtain the local average features and the features are divided into groups in the filter dimension; Then, respectively The feature maps are averaged to obtain the local average feature map 3.2) Time series alignment; in the process of feature extraction, after downsampling, feature L2 out Compressed to T2 in the time dimension T, feature Full L1_out It is compressed to T1 in the time dimension, and the frame_size dimension is copied in the time dimension to obtain Make the temporal dimension of the generated attention-guided mask consistent with the target feature map of the guided full-frame stream, and perform time series alignment on the local clip_size average features; 3.3) Building Attention-Guided Masks: Using L2 out_3 Building spatiotemporal attention-guiding masks, First, create a C1_out Empty mask with the same time and space dimensions empty , According to the clipping frame when the image data is acquired in the local video sequence, the local average feature is calculated to restore the shape in the guidance mask Then, scale the shape of the local average feature from 16×16 to H1×W1; Finally, in the spatial dimension, according to the relative position of the local features in this frame, the G local feature maps are integrated into the mask of one frame. In the temporal dimension, according to the relative position of the local features that change in different frames, an attention-guided mask with the same shape as the target full-frame flow feature map is established frame by frame. attention , attention-guided mask Mask attention It represents the fusion of the local network's understanding of G groups of local spatial information in the time dimension.

5. The dual-stream global-local action recognition method based on video input according to claim 4, characterized in that The specific method of step 4 is as follows: After obtaining the attention-guided mask, the mask is combined with the feature map Full output by the global network Conv1 C1_out Element-wise dot multiplication to obtain guided features After adding the local feature enhancement module, the global feature extraction module and the local feature extraction module are trained collaboratively to obtain the final multi-local extraction dual-stream global-local action recognition network.

6. A dual-stream global-local action recognition system based on video input, characterized in that: Used to implement the dual-stream global-local action recognition method based on any one of claims 1 to 5, including a recognition object key point feature extraction module, a data preprocessing module and a recognition module; The recognition object key point feature extraction module uses the recognition object key point recognition method to extract the recognition object key points in the video and select the extracted recognition object key points, and cuts the recognition object parts according to the key point positions to obtain multiple recognition object parts as input to the local feature extraction module; The data preprocessing module uses the cropping and flipping methods commonly used in the field to preprocess the input data globally and locally; The recognition module includes a local feature extraction module, a global feature extraction module and a local feature enhancement module; the local feature extraction module is constructed using a local bottleneck structure as a basic module, and simultaneously processes multiple local inputs obtained by a data preprocessing module, and finally summarizes and obtains a recognition result; the global feature extraction module processes global features to obtain a recognition result; the local feature enhancement module is trained collaboratively with the global feature extraction module and the local feature extraction module to obtain a global-local action recognition model.

7. A dual-stream global-local action recognition device based on video input, characterized in that: include: memory for storing computer programs; A processor is configured to implement the dual-stream global-local action recognition method based on video multi-local extraction as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the dual-stream global-local action recognition method with multi-local extraction described in any one of claims 1 to 5 is implemented.

Images