An action recognition method, device, apparatus and storage medium

Abstract

The application relates to the technical field of computer vision, and discloses a motion recognition method, device and equipment and a storage medium, which comprises the following steps: acquiring a target image that needs to be subjected to motion recognition; dividing the target image as a whole into a plurality of first image blocks, and performing feature extraction to obtain global visual features; determining a local region where an object to be recognized is located in the target image, dividing the local region into a plurality of second image blocks, and performing feature extraction to obtain local visual features; fusing the global visual features and the local visual features to obtain motion representation features of the target image; and performing motion recognition according to the motion representation features to obtain a motion recognition result of the object to be recognized. The method is suitable for video understanding, intelligent monitoring and human-computer interaction scenes, and through extraction and fusion of the global visual features and the local visual features, the accuracy and robustness of motion recognition are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to an action recognition method, apparatus, device, and storage medium. Background Technology

[0002] Traditional action recognition methods primarily rely on convolutional neural networks or visual Transformers to extract features from images, thereby completing the action classification task. Currently, the mainstream methods for action recognition mainly fall into the following categories: First, methods based on global modeling, which uniformly encode the entire image to obtain global features. However, these methods pay insufficient attention to local key action regions, making it difficult to characterize fine-grained action information. Second, methods based on local regions, which typically obtain people or key regions through object detection or region extraction before performing action recognition. However, these methods lack global contextual information and are prone to semantic ambiguity in complex scenes. Third, multi-scale fusion methods, which integrate information at different scales through feature pyramids or multi-layer feature fusion. However, their fusion methods are usually coarse, lacking an effective correlation mechanism between global and local features. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and storage medium for action recognition, which can improve the accuracy and robustness of human behavior recognition.

[0004] Firstly, an action recognition method is provided, including: Acquire the target image that needs to be performed on action recognition; The target image is divided into multiple first image patches, and features are extracted from multiple first image patches to obtain the global visual features of the target image. The local region where the object to be identified is located in the target image is determined, and the local region is divided into multiple second image blocks. Feature extraction is performed on multiple second image blocks to obtain the local visual features of the target image. By fusing global and local visual features, action representation features of the target image are obtained; Action recognition is performed based on action representation features to obtain the action recognition result of the object to be recognized.

[0005] Optionally, in one embodiment, determining the local region where the object to be identified is located in the target image includes: The target image is detected using a target detection model to obtain the target detection result; Based on the target detection results, the local region where the object to be identified is located in the target image is determined.

[0006] Optionally, in another embodiment, determining the local region in the target image where the object to be identified is located includes: Get the preset object template; The local region where the object to be identified is located in the target image is determined based on the object template.

[0007] Optionally, in one embodiment, global visual features and local visual features are fused to obtain action representation features of the target image, including: By performing cross-attention fusion on global and local visual features, action representation features of the target image are obtained.

[0008] Optionally, in one embodiment, cross-attention fusion is performed on global visual features and local visual features to obtain action representation features of the target image, including: Project local visual features into query features; Project global visual features into key features and value features; Cross-attention processing is performed on query features, key features, and value features to obtain action representation features.

[0009] Optionally, in one embodiment, cross-attention processing is performed on query features, key features, and value features to obtain action representation features, including: Calculate attention weights based on the similarity between query features and key features; The action representation features are obtained by weighting and aggregating the value features based on attention weights.

[0010] Optionally, in one embodiment, action recognition is performed based on action representation features to obtain the action recognition result of the object to be recognized, including: Residual connection features are obtained by performing residual connection between action representation features and local visual features. By mapping the residual connection features to the same feature dimension as the action representation features, we obtain the enhanced action representation features. Action recognition is performed based on enhanced action representation features to obtain the action recognition result of the object to be recognized.

[0011] Secondly, an action recognition device is provided, comprising: The image acquisition module is used to acquire the target image that needs to be performed on action recognition. A global processing module is used to divide the target image into multiple first image blocks and extract features from the multiple first image blocks to obtain the global visual features of the target image. The local processing module is used to determine the local region where the object to be identified is located in the target image, divide the local region into multiple second image blocks, and extract features from the multiple second image blocks to obtain the local visual features of the target image. The feature fusion module is used to fuse the global visual features and the local visual features to obtain the action representation features of the target image; The action recognition module is used to perform action recognition based on the action representation features to obtain the action recognition result of the object to be recognized.

[0012] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described action recognition method.

[0013] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described action recognition method.

[0014] The aforementioned action recognition method, apparatus, device, and storage medium implement the following scheme: The target image to be recognized is divided into multiple first image blocks, and features are extracted from these blocks to obtain global visual features of the target image; the local region where the object to be recognized is located in the target image is determined, and this local region is divided into multiple second image blocks, and features are extracted from these blocks to obtain local visual features of the target image; the global and local visual features are fused to obtain action representation features of the target image; and action recognition is performed based on these action representation features to obtain the action recognition result of the object to be recognized. Therefore, this invention, through the extraction and fusion of global and local visual features, significantly enhances the model's ability to jointly perceive global context and local action regions in complex scenes, compared to existing methods that rely solely on local or global features, thereby improving the accuracy and robustness of human action recognition. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of an action recognition process provided in an embodiment of this application; Figure 2 yes Figure 1 A schematic diagram of a specific implementation method for step S103; Figure 3 yes Figure 1 A flowchart illustrating another specific implementation of step S103; Figure 4 yes Figure 1 A schematic diagram of a specific implementation of step S104; Figure 5 yes Figure 4 A schematic diagram of a specific implementation of step S401; Figure 6 yes Figure 5 A schematic diagram of a specific implementation method for step S503; Figure 7 yes Figure 1 A schematic diagram of a specific implementation of step S105; Figure 8 This is a schematic diagram of the action recognition device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Detailed Implementation To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0017] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0018] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0020] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1 As shown, Figure 1 A flowchart illustrating an action recognition method provided in an embodiment of the present invention includes the following steps: S101: Acquire the target image that needs to be performed on action recognition; S102: Divide the target image into multiple first image blocks and extract features from the multiple first image blocks to obtain the global visual features of the target image; S103: Determine the local region where the object to be identified is located in the target image, divide the local region into multiple second image blocks, and extract features from the multiple second image blocks to obtain the local visual features of the target image; S104: Fusion of global and local visual features to obtain action representation features of the target image; S105: Perform action recognition based on action representation features to obtain the action recognition result of the object to be recognized.

[0024] The target image is an image containing the actions of the object to be identified, which is used for subsequent global feature extraction, local region determination and action recognition processing.

[0025] As an example, in step S101, the device can acquire image frames in real time via a camera, extract keyframes from a video stream, or read pre-saved images from local storage and use them as the target image. The methods for acquiring the target image include, but are not limited to, the methods described above, and can be selected according to the actual application scenario.

[0026] The first image block is an image block unit obtained by uniformly dividing the target image into grids of a preset size, and is used to extract global scene information. The global visual features are a set of features obtained by extracting features from multiple first image blocks, and are used to characterize the overall scene context of the target image.

[0027] As an example, in step S102, the device divides the target image into multiple first image blocks and extracts features from these blocks to obtain the global visual features of the target image. The device can use a preset grid division method to uniformly divide the target image into multiple first image blocks according to a preset pixel size. The device inputs each first image block into a visual encoder for feature extraction, obtaining a feature vector corresponding to each first image block. The device then combines the feature vectors corresponding to all first image blocks to obtain the global visual features of the target image.

[0028] The object to be identified is the main subject in the target image that requires action recognition, including people, animals, or moving objects. The second image patch is an image block unit obtained by uniformly dividing a local region into grids of preset sizes, used to extract local action details. Local visual features are a set of features obtained by extracting features from multiple second image patches, used to characterize the action details of the object to be identified within the local region.

[0029] As an example, in step S103, after determining the local region where the object to be identified is located, the device divides the local region into multiple second image blocks and extracts features from the multiple second image blocks to obtain the local visual features of the target image. The device can use an object detection algorithm or a priori method to determine the local region where the object to be identified is located in the target image. The object detection algorithm can use object detection models such as YOLO or Faster R-CNN to detect the object to be identified in the target image, generate a bounding box containing the object to be identified, and determine the region defined by the bounding box as the local region. The priori method can be based on a preset object template, using a template matching algorithm to search for regions in the target image that match the object template, and determine these regions as local regions. The device crops out this local region from the target image to obtain a local region image containing the object to be identified. The device divides this local region image into a uniform grid according to a preset size to obtain multiple second image blocks. The division size of the second image blocks is consistent with the division size of the first image blocks, but other sizes can also be selected according to actual needs; this embodiment does not limit this. The device inputs each second image block into a visual encoder for feature extraction, obtaining the feature vector corresponding to each second image block. The feature vectors corresponding to all second image blocks are combined to obtain the local visual features of the target image.

[0030] Among them, action representation features refer to the joint feature representation obtained by fusing global visual features and local visual features, which simultaneously contains the overall scene context of the target image and the local action details of the object to be identified.

[0031] As an example, in step S104, after obtaining the global visual features and local visual features, the device needs to fuse the global visual features and local visual features to obtain the action representation features of the target image. The device can use a feature stitching method to stitch the global visual features and local visual features along the feature dimension to obtain the stitched fused features; or a weighted summation method to assign weight coefficients to the global visual features and local visual features respectively, and then sum them according to the weight coefficients to obtain the weighted fused features; or an attention mechanism to calculate the attention weights between the global visual features and local visual features, and then interact and fuse them according to the attention weights to obtain the attention-fused features. The device determines the fused features as the action representation features of the target image. In this example, the fusion method of the global visual features and local visual features can be selected according to actual needs, and this application embodiment does not limit this.

[0032] The action recognition result is the output result obtained after recognizing the action category of the object to be recognized, which may include action category label or action category probability distribution.

[0033] As an example, in step S105, after obtaining the action representation features, the device performs action recognition based on the action representation features to obtain the action recognition result of the object to be recognized. The device inputs the action representation features into the recognition network for processing and outputs the action recognition result corresponding to the object to be recognized. It should be noted that the recognition network can use a fully connected layer combined with a softmax function to map the action representation features to a probability distribution of each action category, and determine the action category corresponding to the maximum probability as the action recognition result. In this example, the type of action recognition task and the structure of the recognition network can be selected according to actual needs, and this application embodiment does not limit this.

[0034] In this embodiment, global visual features are first obtained through global image block segmentation and feature extraction, followed by local visual features through local region segmentation and feature extraction. Finally, the two are fused to obtain action representation features for action recognition. Compared to methods using only single-scale features for action recognition, this embodiment, through global and local dual-branch feature extraction and fusion, can simultaneously capture the overall scene context of the target image and the details of key action parts of the object to be recognized, thereby enhancing the completeness and discriminative power of the action representation features. Furthermore, by independently segmenting local regions into image blocks and extracting features, the problem of action details being diluted by background information in global grid segmentation is avoided, making the response of the action representation features to key action parts more concentrated, thus improving the accuracy and robustness of action recognition.

[0035] In one embodiment, such as Figure 2 As shown, in step S103, determining the local region where the object to be identified is located in the target image includes: S201: The target image is detected using a target detection model to obtain the target detection result; S202: Based on the target detection results, determine the local region where the object to be identified is located in the target image.

[0036] The object detection model is a pre-trained deep learning model used to detect target objects in an image. It can employ object detection algorithms such as YOLO, Faster R-CNN, or SSD, or instance segmentation algorithms such as Mask R-CNN; this embodiment does not impose any limitations on this. The object detection result is the output of the object detection model after detecting the target image, including the location coordinates of the object detection box, the confidence score, and the category information.

[0037] As an example, in step S201, the device inputs the target image into the target detection model, which then detects the object to be identified in the target image and outputs a target detection result containing the object to be identified. It should be noted that the target detection model performs forward inference on the target image, generating multiple candidate detection boxes in the image. Each candidate detection box undergoes category classification and location regression, and detection boxes whose category is the object to be identified and whose confidence score is higher than a preset threshold are selected and output as the target detection result. In this example, the preset threshold can be adjusted according to the actual application scenario, and this embodiment does not impose any limitations on it.

[0038] As an example, in step S202, the device determines the local region where the object to be identified is located from the target image based on the position coordinates of the target detection box in the target detection result. The device defines the rectangular region bounded by the target detection box as the local region where the object to be identified is located, and this local region covers the complete outline of the object to be identified. If the target detection result contains multiple target detection boxes corresponding to multiple objects to be identified, the region bounded by each target detection box is defined as the corresponding local region, and local feature extraction and action recognition are subsequently performed on each local region. If no object to be identified is detected in the target detection result, the device may output a prompt message or end the current processing flow.

[0039] In this embodiment, the local region where the object to be identified is located is determined by the object detection model, and the region defined by the object detection box is directly used as the range for local feature extraction. This local region covers the complete outline of the object, and its boundary is directly determined by the object detection mechanism, unaffected by background complexity and pose changes; it supports the separate localization of multiple objects in the same image and independent subsequent action recognition. Therefore, the determination of the local region is both adaptive and accurate, providing a stable and reliable region range for subsequent local feature extraction, and improving the accuracy and robustness of local visual features.

[0040] In one embodiment, such as Figure 3 As shown, in step S103, determining the local region where the object to be identified is located in the target image includes: S301: Obtain the preset object template; S302: Determine the local region where the object to be identified is located in the target image based on the object template.

[0041] The object template is a pre-stored standard image or feature description used to characterize the appearance features of the object to be identified, and may include the outline image, grayscale template, or feature point descriptor of the object to be identified. In this example, the object template can be pre-constructed according to the actual application scenario. For example, for a human motion recognition scenario, the object template may be a human outline template or a human key point template. This application embodiment does not impose any restrictions on this.

[0042] As an example, in step S301, the device retrieves and loads the corresponding object template from the template library based on the category information of the object to be identified. The template library contains object templates of various categories, and the device selects the appropriate template according to the actual identification requirements.

[0043] As an example, in step S302, the device uses a template matching algorithm to perform pixel-by-pixel or step-by-step sliding matching on the target image using the object template as a sliding window. It calculates the similarity score between the object template and a local region of the target image at each sliding position, and determines the region corresponding to the position with the highest similarity score as the local region. Similarity calculation methods include normalized cross-correlation, squared difference matching, or correlation coefficient matching. If the highest similarity score is lower than a preset matching threshold, the device lowers the matching threshold and performs template matching again; if a match still cannot be found, it is determined that there is no object to be identified in the target image, and a prompt message is output or the current action recognition process is terminated.

[0044] In this embodiment, local regions are determined by template matching using a preset object template. This eliminates the need to pre-collect a large amount of sample data to train the detection model, reducing the data dependency and computational overhead of the method deployment. The template matching process directly calculates similarity based on the object's appearance features, and the region boundaries are directly determined by the template shape, unaffected by changes in background complexity. For multiple objects to be identified in the same target image, corresponding templates can be loaded for multiple matching operations, determining their respective local regions and independently performing subsequent feature extraction and action recognition. Thus, a lightweight local region determination scheme is provided for scenarios lacking training data or with limited computational resources, ensuring the feasibility of the action recognition process.

[0045] In one embodiment, such as Figure 4 As shown, in step S104, the global visual features and the local visual features are fused to obtain the action representation features of the target image, including: S401: Perform cross-attention fusion on the global visual features and the local visual features to obtain the action representation features of the target image.

[0046] Among them, cross-attention fusion is a feature interaction method based on attention mechanism. It calculates the attention weight between the query and the key by taking one feature sequence as the query and another feature sequence as the key and value, and then aggregates the value according to the weight, thereby realizing the information interaction and fusion between the two feature sequences.

[0047] As an example, in step S401, the device inputs local and global visual features into the cross-attention module and outputs fused action representation features. Specifically, the cross-attention module uses local visual features as queries and global visual features as keys and values, calculates the attention weights between the query and the key, and aggregates the values ​​based on the attention weights to obtain the fused action representation features. Through this cross-attention fusion, local visual features can dynamically perceive and selectively aggregate scene context information from global visual features, so that the final action representation features simultaneously contain global scene information and local action details.

[0048] In this embodiment, global and local visual features are fused through a cross-attention mechanism. The global scene context is retrieved with local action details as the primary focus. This allows for adaptive filtering of relevant information and suppression of irrelevant background interference. Compared to direct splicing or fixed-weight fusion, cross-attention fusion can dynamically adjust the aggregation weight of global information, achieving precise alignment between global information and local details, and improving the discriminative and expressive power of action representation features.

[0049] In one embodiment, such as Figure 5 As shown, in step S401, cross-attention fusion is performed on the global visual features and the local visual features to obtain the action representation features of the target image, including: S501: Project the local visual features into query features; S502: Project the global visual features into key features and value features; S503: Perform cross-attention processing on the query features, the key features, and the value features to obtain the action representation features.

[0050] The query feature is a feature representation obtained by linearly projecting local visual features, used to retrieve scene context information related to the current action from global visual features. Both key and value features are feature representations obtained by linearly projecting global visual features. Key features are used to calculate a similarity score with the query feature, while value features are used for weighted aggregation based on attention weights.

[0051] As an example, in step S501, the device inputs local visual features into the query projection layer and maps the local visual features to query features through a linear transformation. Specifically, the query projection layer is a learnable fully connected layer whose input dimension is consistent with the dimension of the local visual features, and whose output dimension is a preset attention head dimension.

[0052] As an example, in step S502, the device inputs the global visual features into the key projection layer and the value projection layer respectively, and maps the global visual features into key features and value features respectively through linear transformation. Specifically, both the key projection layer and the value projection layer are learnable fully connected layers, whose input dimension is consistent with the dimension of the global visual features, and whose output dimension is consistent with the dimension of the query features.

[0053] As an example, in step S503, the device inputs the query features, key features, and value features into the cross-attention layer for processing, and outputs the fused action representation features. Specifically, the cross-attention layer calculates the similarity score between the query features and the key features, normalizes the similarity score to obtain the attention weight, and then performs a weighted summation of the value features according to the attention weight to obtain the fused action representation features. In this example, the cross-attention layer can adopt a multi-head attention mechanism, dividing the query features, key features, and value features into multiple attention heads for processing, and then concatenating and linearly transforming the outputs of each attention head to obtain the final action representation features.

[0054] In this embodiment, by using local visual features as queries and global visual features as keys and values ​​for cross-attention processing, local features can actively retrieve contextual information related to the current action from global features, suppressing responses from regions in the global features that are unrelated to the current action. This embodiment takes local action details as the dominant factor in information retrieval, which can avoid excessive interference of global scene information on action features, achieve accurate enhancement of local features by global information, and improve the discriminative ability of action representation features.

[0055] In one embodiment, such as Figure 6 As shown, in step S503, cross-attention fusion is performed on the global visual features and the local visual features to obtain the action representation features of the target image, including: S601: Calculate the attention weight based on the similarity between the query feature and the key feature; S602: The value features are weighted and aggregated according to the attention weights to obtain the action representation features.

[0056] The formula for calculating attention weights is as follows: The formula for calculating the fusion feature is: in, Key matrix The dimension; The query matrix is ​​obtained by projective transformation of local visual features; The key matrix is ​​obtained by projection transformation of global visual features; The value matrix is ​​obtained by projection transformation of global visual features; This represents the attention weight matrix, where each element represents the degree of association between the corresponding position in the local visual feature and the corresponding position in the global visual feature; This represents the action representation characteristics after fusion.

[0057] As an example, in step S601, the device calculates the similarity score between the query feature and each key feature, and normalizes the similarity score to obtain the attention weight. Specifically, the device calculates the dot product similarity between the query feature and the key feature, and divides the dot product similarity by a scaling factor. The scale is then applied, and the result is normalized using the softmax function to obtain the attention weights corresponding to each key feature.

[0058] As an example, in step S602, the device performs a weighted summation of the value features based on the attention weights calculated in step S601 to obtain the fused action representation features. Specifically, the device multiplies each value feature by its corresponding attention weight, and then sums all the weighted value features to obtain the aggregated feature vector, which is then determined as the action representation feature.

[0059] In this embodiment, attention weights are obtained by calculating the similarity between query features and key features, and value features are weighted and aggregated based on attention weights. This enables local visual features to dynamically filter and aggregate the scene context information most relevant to the current action from global visual features, thereby achieving precise enhancement of local features by global information and improving the discriminative ability and robustness of action representation features.

[0060] In one embodiment, such as Figure 7 As shown, in step S105, action recognition is performed based on the action representation features to obtain the action recognition result of the object to be recognized, including: S701: Perform a residual connection between the action representation features and the local visual features to obtain residual connection features; S702: Map the residual connection features to the same feature dimension as the action representation features to obtain enhanced action representation features; S703: Perform action recognition based on the enhanced action representation features to obtain the action recognition result of the object to be recognized.

[0061] Among them, the residual connection feature is the feature representation obtained by residual connection of action representation features and local visual features, and it contains both fused global-local correlation information and original local action detail information.

[0062] As an example, in step S701, the device adds the action representation features and the local visual features element-wise along the feature dimension. If the two dimensions are inconsistent, the local visual features are first mapped to the same dimension as the action representation features through a linear projection layer, and then element-wise addition is performed to obtain the residual connectivity features.

[0063] Among them, the enhanced action representation feature is the feature representation obtained by linear mapping of the residual connection feature, which is used to unify the feature dimension and optimize the feature distribution to improve the accuracy of action recognition.

[0064] As an example, in step S702, the device inputs the residual connection features into the linear mapping layer, and maps the residual connection features to the same feature dimension as the action representation features through a linear transformation, thereby obtaining enhanced action representation features. The linear mapping layer is a learnable fully connected layer, whose input dimension is the dimension of the residual connection features, and whose output dimension is consistent with the dimension of the action representation features. This linear mapping achieves the alignment of feature dimensions and further optimization of feature representation.

[0065] As an example, in step S703, the device inputs the enhanced action representation features into a classifier for action recognition and outputs the action recognition result corresponding to the object to be recognized. The classifier includes one or more fully connected layers. The output dimension of the last fully connected layer is consistent with the number of action categories. After being activated by the softmax function, it outputs the probability distribution of each action category, and the action category corresponding to the maximum probability is determined as the action recognition result.

[0066] In this embodiment, by performing residual connections between action representation features and local visual features, fine-grained action details in the local visual features can be preserved, compensating for the local information loss that may occur during feature fusion. By unifying feature dimensions and optimizing feature distribution through a linear mapping layer, the integrity and discriminative power of action representation features are enhanced. By implementing a non-linear mapping from features to action category probabilities through a multi-layer fully connected classifier, the accuracy and robustness of action recognition are improved.

[0067] As described above, the action recognition method provided in this embodiment extracts global visual features through global image block segmentation to capture the overall scene context; and extracts local visual features through local region determination and image block segmentation to focus on details of key action parts. After fusing global and local dual-branch features, the action representation features simultaneously contain scene context and action details, enhancing completeness and discriminative power. Compared to methods that only use single-scale features, this embodiment models the scene environment and action details at different spatial granularities, overcoming the inherent contradiction between scene perception and detail capture; by independently extracting local regions, it avoids the omission of action details or the introduction of background noise caused by fixed grids, making the response of action representation features to key parts more concentrated, thus improving the accuracy and robustness of action recognition.

[0068] In one embodiment, an action recognition device 800 is provided, which corresponds one-to-one with the action recognition methods described in the above embodiments. For example... Figure 8 As shown, the action recognition device includes an image acquisition module 801, a global processing module 802, a local processing module 803, a feature fusion module 804, and an action recognition module 805.

[0069] Image acquisition module 801 is used to acquire the target image that needs to be performed on action recognition; The global processing module 802 is used to divide the target image into multiple first image blocks and extract features from the multiple first image blocks to obtain the global visual features of the target image. The local processing module 803 is used to determine the local region where the object to be identified is located in the target image, divide the local region into multiple second image blocks, and extract features from the multiple second image blocks to obtain the local visual features of the target image. The feature fusion module 804 is used to fuse the global visual features and the local visual features to obtain the action representation features of the target image; The action recognition module 805 is used to perform action recognition based on the action representation features to obtain the action recognition result of the object to be recognized.

[0070] For specific limitations regarding the motion recognition device, please refer to the limitations of the motion recognition method above, which will not be repeated here. Each module in the aforementioned motion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0071] In one embodiment, a computer device 900 is provided, such as Figure 9 As shown, it includes a memory 901 and a processor 902, wherein the memory 901 is used to store computer programs; the processor 902 is used to execute the programs stored in the memory 901 to implement the action recognition method described in any embodiment of this application.

[0072] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the action recognition described in the above embodiment.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0075] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0076] It should be noted that when the above embodiments of the present invention are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An action recognition method, characterized in that, include: Acquire the target image that needs to be performed on action recognition; The target image is divided into multiple first image blocks, and features are extracted from the multiple first image blocks to obtain the global visual features of the target image; The local region where the object to be identified is located in the target image is determined, and the local region is divided into multiple second image blocks. Feature extraction is performed on the multiple second image blocks to obtain the local visual features of the target image. The global visual features and the local visual features are fused to obtain the action representation features of the target image; Action recognition is performed based on the action representation features to obtain the action recognition result of the object to be recognized.

2. The action recognition method as described in claim 1, characterized in that, Determining the local region where the object to be identified is located in the target image includes: The target image is detected using a target detection model to obtain the target detection result; Based on the target detection results, the local region where the object to be identified is located in the target image is determined.

3. The action recognition method as described in claim 1, characterized in that, Determining the local region where the object to be identified is located in the target image includes: Get the preset object template; The local region where the object to be identified is located in the target image is determined based on the object template.

4. The action recognition method as described in claim 1, characterized in that, The process of fusing the global visual features and the local visual features to obtain the action representation features of the target image includes: The global visual features and the local visual features are fused across attention to obtain the action representation features of the target image.

5. The action recognition method as described in claim 4, characterized in that, The step of performing cross-attention fusion on the global visual features and the local visual features to obtain the action representation features of the target image includes: Project the local visual features into query features; The global visual features are projected into key features and value features; The query features, key features, and value features are subjected to cross-attention processing to obtain the action representation features.

6. The action recognition method as described in claim 5, characterized in that, The process of performing cross-attention processing on the query features, the key features, and the value features to obtain the action representation features includes: Calculate attention weights based on the similarity between the query features and the key features; The value features are weighted and aggregated according to the attention weights to obtain the action representation features.

7. The action recognition method as described in claim 1, characterized in that, The step of performing action recognition based on the action representation features to obtain the action recognition result of the object to be recognized includes: The action representation features are residually connected to the local visual features to obtain residual connection features; The residual connection features are mapped to the same feature dimension as the action representation features to obtain the enhanced action representation features; Action recognition is performed based on the enhanced action representation features to obtain the action recognition result of the object to be recognized.

8. A motion recognition device, characterized in that, include: The image acquisition module is used to acquire the target image that needs to be performed on action recognition. A global processing module is used to divide the target image into multiple first image blocks and extract features from the multiple first image blocks to obtain the global visual features of the target image. The local processing module is used to determine the local region where the object to be identified is located in the target image, divide the local region into multiple second image blocks, and extract features from the multiple second image blocks to obtain the local visual features of the target image. The feature fusion module is used to fuse the global visual features and the local visual features to obtain the action representation features of the target image; The action recognition module is used to perform action recognition based on the action representation features to obtain the action recognition result of the object to be recognized.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the action recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the action recognition method as described in any one of claims 1 to 7.

Images