Human body action behavior recognition method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a human body action behavior recognition method and device, computer equipment and a storage medium. A first video frame image in video data is acquired, human body motion detection is performed on each frame of the first video frame image to obtain a human body motion area in each frame of the first video frame image, and the human body motion area in the first video frame image is segmented according to a preset block size to obtain a segmented human body motion area in the first video frame image; obtaining a human body image block corresponding to each frame of first video frame image, and finally inputting the human body image blocks and a preset behavior description text into a preset human body action behavior recognition model to obtain an action behavior category of the video data. Visibly, compared with the method of performing down-sampling processing on a high-resolution image in the prior art to obtain a down-sampled image, the method of the invention effectively retains image details under high resolution while satisfying model input, and can achieve the purpose of improving the detection rate of human body action behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, computer device, and storage medium for recognizing human actions and behaviors. Background Technology

[0002] Human action and behavior recognition in surveillance videos is an important task in the field of computer vision. It is usually based on current mainstream models (such as human action and behavior recognition networks based on CNN architecture or transformer architecture) to identify the types of human actions and behaviors in surveillance videos.

[0003] However, as the precision of surveillance equipment increases, the resolution of the surveillance videos acquired by these devices also increases. For example, high-precision surveillance equipment can acquire surveillance videos with a resolution of up to 8 million pixels or even higher, while the input resolution of the model (such as 224×224) is usually much lower than the resolution of the surveillance video. Therefore, existing technologies typically downsample high-resolution surveillance videos to reduce their resolution before performing human action and behavior recognition on the videos.

[0004] However, downsampled surveillance videos lose a lot of detailed information, such as the motion features of small targets in the distance. In particular, after downsampling, small targets in the distance may only occupy a few pixels, making it difficult for the model to capture the motion features of small targets. This leads to a sharp drop in the final detection rate of the model for recognizing human actions. Summary of the Invention

[0005] This invention provides a method, apparatus, computer device, and storage medium for recognizing human actions and behaviors, in order to solve the technical problem of low detection rate in existing model-based human action and behavior recognition methods.

[0006] A method for recognizing human actions and behaviors includes: Obtain the first video frame image from the video data; Human motion detection is performed on each frame of the first video frame to obtain the human motion region in each frame of the first video frame. According to the preset block size, the human motion region in the first video frame image is segmented to obtain human image blocks corresponding to each frame of the first video frame image. The human image block and the preset behavior description text are input into a preset human action behavior recognition model to obtain the action behavior category of the video data.

[0007] Optionally, in the above-mentioned human motion recognition method, the step of segmenting the human motion region in the first video frame image according to a preset block size to obtain a human image block corresponding to each frame of the first video frame image includes: The first video frame image is sampled at preset intervals to obtain the second video frame image; According to the block size, the human motion region in the second video frame image is segmented to obtain human image blocks corresponding to each frame of the second video frame image.

[0008] Optionally, in the above-mentioned human action and behavior recognition method, the human action and behavior recognition model includes a hybrid modality fusion module and a behavior classification module; The step of inputting the human image patch and a preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data includes: The human image blocks are spatiotemporally co-coded to construct a spatiotemporally coded data sequence; The spatiotemporal encoded data sequence and the behavior description text are input into the mixed modality fusion module to obtain action behavior fusion features; The fusion features of the action behavior are input into the behavior classification module to obtain the action behavior category.

[0009] Optionally, in the above-mentioned human action and behavior recognition method, the hybrid modal fusion module includes a visual encoder, a text encoder, and a feature fusion module; The step of inputting the spatiotemporal encoded data sequence and the behavior description text into the mixed modality fusion module to obtain action behavior fusion features includes: The spatiotemporal encoded data sequence is input into the visual encoder to obtain the spatiotemporal joint feature sequence corresponding to the spatiotemporal encoded data sequence; The behavior description text is input into the text encoder to obtain the description text features corresponding to the behavior description text; The spatiotemporal joint feature sequence and the descriptive text features are input into the feature fusion module for feature fusion to obtain the action behavior fusion features.

[0010] Optionally, in the above-mentioned human action and behavior recognition method, the step of inputting the spatiotemporal joint feature sequence and the descriptive text features into the feature fusion module for feature fusion to obtain the action and behavior fusion features includes: The spatiotemporal joint feature sequence is subjected to feature separation to obtain the human body spatial domain feature sequence and the human body temporal domain feature sequence; The human spatial feature sequence, the human temporal feature sequence, and the descriptive text features are input into the feature fusion module for cross-attention feature fusion to obtain spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features. The spatiotemporal fusion feature, the spatial text fusion feature, and the temporal text fusion feature are concatenated to obtain the action behavior fusion feature.

[0011] Optionally, the human action behavior recognition model in the above-mentioned human action behavior recognition method may further include a recommendation module; The step of inputting the fused features of the action behavior into the behavior classification module to obtain the action behavior category includes: The action behavior fusion features are input into the behavior classification module to obtain the classification category corresponding to each human body image block; The classification category is input into the recommendation module to obtain the action / behavior category.

[0012] Optionally, in the above-mentioned human action and behavior recognition method, the step of inputting the classification category into the recommendation module to obtain the action and behavior category includes: The recommendation module determines the target category corresponding to the human image block in the target frame image from the classification categories; the target frame image is the center frame image in the first video frame image; The proportion of each target category is calculated based on the number of human image blocks in the target frame image and the number of each target category. Select the target category corresponding to the largest percentage value as the action behavior category.

[0013] In the above-mentioned human action and behavior recognition method, optionally, the behavior classification module includes a shared classification branch and a differential learning branch; The step of inputting the fused action behavior features into the behavior classification module to obtain the classification category corresponding to each human image patch includes: The action behavior fusion features are respectively input into the shared classification branch and the differential learning branch to obtain the first classification category corresponding to each human image patch output by the shared classification branch and the second classification category corresponding to each human image patch output by the differential learning branch.

[0014] Optionally, the human action behavior recognition model in the above-mentioned human action behavior recognition method is trained in the following manner: Obtain training image patch samples and training text samples; The training image patch samples and training text samples are input into the human action behavior recognition model to obtain the training fusion features output by the hybrid modality fusion module; The training fusion features are input into the shared classification branch and the differential learning branch respectively to obtain the first predicted category output by the shared classification branch and the second predicted category output by the differential learning branch. The first predicted category and the second predicted category are input into the recommendation module to obtain the third predicted category; Based on the training fusion features and the third prediction category, the causal loss value of the human action behavior recognition model is calculated. Determine whether the causal loss value meets the preset termination condition; If the causal loss value meets the preset termination condition, it is determined that the human action behavior recognition model has been sufficiently trained. If the causal loss value does not meet the preset termination condition, the human action behavior recognition model is iteratively trained based on the training image patch samples and the training text samples until the causal loss value meets the preset termination condition, and it is determined that the human action behavior recognition model has been sufficiently trained. During the training of the human action behavior recognition model, the parameters of the shared classification branch are frozen.

[0015] Optionally, in the above-mentioned human action and behavior recognition method, the step of calculating the causal loss value of the human action and behavior recognition model based on the training fusion features and the third prediction category includes: The trained fused features are then subjected to noise processing to obtain interference features; Based on the training fusion features and the third predicted category, the classification loss value is calculated; Based on the interference features and the training fusion features, the causal regularization loss value is calculated; The causal loss value is calculated based on the classification loss value and the causal regularization loss value.

[0016] Optionally, in the above-mentioned human motion recognition method, the step of performing human motion detection on each frame of the first video frame to obtain the human motion region in each frame of the first video frame includes: Human motion detection is performed on each frame of the first video frame to obtain the initial motion region in each frame of the first video frame. Determine whether the size of the initial motion region is an integer multiple of the block size; If the size of the initial motion region is an integer multiple of the block size, the initial motion region is determined to be the human motion region; If the size of the initial motion region is not an integer multiple of the block size, then the size of the initial motion region is adjusted to be an integer multiple of the block size to obtain the human motion region.

[0017] A human motion and behavior recognition device, comprising: The video frame image acquisition module is used to acquire the first video frame image from the video data; The motion region recognition module is used to perform human motion detection on each frame of the first video frame image to obtain the human motion region in each frame of the first video frame image. The image block segmentation module is used to segment the human motion region in the first video frame image according to a preset segmentation size to obtain human image blocks corresponding to each frame of the first video frame image. The action behavior classification module is used to input the human image block and the preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data.

[0018] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the human action behavior recognition method as described above.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the human action behavior recognition method as described above.

[0020] The aforementioned human action and behavior recognition method, apparatus, computer equipment, and storage medium acquire first video frame images from video data, perform human motion detection on each first video frame image to obtain the human motion region in each first video frame image, segment the human motion region in the first video frame image according to a preset block size to obtain human image blocks corresponding to each first video frame image, and finally input the human image blocks and preset behavior description text into a preset human action and behavior recognition model to obtain the action and behavior category of the video data. It can be seen that this application, by downsampling human image blocks compared to the prior art of obtaining downsampled images from high-resolution images, effectively preserves image details at high resolution while satisfying the model input requirements, thereby achieving the goal of improving the detection rate of human action and behavior recognition. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an implementation of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 2 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 3 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 4 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of a human motion behavior recognition model disclosed in an embodiment of the present invention; Figure 6 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 7 This is a schematic diagram of the feature fusion module in a human action behavior recognition model disclosed in an embodiment of the present invention; Figure 8 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 9 This is another structural schematic diagram of the human action behavior recognition model disclosed in an embodiment of the present invention; Figure 10 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 11 This is another structural schematic diagram of the human motion behavior recognition model disclosed in an embodiment of the present invention; Figure 12 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 13 This is a partial implementation flowchart of a human action behavior recognition method disclosed in an embodiment of the present invention; Figure 14 This is a schematic diagram of the structure of a human motion behavior recognition device disclosed in an embodiment of the present invention; Figure 15 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] This application discloses a method, apparatus, computer device, and storage medium for human action and behavior recognition. It acquires a first video frame image from video data and performs human motion detection on each frame to obtain the human motion region in each frame. The human motion region in the first video frame image is then segmented according to a preset block size to obtain a human image block corresponding to each frame. Finally, the human image block and a preset behavior description text are input into a preset human action and behavior recognition model to obtain the action and behavior category of the video data. It is evident that this application, by downsampling the human image block compared to the prior art of downsampling high-resolution images to obtain downsampled images, effectively preserves image details at high resolution while satisfying the model input requirements, thus improving the detection rate of human action and behavior recognition. Specific embodiments are described below.

[0030] In one embodiment, such as Figure 1 The diagram shown is a flowchart illustrating the implementation of a human action behavior recognition method disclosed in this application. This method is applicable to electronic devices with video and image processing capabilities, such as mobile phones, tablets, laptops, personal computers, and servers. The method in this embodiment specifically includes the following steps: S101: Obtain the first video frame image from the video data.

[0031] As can be understood, the video data in this embodiment is a collection of first video frame images over time. The first video frame image is the basic unit constituting the video data, and each first video frame image is an independent static image. These images are played sequentially at specific time intervals (frame rate, such as 32 frames per second, 64 frames per second, etc.) to form the video data.

[0032] In this specific implementation, the video data in this embodiment can be acquired by video acquisition devices or computer devices with video acquisition capabilities, such as surveillance cameras or mobile phones. Since video acquisition devices do not have video and image processing capabilities, the video data needs to be transmitted to devices with video and image processing capabilities, such as servers. The number of first video frame images included in the video data can be set according to actual needs, such as 32 frames or 64 frames. This embodiment does not specifically limit the number of first video frame images.

[0033] For example, when a surveillance camera acquires video data containing 64 first video frames, it sends the video data to the server so that the server can perform human action and behavior recognition based on the video data.

[0034] S102: Perform human motion detection on each first video frame image to obtain the human motion region in each first video frame image.

[0035] The human motion region refers to the area in the first video frame image that is the current frame, where the pixel values ​​of the first video frame image change significantly due to human motion, compared to one or more adjacent first video frame images.

[0036] In a specific implementation, this embodiment can sequentially use each first video frame as the current frame, and then detect the human motion region in each first video frame. The methods for detecting the human motion region in the first video frame include, but are not limited to, using the inter-frame difference method or the optical flow method to perform human motion detection on each first video frame to obtain the human motion region in each first video frame.

[0037] S103: According to the preset block size, the human motion region in the first video frame image is segmented to obtain the human image block corresponding to each first video frame image.

[0038] The block size refers to the image size that the human action behavior recognition model can input, such as a human image block with a width of 16 pixels and a height of 16 pixels, i.e., a 16*16 human image block. The segmented human image blocks can be directly input into the human action behavior recognition model without further downsampling processing. In this embodiment, the specific size of the preset block size is not limited.

[0039] In this specific implementation, this embodiment only segments the human motion region in the first video frame image according to the block size. Other regions in the first video frame image besides the human motion region do not participate in subsequent human action recognition, which helps reduce the inference time of the human action recognition model in this embodiment. Specifically, this embodiment includes, but is not limited to, segmenting the human motion region in the first video frame image according to a preset block size using region growing algorithms and watershed algorithms to obtain human image blocks corresponding to each frame of the first video frame.

[0040] S104: Input the human image block and the preset behavior description text into the preset human action behavior recognition model to obtain the action behavior category of the video data.

[0041] Among them, behavioral description text refers to a detailed, accurate, and standardized written description of various human actions and behaviors.

[0042] The human action and behavior recognition model extracts features from human image blocks and behavior description texts respectively, obtaining human image features corresponding to human image blocks and behavior text features corresponding to behavior text features. The human image features and behavior text features are then fused to obtain action and behavior fusion features for each human image block. Based on the action and behavior fusion features, the classification category corresponding to each human image block is output, and then the classification category with the largest number of occurrences is selected as the action and behavior category.

[0043] In a specific implementation, the human action behavior recognition model in this embodiment may include a hybrid modal fusion module, a behavior classification module, and a recommendation module. The hybrid modal fusion module is used to fuse human image features and behavioral text features to obtain action behavior fusion features. The behavior classification module is used to output the classification category corresponding to each human image patch based on the action behavior fusion features. The recommendation module outputs the action behavior category based on the classification category. The hybrid modal fusion module may include a visual encoder (such as a CLIP visual encoder) and a text encoder (such as a CLIP text encoder). The visual encoder converts human image patches into human image features, and the text encoder converts behavioral description text into descriptive text features. Then, the human image features and descriptive text features are fused to obtain the action behavior fusion features corresponding to each human image patch. It should be noted that the action behavior categories output by the human action behavior recognition model in this embodiment include, but are not limited to, any one of the following: basic actions (such as limb movements, gestures, and head movements), daily activities (such as home activities and work activities), and interactive behaviors (such as character interaction and person-to-person interaction). In other words, the specific human action behavior recognition task performed using the method disclosed in this embodiment is not limited.

[0044] As an extension, in this embodiment, the classification category corresponding to the human image block in the intermediate frame image can be selected from the classification categories corresponding to all human image blocks. Then, the classification category with the largest proportion can be selected from the classification categories corresponding to the human image blocks in the intermediate frame image as the action behavior category output by the human action behavior recognition model.

[0045] For example, taking a video data with 64 frames as the first video frame image, the 32nd frame of the first video frame image is selected as the intermediate frame image. From the classification categories corresponding to all human body image blocks, the classification categories corresponding to the human body image blocks in the intermediate frame image are filtered out. Then, from the classification categories corresponding to the human body image blocks in the intermediate frame image, the classification category with the largest proportion is selected as the action behavior category output by the human action behavior recognition model.

[0046] In summary, the human action behavior recognition method disclosed in this embodiment acquires a first video frame image from video data, performs human motion detection on each first video frame image to obtain the human motion region in each first video frame image, segments the human motion region in the first video frame image according to a preset block size to obtain a human image block corresponding to each first video frame image, and finally inputs the human image block and a preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data. It can be seen that this application, by downsampling the human image block compared to the prior art of downsampling high-resolution images to obtain downsampled images, effectively preserves the image details at high resolution while satisfying the model input requirements, thereby achieving the goal of improving the detection rate of human action behavior recognition.

[0047] In one embodiment, such as Figure 2 As shown, step S103 in this embodiment can be implemented through the following steps: S201: The first video frame image is sampled at intervals according to the preset sampling interval to obtain the second video frame image.

[0048] The sampling interval refers to how many times the first video frame image is sampled to obtain a second video frame image.

[0049] For example, taking a sampling interval of 4 and 64 first video frame images as an example, the first video frame images are sampled starting from frame 1, and the 1st, 5th, 9th... 57th and 61st frames are extracted in sequence, for a total of 16 first video frame images. These 16 first video frame images are the second video frame images.

[0050] S202: According to the block size, the human motion region in the second video frame image is segmented to obtain the human image block corresponding to each second video frame image.

[0051] In this specific implementation, only the human motion region in the second video frame image is segmented according to the block size, while other regions in the second video frame image other than the human motion region do not participate in the subsequent human action behavior recognition.

[0052] In summary, this embodiment obtains a second video frame image with fewer frames by sampling the first video frame image at preset sampling intervals. Then, it segments the human motion region of the second video frame image to extract the human image blocks that need to output the human action behavior recognition model, further reducing the number of human image blocks. This allows the human action behavior recognition model to achieve human action behavior recognition based on fewer human image blocks, reducing the inference time of the human action behavior recognition model and improving the recognition efficiency of the human action behavior recognition model.

[0053] In one embodiment, such as Figure 3 As shown, the human action behavior recognition model in this embodiment includes a hybrid modality fusion module and a behavior classification module. Based on this, step S104 in this embodiment can be implemented through the following steps: S301: Perform spatiotemporal joint coding on human body image blocks to construct a spatiotemporal coded data sequence.

[0054] Spatiotemporal joint coding refers to combining the temporal dimension (frame order) and spatial dimension (relative position of the human image block in the first video frame) of the human image block to generate spatiotemporal joint coded data with spatiotemporal context information. The spatiotemporal coded data sequence contains the spatiotemporal joint coded data corresponding to each human image block.

[0055] In this specific implementation, it is necessary to obtain the temporal position code, spatial position code, and relative frame position code for each human image block. The relative frame position code refers to the offset of the current human image block in the first video frame relative to the intermediate frame. The intermediate frame is the frame that is chronologically centered within the first video frame.

[0056] For example, taking a first video frame image with 64 frames as an example, where frame 32 or frame 33 is an intermediate frame image, and assuming frame 10 contains four human image blocks, then the spatial position codes of these four human image blocks are (0,0), (0,1), (1,0), and (1,1), respectively. The temporal position codes of these four human image blocks are all 10, and the relative frame position codes of these four human image blocks are all 10-32=-22. Therefore, the spatiotemporal joint feature sequence corresponding to frame 10 can be as follows: Token1=[features 0,0 ;0;0;-22;10]; Token2=[features 0,1 ;0;1;-22;10]; Token3=[features 1,0 ;1;0;-22;10]; Token4=[features 1,1 ;1;1;-22;10].

[0057] Taking Token1 as an example, features 0,0The features (such as the original pixel values ​​or CNN features) of a human image patch with a spatial location code of (0,0) are represented by: 0,0 representing the spatial location code of the human image patch in the first video frame; -22 representing the relative frame location code of the human image patch, which is also the relative frame location code of the first video frame relative to intermediate frames; and 10 representing the temporal location code of the human image patch, which is also the temporal location code of the first video frame. Based on this, a spatiotemporal joint feature sequence corresponding to all human image patches can be constructed.

[0058] S302: Input the spatiotemporal encoded data sequence and behavior description text into the mixed modality fusion module to obtain action behavior fusion features.

[0059] The spatiotemporal encoded data sequence and behavioral description text are input into the mixed modality fusion module to obtain the spatiotemporal joint feature sequence corresponding to the spatiotemporal encoded data sequence and the descriptive text features corresponding to the behavioral description text. The spatiotemporal joint features corresponding to each human image patch in the spatiotemporal joint feature sequence are then fused with the descriptive text features to obtain the action behavior fusion features corresponding to each human image patch. The spatiotemporal joint feature sequence contains the spatiotemporal joint features corresponding to each human image patch.

[0060] Specifically, the hybrid modal fusion module in this embodiment may include a visual encoder (such as a CLIP visual encoder) and a text encoder (such as a CLIP text encoder). The visual encoder converts the spatiotemporal encoded data sequence into a spatiotemporal joint feature sequence, and the text encoder converts the behavior description text into description text features. Then, the spatiotemporal joint features corresponding to each human image block in the spatiotemporal joint feature sequence are fused with the description text features to obtain the action behavior fusion features corresponding to each human image block.

[0061] S303: Input the action behavior fusion features into the behavior classification module to obtain the action behavior categories of the video data.

[0062] The action behavior fusion features are input into the behavior classification module to obtain the classification category corresponding to each human body image patch, and then the classification category with the largest number of patches is selected as the action behavior category.

[0063] In a specific implementation, the behavior classification module in this embodiment includes, but is not limited to, any one of a classifier, a fully connected layer, or global average pooling (GAP) combined with Softmax.

[0064] Furthermore, such as Figure 4As shown, the hybrid modality fusion module in this embodiment includes a visual encoder, a text encoder, and a feature fusion module. Based on this, step S302 in this embodiment can be implemented through the following steps: S401: Input the spatiotemporal encoded data sequence into the visual encoder to obtain the spatiotemporal joint feature sequence corresponding to the spatiotemporal encoded data sequence.

[0065] S402: Input the behavior description text into the text encoder to obtain the description text features corresponding to the behavior description text.

[0066] S403: Input the spatiotemporal joint feature sequence and descriptive text features into the feature fusion module for feature fusion to obtain action behavior fusion features.

[0067] Specifically, the visual encoder in this embodiment includes, but is not limited to, any one of the OpenVision encoder, CLIP visual encoder, etc., and the text encoder includes, but is not limited to, any one of the CLIP text encoder, BLIP-2 model, etc.

[0068] In a specific implementation, this embodiment can input the spatiotemporal encoded data sequence into a visual encoder to convert the spatiotemporal encoded data sequence into a spatiotemporal joint feature sequence, input the behavior description text into a text encoder to convert the behavior description text into description text features, and then input the spatiotemporal joint features and description text features corresponding to each human image block in the spatiotemporal joint feature sequence into a feature fusion module to fuse the spatiotemporal joint features corresponding to each human image block in the spatiotemporal joint feature sequence with the description text features respectively to obtain the action behavior fusion features corresponding to each human image block.

[0069] like Figure 5 The diagram shown is a structural schematic of a human action behavior recognition model in this embodiment. The hybrid modal fusion module in the human action behavior recognition model includes a visual encoder, a text encoder, and a feature fusion module. The spatiotemporal encoded data sequence and the behavior description text are respectively input into the visual encoder and text encoder of the hybrid modal fusion module. After feature fusion by the feature fusion module, the action behavior fusion features are output to the behavior classification module to obtain the action behavior category.

[0070] In summary, the hybrid modal fusion module based on the human action behavior recognition model in this embodiment fuses spatiotemporal encoded data sequences and behavior description text into action behavior fusion features. It fully combines the capabilities of spatiotemporal dimensions (including temporal and spatial dimensions) and textual dimensions. Compared with general single-modal or spatiotemporal networks, it emphasizes the multi-dimensional information representation capabilities of text, spatial location, temporal location, and relative frame location, which can further improve the detection rate and accuracy of action behaviors.

[0071] In one embodiment, such as Figure 6 As shown, step S403 in this embodiment can be implemented through the following steps: S601: Perform feature separation on the spatiotemporal joint feature sequence to obtain the human body spatial domain feature sequence and the human body temporal domain feature sequence.

[0072] Understandably, the spatiotemporal joint feature sequence includes joint features from both the spatial and temporal dimensions. Therefore, by performing feature separation on the spatiotemporal joint feature sequence, we can separate the spatial dimension human body spatial feature sequence and the temporal dimension human body temporal feature sequence from it. In a specific implementation, this embodiment can use any method such as 3D convolution decomposition or slow-fast network to separate the spatiotemporal joint feature sequence to obtain the human body spatial domain feature sequence and the human body temporal domain feature sequence.

[0073] S602: The human body spatial feature sequence, human body temporal feature sequence and descriptive text feature input feature fusion module are cross-attention feature fusion to obtain spatiotemporal fusion features, spatial text fusion features and temporal text fusion features.

[0074] Among them, the spatiotemporal fusion feature is obtained by fusing human spatial features in the human spatial feature sequence and human temporal features in the human temporal feature sequence through cross-attention feature fusion; the spatial text fusion feature is obtained by fusing human spatial features and descriptive text features in the human spatial feature sequence through cross-attention feature fusion; and the temporal text fusion feature is obtained by fusing human temporal features and descriptive text features in the human temporal feature sequence through cross-attention feature fusion.

[0075] In a specific implementation, the feature fusion module in this embodiment may include three cross-attention modules: a first module for fusing spatiotemporal fusion features, a second module for fusing spatial text fusion features, and a third module for fusing temporal text fusion features.

[0076] S603: The spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features are concatenated to obtain the action behavior fusion features.

[0077] In this specific implementation, the spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features are first preprocessed, and then the preprocessed spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features are concatenated to obtain the action behavior fusion features. The preprocessing includes aligning the feature dimensions of the spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features. After completing the feature dimension alignment, the spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features are then temporally aligned, thus completing the preprocessing of the spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features. Only after the spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features have been preprocessed can the action behavior fusion features be concatenated.

[0078] like Figure 7 As shown, the feature fusion module in this embodiment may include a spatiotemporal cross-attention module, a spatial text cross-attention module, a temporal text cross-attention module, a spatiotemporal feature separation module, a feedforward neural network, and a splicing module.

[0079] In summary, the human action behavior recognition model in this embodiment performs cross-attention feature fusion on the human spatial feature sequence, human temporal feature sequence, and descriptive text feature input feature fusion module to obtain spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features. These features are then concatenated to form action behavior fusion features. This fully combines the guiding capabilities of temporal, spatial, and text dimensions. Compared to general single-modal or spatiotemporal networks, it emphasizes the ability to represent mixed-dimensional information such as text and spatial location, text and temporal location, and temporal and spatial location, which can further improve the detection rate and accuracy of action behaviors.

[0080] In one embodiment, such as Figure 8 As shown, the human action behavior recognition model also includes a recommendation module. Based on this, step S303 in the above embodiment can be implemented through the following steps: S801: Input the action behavior fusion features into the behavior classification module to obtain the classification category corresponding to each human body image patch.

[0081] The action behavior fusion features are input into the behavior classification module to obtain the classification category corresponding to each human body image patch. The classification module includes, but is not limited to, any one of classifiers, fully connected layers, etc.

[0082] S802: Input the category into the recommendation module to obtain the action / behavior category.

[0083] The recommendation module is used to statistically analyze the classification categories corresponding to each human body image patch in order to identify the action behavior category from the classification categories.

[0084] In its specific implementation, the recommendation module in this embodiment can select the category with the largest proportion of entries as the action / behavior category. For example, Figure 9 The diagram shown is a structural schematic of a human action behavior recognition model in this embodiment, including a behavior recommendation module.

[0085] As an extension, the recommendation module in this embodiment can also filter out the classification categories corresponding to human image patches in intermediate frame images from all the classification categories corresponding to human image patches, and then select the classification category with the largest proportion from the classification categories corresponding to human image patches in intermediate frame images as the action behavior category output by the human action behavior recognition model. This embodiment does not limit the specific method by which the recommendation module statistically derives the action behavior category from the classification categories.

[0086] In one embodiment, such as Figure 10 As shown, step S702 in the above embodiment can be implemented through the following steps: S1001: The recommendation module determines the target category corresponding to the human image patch in the target frame image from the classification categories.

[0087] The target frame image is the center frame image in the first video frame image.

[0088] In a specific implementation, this embodiment can first determine the center frame image in the first video frame image, then determine the temporal position code of the center frame image, and then determine the corresponding human image block based on the temporal position code, thereby determining the target category corresponding to the human image block in the target frame image.

[0089] S1002: Calculate the proportion of each target category based on the number of human image blocks and the number of each target category in the target frame image.

[0090] S1003: Select the target category corresponding to the largest percentage value as the action behavior category.

[0091] The number of each target category is counted, and the number of each target category is used as the numerator, while the number of human image blocks in the target frame image is used as the denominator. The resulting fraction is the proportion of each target category.

[0092] For example, if the number of human image blocks in the target frame image is 4, and three of the human image blocks are classified into the first category and one human image block is classified into the second category, then the proportion of the first category is 3 / 4 and the proportion of the second category is 1 / 4. Since 3 / 4 is greater than 1 / 4, the first category is the action / behavior category.

[0093] In summary, this embodiment determines the final action category by statistically analyzing the classification of human image blocks in intermediate frame images. Compared to statistically analyzing the classification of human image blocks in all video frame images, this effectively reduces the computational load and hardware cost of the human action recognition device, thereby improving processing speed and real-time performance.

[0094] In one embodiment, the behavior classification module includes a shared classification branch and a differential learning branch. Based on this, step S602 in the above embodiment can be implemented in the following way: The action behavior fusion features are input into the shared classification branch and the differential learning branch respectively, to obtain the first classification category corresponding to each human image patch output by the shared classification branch and the second classification category corresponding to each human image patch output by the differential learning branch.

[0095] In this system, the shared classification branch and the differential learning branch execute the classification task in parallel. Each human image patch output by the shared classification branch corresponds to a first classification category, while each human image patch output by the differential learning branch corresponds to a second classification category. In other words, each human image patch corresponds to both a first classification category and a second classification category. Figure 11 The diagram shown is another structural schematic of the human action behavior recognition model disclosed in this embodiment, wherein the behavior classification module includes a shared classification branch and a differential learning branch.

[0096] It should be noted that in this embodiment, only the first classification category corresponding to each human image patch output by the shared classification branch can be input into the recommendation module, or only the second classification category corresponding to each human image patch output by the difference learning branch can be input into the recommendation module, or both the first classification category and the second classification category can be input into the recommendation module at the same time. No specific limitation is made in this embodiment.

[0097] In summary, this embodiment utilizes a shared classification branch and a differential learning branch to execute the classification task in parallel. This effectively combines the generalization ability of the shared classification branch with the task specificity of the differential learning branch, thereby improving the overall performance of the model.

[0098] In one embodiment, such as Figure 12 As shown, the human action behavior recognition model in this embodiment is trained through the following steps: S1201: Obtain training image patch samples and training text samples.

[0099] The training image block samples can be image blocks segmented from video frame images of the training video data, and the training text samples can be the same as the behavior description text in this embodiment.

[0100] S1202: Input the training image patch samples and training text samples into the human action behavior recognition model to obtain the training fusion features output by the hybrid modality fusion module.

[0101] S1203: Input the training fusion features into the shared classification branch and the differential learning branch respectively to obtain the first predicted category output by the shared classification branch and the second predicted category output by the differential learning branch.

[0102] S1204: Input the first and second predicted categories into the recommendation module to obtain the third predicted category.

[0103] The third prediction category is the prediction result actually output by the human action behavior recognition model during the training process.

[0104] S1205: Based on the training fusion features and the third prediction category, the causal loss value of the human action behavior recognition model is calculated.

[0105] S1206: Determine whether the causal loss value meets the preset termination condition.

[0106] S1207: If the causal loss value meets the preset termination condition, it is determined that the human action behavior recognition model has been sufficiently trained.

[0107] If the causal loss value does not meet the preset termination condition, the human action behavior recognition model is iteratively trained based on the training image patch samples and training text samples. That is, the process returns to step S802 and subsequent steps until the causal loss value meets the preset termination condition, thus determining that the human action behavior recognition model has been sufficiently trained.

[0108] In the training process of the human action behavior recognition model, the parameters of the shared classification branch are frozen. This allows the human action behavior recognition model to focus more on learning the parameters of the differential learning branch related to specific tasks, thereby accelerating the training process. At the same time, freezing the shared classification branch can also prevent the human action behavior recognition model from overfitting during training, thus improving the stability of the human action behavior recognition model.

[0109] In one embodiment, such as Figure 13 As shown, step S1205 in the above embodiment can be implemented through the following steps: S1301: Add noise to the training fusion features to obtain interference features.

[0110] S1302: The classification loss value is calculated based on the training fusion features and the third predicted category.

[0111] S1303: The causal regularization loss value is calculated based on the interference features and the training fusion features.

[0112] S1304: The causal loss value is calculated based on the classification loss value and the causal regularization loss value.

[0113] As is understandable, the training fusion features in this embodiment are usually represented in vector form. Therefore, the interference features obtained after adding noise are also represented in vector form.

[0114] In this specific implementation, the training fusion features, the third prediction category, and the interference features can be input into the causal loss calculation formula to obtain the causal loss value.

[0115] The formula for calculating causal loss can be shown below: The formula involves inputting the training fusion features and the third predicted category into the first half of the causal loss calculation formula to calculate the classification loss value, inputting the interference features and the training fusion features into the second half of the causal loss calculation formula to calculate the causal regularization loss value, and then adding the classification loss value and the causal regularization loss value to obtain the causal loss value.

[0116] Furthermore, in the above formula for calculating causal loss... , and They can be expressed by the following formulas respectively: in, Indicates training fusion features, Indicates the third prediction category, This represents the interference features after adding noise to the fused features during training. This represents the weight parameters and bias vector of the differential learning branch. Represents the regularization coefficient. Indicates the number of categories. Represents the weight parameters. This represents the maximum value among all third-prediction categories output by the human action recognition model based on the fused features from training. This indicates the third prediction category index.

[0117] In summary, the causal loss function designed in this embodiment enables the human action behavior recognition model to have a learnable corrective ability for differences in action behavior categories, thereby improving the human action behavior recognition model's ability to correctly distinguish confused human image blocks and achieving the goal of improving the learning effect of the human action behavior recognition model.

[0118] In one embodiment, such as Figure 14 As shown, step S102 in the above embodiment can be implemented through the following steps: S1401: Perform human motion detection on each first video frame image to obtain the initial motion region in each first video frame image.

[0119] S1402: Determine whether the size of the initial motion region is an integer multiple of the block size.

[0120] S1403: If the size of the initial motion region is an integer multiple of the block size, the initial motion region is determined to be the human motion region.

[0121] S1404: If the size of the initial motion region is not an integer multiple of the block size, then adjust the size of the initial motion region to an integer multiple of the block size to obtain the human motion region.

[0122] Understandably, the initial motion region detected from the first video frame may not meet the size requirements. Therefore, it is necessary to determine whether the size of the initial motion region is an integer multiple of the block size. If the size of the initial motion region is an integer multiple of the block size, it means that multiple complete human image blocks can be segmented from the initial motion region, and the initial motion region is determined to be a human motion region. If the size of the initial motion region is not an integer multiple of the block size, it means that after the initial motion region is segmented into multiple complete human image blocks, there will still be a part of the initial motion region that has not been segmented. This part of the image information cannot be applied to human action behavior recognition, which will inevitably affect the final recognition accuracy. Therefore, the size of the initial motion region must be adjusted to an integer multiple of the block size so that multiple complete human image blocks can be segmented from the initial motion region. At this time, the adjusted initial motion region is determined to be a human motion region.

[0123] For example, if the block size is 16*16, and the initial motion area size is 31*31, then the initial motion area size will be adjusted to 32*32, where 32 is an integer multiple of 16.

[0124] In summary, by making the final human motion area an integer multiple of the block size, this embodiment ensures that the human motion area is perfectly divided into human image blocks, avoiding the loss of image information in the human motion area, thereby improving the accuracy of human action behavior recognition.

[0125] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0126] In one embodiment, such as Figure 14 The diagram shown is a structural schematic of a human motion and behavior recognition device disclosed in this application embodiment. This device is suitable for electronic devices with video and image processing capabilities, such as mobile phones, tablets, laptops, personal computers, and servers. The device in this embodiment includes a video frame image acquisition module 1401, a motion region recognition module 1402, an image block segmentation module 1403, and a motion and behavior classification module 1404, as detailed below: The video frame image acquisition module 1401 is used to acquire the first video frame image in the video data; The motion region recognition module 1402 is used to perform human motion detection on each frame of the first video frame image to obtain the human motion region in each frame of the first video frame image. The image block segmentation module 1403 is used to segment the human motion region in the first video frame image according to the preset segmentation size to obtain the human image block corresponding to each first video frame image. The action and behavior classification module 1404 is used to input human image blocks and preset behavior description text into a preset human action and behavior recognition model to obtain the action and behavior categories of video data.

[0127] In summary, this application discloses a human action behavior recognition device that acquires a first video frame image from video data, performs human motion detection on each first video frame image to obtain the human motion region in each first video frame image, segments the human motion region in the first video frame image according to a preset block size to obtain a human image block corresponding to each first video frame image, and finally inputs the human image block and a preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data. It is evident that this application, by downsampling the human image block compared to the prior art of obtaining downsampled images from high-resolution images, effectively preserves image details at high resolution while satisfying the model input requirements, thereby improving the detection rate of human action behavior recognition.

[0128] In one embodiment, the image block segmentation module 1403 can be used for: The first video frame image is sampled at preset intervals to obtain the second video frame image; According to the block size, the human motion region in the second video frame image is segmented to obtain the human image block corresponding to each second video frame image.

[0129] In one embodiment, the human motion behavior recognition model includes a hybrid modality fusion module and a behavior classification module; Action and behavior classification module 1404 can be used for: Spatiotemporal joint coding of human image blocks is performed to construct a spatiotemporal coded data sequence; The spatiotemporal encoded data sequence and behavioral description text are input into the mixed modality fusion module to obtain action behavior fusion features; The action behavior fusion features are input into the behavior classification module to obtain the action behavior category.

[0130] In one embodiment, the hybrid modality fusion module includes a visual encoder, a text encoder, and a feature fusion module; Action and behavior classification module 1404 can also be used for: The spatiotemporal encoded data sequence is input into the visual encoder to obtain the spatiotemporal joint feature sequence corresponding to the spatiotemporal encoded data sequence; Input the behavior description text into the text encoder to obtain the description text features corresponding to the behavior description text; The spatiotemporal joint feature sequence and descriptive text features are input into the feature fusion module for feature fusion to obtain action behavior fusion features.

[0131] In one embodiment, the action classification module 1404 can also be used for: Feature separation is performed on the spatiotemporal joint feature sequence to obtain the human body spatial domain feature sequence and the human body temporal domain feature sequence; The human spatial feature sequence, human temporal feature sequence and descriptive text feature are input into the feature fusion module and cross-attention feature fusion is performed to obtain spatiotemporal fusion features, spatial text fusion features and temporal text fusion features. The spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features are concatenated to obtain the action behavior fusion features.

[0132] In one embodiment, the human action behavior recognition model further includes a recommendation module; Action and behavior classification module 1404 can also be used for: The action behavior fusion features are input into the behavior classification module to obtain the classification category corresponding to each human body image patch; Input the category into the recommendation module to obtain the action / behavior category.

[0133] In one embodiment, the action classification module 1404 can also be used for: The recommendation module determines the target category corresponding to the human image patch in the target frame image from the classification categories; the target frame image is the center frame image in the first video frame image; The proportion of each target category is calculated based on the number of human image blocks and the number of each target category in the target frame image. Select the target category corresponding to the largest percentage value as the action behavior category.

[0134] In one embodiment, the behavior classification module includes a shared classification branch and a differential learning branch; Action and behavior classification module 1404 can also be used for: The action behavior fusion features are input into the shared classification branch and the differential learning branch respectively, to obtain the first classification category corresponding to each human image patch output by the shared classification branch and the second classification category corresponding to each human image patch output by the differential learning branch.

[0135] In one embodiment, the human action recognition model is trained as follows: Obtain training image patch samples and training text samples; The training image patch samples and training text samples are input into the human action behavior recognition model to obtain the training fusion features output by the hybrid modality fusion module; The training fusion features are input into the shared classification branch and the differential learning branch respectively, resulting in the first predicted category output by the shared classification branch and the second predicted category output by the differential learning branch. Input the first and second predicted categories into the recommendation module to obtain the third predicted category; Based on the training fusion features and the third prediction category, the causal loss value of the human action behavior recognition model is calculated. Determine whether the causal loss value meets the preset termination condition; If the causal loss value meets the preset termination condition, it is determined that the human action behavior recognition model has been sufficiently trained. If the causal loss value does not meet the preset termination condition, the human action behavior recognition model is iteratively trained based on training image patch samples and training text samples until the causal loss value meets the preset termination condition, thus determining that the human action behavior recognition model has been sufficiently trained. During the training of the human action and behavior recognition model, the parameters of the shared classification branch are frozen.

[0136] In one embodiment, the causal loss value is calculated using the following formula: in, Indicates training fusion features, Indicates the third prediction category, This represents the interference features after adding noise to the fused features during training. This represents the weight parameters and bias vector of the differential learning branch. Represents the regularization coefficient. Indicates the number of categories. Represents the weight parameters. This represents the maximum value among all third-prediction categories output by the human action recognition model based on the fused features from training. This indicates the third prediction category index.

[0137] In one embodiment, the motion region recognition module 1402 can be used for: Human motion detection is performed on the first video frame of each frame to obtain the initial motion region in the first video frame of each frame. Determine if the size of the initial motion region is an integer multiple of the block size; If the size of the initial motion region is an integer multiple of the block size, the initial motion region is determined to be the human motion region; If the initial motion region size is not an integer multiple of the block size, then adjust the initial motion region size to an integer multiple of the block size to obtain the human motion region.

[0138] For specific limitations regarding the human motion recognition device, please refer to the limitations on the human motion recognition method mentioned above, which will not be repeated here. Each module in the aforementioned human motion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0139] In one embodiment, such as Figure 15 As shown, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the human action behavior recognition method described in the above embodiments, for example... Figure 1 The human motion and behavior recognition method shown, or Figures 2 to 13 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the human motion and behavior recognition device, for example... Figure 14 The human motion and behavior recognition functions shown will not be described again here to avoid repetition.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the human action behavior recognition method described in the above embodiment, for example... Figure 1 The human motion and behavior recognition method shown, or Figures 2 to 13 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the human motion and behavior recognition device, for example... Figure 14 The human motion and behavior recognition functions shown will not be described again here to avoid repetition.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0143] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for recognizing human actions and behaviors, characterized in that, include: Obtain the first video frame image from the video data; Human motion detection is performed on each frame of the first video frame to obtain the human motion region in each frame of the first video frame. According to the preset block size, the human motion region in the first video frame image is segmented to obtain human image blocks corresponding to each frame of the first video frame image. The human image block and the preset behavior description text are input into a preset human action behavior recognition model to obtain the action behavior category of the video data.

2. The human motion behavior recognition method as described in claim 1, characterized in that, The step of segmenting the human motion region in the first video frame image according to a preset block size to obtain human image blocks corresponding to each frame of the first video frame image includes: The first video frame image is sampled at preset intervals to obtain the second video frame image; According to the block size, the human motion region in the second video frame image is segmented to obtain human image blocks corresponding to each frame of the second video frame image.

3. The human motion behavior recognition method as described in claim 1, characterized in that, The human motion behavior recognition model includes a hybrid modality fusion module and a behavior classification module; The step of inputting the human image patch and a preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data includes: The human image blocks are spatiotemporally co-coded to construct a spatiotemporally coded data sequence; The spatiotemporal encoded data sequence and the behavior description text are input into the mixed modality fusion module to obtain action behavior fusion features; The fusion features of the action behavior are input into the behavior classification module to obtain the action behavior category.

4. The human motion behavior recognition method as described in claim 3, characterized in that, The hybrid modality fusion module includes a visual encoder, a text encoder, and a feature fusion module; The step of inputting the spatiotemporal encoded data sequence and the behavior description text into the mixed modality fusion module to obtain action behavior fusion features includes: The spatiotemporal encoded data sequence is input into the visual encoder to obtain the spatiotemporal joint feature sequence corresponding to the spatiotemporal encoded data sequence; The behavior description text is input into the text encoder to obtain the description text features corresponding to the behavior description text; The spatiotemporal joint feature sequence and the descriptive text features are input into the feature fusion module for feature fusion to obtain the action behavior fusion features.

5. The human motion behavior recognition method as described in claim 4, characterized in that, The step of inputting the spatiotemporal joint feature sequence and the descriptive text features into the feature fusion module for feature fusion to obtain the action behavior fusion features includes: The spatiotemporal joint feature sequence is subjected to feature separation to obtain the human body spatial domain feature sequence and the human body temporal domain feature sequence; The human spatial feature sequence, the human temporal feature sequence, and the descriptive text features are input into the feature fusion module for cross-attention feature fusion to obtain spatiotemporal fusion features, spatial text fusion features, and temporal text fusion features. The spatiotemporal fusion feature, the spatial text fusion feature, and the temporal text fusion feature are concatenated to obtain the action behavior fusion feature.

6. The human motion behavior recognition method as described in claim 3, characterized in that, The human motion and behavior recognition model also includes a recommendation module; The step of inputting the fused features of the action behavior into the behavior classification module to obtain the action behavior category includes: The action behavior fusion features are input into the behavior classification module to obtain the classification category corresponding to each human body image block; The classification category is input into the recommendation module to obtain the action / behavior category.

7. The human motion behavior recognition method as described in claim 6, characterized in that, The step of inputting the classification category into the recommendation module to obtain the action behavior category includes: The recommendation module determines the target category corresponding to the human image block in the target frame image from the classification categories; the target frame image is the center frame image in the first video frame image; The proportion of each target category is calculated based on the number of human image blocks in the target frame image and the number of each target category. Select the target category corresponding to the largest percentage value as the action behavior category.

8. The human motion behavior recognition method as described in claim 6, characterized in that, The behavior classification module includes a shared classification branch and a differential learning branch; The step of inputting the fused action behavior features into the behavior classification module to obtain the classification category corresponding to each human image patch includes: The action behavior fusion features are respectively input into the shared classification branch and the differential learning branch to obtain the first classification category corresponding to each human image patch output by the shared classification branch and the second classification category corresponding to each human image patch output by the differential learning branch.

9. The human motion behavior recognition method as described in claim 8, characterized in that, The human motion and behavior recognition model is trained in the following way: Obtain training image patch samples and training text samples; The training image patch samples and training text samples are input into the human action behavior recognition model to obtain the training fusion features output by the hybrid modality fusion module; The training fusion features are input into the shared classification branch and the differential learning branch respectively to obtain the first predicted category output by the shared classification branch and the second predicted category output by the differential learning branch. The first predicted category and the second predicted category are input into the recommendation module to obtain the third predicted category; Based on the training fusion features and the third prediction category, the causal loss value of the human action behavior recognition model is calculated. Determine whether the causal loss value meets the preset termination condition; If the causal loss value meets the preset termination condition, it is determined that the human action behavior recognition model has been sufficiently trained. If the causal loss value does not meet the preset termination condition, the human action behavior recognition model is iteratively trained based on the training image patch samples and the training text samples until the causal loss value meets the preset termination condition, and it is determined that the human action behavior recognition model has been sufficiently trained. During the training of the human action behavior recognition model, the parameters of the shared classification branch are frozen.

10. The human motion behavior recognition method as described in claim 9, characterized in that, The calculation of the causal loss value of the human action behavior recognition model based on the training fusion features and the third prediction category includes: The trained fused features are then subjected to noise processing to obtain interference features; Based on the training fusion features and the third predicted category, the classification loss value is calculated; Based on the interference features and the training fusion features, the causal regularization loss value is calculated; The causal loss value is calculated based on the classification loss value and the causal regularization loss value.

11. The human motion behavior recognition method as described in claim 1, characterized in that, The step of performing human motion detection on each frame of the first video frame to obtain the human motion region in each frame of the first video frame includes: Human motion detection is performed on each frame of the first video frame to obtain the initial motion region in each frame of the first video frame. Determine whether the size of the initial motion region is an integer multiple of the block size; If the size of the initial motion region is an integer multiple of the block size, the initial motion region is determined to be the human motion region; If the size of the initial motion region is not an integer multiple of the block size, then the size of the initial motion region is adjusted to be an integer multiple of the block size to obtain the human motion region.

12. A human motion behavior recognition device, characterized in that, include: The video frame image acquisition module is used to acquire the first video frame image from the video data; The motion region recognition module is used to perform human motion detection on each frame of the first video frame image to obtain the human motion region in each frame of the first video frame image. The image block segmentation module is used to segment the human motion region in the first video frame image according to a preset segmentation size to obtain human image blocks corresponding to each frame of the first video frame image. The action behavior classification module is used to input the human image block and the preset behavior description text into a preset human action behavior recognition model to obtain the action behavior category of the video data.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the human action behavior recognition method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the human action behavior recognition method as described in any one of claims 1 to 11.

Images