Human behavior recognition methods, electronic devices and media

Abstract

This invention relates to the field of image processing technology, specifically providing a human behavior recognition method, electronic device, and medium. The aim is to address the problem that human behavior recognition based on a single modality often encounters difficulties in recognizing similar behaviors or is prone to overfitting. To this end, this invention acquires multi-frame feature information of the object to be recognized from video data, wherein the feature information includes at least RGB images, target bounding box information, and skeleton keypoint information; it then fuses the RGB images, target bounding box information, and skeleton keypoint information to generate a feature fusion image; and finally, it inputs the feature fusion image into a trained human behavior recognition model to obtain the behavior recognition result of the object to be recognized in the video data. This setup enables complementary information from different modal features during behavior recognition, thereby focusing on the actions of the object to be recognized and improving the accuracy and recall rate of behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically providing a human behavior recognition method, electronic device, and medium. Background Technology

[0002] Research on human behavior recognition has explored various modal features, including RGB images, optical flow images, and skeleton sequences. While single-modal feature representations offer some assistance in human behavior recognition, they also present challenges. For instance, when using skeleton sequences for human behavior recognition, although the model is simple, action-focused, and performs well for behaviors with clearly distinguishable actions, the lack of action details in skeleton sequences hinders the recognition of similar behaviors, such as jogging and walking. While RGB image-based human behavior recognition can capture more action details, it requires a large quantity and diversity of data, making it prone to overfitting.

[0003] Accordingly, there is a need in this field for a new method for human behavior recognition to address the aforementioned problems. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems, namely, the difficulty in recognizing similar behaviors or the problem of overfitting when human behavior recognition is based on a single modality feature.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for human behavior recognition, the method comprising the following steps:

[0006] Acquire multi-frame feature information of the object to be identified in video data, wherein the feature information includes at least RGB image, target location box information and skeleton key point information;

[0007] The RGB image, the target location bounding box information, and the skeleton key point information are fused to generate a feature fusion image;

[0008] The feature-fused image is input into a trained human behavior recognition model to obtain the behavior recognition result of the object to be identified in the video data.

[0009] In the optional technical solutions of the above-mentioned human behavior recognition method, the step of "fusing the RGB image, the target location box information, and the skeleton key point information to generate a feature fusion image" includes:

[0010] Extract the feature information from the multiple frames;

[0011] When the number of frames of the extracted feature information reaches a preset number of frames, the extracted RGB image, the target location box information, and the skeleton key point information are fused to generate the feature fusion image of the preset number of frames.

[0012] In the optional technical solutions of the above-mentioned human behavior recognition method, the step of "fusing the RGB image, the target location box information, and the skeleton key point information to generate a feature fusion image" includes:

[0013] The target location box information and the skeleton key point information are fused with the corresponding video frame to generate a first feature fusion image, and a limb heat map is generated based on the first feature fusion image.

[0014] A grayscale image is generated based on the RGB image, and the limb heatmap is fused with the grayscale image to generate the feature fusion image.

[0015] In the optional technical solutions of the above-mentioned human behavior recognition method, the method trains the model based on at least the following steps:

[0016] The training feature information of the training object to be identified in the training video data is obtained in multiple frames. A training feature fusion image is generated based on the training feature information, and the training feature fusion image is used as a training sample of the human behavior recognition model. The training feature information includes at least training RGB image, training target location box information and training skeleton key point information.

[0017] Construct a neural network model and use the neural network model as a human behavior recognition model to be trained;

[0018] The training samples are input into the human behavior recognition model to be trained for training, so as to obtain a trained human behavior recognition model.

[0019] In the optional technical solutions of the above-mentioned human behavior recognition method, the training target location box information includes annotation target location box information and tracking target location box information. The step of "obtaining the training target location box information of the training object to be identified in the training video data" includes:

[0020] Sampling and annotation of the target location bounding box information of the training object to be identified, to obtain the annotated target location bounding box information;

[0021] Based on the annotation location box information, perform forward and / or reverse tracking to generate the first tracking target location box information;

[0022] Obtain the frame distance between the first tracking target location box information and the annotation target location box information, and obtain the tracking weight based on the frame distance, wherein the frame distance is the number of frames between the video frame number where the first tracking target location box information is located and the video frame number where the annotation target location box information is located;

[0023] Based on the first tracking target location box information and the tracking weight, the tracking target location box information is obtained.

[0024] In the optional technical solutions of the above-mentioned human behavior recognition method, the step of "generating a training feature fusion image based on the training feature information" includes:

[0025] The training feature information is extracted based on a preset frame interval;

[0026] The training feature fusion image is generated based on the extracted training feature information and the corresponding training video frames.

[0027] In the optional technical solutions of the above-mentioned human behavior recognition method, the step of "generating the training feature fusion image based on the extracted training feature information and the corresponding training video frames" includes:

[0028] The union range of the training target location boxes is obtained based on the extracted training target location box information;

[0029] The corresponding training video frames are cropped based on the union range of the training target location boxes to obtain training cropped images;

[0030] The training cropped image is combined with the training skeleton key point information to generate a first training feature fusion image, and a training limb heat map is generated based on the first training feature fusion image.

[0031] A training grayscale image is generated based on the training RGB image, and the training limb heatmap is fused with the training grayscale image to generate the training feature fusion image.

[0032] In the optional technical solutions of the above-mentioned human behavior recognition method, the training feature information further includes the confidence level of training skeleton key points. The step of "generating a training limb heatmap based on the fused image of the first training feature" includes:

[0033] Based on the first training feature fusion image and the confidence level of the training skeleton key points, the brightness value of the training skeleton key points is determined.

[0034] A heat map of the training limb is generated based on the brightness values ​​of the key points of the training skeleton.

[0035] In a second aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the human behavior recognition method as described in any one of the above descriptions.

[0036] In a third aspect, the present invention also provides a readable storage medium storing a plurality of program codes adapted to be loaded and run by a processor to perform the human behavior recognition method described in any one of the above.

[0037] Those skilled in the art will understand that the technical solution of this invention involves acquiring multi-frame feature information of the object to be identified in video data, wherein the feature information includes at least RGB images, target location bounding box information, and skeleton keypoint information; fusing the RGB images, target location bounding box information, and skeleton keypoint information to generate a feature fusion image; and inputting the feature fusion image into a trained human behavior recognition model to obtain the behavior recognition result of the object to be identified in the video data. This setup enables complementary information from different modal features during behavior recognition, thereby focusing on the action of the object to be identified and improving the accuracy and recall rate of behavior recognition.

[0038] Furthermore, the step of "fusing RGB images, target bounding box information, and skeleton keypoint information to generate a feature fusion image" includes: extracting feature information from multiple frames; and when the number of extracted feature information frames reaches a preset number, fusing the extracted RGB images, target bounding box information, and skeleton keypoint information to generate a feature fusion image with the preset number of frames. This setup allows for expansion along the timeline using video frames during behavior recognition, reducing the dependence on and sensitivity to tracking performance, thereby improving the overall recall rate of behavior recognition. Attached Figure Description

[0039] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0040] Figure 1 This is a schematic diagram of the main steps of a human behavior recognition method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the main steps in generating a feature-fused image according to an embodiment of the present invention;

[0042] Figure 3This is a schematic diagram of the main steps for training a human behavior recognition model according to an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the main steps for obtaining the training target location box information of the training object to be identified in training video data according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of a training video frame containing annotation target location bounding box information and tracking target location bounding box information according to an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of a training limb heatmap generated based on the brightness values ​​of key points of the training skeleton according to an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of the main structure of an electronic device used to perform the human behavior recognition method of the present invention. Detailed Implementation

[0047] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0048] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0049] As described in the background section, in order to address the problem that human behavior recognition based on a single modality feature is prone to difficulties in recognizing similar behaviors or to overfitting, this invention provides a human behavior recognition method.

[0050] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a human behavior recognition method according to an embodiment of the present invention. Figure 1 As shown, the present invention provides a method for human behavior recognition, the method comprising the following steps:

[0051] Step S101: Obtain multi-frame feature information of the object to be identified in the video data, wherein the feature information includes at least RGB image, target location box information and skeleton key point information.

[0052] Specifically, the RGB color mode is an industry-standard color model. RGB images are obtained by varying the red (R), green (G), and blue (B) color channels and superimposing them to create images of various colors. Target bounding box information refers to the position of the target object's bounding box within the video frame. This can be obtained by using a target tracking algorithm. Skeleton keypoint information includes, but is not limited to, the coordinates of the skeleton keypoints of the target object. This can be obtained by using a skeleton keypoint detection model; and / or by using a motion sensing device, including but not limited to the Kinect v2 sensor, which can effectively capture human motion information and thus obtain the coordinates of various skeletal keypoints.

[0053] For example, the skeletal key points of the object to be identified can be 14, namely the head, neck, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, right hip, left knee, right knee, left ankle, and right ankle. The left hip and right hip are the hip joints of the left and right thighs, respectively, and the hip joint is the turning point of the femur relative to the pelvis. The left ankle and right ankle are the left and right ankle joints, respectively, and the ankle joint is the turning point of the talus relative to the fibula. The above-described method of setting skeletal key points is only an example; in practical applications, it can be selected according to actual needs.

[0054] Step S102: Fuse the RGB image, target location bounding box information, and skeleton key point information to generate a feature fusion image.

[0055] Specifically, the feature fusion image is a three-channel image. It is generated by fusing the RGB image, the target location box information, and the skeleton key point information to form three single-channel images, thereby fusing the three single-channel images to form a three-channel feature fusion image.

[0056] Step S103: Input the feature fusion image into the trained human behavior recognition model to obtain the behavior recognition result of the object to be recognized in the video data.

[0057] Specifically, the behavior recognition results of the object to be identified include, but are not limited to, the recognition of the current behavior and the prediction of the behavior at the next moment. The recognition of the current behavior includes, but is not limited to, identifying the behavior type, duration, speed, and judgment. Behavior types include at least walking, jogging, and running. Behavior judgment refers to determining whether the currently identified behavior type is abnormal, thereby facilitating the protection of the object's health. The prediction of the behavior at the next moment includes, but is not limited to, predicting the behavior type at the next moment, thus also facilitating better recognition of the object's behavior at the next moment. For example, an abnormal type can be trembling and / or convulsions. That is, when the currently identified behavior type is determined to be trembling and / or convulsions, it indicates that the current behavior type is abnormal, thereby allowing for appropriate protective measures to be taken for the object. The above-described method of setting abnormal types is only an example; in practical applications, it can be selected according to actual needs.

[0058] Based on the steps S101 to S103 described above, this invention acquires multi-frame feature information of the object to be identified in video data, wherein the feature information includes at least RGB images, target location bounding box information, and skeleton keypoint information; fuses the RGB images, target location bounding box information, and skeleton keypoint information to generate a feature fusion image; and inputs the feature fusion image into a trained human behavior recognition model to obtain the behavior recognition result of the object to be identified in the video data. This setup enables complementary information from different modal features during behavior recognition, thereby focusing on the action of the object to be identified and improving the accuracy and recall rate of behavior recognition.

[0059] See appendix Figure 2 , Figure 2 This is a schematic diagram illustrating the main steps in generating a feature-fused image according to an embodiment of the present invention. Figure 2 As shown, in some embodiments, fusing the RGB image, target location bounding box information, and skeleton keypoint information to generate a feature fusion image includes the following steps:

[0060] Step S201: Extract feature information from multiple frames.

[0061] Step S202: When the number of frames of extracted feature information reaches the preset number of frames, the extracted RGB image, target location box information and skeleton key point information are fused to generate a feature fusion image with the preset number of frames.

[0062] Specifically, depending on whether the input in human behavior recognition is single-frame or multi-frame information, human behavior recognition can be divided into two-dimensional (2D) behavior recognition and three-dimensional (3D) behavior recognition. When the input is a single frame, human behavior recognition is two-dimensional; when the input is multi-frame information, it is three-dimensional. Human behaviors that are difficult to distinguish using only single-frame information can be identified using a three-dimensional behavior recognition model. Furthermore, compared to two-dimensional behavior recognition models, the input to three-dimensional behavior recognition models mostly consists of consecutive frames from the same target tracking sequence. Therefore, the target tracking performance has a significant impact on the recognition of three-dimensional behavior models. If target tracking is easily interrupted or has a large deviation, it may prevent the formation of the input for a three-dimensional behavior recognition model, leading to missed detections of human behavior. Therefore, this embodiment extracts multi-frame feature information from video data, and generates the input of a three-dimensional behavior recognition model based on the extracted feature information. This setting can expand the extracted video frames on the time axis in human behavior recognition, weaken the dependence and sensitivity of behavior recognition on target tracking effect, and thus improve the overall recall rate of behavior recognition.

[0063] For example, when extracting feature information from multiple frames, it can be extracted every 4 frames based on the tracking sequence of the object to be identified; at the same time, the preset number of frames can be 8 frames, that is, when the number of extracted feature information frames reaches 8 frames, an 8-frame feature fusion image is generated based on the extracted RGB image, target location box information, and skeleton key point information. The feature information extraction method and the preset number of frames described above are only for illustrative purposes, and can be selected according to actual needs in practical applications.

[0064] In some embodiments, after the number of frames of extracted feature information reaches a preset number of frames, and before generating a feature fusion image of the preset number of frames based on the extracted RGB image, target location box information, and skeleton key point information, the method further includes the following steps:

[0065] Step S301: Determine whether the recognition time interval has reached the preset duration.

[0066] Step S302: When the recognition time interval reaches the preset duration, generate a feature fusion image with a preset number of frames based on the extracted RGB image, target location box information and skeleton key point information.

[0067] Specifically, the recognition time interval is the time elapsed between the last human behavior recognition and the current moment. This setting can avoid frequent human behavior recognition and reduce the cost of using behavior recognition.

[0068] For example, the preset duration can be 20 seconds. That is, if the time interval between the last human behavior recognition and the current moment is greater than or equal to 20 seconds, a feature fusion image is generated, and then human behavior is recognized based on the feature fusion image; if the time interval between the last human behavior recognition and the current moment is less than 20 seconds, a feature fusion image is not generated, and therefore no human behavior recognition is performed. The preset duration value described above is only for illustrative purposes, and can be selected according to actual needs in practical applications.

[0069] In some embodiments, fusing RGB images, target location bounding box information, and skeleton keypoint information to generate a feature fusion image includes the following steps:

[0070] Step S401: The target location box information and skeleton key point information are fused with the corresponding video frame to generate a first feature fusion image, and a limb heat map is generated based on the first feature fusion image.

[0071] Step S402: Generate a grayscale image based on the RGB image, and fuse the limb heatmap with the grayscale image to generate a feature fusion image.

[0072] Specifically, the method involves obtaining the union of target location bounding boxes from multiple frames, generating a regular graphic range based on this union. The regular graphic range is the area encompassed by expanding the union of the target location bounding boxes into a regular graphic, which may include, but is not limited to, a square. Multiple video frames are then cropped based on this regular graphic range to generate cropped images, which contain only the regular graphic range from the video frames. Skeleton keypoint information includes, but is not limited to, skeleton keypoint coordinates. This information is fused with the cropped images, aligning the skeleton keypoint coordinates with the coordinates in the cropped images, resulting in a first feature fusion image containing the coordinate information of the skeleton keypoints. The skeleton keypoint information may further include skeleton keypoint confidence. The brightness value of the corresponding skeleton keypoint is determined based on the first feature fusion image and the skeleton keypoint confidence, allowing different skeleton keypoints in the limb heatmap to have different brightness values, thus facilitating the recognition of the object's behavior.

[0073] Generating grayscale images from RGB images involves converting a three-channel RGB color image into a single-channel grayscale image. A limb heatmap is then fused with the grayscale image to generate a feature fusion image. This feature fusion image is a multi-channel image, while both the grayscale image and the limb heatmap are single-channel images. The feature fusion image is generated by merging the image channels of the limb heatmap and the grayscale image.

[0074] For example, the number of limb heat maps generated based on the first feature fusion image can be one or more. For instance, a first limb heat map and a second limb heat map can be generated based on the first feature fusion image. The first limb heat map is the heat map of the left side of the body of the object to be identified, and the second limb heat map is the heat map of the right side of the body of the object to be identified. This allows the movement of the left and right sides of the body of the object to be identified to be determined separately through different heat maps, thereby improving the accuracy of behavior recognition.

[0075] When generating a grayscale image from an RGB image, at least one of the following methods can be used: averaging, max-min averaging, and weighted averaging. Averaging involves averaging the brightness values ​​of the three channels at the same pixel location in the RGB image to obtain the grayscale image. Max-min averaging involves averaging the maximum and minimum brightness values ​​of the three channels at the same pixel location in the RGB image to obtain the grayscale image. Weighted averaging involves averaging the brightness values ​​of the three channels at the same pixel location in the RGB image using weighted averages, where the weighting coefficients for the three channels can be, for example, 0.3, 0.59, and 0.11.

[0076] The settings for the number of limb heat maps and the method for generating grayscale images based on RGB images described above are merely illustrative examples. In practical applications, these settings can be selected according to actual needs.

[0077] It should be noted that although the description here involves first fusing the target location bounding box information and skeleton key point information with the corresponding video frames to generate a first feature fusion image, and then generating a limb heatmap based on the first feature fusion image, this is not limiting. Those skilled in the art can also first generate a complete limb heatmap based on the skeleton key point information and video frames; then obtain a regular graphic range based on the target location bounding box information from multiple frames, and thereby crop the complete limb heatmap based on the regular image range to generate a limb heatmap, wherein the limb heatmap is an image containing only the regular graphic range from the complete limb heatmap. These adjustments do not deviate from the basic principles of the present invention and therefore all fall within the protection scope of the present invention.

[0078] See appendix Figure 3 , Figure 3 This is a schematic diagram illustrating the main steps of training a human behavior recognition model according to an embodiment of the present invention. Figure 3 As shown, in some embodiments, the method trains the model based on at least the following steps:

[0079] Step S501: Obtain multi-frame training feature information of the training object to be identified in the training video data, generate a training feature fusion image based on the training feature information, and use the training feature fusion image as a training sample for the human behavior recognition model. The training feature information includes at least the training RGB image, the training target location box information, and the training skeleton key point information.

[0080] Specifically, the training skeleton key point information includes, but is not limited to, the coordinates of the skeleton key points of the training object to be identified, which can be obtained by calling the skeleton key point detection model and / or body posture perception device.

[0081] Step S502: Construct a neural network model and use the neural network model as a human behavior recognition model to be trained.

[0082] Specifically, because the training samples consist of multi-frame information, a 3D convolutional neural network model is constructed and used as the backbone network of the human behavior recognition model to be trained. For example, the constructed 3D convolutional neural network model could be a slow-only model, a popular 3D-CNN used for RGB-based action recognition, obtained by dilating ResNet layers from 2D to 3D in the last two stages; alternatively, the constructed 3D convolutional neural network model could also be a CSN model, a type of cortical spiking network. The above-described configuration of the 3D convolutional neural network model is merely illustrative; in practical applications, the appropriate model can be selected based on actual needs.

[0083] Step S503: Input the training samples into the human behavior recognition model to be trained for training, so as to obtain the trained human behavior recognition model.

[0084] The training of the human behavior recognition model will be further explained below.

[0085] See appendix Figure 4 , Figure 4 This is a schematic flowchart illustrating the main steps of obtaining the training target location bounding box information of the training object to be identified in training video data according to an embodiment of the present invention. Figure 4 As shown, in some embodiments, the training target location box information includes annotation target location box information and tracking target location box information. Obtaining the training target location box information of the training object to be identified in the training video data includes the following steps:

[0086] Step S601: Sample and annotate the target location bounding box information of the training object to be identified in order to obtain the annotated target location bounding box information.

[0087] Step S602: Perform forward and / or reverse tracking based on the annotation location box information to generate the first tracking target location box information.

[0088] Step S603: Obtain the frame distance between the first tracking target location box information and the annotation target location box information, and obtain the tracking weight based on the frame distance, wherein the frame distance is the number of frames between the video frame number where the first tracking target location box information is located and the video frame number where the annotation target location box information is located.

[0089] Step S604: Based on the first tracking target location box information and the tracking weight, obtain the tracking target location box information.

[0090] Specifically, annotation of training video frames is performed on a single-person, single-session basis. This involves annotating the start and end times of the action to be identified for the training object within each frame, and sampling the target bounding box information of the training object within that time frame. The interval between these sampling annotations can be determined based on the specific training video data. Based on the annotated bounding box information, forward and / or reverse tracking is performed to generate the first tracking target bounding box information. Forward tracking starts from the video frame containing the annotated bounding box and proceeds in ascending order of frame number; reverse tracking starts from the video frame containing the annotated bounding box and proceeds in descending order of frame number. Tracking weights are obtained based on frame distance; the larger the frame distance, the smaller the corresponding tracking weight. This setup eliminates the need for precise annotation of the tracking target location bounding box information, greatly reducing the workload of manual annotation. At the same time, this method of obtaining tracking target location bounding box information is easy to expand, improving the efficiency of extended training. Furthermore, for tracking target location bounding box information that is difficult to track and cannot be obtained, the model can be trained to be robust to tracking deficiencies, while reducing the dependence on tracking performance during recognition.

[0091] For example, when annotating the start and end times of the behavior to be recognized in the training video data, and sampling and annotating the target location bounding box information of the training object to be recognized within the start and end times, A = (f start ,f end ,(f1,loc1),(f2,loc2),...,(f i ,loc i The process is carried out in the form of )), where A refers to the behavior to be identified and annotated, and f start and f end These represent the training video frame numbers corresponding to the occurrence of the behavior to be identified and the end of the behavior to be identified in the training video data, respectively. i ,loc i ) indicates from fstart to f end The target location bounding box information of the training object to be identified is sampled and annotated in this video frame number, where f start ≤f i ≤f end When sampling and annotating the target location bounding box information of a behavior to be identified, it can be done every 5-10 frames. When the human posture of the training object to be identified changes little, the number of sampling and annotations can be reduced; when the human posture of the training object to be identified changes greatly, the number of sampling and annotations can be increased.

[0092] A schematic diagram of training video frames containing annotation target bounding box information and tracking target bounding box information can be shown as follows: Figure 5 As shown. In Figure 5 The first frame in the diagram represents the training video frame number corresponding to the occurrence of the behavior to be identified, and the last frame represents the training video frame number corresponding to the end of the behavior to be identified. Five frames are omitted between the second and third frames, and similarly, five frames are omitted between the last and penultimate frames. Figure 5 In the first frame image, the target location box information is the annotation location box information, and the target location box information in the ninth frame image is also the annotation location box information. The target location box information in the remaining frames image is the tracking target location box information, that is, it is all tracked based on the annotation target location box information in the first frame image and the annotation target location box information in the ninth frame image.

[0093] Taking the acquisition of the tracking target location box information in the second frame image as an example, the process of acquiring the tracking target location box information is explained. Specifically, forward tracking is performed based on the annotation target location box information in the first frame image to obtain the tracking target location box information in the first frame. Reverse tracking is performed based on the annotation target location box information in the ninth frame image to obtain the tracking target location box information in the ninth frame. The tracking target location box information in the first frame can be [Ax1, Ay1, Ax2, Ay2], where (Ax1, Ay1) is the coordinate of the upper left corner of the target location box in the first frame, and (Ax2, Ay2) is the coordinate of the lower right corner of the target location box in the first frame. The tracking target location box information in the ninth frame can be [Bx1, By1, Bx2, By2], where (Bx1, By1) is the coordinate of the upper left corner of the target location box in the ninth frame, and (Bx2, By2) is the coordinate of the lower right corner of the target location box in the ninth frame. The frame distance between the second frame and the first frame is 1 frame, and the frame distance between the second frame and the ninth frame is 7 frames. Therefore, the tracking weight corresponding to the target location box information in the first frame can be set to 7, and the tracking weight corresponding to the target location box information in the ninth frame can be set to 1. Thus, the tracking location box information in the second frame can be calculated using the formulas x1=(Ax1*7+Bx1*1) / (7+1), y1=(Ay1*7+By1*1) / (7+1), x2=(Ax2*7+Bx2*1) / (7+1), and y2=(Ay2*7+By2*1) / (7+1).

[0094] The above-described methods for annotating the start and end times of training video data, sampling annotation of the annotation target location box information of training video data, setting of training video frames containing annotation target location box information and tracking target location box information, and methods for obtaining tracking target location box information are all just illustrative examples. In practical applications, they can be selected according to actual needs.

[0095] In some embodiments, generating a training feature fusion image based on training feature information includes the following steps:

[0096] Step S701: Extract training feature information based on a preset frame interval.

[0097] Step S702: Generate a training feature fusion image based on the extracted training feature information and the corresponding training video frames.

[0098] Specifically, the start and end times of the behaviors to be identified by the training object in the training video data are annotated. A start frame is randomly generated within the annotated start and end time period. Training feature information is extracted based on the randomly generated start frame and a preset frame interval. It is then determined whether the number of frames of extracted training feature information has reached the preset number of training frames. When the number of frames of extracted training feature information has reached the preset number of training frames, a training feature fusion image is generated based on the extracted training feature information and the corresponding training video frames.

[0099] For example, the preset frame interval can be 4 frames, and the preset training frame count can be 8 frames. That is, a starting frame is randomly generated within the annotation start and end time period, and training feature information is extracted every 4 frames starting from the starting frame; when the number of extracted training feature information frames reaches 8 frames, a training feature fusion image is generated based on the extracted training feature information and the corresponding training video frames. The preset frame interval and the preset training frame count described above are only illustrative examples, and can be selected according to actual needs in practical applications.

[0100] In some embodiments, generating a training feature fusion image based on extracted training feature information and corresponding training video frames includes the following steps:

[0101] Step S801: Obtain the union range of the training target location boxes based on the extracted training target location box information.

[0102] Step S802: Crop the corresponding training video frames based on the union range of the training target location boxes to obtain the training cropped image.

[0103] Step S803: Combine the training cropped image with the training skeleton key point information to generate a first training feature fusion image, and generate a training limb heat map based on the first training feature fusion image.

[0104] Step S804: Generate a training grayscale image based on the training RGB image, and fuse the training limb heatmap with the training grayscale image to generate a training feature fusion image.

[0105] The generation of training feature fusion images follows the same principle as the generation of feature fusion images. Specifically, based on the extracted training target location box information from a preset number of training frames, the union range of the training target location boxes is obtained. Then, a training regular graphic range is generated based on this union range. The union range of the training target location boxes is the same as the union range of the training target location boxes from the preset number of training frames, and the training regular graphic range is the range included after expanding the union range of the training target location boxes into a regular graphic, such as a square. Based on the training regular graphic range, the corresponding preset number of training video frames are cropped to generate a training cropped image for that preset number of training frames. This cropped image contains only the training regular graphic range from the training video frames.

[0106] The training skeleton keypoint information includes, but is not limited to, the coordinates of the training skeleton keypoints. The training skeleton keypoint information is then fused with the training cropped image, aligning the coordinates of the training skeleton keypoints with those in the training cropped image. This results in the generated first training feature fusion image containing the coordinate information of the training skeleton keypoints. The training grayscale image generated from the training RGB image is a single-channel image, and the training limb heatmap is also a single-channel image. The training limb heatmap and the training grayscale image are then fused into a training feature fusion image by merging the image channels.

[0107] In some embodiments, the training feature information further includes training skeleton keypoint confidence. Generating a training limb heatmap based on the first training feature fusion image includes the following steps:

[0108] Step S901: Determine the brightness value of the training skeleton key points based on the first training feature fusion image and the confidence of the training skeleton key points.

[0109] Step S902: Generate a heat map of the training limbs based on the brightness values ​​of key points of the training skeleton.

[0110] The generation of training limb heatmaps follows the same principle as the generation of limb heatmaps. For example, the number of training limb heatmaps generated based on the first training feature fusion image can be one or more. For instance, a first training limb heatmap and a second training limb heatmap can be generated based on the first training feature fusion image. The first training limb heatmap is a heatmap of the left side of the training subject's body; the second training limb heatmap is a heatmap of the right side of the training subject's body. This allows for the determination of the movement on the left and right sides of the training subject's body using different heatmaps, improving the accuracy of behavior recognition.

[0111] Each training limb heatmap is a single-channel image, meaning that the brightness value of each training limb heatmap can be represented by 256 numbers between 0 and 255. Furthermore, when a training limb heatmap includes both a first training limb heatmap and a second training limb heatmap, the brightness values ​​of different training limb heatmaps can be used to represent the key points of the training skeleton on different sides. Specifically, the brightness values ​​of the first training limb heatmap can be used to represent the key points of the training skeleton on the left side of the subject's body, and the brightness values ​​of the second training limb heatmap can be used to represent the key points of the training skeleton on the right side of the subject's body.

[0112] Specifically, taking the heatmap of the first training limb and the key points of the training skeleton on the left side of the training subject as examples, the key points of the training skeleton on the left side can include 8 points: head, neck, left shoulder, left elbow, left hand, left hip, left knee, and left ankle. The 256 brightness values ​​of the heatmap of the first training limb are divided into 8 groups, and the brightness values ​​of each group are used as the brightness value range of a key point of the training skeleton on the left side. For example, brightness values ​​0-31, 32-63, 64-95, 96-127, 128-159, 160-191, 192-223, and 224-255 are the brightness value ranges of the head, neck, left shoulder, left elbow, left hand, left hip, left knee, and left ankle, respectively. Furthermore, based on the brightness value range and the corresponding confidence scores of the training skeleton keypoints, initial brightness values ​​of the training skeleton keypoints are obtained. These initial brightness values ​​are then rounded to obtain the final brightness value. The rounding method can include at least one of rounding to the nearest whole number, rounding up, or rounding down. For example, if the confidence score for the left shoulder is 0.8 and the confidence score for the left hand is 0.9, then the initial brightness value for the left shoulder of the training object to be identified is: 63 + 32 * 0.8 = 88.6, and the rounded brightness value is 89; the initial brightness value for the left hand of the training object to be identified is: 127 + 32 * 0.9 = 155.8, and the rounded brightness value is 156.

[0113] The method for determining the brightness values ​​of the key points on the right side of the training limb based on the second training limb heatmap and the key points of the training skeleton on the right side of the subject to be identified is similar to that on the left side, and will not be repeated here. Once the brightness values ​​of the key points on both the left and right sides of the subject's body are determined, a schematic diagram of the training limb heatmap generated based on these brightness values ​​can be shown as follows: Figure 6 As shown.

[0114] The above-described methods for setting up the training limb heatmap, determining the brightness value of the training skeleton key points based on the image fusion of the first training feature and the confidence level of the training skeleton key points, and the generated training limb heatmap are all just illustrative examples. In actual applications, the appropriate method can be selected according to actual needs.

[0115] It should be noted that the videos involving human behavior involved in the embodiments of this disclosure (including but not limited to training video data used for training, video data in actual recognition, etc.) are all videos authorized by the user or fully authorized by all parties.

[0116] The actions involved in the video acquisition and other related actions in this disclosure embodiment are all performed after authorization by the user, the object, or after full authorization by all parties.

[0117] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that, in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in this order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention. Furthermore, all the above-described embodiments can be combined in any way to form optional embodiments of the present invention, which will not be elaborated upon here.

[0118] Furthermore, the present invention also provides an electronic device.

[0119] See appendix Figure 7 , Figure 7 This is a schematic diagram of the main structure of an electronic device used to execute the human behavior recognition method of the present invention. Figure 7 As shown, the present invention also provides an electronic device for executing the human behavior recognition method of the present invention. The electronic device 700 includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the various method embodiments described above. Alternatively, when the processor 701 executes the computer program 703, it implements the functions of each module / unit in the above embodiments.

[0120] For example, the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0121] For example, memory 702 can be an internal storage unit of an electronic device, such as a hard disk or memory; memory 702 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. Furthermore, memory 702 can include both internal and external storage units of the electronic device. Memory 702 is used to store computer programs and other programs and data required by the electronic device, and memory 702 can also be used to temporarily store data that has been output or will be output.

[0122] In some possible implementations, the electronic device may include multiple processors 701 and memory 702. The program executing the human behavior recognition method of the above-described method embodiments can be divided into multiple subroutines, each of which can be loaded and run by a processor 701 to perform different steps of the human behavior recognition method of the above-described method embodiments. Specifically, each subroutine can be stored in a different memory 702, and each processor 701 can be configured to execute programs in one or more memories 702 to jointly implement the human behavior recognition method of the above-described method embodiments. That is, each processor 701 executes different steps of the face occlusion determination method of the above-described method embodiments to jointly implement the human behavior recognition method of the above-described method embodiments.

[0123] The aforementioned multiple processors 701 can be processors deployed on the same device. For example, the aforementioned electronic device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors 701 can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors 701 can also be processors deployed on different devices. For example, the aforementioned electronic device can be a server cluster, and the aforementioned multiple processors 701 can be processors on different servers within the server cluster.

[0124] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The electronic device may include, but is not limited to, the processor 701 and the memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, an electronic device may also include input / output devices, network access devices, buses, etc.

[0125] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that performs the human behavior recognition method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described human behavior recognition method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0126] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0127] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0128] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for recognizing human behavior, characterized in that, The method includes the following steps: Acquire multi-frame feature information of the object to be identified in video data, wherein the feature information includes at least RGB image, target location box information and skeleton key point information; The process of fusing the RGB image, the target location bounding box information, and the skeleton key point information to generate a feature fusion image specifically includes: fusing the target location bounding box information and the skeleton key point information with the corresponding video frame to generate a first feature fusion image, and generating a limb heatmap based on the first feature fusion image; generating a grayscale image based on the RGB image, and fusing the limb heatmap with the grayscale image to generate the feature fusion image. The feature-fused image is input into a trained human behavior recognition model to obtain the behavior recognition result of the object to be identified in the video data; The method trains the model based on at least the following steps: The process involves: acquiring multi-frame training feature information of the target training object from training video data; generating a training feature fusion image based on the training feature information; and using the training feature fusion image as a training sample for the human behavior recognition model. The training feature information includes at least a training RGB image, training target bounding box information, and training skeleton keypoint information. A neural network model is then constructed and used as the human behavior recognition model to be trained. The training samples are input into the human behavior recognition model to be trained for training, resulting in a completed human behavior recognition model. The training target location box information includes annotation target location box information and tracking target location box information. The step of "obtaining the training target location box information of the training object to be identified in the training video data" includes: sampling and annotating the target location box information of the training object to be identified to obtain the annotation target location box information; performing forward and / or reverse tracking based on the annotation target location box information to generate first tracking target location box information; obtaining the frame distance between the first tracking target location box information and the annotation target location box information, and obtaining the tracking weight based on the frame distance, wherein the frame distance is the number of frames between the video frame number where the first tracking target location box information is located and the video frame number where the annotation target location box information is located; and obtaining the tracking target location box information based on the first tracking target location box information and the tracking weight.

2. The human behavior recognition method according to claim 1, characterized in that, The step of "fusing the RGB image, the target location bounding box information, and the skeleton key point information to generate a feature fusion image" includes: Extract the feature information from the multiple frames; When the number of frames of the extracted feature information reaches a preset number of frames, the extracted RGB image, the target location box information, and the skeleton key point information are fused to generate the feature fusion image of the preset number of frames.

3. The human behavior recognition method according to claim 1, characterized in that, The step of "generating a training feature fusion image based on the training feature information" includes: The training feature information is extracted based on a preset frame interval; The training feature fusion image is generated based on the extracted training feature information and the corresponding training video frames.

4. The human behavior recognition method according to claim 3, characterized in that, The step of "generating the training feature fusion image based on the extracted training feature information and the corresponding training video frames" includes: The union range of the training target location boxes is obtained based on the extracted training target location box information; The corresponding training video frames are cropped based on the union range of the training target location boxes to obtain training cropped images; The training cropped image is combined with the training skeleton key point information to generate a first training feature fusion image, and a training limb heat map is generated based on the first training feature fusion image. A training grayscale image is generated based on the training RGB image, and the training limb heatmap is fused with the training grayscale image to generate the training feature fusion image.

5. The human behavior recognition method according to claim 4, characterized in that, The training feature information also includes the confidence level of training skeleton key points. The step of "generating a training limb heatmap based on the image fused with the first training feature" includes: Based on the first training feature fusion image and the confidence level of the training skeleton key points, the brightness value of the training skeleton key points is determined. A heat map of the training limb is generated based on the brightness values ​​of the key points of the training skeleton.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the human behavior recognition method as described in any one of claims 1 to 5.

7. A readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the human behavior recognition method according to any one of claims 1 to 5.

Images