Behavior recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a behavior recognition method and device, electronic equipment and a storage medium. The method comprises the following steps: performing key point detection on a to-be-recognized image to obtain personnel key point position information in the to-be-recognized image; determining personnel posture information based on the personnel key point position information; and determining a behavior recognition result based on the personnel posture information. The behavior recognition method and device, electronic equipment and storage medium provided by the application can quickly and accurately determine a behavior recognition result.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more particularly to a behavior recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] As people place increasing emphasis on food safety, the hygiene and safety of livestock farms, a crucial link in food production, are receiving more and more attention. Due to the poor environment of livestock farms, staff entering and leaving can easily lead to a series of hygiene and safety issues. Therefore, employees need to change their shoes and disinfect themselves before entering and leaving the farm.

[0003] Currently, farm staff rely on changing shoes and disinfecting themselves when entering and leaving the farm. However, the high frequency of entry and exit can easily lead to oversights, resulting in omissions in changing shoes and / or disinfection, which in turn poses a health and safety hazard. Summary of the Invention

[0004] This invention provides a behavior recognition method, device, electronic device, and storage medium to address the shortcomings of existing technologies where farm workers easily overlook changing shoes and / or disinfecting, leading to potential hygiene and safety hazards.

[0005] This invention provides a behavior recognition method, comprising:

[0006] Key point detection is performed on the image to be identified to obtain the location information of key points of people in the image to be identified;

[0007] Based on the personnel's key point location information, determine the personnel's posture information;

[0008] Based on the person's posture information, the behavior recognition result is determined.

[0009] According to a behavior recognition method provided by the present invention, determining the person's posture information based on the person's key point location information includes:

[0010] The person's posture information is determined based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person, and the aspect ratio of the person obtained by target detection of the image to be identified, or based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person.

[0011] According to a behavior recognition method provided by the present invention, determining the person's posture information based on the wrist and ankle coordinates in the person's key point location information includes:

[0012] The wrist-ankle position relationship in the person's posture information is determined based on at least one of the distances between the left wrist coordinate and the left ankle coordinate and the right ankle coordinate, and the distances between the right wrist coordinate and the left ankle coordinate and the right ankle coordinate.

[0013] According to a behavior recognition method provided by the present invention, determining the person's posture information based on the shoulder and hip coordinates in the person's key point location information includes:

[0014] Based on the coordinates of the left and right shoulders in the shoulder-hip coordinate system, determine the coordinates of the shoulder center point.

[0015] Based on the left and right hip coordinates in the shoulder and hip coordinates, determine the coordinates of the center point of the hip.

[0016] Based on the coordinates of the shoulder center point and the coordinates of the hip center point, the tilt angle of the person in the posture information is determined.

[0017] According to a behavior recognition method provided by the present invention, determining the behavior recognition result based on the person's posture information includes:

[0018] Based on the human posture information of multiple consecutive frames of images to be identified, the behavior type of each image to be identified in the multiple consecutive frames of images to be identified is determined;

[0019] If the number of images to be identified with the same behavior type is greater than a preset number, then the behavior type of the images to be identified with the same behavior type is taken as the behavior recognition result.

[0020] According to a behavior recognition method provided by the present invention, determining the behavior recognition result based on the person's posture information includes:

[0021] Based on the personnel posture information, the initial behavior type is determined;

[0022] If the initial behavior type is a target action, then the initial behavior type is verified based on the location information of the target object corresponding to the target action in the image to be identified, and the location information of the key points of the person, to obtain the behavior recognition result.

[0023] According to a behavior recognition method provided by the present invention, the step of verifying the initial behavior type based on the location information of the target object corresponding to the target action in the image to be recognized, and the location information of the key points of the person, to obtain the behavior recognition result includes:

[0024] Based on the location information of the target item and the wrist coordinates in the key point location information of the person, the distance between the target item and the wrist is calculated, and when the distance between the target item and the wrist is less than a threshold, the target action is taken as the behavior recognition result.

[0025] According to a behavior recognition method provided by the present invention, determining the behavior recognition result based on the person's posture information includes:

[0026] The behavior recognition result is determined based on the person's posture information and the image features of the image to be recognized.

[0027] The present invention also provides a behavior recognition device, comprising:

[0028] The detection unit is used to detect key points in the image to be identified, and obtain the location information of key points of people in the image to be identified;

[0029] The determining unit is used to determine the personnel posture information based on the personnel key point position information;

[0030] The recognition unit is used to determine the behavior recognition result based on the person's posture information.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described behavior recognition methods.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described behavior recognition methods.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the behavior recognition methods described above.

[0034] The behavior recognition method, apparatus, electronic device, and storage medium provided by this invention perform key point detection on the image to be recognized to obtain personnel posture information, thereby enabling rapid and accurate determination of behavior recognition results based on the personnel posture information. Furthermore, since the personnel posture information is obtained after key point detection on the image to be recognized, it contains feature information of the key points, thus enabling accurate determination of behavior recognition results based on the personnel posture information, avoiding the problem in traditional methods where noise in the image to be recognized prevents accurate acquisition of behavior recognition results. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the behavior recognition method provided by the present invention;

[0037] Figure 2 This is a flowchart illustrating an implementation of step 120 in the behavior recognition method provided by the present invention;

[0038] Figure 3 This is one of the flowcharts illustrating an implementation of step 130 in the behavior recognition method provided by the present invention;

[0039] Figure 4 This is a second schematic flowchart of an implementation of step 130 in the behavior recognition method provided by the present invention;

[0040] Figure 5 This is the second flowchart of the behavior recognition method provided by the present invention;

[0041] Figure 6 This is a schematic diagram of the behavior recognition device provided by the present invention;

[0042] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Currently, farm staff rely on changing shoes and disinfecting themselves when entering and exiting the farm. However, the high frequency of entry and exit can easily lead to oversights, resulting in omissions of shoe changing and / or disinfection. Some staff may have a sense of complacency, thinking that missing one operation will have little impact on the farm, and thus fail to change shoes and / or disinfect, which can lead to hygiene and safety hazards.

[0045] In addition, traditional methods involve acquiring images of staff through video surveillance, and then manually checking the images to confirm whether the staff have completed the shoe changing and disinfection procedures. However, this method relies entirely on manual inspection, which is not only inefficient, but also prone to false positives or false negatives due to human error.

[0046] Furthermore, traditional methods also involve inputting images of staff captured by video surveillance into a network for behavioral analysis to confirm whether staff have completed the shoe-changing and disinfection procedures. However, the images of staff captured by video surveillance may contain noise, which can affect the accuracy of the network analysis results.

[0047] In response, the present invention provides a behavior recognition method. Figure 1 This is one of the flowcharts illustrating the behavior recognition method provided by the present invention, such as... Figure 1 As shown, this method can be applied to shoe changing and disinfection detection for farm workers, as well as to the recognition of other human actions and behaviors, such as medical personnel wearing gloves and undergoing disinfection detection. For ease of explanation, the following embodiment uses shoe changing and disinfection detection for farm workers as an example. The method includes the following steps:

[0048] Step 110: Perform key point detection on the image to be identified to obtain the location information of key points of people in the image to be identified.

[0049] Here, the image to be identified refers to an image containing human actions, used to determine whether the human action is the target action. For example, the image to be identified could be an image of a worker entering a farm, used to determine whether the worker completed the target action (such as changing shoes or disinfecting) upon entering the farm. Key points refer to points used to characterize human posture information. For full-body movements, key points may include the left eye, right eye, left shoulder, right shoulder, left wrist, right wrist, left elbow, right elbow, left ankle, right ankle, left hip, right hip, left knee, right knee, left ear, right ear, and nose. For partial movements, such as upper-body movements, key points may include the left shoulder, right shoulder, left wrist, right wrist, left elbow, and right elbow. The location information of the human key points characterizes the position of the key points, which can be the coordinates of the key points.

[0050] For example, when changing shoes, a person's hands need to touch their feet, but when disinfecting, their hands usually don't touch their feet. In this case, the hands and feet can be used as key points, and key point detection can be performed to obtain the hand coordinates, foot coordinates, and other key point location information of the person.

[0051] Understandably, the image to be identified can be acquired through a camera device, which can be installed above the shoe cabinet to capture a frontal image of the staff. For example, when a pedestrian walks directly in front of the shoe cabinet, the camera device above the cabinet can use facial recognition to determine whether the pedestrian is a staff member. If so, the camera captures the pedestrian's image as the image to be identified; otherwise, an alarm can be triggered.

[0052] Step 120: Determine personnel posture information based on key personnel location information.

[0053] Specifically, personnel key point location information is used to characterize the position of key points, while personnel posture information is used to characterize the positional relationship information between key points corresponding to a certain action of the personnel. Based on the positions of key points carried in the personnel key point location information, the positional relationship between key points can be determined, thereby obtaining personnel posture information.

[0054] For example, based on the shoulder and hip coordinates in the personnel key point location information, the angle between the straight line formed by the shoulder center and hip center in the personnel posture information and the plumb line can be determined; based on the wrist and ankle coordinates in the personnel key point location information, the distance between the wrist and ankle in the personnel posture information can be determined.

[0055] Step 130: Determine the behavior recognition result based on the person's posture information.

[0056] Specifically, since different behaviors correspond to different posture information, the behavior recognition result can be determined after determining the posture information. For example, when the angle between the straight line formed by the center of the shoulder and the center of the hip in the posture information and the plumb line is greater than 0°, it indicates that the person is in a tilted position. When performing a disinfection action, the person will be in a tilted position. Therefore, the probability of the disinfection action in the image to be recognized is high, that is, the behavior recognition result can be that the disinfection action exists in the image to be recognized. As another example, since the hand needs to touch the foot when changing shoes, the distance between the wrist and ankle is very small. Therefore, when the distance between the wrist and ankle in the posture information is close to 0, the probability of the shoe-changing action in the image to be recognized is high, that is, the behavior recognition result can be that the shoe-changing action exists in the image to be recognized.

[0057] Compared to traditional methods that rely on staff to consciously change shoes and disinfect, this invention determines behavior recognition results based on the image to be identified. This avoids hygiene and safety hazards caused by staff negligence leading to missed shoe-changing and disinfection actions, or by staff taking chances and failing to perform these actions. Compared to traditional methods that manually inspect staff images to confirm whether shoe-changing and disinfection actions have been completed, this invention can automatically and accurately determine behavior recognition results based on staff posture information in the image to be identified, resulting in higher efficiency. Compared to traditional methods that input staff images obtained from video surveillance into the network for behavior analysis to confirm whether staff have changed shoes and disinfected, this invention performs key point detection on the image to be identified to obtain staff posture information and determines behavior recognition results based on this information, avoiding the impact of image noise on the accuracy of behavior recognition results.

[0058] The behavior recognition method provided in this invention performs key point detection on the image to be recognized to obtain personnel posture information, thereby enabling rapid and accurate determination of behavior recognition results based on the personnel posture information. Furthermore, since the personnel posture information is obtained after key point detection on the image to be recognized, it contains feature information of the key points, thus enabling accurate determination of behavior recognition results based on the personnel posture information. This avoids the problem in traditional methods where noise in the image to be recognized prevents accurate acquisition of behavior recognition results.

[0059] Based on the above embodiments, step 120 includes:

[0060] Based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person, and the aspect ratio of the person obtained by target detection in the image to be recognized, or based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person, determine the person's posture information.

[0061] Specifically, when changing shoes, the hands usually need to touch the feet, at which point the distance between the wrist and ankle is very small. When disinfecting, the hands typically do not touch the feet, and the distance between the wrist and ankle is greater than during disinfection. Therefore, the distance between the wrist and ankle can be used to distinguish between shoe-changing and disinfection actions, and thus wrist-ankle coordinates can serve as an indicator for this distinction. Optionally, based on the wrist-ankle coordinates in the personnel's key point location information, the distance between the wrist and ankle can be determined and used as personnel posture information for behavior recognition.

[0062] During disinfection, a person is typically in a tilted position, where the angle between the line formed by the center of the shoulder and the center of the hip and the plumb line is greater than 0°. During shoe changing, a person is typically not tilted, and the angle between the line formed by the center of the shoulder and the center of the hip and the plumb line is 0°. Therefore, the angle between the line formed by the center of the shoulder and the center of the hip and the plumb line can be used to distinguish between shoe changing and disinfection, thus the shoulder-hip coordinates can serve as an indicator for differentiating between these actions. Optionally, based on the shoulder-hip coordinates in the person's key point location information, the angle between the line formed by the center of the shoulder and the center of the hip and the plumb line can be determined, i.e., the person's tilt angle, and this can be used as the person's posture information for behavior recognition.

[0063] Since both wrist-ankle coordinates and shoulder-hip coordinates can be used to distinguish between shoe-changing and disinfection actions, the distance between the wrist and ankle in a person's posture information can be determined based on the wrist-ankle coordinates, and the tilt angle of the person can be determined based on the shoulder-hip coordinates. Furthermore, the behavior recognition result can be determined based on the distance between the wrist and ankle and the tilt angle. For example, when the distance between a person's wrist and ankle in the image to be recognized is close to 0, it indicates that a shoe-changing action may exist in the image. If the tilt angle of the person in the image is 0°, it further proves that the probability of a shoe-changing action in the image is high.

[0064] Furthermore, when changing shoes, people typically sit, squat, or bend over; similarly, they bend over when disinfecting. When people are sitting, squatting, or bending over, their aspect ratio is significantly smaller than when they are standing normally, and this aspect ratio varies depending on the posture. For example, the height in a squatting position is lower than in a sitting position, resulting in a smaller aspect ratio for the squatting position compared to the sitting position. Therefore, based on the wrist-ankle coordinates and / or shoulder-hip coordinates in the key point location information of the person, and determining the distance between the wrist and ankle and / or the person's tilt angle, target detection can be performed on the image to obtain the person's aspect ratio. In other words, based on the wrist-ankle coordinates and / or shoulder-hip coordinates, and the aspect ratio, the person's posture information can be determined. For example, based on wrist and ankle coordinates and / or shoulder and hip coordinates, it can be determined that there may be a shoe-changing action in the image to be identified. If the person is determined to be in a sitting position based on the aspect ratio of the person, it further proves that the probability of shoe-changing action in the image to be identified is high.

[0065] Optionally, the aspect ratio of the person can be determined based on the target detection bounding box obtained after target detection, and the specific calculation formula is as follows: Where R represents the aspect ratio of the person, d w d represents the width of the object detection bounding box. hThis indicates the height of the target detection box.

[0066] The method provided in this invention combines the location information of key points of a person with the aspect ratio of the person obtained from target detection to determine the person's posture information, thereby enabling more accurate acquisition of behavior recognition results based on the person's posture information.

[0067] Based on any of the above embodiments, in step 120, determining the person's posture information based on the wrist and ankle coordinates in the key point location information includes:

[0068] The wrist-ankle position relationship in the person's posture information is determined based on at least one of the distances between the left wrist coordinate and the left ankle coordinate and the right ankle coordinate, and the distances between the right wrist coordinate and the left ankle coordinate and the right ankle coordinate.

[0069] Specifically, when changing shoes, the hands usually need to touch the feet. At this time, the distance between the hands and feet is very small; that is, the distance between the left and right wrists and ankles is very small. For example, when changing shoes on the left foot, the distance between the left and right wrists and the left ankle is very small; when changing shoes on the right foot, the distance between the left and right wrists and the right ankle is very small. Therefore, when the distance between both the left and right wrists and the left ankle is very small, or the distance between both the left and right wrists and the right ankle is very small, it indicates a higher probability that a shoe-changing action exists in the image to be identified.

[0070] During disinfection, the hands typically do not touch the feet, and the disinfection action is usually performed with one hand. In this case, the distance between the hand and the foot is very small; specifically, the distance between the left or right wrist and the ankle is small. For example, when a person uses their left hand to disinfect their left foot, the distance between their left wrist and left ankle is smaller than the distance between their left wrist and right ankle; similarly, when a person uses their left hand to disinfect their right foot, the distance between their left wrist and right ankle is smaller than the distance between their left wrist and left ankle. Therefore, when the distance between the left wrist and the left and right ankles is different, or the distance between the right wrist and the left and right ankles is different, it indicates a higher probability that a disinfection action is present in the image to be identified.

[0071] Therefore, the distances between the left wrist and left ankle, the left wrist and right ankle, the right wrist and left ankle, and the right wrist and right ankle can be used to distinguish between shoe-changing and disinfection actions. Thus, this embodiment of the invention determines the distance between the left wrist and left ankle (first wrist-ankle distance d1) based on the coordinates of the left wrist and left ankle; determines the distance between the right wrist and left ankle (second wrist-ankle distance d2) based on the coordinates of the right wrist and left ankle; determines the distance between the left wrist and right ankle (third wrist-ankle distance d3) based on the coordinates of the left wrist and right ankle; and determines the distance between the right wrist and right ankle (fourth wrist-ankle distance d4) based on the coordinates of the right wrist and right ankle. The first wrist-ankle distance d1, the second wrist-ankle distance d2, the third wrist-ankle distance d3, and the fourth wrist-ankle distance d4 are used as the wrist-ankle position relationships for personnel posture information.

[0072] Based on any of the above embodiments Figure 2 This is a flowchart illustrating an implementation of step 120 in the behavior recognition method provided by the present invention, as shown below. Figure 2 As shown, based on the shoulder and hip coordinates in the personnel key point location information, the personnel posture information is determined, including:

[0073] Step 121: Determine the coordinates of the shoulder center point based on the coordinates of the left and right shoulders in the shoulder-hip coordinate system;

[0074] Step 122: Determine the coordinates of the center point of the hip based on the coordinates of the left and right hips in the shoulder-hip coordinate system;

[0075] Step 123: Based on the coordinates of the shoulder center point and the hip center point, determine the tilt angle of the person in the posture information.

[0076] Specifically, the personnel tilt angle refers to the angle between the straight line formed by the center point of the shoulder and the center point of the hip and the vertical line. During disinfection, the person is usually in a tilted position, with a tilt angle greater than 0°. During shoe changing, the person is usually not tilted, with a tilt angle of 0°. Therefore, the personnel tilt angle can be used to distinguish between shoe changing and disinfection.

[0077] Based on the coordinates of the left and right shoulders, the coordinates of the shoulder center point (x, y) can be determined. P y P Based on the coordinates of the left and right hips, the coordinates of the center point of the hip (x) can be determined. Q y Q Therefore, the tilt angle θ of the person can be calculated based on the following formula:

[0078]

[0079] The tilt angle of the person obtained can be used to determine whether the person is tilted. If the tilt angle is greater than 0°, it indicates that the person is tilted, meaning there is a high probability that a disinfection action exists in the image to be identified. If the tilt angle is 0°, it indicates that the person is not tilted, meaning there is a low probability that a disinfection action exists in the image to be identified. Therefore, based on the tilt angle of the person, it is possible to further determine whether a disinfection action exists in the image to be identified, thus ensuring the accuracy of the behavior recognition results.

[0080] Based on any of the above embodiments, step 130 includes:

[0081] The behavior recognition result is determined based on the person's posture information and the image features of the image to be recognized.

[0082] Specifically, since different behaviors correspond to different posture information, the image features of the image to be identified contain the feature information of the human behavior. Therefore, the human posture information and the image features of the image to be identified can be used as parameters for behavior recognition.

[0083] Optionally, the person's posture information and the image features of the image to be recognized can be input into the behavior recognition model. The behavior recognition model then fuses the person's posture information and the image features of the image to be recognized to obtain a fused feature vector, and determines the behavior recognition result based on the fused feature vector. The person's posture information may include the abscissa vector K of each key point. x , ordinate vector K y The aspect ratio R of the person, the distances between the first and fourth wrists / ankles d1, d2, d3, d4, and the tilt angle θ of the person are used to fuse the person's posture information with the image features n of the image to be identified. The resulting fused feature vector X can be expressed as: X=(n,R,d1,d2,d3,d4,θ,K) x ,K y ).

[0084] Before inputting the person's pose information and the image features of the image to be recognized into the behavior recognition model, the behavior recognition model can be pre-trained. This can be achieved by performing the following steps: First, collect a large number of sample images and perform keypoint detection on each sample image to determine the pose information of the people in the sample. Then, manually label the sample images to determine the corresponding behavior types. Subsequently, train the initial model based on the sample images, the pose information of the people in the sample, and their corresponding behavior types to obtain the behavior recognition model. The initial model can be a YOLOv5s model with low memory overhead. Since the COCO pre-trained model of the YOLOv5s model already contains a large amount of person behavior data, the model can be fine-tuned by freezing the first 9 layers of the basic network.

[0085] Meanwhile, to address situations where a single training category is prone to false detections, the sample images can also include background images without people and easily identifiable error images. When performing keypoint detection on the sample images, the open-source keypoint detection model CenterNet can be used to detect keypoints and obtain their location information. To enable faster model convergence, the keypoint location information can be normalized. For example, the keypoint coordinates used to represent their location can be divided by a preset length and width to achieve proportional normalization.

[0086] It should be noted that when selecting sample images, multiple consecutive frames with the same action can be used as sample images, and the corresponding action type can be labeled. For example, if 10 consecutive frames form a shoe-changing action, then these 10 consecutive frames can be used as sample images, and their action type can be labeled as shoe-changing action.

[0087] Understandably, when performing behavioral recognition on farm workers, the primary goal is to identify whether they have changed shoes and disinfected. Therefore, when labeling sample images with behavioral types, these types can include changing shoes, disinfecting, and other actions. "Other actions" refers to actions other than changing shoes and disinfecting.

[0088] Since the person's posture information is obtained by key point detection of the image to be recognized, the person's posture information carries the feature information of the key points, while the image features of the image to be recognized contain the feature information of the image itself. Therefore, by combining the person's posture information and the image features of the image to be recognized, the behavior recognition result can be further accurately determined.

[0089] Based on any of the above embodiments Figure 3 This is one of the flowcharts illustrating an implementation of step 130 in the behavior recognition method provided by the present invention, as shown below. Figure 3 As shown, step 130 includes:

[0090] Step 131a: Based on the person posture information of multiple consecutive frames of images to be identified, determine the behavior type of each image to be identified in the multiple consecutive frames of images to be identified;

[0091] Step 131b: If the number of images to be identified with the same behavior type is greater than the preset number, then the behavior type of the images to be identified with the same behavior type is taken as the behavior recognition result.

[0092] Specifically, based on the posture information of people in multiple consecutive frames of images to be identified, the behavior type corresponding to each image to be identified can be determined. If the number of images to be identified with the same behavior type is greater than a preset number, it indicates that the probability that the behavior type of multiple consecutive frames of images to be identified is the same as the behavior type of the images to be identified is greater. In other words, the behavior type of the images to be identified with the same behavior type can be used as the behavior recognition result.

[0093] For example, if more than 6 out of 10 consecutive frames of images to be identified have the same behavior type - changing shoes, then the shoe-changing action can be taken as the final behavior recognition result.

[0094] Based on any of the above embodiments Figure 4 This is a second schematic flowchart illustrating an implementation of step 130 in the behavior recognition method provided by the present invention, as shown below. Figure 4 As shown, step 130 includes:

[0095] Step 131b: Determine the initial behavior type based on personnel posture information;

[0096] Step 132b: If the initial behavior type is a target action, the initial behavior type is verified based on the location information of the target object corresponding to the target action in the image to be identified, as well as the location information of key points of the personnel, to obtain the behavior recognition result.

[0097] Specifically, the target object refers to the item used by a person when performing a target action. A person's target action is related not only to their posture information but also to the target object corresponding to that action. If a target action exists in the image to be recognized, the target object corresponding to that action should be detectable in the image, and the distance between the target object and the person's key points should be relatively short. Therefore, based on the location information of the target object and the location information of the person's key points, the relative positional relationship between the target object and the person's key points can be determined, such as the distance between them. If the relative positional relationship between the two meets the requirements, it indicates a high probability that a target action exists in the image to be recognized, and thus the target action can be used as the behavior recognition result.

[0098] Based on any of the above embodiments, step 132b, which verifies the initial behavior type based on the location information of the target object corresponding to the target action in the image to be identified, and the location information of key points of the personnel, to obtain the behavior recognition result, includes:

[0099] Based on the location information of the target item and the wrist coordinates in the key point location information of the person, the distance between the target item and the wrist is calculated, and the target action is taken as the behavior recognition result when the distance between the target item and the wrist is less than a threshold.

[0100] Specifically, when a target object corresponding to a target action is detected in the image to be identified, the distance between the target object and the wrist is calculated, and the target action is taken as the behavior recognition result when the distance is less than a threshold.

[0101] For example, if the initial behavior type is a disinfection action, it is simultaneously detected whether there is a target item corresponding to the disinfection action - disinfectant in the image to be identified. If so, the distance D between the disinfectant and the wrist is calculated, and the distance D is compared with a threshold. If the distance D is less than the threshold, it indicates that there is a high probability that there is a disinfection action in the image to be identified. Therefore, the disinfection action can be used as the final behavior recognition result.

[0102] Based on any of the above embodiments, the present invention also provides a behavior recognition method. Figure 5 This is the second flowchart of the behavior recognition method provided by the present invention, as shown below. Figure 5 As shown, the method includes:

[0103] Multiple consecutive frames of images to be identified are acquired, and personnel detection is performed on each image. The aspect ratio of the personnel is determined based on the detected bounding boxes. Simultaneously, keypoint detection is performed on each image to obtain the location information of these keypoints, and the personnel pose information is determined based on this information. The pose information includes the x-coordinate and y-coordinate vectors of each keypoint, the aspect ratio, the distances between the left wrist and left ankle, right wrist and left ankle, left wrist and right ankle, right wrist and right ankle, and the tilt angle. The pose information and the features from each frame of the images to be identified are fused to obtain a fused feature vector.

[0104] Based on the fused feature vector, the behavior type corresponding to each image to be identified is obtained. If the number of images with the same behavior type is greater than the preset number, the behavior type of the images to be identified with the same behavior type is taken as the behavior identification result.

[0105] If the behavior type of the image to be identified is a disinfection action based on the fusion feature vector, then disinfectant detection is performed on the image to be identified to determine whether disinfectant exists in the image. If so, the distance D between the disinfectant and the wrist is calculated. If the distance D is less than the threshold, then the disinfection action is taken as the behavior recognition result.

[0106] The behavior recognition device provided by the present invention is described below. The behavior recognition device described below and the behavior recognition method described above can be referred to in correspondence.

[0107] Based on any of the above embodiments, the present invention also provides a behavior recognition device. Figure 6 This is a schematic diagram of the behavior recognition device provided by the present invention, as shown below. Figure 6As shown, the device includes:

[0108] The detection unit 610 is used to perform key point detection on the image to be identified, and obtain the location information of key points of people in the image to be identified;

[0109] The determining unit 620 is used to determine the personnel posture information based on the personnel key point position information;

[0110] The recognition unit 630 is used to determine the behavior recognition result based on the person's posture information.

[0111] Based on any of the above embodiments, the determining unit 620 is used for:

[0112] The person's posture information is determined based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person, and the aspect ratio of the person obtained by target detection of the image to be identified, or based on the wrist and ankle coordinates and / or shoulder and hip coordinates in the key point location information of the person.

[0113] Based on any of the above embodiments, the determining unit 620 is used for:

[0114] The wrist-ankle position relationship in the person's posture information is determined based on at least one of the distances between the left wrist coordinate and the left ankle coordinate and the right ankle coordinate, and the distances between the right wrist coordinate and the left ankle coordinate and the right ankle coordinate.

[0115] Based on any of the above embodiments, the determining unit 620 includes:

[0116] The shoulder center determination unit is used to determine the coordinates of the shoulder center point based on the left shoulder coordinates and the right shoulder coordinates in the shoulder-hip coordinate system.

[0117] The hip center determination unit is used to determine the coordinates of the hip center point based on the left and right hip coordinates in the shoulder and hip coordinates.

[0118] The tilt angle determination unit is used to determine the tilt angle of the person in the person's posture information based on the coordinates of the shoulder center point and the coordinates of the hip center point.

[0119] Based on any of the above embodiments, the identification unit 630 is used for:

[0120] The behavior recognition result is determined based on the person's posture information and the image features of the image to be recognized.

[0121] Based on any of the above embodiments, the identification unit 630 includes:

[0122] The behavior type determination unit is used to determine the behavior type of each image to be identified in the continuous multi-frame images to be identified based on the person's posture information.

[0123] The first recognition result determination unit is used to determine the behavior type of the images to be recognized if the number of images to be recognized with the same behavior type is greater than a preset number.

[0124] Based on any of the above embodiments, the identification unit 630 includes:

[0125] An initial behavior type determination unit is used to determine the initial behavior type based on the personnel posture information;

[0126] The second recognition result determination unit is used to, if the initial behavior type is a target action, verify the initial behavior type based on the location information of the target item corresponding to the target action in the image to be recognized and the location information of the key points of the person, and obtain the behavior recognition result.

[0127] Based on any of the above embodiments, the second identification result determining unit is configured to:

[0128] Based on the location information of the target item and the wrist coordinates in the key point location information of the person, the distance between the target item and the wrist is calculated, and when the distance between the target item and the wrist is less than a threshold, the target action is taken as the behavior recognition result.

[0129] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a memory 720, a communications interface 730, and a communications bus 740. The processor 710, memory 720, and communications interface 730 communicate with each other via the communications bus 740. The processor 710 can call logical instructions in the memory 720 to execute a behavior recognition method. This method includes: performing key point detection on an image to be recognized to obtain key point location information of a person in the image; determining person posture information based on the key point location information; and determining a behavior recognition result based on the person posture information.

[0130] Furthermore, the logical instructions in the aforementioned memory 720 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the behavior recognition method provided by the above methods, the method including: performing key point detection on the image to be recognized to obtain the location information of key points of a person in the image to be recognized; determining the person's posture information based on the person's key point location information; and determining the behavior recognition result based on the person's posture information.

[0132] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned behavior recognition methods, the method comprising: performing key point detection on an image to be recognized to obtain key point location information of a person in the image to be recognized; determining person posture information based on the key point location information of the person; and determining a behavior recognition result based on the person posture information.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A behavior recognition method, characterized in that, include: Key point detection is performed on the image to be identified to obtain the location information of key points of people in the image to be identified; Based on the personnel's key point location information, determine the personnel's posture information; The personnel posture information is used to characterize the positional relationship information between key points corresponding to a certain action of a person; Based on the person's posture information, the behavior recognition result is determined; The determination of personnel posture information based on the personnel key point location information includes: Based on the wrist and ankle coordinates in the personnel key point location information, the wrist and ankle positional relationship is determined; Based on the shoulder and hip coordinates in the personnel key point location information, the personnel tilt angle is determined; Based on the wrist-ankle position relationship and the person's tilt angle, the initial person posture information is determined; Based on the aspect ratio of the person, the initial person posture information is verified, and the person posture information is determined.

2. The behavior recognition method according to claim 1, characterized in that, The step of determining the wrist-ankle position relationship based on the wrist-ankle coordinates in the personnel key point location information includes: The wrist-ankle position relationship in the person's posture information is determined based on at least one of the distances between the left wrist coordinate and the left ankle coordinate and the right ankle coordinate, and the distances between the right wrist coordinate and the left ankle coordinate and the right ankle coordinate.

3. The behavior recognition method according to claim 1, characterized in that, Determining the tilt angle of a person based on the shoulder and hip coordinates in the key point location information of the person includes: Based on the coordinates of the left and right shoulders in the shoulder-hip coordinate system, determine the coordinates of the shoulder center point. Based on the left and right hip coordinates in the shoulder and hip coordinates, determine the coordinates of the center point of the hip. Based on the coordinates of the shoulder center point and the coordinates of the hip center point, the tilt angle of the person in the posture information is determined.

4. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The determination of behavior recognition results based on the person's posture information includes: Based on the human posture information of multiple consecutive frames of images to be identified, the behavior type of each image to be identified in the multiple consecutive frames of images to be identified is determined; If the number of images to be identified with the same behavior type is greater than a preset number, then the behavior type of the images to be identified with the same behavior type is taken as the behavior recognition result.

5. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The determination of behavior recognition results based on the person's posture information includes: Based on the personnel posture information, the initial behavior type is determined; If the initial behavior type is a target action, then the initial behavior type is verified based on the location information of the target object corresponding to the target action in the image to be identified, and the location information of the key points of the person, to obtain the behavior recognition result.

6. The behavior recognition method according to claim 5, characterized in that, The initial behavior type is verified based on the location information of the target object corresponding to the target action in the image to be identified, and the location information of the key points of the person, to obtain the behavior recognition result, including: Based on the location information of the target item and the wrist coordinates in the key point location information of the person, the distance between the target item and the wrist is calculated, and when the distance between the target item and the wrist is less than a threshold, the target action is taken as the behavior recognition result.

7. The behavior recognition method according to any one of claims 1 to 3, characterized in that, The determination of behavior recognition results based on the person's posture information includes: The behavior recognition result is determined based on the person's posture information and the image features of the image to be recognized.

8. A behavior recognition device, characterized in that, include: The detection unit is used to detect key points in the image to be identified, and obtain the location information of key points of people in the image to be identified; The determining unit is used to determine the personnel posture information based on the personnel key point position information; The personnel posture information is used to characterize the positional relationship information between key points corresponding to a certain action of a person; The recognition unit is used to determine the behavior recognition result based on the person's posture information; The determination of personnel posture information based on the personnel key point location information includes: Based on the wrist and ankle coordinates in the personnel key point location information, the wrist and ankle positional relationship is determined; Based on the shoulder and hip coordinates in the personnel key point location information, the personnel tilt angle is determined; Based on the wrist-ankle position relationship and the person's tilt angle, the initial person posture information is determined; Based on the aspect ratio of the person, the initial person posture information is verified, and the person posture information is determined.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the behavior recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the behavior recognition method as described in any one of claims 1 to 7.

Images