Abnormal behavior recognition method and device, computer device and storage medium

Abstract

This application relates to an abnormal behavior recognition method, apparatus, computer device, and storage medium. The method includes: performing behavior recognition on an image sequence containing multiple objects; determining a pose type queue and a limb behavior queue for each object; and determining the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue. If the number of entanglements between the target object and other objects reaches a preset number, then determining an initial abnormal behavior recognition result for the target object based on the image sequence to be recognized and a trained abnormal behavior recognition model; and determining a target abnormal behavior recognition result for the target object based on the pose change frequency in the pose type queue, the abnormal behavior frequency in the limb behavior queue, and the initial abnormal behavior recognition result. This method can improve the accuracy of abnormal behavior recognition.

Description

Technical Field

[0001] This application relates to the field of behavior recognition technology, and in particular to an abnormal behavior recognition method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] To maintain social order, servers can analyze and process video images using technologies such as computer vision and deep learning to monitor for abnormal behavior between people. Once abnormal behavior is detected, such as intense physical contact, punching, or kicking, the server can promptly send an alert to staff.

[0003] Currently, technologies for identifying abnormal behavior can determine its presence by analyzing human movements in a single frame of an image or by using the coordinates of multiple key points on a person. However, single-frame images contain limited information and lack temporal correlation, making it difficult to accurately detect abnormal behavior. Furthermore, when using key point coordinates, the coordinates of key points overlap when people are overlapping and struggling, affecting the localization accuracy and resulting in insufficient accuracy in abnormal behavior identification. Summary of the Invention

[0004] Therefore, it is necessary to provide an abnormal behavior identification method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of identification in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for identifying abnormal behavior. The method includes:

[0006] Behavior recognition is performed on an image sequence containing multiple objects to be identified, and a pose type queue and a limb behavior queue for each object are determined. Based on the pose type queue and limb behavior queue of each object, the number of entanglements between the target object and other objects is determined.

[0007] If the number of times the target object is entangled with other objects reaches a preset number, then based on the image sequence to be identified and the trained abnormal behavior recognition model, the initial abnormal recognition result of the target object is determined;

[0008] Based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial anomaly identification result, the target anomaly identification result corresponding to the target object is determined.

[0009] In one embodiment, the step of performing behavior recognition on a sequence of images to be identified containing multiple objects, and determining a pose type queue and a limb behavior queue for each object, includes:

[0010] Multiple objects contained in each image frame of the image sequence to be identified are identified to obtain the position information and skeletal key point information of each object;

[0011] For each of the objects, the image frames containing the current object are identified according to the trained pose recognition model to obtain the pose category of the current object in each of the image frames and determine it as a pose type queue.

[0012] Based on the skeletal key point information of the current object in each of the image frames, the limb behavior of the current object in each of the image frames is determined and identified as a limb behavior queue.

[0013] In one embodiment, the skeletal keypoint information includes information on a first keypoint, a second keypoint, a third keypoint, a fourth keypoint, a fifth keypoint, and a sixth keypoint; determining the limb behavior of the current object in each of the image frames based on the skeletal keypoint information of the current object in each of the image frames includes:

[0014] For each image frame, based on the skeletal key point information, a first angle between the first line segment and the second line segment of the current object, and first height information corresponding to the first key point and the second key point are determined respectively; wherein, the first line segment is a straight line segment between the first key point and the second key point; the second line segment is a straight line segment between the third key point and the fourth key point of the current object;

[0015] If the first included angle and / or the first height information meets the first preset condition, then the limb behavior of the current object in the current image frame is determined to be the first abnormal behavior;

[0016] Based on the skeletal key point information, the second included angle between the third line segment and the second line segment of the current object is determined, as well as the second height information corresponding to the fifth key point and the sixth key point respectively; wherein, the third line segment is the straight line segment between the fifth key point and the sixth key point;

[0017] If the second included angle and / or the second height information meet the second preset conditions, then the limb behavior of the current object in the current image frame is determined to be the second abnormal behavior.

[0018] In one embodiment, determining the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue of each of the objects includes:

[0019] For each image frame in the image sequence to be identified, the overlapping area between the detection bounding box of the target object and the detection bounding boxes of other objects is determined, as well as the facial orientation between the target object and the other objects is determined; the facial orientation is determined based on the skeletal keypoint information of the target object and the other objects; the detection bounding box is determined based on the positional information of each object in the image frame.

[0020] If it is determined that the overlapping area is greater than a preset area value, and it is determined that the facial orientation between the target object and the other objects is face-to-face, and it is determined that at least one of the posture categories of the target object and the other objects is a preset posture category, and it is determined that at least one of the limb behaviors of the target object and the other objects is a preset limb behavior, then it is determined that the target object and the other objects are entangled, and the number of entanglements is updated.

[0021] In one embodiment, determining the initial anomaly recognition result of the target object based on the image sequence to be recognized and the trained anomaly behavior recognition model includes:

[0022] Obtain the current image frame corresponding to the image sequence to be identified. The current image frame is the image frame corresponding to the time when the number of times the target object is entangled with other objects reaches a preset number.

[0023] Based on the detection bounding box of the target object and the detection bounding boxes of other objects, determine the bounding rectangle between the target object and the other objects;

[0024] Based on the circumscribed rectangle, the current image frame is cropped to obtain the target interest region corresponding to the target object;

[0025] The target region of interest is input into the trained abnormal behavior recognition model to determine the initial abnormal recognition result of the target object.

[0026] In one embodiment, the method further includes:

[0027] If the pose recognition model identifies that an image frame in the image sequence to be recognized contains a new object, then a data structure for the new object is created; the data structure is used to store the position information and skeletal key point information of the new object.

[0028] In one embodiment, determining the target anomaly identification result corresponding to the target object based on the frequency of posture changes in the posture type queue, the frequency of abnormal behaviors in the limb behavior queue, and the initial anomaly identification result includes:

[0029] The pose change frequency corresponding to the pose type queue is determined based on the number of adjacent image frames with different pose categories in the pose type queue.

[0030] The frequency of abnormal behavior corresponding to the limb behavior queue is determined based on the number of image frames in the limb behavior queue that contain abnormal limb behaviors.

[0031] Based on preset weight values, the frequency of posture changes, the frequency of abnormal behavior, and the initial anomaly identification result of the target object are weighted and summed to obtain a comprehensive score for the target object, and the target anomaly identification result of the target object is determined based on the comprehensive score.

[0032] Secondly, this application also provides an abnormal behavior recognition device. The device includes:

[0033] The object queue determination module is used to perform behavior recognition on an image sequence containing multiple objects, determine the pose type queue and limb behavior queue of each object, and determine the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue of each object.

[0034] The first identification result determination module is used to determine the initial abnormal identification result of the target object based on the image sequence to be identified and the trained abnormal behavior identification model if the number of entanglements between the target object and other objects reaches a preset number.

[0035] The second identification result determination module is used to determine the target anomaly identification result corresponding to the target object based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial anomaly identification result.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect.

[0037] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0038] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0039] The aforementioned abnormal behavior recognition method, apparatus, computer equipment, storage medium, and computer program product determine the pose type queue and limb behavior queue of each object in the image sequence to be recognized, thereby obtaining the pose category and limb behavior of each object within the same time period. Both pose category and limb behavior can serve as relevant actions to determine whether an object exhibits abnormal behavior. Furthermore, the server determines the initial abnormality recognition result for the target object by identifying entanglement between the target object and other objects. When the server determines that the target object has experienced more than a preset number of entanglements with other objects, the server uses a trained abnormal behavior recognition model to determine the initial abnormality recognition result for the target object. Finally, the initial abnormality recognition result is further updated using the pose change frequency of the pose type queue and the abnormal behavior frequency of the limb behavior queue to obtain the final target abnormality recognition result for the target object. Both the pose type queue and the limb behavior queue contain continuous temporal information and information on a series of action changes occurring within that continuous time. Therefore, the target abnormality recognition result determined based on the pose type queue and the limb behavior queue can improve the recognition accuracy. Moreover, by determining the number of entanglements, the possibility of abnormal behavior in the target object can be initially determined. Based on this, the server can further improve the accuracy of abnormal behavior recognition by performing abnormal behavior recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a diagram illustrating the application environment of an abnormal behavior recognition method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating an abnormal behavior identification method in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the process of determining the posture type queue and the limb behavior queue in one embodiment;

[0044] Figure 4 This is a flowchart illustrating the process of determining limb behavior in one embodiment;

[0045] Figure 5 This is a flowchart illustrating the process of determining the number of entanglements in one embodiment;

[0046] Figure 6 This is a flowchart illustrating the process of determining the initial anomaly identification result in one embodiment;

[0047] Figure 7 This is a flowchart illustrating the process of determining the target anomaly identification result in one embodiment;

[0048] Figure 8 This is a flowchart illustrating an abnormal behavior identification method in another embodiment;

[0049] Figure 9 This is a schematic diagram illustrating abnormal behavior in one embodiment;

[0050] Figure 10 This is a schematic diagram of a posture type queue and a limb behavior queue in one embodiment;

[0051] Figure 11 This is a structural block diagram of an abnormal behavior recognition device in one embodiment;

[0052] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The abnormal behavior identification method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network, and can send the acquired image sequence to be recognized to server 104. The data storage system can store the image sequence to be recognized received by server 104 and the training data used to train the abnormal behavior recognition model. The data storage system can be integrated on server 104 or placed in the cloud or on another network server. Server 104 can identify objects in the image sequence to be recognized, determining the corresponding pose type queue and limb behavior queue for each object. Based on this, server 104 determines the number of entanglements between the target object and other objects through the pose type queue and limb behavior queue. When the number of entanglements reaches a preset number, server 104 can identify the target object through the abnormal behavior recognition model, obtaining the initial abnormal recognition result of the target object. Finally, based on the pose change frequency in the pose type queue and the abnormal behavior frequency in the limb behavior queue, the initial abnormal recognition result is further adjusted to obtain the final target abnormal recognition result.

[0055] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices, and may have video recording capabilities. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc., capable of generating image sequences. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] In one exemplary embodiment, such as Figure 2 As shown, an abnormal behavior identification method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S206. Wherein:

[0057] Step S202: Perform behavior recognition on the image sequence to be identified, which contains multiple objects, determine the pose type queue and limb behavior queue of each object, and determine the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue of each object.

[0058] The image sequence to be identified comprises multiple image frames, and each image frame may contain multiple objects. A pose type queue represents the pose type of an object within a segment of image frames; a limb behavior queue represents the limb behavior of an object within a segment of image frames. Entanglement indicates that two objects overlap, are face-to-face, and exhibit unusual interaction; each instance of overlap and unusual interaction between two objects adds one entanglement count.

[0059] In one example, posture categories can include at least normal upright, leaning forward or backward, swaying left or right, lying down, sitting, and kneeling; limb behaviors can include normal limb behaviors and abnormal limb behaviors, and abnormal limb behaviors can include at least abnormal punching and abnormal kicking.

[0060] Specifically, the server acquires a sequence of images to be recognized, containing multiple image frames, through the terminal. The server can identify multiple objects contained in the image sequence using a preset recognition algorithm or a preset recognition model. For each object, the server can identify the pose type and limb behavior of the current object in each image frame. After completing the recognition of the image sequence, the server can obtain a pose type queue and a limb behavior queue for each object.

[0061] In one example, the pose type queue can contain pose types corresponding to multiple image frames; the limb behavior queue can contain limb behaviors corresponding to multiple image frames. For example, the pose type queue of an object consists of the pose types of four image frames (normal upright, normal upright, normal upright, and lying down) in chronological order. The limb behaviors of an object can consist of the limb behaviors of four image frames (normal limb behavior, normal limb behavior, abnormal limb behavior, and abnormal limb behavior) in chronological order.

[0062] Furthermore, the server can determine whether objects overlap based on their location information; it can also determine whether there are changes in posture or abnormalities in limb behavior based on the object's pose type queue and limb behavior queue. If the server determines that multiple objects overlap, and their postures change or their limb behaviors become abnormal within a certain period before or after the overlap, then entanglement is confirmed, and the number of entanglements is recorded. Repeating this process yields the number of times the target object is entangled with other objects.

[0063] In one example, if the image sequence to be identified contains more than two objects, there may be a situation where one target object and multiple other objects are entangled. The number of entanglements of the target object needs to be determined independently. That is, the number of entanglements can be the number of entanglements between the target object and multiple other objects respectively. For example, there is an independent number of entanglements between target object A and object B, and there is an independent number of entanglements between target object A and object C. The above two types of entanglements cannot be combined as the number of entanglements of target object A.

[0064] Step S204: If the number of times the target object is entangled with other objects reaches a preset number, then the initial anomaly recognition result of the target object is determined based on the image sequence to be recognized and the trained abnormal behavior recognition model.

[0065] The abnormal behavior recognition model is a pre-trained behavior recognition model used to determine whether a target object exhibits abnormal behavior. This model can be a deep learning-based image recognition model. The anomaly recognition result can be represented as the confidence level of the target object's abnormal behavior; a higher confidence level indicates a greater likelihood that the target object is exhibiting abnormal behavior.

[0066] Specifically, the server determines the number of times the target object is entangled with other objects. If the number of entanglements reaches a preset number, the image sequence containing the target object is input into a trained abnormal behavior recognition model. The abnormal behavior recognition model determines the confidence level of the target object's abnormal behavior, which is then used as the initial abnormality recognition result for the target object. The image sequence containing the target object can be the image frame from the first time the target object becomes entangled with other objects to the image frame from the last time the target object becomes entangled with other objects.

[0067] Step S206: Based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial abnormality identification result, determine the target abnormality identification result corresponding to the target object.

[0068] Among them, the frequency of posture changes is determined based on the number of times the posture category changes in the posture type queue; the frequency of abnormal behavior is determined based on the number of times the target object's limb behavior abnormal behavior occurs in the limb behavior queue.

[0069] Specifically, the server counts the frequency of pose changes in the target object's pose type queue and the frequency of abnormal behaviors in the target object's limb behavior queue. The server can determine the target object's first score based on the pose change frequency and a preset calculation strategy, and it can determine the target object's second score based on the abnormal behavior frequency and the preset calculation strategy. Based on this, the server merges the first score, the second score, and the initial anomaly identification result to obtain the target anomaly identification result for the target object.

[0070] In the aforementioned abnormal behavior recognition method, by determining the pose type queue and limb behavior queue of each object in the image sequence to be recognized, the pose category and limb behavior of each object within the same time period are obtained. Both pose category and limb behavior can serve as relevant actions to determine whether an object exhibits abnormal behavior. Furthermore, the server determines the initial abnormality recognition result for the target object by identifying entanglement between the target object and other objects. When the server determines that the target object has experienced more than a preset number of entanglements with other objects, the trained abnormal behavior recognition model determines the initial abnormality recognition result for the target object. Finally, the initial abnormality recognition result is further updated using the pose change frequency of the pose type queue and the abnormal behavior frequency of the limb behavior queue to obtain the final target abnormality recognition result for the target object. Both the pose type queue and the limb behavior queue contain continuous temporal information and a series of action changes occurring within that continuous time. Therefore, the target abnormality recognition result determined based on the pose type queue and the limb behavior queue can improve the recognition accuracy. Moreover, by determining the number of entanglements, the possibility of abnormal behavior in the target object can be initially determined. Based on this, the server can further improve the accuracy of abnormal behavior recognition by performing abnormal behavior recognition.

[0071] In one exemplary embodiment, such as Figure 3 As shown, the specific implementation process of the step "perform behavior recognition on the image sequence to be identified containing multiple objects, and determine the pose type queue and limb behavior queue of each object" includes steps S302 to S306. Wherein:

[0072] Step S302: Identify multiple objects contained in each image frame of the image sequence to be identified, and obtain the position information and skeletal key point information of each object.

[0073] The location information represents the position of the object within the image sequence to be identified, and can be represented using planar coordinates. The skeletal keypoint information represents the location of multiple skeletal keypoints of the object, which may include, but are not limited to, skeletal keypoints of the head, shoulders, elbows, hips, and feet.

[0074] Specifically, the server can identify multiple objects contained in the image sequence to be identified using a preset recognition algorithm or a preset recognition model, and identify multiple skeletal keypoints of each object. The server can first use a preset object recognition algorithm to determine the positional information of each object in each image frame. For each object, the server can input the image within the detection box of each object into a preset skeletal recognition algorithm to obtain the skeletal keypoint information of each object. Based on this, the server can obtain the positional information and skeletal keypoint information of each object in each frame of the image. The preset object recognition algorithm and the preset skeletal recognition algorithm can be pre-trained image recognition algorithms, and are not specifically limited here.

[0075] Step S304: For each object, the pose recognition model is used to identify each image frame containing the current object to obtain the pose category of the current object in each image frame and determine it as a pose type queue.

[0076] The pose recognition model is used to determine the pose category of objects contained in an image frame. The pose recognition model is trained based on pre-labeled first training data; the first training data includes skeletal keypoint information, position information, and pose category of each object. The trained pose recognition model can obtain the position information, skeletal keypoint information, and pose category of each object in the image frame.

[0077] Specifically, for each object, the server can determine multiple image frames to be identified in the image sequence based on a preset interval frame number. For example, if the preset interval frame number is 4 frames and the image sequence to be identified has 25 frames, then frames 1, 5, 10, 15, 20, and 25 can be identified as image frames to be identified.

[0078] The server inputs multiple image frames to be recognized into a pre-trained pose type recognition model to obtain the pose category of the current object in different image frames. The server stores the pose categories in a queue of preset size according to the chronological order of the image frames, forming a pose type queue. If the number of pose categories obtained from the image frames to be recognized exceeds the preset size of the pose type queue, the server can update the pose type queue by using a first-in, first-out (FIFO) data processing method, deleting the pose category stored at the head of the queue and then storing the newly determined pose category at the tail of the queue.

[0079] In one example, the server can use a pre-defined multi-object tracking algorithm to create a tracker for each object in the image frames to be identified, and use the trackers to identify multiple objects across multiple image frames. For the current object, the multi-object tracking algorithm can accurately track the current object across multiple image frames, that is, associate the same object in the previous image frame with the same object in the next image frame. This allows the server to accurately identify the current object when creating and updating the pose type queue and limb behavior queue based on the current object, avoiding the use of the pose categories and limb behaviors of other objects to update the pose type queue and limb behavior queue of the current object.

[0080] Step S306: Based on the skeletal key point information of the current object in each image frame, determine the limb behavior of the current object in each image frame and identify it as a limb behavior queue.

[0081] Specifically, for each object, the server can determine multiple image frames to be recognized in the image sequence according to a preset interval of frames. The server obtains the position information of multiple skeletal keypoints from the current image frame to be recognized, and determines the limb behavior of the current object in the current image frame based on the connection and positional relationships between the skeletal keypoints. For example, if the skeletal keypoints at both ends of the current object's arm are parallel to the ends of the body, it can be determined that the current object is raising its hand vertically above its head. The server can determine the current action of the current object based on the connection and positional relationships of multiple skeletal keypoints, and determine whether the current object's limb behavior is abnormal based on preset action conditions. If the current object's limb behavior is abnormal, the first representation value corresponding to the abnormal limb behavior is input into the limb behavior queue; if the current object's limb behavior is normal, the second representation value corresponding to the normal limb behavior is input into the limb behavior queue. The server stores the first and second representation values ​​of the limb behavior in a queue of a preset size according to the chronological order of the image frames, thus obtaining the limb behavior queue.

[0082] If the number of limb behaviors obtained from the image frame to be identified is greater than the preset size of the limb behavior queue, the limb behaviors stored at the head of the queue can be deleted using the queue's data processing method, i.e., the first-in-first-out rule, and the newly determined limb behaviors can be stored at the tail of the limb behavior queue, thereby completing the update of the limb behavior queue.

[0083] In this embodiment, the location information and skeletal keypoint information of each object are obtained by recognizing the image sequence to be identified. For each object, the server can determine the pose category of the current object in each image frame using a trained pose recognition model, thereby constructing a pose type queue. In addition, the server can determine the limb behavior of the current object in each image frame using the skeletal keypoint information of multiple skeletal points, thereby constructing a limb behavior queue. By determining the pose type queue and limb behavior queue of each object through each frame of static image, the changes in the body movements of the current object can be obtained, resulting in a more accurate pose type queue and limb behavior queue for each object, and thus obtaining a high-accuracy abnormal behavior recognition result.

[0084] In one exemplary embodiment, the skeletal keypoint information includes information on a first keypoint, a second keypoint, a third keypoint, a fourth keypoint, a fifth keypoint, and a sixth keypoint; such as Figure 4 As shown, the specific implementation process of the step "determine the limb behavior of the current object in each image frame based on the skeletal key point information of the current object in each image frame" includes steps S402 to S408. Wherein:

[0085] Step S402: For each image frame, based on the skeletal key point information, determine the first included angle between the first line segment and the second line segment of the current object, as well as the first height information corresponding to the first key point and the second key point respectively.

[0086] The first line segment is the straight line segment between the first and second keypoints; the second line segment is the straight line segment between the third and fourth keypoints of the current object. The first, second, third, fourth, fifth, and sixth keypoints are all skeletal keypoints of the current object.

[0087] Specifically, the server iterates through multiple image frames corresponding to the current object, determining multiple skeletal points of the current object in the current image frames, i.e., obtaining the first keypoint, second keypoint, third keypoint, fourth keypoint, fifth keypoint, and sixth keypoint. Based on the positional information of the multiple skeletal points, the server determines the first height information corresponding to the first and second keypoints, respectively; based on the line connecting the first and second keypoints, the server determines the first line segment of the current object. The first line segment can represent the extension direction and extension length of the current object's arm. Based on the positional information of the third and fourth keypoints, the server can determine the line connecting the third and fourth keypoints, obtaining the second line segment. The second line segment can represent the extension direction and extension length of the current object's body. In one example, the first keypoint could be an elbow keypoint, the second keypoint could be a shoulder keypoint, the third keypoint could be a head keypoint, and the fourth keypoint could be a hip keypoint.

[0088] Step S404: If the first included angle and / or the first height information meet the first preset condition, then determine that the limb behavior of the current object in the current image frame is the first abnormal behavior.

[0089] The first abnormal behavior could be abnormal punching behavior.

[0090] Specifically, the server can determine whether the current object exhibits abnormal punching behavior by using the first angle between the first line segment and the second line segment, as well as the height information of the first key point and the second key point. If the angle between the first line segment and the second line segment is greater than a preset angle value, it can be determined that the current object's limb behavior in the current image frame is abnormal punching behavior; or, if the height of the first key point is greater than the height of the second key point, it can be determined that the current object's limb behavior in the current image frame is abnormal punching behavior; or, if the angle between the first line segment and the second line segment is greater than a preset angle value, and the height of the first key point is greater than the height of the second key point, it can be determined that the current object's limb behavior in the current image frame is abnormal punching behavior.

[0091] Step S406: Based on the skeletal key point information, determine the second included angle between the third line segment and the second line segment of the current object, as well as the second height information corresponding to the fifth key point and the sixth key point respectively.

[0092] The third line segment is the straight line segment between the fifth and sixth key points.

[0093] Specifically, based on the positional information of multiple skeletal keypoints, the server determines the second height information corresponding to the fifth and sixth keypoints, respectively; based on the straight line connecting the fifth and sixth keypoints, the server determines the third line segment of the current object. The third line segment can represent the extension direction and extension length of the current object's legs. In one example, the fifth keypoint is the knee keypoint, and the sixth keypoint is the hip keypoint.

[0094] Step S408: If the second included angle and / or the second height information meet the second preset conditions, then determine that the limb behavior of the current object in the current image frame is the second abnormal behavior.

[0095] The second abnormal behavior could be abnormal foot lifting behavior.

[0096] Specifically, the server can determine whether the current object has abnormal foot lifting behavior by using the second angle between the third and second line segments, as well as the height information of the fifth and sixth key points. If the angle between the third and second line segments is greater than a preset angle value, it can be determined that the current object's limb behavior in the current image frame is abnormal foot lifting behavior; or, if the height of the fifth key point is greater than the height of the sixth key point, it can be determined that the current object's limb behavior in the current image frame is abnormal foot lifting behavior; or, if the angle between the third and second line segments is greater than a preset angle value, and the height of the fifth key point is greater than the height of the sixth key point, it can be determined that the current object's limb behavior in the current image frame is abnormal foot lifting behavior.

[0097] In this embodiment, by using three line segments composed of six key points and the height information corresponding to four key points, it is possible to determine whether the current object has abnormal punching behavior and abnormal foot lifting behavior in the current image frame. If abnormal punching behavior and abnormal foot lifting behavior occur, it is determined that the current object's limb behavior is abnormal. The server can accurately determine the current object's limb behavior based on the position information of the key points.

[0098] In one exemplary embodiment, such as Figure 5 As shown, the specific implementation process of the step "determining the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue of each object" includes steps S502 to S504. Wherein:

[0099] Step S502: For each image frame in the image sequence to be identified, determine the overlap area between the detection boxes of the target object and the detection boxes of other objects, and determine the facial orientation of the target object and other objects.

[0100] Among them, the facial orientation is determined based on the skeletal key point information of the target object and other objects; the detection box can be the position information of each object obtained after the pose recognition model identifies each object, and the detection box is determined based on the position information; the detection box can be determined based on the position information of each object in the image frame.

[0101] Specifically, the server iterates through each image frame in the image sequence to be recognized, determining the bounding boxes for multiple objects contained within each frame. Based on the location information of these bounding boxes, the server can determine the overlap area between the bounding boxes of the target object and those of other objects. The server can acquire the skeletal keypoint information of each object, filter it to obtain the location information of multiple facial keypoints, and determine the facial orientation of the object based on the location information of these facial keypoints. For example, the server can determine the facial orientation of each object based on the location information of the mouth and nose keypoints. Based on this, the facial orientation between two objects can be determined.

[0102] Step S504: If it is determined that the overlapping area is greater than the preset area value, and the facial orientation between the target object and other objects is determined to be face-to-face, and at least one of the posture categories of the target object and other objects is a preset posture category, and at least one of the limb behaviors of the target object and other objects is a preset limb behavior, then it is determined that the target object is entangled with other objects, and the entanglement count is updated.

[0103] Specifically, for each image frame in the image sequence to be identified, the server can determine the overlap area between the target object and other objects. When the overlap area is greater than a preset area value, the next step of judgment can be performed. The server can determine the facial orientation of the target object and the facial orientation of other objects. If the facial orientation of the target object is opposite to that of other objects, and the image in the direction of the target object's facial orientation contains other objects, then the facial orientation between the target object and other objects is determined to be face-to-face. When the facial orientation between the target object and other objects is face-to-face, the next step of judgment can be performed. The server can obtain the pose category of the target object and the pose categories of other objects in the current image frame, and when it determines that at least one of the pose categories of the target object and other objects is an abnormal pose category, the next step of judgment can be performed. The server can obtain the limb behavior of the target object and other objects in the current image frame, and when it determines that at least one of the limb behavior of the target object and other objects is an abnormal limb behavior, it can determine that there is an entanglement relationship between the target object and other objects, and update the entanglement count corresponding to the target object.

[0104] Based on this, the target object can be entangled with multiple other objects. Therefore, the one-to-many entanglement determination method can be broken down into a one-to-one entanglement determination method, thereby determining the number of times the target object is entangled with each of the other objects. In other words, the number of times the target object is entangled with multiple other objects needs to be determined and updated separately.

[0105] It should be understood that the order of the above judgment content can be customized, and the judgment method can be at least one of the above four judgment contents. This application embodiment does not make specific limitations.

[0106] In one example, the server can use a pre-defined multi-object tracking algorithm to create a tracker for each object in the image frames to be identified. The tracker then identifies each object associated with multiple image frames. Based on this multi-object tracking algorithm, the server can accurately distinguish the target object from other objects across multiple image frames, preventing the target object from becoming entangled with other objects and thus preventing the tracking of the correct object from continuing.

[0107] In this embodiment, whether the target object is entangled with other objects is determined by four aspects: the overlapping area of ​​the target object with other objects, the facial orientation of the target object with other objects, the posture category of the target object with other objects, and the limb behavior of the target object with other objects. This allows for the determination of the number of times the target object is entangled within a certain time range. By using multiple judgment conditions such as physical location, object posture, and object orientation to determine whether entanglement has occurred, the accuracy of determining the number of times entanglement has been improved.

[0108] In one exemplary embodiment, such as Figure 6 As shown, the specific implementation process of the step "determine the initial anomaly recognition result of the target object based on the image sequence to be recognized and the trained anomaly behavior recognition model" includes steps S602 to S608. Wherein:

[0109] Step S602: Obtain the current image frame corresponding to the image sequence to be identified.

[0110] The current image frame is the image frame that appears when the target object is entangled with other objects a preset number of times.

[0111] Specifically, when the server determines that the number of times the target object is entangled with other objects has reached a preset number, it determines the current image frame corresponding to the current moment in the image sequence to be identified.

[0112] Step S604: Based on the detection bounding box of the target object and the detection bounding boxes of other objects, determine the bounding rectangle between the target object and other objects.

[0113] The detection box can be the location information of each object obtained after the pose recognition model identifies each object, and the detection box is determined based on the location information. The circumscribed rectangle can be a rectangle that includes the detection box of the target object and the detection boxes of other objects.

[0114] Specifically, the server can determine the detection bounding box of the target object and the detection bounding boxes of other objects in the current image frame based on the detection bounding boxes of each object output by the pose recognition model. The server can then determine the bounding rectangle containing the detection bounding boxes of the target object and other objects, based on the principle of creating the minimum bounding rectangle.

[0115] Optionally, the server can use the target object's location information and the location information of all its skeletal keypoints as constraints for the outer matrix, creating a detection box that includes the target object's location information and the location information of all its skeletal keypoints. Furthermore, if the target object is entangled with other objects, the location information of the other objects and the location information of all their skeletal keypoints need to be determined and used as constraints for building the outer matrix, thereby creating a detection box that includes the location information of the other objects and the location information of all their skeletal keypoints.

[0116] In one example, a two-dimensional coordinate system is established for the image, and four coordinate points are generated around the location of the target object. The server can expand the positions of the four coordinate points outward based on the position information of the skeletal keypoints until an outer rectangle containing the coordinates of the skeletal keypoints is obtained, which serves as the detection box for the target object.

[0117] Step S606: Cropping the current image frame based on the circumscribed rectangle to obtain the target interest region corresponding to the target object.

[0118] Specifically, the server can crop the current image frame using the bounding rectangle between the target object and other objects to obtain the target region of interest, which includes the target object and other entangled objects.

[0119] Step S608: Input the target region of interest into the trained abnormal behavior recognition model to determine the initial abnormal recognition result of the target object.

[0120] The abnormal behavior recognition model is trained based on pre-labeled second training data to obtain a trained abnormal behavior recognition model. The second training data includes images of multiple objects without abnormal behavior and images of multiple objects with abnormal behavior.

[0121] Specifically, the server inputs the target interest region of the target object into the trained abnormal behavior recognition model to obtain the initial abnormal recognition result output by the abnormal behavior recognition model, namely the abnormal behavior confidence score.

[0122] In this embodiment, the current image frame in the image sequence to be identified is cropped using an outer rectangle to obtain a target region of interest (ROI) containing the target object and other objects entangled with it. This ROI is then input into a trained abnormal behavior recognition model to obtain an initial anomaly recognition result for the target object. Since the ROI only contains the target object and other objects, redundant image data is removed, allowing the abnormal behavior recognition model to output accurate recognition results, thereby improving the accuracy of abnormal behavior recognition.

[0123] In one exemplary embodiment, the abnormal behavior identification method further includes:

[0124] If the pose recognition model identifies a new object in the image frame of the image sequence to be recognized, a data structure for the new object is created; the data structure is used to store the location information and skeletal key point information of the new object.

[0125] In this context, a newly added object is an object that appears in a new image frame compared to the objects contained in previous image frames. Since the identification is performed using a pose recognition model, the server already has multiple objects contained in previous image frames. When a newly added object appears in the next image frame, a corresponding data structure needs to be created for the newly added object.

[0126] Specifically, if the pose recognition model identifies that an image frame in the image sequence to be recognized contains a new object, it determines the location information of the new object and the information of multiple skeletal key points of the new object, and stores them in the data structure corresponding to the new object.

[0127] In this embodiment, by creating a data structure for the newly added object, a copy of the object data can be retained for all objects that have appeared in the image sequence to be identified. This allows the previously established data structure to be used as a reference when the object is identified in the future, thereby obtaining the complete life cycle of each object and ensuring the integrity of the object data.

[0128] In one exemplary embodiment, such as Figure 7 As shown, the specific implementation process of the step "determining the target anomaly identification result corresponding to the target object based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial anomaly identification result" includes steps S702 to S706. Wherein:

[0129] Step S702: Determine the pose change frequency corresponding to the pose type queue based on the number of adjacent image frames with different pose categories in the pose type queue.

[0130] Specifically, the server can traverse the pose type queue in chronological order to determine the current pose category and the next pose category of the target object. If the current pose category and the next pose category are different, it is determined that the target object has completed a pose change, and the pose change frequency of the target object is updated. If the current pose category and the next pose category are the same, it is determined that the target object has not performed a pose change, and the pose change frequency of the target object is not updated. After the server completes the traversal of the pose type queue, it obtains the pose change frequency of the pose type queue.

[0131] Step S704: Determine the frequency of abnormal behaviors corresponding to the limb behavior queue based on the number of image frames in the limb behavior queue that are abnormal behaviors.

[0132] Specifically, the server can traverse the limb behavior queue in chronological order to determine the current limb behavior of the target object. If the current limb behavior is an abnormal limb behavior, the abnormal behavior frequency of the target object is updated. If the current limb behavior is a normal limb behavior, the abnormal behavior frequency of the target object is not updated. After the server completes the traversal of the limb behavior queue, it obtains the abnormal behavior frequency of the limb behavior queue.

[0133] Step S706: Based on preset weight values, the frequency of posture changes, the frequency of abnormal behavior, and the initial anomaly identification results of the target object are weighted and summed to obtain the comprehensive score of the target object, and the target anomaly identification result of the target object is determined based on the comprehensive score.

[0134] Specifically, the server acquires a first weight value corresponding to the frequency of posture changes, a second weight value corresponding to the frequency of abnormal behavior, and a third weight value corresponding to the initial anomaly identification result. The server can determine a first quotient between the frequency of posture changes and the length of the posture type queue, and a second quotient between the frequency of abnormal behavior and the length of the limb behavior queue. Then, the server calculates the product of the first weight value and the first quotient to obtain a first score; the server calculates the product of the second weight value and the second quotient to obtain a second score; and the server calculates the product of the third weight value and the initial anomaly identification result to obtain a third score. Finally, the server sums the first, second, and third scores to obtain the target anomaly identification result for the target object. If the target anomaly identification result is greater than a preset score, the target object is determined to have abnormal behavior; if the target anomaly identification result is less than or equal to the preset score, the target object is determined not to have abnormal behavior.

[0135] In this embodiment, the target anomaly identification result of the target object is determined by the frequency of posture changes, the frequency of limb behavior, and the initial anomaly identification result. The posture change information and the abnormal behavior information in multiple image frames can be merged into the initial anomaly identification result, thereby obtaining a more accurate target anomaly identification result and improving the accuracy of abnormal behavior identification.

[0136] like Figure 8 As shown, the specific execution process of the above-mentioned abnormal behavior identification method is described in detail below with reference to a specific embodiment. This method is applicable and has high accuracy in various public places, such as campuses, subway stations, and train stations. The method includes the following steps:

[0137] Step 10: Label pedestrian (object) data. Based on pedestrian posture, subdivide pedestrians into multiple categories such as normal upright (category 0), leaning forward or backward (category 1), swaying left or right (category 2), lying down (category 3), lying down (category 4), sitting (category 5), and kneeling (category 6). Then, label the location and human skeleton key points of each pedestrian. The labeled training data can be used to train the yolov8s-pose human detection and posture estimation model to obtain the trained human detection and posture estimation model.

[0138] Step 20: Create an abnormal behavior recognition dataset. Confine and cut out images of two or more people engaged in a fight to create the abnormal behavior dataset. Confine and cut out images of two or more normal pedestrians to create the normal behavior dataset. The server can train an abnormal behavior recognition model using the abnormal behavior dataset and the normal behavior dataset. This model can then classify pedestrians in images as either abnormal pedestrians or those engaging in normal behavior. In other words, the abnormal behavior recognition model is a binary classification model. In one example, the classification network can use neural network models such as ResNet, VGG, or MobileNet.

[0139] Step 30: Use yolov8s-pose to perform detailed pedestrian classification detection on the image sequence in the surveillance video. The video frame rate is typically 25 frames per second, so we can set it to detect every 6 frames, resulting in 4 detection results per second. For each image frame in the image sequence, the server can create a data structure to store the location and key points of the human skeleton for newly added objects, and put the identified pose category into a pose type queue of length L (when the queue is full, delete the data at the head of the queue and add data at the tail). For each existing object, calculate and determine whether the object has abnormal punching or kicking behavior based on the key points of the human skeleton. Normal behavior is assigned a value of 0, and abnormal punching and kicking are assigned a value of 1. The corresponding values ​​for the limb behaviors are put into a limb behavior queue of length L (when the queue is full, delete the data at the head of the queue and add data at the tail). Each time detailed pedestrian classification detection is performed, the pose type queue and limb behavior queue for each object can be updated.

[0140] like Figure 9 As shown, abnormal punches can be judged by the angle between the arm and the body, or by the height difference between the elbow and shoulder; abnormal leg raises can be judged by the angle between the leg and the body, or by the height difference between the hip and knee. The judgment method needs to be adjusted according to the position of the object and the direction it is facing.

[0141] Step 40: Based on the detection results of the above pedestrian sub-classification, determine whether two people are entangled together. The rules for determining whether two people are entangled are as follows:

[0142] a. The detection frames of the two individuals overlap, and the overlapping area accounts for more than 40% of the detection frame area;

[0143] b. At least one of the two individuals' subcategories has a non-zero value;

[0144] c. At least one of the two individuals exhibits abnormal foot lifting or punching behavior;

[0145] d. Whether the two people are facing each other can be determined based on the positional information of key points in the human skeleton. In one example, the angles between the line connecting the midpoints of the two people's shoulders and the perpendicular bisectors of each person's shoulders are α1 and α2, respectively. Both α1 and α2 must be less than a pre-set threshold; only when both are less than the threshold can the two people be facing each other. This further restricts the position of the two people and reduces misjudgments of entanglement relationships caused by visual differences. Each object can use a Map table to record the number of times it is entangled with other objects.

[0146] Step 50: The server can create a tracker for each detected object using a multi-object tracking algorithm and track it in real time. When the number of entanglements with other objects reaches a preset threshold, the bounding rectangles of the current object and other entangled objects are obtained. The image is then extracted based on the position of the bounding rectangles and fed into the abnormal behavior recognition model to determine if there is any abnormal behavior. For details, please refer to [link / reference]. Figure 8 .

[0147] Step 60: Determine whether the object has abnormal behavior based on the posture type queue, limb behavior queue, and abnormal behavior recognition results.

[0148] Furthermore, step 60 specifically includes:

[0149] Step 61, the pose type queue is [P1, P2, P3, P4, ..., P L If two adjacent poses are not the same, the pose change frequency N is incremented by 1.

[0150] Step 62, the limb behavior queue is [S1,S2,S3,S4,...,S] L The frequency M of abnormal behaviors (i.e., the number of 1s) in the limb behavior queue is counted. The posture type queue and the limb behavior queue can be entered into... Figure 10 As shown, the posture type queue determines the frequency of posture changes by the types of two adjacent postures in the queue; the limb behavior queue determines the frequency of abnormal behavior by the number of times abnormal limb behaviors occur in the queue.

[0151] Step 63, calculate the comprehensive score Z for abnormal behavior, using the following formula:

[0152]

[0153] Abnormal behavior can be determined when Z is greater than a preset threshold.

[0154] In this embodiment, the proposed fight recognition method based on human posture type queue can effectively determine the abnormal behavior of each object appearing in the video. Furthermore, this embodiment has high real-time performance and can quickly and accurately determine the abnormal behavior area and the object exhibiting the abnormal behavior.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides an abnormal behavior recognition device for implementing the abnormal behavior recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the abnormal behavior recognition device provided below can be found in the limitations of the abnormal behavior recognition method described above, and will not be repeated here.

[0157] In one exemplary embodiment, such as Figure 11 As shown, an abnormal behavior recognition device 1100 is provided, including: an object queue determination module 1101, a first recognition result determination module 1102, and a second recognition result determination module 1103, wherein:

[0158] The object queue determination module 1101 is used to perform behavior recognition on an image sequence containing multiple objects, determine the pose type queue and limb behavior queue of each object, and determine the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue of each object.

[0159] The first recognition result determination module 1102 is used to determine the initial abnormal recognition result of the target object based on the image sequence to be recognized and the trained abnormal behavior recognition model if the number of times the target object is entangled with other objects reaches a preset number.

[0160] The second identification result determination module 1103 is used to determine the target anomaly identification result corresponding to the target object based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial anomaly identification result.

[0161] Furthermore, the object queue determination module 1101 is specifically used for: identifying multiple objects contained in each image frame of the image sequence to be identified, and obtaining the position information and skeletal key point information of each object; for each object, identifying each image frame containing the current object according to the trained pose recognition model, obtaining the pose category of the current object in each image frame, and determining it as a pose type queue; and determining the limb behavior of the current object in each image frame based on the skeletal key point information of the current object in each image frame, and determining it as a limb behavior queue.

[0162] Further, the skeletal keypoint information includes information on a first keypoint, a second keypoint, a third keypoint, a fourth keypoint, a fifth keypoint, and a sixth keypoint; the object queue determination module 1101 is specifically used for: for each image frame, based on the skeletal keypoint information, determining a first angle between a first line segment and a second line segment of the current object, and first height information corresponding to the first keypoint and the second keypoint respectively; wherein, the first line segment is a straight line segment between the first keypoint and the second keypoint; the second line segment is a straight line segment between the third keypoint and the fourth keypoint of the current object; if the first angle and / or the first height information meets a first preset condition, then the limb behavior of the current object in the current image frame is determined to be a first abnormal behavior; based on the skeletal keypoint information, determining a second angle between a third line segment and a second line segment of the current object, and second height information corresponding to the fifth keypoint and the sixth keypoint respectively; wherein, the third line segment is a straight line segment between the fifth keypoint and the sixth keypoint; if the second angle and / or the second height information meets a second preset condition, then the limb behavior of the current object in the current image frame is determined to be a second abnormal behavior.

[0163] Furthermore, the object queue determination module 1101 is specifically used to: for each image frame in the image sequence to be identified, determine the overlapping area between the detection boxes of the target object and the detection boxes of other objects, and determine the facial orientation between the target object and other objects; the facial orientation is determined based on the skeletal key point information of the target object and other objects; the detection boxes are determined based on the position information of each object in the image frame; if the overlapping area is determined to be greater than a preset area value, and the facial orientation between the target object and other objects is determined to be face-to-face, and at least one of the pose categories of the target object and other objects is determined to be a preset pose category, and at least one of the limb behaviors of the target object and other objects is determined to be a preset limb behavior, then it is determined that the target object and other objects are entangled, and the entanglement count is updated.

[0164] Further, the first recognition result determination module 1102 is specifically used for: obtaining the current image frame corresponding to the image sequence to be recognized, wherein the current image frame is the image frame corresponding to the number of times the target object is entangled with other objects to a preset number; determining the bounding rectangle between the target object and other objects based on the detection boxes of the target object and other objects; cropping the current image frame based on the bounding rectangle to obtain the target interest region corresponding to the target object; and inputting the target interest region into the trained abnormal behavior recognition model to determine the initial abnormal recognition result of the target object.

[0165] Furthermore, the device also includes a data structure creation module, specifically used to: if the pose recognition model recognizes that the image frame in the image sequence to be recognized contains a new object, then create a data structure for the new object; the data structure is used to store the position information and skeletal key point information of the new object.

[0166] Furthermore, the second recognition result determination module 1103 is specifically used to: determine the pose change frequency corresponding to the pose type queue based on the number of adjacent image frames with different pose categories in the pose type queue; determine the abnormal behavior frequency corresponding to the limb behavior queue based on the number of image frames in the limb behavior queue where the limb behavior is abnormal; and, based on preset weight values, perform weighted summation on the pose change frequency, abnormal behavior frequency, and the initial abnormal recognition result of the target object to obtain a comprehensive score for the target object, and determine the target abnormal recognition result of the target object based on the comprehensive score.

[0167] Each module in the aforementioned abnormal behavior recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0168] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores training data for training an abnormal behavior recognition model, training data for training a pose recognition model, and image sequences to be recognized sent by the terminal. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an abnormal behavior recognition method.

[0169] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0171] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0172] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0174] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for identifying abnormal behavior, characterized in that, The method includes: Behavior recognition is performed on an image sequence containing multiple objects to be identified, determining a pose type queue and a limb behavior queue for each object, and determining the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue for each object. This includes: for each image frame in the image sequence to be identified, determining the overlapping area between the detection boxes of the target object and the detection boxes of other objects, and determining the facial orientation between the target object and the other objects; the facial orientation is determined based on the skeletal key point information of the target object and the other objects; if the overlapping area is determined to be greater than a preset area value, and the facial orientation between the target object and the other objects is face-to-face, and at least one of the pose categories of the target object and the other objects is a preset pose category, and at least one of the limb behaviors of the target object and the other objects is a preset limb behavior, then it is determined that the target object and the other objects are entangled, and the number of entanglements is updated. If the number of times the target object is entangled with other objects reaches a preset number, then based on the image sequence to be identified and the trained abnormal behavior recognition model, the initial abnormal recognition result of the target object is determined; Based on the frequency of posture changes in the posture type queue, the frequency of abnormal behavior in the limb behavior queue, and the initial anomaly identification result, the target anomaly identification result corresponding to the target object is determined, including: determining the frequency of posture changes corresponding to the posture type queue based on the number of adjacent image frames with different posture categories in the posture type queue; determining the frequency of abnormal behavior corresponding to the limb behavior queue based on the number of image frames with abnormal limb behavior in the limb behavior queue; and weighting and summing the frequency of posture changes, the frequency of abnormal behavior, and the initial anomaly identification result of the target object based on preset weight values ​​to obtain a comprehensive score for the target object, and determining the target anomaly identification result of the target object based on the comprehensive score.

2. The method according to claim 1, characterized in that, The step of performing behavior recognition on a sequence of images containing multiple objects to determine the pose type queue and limb behavior queue for each object includes: Multiple objects contained in each image frame of the image sequence to be identified are identified to obtain the position information and skeletal key point information of each object; For each of the objects, the image frames containing the current object are identified according to the trained pose recognition model to obtain the pose category of the current object in each of the image frames and determine it as a pose type queue. Based on the skeletal key point information of the current object in each of the image frames, the limb behavior of the current object in each of the image frames is determined and identified as a limb behavior queue.

3. The method according to claim 2, characterized in that, The skeletal key point information includes information on the first key point, the second key point, the third key point, the fourth key point, the fifth key point, and the sixth key point; The step of determining the limb behavior of the current object in each of the image frames based on the skeletal key point information of the current object includes: For each image frame, based on the skeletal key point information, a first angle between the first line segment and the second line segment of the current object, and first height information corresponding to the first key point and the second key point are determined respectively; wherein, the first line segment is a straight line segment between the first key point and the second key point; the second line segment is a straight line segment between the third key point and the fourth key point of the current object; If the first included angle and / or the first height information meets the first preset condition, then the limb behavior of the current object in the current image frame is determined to be the first abnormal behavior; Based on the skeletal key point information, the second included angle between the third line segment and the second line segment of the current object is determined, as well as the second height information corresponding to the fifth key point and the sixth key point respectively; wherein, the third line segment is the straight line segment between the fifth key point and the sixth key point; If the second included angle and / or the second height information meet the second preset condition, then the limb behavior of the current object in the current image frame is determined to be the second abnormal behavior.

4. The method according to claim 2, characterized in that, The step of determining the initial anomaly recognition result of the target object based on the image sequence to be recognized and the trained anomaly behavior recognition model includes: Obtain the current image frame corresponding to the image sequence to be identified. The current image frame is the image frame corresponding to the time when the number of times the target object is entangled with other objects reaches a preset number. Based on the detection bounding box of the target object and the detection bounding boxes of other objects, determine the bounding rectangle between the target object and the other objects; Based on the circumscribed rectangle, the current image frame is cropped to obtain the target interest region corresponding to the target object; The target region of interest is input into the trained abnormal behavior recognition model to determine the initial abnormal recognition result of the target object.

5. The method according to claim 2, characterized in that, The method further includes: If the pose recognition model identifies that an image frame in the image sequence to be recognized contains a new object, then a data structure for the new object is created; the data structure is used to store the position information and skeletal key point information of the new object.

6. An abnormal behavior recognition device, characterized in that, The device includes: An object queue determination module is used to perform behavior recognition on an image sequence containing multiple objects, determine a pose type queue and a limb behavior queue for each object, and determine the number of entanglements between the target object and other objects based on the pose type queue and limb behavior queue for each object. This includes: for each image frame in the image sequence to be recognized, determining the overlapping area between the detection boxes of the target object and the detection boxes of other objects, and determining the facial orientation between the target object and the other objects; the facial orientation is determined based on the skeletal keypoint information of the target object and the other objects; if the overlapping area is determined to be greater than a preset area value, and the facial orientation between the target object and the other objects is face-to-face, and at least one of the pose categories of the target object and the other objects is a preset pose category, and at least one of the limb behaviors of the target object and the other objects is a preset limb behavior, then it is determined that the target object and the other objects are entangled, and the number of entanglements is updated. The first identification result determination module is used to determine the initial abnormal identification result of the target object based on the image sequence to be identified and the trained abnormal behavior identification model if the number of entanglements between the target object and other objects reaches a preset number. The second identification result determination module is used to determine the target anomaly identification result corresponding to the target object based on the posture change frequency in the posture type queue, the abnormal behavior frequency in the limb behavior queue, and the initial anomaly identification result. This includes: determining the posture change frequency corresponding to the posture type queue based on the number of adjacent image frames with different posture categories in the posture type queue; determining the abnormal behavior frequency corresponding to the limb behavior queue based on the number of image frames in the limb behavior queue where the limb behavior is abnormal; and performing a weighted summation of the posture change frequency, the abnormal behavior frequency, and the initial anomaly identification result of the target object based on preset weight values ​​to obtain a comprehensive score for the target object, and determining the target anomaly identification result of the target object based on the comprehensive score.

7. The apparatus according to claim 6, characterized in that, The object queue determination module is also used for: Multiple objects contained in each image frame of the image sequence to be identified are identified to obtain the position information and skeletal key point information of each object; For each of the objects, the image frames containing the current object are identified according to the trained pose recognition model to obtain the pose category of the current object in each of the image frames and determine it as a pose type queue. Based on the skeletal key point information of the current object in each of the image frames, the limb behavior of the current object in each of the image frames is determined and identified as a limb behavior queue.

8. The apparatus according to claim 7, characterized in that, The skeletal keypoint information includes information on the first keypoint, the second keypoint, the third keypoint, the fourth keypoint, the fifth keypoint, and the sixth keypoint; the object queue determination module is further used for: For each image frame, based on the skeletal key point information, a first angle between the first line segment and the second line segment of the current object, and first height information corresponding to the first key point and the second key point are determined respectively; wherein, the first line segment is a straight line segment between the first key point and the second key point; the second line segment is a straight line segment between the third key point and the fourth key point of the current object; If the first included angle and / or the first height information meets the first preset condition, then the limb behavior of the current object in the current image frame is determined to be the first abnormal behavior; Based on the skeletal key point information, the second included angle between the third line segment and the second line segment of the current object is determined, as well as the second height information corresponding to the fifth key point and the sixth key point respectively; wherein, the third line segment is the straight line segment between the fifth key point and the sixth key point; If the second included angle and / or the second height information meet the second preset conditions, then the limb behavior of the current object in the current image frame is determined to be the second abnormal behavior.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images