Abnormal behavior recognition method, device, equipment and medium

Abstract

Embodiments of the present disclosure provide an abnormal behavior recognition method and device, equipment and medium, relating to the field of artificial intelligence and big data. The method comprises: obtaining a to-be-processed video stream, sampling the to-be-processed video stream to obtain a plurality of video frame images of the to-be-processed video stream; performing target detection and recognition on the plurality of video frame images to obtain image time sequence data of each target object and key point time sequence data of each target object; based on a pre-trained abnormal behavior recognition model, determining individual abnormal behavior labels of each target object and group abnormal behavior labels of a plurality of target objects according to the image time sequence data of each target object and the key point time sequence data of each target object, wherein the abnormal behavior recognition model comprises an individual abnormal behavior recognition model and a group abnormal behavior recognition model. The abnormal behavior recognition method has strong generalization ability, can recognize complex and continuous abnormal behaviors, is simple to implement, and has high recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence and big data, and in particular, to an abnormal behavior recognition method, an abnormal behavior recognition device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of information technology, security cameras that are ubiquitous in the streets and alleys provide a strong boost for the improvement of urban public security. Through artificial monitoring, intelligent prevention and control and other methods, various abnormal behaviors that interfere with public security can be alarmed in the process and traced back afterwards, further improving the urban public security environment.

[0003] In related technologies, an abnormal behavior recognition is performed based on single-frame RGB (a color standard representing the colors of red, green and blue three channels) image classification or database comparison method. However, the above method has the problems of weak generalization ability and inability to recognize complex continuous abnormal behaviors. The method of adopting different strategies for individual and group abnormal behavior recognition has complex logic and high implementation difficulty, and has low recognition accuracy.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present disclosure provides an abnormal behavior recognition method, an abnormal behavior recognition device, an electronic device and a computer readable storage medium, which solve the problem of weak generalization ability and inability to recognize complex continuous abnormal behaviors in related technologies, have simple implementation and high recognition accuracy.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, an abnormal behavior recognition method is provided, which includes: acquiring a to-be-processed video stream, sampling the to-be-processed video stream to obtain a plurality of video frame images of the to-be-processed video stream; performing target detection and recognition on the plurality of video frame images to obtain image time sequence data of each target object and key point time sequence data of each target object; based on a pre-trained abnormal behavior recognition model, determining individual abnormal behavior labels of each target object and group abnormal behavior labels of a plurality of target objects according to the image time sequence data of each target object and the key point time sequence data of each target object, wherein the abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model.

[0008] In some embodiments of the present disclosure, the individual abnormal behavior recognition model comprises a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and the group abnormal behavior recognition model comprises a backbone network for group behavior recognition and a classification head for group behavior recognition; wherein the pre-trained abnormal behavior recognition model determines the individual abnormal behavior label of each target object and the group abnormal behavior label of multiple target objects according to the image time series data of each target object and the key point time series data of each target object, comprising: obtaining the individual action features of each target object by using the backbone network for individual behavior recognition according to the image time series data of each target object and the key point time series data of each target object; inputting the individual action features of each target object into the classification head for individual behavior recognition respectively, and outputting the individual abnormal behavior label of each target object; adding the individual action features of each target object and the pedestrian box encoding of each target object respectively to obtain the fusion features of each target object; inputting the fusion features of each target object into the backbone network for group behavior recognition to obtain the group behavior features of the multiple target objects; and inputting the group behavior features of the multiple target objects into the classification head for group behavior recognition to output the group abnormal behavior label of the multiple target objects.

[0009] In some embodiments of the present disclosure, the backbone network for individual behavior recognition comprises a first backbone network and a second backbone network, the first backbone network is a backbone network of a time domain convolution network, and the second backbone network is a backbone network of a space-time graph convolution network; wherein the backbone network for individual behavior recognition is used to obtain the individual action features of each target object according to the image time series data of each target object and the key point time series data of each target object, comprising: inputting the image time series data of each target object into the first backbone network respectively to obtain the image time series features of each target object; inputting the key point time series data of each target object into the second backbone network respectively to obtain the key point time series features of each target object; and concatenating the image time series features of each target object and the key point time series features of each target object respectively to obtain the individual action features of each target object.

[0010] In some embodiments of the present disclosure, the individual abnormal behavior recognition model is trained according to the following method: obtaining the image time series data of a sample object, the key point time series data of the sample object, and the individual action category of the sample object; and training the backbone network for individual behavior recognition and the classification head for individual behavior recognition comprised in the individual abnormal behavior recognition model according to the image time series data of the sample object, the key point time series data of the sample object, and the individual action category of the sample object.

[0011] In some embodiments of the present disclosure, the group abnormal behavior recognition model is trained according to the following method: obtaining image time series data of multiple sample objects, key point time series data of the multiple sample objects, and group action categories of the multiple sample objects; based on the image time series data of the multiple sample objects, the key point time series data of the multiple sample objects, and the group action categories of the multiple sample objects, and using the backbone network of individual behavior recognition obtained through training, training the group abnormal behavior recognition model including the backbone network of group behavior recognition and the classification head of group behavior recognition.

[0012] In some embodiments of the present disclosure, the method further includes: performing mask reconstruction on the image timing data of the sample object according to a first preset proportional frame number, and performing mask reconstruction on the pixels in the image timing data of the sample object; performing mask reconstruction on the key point timing data of the sample object according to a second preset proportional frame number, and performing mask reconstruction on the key points in the key point timing data of the sample object.

[0013] In some embodiments of the present disclosure, target detection and recognition are performed on the multiple video frame images to obtain image timing data of each target object and key point timing data of each target object, including: detecting and tracking pedestrians in the multiple video frame images to obtain pedestrian detection frames of each target object, and obtaining image timing data of each target object; and inputting the images in the pedestrian detection frames of each target object into a key point recognition model respectively to obtain key point timing data of each target object.

[0014] According to another aspect of the present disclosure, an abnormal behavior identification device is provided, which includes: a sampling module for acquiring a video stream to be processed, sampling the video stream to be processed, and obtaining multiple video frame images of the video stream to be processed; a target detection and identification module for performing target detection and identification on the multiple video frame images, and obtaining image time series data of each target object and key point time series data of each target object; a behavior recognition module for determining, based on a pre-trained abnormal behavior recognition model, individual abnormal behavior labels of each target object and group abnormal behavior labels of multiple target objects according to the image time series data of each target object and the key point time series data of each target object, wherein the abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model.

[0015] In some embodiments of the present disclosure, the individual abnormal behavior recognition model comprises a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and the group abnormal behavior recognition model comprises a backbone network for group behavior recognition and a classification head for group behavior recognition; wherein the behavior recognition module is further configured to: obtain individual action features of each target object by using the backbone network for individual behavior recognition according to image time sequence data of each target object and key point time sequence data of each target object; input the individual action features of each target object into the classification head for individual behavior recognition respectively, and output individual abnormal behavior labels of each target object; add individual action features of each target object and pedestrian box encoding of each target object respectively to obtain fusion features of each target object; input the fusion features of each target object into the backbone network for group behavior recognition to obtain group behavior features of the plurality of target objects; and input the group behavior features of the plurality of target objects into the classification head for group behavior recognition to output group abnormal behavior labels of the plurality of target objects.

[0016] In some embodiments of the present disclosure, the backbone network for individual behavior recognition comprises a first backbone network and a second backbone network, the first backbone network is a backbone network of a time domain convolution network, and the second backbone network is a backbone network of a space-time graph convolution network; wherein the behavior recognition module is further configured to: input image time sequence data of each target object into the first backbone network respectively to obtain image time sequence features of each target object; input key point time sequence data of each target object into the second backbone network respectively to obtain key point time sequence features of each target object; and concatenate the image time sequence features of each target object and the key point time sequence features of each target object respectively to obtain individual action features of each target object.

[0017] In some embodiments of the present disclosure, the device further comprises a model training module configured to train an individual abnormal behavior recognition model according to the following method: obtaining image time sequence data of a sample object, key point time sequence data of the sample object, and individual action categories of the sample object; and training a backbone network for individual behavior recognition and a classification head for individual behavior recognition comprised in the individual abnormal behavior recognition model according to the image time sequence data of the sample object, the key point time sequence data of the sample object, and the individual action categories of the sample object.

[0018] In some embodiments of the present disclosure, the model training module is further configured to train the group abnormal behavior recognition model according to the following method: obtaining image time series data of a plurality of sample objects, key point time series data of the plurality of sample objects, and group action categories of the plurality of sample objects; and training a group behavior recognition backbone network and a group behavior recognition classification head included in the group abnormal behavior recognition model according to the image time series data of the plurality of sample objects, the key point time series data of the plurality of sample objects, and the group action categories of the plurality of sample objects, and using the trained individual behavior recognition backbone network.

[0019] In some embodiments of the present disclosure, the model training module is further configured to: mask reconstruct the image time series data of the sample object according to a first preset proportion of frame numbers, and mask reconstruct pixels in the image time series data of the sample object; and mask reconstruct the key point time series data of the sample object according to a second preset proportion of frame numbers, and mask reconstruct key points in the key point time series data of the sample object.

[0020] In some embodiments of the present disclosure, the target detection and recognition module is further configured to: detect and track pedestrians in the plurality of video frame images, obtain pedestrian detection boxes of each target object, and obtain image time series data of each target object; input images in the pedestrian detection boxes of each target object into a key point recognition model respectively, and obtain key point time series data of each target object.

[0021] According to yet another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the abnormal behavior recognition method as described in the above embodiments.

[0022] According to yet another aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, the program being executed by a processor to implement the abnormal behavior recognition method as described in the above embodiments.

[0023] The method for identifying abnormal behavior provided by the embodiments of the present disclosure first samples a to-be-processed video stream to obtain a plurality of video frame images, then detects and identifies the plurality of video frame images to obtain image time sequence data and key point time sequence data of each target object contained in the images, and then uses a pre-trained individual abnormal behavior identification model and a group abnormal behavior identification model to determine the individual abnormal behavior of each target object and the group abnormal behavior of a plurality of target objects in combination with the image time sequence data and the key point time sequence data of each target object. In this way, the abnormal behavior is identified according to the image time sequence data and the key point time sequence data, the RGB image features and the key point features are considered, and the generalization ability of the abnormal behavior identification model in different scenes is improved; the plurality of video frame images are detected and identified to obtain the time sequence data of the target objects, and then the abnormal behavior is identified, so that the complex abnormal behavior can be identified through the multi-frame time sequence features; and the individual and group abnormal behavior identification models are used to identify the abnormal behavior, so that the support for identifying the individual and group abnormal behavior is realized.

[0024] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings incorporated in the specification and forming a part of it illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0026] Figure 1 is a schematic diagram of a system architecture of a method for identifying abnormal behavior according to an exemplary embodiment;

[0027] Figure 2 is a flowchart of a method for identifying abnormal behavior according to an exemplary embodiment;

[0028] Figure 3 is a flowchart of training an individual abnormal behavior identification model according to an exemplary embodiment;

[0029] Figure 4 is a flowchart of training a group abnormal behavior identification model according to an exemplary embodiment;

[0030] Figure 5 is a flowchart of determining an abnormal behavior label based on an abnormal behavior identification model according to an exemplary embodiment;

[0031] Figure 6 is a flowchart of an individual and group abnormal behavior identification method fusing RGB image time sequence features and key point time sequence features according to an exemplary embodiment;

[0032] Figure 7 is a flow chart of an abnormal behavior recognition method based on an individual and group abnormal behavior recognition model according to an example embodiment;

[0033] Figure 8 is a structural schematic diagram of an abnormal behavior recognition device according to an example embodiment;

[0034] Figure 9 is a structural block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0035] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings.

[0036] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0037] The example implementations described below can be implemented in various forms and should not be construed as being limited to the examples set forth herein; rather, these implementations are provided so that the present disclosure will be more comprehensive and complete, and the ideas of the example implementations will be fully conveyed to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more implementations. In the following description, many specific details are provided to give a sufficient understanding of the implementations of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring the aspects of the present disclosure.

[0038] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities, which do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] It should be noted that in the technical solutions of the present disclosure, the collection, collection, updating, analysis, processing, use, transmission, storage, etc. of user personal information involved in the technical solutions comply with relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data, and the security of user personal information, network security and national security are maintained.

[0040] Figure 1 A system architecture diagram of an abnormal behavior recognition method according to an exemplary embodiment is shown, which includes a server 101, a network 102 and a client 103. The network 102 is a medium for providing a communication link between the client 103 and the server 101. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0041] The server 101 can be a standalone physical server, a server cluster or a distributed system formed by multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. In some actual applications, the server 101 can also be a server of a network platform, which can be a transaction platform, a live broadcast platform, a social platform or a music platform, etc., and the embodiments of the present disclosure do not limit the same. The server can be a server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0042] The client 103 can include but is not limited to smart phones, desktop computers, tablet computers, notebook computers, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, smart wearable devices, etc. Type of electronic device, or the client 103 can also be a personal computer, such as a laptop computer and a desktop computer, etc. Optionally, the operating system running on the electronic device can include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0043] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the steps of the abnormal behavior recognition method of the embodiments of the present disclosure will be described in more detail below in conjunction with the drawings and embodiments.

[0044] Figure 2is a flowchart of an abnormal behavior recognition method according to an exemplary embodiment, see Figure 2 The method is applied to Figure 1 The server shown in the figure specifically includes the following steps.

[0045] Step S201: Obtain a to-be-processed video stream, sample the to-be-processed video stream, and obtain multiple frame images of the to-be-processed video stream.

[0046] The to-be-processed video stream can be a video in a period of time. After the to-be-processed video stream is obtained, the frame extraction index of the video stream can be calculated, and then the to-be-processed video stream is sampled by using the calculated frame extraction index to obtain multiple frame images of the to-be-processed video stream. The multiple frame images can be understood as multiple frame images obtained by sampling the to-be-processed video stream. In the embodiment of the present disclosure, the frame extraction index is dynamically calculated for the video stream with different frame rates, and uniform sampling of the video stream is realized.

[0047] Step S202: Perform target detection and recognition on the multiple frame images to obtain image time series data of each target object and key point time series data of each target object.

[0048] The target detection and recognition specifically refers to detecting and recognizing pedestrians in the images. The target object can be understood as a pedestrian in the multiple frame images, i.e., a pedestrian in the to-be-processed video stream. The image time series data of the target object is the RGB image time series data of the pedestrian, and the key point time series data of the target object is the key point time series data of the pedestrian. For ease of description, the target object is taken as a pedestrian in the following description.

[0049] In this step, the multiple frame images can be subjected to pedestrian detection and recognition to obtain the RGB image time series data of each pedestrian and the key point time series data of each pedestrian, and then the RGB image time series data and the key point time series data of these pedestrians can be analyzed to determine the abnormal behavior of the pedestrians.

[0050] In the embodiment of the present disclosure, the target detection and recognition of the multiple frame images to obtain the image time series data of each target object and the key point time series data of each target object can include: detecting and tracking the pedestrians in the multiple frame images to obtain a pedestrian detection frame of each target object and to obtain the image time series data of each target object; and inputting the images in the pedestrian detection frame of each target object into a key point recognition model to obtain the key point time series data of each target object.

[0051] Specifically, based on the target detection algorithm and the tracking algorithm, the pedestrians in the multiple frames of video frame images are detected and tracked, the detection boxes of the pedestrians contained in the images are obtained, the RGB images in each pedestrian box are obtained, and then the time sequence data of the RGB images of each pedestrian are obtained. In addition, the RGB images in each pedestrian box can be input into a key point recognition model to identify the key points in the RGB images and obtain the time sequence data of the key points of each pedestrian. The target detection algorithm, the tracking algorithm and the key point recognition model are all mature technologies, and the embodiments of the present disclosure do not expand the description.

[0052] It should be noted that the multiple frames of video frame images contain one or more pedestrians, and the embodiments of the present disclosure can obtain all the pedestrians in the multiple frames of video frame images based on the target detection algorithm and the tracking algorithm, and obtain the time sequence data of the RGB images of each pedestrian and the time sequence data of the key points of each pedestrian.

[0053] In the embodiments of the present disclosure, after obtaining the image time sequence data of each target object and the key point time sequence data of each target object, the abnormal behavior recognition method further includes: pre-processing the image time sequence data of each target object and the key point time sequence data of each target object. Specifically, the useless data in the time sequence data is deleted, and the data is normalized and the like.

[0054] Step S203: Based on the pre-trained abnormal behavior recognition model, the individual abnormal behavior label of each target object and the group abnormal behavior label of multiple target objects are determined according to the image time sequence data of each target object and the key point time sequence data of each target object.

[0055] The abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model. The individual abnormal behavior recognition model is used to output the individual abnormal behavior of a single pedestrian, such as kneeling, running, slapping, kicking, lifting the leg, punching and the like. The group abnormal behavior recognition model is used to output the group abnormal behavior of multiple pedestrians, such as fighting, crowd gathering, and personnel pursuit.

[0056] In this step, for each pedestrian, the RGB image time sequence data of the pedestrian and the key point time sequence data of the pedestrian are input into the individual abnormal behavior recognition model, and the individual abnormal behavior label of the pedestrian is output. For multiple pedestrians, the RGB image time sequence data of each pedestrian and the key point time sequence data of each pedestrian are input into the group abnormal behavior recognition model, and the group abnormal behavior label of the multiple pedestrians is output.

[0057] The method for identifying abnormal behavior provided by the embodiments of the present disclosure first samples a to-be-processed video stream to obtain a plurality of video frame images, then detects and identifies the plurality of video frame images to obtain image time sequence data and key point time sequence data of each target object contained in the images, and then determines individual abnormal behavior of each target object and group abnormal behavior of a plurality of target objects by using a pre-trained individual abnormal behavior identification model and a group abnormal behavior identification model in combination with the image time sequence data and the key point time sequence data of each target object. In this way, the abnormal behavior is identified according to the image time sequence data and the key point time sequence data, the RGB image features and the key point features are considered, and the generalization ability of the abnormal behavior identification model in different scenes is improved. The plurality of video frame images are detected and identified to obtain the time sequence data of the target objects, and then the abnormal behavior is identified, so that the complex abnormal behavior can be identified through the multi-frame time sequence features. In addition, the individual and group abnormal behavior identification models are proposed to identify the abnormal behavior, and the support for the individual and group abnormal behavior identification capabilities is realized.

[0058] In the embodiments of the present disclosure, the individual abnormal behavior identification model includes a backbone network for individual behavior identification and a classification head for individual behavior identification. The backbone network for individual behavior identification includes a first backbone network and a second backbone network. The first backbone network is a backbone network of a time domain convolution network, and the second backbone network is a backbone network of a space-time graph convolution network. The first backbone network is used to obtain time sequence features of an RGB image, and can be a backbone network of a TCN (Time Convolution Network). The TCN is a novel algorithm that can be used to solve time series prediction. The second backbone network is used to obtain key point time sequence features, and can be a backbone network of an STGCN (Space-Time Graph Convolutional Neural Network). The STGCN is a novel deep learning framework that can be used to solve time series prediction problems in the field of transportation.

[0059] After obtaining the time sequence features of the RGB image and the key point time sequence features by using the first backbone network and the second backbone network, the time sequence features of the RGB image and the key point time sequence features can be concatenated to obtain individual action features. The classification head for individual behavior identification is used to determine individual abnormal behavior according to the individual action features. In the embodiments of the present disclosure, the classification head for individual behavior identification can be a fully connected layer (fc layer) of a TCN.

[0060] Figure 3 FIG. 1 is a flowchart illustrating a process of training an individual abnormal behavior identification model according to an example embodiment. As shown in FIG. 1, the training of the individual abnormal behavior identification model is implemented in the following manner. Figure 3

[0061] Step S301: Obtain image time sequence data of a sample object, key point time sequence data of the sample object, and an individual action category of the sample object. ​

[0062] The image time series data of the sample object and the key point time series data of the sample object can be pedestrian RGB image time series data and pedestrian key point time series data in a period of time. The individual object category of the sample object can be an individual action category of a pedestrian.

[0063] It should be noted that the image time series data and the key point time series data of the sample object in the embodiments of the present disclosure can be obtained by the following method: obtaining a sample video stream in a period of time, sampling the sample video, obtaining a plurality of video frame images corresponding to the sample video stream, and then detecting and recognizing the plurality of video frame images and performing tracking processing to obtain RGB image data and key point time series data of the sample object. The sample object is a pedestrian in the sample video stream.

[0064] In the embodiments of the present disclosure, the abnormal behavior recognition method can further include: performing mask reconstruction on the image time series data of the sample object according to a first preset proportion of frame numbers, and performing mask reconstruction on the pixels in the image time series data of the sample object; and performing mask reconstruction on the key point time series data of the sample object according to a second preset proportion of frame numbers, and performing mask reconstruction on the key points in the key point time series data of the sample object.

[0065] In order to enhance the generalization ability of the model and improve the accuracy of the model, in the embodiments of the present disclosure, during the training of the individual abnormal behavior recognition model, a self-supervised pre-training method can be used first, and then a supervised training method can be used. The self-supervised training method can be a mask reconstruction strategy:

[0066] (1) The image time series data of the sample object is reconstructed according to a first preset proportion of frame numbers, and the pixels in the image time series data of the sample object are reconstructed. The first preset proportion of frame numbers can be randomly set or set according to experience. In addition, part of the pixels in the image time series data can be reconstructed.

[0067] (2) The key point time series data of the sample object is reconstructed according to a second preset proportion of frame numbers, and the key points in the key point time series data of the sample object are reconstructed. The second preset proportion of frame numbers can be randomly set or set according to experience. In addition, part of the key points in the key point time series data can be reconstructed.

[0068] Step S302: Training the individual behavior recognition backbone network and the individual behavior recognition classification head included in the individual abnormal behavior recognition model according to the image time series data of the sample object, the key point time series data of the sample object, and the individual action category of the sample object.

[0069] After obtaining the image time series data, key point time series data and corresponding individual action category of the sample object, model training is performed to obtain an individual abnormal behavior recognition model, i.e., a backbone network for individual behavior recognition and a classification head for individual behavior recognition are trained.

[0070] In the embodiments of the present disclosure, the group abnormal behavior recognition model includes a backbone network for group behavior recognition and a classification head for group behavior recognition. The backbone network for group behavior recognition is used to obtain group behavior features, and can be a Transformer model. The classification head for group behavior recognition is used to determine group abnormal behavior according to the group behavior features, and can specifically be a fully connected layer, i.e., an fc layer.

[0071] Figure 4 is a flowchart for training a group abnormal behavior recognition model according to an exemplary embodiment. As shown in Figure 4 , the training of the group abnormal behavior recognition model is implemented in the following manner.

[0072] Step S401: Obtain image time series data of a plurality of sample objects, key point time series data of the plurality of sample objects, and group action categories of the plurality of sample objects.

[0073] The image time series data of the plurality of sample objects and the key point time series data of the plurality of sample objects can be a plurality of pedestrian RGB image time series data and a plurality of pedestrian key point time series data in a period of time. The individual object categories of the plurality of sample objects can be group action categories of a plurality of pedestrians.

[0074] Step S402: According to the image time series data of the plurality of sample objects, the key point time series data of the plurality of sample objects, and the group action categories of the plurality of sample objects, and using the trained backbone network for individual behavior recognition, train the backbone network for group behavior recognition and the classification head for group behavior recognition included in the group abnormal behavior recognition model.

[0075] During the training of the backbone network for group abnormal behavior recognition, the group action features are obtained using individual action features, and the individual action features are obtained by the first backbone network and the second backbone network of the backbone network for individual behavior recognition. Specifically, the first backbone network is used to obtain the time series features of the RGB image, and the second backbone network is used to obtain the key point time series features. After obtaining the time series features of the RGB image and the key point time series features using the first backbone network and the second backbone network, the time series features of the RGB image and the key point time series features can be concatenated to obtain the individual action features.

[0076] Therefore, in the embodiments of the present disclosure, the backbone network of individual behavior recognition and the classification head of individual behavior recognition of the individual anomaly recognition model are first trained, the model weight of the part of the trained backbone network of individual behavior recognition is frozen, the classification head of individual behavior recognition is removed, and then the backbone network of group abnormal behavior recognition and the classification head of group abnormal behavior recognition of the group anomaly recognition model are trained.

[0077] In the related art, different strategies are used for individual and group abnormal behavior recognition, which is logically complex and difficult to implement. In the embodiments of the present disclosure, an individual abnormal behavior recognition model and a group abnormal behavior recognition model are proposed, which improves the simplicity of the abnormal behavior recognition method and can be further developed into a single-stage training individual and group abnormal behavior recognition method.

[0078] Figure 5 is a flowchart of determining an abnormal behavior label based on an abnormal behavior recognition model according to an exemplary embodiment. As shown in Figure 5 Based on the pre-trained abnormal behavior recognition model, the individual abnormal behavior label of each target object and the group abnormal behavior label of multiple target objects are determined according to the image time series data of each target object and the key point time series data of each target object, which can include:

[0079] Step S501: obtaining individual action features of each target object by using the backbone network of individual behavior recognition according to the image time series data of each target object and the key point time series data of each target object.

[0080] Further, obtaining individual action features of each target object by using the backbone network of individual behavior recognition according to the image time series data of each target object and the key point time series data of each target object can include: inputting the image time series data of each target object into the first backbone network respectively to obtain the image time series features of each target object; inputting the key point time series data of each target object into the second backbone network respectively to obtain the key point time series features of each target object; and concatenating the image time series features of each target object and the key point time series features of each target object respectively to obtain the individual action features of each target object.

[0081] As explained above, the backbone network of individual behavior recognition includes a first backbone network and a second backbone network, the first backbone network is a backbone network of a time domain convolution network, and the second backbone network is a backbone network of a space-time graph convolution network. The first backbone network is used to obtain the time series features of the RGB image, and the second backbone network is used to obtain the key point time series features. After obtaining the time series features of the RGB image and the key point time series features, the features can be concatenated to obtain the individual action features.

[0082] For each pedestrian (i.e., target object) in multiple video frames, the pedestrian's RGB image time series data can be input into the first backbone network to obtain the pedestrian's image time series features, and the pedestrian's key point time series data can be input into the second backbone network to obtain the pedestrian's key point time series features. The pedestrian's image time series features and the pedestrian's key point time series features are then cascaded to obtain the pedestrian's individual motion features. In this way, the individual motion features of each pedestrian in multiple video frames can be obtained.

[0083] Step S502: Input the individual action features of each target object into the classification head of individual behavior recognition, and output the individual abnormal behavior label of each target object.

[0084] For each pedestrian (i.e., target object) in the multi-frame video images, the pedestrian's individual motion features are input into the classification head for individual behavior recognition, and the pedestrian's individual abnormal behavior is output. In this way, the individual abnormal behavior of each pedestrian can be obtained.

[0085] Step S503: Adding the individual motion features of each target object and the pedestrian frame code of each target object respectively to obtain the fusion features of each target object.

[0086] In this step, for each pedestrian (i.e., target object) in the multi-frame video image, the pedestrian's individual motion features are added to the pedestrian frame code of the pedestrian to obtain the pedestrian's fusion features. In this way, the fusion features of each pedestrian in the multi-frame video image can be obtained.

[0087] Step S504: inputting the fusion features of each target object into the backbone network of group behavior recognition to obtain group behavior features of multiple target objects.

[0088] In this step, the fusion features of each of the multiple pedestrians (i.e., all target objects in the multiple video frame images) are input into the backbone network of group behavior recognition, and the group behavior features of the multiple pedestrians are output.

[0089] Step S505: inputting the group behavior features of the multiple target objects into a classification head for group behavior recognition, and outputting abnormal group behavior labels of the multiple target objects.

[0090] In this step, the group behavior features of multiple pedestrians are input into the classification head of group behavior recognition, and the abnormal group behavior of multiple pedestrians is output.

[0091] In the abnormal behavior recognition process based on the abnormal behavior recognition model, the image time series data and the key point time series data of the target object can be analyzed by using the backbone network of individual behavior recognition to output the individual action features of the target object, and then the individual abnormal behavior can be output by using the classification head of individual behavior recognition. In addition, after obtaining the individual action features of each target object, the fusion features of each target object can be obtained by combining the pedestrian box encoding of each target object, and then the group action features of multiple target objects can be output by using the backbone network of group behavior recognition, and then the group abnormal behavior can be output by using the classification head of group abnormal behavior recognition. In this way, the abnormal behavior recognition is performed according to the image time series data and the key point time series data, and the dependence of the action recognition result on the RGB image features can be reduced by combining the RGB image features and the key point features, the generalization ability of the abnormal behavior recognition model in different scenes can be improved, and the individual and group abnormal behavior recognition model is proposed to perform abnormal behavior recognition, and the individual and group abnormal behavior recognition capability is supported.

[0092] The following specific embodiments are listed to describe the abnormal behavior recognition method provided in the embodiments of the present disclosure.

[0093] Figure 6 is a flowchart of an individual and group abnormal behavior recognition method combining RGB image time series features and key point time series features according to an example embodiment. As shown in Figure 6 , the specific implementation of the method is: obtaining a to-be-processed video stream in a period of time, sampling the to-be-processed video stream to obtain multiple video frame images; then, detecting and tracking pedestrians in the multiple video frame images to obtain the RGB image action sequence (i.e. RGB image time series data) of each pedestrian, and identifying the key points of the RGB image in the pedestrian detection box of each pedestrian based on a key point recognition model to obtain the key point action sequence (i.e. key point time series data) of each pedestrian; finally, performing abnormal behavior recognition according to the RGB image action sequence of each pedestrian and the key point action sequence of each pedestrian.

[0094] Figure 7 is a flowchart of an abnormal behavior recognition method based on an individual and group abnormal behavior recognition model according to an example embodiment. Figure 7 In the method, the multiple frames of key points (i.e. key point time series data) of a single pedestrian are input into the STGCN backbone network to output the key point time series features of the pedestrian, the multiple frames of RGB time series images (i.e. RGB image time series data) of the single pedestrian are input into the TCN backbone network to output the image time series features of the pedestrian, then the key point time series features of the pedestrian and the image time series features of the pedestrian are concatenated to obtain the individual action features of the pedestrian, and then the individual action features of the pedestrian are input into the Head-FC layer (i.e. the classification head of individual behavior recognition) to output the abnormal behavior label of the single pedestrian.

[0095] In addition, Figure 7 In the middle, after obtaining the individual action features of a single pedestrian, the pedestrian's detection box position code is combined to obtain the fusion features of the pedestrian; after obtaining the fusion features of multiple pedestrians, the fusion features of these pedestrians are input into the transformer (i.e., the backbone network of group behavior recognition), and the group action features are output; then, the group action features are input into the Head-FC layer (i.e., the classification head of group behavior recognition), and the group abnormal behavior label is output.

[0096] The abnormal behavior recognition method provided by the embodiments of the present disclosure performs abnormal behavior recognition according to image time series data and key point time series data, considers RGB image features and key point features, and improves the generalization ability of the abnormal behavior recognition model in different scenes; the multiple frame video frame images are detected and recognized to obtain the time series data of the target object, and then the abnormal behavior recognition is performed, which can realize the recognition of complex abnormal behaviors through multiple frame time series features; and the individual and group abnormal behavior recognition models are proposed to perform abnormal behavior recognition, thereby supporting the individual and group abnormal behavior recognition capabilities.

[0097] Based on the same inventive concept, the embodiments of the present disclosure provide an abnormal behavior recognition device, as described in the following embodiments. Since the principle of solving problems of the abnormal behavior recognition device embodiments is similar to the above-mentioned method embodiments, the implementation of the abnormal behavior recognition device embodiments can be referred to the implementation of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0098] Figure 8 is a structural schematic diagram of an abnormal behavior recognition device according to an example embodiment. As shown in Figure 8 The abnormal behavior recognition device 800 can include a sampling module 810, a target detection and recognition module 820, and a behavior recognition module 830.

[0099] The sampling module 810 can be configured to obtain a to-be-processed video stream, sample the to-be-processed video stream, and obtain multiple frame video frame images of the to-be-processed video stream; the target detection and recognition module 820 can be configured to perform target detection and recognition on the multiple frame video frame images, and obtain image time series data of each target object and key point time series data of each target object; and the behavior recognition module 830 can be configured to determine individual abnormal behavior labels of each target object and group abnormal behavior labels of multiple target objects based on a pre-trained abnormal behavior recognition model and according to the image time series data of each target object and the key point time series data of each target object. The abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model.

[0100] In some embodiments of the present disclosure, the individual abnormal behavior recognition model comprises a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and the group abnormal behavior recognition model comprises a backbone network for group behavior recognition and a classification head for group behavior recognition. The behavior recognition module 830 can be further configured to: obtain individual action features of each target object by using the backbone network for individual behavior recognition according to the image time sequence data of each target object and the key point time sequence data of each target object; input the individual action features of each target object into the classification head for individual behavior recognition respectively, and output individual abnormal behavior labels of each target object; add the individual action features of each target object and the pedestrian box encoding of each target object respectively to obtain fusion features of each target object; input the fusion features of each target object into the backbone network for group behavior recognition to obtain group behavior features of the plurality of target objects; and input the group behavior features of the plurality of target objects into the classification head for group behavior recognition to output group abnormal behavior labels of the plurality of target objects.

[0101] In some embodiments of the present disclosure, the backbone network for individual behavior recognition comprises a first backbone network and a second backbone network, the first backbone network is a backbone network of a time domain convolution network, and the second backbone network is a backbone network of a space-time graph convolution network. The behavior recognition module 830 can be further configured to: input the image time sequence data of each target object into the first backbone network respectively to obtain image time sequence features of each target object; input the key point time sequence data of each target object into the second backbone network respectively to obtain key point time sequence features of each target object; and concatenate the image time sequence features of each target object and the key point time sequence features of each target object respectively to obtain individual action features of each target object.

[0102] In some embodiments of the present disclosure, the abnormal behavior recognition device 800 can further comprise a model training module 840, which can be configured to train the individual abnormal behavior recognition model according to the following method: obtaining image time sequence data of a sample object, key point time sequence data of the sample object, and individual action categories of the sample object; and training the backbone network for individual behavior recognition and the classification head for individual behavior recognition comprised in the individual abnormal behavior recognition model according to the image time sequence data of the sample object, the key point time sequence data of the sample object, and the individual action categories of the sample object.

[0103] In some embodiments of the present disclosure, the model training module 840 can be further configured to train the group abnormal behavior recognition model according to the following method: obtaining image time series data of a plurality of sample objects, key point time series data of the plurality of sample objects, and group action categories of the plurality of sample objects; and training a group behavior recognition backbone network and a group behavior recognition classification head included in the group abnormal behavior recognition model according to the image time series data of the plurality of sample objects, the key point time series data of the plurality of sample objects, and the group action categories of the plurality of sample objects, and using a trained individual behavior recognition backbone network.

[0104] In some embodiments of the present disclosure, the model training module 840 can be further configured to: mask reconstruct the image time series data of the sample object according to a first preset proportion of frame numbers, and mask reconstruct pixels in the image time series data of the sample object; and mask reconstruct the key point time series data of the sample object according to a second preset proportion of frame numbers, and mask reconstruct key points in the key point time series data of the sample object.

[0105] In some embodiments of the present disclosure, the target detection and recognition module 820 can be further configured to: detect and track pedestrians in a plurality of video frame images, obtain pedestrian detection boxes of each target object, and obtain image time series data of each target object; input images in the pedestrian detection boxes of each target object into a key point recognition model respectively, and obtain key point time series data of each target object.

[0106] Figure 9 is a structural block diagram of an electronic device according to an exemplary embodiment. The electronic device 900 according to this embodiment of the present application will be described below with reference to Figure 9 Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0107] As shown in Figure 9 , the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 can include, but are not limited to, the at least one processing unit 910 described above, the at least one storage unit 920 described above, a bus 930 connecting different system components including the storage unit 920 and the processing unit 910, and a display unit 940.

[0108] ​The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments according to the present application described in the above “Exemplary Method” section of the specification. Specifically, the electronic device 910 provided in the embodiments of the present disclosure can perform the following steps in the above embodiments: step S201, obtaining a to-be-processed video stream, sampling the to-be-processed video stream to obtain a plurality of video frame images of the to-be-processed video stream; step S202, performing target detection and recognition on the plurality of video frame images to obtain image time sequence data of each target object and key point time sequence data of each target object; and step S203, determining individual abnormal behavior labels of each target object and group abnormal behavior labels of a plurality of target objects based on a pre-trained abnormal behavior recognition model according to the image time sequence data of each target object and the key point time sequence data of each target object. The abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model.

[0109] The storage unit 920 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 9201 and / or a cache memory unit 9202, and can further include a read-only memory (ROM) 9203.

[0110] The storage unit 920 can further include program / utility 9204 having a set of at least one program modules 9205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, and each of these examples, or some combination thereof, can include implementation of a network environment.

[0111] The bus 930 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0112] The electronic device 900 can also communicate with one or more external devices 970 such as a keyboard or pointing device, a Bluetooth device, or a database, etc.; and / or can communicate with one or more devices that enable a user to interact with the electronic device 900; and / or can communicate with any devices (such as a router, a modem, a server, etc.) that enable the electronic device 900 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 950. Still yet, the electronic device 900 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via the network adapter 960. As depicted, the network adapter 960 communicates with the other components of the electronic device 900 via the bus 930. It should be appreciated that although the network adapter 960 is depicted as a single component, the network adapter 960 can comprise two or more components that operate together to facilitate the communication of data and / or signals between the electronic device 900 and one or more other devices.

[0113] In the example embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described methods of the present specification is also provided. In some possible implementations, various aspects of the present disclosure can also be implemented as a program product in the form of a computer readable storage medium having program code portions stored thereon, the program code portions being capable of implementing the steps described in the above "Example Methods" section of the present specification according to various example embodiments of the present disclosure when the program product is run on a terminal device.

[0114] The program product for implementing the above-described methods according to embodiments of the present disclosure can take the form of a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The program product can take any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0116] A computer readable signal medium can include a propagated data signal with computer executable program code embodied therein. The propagated data signal can take any of a variety of forms, including but not limited to radio frequency signals, light signals, infrared signals, and any suitable combination thereof. The computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program code.

[0117] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0118] Computer program code for carrying out operations for aspects of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider. The application program code can be downloaded to the user's computing device from an external computing device or server through any type of network, including a local area network, a wide area network, or the Internet using a browser or other software.

[0119] It should be noted that, although the above detailed description refers to several modules or units of the device for performing actions, this division is not mandatory. Indeed, according to embodiments of the present disclosure, features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functionalities of one module or unit described above can be further divided into several modules or units.

[0120] Moreover, although the various steps of the methods of the present disclosure are described in a particular order in the figures, this is not mandatory. Indeed, the steps can be performed in a different order, or not all of the steps shown can be performed, or a step can be split into several steps, etc.

[0121] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0122] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0123] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method for identifying abnormal behavior, characterized in that: The method comprises: Acquire a video stream to be processed, sample the video stream to be processed, and obtain multiple video frame images of the video stream to be processed; Performing target detection and recognition on the multiple video frame images to obtain image time series data of each target object and key point time series data of each target object; Based on a pre-trained abnormal behavior recognition model, individual abnormal behavior labels of each target object and group abnormal behavior labels of multiple target objects are determined according to the image time series data of each target object and the key point time series data of each target object; wherein the abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model, the individual abnormal behavior recognition model includes a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and the group abnormal behavior recognition model includes a backbone network for group behavior recognition and a classification head for group behavior recognition; Furthermore, the abnormal behavior recognition model based on pre-training determines the individual abnormal behavior label of each target object and the group abnormal behavior label of multiple target objects according to the image time series data of each target object and the key point time series data of each target object, including: According to the image time series data and key point time series data of each target object, the individual action features of each target object are obtained by using the backbone network of the individual behavior recognition; the individual action features of each target object are respectively input into the classification head of the individual behavior recognition, and the individual abnormal behavior labels of each target object are output; the individual action features of each target object and the pedestrian frame codes of each target object are respectively added to obtain the fusion features of each target object; the fusion features of each target object are input into the backbone network of the group behavior recognition to obtain the group behavior features of the multiple target objects; the group behavior features of the multiple target objects are input into the classification head of the group behavior recognition, and the group abnormal behavior labels of the multiple target objects are output.

2. The method according to claim 1, characterized in that The backbone network for individual behavior recognition includes a first backbone network and a second backbone network, wherein the first backbone network is a backbone network of a time domain convolutional network, and the second backbone network is a backbone network of a spatiotemporal graph convolutional network; The method of obtaining the individual action features of each target object by using the backbone network for individual action recognition based on the image time series data and the key point time series data of each target object includes: Inputting the image time series data of each target object into the first backbone network respectively to obtain the image time series features of each target object; Inputting the key point time series data of each target object into the second backbone network respectively to obtain the key point time series features of each target object; The image temporal features of each target object and the key point temporal features of each target object are cascaded respectively to obtain the individual action features of each target object.

3. The method according to claim 1 or 2, characterized in that The individual abnormal behavior recognition model is trained according to the following method: Acquire image time series data of a sample object, key point time series data of the sample object, and individual action categories of the sample object; The individual abnormal behavior recognition model includes a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and is trained according to the image time series data of the sample object, the key point time series data of the sample object, and the individual action category of the sample object.

4. The method according to claim 3, characterized in that The group abnormal behavior recognition model is trained according to the following method: Acquire image time series data of a plurality of sample objects, key point time series data of the plurality of sample objects, and group action categories of the plurality of sample objects; According to the image time series data of the multiple sample objects, the key point time series data of the multiple sample objects and the group action categories of the multiple sample objects, and using the backbone network for individual behavior recognition obtained through training, the group abnormal behavior recognition model includes a backbone network for group behavior recognition and a classification head for group behavior recognition.

5. The method according to claim 3, characterized in that The method further comprises: Performing mask reconstruction on the image time series data of the sample object according to a first preset ratio of frames, and performing mask reconstruction on pixels in the image time series data of the sample object; Mask reconstruction is performed on the key point time series data of the sample object according to a second preset ratio of frames, and mask reconstruction is performed on the key points in the key point time series data of the sample object.

6. The method according to claim 1, characterized in that The performing target detection and recognition on the multiple video frame images to obtain image time series data of each target object and key point time series data of each target object includes: Detecting and tracking pedestrians in the multiple video frames to obtain pedestrian detection frames of each target object and acquiring image time series data of each target object; The images in the pedestrian detection frames of the target objects are respectively input into the key point recognition model to obtain the key point time series data of the target objects.

7. An abnormal behavior recognition device, characterized in that: The device comprises: A sampling module is used to obtain a video stream to be processed, sample the video stream to be processed, and obtain multiple video frame images of the video stream to be processed; A target detection and recognition module is used to perform target detection and recognition on the multiple video frame images to obtain image time series data of each target object and key point time series data of each target object; A behavior recognition module is configured to determine, based on a pre-trained abnormal behavior recognition model and according to the image time series data and key point time series data of each target object, individual abnormal behavior labels of each target object and group abnormal behavior labels of multiple target objects; wherein the abnormal behavior recognition model includes an individual abnormal behavior recognition model and a group abnormal behavior recognition model, the individual abnormal behavior recognition model includes a backbone network for individual behavior recognition and a classification head for individual behavior recognition, and the group abnormal behavior recognition model includes a backbone network for group behavior recognition and a classification head for group behavior recognition; In addition, the behavior recognition module is further used to obtain the individual action features of each target object based on the image time series data and the key point time series data of each target object, using the backbone network of the individual behavior recognition; input the individual action features of each target object into the classification head of the individual behavior recognition, and output the individual abnormal behavior labels of each target object; add the individual action features of each target object and the pedestrian frame code of each target object respectively to obtain the fusion features of each target object; input the fusion features of each target object into the backbone network of the group behavior recognition to obtain the group behavior features of the multiple target objects; input the group behavior features of the multiple target objects into the classification head of the group behavior recognition, and output the group abnormal behavior labels of the multiple target objects.

8. An electronic device, characterized in that: include: one or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images