Abnormal action detection method and abnormal action detection model generation method

Abstract

The invention discloses an abnormal action detection method and an abnormal action detection model generation method, and belongs to the field of intelligent equipment. According to the method, the action tensor of the preset target is obtained, and the action tensor is obtained based on the image containing the preset target; inputting the action tensor to an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor; and obtaining an output result output by the abnormal action detection model based on the action tensor, and judging whether a corresponding abnormal action exists or not based on the output result. The problem that the recognition rate of a current model for intelligently detecting abnormal actions is low due to sample imbalance can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent devices, and in particular to an abnormal action detection method and a method for generating an abnormal action detection model. Background Art

[0002] Household cleaning robots are particularly popular among users because they can free users' hands and can clean and recharge autonomously. With the development of household cleaning robot products, a variety of products with different sensor configurations have emerged. Among them, cleaning robot products represented by video surveillance systems are popular among families of all ages due to people's increasing safety awareness year by year.

[0003] In order to realize the intelligence of video surveillance system, there is currently a task of intelligently detecting abnormal actions by computers. By using normal and abnormal action data at the same time to train machine learning models, a normal action and abnormal action classifier is obtained.

[0004] However, in reality, the probability of abnormal events is small, and there is a problem of difficulty in collecting data. Using data from normal and abnormal actions for model training is likely to cause sample imbalance.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide an abnormal action detection method and a method for generating an abnormal action detection model, aiming to solve the technical problem that the current intelligent abnormal action detection model uses both normal and abnormal action data for model training, which easily causes sample imbalance and leads to low abnormal action recognition rate.

[0007] To achieve the above object, the present application provides an abnormal action detection method, which comprises the following steps:

[0008] Acquire an action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target;

[0009] Inputting the action tensor into an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor;

[0010] The output result of the abnormal action detection model based on the action tensor output is obtained, and whether there is a corresponding abnormal action based on the output result is determined.

[0011] Optionally, the abnormal action detection model includes multiple submodules, and different submodules have different recognition principles for the action tensor. The step of obtaining the output result of the abnormal action detection model based on the action tensor output includes:

[0012] Obtain sub-results obtained by each of the sub-modules identifying the action tensor based on its own identification principle;

[0013] Based on the preset calculation method and the sub-results, an output result is calculated.

[0014] Optionally, the recognition principle of the action tensor by the submodule includes one or more combinations of the following methods:

[0015] Identify the real action of each frame in the action tensor, and predict the predicted action of the next frame corresponding to the real action of the current frame in sequence; output the sub-result corresponding to the action tensor based on the similarity between the predicted action and the real action of the next frame;

[0016] Identify key points of preset frames in the action tensor, wherein the preset frames are at least two sequential frames, and the key points are used to characterize the action posture corresponding to the preset frames; determine the change amplitude of the action posture corresponding to the preset frames based on the key points, and output the sub-result corresponding to the action tensor based on the change amplitude;

[0017] Acquire a feature vector corresponding to the action tensor, and calculate the similarity between the feature vector and a pre-stored memory module, wherein the pre-stored memory module is used to characterize the characteristics of a preset normal action; based on the similarity, output a sub-result corresponding to the action tensor.

[0018] Optionally, the step of obtaining the action tensor of the preset target includes:

[0019] Based on the received task instruction, obtaining a video stream corresponding to the task instruction, and extracting an image of a preset target corresponding to the video stream;

[0020] Based on the image, an action tensor of the preset target is generated.

[0021] Optionally, the step of extracting an image of a preset target corresponding to the video stream includes:

[0022] Extracting a key frame in which a preset target exists in the video stream, and determining the coordinates of the preset target in the key frame;

[0023] Based on the coordinates, construct a detection frame corresponding to the preset target, and extract an image corresponding to the preset target based on the detection frame;

[0024] The step of generating the action tensor of the preset target based on the image comprises:

[0025] Obtain the number of image channels of the image, the size information of the image, and the length of the sequence frames corresponding to the image;

[0026] Based on the number of image channels, the size information and the length of the sequence frames, an action tensor of the preset target is generated.

[0027] Optionally, the step of judging whether the preset target has an abnormal action based on the output result of the abnormal action detection model includes:

[0028] If there is at least one preset target in the video stream, taking the average or minimum value of the output result of the abnormal action detection model to obtain a frame-level output result, wherein the abnormal action detection model outputs at least one output result based on the at least one preset target;

[0029] Calling a first-order digital low-pass filter to perform smoothing filtering on multiple output results of the abnormal action detection model based on continuous frames;

[0030] By judging whether the number of the plurality of output results after smoothing filtering that is lower than the preset score threshold reaches a preset number, it is determined whether the preset target has an abnormal action.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides an abnormal action detection device, the device comprising:

[0032] An acquisition module, used for acquiring an action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target;

[0033] An input module, used for inputting the action tensor into an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor;

[0034] A judgment module is used to obtain an output result of the abnormal action detection model based on the action tensor output, and judge whether there is a corresponding abnormal action based on the output result.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides an abnormal action detection device, which includes: a memory, a processor, and an abnormal action detection program stored on the memory and executable on the processor, wherein the abnormal action detection program is configured to implement the steps of the above-mentioned abnormal action detection method.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a method for generating an abnormal action detection model, and the method for generating an abnormal action detection model includes:

[0037] Acquire training samples and supervision images, wherein the training samples are obtained based on images annotated with preset normal actions and the model parameters, and the supervision images include the images annotated with preset normal actions;

[0038] Inputting the training sample into an initial detection model to obtain an output result of the initial detection model;

[0039] Determine the loss parameters of the output result and the supervision image based on the loss function;

[0040] The initial detection model is feedback updated according to the loss parameter until it is trained into an abnormal action detection model.

[0041] Optionally, the training sample includes a disordered sequence obtained by scrambling the image labeled with the preset normal action in the time dimension, and inputting the training sample into the initial detection model to obtain the output result of the initial detection model includes:

[0042] The scrambled sequence is input into the initial detection model as the training sample, wherein after receiving the scrambled sequence, the initial detection model identifies the real action of each frame in the scrambled sequence, and sequentially predicts the predicted action of the next frame corresponding to the real action of the current frame, to obtain the output result of the initial detection model.

[0043] Optionally, the step of inputting the training sample into an initial detection model to obtain an output result of the initial detection model further includes:

[0044] The training sample is input into an initial detection model, wherein after receiving the training sample, the initial detection model identifies the key points of a preset frame in the training sample to obtain an output result of the initial detection model, wherein the preset frame is at least two frames in sequence, and the key points are used to characterize the action posture corresponding to the preset frame.

[0045] Optionally, the step of inputting the training sample into an initial detection model to obtain an output result of the initial detection model further includes:

[0046] Input the training sample into an initial detection model, wherein after receiving the training sample, the initial detection model identifies a feature vector of a preset frame in the training sample to obtain an output result of the initial detection model, wherein the output result includes an initial feature vector having the highest similarity to the feature vector in a pre-stored memory module;

[0047] Feedback updating the initial detection model according to the loss parameter until training the abnormal action detection model includes:

[0048] A preset memory module is updated based on the feature vector and the initial feature vector until the preset memory module is no longer updated.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a device for generating an abnormal action model, the device comprising:

[0050] An acquisition module, used to acquire training samples and supervisory images, wherein the training samples are obtained based on images annotated with preset normal actions and the model parameters, and the supervisory images include the images annotated with preset normal actions;

[0051] An input module, used for inputting the training sample into an initial detection model to obtain an output result of the initial detection model;

[0052] A determination module, used to determine the loss parameters of the output result and the supervision image based on the loss function;

[0053] The training module is used to perform feedback update on the initial detection model according to the loss parameter until it is trained into an abnormal action detection model.

[0054] In addition, to achieve the above-mentioned purpose, the present application also provides a device for generating an abnormal action detection model, the device comprising: a memory, a processor, and an abnormal action detection program stored on the memory and executable on the processor, the abnormal action detection program being configured to implement the steps of the above-mentioned abnormal action detection method.

[0055] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, on which is stored an abnormal action detection program and / or a program for generating an abnormal action detection model. When the abnormal action detection program is executed by a processor, the steps of the above-mentioned abnormal action detection method are implemented. When the abnormal action detection model generation program is executed by a processor, the steps of the above-mentioned abnormal action detection model generation method are implemented.

[0056] In the present application, in order to solve the technical problem that the current intelligent abnormal action detection model is prone to sample imbalance due to the use of normal and abnormal action data for model training at the same time, resulting in a low abnormal action recognition rate, the present application obtains the action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target; inputs the action tensor to the abnormal action detection model, the abnormal action detection model is pre-trained with a preset normal action tensor; obtains the output result of the abnormal action detection model based on the output of the action tensor, and determines whether there is a corresponding abnormal action based on the output result. Since the abnormal action detection model used in the present application is only pre-trained with a preset normal action tensor, and does not need to use abnormal actions as training samples, it can effectively solve the problem of low recognition rate of the current intelligent abnormal action detection model due to sample imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a structural diagram of an abnormal action detection device in a hardware operating environment involved in an embodiment of the present application;

[0058] Figure 2 This is a flow chart of the first embodiment of the abnormal action detection method of the present application;

[0059] Figure 3 This is a flow chart of the second embodiment of the abnormal action detection method of the present application;

[0060] Figure 4 This is a schematic diagram of the model structure of the second embodiment of the abnormal action detection method of the present application;

[0061] Figure 5 This is a flow chart of the third embodiment of the abnormal action detection method of the present application;

[0062] Figure 6 This is a functional module diagram of the first embodiment of the abnormal action detection device of the present application;

[0063] Figure 7 It is a structural schematic diagram of a device for generating an abnormal action detection model of a hardware operating environment involved in an embodiment of the present application;

[0064] Figure 8 This is a flow chart of a first embodiment of a method for generating an abnormal action detection model of the present application;

[0065] Fig. 9 This is a functional module diagram of the first embodiment of the device for generating an abnormal action detection model of the present application.

[0066] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of an abnormal action detection device in a hardware operating environment involved in an embodiment of the present application.

[0069] like Figure 1 As shown, the abnormal action detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0070] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the abnormal action detection device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0071] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and an abnormal action detection program.

[0072] exist Figure 1In the abnormal action detection device shown, the network interface 1004 is mainly used for data communication with other devices, wherein the other devices may be but are not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the abnormal action detection device of the present application can be set in the abnormal action detection device, and the abnormal action detection device calls the abnormal action detection program stored in the memory 1005 through the processor 1001, and executes the abnormal action detection method provided in the embodiment of the present application.

[0073] The present application provides a method for detecting abnormal behavior. Figure 2 , Figure 2 This is a flow chart of a first embodiment of an abnormal motion detection method of the present application, and the method is described by taking the application of the method in an abnormal motion detection device as an example.

[0074] In this embodiment, the abnormal action detection method includes:

[0075] Step S10: Acquire an action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target;

[0076] Furthermore, the step S10 includes:

[0077] Step S11: based on the received task instruction, obtaining a video stream corresponding to the task instruction, and extracting an image of a preset target corresponding to the video stream;

[0078] Optionally, the abnormal motion detection method provided in the present application can be applied to a cleaning device, which can be a self-moving robot or a self-moving robot and a base station. The self-moving robot can specifically be an automatic sweeping robot, an automatic mopping robot, etc. The execution subject of the present application can be a host, a server, or a base station. Therefore, the execution subject can be a cleaning device or a server.

[0079] The self-moving robot can be provided with a navigation system, which can detect and determine the working area by itself and determine the specific position of the cleaning robot in the working area. The cleaning robot can be provided with various sensors, such as ranging sensors such as laser radar, TOF sensors, visual sensors such as cameras, etc., for real-time detection of the ground debris conditions in the working area. In other embodiments, the automatic cleaning device can be provided with a touch-sensitive display to receive operation instructions input by the user. The automatic cleaning device can also be provided with wireless communication modules such as WIFI modules and Bluetooth modules to connect to the smart terminal, and receive the operation instructions transmitted by the user using the smart terminal through the wireless communication module.

[0080] In this embodiment, the self-propelled robot is equipped with a visual sensor, a SLAM (Simultaneous Localization And Mapping) module, a mobile navigation module, and other modules. The visual sensor is used to obtain a real-time video stream. The SLAM module is used to build a map and estimate the position and posture, which is used to control the movement of the cleaning device.

[0081] As an implementation method, the video stream acquired by the sensor is transmitted to the host, and the host processes the video stream to obtain a processing result.

[0082] As another implementation, the video stream acquired by the sensor is transmitted to a server, the server processes the video stream to obtain a processing result, and the host obtains the result from the server for positioning, identification, etc.

[0083] As another implementation, the video stream acquired by the sensor is transmitted to a base station, the base station processes the image, obtains a processing result, and instructs the host to perform corresponding movements.

[0084] If the cleaning device receives a task instruction, it will go to the designated indoor location corresponding to the task instruction based on the indoor map constructed by the SLAM module and the mobile navigation module, and obtain a real-time monitoring video stream through the visual sensor.

[0085] As an implementation method, the user can issue a task instruction to the cleaning robot to go to the indoor restaurant area to perform an abnormal motion detection task during lunch time, or to go to the entrance and exit to perform an abnormal motion detection task during off-duty time.

[0086] In addition, real-time monitoring video streams are obtained through visual sensors, in which the instructions issued by users can be executed in real time or on a scheduled basis.

[0087] In this embodiment, the video stream is composed of continuous images taken by the cleaning robot, and key frames with preset targets are obtained by real-time monitoring of the video stream.

[0088] Optionally, continuous images are input into a target detection model to detect a preset target. The target detection model may adopt currently popular deep learning models such as Faster-RCNN and YOLOv5. When the target detection model detects a preset target, data preprocessing is performed. First, a detection frame corresponding to the preset target is constructed according to the coordinates of the preset target in the key frame, and the images of the same preset target in the t+1 frames before and after the preset frame are cropped according to the detection frame, and then the cropped t+1 frame image is subjected to background removal, and only the preset target information is retained.

[0089] In addition, since the goal of the present application is to detect abnormal actions, images in which the preset target is not detected are not processed. That is, if the target detection model does not detect the preset target, the next frame of image detection continues until the preset target appears.

[0090] Optionally, assuming that the target detection model detects a preset target for the i-th frame image, the coordinates x1, y1, x2, y2 of each preset target frame will be output, where (x1, y1) is the coordinate of the upper left corner of the target detection frame in the image, and (x2, y2) is the coordinate of the lower right corner of the target detection frame in the image. Since each complete action will be captured by multiple consecutive frames of images in the video stream, in order to better analyze the action features, this solution uses continuous multiple frames of image information as model input. For example, the t frames before and after the i-th frame are selected as a sample, that is, the {it, ..., i-1, i, i+1, ..., i+t} frames (a total of 2t+1 frames) images form an abnormal action detection sequence (t can take values ​​of 2, 3, 4, etc.). Since the motion changes in each frame are relatively small, this scheme assumes that the detection frame position of the same preset target in the previous and next t frames of images is relatively close. Therefore, the length and width of the preset target frame of the i-th frame are slightly expanded by d pixels on one side to construct the detection frame, and the previous and next t frames of images are cropped according to the position of the detection frame. The advantage of this is that it saves the computing resources required for the target tracking algorithm of the preset target and improves the running frame rate of this application.

[0091] Specifically, the 2t+1 frames of images in each abnormal action detection sequence are cropped, and each image is cropped according to the preset target frame coordinates x1-d, y1-d, x2+d, y2+d of the i-th frame. The height of the cropped image is (y2-y1+2d), and the width is (x2-x1+2d), where d is the size of the preset target frame. If the expanded preset target frame exceeds the size of the original image, d=0, that is, the preset target frame is not expanded.

[0092] And since the size of each preset target box is different, this solution can use bilinear interpolation or cubic interpolation to resize the cropped image to 96 pixels × 96 pixels to unify the input image size.

[0093] Optionally, since the object of this scheme is abnormal action, the background that is completely unrelated to the preset target has no effect on this scheme, and even the background information may interfere with the model detection. Therefore, the background information in the image is masked with a gray pixel mask, that is, the RGB channel information other than the preset target is shielded. The method of background removal can specifically adopt the following method, such as using the background subtraction method to subtract the RGB image containing the preset target from the RGB image without the preset target. The pixel points with a difference value less than the threshold can be judged as the background (because the pixels of the background have almost no change, while the pixels of the area where the preset target appears are huge). For such background areas, gray pixel values ​​(114, 114, 114) are used to replace them; you can also use an instance segmentation network (such as OrienMask) to segment the preset target area, and replace the background area with gray pixel values.

[0094] In this embodiment, data preprocessing is performed on the real-time video frames in which new people are detected through image cropping and background removal, thereby avoiding the negative impact of background information unrelated to the human body on abnormal action detection, saving the computing resources required for the target tracking algorithm of the preset target, and being able to effectively improve the operating frame rate when implementing the technical solution of this application.

[0095] Step S12: Based on the image, generate an action tensor of the preset target.

[0096] Furthermore, the step S12 includes:

[0097] Step S13: obtaining the number of image channels of the image, the size information of the image, and the length of the sequence frames corresponding to the image;

[0098] Step S14: Based on the number of image channels, the size information and the length of the sequence frames, an action tensor of the preset target is generated.

[0099] Optionally, the number of image channels is the number of RGB channels, and the size information includes the height and width of the image. Based on the batch processing quantity of the abnormal action detection model, the number of RGB channels, the height, the width, and the sequence frame length, a corresponding action tensor is generated.

[0100] In this embodiment, the shape of the action tensor is (B, C, T, H, W), where B is the batch size of the neural network model inference; C is the number of channels of the RGB image; T is the length of the sequence frame number of the action detection sample; H and W are the height and width of each frame image.

[0101] Step S30: inputting the action tensor into an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor;

[0102] In this embodiment, the samples pre-trained and learned by the abnormal action detection model are normal action tensors. The model structure consists of Backbone (backbone network) and Task Head, and Task Head can be composed of a fully connected layer and a non-linear activation function. Backbone consists of a traditional series of operators: 3D convolution, IN layer, non-linear activation function ReLU stacking, and 3 3D maximum pooling layers are interspersed in Backbone to extract features, adjust channels and downsample the input tensor 3 times. For example, if the model inputs a tensor of shape (B, C, T, H, W), after passing through the Backbone module of the model, a feature map downsampled 8 times will be obtained, with a shape of (BS, Cout, 1, H / 8, W / 8), where Cout is the number of output channels adjusted by Backbone. Head is a network that obtains the output content of the network and makes predictions using the previously extracted features.

[0103] Step S40: Obtain the output result of the abnormal action detection model based on the action tensor output, and determine whether there is a corresponding abnormal action based on the output result.

[0104] Optionally, the model training uses normal action data as the training data of the model, such as walking, sitting, standing, etc., and the actions not included in the training data are classified as abnormal actions. When the abnormal action detection model infers action data (abnormal action data) that has not been seen during its model training, the performance of the model will drop sharply, and the predicted output of the model will be very different from the predicted output when inferring normal actions, thereby detecting abnormal actions. This application designs and integrates multiple anomaly detection tasks according to actual application needs to obtain an abnormal action detection model, and determines whether the input action sequence is an abnormal action based on the normal score output of all tasks during reasoning.

[0105] In this embodiment, in order to solve the technical problem that the current intelligent abnormal action detection model is prone to sample imbalance due to the use of normal and abnormal action data for model training at the same time, resulting in a low abnormal action recognition rate, the present application obtains the action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target; inputs the action tensor to the abnormal action detection model, the abnormal action detection model is pre-trained by a preset normal action tensor; obtains the output result of the abnormal action detection model based on the action tensor output, and determines whether there is a corresponding abnormal action based on the output result. It can effectively solve the problem of low recognition rate caused by sample imbalance in the current intelligent abnormal action detection model.

[0106] Based on the above embodiments, refer to Figure 3 and Figure 4, providing a second embodiment of the present application, further, the abnormal action detection model includes a plurality of sub-modules, different sub-modules have different recognition principles for the action tensor, and the step of obtaining the output result of the abnormal action detection model based on the output of the action tensor in step S40 includes:

[0107] Step S41: obtaining sub-results obtained by each of the sub-modules identifying the action tensor based on its own identification principle;

[0108] Optionally, the submodules of the abnormal multi-action detection model include abnormal action detection tasks that disrupt the time series and / or abnormal action detection tasks based on key point reconstruction and / or abnormal action detection tasks based on memory modules. Each submodule is divided into model training and model reasoning, refer to Figure 4 , Figure 4 This is a schematic diagram of the model structure of the second embodiment of the abnormal action detection method of the present application.

[0109] Optionally, during the training of the abnormal action detection task of disrupting the time series, the frame order of the normal action sequence samples is disrupted to obtain a random sequence, and the random sequence is used as input. The predicted value output by the task branch is the frame order after the random sequence is rearranged, and the true value is the correct frame order of the sample sequence. The model learns the continuous features of normal actions by minimizing the model prediction value and the true value. During reasoning, if the input is a trained normal action sample, the model can predict the frame order very well; if the input is an untrained abnormal action sample, it is very difficult for the neural network model to correctly predict the unknown action sequence, that is, the similarity between the predicted value and the true value output by the abnormal data model is very low, and the output sub-result can be the minimum value (similarity) of the probability of the predicted value at the true value position. When the probability is higher, it indicates that the network predicts the correct score higher; when the probability is lower, it indicates that the action sequence is more likely to be an abnormal action.

[0110] Optionally, for many preset normal actions such as standing, sitting, and walking, the amplitude of the action between consecutive frames of the abnormal action detection task based on key point reconstruction varies little; while for abnormal actions such as falling and fighting, the amplitude of the action between consecutive frames varies greatly. For unknown abnormal actions that change rapidly, it is difficult for the model to capture the key point feature information, making it difficult to reconstruct the key points. Therefore, in response to these abnormal action change amplitudes, the present application designs an abnormal action detection task based on key point reconstruction, that is, abnormal action detection and judgment is performed by predicting the key point positions of the middle frames of the action sequence samples. Specifically, during model training, the normal sequence samples are input into the model, and the key point positions of the second frame image in the sequence and the key point positions of the penultimate frame are output in the task branch, and the true value output by the task branch is the true value of the key points of these two frames, and the objective function is the MSE (Mean-Squared Loss, loss function) between the predicted value and the true value of the task branch. During reasoning, when the model inputs the preset target's limbs undergoing rapid abnormal movements, the neural network model's prediction of the key points of the task branch for such rapid and unknown abnormal movement data will have a larger variation range compared to the movement posture corresponding to the true value, and the output sub-result is the cosine similarity between the predicted value and the true value.

[0111] Optionally, the abnormal action detection task based on the memory module outputs a feature vector p of length 512, which is used to describe the features learned by the neural network for normal actions, and uses M memory modules qm of feature vectors of length 512 to store the features of normal actions. When the model is training, a normal action sequence is input and a feature vector of length 512 is output. The cosine similarity and softmax operation are performed between the feature vector and the m memory modules to obtain the similarity between the feature vector p and the m memory modules qm, that is, the weight Wm (m = 0, ..., M-1). The higher the cosine similarity between the feature vector and a certain memory module, the larger the corresponding weight Wm. Each memory module is updated by calculating the weighted average: qm = Wm·p.

[0112] In the inference phase, the feature vector calculated by the unknown action is calculated with the cosine similarity calculated by the M memory modules, and the output sub-result is the minimum value of the cosine similarity. The lower the minimum value of the cosine similarity, the greater the possibility that the unknown action is an abnormal action.

[0113] In this embodiment, the output of each sub-module may not be a numerical value calculated by the model based on the input action tensor, but a direct judgment result, whether there is an abnormal action.

[0114] Step S42: Based on the preset calculation method and the sub-results, an output result is calculated.

[0115] Optionally, the preset calculation method can be to take the average or minimum value according to the output sub-results of each sub-module. Then compare it with the preset critical value to finally determine whether there is an output result of abnormal action. The preset critical value is a value set based on the output result of the preset normal action output during model training to determine the abnormality. When the output result is that there is an abnormal action, an alarm measure is taken, which can be to sound an alarm or turn on a warning light or notify an administrator, etc.

[0116] In this embodiment, compared with a single detection model that is easily restricted by scenes, the abnormal action detection model designed by this application makes joint decisions in a multi-task detection integrated manner. The multi-task uses the RGB image of the action sequence and the key points of the human skeleton to jointly train the network, so that the network learns more effective action features and can effectively improve the recognition rate. And the submodules in the abnormal action detection model of this application only need to use normal data to train the model, which can effectively avoid the problem of data collection difficulties faced by the existing technology of collecting abnormal data for training.

[0117] Based on the above embodiments, refer to Figure 5 , Figure 5 The third embodiment of the abnormal action detection method of the present application is provided as a flow chart of the third embodiment of the present application, wherein the step S40 further includes:

[0118] Step S50: if there is at least one preset target in the video stream, taking the average or minimum value of the output result of the abnormal action detection model to obtain a frame-level output result, wherein the abnormal action detection model outputs at least one output result based on the at least one preset target;

[0119] Step S51: calling a first-order digital low-pass filter to perform smoothing filtering on multiple output results of the abnormal action detection model based on continuous frames;

[0120] Step S52: determining whether there is any abnormal action of the preset target by judging whether the number of the multiple output results after smoothing filtering that is lower than the preset score threshold reaches a preset number.

[0121] Optionally, if there is at least one preset target in the video stream, the output result of the abnormal action detection model is averaged or minimized to obtain a frame-level output result, wherein the abnormal action detection model outputs at least one output result based on the at least one preset target.

[0122] Since the batch processing times of the abnormal action detection model are limited, in the actual application process, to determine whether there is an abnormal action, real-time and continuous detection will be performed based on the video stream, and the results will be continuously output.

[0123] Optionally, a first-order digital low-pass filter is called to smooth the above-mentioned multiple continuous output results to reduce interference and make the results more accurate.

[0124] Optionally, the preset score threshold is a numerical value for determining abnormalities that is set based on the output results of the preset normal action output during model training. By judging whether the number of multiple output results that have undergone smoothing filtering and are lower than the preset score threshold reaches a preset number, it is determined whether the preset target has an abnormal action.

[0125] For example, if the output results of the continuous output are: abnormal action exists, abnormal action does not exist, abnormal action exists, abnormal action exists, abnormal action exists, and abnormal action exists. With 5 output results as a batch and the preset number is 4, the final judgment is that abnormal action exists.

[0126] In this embodiment, a solution for detecting abnormal actions is provided for scenarios where multiple preset targets exist, and the introduction of a digital filter and a determination mechanism based on multiple output results can effectively improve detection accuracy.

[0127] In addition, the present application also provides an abnormal action detection device, referring to Figure 6 , Figure 6 This is a functional module diagram of the first embodiment of the abnormal action detection device of the present application. The abnormal action detection device comprises:

[0128] An acquisition module 10, configured to acquire an action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target;

[0129] An input module 20, used for inputting the action tensor into an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor;

[0130] The judgment module 30 is used to obtain the output result of the abnormal action detection model based on the action tensor output, and judge whether there is a corresponding abnormal action based on the output result.

[0131] Among them, the specific embodiments executed by each module in the abnormal action detection device of the present application are basically the same as the embodiments of the above-mentioned abnormal action detection method, and will not be repeated here.

[0132] In addition, refer to Figure 7 , Figure 7 A schematic diagram of the structure of a device for generating a model for generating an abnormal action detection model for a hardware operating environment involved in an embodiment of the present application.

[0133] like Figure 7As shown, the generation device of the abnormal action detection model may include: a processor 2001, such as a central processing unit (CPU), a communication bus 2002, a user interface 2003, a network interface 2004, and a memory 2005. Among them, the communication bus 2002 is used to realize the connection and communication between these components. The user interface 2003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 2003 may also include a standard wired interface and a wireless interface. The network interface 2004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 2005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 2005 may also be a storage device independent of the aforementioned processor 2001.

[0134] Those skilled in the art will understand that Figure 7 The structure shown in does not constitute a limitation on the device for generating an abnormal action detection model, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0135] like Figure 7 As shown, the memory 2005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a program for generating an abnormal action detection model.

[0136] exist Figure 7 In the generation device of the abnormal action detection model shown, the network interface 2004 is mainly used for data communication with other devices, wherein the other devices may be but are not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices; the user interface 2003 is mainly used for data interaction with the user; the processor 2001 and the memory 2005 in the generation device of the abnormal action detection model of the present application can be set in the generation device of the abnormal action detection model, and the generation device of the abnormal action detection model calls the generation program of the abnormal action detection model stored in the memory 2005 through the processor 2001, and executes the generation method of the abnormal action detection model provided in the embodiment of the present application.

[0137] The present application embodiment provides a method for generating an abnormal action detection model, referring to Figure 2 , Figure 2This is a flow chart of a first embodiment of a method for generating an abnormal action detection model of the present application, and is described by taking the method applied to a device for generating an abnormal action detection model as an example.

[0138] In this embodiment, refer to Figure 8 , Figure 8 The flowchart of the first embodiment of the method for generating an abnormal action detection model of the present application is shown in FIG. The method for generating an abnormal action detection model includes:

[0139] Step S60: Acquire training samples and supervisory images, wherein the training samples are obtained based on images annotated with preset normal actions and the model parameters, and the supervisory images include the images annotated with preset normal actions;

[0140] Step S61: inputting the training sample into the initial detection model to obtain the output result of the initial detection model;

[0141] Step S62: determining a loss parameter of the output result and the supervision image based on a loss function;

[0142] Step S63: Feedback update is performed on the initial detection model according to the loss parameter until the abnormal action detection model is trained.

[0143] Optionally, the executor of the abnormal action detection model training can be a cleaning device, or other cleaning devices different from the cleaning device, or a server; when using the model, the corresponding model can be retrieved from the server for processing; the model can also be loaded locally for processing.

[0144] Optionally, the present application uses a supervised learning method to train an abnormal action detection model. Supervised learning refers to the process of adjusting the parameters of a classifier using a set of samples of known categories (supervised images) to achieve the required performance, also known as supervised training or teacher learning. Supervised learning is a machine learning task that infers a function from labeled training data. The supervised learning algorithm analyzes the training samples and generates an inferred function that can be used to map out new instances.

[0145] Optionally, the abnormal action detection model includes an abnormal action detection task that disrupts the time series. The purpose of this task is to let the model learn the feature connection of continuous frames of normal actions, while for unknown actions (abnormal actions), it is difficult for the model to correctly predict the normal order of the action sequence.

[0146] The training process of this task is as follows: the images labeled with preset normal actions are shuffled in the time dimension to obtain a shuffled sequence, and the shuffled sequence is used as a training sample as the input of the initial detection model. After receiving the shuffled sequence, the initial detection model identifies the real action of each frame in the shuffled sequence, and predicts and outputs the predicted action of the next frame corresponding to the real action of the current frame in sequence, determines the loss parameter of the output result and the supervision image based on the loss function, and feeds back and updates the initial detection model according to the loss parameter until it is trained into an abnormal action detection model.

[0147] For example, the initial detection model inputs a tensor of shape (B, C, T, H, W), where T is the length of the sequence frame number of the action detection sample. Taking T=7 as an example, the normal frame order is [i-3, i-2, i-1, i, i+1, i+2, i+3]; now it is randomly shuffled in the time dimension, for example, the action sequence of [i+2, i, i-1, i+3, i+1, i-2, i-3] is obtained. The action sequence is input into the model, and the task branch will predict the position of the shuffled sequence in the correct sequence. For example, the position of the above shuffled sequence in the correct sequence is [5, 3, 2, 6, 4, 1, 0]. Therefore, the model can learn the continuous features of normal actions by minimizing the model prediction value and the true value. When inferring abnormal actions, the model has never learned the continuous features of unknown (abnormal) actions, so the predicted value of abnormal actions will be very different from the true value.

[0148] Optionally, the abnormal action detection model includes an abnormal action detection task based on key point reconstruction, the purpose of which is to allow the model to learn how to reconstruct key points of normal actions; when rapidly changing unknown abnormal actions are input, it is difficult for the model to capture effective key point feature information, resulting in a large reconstruction error.

[0149] The training process of this task: the initial detection model inputs a tensor of shape (B, C, T, H, W), where T is the length of the sequence frame number of the action detection sample. Taking T=7 as an example, it is the [i-3, i-2, i-1, i, i+1, i+2, i+3]th frame. After receiving the training sample, the initial detection model predicts the key points of the preset frame in the training sample and outputs it. The preset frame is at least two frames before and after the sequential frame (i-2, i+2 frame), and the key point is used to characterize the action posture corresponding to the preset frame. The prediction vector corresponding to the predicted key point is used as the output of the initial detection model; based on the loss function, the loss parameter corresponding to the prediction vector and the true value vector of the supervision image is determined, and the initial detection model is feedback updated according to the loss parameter until the loss parameter no longer changes and tends to the minimum value.

[0150] Specifically, the key point position information of the preset target can be given by inference from a pre-trained human key point model. For example, the Yolo-Pose model predicts the coordinates (x, y) and confidence conf of the 17 key points of the human body in the image based on the MS-COCO dataset, that is, a true value vector with a length of 51 is predicted for a preset target in a picture. Since this task predicts the key point position information of the preset target in the i-2 and i+2 frames, the task branch needs to give a prediction vector with a length of 102, and the network is trained by minimizing the MSE function of the prediction vector and the true value vector.

[0151] Optionally, the abnormal action detection model includes an abnormal action detection task based on a memory module, which uses M memory modules to store the feature vector of normal action. When the task detects the i-th frame of the video stream, the preset target is detected. The initial detection model inputs a tensor of shape (B, C, T, H, W), where T is the length of the sequence frame number of the action detection sample. After receiving the training sample, the initial detection model identifies the feature vector of the preset frame in the training sample and generates annotation information, the annotation information includes the initial feature vector with the highest similarity to the feature vector in the pre-stored memory module; the annotation information is used as the output result of the initial detection model; the preset memory module is updated based on the feature vector and the initial feature vector; the above steps are repeated until the preset memory module is no longer updated, and the initial detection model is trained into an abnormal action detection model.

[0152] In this embodiment, compared with the prior art, abnormal data needs to be used as training data. In reality, the probability of abnormal events is small, and the difficulty of collecting data is much greater than that of collecting normal data. Using both normal and abnormal action data for model training is likely to cause sample imbalance. In this embodiment, only normal data is needed to train the model.

[0153] At the same time, compared with the existing technology that only uses a single task based on key point reconstruction for abnormal action detection, it relies too much on the performance of the human skeleton key point model, resulting in great limitations in application for some scenes, such as occlusion, missing or predicted position errors of human skeleton key points. This application uses the RGB image of the action sequence and the human skeleton key points to jointly train the network, so that the network learns more effective action features and improves the recognition rate of abnormal actions.

[0154] Finally, this application can use a multi-task detection integration approach to make joint decisions, further improving the recognition rate.

[0155] In addition, the present application also provides a device for generating an abnormal action detection model, referring to Fig. 9 , Fig. 9This is a functional module diagram of the first embodiment of the device for generating an abnormal action detection model of the present application. The device for generating an abnormal action detection model comprises:

[0156] An acquisition module 100 is used to acquire training samples and supervisory images, wherein the training samples are obtained based on images annotated with preset normal actions and the model parameters, and the supervisory images include the images annotated with preset normal actions;

[0157] An input module 200 is used to input the training sample into an initial detection model to obtain an output result of the initial detection model;

[0158] A determination module 300, configured to determine a loss parameter of the output result and the supervision image based on a loss function;

[0159] The training module 400 is used to perform feedback update on the initial detection model according to the loss parameter until it is trained into an abnormal action detection model.

[0160] Among them, the specific embodiments executed by each module in the generation device of the abnormal action detection model of the present application are basically the same as the embodiments of the generation method of the abnormal action detection model mentioned above, and will not be repeated here.

[0161] In addition, an embodiment of the present application also provides a storage medium.

[0162] The storage medium of the present application stores an abnormal action detection program and / or a program for generating an abnormal action detection model. When the abnormal action detection program is executed by a processor, the steps of the abnormal action detection method described above are implemented. When the abnormal action detection model generation program is executed by a processor, the steps of the abnormal action detection model generation method described above are implemented.

[0163] Among them, the specific embodiments of the abnormal action detection program stored in the storage medium of the present application being executed by the processor are basically the same as the embodiments of the above-mentioned abnormal action detection method, and the specific embodiments of the abnormal action detection model generation program being executed by the processor are basically the same as the embodiments of the above-mentioned abnormal action detection model generation method, which will not be repeated here.

[0164] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, the elements defined by the sentence "including an abnormal action detection" do not exclude the existence of other identical elements in the process, method, article or system including the element.

[0165] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting abnormal motion, characterized in that: The abnormal action detection method comprises the following steps: Acquire an action tensor of a preset target, wherein the action tensor is obtained based on an image containing the preset target; Inputting the action tensor into an abnormal action detection model, wherein the abnormal action detection model is pre-trained by a preset normal action tensor; The output result of the abnormal action detection model based on the action tensor output is obtained, and whether there is a corresponding abnormal action based on the output result is determined.

2. The abnormal motion detection method according to claim 1, characterized in that: The abnormal action detection model includes a plurality of submodules, and different submodules have different recognition principles for the action tensor. The step of obtaining the output result of the abnormal action detection model based on the output of the action tensor includes: Obtain sub-results obtained by each of the sub-modules identifying the action tensor based on its own identification principle; Based on the preset calculation method and the sub-results, an output result is calculated.

3. The abnormal motion detection method according to claim 2, characterized in that: The recognition principle of the action tensor by the submodule includes one or more combinations of the following methods: Identify the real action of each frame in the action tensor, and predict the predicted action of the next frame corresponding to the real action of the current frame in sequence; output the sub-result corresponding to the action tensor based on the similarity between the predicted action and the real action of the next frame; Identify key points of preset frames in the action tensor, wherein the preset frames are at least two sequential frames, and the key points are used to characterize the action posture corresponding to the preset frames; determine the change amplitude of the action posture corresponding to the preset frames based on the key points, and output the sub-result corresponding to the action tensor based on the change amplitude; Acquire a feature vector corresponding to the action tensor, and calculate the similarity between the feature vector and a pre-stored memory module, wherein the pre-stored memory module is used to characterize the characteristics of a preset normal action; based on the similarity, output a sub-result corresponding to the action tensor.

4. The abnormal motion detection method according to claim 1, characterized in that: The step of obtaining the action tensor of the preset target includes: Based on the received task instruction, obtaining a video stream corresponding to the task instruction, and extracting an image of a preset target corresponding to the video stream; Based on the image, an action tensor of the preset target is generated.

5. The abnormal motion detection method according to claim 4, characterized in that: The step of extracting an image of a preset target corresponding to the video stream comprises: Extracting a key frame in which a preset target exists in the video stream, and determining the coordinates of the preset target in the key frame; Based on the coordinates, construct a detection frame corresponding to the preset target, and extract an image corresponding to the preset target based on the detection frame; The step of generating the action tensor of the preset target based on the image comprises: Obtain the number of image channels of the image, the size information of the image, and the length of the sequence frames corresponding to the image; Based on the number of image channels, the size information and the length of the sequence frames, an action tensor of the preset target is generated.

6. The abnormal motion detection method according to claim 4, characterized in that: The step of judging whether the preset target has an abnormal action based on the output result of the abnormal action detection model comprises: If there is at least one preset target in the video stream, taking the average or minimum value of the output result of the abnormal action detection model to obtain a frame-level output result, wherein the abnormal action detection model outputs at least one output result based on the at least one preset target; Calling a first-order digital low-pass filter to perform smoothing filtering on multiple output results of the abnormal action detection model based on continuous frames; By judging whether the number of the plurality of output results after smoothing filtering that is lower than the preset score threshold reaches a preset number, it is determined whether the preset target has an abnormal action.

7. The abnormal motion detection method according to any one of claims 1 to 6, characterized in that: The method for generating the abnormal action detection model includes: Acquire training samples and supervision images, wherein the training samples are obtained based on images annotated with preset normal actions and the model parameters, and the supervision images include the images annotated with preset normal actions; Inputting the training sample into an initial detection model to obtain an output result of the initial detection model; Determine the loss parameters of the output result and the supervision image based on the loss function; The initial detection model is feedback updated according to the loss parameter until it is trained into an abnormal action detection model.

8. The method for generating an abnormal action detection model according to claim 7, characterized in that: The training samples include a disordered sequence obtained by scrambling the images with the preset normal action in the time dimension, and inputting the training samples into the initial detection model to obtain the output result of the initial detection model, including: The scrambled sequence is input into the initial detection model as the training sample, wherein after receiving the scrambled sequence, the initial detection model identifies the real action of each frame in the scrambled sequence, and sequentially predicts the predicted action of the next frame corresponding to the real action of the current frame, to obtain the output result of the initial detection model.

9. The method for generating an abnormal action detection model according to claim 7, wherein: The step of inputting the training sample into the initial detection model to obtain the output result of the initial detection model further includes: The training sample is input into an initial detection model, wherein after receiving the training sample, the initial detection model identifies the key points of a preset frame in the training sample to obtain an output result of the initial detection model, wherein the preset frame is at least two frames in sequence, and the key points are used to characterize the action posture corresponding to the preset frame.

10. The method for generating an abnormal action detection model according to claim 7, wherein: The step of inputting the training sample into the initial detection model to obtain the output result of the initial detection model further includes: Input the training sample into an initial detection model, wherein after receiving the training sample, the initial detection model identifies a feature vector of a preset frame in the training sample to obtain an output result of the initial detection model, wherein the output result includes an initial feature vector having the highest similarity to the feature vector in a pre-stored memory module; Feedback updating the initial detection model according to the loss parameter until training the abnormal action detection model includes: A preset memory module is updated based on the feature vector and the initial feature vector until the preset memory module is no longer updated.

Images