Abnormal behavior recognition method and device

Abstract

The application belongs to the technical field of image processing, and specifically discloses an abnormal behavior recognition method and device, wherein the method comprises the following steps: acquiring an initial image sequence, wherein the images in the initial image sequence are minimum rectangular images containing a target; inputting the initial image sequence into a target silhouette deep learning model to generate an initial silhouette sequence of the target; the target silhouette deep learning model is obtained after being trained based on sample images and pre-determined binary silhouette label information; intercepting part of the initial silhouette sequence that is irregular as an initial behavior silhouette sequence; acquiring an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model; and the abnormal behavior deep learning model is used to determine the abnormal behavior classification result of the input silhouette sequence. The application realizes the recognition of abnormal behaviors that are short in duration and sudden, and avoids excessive redundant background information caused by directly taking the initial silhouette sequence as input.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more particularly, to an abnormal behavior recognition method and device. BACKGROUND

[0002] With the rapid development of artificial intelligence and image processing technology, the research on intelligent video monitoring system based on deep learning is of great significance. With the development of technology, intelligent monitoring technology is also constantly progressing, and the demand for human abnormal behavior recognition and detection technology in social production and life is also becoming more and more widespread. In the field of medical care, through abnormal behavior recognition and detection technology, real-time monitoring of unattended patients or the elderly can be realized, and it can be judged whether the target falls or other accidents occur, and timely alarm and help can be provided to ensure that they receive treatment and help in the first time; in the field of traffic supervision, human abnormal behavior recognition and detection technology can be used to monitor abnormal situations inside and outside the vehicle including the behavior of the driver, and reduce the risk of accidents; in the field of public security, this technology can be used for abnormal situation detection of personnel in public places to judge behaviors such as fighting.

[0003] However, compared with the actions with long duration such as running, fast walking and retreating, the recognition effect of the actions with short duration and sudden occurrence still needs to be improved. SUMMARY

[0004] In view of the defects in the related art, the embodiments of the present application provide an abnormal behavior recognition method and device, aiming to solve the problem of poor recognition effect of actions with short duration and sudden occurrence.

[0005] In a first aspect, the embodiments of the present application provide an abnormal behavior recognition method, comprising:

[0006] obtaining an initial image sequence, the images in the initial image sequence being minimum rectangular images containing a target;

[0007] inputting the initial image sequence into a target silhouette deep learning model to generate an initial silhouette sequence of the target; wherein the target silhouette deep learning model is obtained based on sample images and pre-determined binary silhouette label information after training;

[0008] cutting the irregular part of the initial silhouette sequence as an initial behavior silhouette sequence;

[0009] obtaining an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model; wherein the abnormal behavior deep learning model is used to determine the abnormal behavior classification result of the input silhouette sequence.

[0010] In a second aspect, the embodiments of the present application also provide an abnormal behavior recognition device, comprising:

[0011] The first obtaining module is configured to obtain an initial image sequence, and each image in the initial image sequence is a minimum rectangular image containing a target.

[0012] The generating module is configured to input the initial image sequence into a target silhouette deep learning model to generate an initial silhouette sequence of the target, wherein the target silhouette deep learning model is obtained based on sample images and pre-determined binary silhouette annotation information after training.

[0013] The intercepting module is configured to intercept a non-regular part of the initial silhouette sequence as an initial behavior silhouette sequence.

[0014] The second obtaining module is configured to obtain an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model, wherein the abnormal behavior deep learning model is used to determine an abnormal behavior classification result of an input silhouette sequence.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, including: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, and when the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0018] The abnormal behavior recognition method and device provided by the embodiment of the present application use the target silhouette deep learning model to obtain the initial silhouette sequence corresponding to the initial image sequence of the target, intercept the non-regular part of the initial silhouette sequence to obtain a short-time behavior sequence, use the short-time behavior sequence as the input basis of the abnormal behavior deep learning model, which is more in line with the actual demand of recognizing the abnormal behavior with a short duration and suddenness, realizes the recognition of the abnormal behavior with a short duration and suddenness, and avoids excessive redundant background information caused by directly using the initial silhouette sequence as the input, so that the extracted image information is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the technical solutions in the application or the related art clearer, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort on the basis of these accompanying drawings.

[0020] Figure 1 is a flowchart of an abnormal behavior recognition method provided by an embodiment of the application;

[0021] Figure 2 is a flowchart of a process of obtaining a fitted behavior silhouette sequence provided by an embodiment of the application;

[0022] Figure 3 is a structural diagram of a target silhouette deep learning model provided by an embodiment of the application;

[0023] Figure 4 is an architectural diagram of an abnormal behavior deep learning model provided by an embodiment of the application;

[0024] Figure 5 is a structural diagram of an abnormal behavior recognition device provided by an embodiment of the application;

[0025] Figure 6 is a structural diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0026] In order to make the technical solutions in the application or the related art clearer, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort on the basis of these accompanying drawings.

[0027] In different application scenarios, the types of abnormal behaviors to be detected are also different, especially the actions with short duration and sudden occurrence, which are difficult to be detected. However, some actions with short duration and sudden occurrence can already achieve the purpose of transmitting abnormal information or causing serious consequences such as security threats, so the conventional abnormal behavior recognition has been difficult to meet the actual recognition needs.

[0028] To solve the above problems existing in the related art, the embodiment of the present application provides an abnormal behavior recognition method and device, which obtains an initial silhouette sequence corresponding to an initial image sequence of a target by using a target silhouette deep learning model, obtains a short-time behavior sequence by intercepting an irregular part in the initial silhouette sequence, and takes the short-time behavior sequence as an input basis of an abnormal behavior deep learning model, so as to better meet the actual demand for recognizing an abnormal behavior with a short duration and suddenness, realize the recognition of the abnormal behavior with the short duration and suddenness, and avoid excessive redundant background information caused by directly taking the initial silhouette sequence as an input, so that the extracted image information is more accurate.

[0029] Figure 1 FIG. 1 is a flowchart of an abnormal behavior recognition method provided by the embodiment of the present application, as shown in FIG. 1, the method at least includes the following steps (Step): Figure 1

[0030] S101, obtaining an initial image sequence, and images in the initial image sequence are minimum rectangular images containing a target.

[0031] Specifically, the initial image sequence is generally obtained by preprocessing a video (stream) captured by a camera.

[0032] Optionally, the preprocessing of the video captured by the camera includes: performing real-time frame extraction on the video (stream), and performing target position detection, target tracking and subsequent abnormal behavior recognition only on the extracted frame images. Considering that the gait feature of the target does not change significantly in a very short duration, the frame extraction processing can reduce the overall operation amount and operation complexity of the system without affecting the final recognition effect.

[0033] Optionally, the preprocessing of the video captured by the camera includes: after determining the frame images containing the target, the frame images are cropped to obtain the minimum rectangular images containing the target, and the initial image sequence corresponding to the target is arranged in time sequence. The target can be a human body, an animal body or the like.

[0034] Optionally, the preprocessing of the video captured by the camera includes: detecting the positions of different targets in the frame images of the video by using a target position detection algorithm (for example, a position detection deep learning model), and tracking each detected target by using a target tracking algorithm (for example, a general DeepSort algorithm), that is, locating the positions of the same target in different frame images. The target position detection algorithm and the target tracking algorithm are not limited in the embodiment of the present application.

[0035] ​The initial image sequence is determined when it is detected that the target leaves the field of view of the camera or the number of images (real-time frame-extracted images) accumulates to a certain threshold. The video data captured by the camera can be real-time video data or historical video data; the video data can be collected and preprocessed by itself, or the processed initial image sequence can be directly obtained.

[0036] S102, inputting the initial image sequence to a target silhouette deep learning model to generate an initial silhouette sequence of the target.

[0037] The target silhouette deep learning model is obtained based on sample images and pre-determined binary silhouette annotation information after training.

[0038] Specifically, the images in the initial image sequence are sequentially input to the pre-trained target silhouette deep learning model to generate the initial silhouette sequence of the target. Before performing S102, the target silhouette deep learning model needs to be constructed and trained.

[0039] The training process of the target silhouette deep learning model is as follows: first, sample images collected under different states are obtained; then, the binary silhouette annotation information corresponding to the samples in the images is determined; a plurality of sample images and corresponding binary silhouette annotation information form training samples. Optionally, the target silhouette deep learning model uses a segmentation convolutional network, which includes an encoder module, a decoder module, an encoding-decoding connection model, and a pixel-level classification layer. After obtaining the training samples and determining the type of the target silhouette deep learning model, the model is trained until the model converges.

[0040] When performing binary silhouette annotation on the samples, the edge contour of each collected sample is annotated using a polygon based on the sample position in the image. Using the edge contour of the collected sample, it can be determined whether a pixel point in the image is a foreground pixel point or a background pixel point. For example, by using python code to load the saved json file, the contour coordinates can be parsed, and by using the pointPolygonTest method of opencv, it can be determined whether a pixel point outside the contour (background pixel point) or a pixel point inside the contour (foreground pixel point) in the image.

[0041] Different color values are attached to the background pixel points and foreground pixel points in the image, for example, the background pixel points are set to 0 and the foreground pixel points are set to 255, and the image is saved as a picture, which can obtain a binary silhouette image with black background and white collected sample silhouette.

[0042] S103, intercepting the irregular part of the initial silhouette sequence as an initial behavior silhouette sequence.

[0043] Specifically, for normal behaviors of the target, such as normal walking, running, standing still, etc., the silhouette sequence presents certain regular changes, for example, the walking and running change periodically, and the standing still basically remains unchanged. Therefore, the irregular part in the initial silhouette sequence is intercepted as the initial behavior silhouette sequence, that is, the initial short-time behavior sequence is obtained, and the short-time behavior sequence contains the abnormal behavior characteristics of the target.

[0044] In S104, an abnormal behavior classification result is obtained based on the initial behavior silhouette sequence and the pre-trained abnormal behavior deep learning model.

[0045] The abnormal behavior deep learning model is used to determine the abnormal behavior classification result of the input silhouette sequence.

[0046] Specifically, the initial behavior silhouette sequence intercepted is used to obtain the abnormal behavior classification result by using the pre-trained abnormal behavior deep learning model.

[0047] Optionally, the initial behavior silhouette sequence is input into the pre-trained abnormal behavior deep learning model to obtain the abnormal behavior classification result.

[0048] Optionally, the initial behavior silhouette sequence is fitted, and the fitted behavior silhouette sequence is input into the pre-trained abnormal behavior deep learning model to obtain the abnormal behavior classification result.

[0049] The abnormal behavior recognition method provided by the embodiment of the application uses the target silhouette deep learning model to obtain the initial silhouette sequence corresponding to the initial image sequence of the target, intercepts the irregular part in the initial silhouette sequence to obtain a short-time behavior sequence, uses the short-time behavior sequence as the input basis of the abnormal behavior deep learning model, and is more in line with the actual needs of recognizing the abnormal behavior with a short duration and suddenness, realizes the recognition of the abnormal behavior with a short duration and suddenness, and avoids excessive redundant background information caused by directly using the initial silhouette sequence as the input, so that the extracted image information is more accurate.

[0050] In some embodiments, the initial behavior silhouette sequence is obtained by intercepting the irregular part of the initial silhouette sequence in S103, and the method includes the following steps.

[0051] The contour centroid position of the target in each silhouette image in the initial silhouette sequence is determined.

[0052] The part of the mutation sequence in the initial silhouette sequence is intercepted based on the contour centroid position, and the initial behavior silhouette sequence is obtained.

[0053] Specifically, after obtaining the initial silhouette sequence output by the target silhouette deep learning model, the initial silhouette sequence is analyzed to find out the abnormal behavior of the target in the long-time normal behavior, such as a pedestrian suddenly bending over while walking normally, or a person suddenly jumping while standing, and the like, and thus the irregular part in the initial silhouette sequence needs to be cut out as the initial behavior silhouette sequence.

[0054] The contour centroid position of the target in each silhouette image in the initial silhouette sequence is determined, and part of the mutation sequence in the initial silhouette sequence is cut out based on the contour centroid position as the initial behavior silhouette sequence. The main basis for cutting out is the contour centroid position of the target in the silhouette image, that is, the pixel point position where the centroid of the external contour of the target is located. It is conceivable that in addition to the centroid, other points of interest can also be considered. Considering that the position of the centroid changes relatively stably in the regular motion of the target for a long duration, the contour centroid position is selected as the basis for cutting out the irregular part sequence, rather than a certain pixel point of the head or limb position.

[0055] In some embodiments, cutting out part of the mutation sequence in the initial silhouette sequence based on the contour centroid position comprises:

[0056] The silhouette images in the initial silhouette sequence are processed by equal width, and part of the mutation sequence in the initial silhouette sequence is cut out based on the height change of the contour centroid position of the target in the initial silhouette sequence processed by equal width as the initial behavior silhouette sequence; or,

[0057] The silhouette images in the initial silhouette sequence are processed by equal height, and part of the mutation sequence in the initial silhouette sequence is cut out based on the width change of the contour centroid position of the target in the initial silhouette sequence processed by equal height as the initial behavior silhouette sequence.

[0058] Specifically, the contour centroid position of the target in the silhouette image can be calculated by the following formula:

[0059]

[0060] wherein x and y represent the x coordinate and y coordinate of the contour centroid, respectively, x i represents the x coordinate of all pixel points in the target contour, and y i represents the y coordinate of all pixel points in the target contour.

[0061] Since the silhouette is a binary image, it is considered that all the masses of the targets are the same, the x coordinate of the centroid is the average value of the x coordinates of all pixel points in the target contour, and the y coordinate of the centroid is the average value of the y coordinates of all pixel points in the target contour. By the position change of the centroid in all silhouette images in the initial silhouette sequence, the mutated image can be determined.

[0062] If the silhouette images in all initial silhouette images are processed with equal width, for example, all images are adjusted to be high-ratio scaled images with a width of 256 pixels and a height of 256 pixels, that is, the width and height of the original image are w and h respectively, and the width and height after adjustment are 256 and (h*256) / w respectively, only the height change (corresponding to the y coordinate) of the centroid needs to be considered, and the part sequence in which the centroid position of the contour suddenly changes on the y axis is taken as the initial behavior silhouette sequence. If the silhouette images in the initial silhouette images are processed with equal height, for example, all images are adjusted to be high-ratio scaled images with a width of (w*256) / h and a height of 256, that is, the width and height of the original image are w and h respectively, and the width and height after adjustment are (w*256) / h and 256 respectively, only the width change (corresponding to the x coordinate) of the centroid needs to be considered, and the part sequence in which the centroid position of the contour suddenly changes on the x axis is taken as the initial behavior silhouette sequence.

[0063] The image in which the sudden change occurs is the starting image of the initial behavior silhouette sequence, corresponding to the start of the abnormal behavior; and the image in which the sudden change ends is the ending image of the initial behavior silhouette sequence, corresponding to the end of the abnormal behavior.

[0064] The abnormal behavior recognition method provided by the embodiments of the present application extracts the irregular part in the initial behavior silhouette sequence output by the target silhouette deep learning model through the change of the contour centroid position of the target in the silhouette image, obtains a short-time behavior sequence, and avoids the problem of too much redundant background information caused by directly inputting the initial behavior silhouette sequence for abnormal behavior recognition.

[0065] In some embodiments, S105 specifically includes:

[0066] fitting the initial behavior silhouette sequence to obtain a fitted behavior silhouette sequence;

[0067] inputting the fitted behavior silhouette sequence into the pre-trained abnormal behavior deep learning model to obtain an abnormal behavior classification result.

[0068] Specifically, in the same behavior sequence, the pixel points in the binary silhouette of the target do not suddenly change, and therefore the initial behavior silhouette sequence needs to be fitted to correct the pixel point positions in the sudden change image, and a fitted behavior silhouette sequence is obtained. The fitted behavior silhouette sequence is input into the pre-trained abnormal behavior deep learning model to obtain an abnormal behavior classification result, and the abnormal behavior recognition effect is more accurate.

[0069] In some embodiments, fitting the initial behavior silhouette sequence specifically includes:

[0070] processing the silhouette images in the initial behavior silhouette sequence with equal height and equal width, and determining the positions of at least two key points of the target in the processed silhouette images;

[0071] In the behavior sequence corresponding to the regular part in the initial silhouette sequence, based on the positions of the first key points in the time-sequentially continuous silhouette images, a first average rate of change in the x direction and a second average rate of change in the y direction of the first key points are determined.

[0072] Based on the first average rate of change, a first threshold value in the x direction is set; and based on the second average rate of change, a second threshold value in the y direction is set.

[0073] If the first rate of change in the x direction of a current silhouette image in the initial behavior silhouette sequence, compared with a time-sequentially continuous previous silhouette image, exceeds the first threshold value, and the rate of change in the x direction of a next silhouette image time-sequentially continuous with the current silhouette image, compared with the current silhouette image, exceeds the first threshold value, the current silhouette image is fitted in the x direction, and is adjusted to the average of the coordinate values in the x direction of the previous silhouette image and the next silhouette image.

[0074] If the third rate of change in the y direction of the current silhouette image in the initial behavior silhouette sequence, compared with the time-sequentially continuous previous silhouette image, exceeds the second threshold value, and the rate of change in the y direction of the next silhouette image time-sequentially continuous with the current silhouette image, compared with the current silhouette image, exceeds the second threshold value, the current silhouette image is fitted in the y direction, and is adjusted to the average of the coordinate values in the y direction of the previous silhouette image and the next silhouette image.

[0075] Specifically, the silhouette images in the initial silhouette sequence are processed in equal height and equal width, for example, all the silhouette images in the initial silhouette sequence are adjusted to images with a width and height of 256 pixels, so that all the images in the sequence have the same size.

[0076] The positions of at least two key points of the target in the processed silhouette image are determined. The selection of the key points can be the contour centroid, the highest point of the head of the target, the edge points of the limbs of the target, etc. Since the original images are all minimum rectangular images containing complete targets, i.e., the contour positions of the targets in the images will occupy the entire image. Taking a pedestrian as an example, the highest point of the head of the pedestrian is the white position closest to the upper edge in the binary image, the lowest point of the foot is the white position closest to the lower edge in the binary image, and similarly, the white positions closest to the left and right edges are the limbs. To distinguish between the upper limbs and the lower limbs, it can be considered that the lower limbs are below 0.3 of the image height and the upper limbs are above 0.3 of the image height according to the human body structure. Therefore, the positions of the head and the limbs of the pedestrian in an image can be determined, and the key points can be selected. The selected at least two key points can be five key points, which are the highest point of the head of the target and the edge pixel points of the limbs of the target.

[0077] In the process of the target continuously behaving normally, the positions of the key points should change approximately regularly, that is, the positions of the key points will not mutate, and the change trend of the position of each key point is analyzed according to the position of each key point in each silhouette image.

[0078] For any one of the key points, denoted as a first key point, in the behavior sequence corresponding to the regular part in the initial silhouette sequence, a first average change rate of the first key point in the x direction, denoted as σ x , is determined based on the positions of the first key point in the time-sequentially continuous silhouette images in the behavior sequence; a second average change rate of the first key point in the y direction, denoted as σ y , is determined. A first threshold value, for example, 2 x or other values, is set based on the first average change rate; and a second threshold value, for example, 2 y or other values, is set based on the second average change rate.

[0079] In the irregular initial behavior silhouette sequence, if a first change rate of a current silhouette image A in the x direction exceeds 2 x as compared with a time-sequentially continuous previous silhouette image A0, and a change rate of a time-sequentially continuous next silhouette image A1 in the x direction exceeds 2 x as compared with the current silhouette image A, it is considered that the first key point in the x direction needs to be fitted in the current silhouette image A, and a specific fitting manner can be adjusting to an average value of the coordinate values in the x direction of the previous silhouette image A0 and the next silhouette image A1.

[0080] In the irregular initial behavior silhouette sequence, if a first change rate of a current silhouette image A in the x direction exceeds 2 x as compared with a time-sequentially continuous previous silhouette image A0, and a second change rate of a time-sequentially continuous next silhouette image A1 in the x direction exceeds 2 x as compared with the current silhouette image A, it is considered that the first key point in the x direction needs to be fitted in the current silhouette image A, and a specific fitting manner can be adjusting to an average value of the coordinate values in the x direction of the previous silhouette image A0 and the next silhouette image A1.

[0081] In the irregular initial behavior silhouette sequence, if a third change rate of a current silhouette image A in the y direction exceeds 2 y as compared with a time-sequentially continuous previous silhouette image A0, and a fourth change rate of a time-sequentially continuous next silhouette image A1 in the y direction exceeds 2 yIf the first key point in the current silhouette image A needs to be fitted in the y direction, the specific fitting method can be: adjusting the average value of the coordinate values in the y direction of the previous silhouette image A0 and the next silhouette image A1.

[0082] Figure 2 is a flowchart of a process for obtaining the fitted behavior silhouette sequence provided by the embodiment of the present application, as Figure 2 shown, after obtaining the initial silhouette sequence output by the target silhouette deep learning model, the irregular part of the sequence is intercepted, the key point position is fitted, and the fitted behavior silhouette sequence is obtained as the input of the abnormal behavior deep learning model.

[0083] The abnormal behavior recognition method provided by the embodiment of the present application, after intercepting the irregular part of the initial silhouette sequence of the target output by the target silhouette deep learning model, obtains a short-time behavior sequence, and then uses the continuity of the normal action behavior of the target to fit the short-time behavior sequence, considers more comprehensive dimensions, so that the subsequent extracted image information is more accurate and more meets the requirements of recognizing behaviors with shorter duration.

[0084] In some embodiments, the target silhouette deep learning model uses a segmentation convolutional network, which includes an encoder network, a decoder network, a coding-decoding connection module, and a pixel-level classification layer. The encoder network includes a convolutional layer and a maximum pooling layer, and the decoder network includes a convolutional layer and an up-sampling layer. The network structures of the encoder network and the decoder are symmetrical. The coding-decoding connection module uses a skip connection, and each convolutional layer before a maximum pooling layer in the encoder network is connected to the last convolutional layer of the corresponding size in the decoder network. The encoder network records boundary information through a pooling index, and the decoder network realizes non-linear up-sampling through a pooling index derived from the corresponding maximum pooling layer.

[0085] Specifically, the target silhouette deep learning model uses a segmentation convolutional neural network to improve the performance of image segmentation, and uses a set index in the segmentation convolutional network to further improve the accuracy of the segmentation boundary.

[0086] The segmentation convolutional network includes an encoder module, a decoder network, a coding-decoding connection module, and a pixel-level classification layer. The encoder module includes a convolutional layer and a maximum pooling layer, and the decoder module includes a convolutional layer and an up-sampling layer. The encoder module and the decoder module use a symmetrical network structure, i.e., each layer in the encoder module has a corresponding layer in the decoder.

[0087] The encoder module records boundary information through a pooling index. During the encoding process, the size of the feature map changes after the max-pooling operation of the pooling layer, and the low-resolution features generated by the down-sampling operation lose some information; in order to retain more details, the segmentation convolutional network uses the pooling index to record the boundary information, that is, the pixel position of the maximum value selected by the pooling layer in the max-pooling operation. During the decoding process, the up-sampling operation is the inverse process of the pooling operation, the feature map is first enlarged, and then the data of the feature map is put into the corresponding position according to the pooling index.

[0088] The decoder module realizes the non-linear up-sampling through the pooling index derived from the corresponding max-pooling layer, so that the up-sampling layer does not need to be learned, and the training parameters can be reduced. Due to the introduction of the pooling index, more boundary information can be obtained, and the segmentation performance of the model can be improved. The role of the convolution operation is to enrich the information of the sparse feature map generated by the up-sampling layer, and to generate a more continuous feature map.

[0089] In some embodiments, the encoding-decoding connection module adopts a jump connection, and the convolution layer before each max-pooling layer in the encoder module is connected to the last convolution layer of the corresponding size in the decoder module.

[0090] Specifically, the segmentation convolutional network also implements a jump layer fusion mechanism, that is, different scale feature maps generated by different layers are fused into a fusion map. After each down-sampling or up-sampling operation, the scale of the feature map changes, so there are multiple scales of the feature map in the segmentation convolutional network.

[0091] In order to utilize the sparse feature maps and continuous feature maps of different scales, a cross-layer fusion mechanism is established to connect the encoder network and the decoder network. The encoding-decoding connection module adopts a jump connection, and the convolution layer before each max-pooling layer in the encoder module is connected to the last convolution layer of the corresponding size in the decoder network.

[0092] Figure 3 is a structural schematic diagram of a target silhouette deep learning model provided by an embodiment of the present application, as Figure 3 As shown in the figure, the network structure of the encoder module is consistent with VGG16, and contains 13 convolution layers and 5 pooling layers; the network structure of the decoder module is symmetrical to the network structure of the encoder module, and contains 13 convolution layers and 5 up-sampling layers, and each layer of the encoder module has a corresponding layer in the decoder module. The decoder can realize the non-linear up-sampling operation through the pooling index derived from the corresponding max-pooling layer, so that the up-sampling layer does not need to be learned, and the training parameters can be reduced.

[0093] Meanwhile, more boundary information can be obtained due to the introduction of the pooling index, and the performance of the segmentation is improved. In the encoding process, the size of the feature map will change after the max-pooling operation, and some information will be lost in the low-resolution feature generated by the down-sampling operation; in order to preserve more details, the segmentation convolutional network uses the pooling index to record the boundary information, that is, the pixel position of the maximum value selected by the pooling layer in the max-pooling operation. In the decoding process, the up-sampling operation is the inverse process of the pooling operation, the feature map is first enlarged, and then the data of the feature map is put into the corresponding position according to the pooling index. The role of the convolution operation is to enrich the information of the sparse feature map generated by the up-sampling layer, and to generate a more continuous feature map.

[0094] Meanwhile, a skip layer fusion mechanism is implemented on the segmentation convolutional network, that is, different scale feature maps generated by different layers are fused into a fusion map. Each down-sampling or up-sampling operation will change the scale of the feature map, so there are 5 scales of the feature map in the segmentation convolutional network. In order to utilize the sparse feature maps and continuous feature maps of different scales, a cross-layer fusion mechanism is established to connect the encoder network and the decoder network, and the convolutional layer before each max-pooling layer in the encoder network is connected to the last convolutional layer of the corresponding scale in the decoder network.

[0095] In some embodiments, the abnormal behavior deep learning model adopts a twin structure to extract features from the behavior silhouette sequence and the randomly enhanced behavior silhouette sequence at the same time, and outputs the feature images obtained after the two branches pass through the attention mechanism after feature fusion, the twin structure shares parameters; the output result of the feature fusion twin structure is input into the same feature extraction network as the behavior silhouette sequence and is fused through horizontal connection, and the output is the abnormal behavior category after passing through the full connection layer and the classifier layer.

[0096] Figure 4 is the architecture schematic diagram of the abnormal behavior deep learning model provided by the embodiments of the present application, as Figure 4 shown, a twin structure is introduced in the network model input stage. One branch takes the behavior silhouette sequence (initial behavior silhouette sequence or fitted behavior silhouette sequence) as input, and the other branch takes the randomly enhanced behavior silhouette sequence as input. The parameters are shared in the two branches, which greatly improves the network training efficiency.

[0097] At the input, different disturbances are randomly generated for the data, mainly random enhancement is adopted for the image input by one branch, including random cropping, random flipping and random erasing. Then, random regularization (Dropout) is adopted for the output features of the two branches.

[0098] After the data passes through the twin structure, the output result of the twin structure after feature fusion (slow channel) and the behavior silhouette sequence (fast channel) are input into the same feature extraction network, such as a ResNet network, for feature fusion through lateral connection. Here, the output result of the fast channel is sent to the slow channel through lateral connection, but the data sample shapes of the two channels are different, so the output shape of the fast channel is transformed before fusion.

[0099] The feature fusion module includes an attention mechanism and an optimization part. The attention mechanism is a simple and effective attention module that can be directly applied to a feedforward convolutional neural network, and is composed of a channel attention module and a spatial attention module. For the feature map generated by the convolutional neural network, the attention mechanism calculates an attention map of the feature map from the channel and spatial dimensions, and then multiplies the attention map and the corresponding elements of the feature map to perform adaptive learning of the features.

[0100] Optionally, the feature map obtained after the attention mechanism is subjected to Concat fusion, and the two features are directly connected. If the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p+q. At this time, due to weight sharing, the efficiency of the network training process will not be reduced. After the Concat operation, a Bn function and a ReLu function can be used. From the perspective of data saturation, if ReLu is performed first, some units will be deactivated when Bn is performed, causing Bn to be unstable and affecting the performance of the model; Bn standardizes the data distribution, so that the data returns to the non-saturated region, and then passes through ReLu to control the saturation degree of activation. Therefore, in the embodiments of the present application, the operation of placing Bn before ReLu is adopted.

[0101] Finally, a fully connected layer is used to output an n (number of recognized abnormal behaviors) dimensional vector O, and a softmax operation is performed on the vector to obtain an n dimensional vector O ′ , representing the probability of each abnormal behavior, and setting a threshold of 2 / n, if the maximum value O ′ in O ′ max >2 / n, the abnormal behavior result represented by the vector position of the maximum value is output, otherwise the category is other category.

[0102] The abnormal behavior recognition method provided in the embodiments of the present application directly uses the behavior silhouette sequence as the input of the abnormal behavior deep learning model, uses the twin structure to process the original sequence data and the enhanced data at the same time when inputting, and the two structures share parameters and perform feature fusion subsequently, so that the training speed is not affected and the robustness is greatly improved, and since the semantic information is already included in the input sequence information, there is no need to provide a skip frame, the feature information is fused through lateral connection, and the behavior features can be better extracted.

[0103] Figure 5 is a structural schematic diagram of an abnormal behavior recognition device provided by an embodiment of the present application, as shown in the figure, the device at least includes: Figure 5

[0104] The first acquisition module 501 is configured to acquire an initial image sequence, and an image in the initial image sequence is a minimum rectangular image containing a target.

[0105] The generation module 502 is configured to input the initial image sequence to a target silhouette deep learning model to generate an initial silhouette sequence of the target, wherein the target silhouette deep learning model is obtained after training based on sample images and pre-determined binary silhouette annotation information.

[0106] The intercepting module 503 is configured to intercept a part of the initial silhouette sequence that is irregular as an initial behavior silhouette sequence.

[0107] The second acquisition module 504 is configured to acquire an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model, wherein the abnormal behavior deep learning model is used to determine an abnormal behavior classification result of an input silhouette sequence.

[0108] In some embodiments, the intercepting module 503 includes:

[0109] The determination unit is configured to determine a contour centroid position of the target in each silhouette image in the initial silhouette sequence.

[0110] The intercepting unit is configured to intercept a part of the mutation sequence in the initial silhouette sequence as the initial behavior silhouette sequence based on the contour centroid position.

[0111] In some embodiments, the intercepting unit is specifically configured to:

[0112] perform equal-width processing on the silhouette images in the initial silhouette sequence, intercept a part of the mutation sequence in the initial silhouette sequence as the initial behavior silhouette sequence based on a height change of the contour centroid position of the target in the initial silhouette sequence after the equal-width processing, or

[0113] perform equal-height processing on the silhouette images in the initial silhouette sequence, intercept a part of the mutation sequence in the initial silhouette sequence as the initial behavior silhouette sequence based on a width change of the contour centroid position of the target in the initial silhouette sequence after the equal-height processing.

[0114] In some embodiments, the second acquisition module 504 includes:

[0115] The fitting unit is configured to fit the initial behavior silhouette sequence to obtain a fitted behavior silhouette sequence.

[0116] ​The acquisition unit is configured to input the fitted behavior silhouette sequence into a pre-trained abnormal behavior deep learning model to obtain an abnormal behavior classification result.

[0117] In some embodiments, the fitting unit is specifically configured to:

[0118] performing contour and width processing on the silhouette images in the initial silhouette sequence, and determining positions of at least two key points of the target in the processed silhouette images;

[0119] determining, in a behavior sequence corresponding to the regular part in the initial silhouette sequence, a first average change rate of the first key point in the x direction and a second average change rate of the first key point in the y direction based on positions of the first key point in the time-sequentially continuous silhouette images;

[0120] setting a first threshold value in the x direction based on the first average change rate, and setting a second threshold value in the y direction based on the second average change rate;

[0121] if a first change rate of a current silhouette image in the initial behavior silhouette sequence in the x direction compared with a time-sequentially continuous previous silhouette image exceeds the first threshold value, and a change rate of a next silhouette image time-sequentially continuous with the current silhouette image in the x direction compared with the current silhouette image exceeds the first threshold value, fitting the current silhouette image in the x direction and adjusting to an average of coordinate values of the previous silhouette image and the next silhouette image in the x direction;

[0122] if a third change rate of the current silhouette image in the initial behavior silhouette sequence in the y direction compared with the time-sequentially continuous previous silhouette image exceeds the second threshold value, and a change rate of the next silhouette image time-sequentially continuous with the current silhouette image in the y direction compared with the current silhouette image exceeds the second threshold value, fitting the current silhouette image in the y direction and adjusting to an average of coordinate values of the previous silhouette image and the next silhouette image in the y direction.

[0123] In some embodiments, the target silhouette deep learning model adopts a segmentation convolutional network, which includes an encoder module, a decoder module, an encoding-decoding connection module and a pixel-level classification layer.

[0124] The encoder module includes a convolutional layer and a max-pooling layer, and the decoder module includes a convolutional layer and an up-sampling layer, and the network structures of the encoder module and the decoder module are symmetrical.

[0125] The encoder module records boundary information through a pooling index, and the decoder module realizes nonlinear up-sampling through a pooling index derived from a corresponding max-pooling layer.

[0126] The coding connection module adopts a skip connection, and each convolutional layer before a maximum pooling layer in the encoder module is connected to the last convolutional layer of the corresponding size in the decoder module.

[0127] In some embodiments, the abnormal behavior deep learning model adopts a twin structure to extract features from the behavior silhouette sequence and the randomly enhanced behavior silhouette sequence at the same time, the feature images obtained after the two branches pass through the attention mechanism are fused and output, and the twin structure shares parameters; the output results of the feature fusion twin structure and the behavior silhouette sequence are respectively input into the same feature extraction network and fused through horizontal connection, and the abnormal behavior category is output after passing through the full connection layer and the classifier layer.

[0128] It can be understood that the detailed function implementation of each unit / module described above can refer to the description in the foregoing method embodiments, which will not be repeated here.

[0129] It should be understood that the above device is used to execute the method in the above embodiments, the corresponding program modules in the device, the implementation principles and technical effects are similar to the description in the above method, and the working process of the device can refer to the corresponding process in the above method, which will not be repeated here.

[0130] Based on the method in the above embodiments, the embodiments of the present application provide an electronic device. The device can include at least one memory for storing programs and at least one processor for executing the programs stored in the memory. Wherein, when the program stored in the memory is executed, the processor is used to execute the method described in the above embodiments.

[0131] Figure 6 is a structural schematic diagram of an electronic device provided by the embodiments of the present application, as shown in Figure 6 The electronic device can include a processor 601, a communications interface 620, a memory 603, and a communications bus 604, wherein the processor 601, the communications interface 602, and the memory 603 complete mutual communication through the communications bus 604. The processor 601 can invoke software instructions in the memory 603 to execute the method described in the above embodiments.

[0132] In addition, the logic instructions in the memory 603 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the related art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application.

[0133] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and when the computer program runs on a processor, the processor executes the method in the above embodiments.

[0134] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product, and when the computer program product runs on a processor, the processor executes the method in the above embodiments.

[0135] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0136] The method steps in the embodiments of the present application can be implemented by means of hardware, or by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a Random Access Memory (RAM), a flash memory, a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0137] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a Solid State Disk (SSD)), etc.

[0138] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of differentiation, and do not limit the scope of the embodiments of the present application.

[0139] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present application, and are not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An abnormal behavior recognition method characterized by comprising: The method comprises the following steps: acquiring an initial image sequence, wherein images in the initial image sequence are minimum rectangular images containing a target; inputting the initial image sequence into a target silhouette deep learning model to generate an initial silhouette sequence of the target; wherein the target silhouette deep learning model is obtained after training based on sample images and pre-determined binary silhouette annotation information; cutting irregular part of the initial silhouette sequence as an initial behavior silhouette sequence; acquiring an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model; wherein the abnormal behavior deep learning model is used to determine the abnormal behavior classification result of an input silhouette sequence; the step of acquiring an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model comprises: fitting the initial behavior silhouette sequence to obtain a fitted behavior silhouette sequence; inputting the fitted behavior silhouette sequence into the pre-trained abnormal behavior deep learning model to obtain an abnormal behavior classification result; the step of fitting the initial behavior silhouette sequence comprises: performing contour and width processing on the silhouette images in the initial silhouette sequence, and determining the positions of at least two key points of the target in the processed silhouette images; in a behavior sequence corresponding to a regular part of the initial silhouette sequence, based on the positions of a first key point in time-sequentially continuous silhouette images, determining a first average change rate of the first key point in an x direction and a second average change rate of the first key point in a y direction; based on the first average change rate, setting a first threshold value in the x direction; based on the second average change rate, setting a second threshold value in the y direction; if a first change rate of a current silhouette image in the initial behavior silhouette sequence in the x direction exceeds the first threshold value compared with a time-sequentially continuous previous silhouette image, and a change rate of a next silhouette image time-sequentially continuous with the current silhouette image in the x direction exceeds the first threshold value compared with the current silhouette image, then fitting the current silhouette image in the x direction and adjusting it to an average value of coordinate values of the previous silhouette image and the next silhouette image in the x direction; if a third change rate of a current silhouette image in the initial behavior silhouette sequence in the y direction exceeds the second threshold value compared with a time-sequentially continuous previous silhouette image, and a change rate of a next silhouette image time-sequentially continuous with the current silhouette image in the y direction exceeds the second threshold value compared with the current silhouette image, then fitting the current silhouette image in the y direction and adjusting it to an average value of coordinate values of the previous silhouette image and the next silhouette image in the y direction.

2. The abnormal behavior recognition method according to claim 1, characterized by, the step of cutting irregular part of the initial silhouette sequence as an initial behavior silhouette sequence comprises: determining the contour centroid positions of the target in each silhouette image in the initial silhouette sequence; cutting part of the mutation sequence in the initial silhouette sequence based on the contour centroid positions as the initial behavior silhouette sequence.

3. The abnormal behavior recognition method according to claim 2, characterized by, the step of cutting part of the mutation sequence in the initial silhouette sequence based on the contour centroid positions comprises: The silhouette images in the initial silhouette sequence are processed by equal width, and based on the height change of the profile centroid position of the target in the initial silhouette sequence after the equal width processing, part of the mutation sequence in the initial silhouette sequence is intercepted as the initial behavior silhouette sequence; or, The silhouette images in the initial silhouette sequence are processed by equal height, and based on the width change of the profile centroid position of the target in the initial silhouette sequence after the equal height processing, part of the mutation sequence in the initial silhouette sequence is intercepted as the initial behavior silhouette sequence.

4. The abnormal behavior recognition method according to claim 1, characterized by, The target silhouette deep learning model adopts a segmentation convolutional network, which includes an encoder module, a decoder module, a coding and decoding connection module, and a pixel-level classification layer; The encoder module includes a convolutional layer and a maximum pooling layer, the decoder module includes a convolutional layer and an up-sampling layer, and the network structures of the encoder module and the decoder module are symmetrical; The encoder module records boundary information through a pooling index, and the decoder module realizes nonlinear up-sampling through a pooling index derived from the corresponding maximum pooling layer; The coding and decoding connection module adopts a jump connection, and each convolutional layer before each maximum pooling layer in the encoder module is connected to the last convolutional layer of the corresponding size in the decoder module.

5. The abnormal behavior recognition method of claim 1, wherein, The abnormal behavior deep learning model adopts a twin structure to simultaneously extract features from the behavior silhouette sequence and the randomly enhanced behavior silhouette sequence, and outputs the feature images obtained after the two branches pass through the attention mechanism after feature fusion, the twin structure shares parameters; the output results of the twin structure after feature fusion are input into the same feature extraction network as the behavior silhouette sequence and are fused through horizontal connection, and the abnormal behavior category is output after passing through the full connection layer and the classifier layer.

6. An abnormal behavior recognition apparatus characterized by comprising: It includes: The first acquisition module is used for acquiring an initial image sequence, and the images in the initial image sequence are minimum rectangular images containing a target; The generation module is used for inputting the initial image sequence into a target silhouette deep learning model to generate an initial silhouette sequence of the target; wherein the target silhouette deep learning model is obtained based on sample images and pre-determined binary silhouette annotation information after training; The cutting module is used for cutting part of the initial silhouette sequence that is irregular as an initial behavior silhouette sequence; The second acquisition module is used for acquiring an abnormal behavior classification result based on the initial behavior silhouette sequence and a pre-trained abnormal behavior deep learning model; wherein the abnormal behavior deep learning model is used to determine the abnormal behavior classification result of the input silhouette sequence; The acquisition of the abnormal behavior classification result based on the initial behavior silhouette sequence and the pre-trained abnormal behavior deep learning model includes: Fitting the initial behavior silhouette sequence to obtain a fitted behavior silhouette sequence; Inputting the fitted behavior silhouette sequence into the pre-trained abnormal behavior deep learning model to obtain an abnormal behavior classification result; The fitting of the initial behavior silhouette sequence includes: performing contouring and equal-width processing on the silhouette images in the initial silhouette sequence, and determining positions of at least two key points of the target in the processed silhouette images; in a behavior sequence corresponding to the regular part in the initial silhouette sequence, determining a first average change rate of the first key point in the x direction and a second average change rate of the first key point in the y direction based on positions of the first key point in time-sequentially continuous silhouette images; setting a first threshold value in the x direction based on the first average change rate, and setting a second threshold value in the y direction based on the second average change rate; if a first change rate of a current silhouette image in the initial behavior silhouette sequence in the x direction exceeds the first threshold value compared with a time-sequentially previous silhouette image, and a change rate of a time-sequentially next silhouette image in the x direction exceeds the first threshold value compared with the current silhouette image, fitting the current silhouette image in the x direction and adjusting to an average of coordinate values in the x direction of the previous silhouette image and the next silhouette image; if a third change rate of a current silhouette image in the initial behavior silhouette sequence in the y direction exceeds the second threshold value compared with a time-sequentially previous silhouette image, and a change rate of a time-sequentially next silhouette image in the y direction exceeds the second threshold value compared with the current silhouette image, fitting the current silhouette image in the y direction and adjusting to an average of coordinate values in the y direction of the previous silhouette image and the next silhouette image.

7. An electronic device / image signal generator / network device / transmitter / terminal / base station / industrial computer, characterized by comprising: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method as claimed in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: when the computer program runs on the processor, the processor is caused to execute the method as claimed in any one of claims 1-5.

Images