Abnormal behavior detection method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses an abnormal behavior detection method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; a prediction image is generated according to the target saliency map, the Euclidean distance between the prediction image and a second image in the to-be-recognized video is determined, the Euclidean distance is used for reflecting the similarity between the prediction image and the second image, and the second image is a next frame image adjacent to the first image in the to-be-recognized video; and determining whether an abnormal behavior exists in the to-be-identified video according to the Euclidean distance and a preset distance threshold. According to the method and the device, the technical problems that the abnormal behavior judgment process is relatively high in cost and the accuracy cannot be guaranteed due to a method for manually judging whether the abnormal behavior exists in the video or not in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of video data processing, and in particular, to an abnormal behavior detection method, device, non-volatile storage medium and electronic device. Background Art

[0002] Video anomaly detection technology plays a vital role in the judicial field, including supervision of correctional personnel, prediction of extreme behaviors caused by escalating conflicts among the people, mass gatherings, upward risks, etc. However, existing video anomaly detection usually requires manual judgment to determine whether there is abnormal behavior in the video. This results in a high cost of the video anomaly detection process and the accuracy cannot be guaranteed.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an abnormal behavior detection method, device, non-volatile storage medium and electronic device to at least solve the technical problem that the abnormal behavior judgment process is costly and the accuracy cannot be guaranteed due to the method of manually judging whether there is abnormal behavior in the video in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for detecting abnormal behavior is provided, including: determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; generating a predicted image based on the target saliency map, and determining a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0006] Optionally, determining a local feature prior information saliency map of a first image in a video to be identified includes: performing superpixel segmentation processing on the first image to determine a plurality of superpixel blocks in the first image; determining a plurality of local feature prior information saliency maps of the first image based on the plurality of superpixel blocks or the first image, wherein the plurality of local feature prior information saliency maps include a center prior saliency map, a target prior saliency map and a dark channel information saliency map.

[0007] Optionally, fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain the target saliency map includes: calculating the Kronecker product matrix of multiple local feature prior information saliency maps, and using the Kronecker product matrix as the fusion result of the multiple local feature prior information saliency maps; and fusing the fusion result and the global feature prior information saliency map using a linear fusion method to obtain the target saliency map.

[0008] Optionally, determining a global feature prior information saliency map of a first image in a to-be-identified video includes: extracting global saliency features of the first image through a visual geometry group network; and generating a global feature prior information saliency map based on the global saliency features of the first image.

[0009] Optionally, the visual geometry group network includes a feature extraction module, a pixel reasoning module and an output module, wherein the feature extraction module includes multiple convolutional layers, and the number of convolutional layers in the feature extraction module is not less than 15 layers, wherein the feature extraction module is also provided with a first preset convolutional layer connected to the fourteenth convolutional layer and a second preset convolutional layer connected to the fifteenth convolutional layer, and the convolution kernel sizes of the first preset convolutional layer and the second preset convolutional layer are different; the feature extraction module also includes a sparse sampling unit, and the sparse sampling unit is used to perform sparse sampling on the features extracted by the first thirteen convolutional layers of the multiple convolutional layers; the pixel reasoning module includes a random deactivation layer with a deactivation rate of a preset value; the output module includes a full convolutional layer and a deconvolutional layer.

[0010] Optionally, determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold includes: determining whether the second image is an abnormal frame based on the Euclidean distance and the preset distance threshold; and determining that there is abnormal behavior in the video to be identified when the second image is determined to be an abnormal frame.

[0011] Optionally, generating a predicted image based on the target saliency map includes: inputting the target saliency maps of multiple consecutive frames of the first image into a video prediction framework network of a parallel prefix scanning attention structure, and obtaining a predicted image output by the video prediction framework network.

[0012] According to another aspect of an embodiment of the present application, an abnormal behavior detection device is also provided, including: a first processing module, used to determine a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; a second processing module, used to fuse the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; a third processing module, used to generate a predicted image based on the target saliency map, and determine the Euclidean distance between the predicted image and the second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; a fourth processing module, used to determine whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute an abnormal behavior detection method.

[0014] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the abnormal behavior detection method is executed when the program is run.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the abnormal behavior detection method is implemented.

[0016] In an embodiment of the present application, a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified are determined; the local feature prior information saliency map and the global feature prior information saliency map are fused to obtain a target saliency map; a predicted image is generated based on the target saliency map, and the Euclidean distance between the predicted image and a second image in the video to be identified is determined, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; a method of determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold is used, by generating a predicted image based on the saliency map of the first image, and comparing the predicted image with the actual second image to determine whether there is abnormal behavior in the video, thereby achieving the purpose of automatically determining whether there is abnormal behavior in the video to be identified, thereby achieving the technical effect of reducing the cost of judging abnormal behavior in the video and improving the accuracy, thereby solving the technical problem that the cost of the abnormal behavior judgment process is high and the accuracy cannot be guaranteed due to the method of manually judging whether there is abnormal behavior in the video in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a structural schematic diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an abnormal behavior detection method provided according to an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a process for generating a local feature saliency map according to an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of a process for generating a global feature saliency map according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a target saliency map provided according to an embodiment of the present application;

[0023] Figure 6 It is a structural diagram of an Inception network module with dimensionality reduction provided according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of a second image when it is determined that there is no abnormal behavior according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of a second image when it is determined that abnormal behavior exists according to an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of a threshold determination method provided according to an embodiment of the present application;

[0027] Fig.10 It is a structural schematic diagram of an abnormal behavior detection device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] Salient features: Based on the external visual information obtained by the visual system, a certain prominence result is formed under the competition between neurons. Through feature representation, it can achieve a prominent effect in space and truly reflect the significant features in the region.

[0032] Parallel Prefix Scanning Attention Structure: It is a special recurrent neural network (RNN) structure used to learn long-term dependencies. In standard RNNs, due to the problem of gradient vanishing or gradient exploding, the model often has difficulty capturing dependencies over long time intervals. The video prediction framework of the parallel prefix scanning attention structure solves this problem by introducing a gating mechanism, allowing the network to store and access information in longer sequences.

[0033] Inception network: The Inception network is used to extract image features. It optimizes the traditional convolutional neural network structure, adds a normalization layer, reduces input data noise, and maintains the feature space consistency of the output data.

[0034] Reconstruction error: By calculating the difference or loss between the reconstructed data and the original data, we can determine the model’s ability to restore the input data and identify abnormal data accordingly.

[0035] Superpixel segmentation: Superpixel segmentation is an image segmentation method that divides an image into small blocks with similar features, called superpixels. By performing superpixel segmentation on an image, various features of the image can be obtained, such as color, texture, shape, etc.

[0036] Central Prior Significance Map: Central Prior Significance Map is a graphical representation method used to display central prior information. In statistics, central prior refers to prior information about a parameter, which is usually used in Bayesian statistical inference. The central prior significance map displays the central prior information in the form of a graph, which can help researchers understand the prior distribution of parameters more intuitively, and thus better perform statistical inference and decision-making. This graph usually includes information such as the prior distribution curve, mean, confidence interval, etc. of the parameter. Through the central prior significance map, you can see the prior information of the parameter more clearly, so as to perform statistical inference analysis more accurately.

[0037] Target Prior Saliency Map: Target Prior Saliency Map is an image processing technique used to study how the human visual system selects and focuses on specific targets. This technique can determine which areas humans usually focus on when viewing images by analyzing a large amount of visual data and experimental results, and mark these areas as "salient" areas. Target Prior Saliency Map can be used to improve the performance of computer vision systems, enabling them to better mimic human visual behavior, thereby improving the accuracy and efficiency of image recognition and analysis.

[0038] Local saliency map of dark channel information: In the field of image processing, the local saliency map of dark channel information refers to an image that reflects the local saliency of an image by processing the dark channel information. Dark channel information refers to the channel value of the minimum value among the surrounding pixels at each pixel position in the image. By calculating the dark channel information and performing appropriate processing, the local saliency information of each area in the image can be obtained. Local saliency maps can be used in applications such as image segmentation and target detection.

[0039] Global feature prior information saliency map: The global feature prior information of an image refers to some general rules or features in the image. These features exist in most images and can help better understand and process the image. The saliency map is an image generated by calculating the saliency of each pixel in the image based on this global feature prior information. The saliency map can help identify important objects or areas in the image more quickly, thereby improving the efficiency of image analysis and understanding.

[0040] At present, video anomaly detection technology plays a vital role in the judicial field, including the supervision of correctional personnel, predicting extreme behaviors caused by escalating conflicts among the people, mass gatherings, upward risks, etc. Its background technology includes the widespread application of video surveillance technology, intelligent analysis of massive video data, and detection technology for timely detection of abnormal states or events in videos. The development of these technologies has promoted the improvement of judicial intelligence and wisdom.

[0041] Specifically, abnormal behavior detection has become a research hotspot in computer vision. Abnormal behavior detection aims to model and analyze the behavior of pedestrians in the video, distinguish normal and abnormal behaviors in the crowd, and promptly discover and correct or mediate abnormal behaviors of relevant personnel or group events. The task of anomaly detection is to find the boundary between normal samples and abnormal samples, and to separate normal samples from abnormal samples as much as possible. Abnormal behavior in a video refers to an event in which a video appears that is significantly different from the video under normal circumstances, such as: a group gathering of P2P thunderstorms, a parade caused by unfinished real estate projects after mediation is invalid, etc. In addition, video anomaly detection is limited by the multiple influences of the environment, action, and object. For example, the action of "chopping wood with an axe" is likely to be an armed injury incident if it occurs on a city street, but it is likely to be a normal and common behavior if it occurs in a farmyard. Therefore, the cost of manual identification of abnormal behavior detection is high. Compared with manual identification, intelligent video systems are stable, reliable, cheap and practical. Manual identification requires labor costs and the effect is unstable, and it is easily affected by the human state. Intelligent video systems can use machines to perform automatic identification, saving labor costs and achieving stable results. Therefore, it is of great significance to apply intelligent video systems to detect abnormal behavior. If abnormal behavior is found, responses should be taken according to the degree of danger of the abnormal behavior and the scenario, so as to promptly stop the risk of extreme behavior or judicial escalation.

[0042] In order to solve the problem of high cost and uncertain accuracy caused by manual identification of whether there is abnormal behavior in a video, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0043] According to an embodiment of the present application, a method embodiment of an abnormal behavior detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an abnormal behavior detection method. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0045] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal behavior detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned abnormal behavior detection method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0049] In the above operating environment, the embodiment of the present application provides a method for detecting abnormal behavior, such as Figure 2 As shown, the method comprises the following steps:

[0050] Step S202, determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified;

[0051] In the technical solution provided in step S202, the step of determining a local feature prior information saliency map of the first image in the video to be identified includes: performing superpixel segmentation processing on the first image to determine multiple superpixel blocks in the first image; determining multiple local feature prior information saliency maps of the first image based on the multiple superpixel blocks or the first image, wherein the multiple local feature prior information saliency maps include a center prior saliency map, a target prior saliency map and a dark channel information saliency map.

[0052] As an optional implementation, a multi-granularity segmentation strategy may be used when performing superpixel segmentation. Specifically, during segmentation, the peak distribution of the image color component histogram in common data may be counted, and the number of pixels in the superpixel set may be set to values ​​of different granularities, such as {100, 150, 200, 250}, to perform superpixel segmentation on the image.

[0053] When performing segmentation, the main peaks with obvious peak trends can be identified from the histogram of the image. Then, the number of initial superpixels can be preliminarily determined based on the number and distribution of the main peaks. Then, based on the peak distribution of the histogram, the cluster centers are initialized to cover the main colors or color intervals in the image to ensure the representativeness of the superpixels. Finally, clustering methods such as K-means can be used for iterative clustering to assign the pixels in the image to each cluster center to form the initial superpixel block, thereby completing the superpixel segmentation.

[0054] As an optional implementation, Figure 3 As shown in , based on the superpixel segmentation results, a local saliency map based on center prior, target prior, and dark channel information can be calculated, and then the Kronecker product method is used to fuse the local saliency maps. Figure 3 It can be seen that by performing superpixel segmentation on the image, the texture features, RGB features and CELab features (image features based on color, contrast and structure) of the image can be obtained.

[0055] Specifically, when generating the center prior saliency map, the following formula can be used to describe the degree of deviation of each pixel area in the image, and the center prior saliency weight can be obtained, w x 、w y Respectively represent the length and width of the image, Ce(r i ) represents the final prior weight coefficient:

[0056] Ce(r i )=exp(-(x-x0) 2 / (2w x 2 )-(y-y0) 2 / (2w y 2 ))

[0057] In the above formula, (x, y) is the coordinate of any pixel point, (x0, y0) is the coordinate of the center point of the image. The coordinate system is a plane rectangular coordinate system established in the plane where the image is located, and then the center prior saliency map can be generated according to the prior weight coefficient.

[0058] When generating a target priori saliency map, the center point can be set as any pixel q. On this basis, several windows containing the point are selected, and the occurrence frequency of the salient target in each window is calculated, thereby obtaining the actual row-by-row occurrence rate of the window target. Subsequently, all probability values ​​are summarized and summed to obtain the probability that the target exists at pixel q. It can be expressed by the following formula, where Ob(q) refers to the probability measure of the existence of the target at pixel q; p(w q ) refers to the window w q The probability value of the existence of the target, W represents the size of the entire image, and w represents the window size.

[0059]

[0060] Then the central prior saliency map is generated according to the probability value.

[0061] The saliency map of the dark channel information is mainly reflected in most non-sky image blocks. The RGB values ​​of individual pixels in the salient area will have a minimum value close to 0. The following is the specific calculation formula, where patch (p) refers to a specific image block centered on q, K refers to the number of image blocks, p describes the image block centered on q, d describes the dark channel grayscale value, and I ch (q) refers to the true color value of q. The following is the calculation formula for calculating the average value of the .ir dark channel:

[0062]

[0063] Then, a saliency map of the dark channel information can be generated based on the dark channel average.

[0064] In some embodiments of the present application, a flowchart for determining a global feature prior information saliency map of a first image in a video to be identified is as follows: Figure 4 As shown, it includes: extracting global salient features of the first image through a visual geometry group network; generating a global feature prior information salient map based on the global salient features of the first image.

[0065] Optionally, the visual geometry group network includes a feature extraction module, a pixel reasoning module and an output module, wherein the feature extraction module includes multiple convolutional layers, and the number of convolutional layers in the feature extraction module is not less than 15, wherein the feature extraction module is also provided with a first preset convolutional layer connected to the fourteenth convolutional layer and a second preset convolutional layer connected to the fifteenth convolutional layer, and the convolution kernel sizes of the first preset convolutional layer and the second preset convolutional layer are different; the feature extraction module also includes a sparse sampling unit, and the sparse sampling unit is used to perform sparse sampling on the features extracted by the first thirteen convolutional layers of the multiple convolutional layers; the pixel reasoning module includes a random deactivation layer with a deactivation rate of a preset value; the output module includes a full convolutional layer and a deconvolutional layer.

[0066] The above-mentioned visual geometry group network (VGG network model) can reasonably extract deep feature values ​​with the help of convolutional networks, and at the same time predict salient values ​​with the help of convolutional networks, and form a salient map with basically the same size as the image.

[0067] Specifically, Figure 4As shown, the abnormal behavior detection method provided in the embodiment of the present application will perform sparse sampling on the feature maps of the first 13 layers to improve the generalization processing capability of the network. Two convolution layers of different sizes are added to the adjacent areas of the 14th and 15th layers of the convolution part, and the convolution kernel sizes are 1*1 and 7*7 respectively. In order to ensure the rationality of the prediction, a random inactivation layer is introduced in the pixel reasoning part with a inactivation rate of 60%. Secondly, in order to ensure that the sizes of the input and output images are basically the same and the globality is preserved intact, a full convolution layer and one deconvolution layer are reasonably introduced here. Among them, the kernel size of the full convolution layer is 1*1, and the size of the kernel of the deconvolution layer is: 63*63, step size 31, Padding14.

[0068] It should be noted that the purpose of saliency detection is to make a reasonable inference as to whether the pixels in the image belong to the background image or the foreground image. Therefore, in the embodiment of the present application, a cross entropy loss is designed in the network output part of the above-mentioned visual geometry group network to classify the relevant salient features of the output layer, thereby forming a corresponding saliency map.

[0069] Step S204, fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map;

[0070] In the technical solution provided in step S204, the step of fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain the target saliency map includes: calculating the Kronecker product matrix of multiple local feature prior information saliency maps, and using the Kronecker product matrix as the fusion result of the multiple local feature prior information saliency maps; fusing the fusion result and the global feature prior information saliency map by linear fusion to obtain the target saliency map. Figure 5 The target saliency map shown. Among them, Figure 5 The local saliency map in represents the fusion result of each local feature prior information saliency map, the depth saliency map represents the global feature prior information saliency map, and the target saliency map represents the target saliency map obtained by linearly fusion of the local saliency map and the depth saliency map.

[0071] As an optional implementation, in addition to using the Kronecker product (Kronecker product matrix) to fuse the local feature prior information saliency map, the average of the saliency values ​​of each pixel point in the first image in different local feature prior information saliency maps can also be directly used as its saliency value to obtain the above-mentioned fusion result.

[0072] Step S206, generating a predicted image according to the target saliency map, and determining a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified;

[0073] In the technical solution provided in step S206, the step of generating a predicted image according to the target saliency map includes: inputting the target saliency maps of the continuous plurality of frames of the first image into the Figure 6 The Inception network with dimensionality reduction shown in Figure 1 and obtain the predicted image output by the Inception network with dimensionality reduction.

[0074] In some embodiments of the present application, Figure 6 The data set used in the training process of the network shown in FIG includes positive samples that do not contain abnormal behaviors and negative samples that contain abnormal behaviors. In addition, it should be noted that since the same behavior may be normal or abnormal in different scenarios, the positive and negative samples mentioned above also carry scene labels.

[0075] As an optional implementation, a video prediction framework network structure can be used to perform predictions based on convolutional feature extraction calculations and input into a parallel prefix scanning attention structure, and finally the detection of abnormal frames in the video can be achieved by using reconstruction errors. However, since the convolutional network has a high feature conversion mapping capability, the convolutional network is also easy to map abnormal frames to the normal feature domain, resulting in a lower recall rate of abnormal frames. Therefore, feature fusion is used in the convolutional calculation to extract features, so that the feature extraction model is robust in learning abnormal areas.

[0076] Step S208: determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0077] In the technical solution provided in step S208, the step of determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and the preset distance threshold includes: determining whether the second image is an abnormal frame based on the Euclidean distance and the preset distance threshold; and determining that there is abnormal behavior in the video to be identified if the second image is an abnormal frame. Figure 7 When the second image is a normal image as shown, the judgment result watermark and the abnormal score watermark can be stamped on the second image according to the calculated distance. The abnormal score can be determined by the Euclidean distance. Figure 8 When an abnormal image is shown, a watermark indicating abnormal judgment result and abnormal score can also be stamped.

[0078] As an optional implementation, after determining the abnormal frames in the video to be identified, a comprehensive judgment can be made on each abnormal frame to determine the area where the abnormal behavior occurs. Fig. 9The time of each frame image and the time of the anomaly score - the anomaly degree value curve can accurately reflect the degree of anomaly at the corresponding moment in the video. The resulting significant area can identify the area where abnormal behavior occurs, thereby providing timely feedback and early warning prompts for predicting the escalation of conflicts among the people, leading to extreme behaviors, mass gatherings, and upward risks in the integrated application of the rule of law government.

[0079] The above anomaly score can be determined by Euclidean distance, and the closer the Euclidean distance is, the smaller the degree of anomaly is, and vice versa. The Euclidean distance can also be directly used as an indicator of the degree of anomaly.

[0080] In some embodiments of the present application, Fig. 9 As shown, the anomaly score threshold can be determined according to the anomaly score corresponding to each frame image in the video ( Fig. 9 ), and determine whether a frame of image is an abnormal image based on the threshold. Specifically, it can be determined that an image with an abnormal score greater than the abnormal score threshold is an abnormal image.

[0081] The invention adopts the method of determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; generating a predicted image based on the target saliency map, and determining the Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold, generating a predicted image based on the saliency map of the first image, and comparing the predicted image with the actual second image to determine whether there is abnormal behavior in the video, thereby achieving the purpose of automatically determining whether there is abnormal behavior in the video to be identified, thereby achieving the technical effect of reducing the cost of abnormal behavior judgment in the video and improving the accuracy, thereby solving the technical problem that the cost of the abnormal behavior judgment process is high and the accuracy cannot be guaranteed due to the method of manually judging whether there is abnormal behavior in the video in the related art.

[0082] The present application embodiment provides an abnormal behavior detection device, Fig.10 is a schematic diagram of the structure of the device, such as Fig.10As shown, the device includes: a first processing module 100, used to determine a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; a second processing module 102, used to fuse the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; a third processing module 104, used to generate a predicted image based on the target saliency map, and determine the Euclidean distance between the predicted image and the second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; a fourth processing module 106, used to determine whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0083] In some embodiments of the present application, the step of the first processing module 100 determining the local feature prior information saliency map of the first image in the video to be identified includes: performing superpixel segmentation processing on the first image to determine multiple superpixel blocks in the first image; determining multiple local feature prior information saliency maps of the first image based on the multiple superpixel blocks or the first image, wherein the multiple local feature prior information saliency maps include a center prior saliency map, a target prior saliency map and a dark channel information saliency map.

[0084] In some embodiments of the present application, the first processing module 100 determines the global feature prior information saliency map of the first image in the video to be identified, including: extracting the global saliency features of the first image through a visual geometry group network; and generating a global feature prior information saliency map based on the global saliency features of the first image.

[0085] In some embodiments of the present application, the visual geometry group network includes a feature extraction module, a pixel reasoning module and an output module, wherein the feature extraction module includes multiple convolutional layers, and the number of convolutional layers in the feature extraction module is not less than 15 layers, wherein the feature extraction module is also provided with a first preset convolutional layer connected to the fourteenth convolutional layer and a second preset convolutional layer connected to the fifteenth convolutional layer, and the convolution kernel sizes of the first preset convolutional layer and the second preset convolutional layer are different; the feature extraction module also includes a sparse sampling unit, and the sparse sampling unit is used to perform sparse sampling on the features extracted by the first thirteen convolutional layers in the multiple convolutional layers; the pixel reasoning module includes a random deactivation layer with a deactivation rate of a preset value; the output module includes a full convolutional layer and a deconvolution layer.

[0086] In some embodiments of the present application, the second processing module 102 fuses the local feature prior information saliency map and the global feature prior information saliency map to obtain the target saliency map, including: calculating the Kronecker product matrix of multiple local feature prior information saliency maps, and using the Kronecker product matrix as the fusion result of the multiple local feature prior information saliency maps; and fusing the fusion result and the global feature prior information saliency map using a linear fusion method to obtain the target saliency map.

[0087] In some embodiments of the present application, the step in which the fourth processing module 106 determines whether there is abnormal behavior in the video to be identified based on the Euclidean distance and the preset distance threshold includes: determining whether the second image is an abnormal frame based on the Euclidean distance and the preset distance threshold; if it is determined that the second image is an abnormal frame, determining that there is abnormal behavior in the video to be identified.

[0088] In some embodiments of the present application, the step of the fourth processing module 106 generating a predicted image based on the target saliency map includes: inputting the target saliency maps of multiple consecutive frames of the first image into a video prediction framework network of a parallel prefix scanning attention structure, and obtaining the predicted image output by the video prediction framework network.

[0089] It should be noted that the various modules in the above-mentioned abnormal behavior detection device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0090] An embodiment of the present application provides a non-volatile storage medium, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following abnormal behavior detection method: determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; generating a predicted image based on the target saliency map, and determining a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0091] An embodiment of the present application provides an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the following abnormal behavior detection method is executed when the program is run: determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; generating a predicted image based on the target saliency map, and determining a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0092] An embodiment of the present application provides a computer program product, including a computer program, which implements the following abnormal behavior detection method when executed by a processor: determining a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; generating a predicted image based on the target saliency map, and determining a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is the next frame image adjacent to the first image in the video to be identified; determining whether there is abnormal behavior in the video to be identified based on the Euclidean distance and a preset distance threshold.

[0093] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0098] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting abnormal behavior, characterized in that: include: Determine a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; Fusing the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; Generate a predicted image according to the target saliency map, and determine a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is a next frame image adjacent to the first image in the video to be identified; It is determined whether there is abnormal behavior in the video to be identified according to the Euclidean distance and a preset distance threshold.

2. The abnormal behavior detection method according to claim 1, characterized in that: Determining a saliency map of local feature prior information of a first image in a video to be identified includes: Performing superpixel segmentation processing on the first image to determine a plurality of superpixel blocks in the first image; A plurality of the local feature prior information saliency maps of the first image are determined based on the plurality of the superpixel blocks or the first image, wherein the plurality of the local feature prior information saliency maps include a center prior saliency map, a target prior saliency map and a dark channel information saliency map.

3. The abnormal behavior detection method according to claim 2, characterized in that: Fusion of the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map includes: Calculating the Kronecker product matrix of the plurality of local feature prior information saliency maps, and using the Kronecker product matrix as a fusion result of the plurality of local feature prior information saliency maps; The fusion result and the global feature prior information saliency map are fused in a linear fusion manner to obtain the target saliency map.

4. The abnormal behavior detection method according to claim 1, characterized in that: Determining a global feature prior information saliency map of a first image in a video to be identified includes: Extracting global saliency features of the first image through a visual geometry group network; The global feature prior information saliency map is generated according to the global saliency features of the first image.

5. The abnormal behavior detection method according to claim 4, characterized in that: The visual geometry group network includes a feature extraction module, a pixel reasoning module and an output module, wherein: The feature extraction module includes a plurality of convolutional layers, and the number of convolutional layers in the feature extraction module is not less than 15 layers, wherein the feature extraction module is further provided with a first preset convolutional layer connected to the fourteenth convolutional layer and a second preset convolutional layer connected to the fifteenth convolutional layer, and the convolution kernel sizes of the first preset convolutional layer and the second preset convolutional layer are different; The feature extraction module also includes a sparse sampling unit, which is used to perform sparse sampling on the features extracted by the first thirteen convolutional layers of the multiple convolutional layers; The pixel reasoning module includes a random deactivation layer with a deactivation rate of a preset value; The output module includes a full convolution layer and a deconvolution layer.

6. The abnormal behavior detection method according to claim 1, characterized in that: Determining whether there is abnormal behavior in the video to be identified according to the Euclidean distance and the preset distance threshold includes: determining whether the second image is an abnormal frame according to the Euclidean distance and a preset distance threshold; When it is determined that the second image is an abnormal frame, it is determined that abnormal behavior exists in the video to be identified.

7. The abnormal behavior detection method according to claim 1, characterized in that: Generating a predicted image according to the target saliency map includes: The target saliency map of the first image of multiple consecutive frames is input into a video prediction framework network of a parallel prefix scanning attention structure, and a predicted image output by the video prediction framework network is obtained.

8. An abnormal behavior detection device, characterized in that: include: A first processing module, used to determine a local feature prior information saliency map and a global feature prior information saliency map of a first image in a video to be identified; A second processing module is used to fuse the local feature prior information saliency map and the global feature prior information saliency map to obtain a target saliency map; a third processing module, configured to generate a predicted image according to the target saliency map, and determine a Euclidean distance between the predicted image and a second image in the video to be identified, wherein the Euclidean distance is used to reflect the similarity between the predicted image and the second image, and the second image is a next frame image adjacent to the first image in the video to be identified; The fourth processing module is used to determine whether there is abnormal behavior in the video to be identified according to the Euclidean distance and a preset distance threshold.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the abnormal behavior detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the abnormal behavior detection method according to any one of claims 1 to 7 is executed when the program is run.

11. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the abnormal behavior detection method according to any one of claims 1 to 7.

Images