Traffic anomaly detection method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a traffic anomaly detection method, electronic equipment and a storage medium. The method comprises the following steps: acquiring continuous multi-frame traffic videos to be detected; the RGB images and the optical flow images of the previous several frames of videos are coded through the two encoders respectively; performing multi-scale attention fusion on the global RGB features and the global optical flow features output by the last layer of the two encoders to obtain deep fusion features of the RGB and the optical flow; reconstructing the deep fusion feature and the global optical flow feature according to normal features recorded in the two memory modules; respectively decoding the two reconstruction features through two decoders, and respectively restoring an RGB image and an optical flow image of the last frame of video; and according to the difference between each restored image and the original image and the difference between each reconstructed feature and the normal feature, judging whether the last frame of video has a traffic abnormal risk. According to the embodiment, the accuracy of traffic anomaly detection is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of traffic detection, and in particular to a traffic anomaly detection method, electronic equipment and storage medium. Background Art

[0002] With the development of transportation, traffic accidents, congestion, nighttime environment and other traffic anomalies have an increasingly significant impact on road safety and traffic efficiency. How to use traffic videos to accurately detect traffic anomalies is an urgent problem to be solved.

[0003] There are also some methods in the prior art that use video data to automatically detect traffic anomalies. For example, patent CN105405297B provides an automatic detection method for traffic accidents based on surveillance videos, and patent CN113221716A provides an unsupervised abnormal traffic behavior detection method based on foreground target detection, but the accuracy of detection still needs to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a traffic anomaly detection method, an electronic device and a storage medium to improve the accuracy of traffic anomaly detection.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting traffic anomalies, comprising: S110, obtaining a continuous multi-frame traffic video to be detected; S120, respectively encoding the RGB images and optical flow images of the first few frames of video through two encoders, and each layer of each encoder outputs multi-layer RGB features and multi-layer optical flow features respectively; wherein, when encoding the RGB image, the RGB features output by each layer of the RGB encoder are added to the optical flow features output by the corresponding layer of the optical flow encoder and then input into the next layer of the RGB encoder, so as to realize the shallow information interaction between RGB and optical flow; S130, performing multi-scale attention fusion on the global RGB features and the global optical flow features output by the last layer of the two encoders to obtain deep fusion features of RGB and optical flow; S140, reconstructing the deep fusion feature and the global optical flow feature respectively according to the normal features recorded in the two memory modules, wherein the normal features in the two memory modules respectively represent the characteristics of the deep fusion feature and the global optical flow feature of the normal traffic video; S150, respectively decoding the two reconstructed features through two decoders to restore the RGB image and the optical flow image of the last frame of video respectively; S160: judging whether the last frame of video has a traffic abnormality risk according to the difference between each restored image and the original image, and the difference between each reconstructed feature and the normal feature.

[0006] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the traffic anomaly detection method described in any embodiment.

[0007] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the traffic anomaly detection method described in any embodiment.

[0008] In summary, the embodiments of the present invention provide a method, electronic device and storage medium for detecting traffic anomalies, introduce optical flow data, use information interaction between streams to fully fuse RGB and optical flow data, so as to more accurately capture abnormal features, and perform multi-scale attention fusion on the captured dual-stream features to better reflect contextual information and multi-scale information; then, feature reconstruction is completed through a memory module matched with the dual-stream framework, further reducing the probability of false detection in normal situations. This embodiment uses the end-to-end structure of the dual-stream network to train the model, which greatly improves the efficiency of model training and prediction; and uses a depth-based prediction loss function to adapt to the regular shape of the vehicle and events at different distances under the surveillance camera, enhancing the accuracy and robustness of the model in detecting anomalies. At the same time, considering that the data set dedicated to traffic anomaly detection usually only includes a single type of anomaly (i.e., traffic accidents), this embodiment introduces a traffic monitoring data set for full-class anomaly detection model verification, and a full-class anomaly detection in traffic scenes can be achieved through a model, further improving detection efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 is a flow chart of a traffic anomaly detection method provided by an embodiment of the present invention; Figure 2 is a schematic diagram of multiple frames of RGB images and optical flow images provided by an embodiment of the present invention; Figure 3 is a flow chart of another traffic anomaly detection method provided by an embodiment of the present invention; Figure 4A schematic diagram of a multi-scale attention fusion module provided by an embodiment of the present invention; Figure 5 A schematic diagram of feature reconstruction through a memory module provided by an embodiment of the present invention; Figure 6 A comparison diagram of the TSD provided by the embodiment of the present invention and the existing public transportation data set, wherein: Figure 6 (a) is a schematic diagram of the AI ​​City dataset. Figure 6 (b) is a schematic diagram of the DoTA dataset. Figure 6 (c) is a schematic diagram of the TSD of this embodiment; Figure 7 A schematic diagram of a training method for a traffic anomaly detection model provided by an embodiment of the present invention; Figure 8 A schematic diagram of updating a memory module provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0012] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0013] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0014] Figure 1 is a flow chart of a traffic anomaly detection method provided by an embodiment of the present invention. The method is applicable to traffic anomaly detection on roads where video acquisition equipment is deployed, and is particularly applicable to traffic anomaly detection on highways. The method is executed by an electronic device, such as Figure 1 As shown, the method specifically includes: S110, obtaining a plurality of continuous frames of traffic video to be detected, wherein each frame of the video includes an RGB image and an optical flow image.

[0015] This embodiment will use RGB images and optical flow images to jointly detect road traffic anomalies. Figure 2 The RGB image and optical flow image of multiple frames of video in the traffic scene are shown as an example, where T represents the time of each frame of video. Optionally, the RGB image and optical flow image of multiple frames of video can be obtained through roadside monitoring as the data source of the whole method.

[0016] S120. Encode the RGB images and optical flow images of the first few frames of video respectively through two encoders, and each layer of each encoder outputs multi-layer RGB features and multi-layer optical flow features respectively; wherein, when encoding the RGB image, the RGB features output by each layer of the RGB encoder are added to the optical flow features output by the corresponding layer of the optical flow encoder and then input into the next layer of the RGB encoder, so as to realize shallow information interaction between RGB and optical flow.

[0017] This embodiment uses a dual-stream architecture to process the RGB images and optical flow images of the first few frames of video respectively to extract traffic anomaly information in the data, and predict the RGB image and optical flow image of the last frame of video based on the information. The entire data processing flow is as follows: Figure 3 As shown, this step corresponds to the encoding module in the figure. Figure 3 , perform the following operations in the encoding module: First, the RGB images of the first few frames of video are encoded by encoder 1, and each layer of encoder 1 outputs a layer of RGB features, wherein the size of the RGB features gradually decreases as the number of layers increases. At the same time, the optical flow images of the first few frames of video are encoded by encoder 2, and each layer of encoder 2 outputs a layer of optical flow features, wherein the size of the optical flow features gradually decreases as the number of layers increases. Furthermore, encoder 1 and encoder 2 have the same structure, and the feature size outputted by the same layer is also the same. In order to realize the shallow information interaction between RGB and optical flow, the output features of each layer of encoder 1 are first added to the output features of the same layer in encoder 2 before entering the next layer, and then the result of the addition is input to the next layer of encoder 1. In this way, shallow optical flow feature maps of different scales can be integrated into the feature extraction process of the RGB feature encoder, thereby realizing comprehensive information interaction between RGB and optical flow. The entire operation can be expressed as follows:

[0018] in, represents the original sequence optical flow image (i.e., the optical flow images of the first few frames of video). For example, assuming that N frames of video are obtained in S110, then The optical flow image of the first N-1 frames of video; represents the encoding operation of encoder 2; Represents the global optical flow feature map output by the last layer of encoder 2, Before encoder 2 b Layer output Optical flow features of different sizes, b +1 is the total number of encoder layers; Represents the encoding operation of encoder 1; represents the global RGB features output by the last layer of encoder 1, ,in are the number of channels, width and height respectively. Optionally, the two encoders can adopt a 10-layer convolution and pooling structure that is mixed with each other. b +1=10. In practical applications, other encoder structures may also need to be selected, and this embodiment does not make any specific limitation.

[0019] S130, performing multi-scale attention fusion on the global RGB features and global optical flow features output by the last layer of the two encoders to obtain a deep fusion feature of RGB and optical flow :

[0020] in, Represents a multi-scale attention fusion operation. Optional, Figure 4A schematic diagram of a multi-scale attention fusion module provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, in this module, first and They are used as initial features, and self-attention and normalization are performed to enhance the RGB and optical flow feature representations respectively; then a simple element-by-element addition is performed to achieve initial feature fusion, as detailed below:

[0021] in, represents the initial fused features (or called preliminary fused features), represents the self-attention mechanism, represents batch normalization, Indicates the addition of corresponding elements.

[0022] Then, the preliminary fusion features are convolved to extract local features. In order to cope with the scale changes of different information in real traffic scenes, especially to improve the analysis ability of small objects, this step combines contextual information with different receptive fields to integrate the preliminary fusion features. The multi-scale information in the convolution operation can be used to enhance the performance of the entire framework when dealing with multi-scale and small-sized objects. Optionally, local features of contextual information can be extracted through point-by-point convolution operations. :

[0023] in, Represent two convolutional layers respectively. represent Linear activation function.

[0024] Next, the local features Perform global average pooling to extract global features Then, based on the preliminary fusion features and global features, a deep fusion feature map of RGB and optical flow is generated. :

[0025] in, Represents the multiplication of corresponding matrix elements.

[0026] S140. Reconstruct the deep fusion features and the global optical flow features respectively according to the normal features in the two memory modules, wherein the normal features in the two memory modules respectively represent the characteristics of the deep fusion features and the global optical flow features of the normal traffic video.

[0027] This embodiment introduces a memory module to store the diverse features of normal traffic videos and prevent abnormal videos from being mistakenly identified as normal modes. Specifically, the memory module pre-stores multiple feature representations of normal traffic videos (referred to as normal features for short), which can be obtained in advance through normal traffic video training, can reflect the important characteristics of normal videos, and can also ensure the diversity and comprehensiveness of normal modes. The specific training process will be described in detail in subsequent implementations.

[0028] Furthermore, in order to adapt to the dual-stream processing framework of RGB images and shunt images, this embodiment constructs a memory module for RGB images and shunt images respectively. Among them, the memory module 1 corresponding to the RGB image stores normal features that can be compared with the above-mentioned deep fusion features, which can be constructed based on the deep fusion features generated by normal traffic videos during the training process. The memory module 2 corresponding to the optical flow image stores normal features that can be compared with the above-mentioned global optical flow features, which can be constructed based on the global optical flow features generated by normal traffic videos during the training process.

[0029] For any memory module, Figure 5 A schematic diagram of a method for reconstructing features through a memory module provided by an embodiment of the present invention. Figure 5 , the memory module stores M normal features, where the mth normal feature is recorded as . For the feature to be reconstructed (global optical flow feature or deep fusion feature), it can be divided into K blocks (or K items), and the distance between each item and each normal feature in the memory module is calculated respectively; the sum of the distances between the K items and the same normal feature is taken as the distance between the feature to be reconstructed and the normal feature. Finally, according to the distance between the feature to be reconstructed and each normal feature, the weighted sum of each normal feature is performed to obtain the reconstructed feature of the global optical flow feature. Optionally, the distance can be measured by cosine similarity and can also be normalized by a softmax function, such as Figure 5 shown.

[0030] S150, respectively decode the two reconstructed features through two decoders to restore the RGB image and the optical flow image of the last frame of video respectively.

[0031] This step corresponds to Figure 3 The decoding module in the decoder uses decoder 1 to decode the reconstructed features of the deep fusion features to restore the RGB image of the last frame of video; at the same time, decoder 2 is used to decode the reconstructed features of the global optical flow features to restore the optical flow image of the last frame of video.

[0032] S160: judging whether the last frame of video has a traffic abnormality risk according to the difference between each restored image and the original image, and the difference between each reconstructed feature and the normal feature.

[0033] The reconstructed features here still refer to the deep fusion features and the global optical flow features. As described in the memory module, the entire data processing architecture of this embodiment ( Figure 3 ) is constructed based on normal traffic videos. The above two differences corresponding to normal traffic videos will be significantly different from the above two differences of abnormal traffic videos, and thus can be used as a basis for determining whether the input video has a risk of traffic abnormality.

[0034] Optionally, this step calculates an anomaly score for each of the RGB restored image and the optical flow restored image. Taking the RGB restored image as an example, the predicted N-th frame RGB image can be calculated. Compared with the actual Nth frame RGB image Peak Signal-to-Noise Ratio (PSNR):

[0035] in, Represents the number of pixels in the video frame. The larger the PSNR, the greater the difference between the two images.

[0036] At the same time, the deep fusion feature and the nearest normal feature are calculated Distance:

[0037] in, represents the reconstructed feature, Represents the reconstructed feature The larger the distance, the greater the difference between the two features.

[0038] Finally, based on the PSNR and distance, the anomaly score for the RGB image is calculated. :

[0039] in, Normalization, represents the equilibrium parameter.

[0040] Regarding the anomaly score of the optical flow image, its calculation method is similar to that of the RGB image. The Nth frame RGB image is replaced with the Nth frame optical flow image, and the deep fusion feature is replaced with the global optical flow feature. After obtaining the two anomaly scores, the weighted average or the maximum value of the two can be used as the final anomaly score. When the final anomaly score is higher than the set threshold, it is considered that there is a risk of traffic anomaly in the video data to be detected. In practical applications, the method of this embodiment can be deployed on edge devices, and when the roadside monitoring device obtains the current time tAfter the video data is received, the edge device t Fast prediction of several frames of video data before the moment t The video data at each moment, and t Compare with the real video data at the moment to judge t Whether there is a risk of traffic abnormality at any time, and realize real-time detection of traffic abnormalities.

[0041] Furthermore, the entire Figure 3 The data processing flow constitutes a traffic anomaly detection model, and the training process of the model is expanded below. In a specific implementation, first, a traffic surveillance dataset (Traffic Surveillance dataset, TSD) is constructed. Specifically, the video anomaly datasets in the prior art focus more on pedestrians, lack attention to vehicle anomalies in road traffic scenes, and cannot effectively verify the anomaly detection of the road traffic environment. To this end, this embodiment constructs a new TSD for full-class anomaly detection in real traffic scenes. TSD consists of high-quality, unedited videos from real monitoring angles, covering various traffic anomalies occurring in 52 real-life scenes, including accidents, traffic jams, and some night shots. Table 1 lists the comparison of TSD with commonly used datasets for full-class anomaly detection. Figure 6 The intuitive comparison between TSD and some existing public transportation datasets is shown. Figure 6 The AICity dataset in (a) contains many non-road anomalies (such as camera shake), which will interfere with it; Figure 6 The DoTA dataset in (b) is based on the perspective of autonomous driving and cannot integrate abnormal data from the entire road network; Figure 6 The TSD of this embodiment in (c) is constructed based on the video surveillance angle and is a high-quality traffic scene full-category anomaly detection dataset.

[0042] Table 1

[0043] Then, from the normal traffic video data set of TSD, multiple frames of continuous video are extracted as training samples to obtain the training sample set of the traffic anomaly detection model. Due to the rarity of abnormal events, this embodiment uses part of the normal video data in TSD for model training. During the training process, the operations of S120-S150 are performed on each sample respectively, and the loss function is constructed by minimizing the difference between each restored image and the original image, and minimizing the difference between the reconstructed features and the normal features. Since there are large differences between normal traffic videos and abnormal traffic videos, through the above training process, it can be ensured that the trained model obtains a lower anomaly score for normal traffic videos and a higher anomaly score for abnormal traffic videos in actual detection, thereby effectively identifying whether there is a risk of traffic anomalies. After the training is completed, a part of the test samples can be collected from both the normal traffic video and the abnormal traffic video in the testing phase to verify the model's ability to recognize traffic anomalies.

[0044] Optionally, the following loss function can be constructed during training:

[0045] in, L represents the total loss, represent the depth-based prediction losses of RGB images and optical flow images, respectively. They represent the compactness loss of RGB normal features and optical flow normal features in the memory module, respectively. Respectively represent the separability loss of RGB normal features and optical flow normal features in the memory module. These losses are separated by hyperparameters Balance. The calculation method of each loss is explained in detail below.

[0046] First, the depth-based prediction loss is explained. In general, the prediction loss penalizes the inconsistency between the prediction and the actual situation, ensuring that the decoder's prediction closely matches the actual situation and that each pixel has the same weight. However, in a typical traffic monitoring scenario, the camera has a certain depth of field, which means that for pixel blocks of the same size, pixel blocks farther from the camera contain more scene information. Therefore, this embodiment adopts a depth-based prediction loss strategy to assign greater weights to pixels farther from the camera to increase their contribution, so that the adjustment process of the model is consistent with the real-world monitoring scene, which helps the model better understand and process real-world scene data and improve detection performance.

[0047] Optional, such as Figure 7As shown, for any restored image, it is first layered along the height direction, and then the pixels of each layer are average pooled at different scales according to the distance between each layer and the camera. For example, the bottom layer image is closest to the camera, so the bottom layer image is average pooled in a 64×64 block, and the upper layer image is relatively far from the camera, so the layer image is average pooled in a 32×32 block. In general, the closer to the camera, the larger the scale of the average pooling. This is because the main subjects in real traffic scenes (especially highways) are vehicles, and average pooling of different scales can adapt to the regular shape characteristics of traffic vehicle data. Taking RGB images as an example, the operation can be expressed as:

[0048] in, The image is divided into A layer; Indicates a The average pooling strategy used within the layer; Indicates the height of each layer.

[0049] Then, the average pooled image is divided into blocks, and the difference between each block and the original image block is weighted averaged according to the distance from each block to the camera to construct a depth-based prediction loss function. Still taking the RGB image as an example:

[0050] in, Respectively represent the number of pixel blocks into which the image is divided in the width and height dimensions; is assigned to the coordinate position The weight of the pixel block. The farther away from the camera, the greater the weight. Respectively represent the coordinate positions of the original image and the restored image The pixel value at . Through The L2 distance between the decoder output and the true value is scaled by the constraint Minimizing the distance between the decoder output and the true value can be minimized.

[0051] Then, combined with the update operation of the memory module during the training process, the feature compactness loss and feature separation loss are explained. Specifically, during the training process, after each sample completes the reconstruction of the global optical flow feature and the deep fusion feature, the normal features in the memory module will be updated. Figure 8 , the deep fusion features of any sample include K items, first select the normal feature farthest from the deep fusion feature I uAs the update object, then according to the items in the deep fusion feature and the normal feature I u The distance is weighted averaged for each item, and I u The weighted average result is replaced. Through feature updating, various normal features can be effectively recorded, thereby ensuring the diversity and time correlation of normal features.

[0052] Based on the reconstruction and update process of the memory module, taking the deep fusion feature as an example, the following feature compactness loss can be constructed:

[0053] in, Represents the normal feature closest to the reconstructed feature in the memory module. Feature compactness loss can make the reconstructed feature closer to the nearest normal feature in the memory module, thereby reducing the feature differences within the normal category.

[0054] At the same time, the following feature separability loss can be constructed:

[0055] in, represents the normal feature in the memory module that is the second closest to the reconstructed feature. A hyperparameter to maintain the basic distance between classes. The feature separability loss allows the normal features in the memory module to record different normal modes in various scenarios, thus maintaining the diversity of normal scenarios.

[0056] In the iterative training of each sample, the model parameters can be updated by minimizing the total loss L, wherein the depth-based prediction loss can ensure that the difference between each restored image and the original image is minimized, the feature compactness loss can ensure that the difference between the optical flow features or deep fusion features corresponding to each restored image and the normal features closest to their respective memory modules is minimized, and the feature separability loss can maximize the difference between each normal feature in each memory module.

[0057] In summary, this embodiment provides a method, electronic device and storage medium for traffic anomaly detection, introduces optical flow data, and uses information interaction between streams to fully fuse RGB and optical flow data to more accurately capture abnormal features, and fuses the captured dual-stream features with multi-scale attention to better reflect context and multi-scale information; the memory module matching the dual-stream framework completes feature reconstruction, further reducing the probability of false detection in normal situations. This embodiment uses the end-to-end structure of the dual-stream network to train the model, which greatly improves the efficiency of model training and prediction; and uses a depth-based prediction loss function to adapt to the regular shape of the vehicle and events at different distances under the surveillance camera, enhancing the accuracy and robustness of the model to detect anomalies. At the same time, considering that the data set dedicated to traffic anomaly detection usually only includes a single type of anomaly (i.e., traffic accidents), this embodiment introduces a traffic monitoring data set for full-class anomaly detection model verification, and a full-class anomaly detection in traffic scenes can be achieved through one model, further improving detection efficiency and accuracy.

[0058] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Fig. 9 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0059] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the traffic anomaly detection method in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, implementing the above-mentioned traffic anomaly detection method.

[0060] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0062] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the traffic anomaly detection method of any embodiment is implemented.

[0063] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0064] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0065] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0066] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, python, and conventional procedural programming languages, such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A traffic anomaly detection method, characterized in that: include: S110, obtaining a continuous multi-frame traffic video to be detected; S120, respectively encoding the RGB images and optical flow images of the first few frames of video through two encoders, and each layer of each encoder outputs multi-layer RGB features and multi-layer optical flow features respectively; wherein, when encoding the RGB image, the RGB features output by each layer of the RGB encoder are added to the optical flow features output by the corresponding layer of the optical flow encoder and then input into the next layer of the RGB encoder, so as to realize the shallow information interaction between RGB and optical flow; S130, performing multi-scale attention fusion on the global RGB features and the global optical flow features output by the last layer of the two encoders to obtain deep fusion features of RGB and optical flow; S140, reconstructing the deep fusion feature and the global optical flow feature respectively according to the normal features recorded in the two memory modules, wherein the normal features in the two memory modules respectively represent the characteristics of the deep fusion feature and the global optical flow feature of the normal traffic video; S150, respectively decoding the two reconstructed features through two decoders to restore the RGB image and the optical flow image of the last frame of video respectively; S160: judging whether the last frame of video has a traffic abnormality risk according to the difference between each restored image and the original image, and the difference between each reconstructed feature and the normal feature.

2. The method according to claim 1, characterized in that The multi-scale attention fusion of the global RGB features and the global optical flow features output by the last layer of the two encoders to obtain the deep fusion features of RGB and optical flow includes: Performing self-attention operations on the global RGB features and the global optical flow features respectively, and fusing the two self-attention results into a preliminary fused feature; Performing a convolution operation on the preliminary fusion features to extract local features; Perform global average pooling on each local feature to extract global features; According to the preliminary fusion features and the global features, a deep fusion feature of RGB and optical flow is generated.

3. The method according to claim 1, characterized in that The reconstructing the deep fusion feature and the global optical flow feature respectively according to the normal features in the two memory modules includes: Calculating the distance between the global optical flow feature and each normal feature in the corresponding memory module respectively; The normal features are weighted and summed according to the distances to obtain the reconstructed features of the global optical flow features.

4. The method according to claim 1, characterized in that: The determining whether the last frame of video has a traffic abnormality risk based on the difference between each restored image and the original image, and the difference between each reconstructed feature and the normal feature, includes: Calculate the PSNR between the RGB restored image and the original image; Calculating the distance between the deep fusion feature and the nearest normal feature; Based on the PSNR and distance, an anomaly score is calculated for the RGB image.

5. The method according to claim 1, characterized in that Before respectively encoding the RGB images and the optical flow images of the first few frames of video by two encoders, the method further includes: Obtain a normal traffic video dataset and extract multiple consecutive frames of video as samples; Each encoder and decoder is trained using each sample. During the training process, operations S120-S150 are performed on each sample respectively, and a loss function is constructed by minimizing the difference between each restored image and the original image, and minimizing the difference between each reconstructed feature and the normal feature.

6. The method according to claim 5, characterized in that The loss function is constructed by minimizing the difference between each restored image and the original image, and minimizing the difference between each reconstructed feature and the normal feature, including: Layer any restored image and the original image along the height direction respectively; According to the distance from each layer to the camera, average pooling of pixels in each layer is performed at different scales; Divide each average pooled image into blocks; According to the distance from each block to the camera, the difference between each block and the original image block is weighted averaged; Construct the loss function to minimize the weighted average difference.

7. The method according to claim 5, characterized in that The loss function is constructed by minimizing the difference between each restored image and the original image, and minimizing the difference between each reconstructed feature and the normal feature, including: The loss function is constructed by minimizing the difference between each restored image and the original image, minimizing the difference between the optical flow features or deep fusion features corresponding to each restored image and the normal features closest to each memory module, and maximizing the difference between each normal feature in each memory module.

8. The method according to claim 5, characterized in that After performing the operations of S120 to S150 on each sample respectively during the training process, the method further includes: Decompose the deep fusion features of any sample into K items; Calculate the distance between each item and each normal feature in the corresponding memory module respectively; Determine the distance between the deep fusion feature and the same normal feature according to the distance between each item and the same normal feature; The normal feature that is farthest from the deep fusion feature is updated to a result of weighted averaging each item according to its distance from the normal feature.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the traffic anomaly detection method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the traffic anomaly detection method described in any one of claims 1-8 is implemented.

Images