Video detection method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video detection method and device, electronic equipment, a storage medium and a program product. Relates to the technical field of video processing. The method comprises the following steps: acquiring to-be-detected video data; processing the video data by using the trained detection model to obtain a detection result corresponding to the video data; the detection model is obtained by training a multi-branch generative adversarial network by adopting a training data set; the training data set comprises a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by carrying out abnormal processing on the historical video data by adopting a pre-constructed pseudo-abnormal processing module; and when the detection result is that the abnormal behavior exists in the video data, determining target video data corresponding to the abnormal behavior in the video data, and outputting the target video data. According to the method, the technical problem of poor generalization ability of the existing detection model is solved, and the generalization ability of the video anomaly detection model is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video detection method, device, electronic device, storage medium and program product. Background Art

[0002] With the progress of society and the development of science and technology, surveillance cameras have been widely deployed in public places, transportation hubs, commercial areas, residential areas, etc. Detecting abnormal behaviors in surveillance videos is a crucial task, which can ensure the timely detection of potential security risks or abnormal situations in the current environment and ensure environmental safety.

[0003] Traditional monitoring systems rely on manual monitoring, which not only consumes a lot of human resources, but is also prone to missed detection and false detection. In order to improve monitoring efficiency and accuracy, automated monitoring video detection technology for abnormal behavior has emerged, such as detection models based on neural network training.

[0004] However, due to the scarcity of abnormal samples, the existing trained detection models have the technical problem of poor generalization ability, resulting in low accuracy and high false negative rate in the detection of abnormal behaviors. Summary of the invention

[0005] The present application provides a video detection method, device, electronic device, storage medium and program product to solve the technical problem of poor generalization ability of existing detection models, improve the generalization ability of video anomaly detection models, and enable them to adapt to different video surveillance scenarios.

[0006] In a first aspect, the present application provides a video detection method, comprising:

[0007] Obtain the video data to be detected;

[0008] The video data is processed using the trained detection model to obtain a detection result corresponding to the video data; the detection model is obtained by training a multi-branch generative adversarial network using a training data set; the training data set includes a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module;

[0009] When the detection result indicates that abnormal behavior exists in the video data, target video data corresponding to the abnormal behavior in the video data is determined, and the target video data is output.

[0010] In an optional implementation, the video data includes a plurality of video frames;

[0011] Accordingly, the video data is processed using the trained detection model to obtain the detection result corresponding to the video data, including:

[0012] For any video frame, determine the previous video frame and the subsequent video frame corresponding to the current video frame;

[0013] According to the previous video frame, the current video frame is predicted to obtain a corresponding predicted video frame;

[0014] According to the subsequent video frame, review processing is performed on the subsequent video frame to obtain a corresponding review video frame;

[0015] According to the predicted video frame and the reviewed video frame, the current video frame is subjected to detection processing to obtain a frame detection result corresponding to the current video frame;

[0016] The detection result corresponding to the video data is obtained according to the frame detection result corresponding to each of the video frames.

[0017] In an optional implementation, performing detection processing on the current video frame according to the predicted video frame and the reviewed video frame to obtain a frame detection result corresponding to the current video frame includes:

[0018] Extracting frame features of the predicted video frame, the reviewed video frame, and the current video frame respectively, and determining corresponding frame feature scores according to the frame features;

[0019] Determining corresponding frame quality scores according to the predicted video frame, the reviewed video frame and the current video frame;

[0020] A frame detection result corresponding to the video frame according to the frame feature score and the frame quality score.

[0021] In an optional implementation, the detection model includes a first generator, a second generator and a discriminator;

[0022] The method further comprises:

[0023] Using historical video data in the training data set, the first generator and the second generator in the multi-branch generative adversarial network are respectively trained in the first phase of iteration;

[0024] The historical video data in the training data set, the historical pseudo-abnormal video data, the first generator and the second generator are used to perform a second stage of iterative training on the discriminator in the multi-branch adversarial network.

[0025] In an optional implementation, the iterative training of the second stage includes a discriminator training stage and a discriminator optimization stage;

[0026] Using the historical video data in the training data set, the historical pseudo-abnormal video data, the first generator, and the second generator, the discriminator in the multi-branch adversarial network is subjected to a second-stage iterative training, including:

[0027] In the discriminator training stage, the historical video data, the predicted video data output by the first generator, and the reviewed video data output by the second generator are used to train the discriminator; wherein the training goal is to enable the discriminator to distinguish between the predicted video data and the reviewed video data and the video data generated from the historical video data;

[0028] In the discriminator optimization stage, the historical video data and the historical pseudo-abnormal video data are used to optimize the discrimination; wherein the optimization goal is to enable the discriminator to distinguish the abnormal video data in the historical video data and the historical pseudo-abnormal video data.

[0029] In an optional implementation, the first loss function used in the iterative training of the first stage is determined based on the current video frame in the historical video data, the predicted video frame obtained after the current video frame is input into the first generator, and the reviewed video frame obtained after the current video frame is input into the second generator;

[0030] The second loss function used in the iterative training of the second stage is determined based on the current video frame, the predicted video frame, the reviewed video frame in the historical video data, and the abnormal predicted video frame obtained after the historical pseudo-abnormal video data is input into the discriminator.

[0031] In a second aspect, the present application provides a video detection device, comprising:

[0032] A video data acquisition module, used to acquire video data to be detected;

[0033] A video data detection module is used to process the video data using a trained detection model to obtain a detection result corresponding to the video data; the detection model is obtained by training a multi-branch generative adversarial network using a training data set; the training data set includes a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module;

[0034] The detection result output module is used to determine target video data corresponding to the abnormal behavior in the video data when the detection result is that the abnormal behavior exists in the video data, and output the target video data.

[0035] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0036] The memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0039] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0040] The video detection technology provided by the present application inputs the acquired video data into the detection model for a series of processing to obtain the detection result corresponding to the video data, and when the detection result obtained by the detection model is that there is abnormal behavior in the video data, the target video data corresponding to the abnormal behavior in the video data is determined, and the target video data is output, which can play the role of an alarm prompt, so that the maintenance personnel in the current scene can determine that there is an abnormal situation in the monitoring data corresponding to the video data when receiving the output video data, and can immediately start the corresponding processing measures. This design can effectively achieve the effect of timely discovery and timely processing of problems; and, since the training data set used in the training of the detection model is a plurality of historical video data without abnormal behavior, that is, only positive sample training is used to train the multi-branch generative adversarial network to obtain the detection model, so that a large number of samples can be used to train the model, thereby improving the accuracy of the detection results when the model is applied; and in the training process, a pseudo-anomaly processing module is also used to perform pseudo-anomaly processing on the positive samples, and a variety of indefinite negative samples are generated to train the model, so as to obtain a detection model with enhanced generalization ability, so that when applied, the underreporting rate of abnormal behavior in the video can be reduced, thereby further improving the accuracy of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 An application scenario diagram of the video detection method provided in this application;

[0043] Figure 2 A schematic diagram of a flow chart of a video detection method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a flow chart of another video detection method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a detection model provided in an embodiment of the present application;

[0046] Figure 5 A flow chart of a detection model training process provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a detection model during training provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of a video detection device provided by the present application according to an embodiment;

[0049] Figure 8 It is a block diagram of an electronic device shown in an embodiment of the present application.

[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0053] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0054] In practical applications, with the progress of society and the development of science and technology, surveillance cameras have been widely deployed in public places, transportation hubs, commercial areas, residential areas, etc. Detecting abnormal behaviors in surveillance videos is a crucial task, which can ensure the timely discovery of potential security risks or abnormal situations in the current environment and ensure environmental safety.

[0055] In an optional implementation, the YOLOv3 network is trained using video data containing abnormal behavior and corresponding labels to obtain a YOLOv3 target detection algorithm; then, the algorithm is used to process the foreground data of the target object in the video data, and the foreground data is segmented, feature extracted, etc. to obtain the abnormal score of the target object, and finally the SVM classifier is used to achieve fast and accurate classification to meet the real-time requirements. The above method combines target detection technology, convolutional autoencoders, deep learning and machine learning methods, which can effectively detect abnormal behavior in video surveillance and improve the accuracy and real-time performance of detection.

[0056] However, the above implementation has the following problems during implementation: 1. Dependence on the accuracy of the target detection algorithm: The method relies on the YOLOv3 target detection algorithm to detect foreground targets, and the accuracy of the algorithm directly affects the effect of abnormal behavior detection. 2. Restrictions on the definition of abnormal behavior: The method may be restricted by the definition of abnormal behavior, and may not be able to effectively detect some new or complex abnormal behaviors. 3. The impact of data set division: The method divides the video data set into a training set and a test set. If the data set division is unreasonable or there is a lack of diverse data in the training set, it may affect the generalization ability of the model. 4. Limitations of feature extraction and clustering: The effect of feature extraction and clustering depends on the choice of algorithms and parameters. There may be insufficient feature extraction or inaccurate clustering. 5. Limitations of SVM classifier: Although the SVM classifier can classify quickly and accurately, it may have high computational complexity when processing large-scale data.

[0057] In another optional implementation, a video frame sequence diagram of a surveillance video image is generated by using time attributes and associated attributes, video single-frame features of the video frame sequence diagram are extracted, and abnormal data in the video single-frame features are detected to obtain a video abnormal sequence. The video abnormal sequence is further classified and identified to obtain a video classification abnormal sequence, and the abnormal value of the video classification abnormal sequence is determined according to the classification coefficient; the sequence abnormality critical value of the video abnormal sequence is calculated according to the abnormal value and the dynamic weight by a dynamic weight algorithm, and the safety warning value of the surveillance video image is calculated according to the sequence abnormality critical value, thereby determining the data security of the surveillance video image. The above method utilizes intelligent video surveillance technology, and improves the accuracy of data security warnings by extracting features and detecting abnormalities of video data in combination with a dynamic weight algorithm, thereby ensuring the data security of surveillance video images.

[0058] However, there are also some problems in the above implementation process: 1. The processing ability for complex scenes or diverse abnormal situations is limited, which may lead to the inability to accurately identify all abnormal situations. 2. The presets of some parameters may not be flexible enough to adapt to the actual needs under different monitoring environments. 3. The processing efficiency of large-scale monitoring video data may be low, requiring more computing resources and time. 4. In actual applications, there may be a certain false alarm rate or missed alarm rate, which affects the accuracy and reliability of data security warnings. 5. It is necessary to rely on the preset anomaly detection algorithm and classification recognition model, and the algorithm model may need to be continuously updated and optimized for new abnormal situations.

[0059] In another optional implementation, video anomaly detection can also be performed based on graph domain adversarial learning; specifically, the method includes: constructing a domain feature invariant learning module: constructing a graph-based domain feature invariant learning module using graph convolution operations, and learning domain invariant features through the strength of associations between domain nodes. Constructing a domain discriminator: constructing a domain discriminator, predicting the domain to which the input sample belongs through graph convolution operations, and learning domain-specific features. Training model: using a training data set to train anomaly detection network, while optimizing domain-invariant features and domain discriminators. Domain adversarial learning: through domain adversarial learning, the domain-specific feature components in the deep features learned by the model are reduced, and the domain-invariant feature components are increased. Model training and optimization: using a gradient descent algorithm to optimize the domain adversarial learning model, including forward propagation and back propagation processes. Anomaly detection: using a trained network model to perform anomaly detection on any data set to detect whether anomalies occur. System implementation: including system implementation on the client and server sides, the client is responsible for video acquisition and information transmission, and the server side is responsible for information processing and storage. Real-time video training and detection: providing training and detection for real-time video, enhancing the feasibility and portability of the method in practical applications.

[0060] Although the above process is innovative in improving cross-domain generalization capabilities, it may also have some disadvantages and limitations: domain adversarial learning involves multiple hyperparameters (such as η in the gradient reversal layer and β in the loss function). The selection of these hyperparameters has a significant impact on model performance and needs to be adjusted through methods such as cross-validation, which increases the complexity of model tuning. In addition, the domain discriminator is designed to learn domain-specific features, but in practical applications, over-learning domain-specific features may affect the generalization ability of the model.

[0061] In summary, although the above method emphasizes cross-domain generalization capabilities, the generalization ability of the model may still be limited for different types of abnormal behaviors, especially for abnormal types that have not appeared in the training data, resulting in problems such as low accuracy and high false negative rate in the detection of abnormal behaviors.

[0062] The video detection method provided by the present application aims to solve the above technical problems of the prior art. Specifically, historical video data containing normal behaviors is used as a training data set, and a detection model obtained by training a multi-branch generative adversarial network is used to detect the video data to be detected, so as to obtain more accurate detection results; the reason is that due to the scarcity of abnormal samples, the use of positive samples as training data can provide a large amount of training data for model training, thereby improving the accuracy of the detection results of the detection model when it is applied; in order to avoid the limitation of the trained model caused by the detection model being able to learn only a small amount of abnormal behaviors contained in the training data set during the training process, during the training process, pseudo-anomaly processing is performed on positive samples by using a pseudo-anomaly processing module, and multiple types of indefinite negative samples are generated to train the model, so as to obtain a detection model with enhanced generalization ability, so that the underreporting rate of abnormal behaviors in the video can be reduced when applied, thereby further improving the accuracy of the detection results.

[0063] Figure 1 This is an application scenario diagram of the video detection method provided in this application. Figure 1 The application scenarios to which the embodiments of the present application are applicable are described. The video detection method provided by the present application has broad application prospects in the fields of security monitoring, intelligent transportation, public safety, etc. For example, in public places such as squares, subway stations, airports, event venues, etc., it can detect abnormal crowd gatherings to prevent stampedes; in banks and financial institutions, it can detect suspicious behaviors such as long stays, abnormal withdrawals, etc. to ensure financial security; in factories and warehouses, it can detect abnormal equipment operation, such as fires, water leaks, etc., and take timely measures to prevent accidents.

[0064] In short, video detection technology has important applications in many fields, improving the efficiency and accuracy of monitoring systems, timely detecting and handling various abnormal situations, and ensuring the safety and stability of public safety, commercial security, industrial monitoring, etc. Moreover, with the continuous advancement of technology, the application scenarios of video detection will become more extensive and in-depth.

[0065] See also Figure 1 , in any of the above scenarios, the implementation of the video detection technology involves a video data acquisition device 11 and a video detection device 12;

[0066] Specifically, the video data acquisition device 11 is used to obtain the video data to be detected, and the obtained video data is transmitted to the video detection device 12 for video detection processing to obtain the corresponding detection result.

[0067] The video detection device 12 is provided with a pre-trained detection model. Specifically, the video data is input into the detection model for a series of processing to obtain a detection result corresponding to the video data. Optionally, when the detection result obtained by the detection model is that there is an abnormal behavior in the video data, the video detection device 12 determines the target video data corresponding to the abnormal behavior in the video data and outputs the target video data.

[0068] It should be noted that the above-mentioned video detection device only outputs when an abnormality is detected, which can serve as an alarm prompt, so that the maintenance personnel in the current scene can determine that there is an abnormality in the monitoring data corresponding to the video data when receiving the output video data, and can immediately initiate corresponding processing measures. This design can effectively achieve the effect of timely detection and timely processing of problems.

[0069] It should also be noted that the training data set used in the training of the detection model is a plurality of historical video data without abnormal behavior, that is, only positive sample training is used to train the multi-branch generative adversarial network to obtain the detection model. In this way, a large number of samples can be used to train the model, thereby improving the accuracy of the detection results when the model is applied. In addition, during the training process, a pseudo-anomaly processing module is used to perform pseudo-anomaly processing on the positive samples, and a variety of indefinite negative samples are generated to train the model, so as to obtain a detection model with enhanced generalization ability. In this way, when applied, the underreporting rate of abnormal behavior in the video can be reduced, thereby further improving the accuracy of the detection results.

[0070] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0071] Figure 2 The flowchart of a video detection method provided in an embodiment of the present application is shown in FIG. The method can be executed by a video detection device, which can be a server or an electronic device. The following description takes an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method comprises the following steps:

[0072] S210: Obtain video data to be detected.

[0073] In the present application, video data refers to a sequence of images captured by a camera device, and these image sequences can also be understood as a plurality of continuous video frames.

[0074] Since the original video data captured by the camera device is usually large in size, it can be compressed using a video encoder and the compressed video data can be transmitted to a cloud server via the network for storage, so that it can be played back and retrieved when needed to view historical events.

[0075] At the same time, if the current camera equipment does not have a video detection function, it is also necessary to transmit the video data to a corresponding video detection device to perform abnormal analysis on the collected video data.

[0076] Correspondingly, when receiving video data, the video detection device uses a video decoder to decode the compressed video data to obtain the original video data, that is, multiple continuous video frames, and detects abnormal behavior (such as intrusion, fire, etc.) to obtain detection results corresponding to the video data.

[0077] S220: Process the video data using the trained detection model to obtain detection results corresponding to the video data.

[0078] Specifically, the video data is input into the detection model, and a series of abnormal behavior detection processes are performed on each video frame in the video data in the detection model to obtain a frame detection result corresponding to each video frame, and a video detection result corresponding to the video data is obtained based on the detection results of each frame.

[0079] In the present application, the detection model is obtained by training a multi-branch generative adversarial network using a training data set; the training data set includes a plurality of historical video data without abnormal behavior, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module.

[0080] Optionally, the detection model can be trained in a video data acquisition device, or can be trained on other devices and then set in the current device. There is no limitation on the training device of the model.

[0081] Specifically, the training data set used in the training of the detection model is a plurality of historical video data without abnormal behavior. In other words, in the present application, only positive sample training is used to train the multi-branch generative adversarial network to obtain the detection model. In this way, a large number of samples can be used to train the model, thereby improving the accuracy of the detection results when the model is applied. In addition, during the training process, a pseudo-anomaly processing module is used to perform pseudo-anomaly processing on the positive samples, and a variety of indefinite negative samples are generated to train the model, so as to obtain a detection model with enhanced generalization ability. In this way, when applied, the underreporting rate of abnormal behavior in the video can be reduced, thereby further improving the accuracy of the detection results.

[0082] S230: When the detection result is that abnormal behavior exists in the video data, determine target video data corresponding to the abnormal behavior in the video data, and output the target video data.

[0083] In the present application, the detection model obtains the detection results of the video data based on the frame detection results corresponding to each video frame.

[0084] Optionally, determine whether the detection result contains abnormal behavior; if not, store the detection result; otherwise, if contained, determine target video data corresponding to the abnormal behavior in the video data, and output the target video data.

[0085] Since most of the video data in the video data is video data corresponding to normal behavior, if the above detection results are output, the maintenance personnel will pay ineffective attention to the video data corresponding to normal behavior, and when the detection results of abnormal behavior are output, due to the decrease in energy caused by long-term attention, they may not respond immediately, resulting in failure to respond to abnormal situations in time; if only the data corresponding to abnormal behavior is output, the maintenance personnel will receive the output detection results, which means that there is abnormal behavior in the video data, and immediately initiate corresponding processing measures. This design can effectively achieve the effect of timely discovery and timely processing of problems, and improve the efficiency of responding to emergencies. On the other hand, by intelligently filtering normal events and only recording and storing abnormal events, the demand for storage space can be reduced and storage costs can be reduced.

[0086] In the above technical solution, the acquired video data is input into the detection model for a series of processing to obtain the detection result corresponding to the video data. When the detection result obtained by the detection model is that there is abnormal behavior in the video data, the target video data corresponding to the abnormal behavior in the video data is determined and the target video data is output, which can play the role of an alarm prompt, so that the maintenance personnel in the current scene can determine that there is an abnormal situation in the monitoring data corresponding to the video data when receiving the output video data, and can immediately start the corresponding processing measures. This design can effectively achieve the effect of timely discovery and timely processing of problems; and, since the training data set used in the training of the detection model is a plurality of historical video data without abnormal behavior, that is, only positive sample training is used to train the multi-branch generative adversarial network to obtain the detection model, so that a large number of samples can be used to train the model, and the accuracy of the detection result when the model is applied can be improved; and in the training process, a pseudo-anomaly processing module is also used to perform pseudo-anomaly processing on the positive samples, and a variety of indefinite negative samples are generated to train the model, so as to obtain a detection model with enhanced generalization ability, so that the underreporting rate of abnormal behavior in the video can be reduced when applied, thereby further improving the accuracy of the detection result.

[0087] Figure 3 A schematic diagram of a video detection method provided in an embodiment of the present application. Figure 3 The specific process of video detection is introduced in detail. Specifically, the video detection process includes the following steps:

[0088] S310: For any video frame, determine a previous video frame and a subsequent video frame corresponding to the current video frame; perform prediction processing on the current video frame according to the previous video frame to obtain a corresponding predicted video frame.

[0089] In the present application, the detection model is a model obtained by training a multi-branch adversarial network. Figure 4 A schematic diagram of the structure of a detection model provided in an embodiment of the present application. Figure 4 The detection model includes two generators and a discriminator, namely, a first generator, a second generator and a discriminator. In this application, these two generators are used to collaboratively regenerate the video frames to be detected through context learning technology, and the generated video frames and the video frames to be detected are input into the discriminator together to verify the authenticity.

[0090] Since the video data contains multiple video frames, and the context information of the video frames needs to be referenced when monitoring the video frames in the technical solution of this application, the process of detecting abnormal behavior in the video frames is described in detail below using any intermediate video frame as an example. The detection of the first video frame and the last video frame in the video data can be processed as context information, or other methods can be used to generate context information, which is not specifically limited.

[0091] Specifically, for the current video frame to be detected, multiple preceding video frames before the current video frame are input into the first generator for video frame prediction to obtain a predicted video frame corresponding to the current video frame. Video frame prediction can be understood as a process of predicting the next video frame based on multiple preceding video frames.

[0092] S320: According to the subsequent video frames, review the subsequent video frames to obtain corresponding review video frames.

[0093] Specifically, for the current video frame to be detected, multiple subsequent video frames after the current video frame are input into the second generator for video frame review processing to obtain a review video frame corresponding to the current video frame. The video frame review can be understood as a process of reviewing a previous video frame based on multiple subsequent video frames.

[0094] In an optional implementation, since the video frame review can also be understood as reverse-order video frame prediction, a plurality of subsequent video frames can be input into a corresponding generator in reverse order for processing to obtain corresponding review video frames.

[0095] S330 , performing detection processing on the current video frame according to the predicted video frame and the reviewed video frame to obtain a frame detection result corresponding to the current video frame.

[0096] The obtained predicted video frame and reviewed video frame are input into the discriminator together with the current video frame to realize detection processing on the current video frame and obtain the frame detection result corresponding to the current video frame.

[0097] In an optional embodiment, for the discrimination of video frames, in order to quickly and accurately obtain the discrimination results, when the discriminator in the present application performs discrimination processing, the three video frames can be segmented and processed to obtain multiple sub-video frames, and each sub-video frame can be independently discriminated, and finally the frame discrimination result of the entire video frame can be obtained comprehensively.

[0098] When the frame detection results corresponding to the video frames are obtained, the frame detection results are spliced ​​or combined according to the time sequence of the video frames to obtain the detection results corresponding to the video data.

[0099] On the basis of the above, optionally, the current video frame is detected and processed to obtain a frame detection result corresponding to the current video frame, including: respectively extracting frame features of the predicted video frame, the reviewed video frame and the current video frame, and determining the corresponding frame feature score according to the frame features; determining the corresponding frame quality score according to the predicted video frame, the reviewed video frame and the current video frame; and obtaining the frame detection result corresponding to the video frame according to the frame feature score and the frame quality score.

[0100] Specifically, potential feature extraction operations are performed on the predicted video frame, the reviewed video frame and the current video frame to obtain frame features corresponding to each video frame; feature fusion can be performed on the frame features corresponding to the predicted video frame and the reviewed video frame to obtain fused frame features; the correlation between the fused frame features and the frame features corresponding to the current video frame is further calculated to obtain the frame feature score corresponding to the current video frame.

[0101] The correlation between the predicted video frame and the reviewed video frame and the current video frame may also be calculated respectively, and then the frame quality score of the current video frame may be obtained according to the correlation calculation result.

[0102] Exemplarily, a first peak signal-to-noise ratio between the predicted video frame and the current video frame can be calculated, and a second peak signal-to-noise ratio between the reviewed video frame and the current video frame can be calculated; and a peak signal-to-noise ratio corresponding to the current video frame is calculated based on the first peak signal-to-noise ratio and the second peak signal-to-noise ratio, and the peak signal-to-noise ratio is used as a frame quality score corresponding to the current video frame.

[0103] By performing weighted summation on the frame feature score and the frame quality score, the frame score corresponding to the current video frame is obtained, and then the detection result of the current video frame is determined according to the frame score. In this way, the current video frame is detected by integrating multiple features and data, making the obtained detection result more accurate.

[0104] Optionally, the frame score can be determined to be greater than a preset abnormal score threshold to obtain a detection result of whether the current video frame has abnormal behavior; optionally, if it exceeds, it indicates that the detection result is that abnormal behavior exists in the video frame; otherwise, it indicates that the detection result is that the video frame has normal behavior.

[0105] It should be understood that the above-mentioned specific implementation methods for determining the search results are exemplary introductions to the technical solutions of the present application, and the present application may also adopt other methods to determine the search results, which are not specifically limited.

[0106] S340. Obtain detection results corresponding to the video data according to the frame detection results corresponding to each video frame.

[0107] On the basis of the above implementation, the detection result corresponding to the video data is obtained by performing positive deterioration on the frame detection result of each video frame according to the time sequence of each video frame.

[0108] The above method of detecting the video data frame by frame and integrating the results can accurately and quickly obtain the detection results, thereby improving the accuracy of the detection results.

[0109] Figure 5 A flow chart of a detection model training process provided in an embodiment of the present application. Figure 5 , the training process of the detection model in this application is introduced in detail. The specific model training process includes the following steps:

[0110] S510, using the historical video data in the training data set, respectively perform the first phase of iterative training on the first generator and the second generator in the multi-branch generative adversarial network.

[0111] Due to the scarcity of video samples containing abnormal behaviors, the historical video data used to train the multi-branch generative adversarial network in this application is video data that does not contain abnormal behaviors, that is, positive sample data. This can provide a large number of training data sets for training, so that the obtained detection model can have more accurate detection results when applied.

[0112] During the training process, this application adopts staged training. In the first stage, the two generators are trained to achieve that the two generators can generate high-quality video frames; in the second stage, the discriminator is trained to achieve that the discriminator can distinguish the video frames generated by the generator.

[0113] In the training process of any iterative round in the first stage, the first loss function is used to implement the training of the generator. The first loss function is determined based on the current video frame in the historical video data, the predicted video frame obtained after the current video frame is input to the first generator, and the reviewed video frame obtained after the current video frame is input to the second generator. Specifically, the first loss function includes gradient loss, intensity loss, appearance loss, optical flow loss, generator adversarial loss, generator potential loss and generator frame loss.

[0114] For example, 2T+1 consecutive video frames are extracted from the training sample set, that is, {I t-T ,…,I t ,…,I t+T}; It should be understood that the above-mentioned multiple video frames are all video frames containing normal behavior. t-T ,…,I t-1}As I tThe previous video frame is input to the first generator for video frame prediction, and the predicted video frame output by the first generator is obtained. And, {I in the above video frame t+1 ,…,I t+T}As I t The subsequent video frames are input to the second generator for video frame review. To facilitate the second generator to perform video frame review processing, the subsequent video frames can be input to the second generator in reverse order, that is, {I t+T ,…,I t+1} Input to the second generator to obtain the review video frame output by the second generator

[0115] On this basis, according to the predicted video frame Review video frames With the current video frame I t Calculate the gradient loss; wherein the gradient loss is used to characterize the difference between the generated predicted video frame, the reviewed video frame and the current video frame in the local image. Exemplarily, the expression for calculating the gradient loss is as follows:

[0116]

[0117] in, Characterize the gradient loss.

[0118] Based on the predicted video frame Review video frames With the current video frame I t Calculate the intensity loss; wherein the intensity loss is used to characterize the difference in pixel intensity between the generated predicted video frame, the reviewed video frame, and the current video frame. Exemplarily, the expression for calculating the intensity loss is as follows:

[0119]

[0120] in, Characterizes strength loss.

[0121] Based on the predicted video frame Review video frames With the current video frame I t Calculate the appearance loss; wherein the appearance loss is used to characterize the difference in image appearance between the generated predicted video frame, the reviewed video frame and the current video frame; the appearance loss is determined based on the gradient loss and the intensity loss. Exemplarily, the expression for calculating the appearance loss is as follows:

[0122]

[0123] in, Characterizes appearance loss.

[0124] Based on the predicted video frame Review video frames With the current video frame I t Calculate the optical flow loss; wherein the optical flow loss is used to characterize the difference in image optical flow between the generated predicted video frame, the reviewed video frame and the current video frame. Exemplarily, the expression for calculating the optical flow loss is as follows:

[0125]

[0126] in, Characterize optical flow loss; Characterization and The light flow between Characterization and The optical flow between t Characterization I t-1 and I t The light flow between.

[0127] Based on the predicted video frame Review video frames With the current video frame I t Calculate the generator adversarial loss; wherein the generator adversarial loss is used to characterize the difference in image quality between the predicted video frame, the reviewed video frame, and the current video frame. Exemplarily, the expression for calculating the generator adversarial loss is as follows:

[0128]

[0129] in, Represents the generator adversarial loss; DI(·) represents the final output layer of the discriminator, which is usually responsible for providing a judgment on whether the input sample is real data. In some GAN variants, D_I may refer to the score of the output layer, which will be converted into a probability value indicating the possibility of the input being judged as real data; DE(·) represents the output of an intermediate layer or feature extraction layer in the discriminator, and E may stand for Extractor. During the training process of the generative adversarial network, using the output of the intermediate layers can help the generator better learn how to generate more realistic data because these intermediate layers capture certain characteristics of the data.

[0130] Based on the predicted video frame Review video frames With the current video frame I t Calculate the potential loss of the generator; wherein the potential loss of the generator is used to characterize the difference in image features between the predicted video frame, the reviewed video frame and the current video frame. Exemplarily, the expression for calculating the potential loss of the generator is as follows:

[0131]

[0132] in, Characterizes the latent loss of the generator; DL(DE(·)) characterizes the function that extracts features from the latent space after the backbone network of the discriminator extracts features.

[0133] Based on the above implementation, any one of the preceding video frames I is also t-i Input into the first generator to predict the video frame and get the current video frame I t The corresponding predicted video frame And, any one of the subsequent video frames I t+i Input to the second generator to review the video frame and get the current video frame I t Corresponding review video frames

[0134] On this basis, according to the predicted video frame generated above Review video frames With the current video frame I t Calculate the generator frame loss; wherein the generator frame loss is used to characterize the image difference between the predicted video frame, the reviewed video frame and the current video frame generated by the generator based on a single video frame. Exemplarily, the expression for calculating the generator frame loss is as follows:

[0135]

[0136] in, Characterizing the generator frame loss.

[0137] On the basis of calculating each loss function, the network parameters of the first generator and the second generator are jointly adjusted based on each loss function to achieve the training of the current iteration round.

[0138] S520, using the historical video data, the historical pseudo-abnormal video data, the first generator and the second generator in the training data set, to perform a second phase of iterative training on the discriminator in the multi-branch adversarial network.

[0139] In the present application, on the basis of completing the training of the first generator and the second generator based on the above-mentioned implementation method, the first generator and the second generator are respectively used to perform video frame prediction and video frame review, and the discriminator is trained according to the output predicted video frames and reviewed video frames; in order to enhance the detection results of the discriminator on abnormal behavior in video frames, a pseudo-anomaly module is used to perform pseudo-anomaly processing on historical video data, and the pseudo-anomaly video data obtained is input into the discriminator for optimization processing, so as to achieve a discriminator with better performance when applied.

[0140] In the training process of any iterative round in the second stage, the second loss function is used to realize the overall training of the discriminator. The second loss function is determined based on the current video frame, the predicted video frame, the reviewed video frame in the historical video data, and the abnormal predicted video frame obtained after the historical pseudo-abnormal video data is input into the discriminator.

[0141] Specifically, during the second stage of training, in the discriminator training stage, the historical video data, the predicted video data output by the first generator, and the reviewed video data output by the second generator are used to train the discriminator; wherein the training goal is to enable the discriminator to distinguish between the predicted video data and the reviewed video data and the video data generated from the historical video data; in the discriminator optimization stage, the historical video data and the historical pseudo-abnormal video data are used to optimize the discrimination; wherein the optimization goal is to enable the discriminator to distinguish between the abnormal video data in the historical video data and the historical pseudo-abnormal video data.

[0142] Specifically, the loss when training the discriminator includes the discriminator adversarial loss and the discriminator potential loss.

[0143] For example, the above 2T+1 consecutive video frames are still used as an example for processing, that is, the preceding video frame {I t-T ,…,I t-1} Input to the first generator after training to predict the video frame, and obtain the predicted video frame output by the first generator And, the subsequent video frames {I t+1 ,…,I t+T} is input to the trained second generator for video frame review. To facilitate the second generator to review the video frames, the subsequent video frames can be input to the second generator in reverse order, that is, {I t+T ,…,I t+1} is input to the second generator to obtain the review video frame output by the second generator

[0144] On this basis, according to the predicted video frame Review video frames With the current video frame I t Calculate the discriminator adversarial loss; wherein the discriminator adversarial loss is used to characterize the difference in image quality between the predicted video frame, the reviewed video frame, and the current video frame. Exemplarily, the expression for calculating the discriminator adversarial loss is as follows:

[0145]

[0146] in, Characterizing the discriminator adversarial loss.

[0147] According to the predicted video frame Review video frames With the current video frame I t Calculate the discriminator potential loss; wherein the discriminator potential loss is used to characterize the difference in image features between the predicted video frame, the reviewed video frame and the current video frame. Exemplarily, the expression for calculating the discriminator potential loss is as follows:

[0148]

[0149] in, Characterize the discriminator's potential loss.

[0150] Furthermore, the discriminator is trained based on the loss function obtained in the above implementation mode to obtain a trained discriminator.

[0151] Figure 6 A schematic diagram of the structure of a detection model in the training process provided in an embodiment of the present application. Figure 6 As shown, in order to enhance the discriminator's ability to identify abnormal behavior, the present application uses a pseudo-anomaly module to generate non-existent abnormal event data for video detection, and optimizes and trains the discriminator based on the processed data.

[0152] Specifically, the loss function used in the discriminator optimization process is a pseudo-anomaly loss, which is determined based on historical video data and historical pseudo-anomaly video data.

[0153] Exemplarily, any two non-adjacent video frames {I t-i ,I t+i} Input to the pseudo-anomaly module for pseudo-anomaly processing, and obtain the output pseudo-anomaly video frame and will and the current video frame I t The two are input into the discriminator together to optimize it using the generated pseudo-anomaly loss function.

[0154] On this basis, according to the pseudo abnormal video frame and the current video frame I t Calculate the pseudo-anomaly loss of the discriminator; wherein the pseudo-anomaly loss is used to optimize the discriminator and improve its ability to identify abnormal video frames. Exemplarily, the calculation expression of the pseudo-anomaly loss is as follows:

[0155]

[0156] in, Characterizing pseudo-anomaly losses.

[0157] Furthermore, the discriminator is optimized and trained based on the loss function obtained in the above implementation mode to obtain a trained discriminator. In the present application, data optimization training of the pseudo-anomaly module helps to improve the detection model's ability to identify unknown anomaly types and reduce misjudgments and missed judgments.

[0158] In the above implementation, a variety of loss functions are used to perform stage-by-stage training, and during the training process, hyperparameters such as learning rate, batch size, and training cycle are carefully adjusted to achieve the optimal training effect.

[0159] In another optional implementation, two representative public video surveillance datasets are selected: CUHK AvenueC and ShanghaiTech Campus. These datasets cover different surveillance scenarios and abnormal event types. All video frames are normalized to eliminate the effects of lighting and contrast. During the training process, accuracy, recall, precision, and F1 score are also used as the main performance evaluation indicators. In addition, the inference time of the model is also recorded to evaluate its applicability in real-time surveillance scenarios.

[0160] Specifically, for the CUHK Avenue dataset, the experimental results of the model after training include: accuracy: 96.3%, which maintains high accuracy in complex urban environments; recall rate: 94.2%, with high coverage of anomaly detection; precision: 95.1%, with further reduction in false alarm rate; F1 score: 94.6%, with stable overall performance.

[0161] For the ShanghaiTech Campus dataset, the experimental results of the trained model include: accuracy: 94.1%, which slightly decreased in large-scale scenarios but was still at a high level; recall: 90.5%, which still had high sensitivity for abnormal event detection in large-scale scenarios; precision: 93.6%, which maintained a low false alarm rate; F1 score: 92.0%, which had good overall performance.

[0162] Based on the above implementation, the technical solution of the present application also carried out robustness testing and real-time performance testing. Specifically, under different lighting conditions and camera viewing angles, the detection model obtained by training the multi-branch generative adversarial network showed good robustness. Even in the case of insufficient lighting or changes in viewing angle, the decrease in the accuracy and recall rate of the model was within an acceptable range.

[0163] For example, on a Tesla-V100 GPU, the average processing speed of the detection model trained by the multi-branch generative adversarial network reaches 45 frames per second, meeting the needs of real-time monitoring. Moreover, compared with the existing video detection technology, the detection model trained by the multi-branch generative adversarial network shows at least 5% performance improvement in all evaluation indicators, proving its potential advantages in practical applications.

[0164] In summary, the detection model trained by multi-branch generative adversarial networks shows excellent performance on multiple standard datasets. The model not only performs well in accuracy and recall, but also meets the needs of practical applications in terms of real-time processing and robustness. These results provide strong support for the application of detection models trained by multi-branch generative adversarial networks in the field of intelligent video surveillance.

[0165] Figure 7 This is a schematic diagram of the structure of a video detection device provided by the present application according to an embodiment. Figure 7 The video detection device 70 includes: a video data acquisition module 710, a video data detection module 720, and a detection result output module 730; wherein,

[0166] The video data acquisition module 710 is used to acquire the video data to be detected;

[0167] The video data detection module 720 is used to process the video data using the trained detection model to obtain the detection result corresponding to the video data; the detection model is obtained by training the multi-branch generative adversarial network using the training data set; the training data set includes a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module;

[0168] The detection result output module 730 is used to determine target video data corresponding to the abnormal behavior in the video data and output the target video data when the detection result is that there is an abnormal behavior in the video data.

[0169] In an optional embodiment, the video data includes a plurality of video frames;

[0170] Accordingly, the video data detection module 720 includes:

[0171] A video frame determination submodule, used for determining, for any video frame, a preceding video frame and a subsequent video frame corresponding to the current video frame;

[0172] The video frame prediction submodule is used to predict the current video frame according to the previous video frame to obtain the corresponding predicted video frame;

[0173] The video frame review submodule is used to review the subsequent video frames according to the subsequent video frames to obtain corresponding review video frames;

[0174] The frame detection result determination submodule is used to perform detection processing on the current video frame according to the predicted video frame and the reviewed video frame to obtain the frame detection result corresponding to the current video frame;

[0175] The detection result determination submodule is used to obtain the detection result corresponding to the video data according to the frame detection result corresponding to each video frame.

[0176] In an optional implementation, the frame detection result determination submodule includes:

[0177] A frame feature score determination unit, used to extract frame features of the predicted video frame, the reviewed video frame and the current video frame, respectively, and determine corresponding frame feature scores according to the frame features;

[0178] A frame quality score determination unit, configured to determine a corresponding frame quality score according to the predicted video frame, the reviewed video frame and the current video frame;

[0179] The frame detection result determination unit is used to determine the frame detection result corresponding to the video frame according to the frame feature score and the frame quality score.

[0180] In an optional implementation, the detection model includes a first generator, a second generator, and a discriminator;

[0181] The device also includes:

[0182] A first training module is used to use the historical video data in the training data set to perform a first phase of iterative training on the first generator and the second generator in the multi-branch generative adversarial network respectively;

[0183] The second training module is used to use the historical video data, historical pseudo-abnormal video data, the first generator and the second generator in the training data set to perform the second stage of iterative training on the discriminator in the multi-branch adversarial network.

[0184] In an optional implementation, the first loss function used in the iterative training of the first stage is determined based on the current video frame in the historical video data, the predicted video frame obtained after the current video frame is input into the first generator, and the reviewed video frame obtained after the current video frame is input into the second generator;

[0185] The second loss function used in the iterative training of the second stage is determined based on the current video frame, the predicted video frame, the reviewed video frame in the historical video data, and the abnormal predicted video frame obtained after the historical pseudo-abnormal video data is input into the discriminator.

[0186] In an optional implementation, the iterative training of the second phase includes a discriminator training phase and a discriminator optimization phase;

[0187] The second training module includes:

[0188] a discriminator training sub-stage, for training the discriminator using the historical video data, the predicted video data output by the first generator, and the reviewed video data output by the second generator in the discriminator training stage; wherein the training goal is to enable the discriminator to distinguish the predicted video data and the reviewed video data from the video data generated from the historical video data;

[0189] The discriminator optimization sub-stage is used to optimize the discrimination using the historical video data and the historical pseudo-anomaly video data during the discriminator optimization stage; wherein the optimization goal is to enable the discriminator to distinguish the abnormal video data in the historical video data and the historical pseudo-anomaly video data.

[0190] Figure 8 It is a block diagram of an electronic device shown in an embodiment of the present application, and the device may be a computer, a digital broadcast terminal, etc.

[0191] See also Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0192] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0193] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0194] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0195] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0196] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0197] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0198] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of the components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a solid image (Complementary Metal Oxide Semiconductor, CMOS) sensor or a semiconductor image (Charge-coupled Device, CCD) sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0199] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0200] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0201] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0202] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned aircraft collision prediction method.

[0203] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the aircraft collision prediction method in the above embodiment.

[0204] An embodiment of the present application further provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed on a computer, the computer executes the technical solution of the aircraft collision prediction method in the above embodiment.

[0205] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the collision prediction method of the aircraft in the above embodiment.

[0206] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0207] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0208] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0209] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video detection method, characterized in that: The method comprises: Obtain the video data to be detected; The video data is processed using the trained detection model to obtain a detection result corresponding to the video data; the detection model is obtained by training a multi-branch generative adversarial network using a training data set; the training data set includes a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module; When the detection result indicates that abnormal behavior exists in the video data, target video data corresponding to the abnormal behavior in the video data is determined, and the target video data is output.

2. The method according to claim 1, characterized in that The video data includes a plurality of video frames; Accordingly, the video data is processed using the trained detection model to obtain the detection result corresponding to the video data, including: For any video frame, determine the previous video frame and the subsequent video frame corresponding to the current video frame; According to the previous video frame, the current video frame is predicted to obtain a corresponding predicted video frame; According to the subsequent video frame, review processing is performed on the subsequent video frame to obtain a corresponding review video frame; According to the predicted video frame and the reviewed video frame, the current video frame is subjected to detection processing to obtain a frame detection result corresponding to the current video frame; The detection result corresponding to the video data is obtained according to the frame detection result corresponding to each of the video frames.

3. The method according to claim 2, characterized in that According to the predicted video frame and the reviewed video frame, the current video frame is detected and processed to obtain a frame detection result corresponding to the current video frame, including: Extracting frame features of the predicted video frame, the reviewed video frame, and the current video frame respectively, and determining corresponding frame feature scores according to the frame features; Determining corresponding frame quality scores according to the predicted video frame, the reviewed video frame and the current video frame; A frame detection result corresponding to the video frame according to the frame feature score and the frame quality score.

4. The method according to any one of claims 1 to 3, characterized in that The detection model includes a first generator, a second generator and a discriminator; The method further comprises: Using historical video data in the training data set, the first generator and the second generator in the multi-branch generative adversarial network are respectively trained in the first phase of iteration; The historical video data in the training data set, the historical pseudo-abnormal video data, the first generator and the second generator are used to perform a second stage of iterative training on the discriminator in the multi-branch adversarial network.

5. The method according to claim 4, characterized in that The first loss function used in the iterative training of the first stage is determined based on the current video frame in the historical video data, the predicted video frame obtained after the current video frame is input into the first generator, and the reviewed video frame obtained after the current video frame is input into the second generator; The second loss function used in the iterative training of the second stage is determined based on the current video frame, the predicted video frame, the reviewed video frame in the historical video data, and the abnormal predicted video frame obtained after the historical pseudo-abnormal video data is input into the discriminator.

6. The method according to claim 4, characterized in that The iterative training of the second stage includes a discriminator training stage and a discriminator optimization stage; Using the historical video data in the training data set, the historical pseudo-abnormal video data, the first generator, and the second generator, the discriminator in the multi-branch adversarial network is subjected to a second-stage iterative training, including: In the discriminator training stage, the historical video data, the predicted video data output by the first generator, and the reviewed video data output by the second generator are used to train the discriminator; wherein the training goal is to enable the discriminator to distinguish between the predicted video data and the reviewed video data and the video data generated from the historical video data; In the discriminator optimization stage, the historical video data and the historical pseudo-abnormal video data are used to optimize the discrimination; wherein the optimization goal is to enable the discriminator to distinguish the abnormal video data in the historical video data and the historical pseudo-abnormal video data.

7. A video detection device, characterized in that: The device comprises: A video data acquisition module, used to acquire video data to be detected; A video data detection module is used to process the video data using a trained detection model to obtain a detection result corresponding to the video data; the detection model is obtained by training a multi-branch generative adversarial network using a training data set; the training data set includes a plurality of historical video data without abnormal behaviors, and historical pseudo-abnormal video data obtained by performing abnormal processing on the historical video data using a pre-built pseudo-abnormal processing module; The detection result output module is used to determine target video data corresponding to the abnormal behavior in the video data when the detection result is that the abnormal behavior exists in the video data, and output the target video data.

8. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When executing the computer-executable instructions, the processor is used to implement the video detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the video detection method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images