Video-based anomaly detection method, device and equipment, and storage medium

Abstract

The application provides a video-based anomaly detection method, device and equipment and a storage medium, comprising: acquiring an image sequence of a video to be detected; inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on an image sequence sample and a vehicle running angle of a target vehicle in the image sequence sample; and determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter. The application trains the behavior detection model by combining the vehicle running angle of the image sequence sample, so that the model can accurately distinguish dangerous driving situations such as vehicle rollover, sudden stop, lane change and sharp turn, thereby improving the accuracy of the model in detecting abnormal vehicle driving behavior.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic safety, and in particular to an abnormality detection method and device based on video, equipment and a storage medium. BACKGROUND

[0002] Dangerous driving behavior poses a great threat to urban traffic and pedestrian safety. At present, the vehicle driving image data is usually obtained based on the monitoring camera installed on the road, and then the vehicle running speed is obtained by target recognition on the vehicle driving image data, and then the vehicle running speed is directly compared with a pre-set threshold value to determine the dangerous driving behavior according to the comparison result, or the distance between the road marking and the vehicle position is calculated, and the distance and the pre-set distance threshold value are compared to determine the dangerous driving behavior. The abnormality judgment index is too single, and there is a certain randomness, which leads to low accuracy of vehicle abnormal driving behavior detection. SUMMARY

[0003] The present application provides an abnormality detection method and device based on video, equipment and a storage medium, which aims to improve the accuracy of vehicle abnormal driving behavior detection.

[0004] The present application provides an abnormality detection method based on video, comprising:

[0005] obtaining an image sequence of a video to be detected;

[0006] inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on an image sequence sample and a vehicle running angle of a target vehicle in the image sequence sample;

[0007] determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold value.

[0008] Optionally, according to the abnormality detection method based on video provided by the present application, the behavior detection model is obtained by training based on the following steps:

[0009] obtaining an image sequence sample in a driving behavior video;

[0010] inputting each training image in the image sequence sample into a pose estimation model to obtain a vehicle running angle of a target vehicle in each training image output by the pose estimation model;

[0011] for any one training image in the image sequence sample, inputting the training image into an initial network model to obtain a reconstructed image output by the initial network model;

[0012] calculate a first model loss value based on the reconstructed image, the training image, and a vehicle running angle of the target vehicle in each of the training images;

[0013] update the initial network model based on the first model loss value obtained in each iteration to obtain the behavior detection model.

[0014] Optionally, according to the video-based anomaly detection method provided in the present application, the calculation of the first model loss value based on the reconstructed image, the training image, and a vehicle running angle of the target vehicle in each of the training images comprises:

[0015] calculate an image loss value based on the reconstructed image and the training image;

[0016] calculate a vehicle angle change value based on a vehicle running angle of the target vehicle in each of the training images;

[0017] add the image loss value and the vehicle angle change value to obtain the first model loss value.

[0018] Optionally, according to the video-based anomaly detection method provided in the present application, the pose estimation model is trained based on the following steps:

[0019] input a training image in the image sequence sample into a to-be-trained pose estimation model to obtain a vehicle running angle of the target vehicle in the training image output by the to-be-trained pose estimation model;

[0020] calculate a second model loss value based on the vehicle running angle and an angle label of the target vehicle in the training image;

[0021] update the to-be-trained pose estimation model based on the second model loss value obtained in each iteration to obtain the pose estimation model.

[0022] Optionally, according to the video-based anomaly detection method provided in the present application, the to-be-trained pose estimation model comprises a feature extraction network, a target detection network, and an angle estimation network;

[0023] the input of the training image in the image sequence sample into the to-be-trained pose estimation model to obtain the vehicle running angle of the target vehicle in the training image output by the to-be-trained pose estimation model comprises:

[0024] input the training image in the image sequence sample into the feature extraction network to obtain feature information of a plurality of candidate regions output by the feature extraction network;

[0025] Input the feature information of each of the candidate regions into a target detection network to obtain a target detection result output by the target detection network, the target detection result including target category information;

[0026] Input the feature information of the candidate region corresponding to the vehicle category to which the target category information belongs into an angle estimation network to obtain a vehicle running angle output by the angle estimation network.

[0027] Optionally, according to the video-based abnormality detection method provided in the application, after the vehicle behavior detection result corresponding to the image sequence is determined based on the detection result and a preset threshold parameter, the method further includes:

[0028] If the vehicle behavior detection result is an abnormal driving behavior, an abnormal behavior alarm record is generated based on license plate information corresponding to the abnormal driving behavior, and the abnormal behavior alarm record and the to-be-detected video are cached in a preset database, so as to query the abnormal behavior alarm record and the to-be-detected video corresponding to the license plate information based on the license plate information.

[0029] Optionally, according to the video-based abnormality detection method provided in the application, after the vehicle behavior detection result corresponding to the image sequence is determined based on the detection result and a preset threshold parameter, the method further includes:

[0030] If the vehicle behavior detection result is an abnormal driving behavior, a stop instruction is sent to a detection vehicle corresponding to the abnormal driving behavior to control the detection vehicle to stop driving.

[0031] The application further provides a video-based abnormality detection device, including:

[0032] An acquisition module is configured to acquire an image sequence of a to-be-detected video.

[0033] A detection module is configured to input the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model, wherein the behavior detection model is obtained by training based on image sequence samples and vehicle running angles of target vehicles in the image sequence samples.

[0034] A determination module is configured to determine a vehicle behavior detection result corresponding to the image sequence based on the detection result and a preset threshold parameter.

[0035] The application further provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video-based abnormality detection method according to any one of the above embodiments when executing the program.

[0036] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the video-based anomaly detection method according to any one of the above.

[0037] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the video-based anomaly detection method according to any one of the above.

[0038] The video-based anomaly detection method, device, equipment and storage medium provided by the application can train a behavior detection model by combining the vehicle running angle of the image sequence sample, so that the model can accurately distinguish dangerous driving situations such as vehicle rollover, sudden stop, lane change and sharp turn, thereby improving the accuracy of the behavior detection model in detecting abnormal driving behaviors of vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description one by one. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 is one of the flowcharts of the video-based anomaly detection method provided by the application;

[0041] Figure 2 is another flowchart of the video-based anomaly detection method provided by the application;

[0042] Figure 3 is a structural schematic diagram of an initial network model provided by the embodiment of the application;

[0043] Figure 4 is a third flowchart of the video-based anomaly detection method provided by the application;

[0044] Figure 5 is a flowchart of identifying the vehicle running angle of the target vehicle provided by the embodiment of the application;

[0045] Figure 6 is a structural schematic diagram of the video-based anomaly detection device provided by the application;

[0046] Figure 7 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0048] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "an" and "the" used in one or more embodiments of the present application are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present application means and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, etc. can be used in one or more embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determination" or "in response to determining".

[0050] The technical solutions of the present application will be described below in combination with Figures 1-5 The example embodiments of the present application are described in detail.

[0051] Figure 1 is one of the flow diagrams of the video-based anomaly detection method provided by the present application. As shown in Figure 1 The video-based anomaly detection method comprises the following steps:

[0052] Step 11, acquiring an image sequence of a video to be detected;

[0053] It should be noted that the image sequence represents an image sequence formed in time sequence in the to-be-detected video. Specifically, the shooting video of each traffic camera of the required detection section (i.e., the to-be-detected video in the embodiment) is acquired, and each image frame in the to-be-detected video is sent to a behavior detection model for behavior detection. In the embodiment, the image frames can be extracted from the to-be-detected video by a pre-set video frame extraction device. Preferably, the video frame extraction device is provided with a license plate recognition function and a vehicle position recognition function, so as to send the image sequence of the to-be-detected video and the license plate information and vehicle position information recognized in each image to the behavior detection model. If the video frame extraction device does not have the license plate recognition function, the license plate information of the image frame can be recognized by a pre-set license plate recognition algorithm after the image frame is extracted. The license plate recognition algorithm includes projection statistics using gradient information, license plate region scanning connection algorithm, using region features to train a classifier to recognize a license plate, and optical character recognition algorithm, etc. In addition, the image sequence sent to the behavior detection model also includes the traffic camera identifier of the shooting video and the timestamp of each image in the to-be-detected video.

[0054] Step 12, inputting the image sequence into the behavior detection model to obtain a detection result output by the behavior detection model.

[0055] The behavior detection model is obtained by training based on the image sequence sample and the vehicle running angle of the target vehicle in the image sequence sample.

[0056] Specifically, the image sequence is input into the behavior detection model, and the detection result is obtained according to the output result of the behavior detection model. The behavior detection model is obtained by training based on the image sequence sample and the vehicle running angle of the image sequence sample. It should be noted that the vehicle running angle is obtained by angle prediction of each training image in the image sequence sample by a pose estimation model, and the pose estimation model is obtained by iterative training based on the image sequence sample and the angle label of each training image in the image sequence sample. It can be understood that the behavior detection model can effectively identify the driving behavior of the vehicle in the image after training, and obtain the detection result.

[0057] Step 13, determining the vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter.

[0058] It should be noted that the threshold parameter is set according to the actual situation in the model verification stage, and the threshold parameter is specifically set as follows: after obtaining the behavior detection model through training, a plurality of unlabeled verification image sequences are input into the behavior detection model to obtain a model output result, and then the model output result is compared with a preset initial threshold to obtain a behavior detection result, and then the accuracy of the behavior detection result of the plurality of verification image sequences is calculated, if the accuracy meets the preset accuracy, the initial threshold is taken as the threshold parameter, and if the accuracy does not meet the preset accuracy, the initial threshold is adjusted until the threshold parameter meeting the preset accuracy is obtained.

[0059] Specifically, the detection result is compared with the threshold parameter, if the detection result is greater than the threshold parameter, it is determined that the vehicle behavior detection result is abnormal driving behavior, and if the detection result is not greater than the threshold parameter, it is determined that the vehicle behavior detection result is normal driving behavior.

[0060] The embodiment of the application obtains an image sequence of a to-be-detected video, inputs the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model, wherein the behavior detection model is obtained by training based on an image sequence sample and a vehicle running angle of a target vehicle in the image sequence sample, and determines a vehicle behavior detection result corresponding to the image sequence based on the detection result and a preset threshold parameter. The training of the behavior detection model is realized by combining the vehicle running angle of the image sequence sample, so that the model can accurately distinguish dangerous driving situations such as vehicle rollover, sudden stop, lane change and sharp turn, thereby improving the accuracy of the behavior detection model in video-based abnormal detection of vehicles.

[0061] In an embodiment of the application, Figure 2 is a flowchart of a video-based abnormal detection method provided by the application, as shown in Figure 2 The behavior detection model is obtained by training based on the following steps:

[0062] Step 21, obtaining an image sequence sample in a driving behavior video;

[0063] Step 22, respectively inputting each training image in the image sequence sample into a pose estimation model to obtain a vehicle running angle of a target vehicle in each training image output by the pose estimation model;

[0064] Step 23, for any one training image in the image sequence sample, inputting the training image into an initial network model to obtain a reconstructed image output by the initial network model;

[0065] Step 24, based on the reconstructed image, the training image and the vehicle running angle of the target vehicle in each training image, a first model loss value is calculated;

[0066] Step 25, based on the first model loss value obtained in each iteration, the initial network model is updated in parameters to obtain the behavior detection model.

[0067] It should be noted that the driving behavior video includes videos collected at different times and places according to different road scenes and vehicle types. The driving behavior video contains videos of abnormal driving behaviors to be detected, such as sudden acceleration, sudden deceleration, and sudden lane change.

[0068] It should be further noted that the pose estimation model is trained based on the training image and the angle label of the target vehicle in the image. The to-be-trained pose estimation model includes a feature extraction network, a target detection network and an angle estimation network, Figure 3 is a structural schematic diagram of the initial network model provided by the embodiment of the application, as Figure 3 shown, the initial network model is an encoder-decoder architecture network model.

[0069] Specifically, the driving behavior videos of different scene road scenes, different vehicle types and collected at different times and places can be collected by traffic cameras or searched on the network, and then each training image in the driving behavior video is labeled. Specifically, the lane lines, vehicles, pedestrians and other traffic information in each training image in the driving behavior video, the vehicle position information of the target vehicle in the training image and the angle label are labeled. Then, according to the time sequence of the training images in the driving behavior video, an image sequence sample is formed.

[0070] Further, for each training image in the image sequence sample, the following steps are performed: based on the training image input to the pose estimation model, the feature extraction network in the pose estimation model is used to extract features of the training image to obtain feature information of a plurality of candidate regions in the training image, the feature information of each candidate region is input to the target detection network, the target detection network is used for target detection to obtain a target detection result, and the feature information of the candidate region corresponding to the vehicle category to which the target category information belongs is input to the angle estimation network. The angle estimation network is used for pose estimation to obtain the vehicle running angle of the target vehicle in the training image.

[0071] Further, the training image is input into an encoder in the initial network model to extract feature information in the training image by the encoder, so as to compress the features in a high-dimensional space into low-dimensional features, and then the feature information is input into a decoder in the initial network model to decode the feature information by the decoder to obtain the reconstructed image. Then, based on the reconstructed image and the training image, an image loss value is calculated, and based on a vehicle running angle of a target vehicle in the training image and a vehicle running angle corresponding to a training image adjacent to the training image, an angle change value of the target vehicle in the training image is calculated. Further, the image loss value and the vehicle angle change value are added to obtain a first model loss value. After the first model loss value is calculated, the current training process is ended, and the model parameters in the initial network model are updated by using the error back propagation algorithm. Then, the next training is performed until the initial network model meets a preset training end condition to obtain a trained initial network model. It should be noted that the preset training end condition includes loss convergence and reaching a maximum iteration number threshold. Then, the decoder in the trained initial network model is removed to obtain the behavior detection model. That is, the network structure of the behavior detection model does not include the decoder, so that when an image is input into the behavior detection model for behavior recognition, a detection score output by the behavior detection model is obtained, and the detection score is taken as a detection result.

[0072] In the embodiment of the present application, the vehicle running angle of the target vehicle in the training image is identified based on the pose estimation model, and the vehicle running angles of continuous frames of images are added to the training process of the behavior detection model, so that dangerous driving situations such as vehicle rollover, sudden stop, lane change and sharp turn can be accurately distinguished, thereby facilitating improvement of the accuracy of the behavior detection model in behavior detection.

[0073] In an embodiment of the present application, the step 24 of calculating a first model loss value based on the reconstructed image, the training image and the vehicle running angles of the target vehicle in each training image comprises:

[0074] The image loss value is calculated based on the reconstructed image and the training image, the vehicle angle change value is calculated based on the vehicle running angles of the target vehicle in each training image, and the image loss value and the vehicle angle change value are added to obtain the first model loss value.

[0075] Specifically, the image loss value is calculated based on the reconstructed image and the training image by using a preset loss function, wherein the preset loss function includes an L1 loss function and an L2 loss function, and the image loss value is calculated as follows:

[0076] L res= |I - I net |

[0077] wherein, I represents a training image, I net represents a reconstructed image, L res represents an image loss value.

[0078] Further, based on the vehicle running angle of the target vehicle in the current training image and the vehicle running angle of the target vehicle in the training image adjacent to the training image, a vehicle angle change value is calculated in a second-order differential manner, wherein the second-order differential manner is as follows:

[0079] L pose = ∑ |θ t+1 - θ t | |θ t - θ t-1 |

[0080] wherein, L pose represents a vehicle angle change value, θ t represents the vehicle running angle of the current training image, θ t+1 and θ t-1 represent the vehicle running angles of two training images adjacent to the current training image. Further, the image loss value and the vehicle angle change value are added to obtain the first model loss value.

[0081] The vehicle angle change value is calculated by combining the vehicle running angles of consecutive frames of images in the embodiment of the application, so that the vehicle angle change value is introduced as the initial network model for abnormality detection, which makes the model able to accurately distinguish dangerous driving situations such as vehicle rollover, sudden stop, lane change and sharp turn, thereby improving the accuracy of behavior detection by the behavior detection model.

[0082] In an embodiment of the application, Figure 4 is a third flowchart of the abnormality detection method based on video provided by the application, as Figure 4 shown, the pose estimation model is trained based on the following steps:

[0083] Step 41, input the training image in the image sequence sample into the to-be-trained pose estimation model to obtain the vehicle running angle of the target vehicle in the training image output by the to-be-trained pose estimation model;

[0084] wherein, the above step 41 comprises:

[0085] inputting the training image in the image sequence sample into the feature extraction network to obtain feature information of a plurality of candidate regions output by the feature extraction network; inputting the feature information of each candidate region into a target detection network to obtain a target detection result output by the target detection network, the target detection result including target category information; inputting feature information of a candidate region corresponding to the target category information belonging to a vehicle category into an angle estimation network to obtain a vehicle running angle output by the angle estimation network.

[0086] Step 42, based on the vehicle running angle and the angle label of the target vehicle in the training image, a second model loss value is calculated;

[0087] Step 43, based on the second model loss value obtained in each iteration, the parameter of the to-be-trained pose estimation model is updated to obtain the pose estimation model.

[0088] It should be noted that the feature extraction network can be used to extract image features, and the feature extraction network is a convolutional neural network pre-trained based on a large amount of image data set, for example, ResNet-50 and Darknet53 neural networks. Specifically, the following operations are performed for each training image:

[0089] As shown in Figure 5 , Figure 5 is a flowchart of identifying the vehicle running angle of the target vehicle provided by the embodiment of the application. The training image is input into the feature extraction network to extract the features in the training image using the feature extraction network to obtain the feature information of a plurality of candidate regions output by the feature extraction network. The feature information of each candidate region is input into a target detection network to obtain a target detection result output by the target detection network. The target detection result includes boundary box information and confidence information corresponding to the candidate region. The boundary box information represents the position information of the candidate region in the training image, the width and length of the candidate region. The confidence information represents the target category information corresponding to the candidate region. Then, it is judged whether the target category information is a vehicle category. The feature information of the candidate region corresponding to the target category information belonging to the vehicle category is input into an angle estimation network to obtain a vehicle running angle of the target vehicle output by the angle estimation network. Preferably, the vehicle running angle is represented by Euler angle, wherein the Euler angle represents the rotation angle on the spatial coordinate axis. Generally, roll, pitch, yaw are used to represent the rotation values of these components, for example: 0 to 360° or 0 to 2 values are used to represent the rotation values respectively, wherein pitch is the rotation around the X axis, called the pitch angle, yaw is the rotation around the Y axis, called the yaw angle, and roll is the rotation around the Z axis, called the roll angle.

[0090] Additionally, since the feature information of the candidate region is subjected to Pooling quantization processing in the target detection process, resulting in a large error when the final obtained bounding box information is mapped to the training image, therefore, the feature information of the candidate region needs to be subjected to ROIAlign processing, wherein the ROIAlign processing represents a processing manner of calculating the pixel value corresponding to the floating-point coordinate value by using the bilinear interpolation method to fill the vacancy of the floating-point coordinate value pixel, so that the pixel values corresponding to the adjacent two integer coordinate values are continuous, avoiding the error of pooling quantization.

[0091] Further, based on the difference between the vehicle running angle and the angle label of the target vehicle in the training image, a second model loss value is calculated, and then based on the second model loss value obtained in each iteration, the training process is ended after the second model loss value is calculated, and the model parameters in the to-be-trained pose estimation model are updated using the error back propagation algorithm, and then the next training is performed until the to-be-trained pose estimation model meets the preset training end condition, the pose estimation model is obtained for parameter updating, and the pose estimation model is obtained.

[0092] The embodiment of the present application identifies the feature information of the candidate region corresponding to the vehicle class through the target detection network, calculates the vehicle running angle of the target vehicle in the training image based on the angle estimation network, updates the parameters of the model according to the vehicle running angle and the angle label of the target vehicle in the training image, and controls the loss value of the pose estimation model within a preset range, thereby facilitating to improve the precision of the pose estimation model in angle estimation.

[0093] In an embodiment of the present application, after determining the vehicle behavior detection result corresponding to the image sequence based on the detection result and the pre-set threshold parameter, the method further comprises:

[0094] If the vehicle behavior detection result is an abnormal driving behavior, an abnormal behavior alarm record is generated based on the license plate information corresponding to the abnormal driving behavior, and the abnormal behavior alarm record and the to-be-detected video are cached in a preset database, so as to query the abnormal behavior alarm record and the to-be-detected video corresponding to the license plate information based on the license plate information.

[0095] Specifically, since the image sent to the behavior detection model for behavior detection carries the timestamp corresponding to the image and the license plate information, if the vehicle behavior detection result is abnormal driving behavior, an abnormal behavior alarm record is generated based on the license plate information and the timestamp corresponding to the abnormal driving behavior, and the abnormal behavior alarm record and the to-be-detected video are cached in a preset database, so that the abnormal behavior alarm record corresponding to the license plate information can be queried according to the license plate information, the to-be-detected video corresponding to the abnormal behavior can be viewed, and the abnormal behavior alarm record and the to-be-detected video occurring at the time can be queried based on the time, thereby improving the intelligent level of traffic safety management.

[0096] In an embodiment of the present application, after the vehicle behavior detection result corresponding to the image sequence is determined based on the detection result and the preset threshold parameter, the method further comprises:

[0097] If the vehicle behavior detection result is abnormal driving behavior, a stop instruction is sent to the detection vehicle corresponding to the abnormal driving behavior to control the detection vehicle to stop driving.

[0098] Specifically, when the detection vehicle is a vehicle with a receiving control instruction function, for example, a special vehicle or an unmanned vehicle. If the vehicle behavior detection result of the detection vehicle is abnormal driving behavior, a stop instruction is sent to the unmanned vehicle by using a 5G network or a special vehicle networking, or according to the number or frequency of abnormal driving behavior of the detection vehicle, when the number or frequency of abnormal driving behavior meets a preset threshold, a stop instruction is sent to the unmanned vehicle to control the detection vehicle to stop driving. The preset threshold can be set according to actual conditions, which is not limited here. Thus, the unmanned vehicle with abnormal driving (for example, frequent lane switching and sudden acceleration and deceleration) is monitored in real time, and the vehicle is prevented from driving before an accident occurs.

[0099] The video-based anomaly detection device provided by the present application is described below. The video-based anomaly detection device described below can be referred to in correspondence with the video-based anomaly detection method described above.

[0100] Figure 6 is a structural schematic diagram of the video-based anomaly detection device provided by the present application, as Figure 6 shown, the video-based anomaly detection device of the embodiment of the present application comprises:

[0101] The acquisition module 61 is configured to acquire an image sequence of a to-be-detected video.

[0102] The detection module 62 is configured to input the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model, wherein the behavior detection model is trained based on image sequence samples and vehicle running angles of target vehicles in the image sequence samples.

[0103] The determination module 63 is configured to determine a vehicle behavior detection result corresponding to the image sequence based on the detection result and a preset threshold parameter.

[0104] The video-based anomaly detection device further comprises:

[0105] An image sequence sample in a driving behavior video is obtained.

[0106] Each training image in the image sequence sample is input into a pose estimation model respectively to obtain a vehicle running angle of a target vehicle in each training image output by the pose estimation model.

[0107] For any one training image in the image sequence sample, the training image is input into an initial network model to obtain a reconstructed image output by the initial network model.

[0108] Based on the reconstructed image, the training image and the vehicle running angle of the target vehicle in each training image, a first model loss value is calculated.

[0109] The initial network model is updated based on the first model loss value obtained in each iteration to obtain the behavior detection model.

[0110] The video-based anomaly detection device further comprises:

[0111] Based on the reconstructed image and the training image, an image loss value is calculated.

[0112] Based on the vehicle running angle of the target vehicle in each training image, a vehicle angle change value is calculated.

[0113] The image loss value and the vehicle angle change value are added to obtain the first model loss value.

[0114] The video-based anomaly detection device further comprises:

[0115] A training image in the image sequence sample is input into a to-be-trained pose estimation model to obtain a vehicle running angle of a target vehicle in the training image output by the to-be-trained pose estimation model.

[0116] Based on the vehicle running angle and an angle label of the target vehicle in the training image, a second model loss value is calculated.

[0117] Based on the second model loss value obtained in each iteration, the parameter of the to-be-trained pose estimation model is updated to obtain the pose estimation model.

[0118] The video-based anomaly detection device further comprises:

[0119] The to-be-trained pose estimation model comprises a feature extraction network, a target detection network and an angle estimation network.

[0120] The training image in the image sequence sample is input into the feature extraction network to obtain the feature information of a plurality of candidate regions output by the feature extraction network.

[0121] The feature information of each candidate region is input into the target detection network to obtain a target detection result output by the target detection network, wherein the target detection result comprises target category information.

[0122] The feature information of the candidate region corresponding to the vehicle category to which the target category information belongs is input into the angle estimation network to obtain a vehicle running angle output by the angle estimation network.

[0123] The video-based anomaly detection device further comprises:

[0124] If the vehicle behavior detection result is an abnormal driving behavior, an abnormal behavior alarm record is generated based on the license plate information corresponding to the abnormal driving behavior, and the abnormal behavior alarm record and the to-be-detected video are cached in a preset database, so as to query the abnormal behavior alarm record and the to-be-detected video corresponding to the license plate information based on the license plate information.

[0125] The video-based anomaly detection device further comprises:

[0126] If the vehicle behavior detection result is an abnormal driving behavior, a stop instruction is sent to the detection vehicle corresponding to the abnormal driving behavior to control the detection vehicle to stop driving.

[0127] It should be noted that the above device provided by the embodiment of the present application can realize all method steps realized by the above method embodiment, and can achieve the same technical effect. Therefore, the same parts and beneficial effects of the method embodiment will not be described in detail.

[0128] Figure 7 is a structural schematic diagram of an electronic device provided by the present application, such as Figure 7As shown, the electronic device can include a processor 710, a memory 720, a communications interface 730, and a communications bus 740, wherein the processor 710, the memory 720, and the communications interface 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logic instruction in the memory 720 to execute a video-based anomaly detection method, which includes: acquiring an image sequence of a to-be-detected video; inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on an image sequence sample and a vehicle running angle of a target vehicle in the image sequence sample; and determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter.

[0129] In addition, the logic instruction in the memory 720 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0130] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a video-based anomaly detection method provided by the above-mentioned methods, which includes: acquiring an image sequence of a to-be-detected video; inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on an image sequence sample and a vehicle running angle of a target vehicle in the image sequence sample; and determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter.

[0131] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to cause a computer to execute the video-based anomaly detection method provided by any of the above methods, the method comprising: obtaining an image sequence of a video to be detected; inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on image sequence samples and a vehicle running angle of a target vehicle in the image sequence samples; and determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter.

[0132] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0133] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in the sense of contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0134] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of video-based anomaly detection, characterized in that, The method comprises the following steps: obtaining an image sequence of a video to be detected; inputting the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model; wherein the behavior detection model is obtained by training based on image sequence samples and vehicle running angles of target vehicles in the image sequence samples; determining a vehicle behavior detection result corresponding to the image sequence based on the detection result and a pre-set threshold parameter; the behavior detection model is obtained by training based on the following steps: obtaining image sequence samples in a driving behavior video; inputting each training image in the image sequence samples into a pose estimation model to obtain a vehicle running angle of a target vehicle in each training image output by the pose estimation model; for any one training image in the image sequence samples, inputting the training image into an initial network model to obtain a reconstructed image output by the initial network model; based on the reconstructed image, the training image and the vehicle running angle of the target vehicle in each training image, a first model loss value is calculated; based on the first model loss value obtained in each iteration, the parameters of the initial network model are updated to obtain the behavior detection model; the first model loss value is calculated based on the reconstructed image, the training image and the vehicle running angle of the target vehicle in each training image, which comprises: based on the reconstructed image and the training image, an image loss value is calculated; based on the vehicle running angle of the target vehicle in each training image, a vehicle angle change value is calculated; the image loss value and the vehicle angle change value are added to obtain the first model loss value.

2. The video-based anomaly detection method of claim 1, wherein, the pose estimation model is obtained by training based on the following steps: inputting a training image in the image sequence samples into a to-be-trained pose estimation model to obtain a vehicle running angle of a target vehicle in the training image output by the to-be-trained pose estimation model; based on the vehicle running angle and an angle label of the target vehicle in the training image, a second model loss value is calculated; based on the second model loss value obtained in each iteration, the parameters of the to-be-trained pose estimation model are updated to obtain the pose estimation model.

3. The video-based anomaly detection method of claim 2, wherein, the to-be-trained pose estimation model comprises a feature extraction network, a target detection network and an angle estimation network; inputting a training image in the image sequence samples into the to-be-trained pose estimation model to obtain a vehicle running angle of a target vehicle in the training image output by the to-be-trained pose estimation model, which comprises: inputting the training image in the image sequence samples into the feature extraction network to obtain feature information of a plurality of candidate regions output by the feature extraction network; inputting the feature information of each candidate region into the target detection network to obtain a target detection result output by the target detection network, the target detection result comprising target category information; inputting the feature information of the candidate region corresponding to the vehicle category of the target category information into the angle estimation network to obtain the vehicle running angle output by the angle estimation network.

4. The video-based anomaly detection method of claim 1, wherein, The method further comprises: If the vehicle behavior detection result is an abnormal driving behavior, an abnormal behavior warning record is generated based on license plate information corresponding to the abnormal driving behavior, and the abnormal behavior warning record and the to-be-detected video are cached in a preset database to query the abnormal behavior warning record and the to-be-detected video corresponding to the license plate information based on the license plate information.

5. The video-based anomaly detection method of claim 1, wherein, The method further comprises: If the vehicle behavior detection result is an abnormal driving behavior, a stop instruction is sent to a detection vehicle corresponding to the abnormal driving behavior to control the detection vehicle to stop driving.

6. A video-based anomaly detection apparatus characterized by comprising: The method comprises: An acquisition module is configured to acquire an image sequence of a to-be-detected video. A detection module is configured to input the image sequence into a behavior detection model to obtain a detection result output by the behavior detection model, wherein the behavior detection model is obtained by training based on image sequence samples and vehicle running angles of target vehicles in the image sequence samples. A determination module is configured to determine a vehicle behavior detection result corresponding to the image sequence based on the detection result and a preset threshold parameter. The behavior detection model is obtained by training based on the following steps: Image sequence samples in a driving behavior video are acquired. Each training image in the image sequence samples is input into a pose estimation model to obtain a vehicle running angle of a target vehicle in each training image output by the pose estimation model. For any one training image in the image sequence samples, the training image is input into an initial network model to obtain a reconstructed image output by the initial network model. A first model loss value is calculated based on the reconstructed image, the training image, and the vehicle running angle of the target vehicle in each training image. The initial network model is updated based on the first model loss value obtained in each iteration to obtain the behavior detection model. The first model loss value is calculated based on the reconstructed image, the training image, and the vehicle running angle of the target vehicle in each training image, and comprises: An image loss value is calculated based on the reconstructed image and the training image. A vehicle angle change value is calculated based on the vehicle running angle of the target vehicle in each training image. The image loss value and the vehicle angle change value are added to obtain the first model loss value.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the video-based anomaly detection method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the video-based anomaly detection method of any one of claims 1 to 5.

Images