End-to-end vehicle behavior detection method, system, device and storage medium

Abstract

The present invention provides an end-to-end vehicle behavior detection method, system, device and storage medium, wherein the method includes: acquiring collected real-time video data and single-frame images; identifying and classifying target objects in single-frame images through convolutional neural networks; cropping the single-frame images for regions of interest according to the target objects to obtain at least one target image; adopting a multi-target tracking method to extract feature data, and tracking ID classification of target images according to the feature data to obtain ID information; screening out target images whose target objects are vehicles, obtaining corresponding continuous video clips, and performing interactive aggregation operations to obtain interactive relationships of target objects; combining ID information, continuous video clips and interactive relationships to perform behavior detection on target objects, outputting behavior information of target objects, and locating vehicles. The present invention can improve the accuracy, real-time performance and generalization ability of vehicle behavior detection, and has a wide range of applications.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle behavior detection, and in particular to an end-to-end vehicle behavior detection method, system, device and storage medium. Background Art

[0002] In recent years, with the continuous increase in the number of cars, traffic congestion and traffic violations have occurred frequently. How to use modern scientific and technological means to achieve efficient and accurate vehicle behavior detection has become a research hotspot in the current transportation field. Among them, because the vehicle behavior detection method based on computer vision is simple and easy to implement, it has been widely used in the fields of intelligent security, traffic monitoring and unmanned system camera detection. At present, the common image-based detection of parking violations in fixed areas includes the use of optical flow combined with target detection to determine whether the target exists, but the use scenario of this method is limited, and the vehicle behavior that can be detected is relatively single.

[0003] Some scholars use traditional image processing methods, such as background difference and optical flow trajectory tracking to realize the recognition of vehicle trajectories, but the accuracy of this method is not high, the generalization ability is poor, and it cannot be widely used in various scenarios. Some studies also combine lane line detection, obstacle detection, and traffic sign detection with vehicle position detection to realize the judgment of vehicle behavior. This method is highly complex and depends on the accuracy of information from multiple sensors. It cannot achieve end-to-end behavior detection and is difficult to apply in scenarios where traffic signs are not obvious or weather conditions are poor. Due to the development of neural networks, some scholars use convolutional neural networks to predict and identify vehicle behavior, but most of the functions tend to focus on the current motion state of a certain vehicle or the state of the driver, and lack the ability to recognize the behavior of traffic vehicle formations. Therefore, the prior art lacks a vehicle behavior detection method with a wide range of applications, high recognition accuracy, and strong generalization ability. Summary of the invention

[0004] Based on this, it is necessary to provide an end-to-end vehicle behavior detection method, system, device and storage medium to address the above technical problems.

[0005] An end-to-end vehicle behavior detection method comprises the following steps: acquiring collected real-time video data, and acquiring a single-frame image according to the real-time video data; identifying and classifying a target object in the single-frame image by a convolutional neural network; cropping an area of ​​interest of the single-frame image according to the target object to acquire at least one target image; extracting feature data of a target object in the target image by a multi-target tracking method, and performing tracking ID classification on the target image according to the feature data to acquire ID information; screening out a target image whose target object is a vehicle, acquiring continuous video segments according to the single-frame image corresponding to the target image, and performing interactive aggregation according to the single-frame image and the continuous video segments to acquire an interactive relationship between the target object and the remaining target objects in the single-frame image; performing behavior detection on the target object in combination with the ID information, the continuous video segments and the interactive relationship, outputting the behavior information of the target object according to the detection result, and locating the vehicle.

[0006] In one of the embodiments, the acquiring of the collected real-time video data and acquiring a single-frame image based on the real-time video data specifically includes: splitting the real-time video data according to seconds to acquire multiple single-frame images of one second; filtering out continuous video segments corresponding to the single-frame images based on the single-frame images, and saving the continuous video segments to a memory pool.

[0007] In one embodiment, the identifying and classifying the target object in the single-frame image by a convolutional neural network specifically includes: obtaining a number of historical video data, and splitting them into corresponding historical single-frame images, using the historical single-frame images as training sets, and the target objects corresponding to the historical single-frame images as test sets; constructing an initial target detection model based on the YOLOV5 convolutional neural network, training the initial target detection model according to the training set and the test set, and obtaining the target detection model after the training is completed; inputting the single-frame image into the target detection model, and outputting the corresponding target object; classifying the target objects in all single-frame images by a classifier, and storing the single-frame images according to the classification.

[0008] In one of the embodiments, the multi-target tracking method is used to extract feature data of the target object in the target image, and the target image is tracked and classified according to the feature data to obtain ID information, which specifically includes: classifying all target images according to the feature data, screening out target images with the same target object, storing the same ID tag for the target image, and obtaining corresponding ID information; classifying all target images according to the ID tag, and the classified target images have the behavioral characteristics of the target object.

[0009] In one of the embodiments, the target object is a target image of a vehicle, a continuous video segment is obtained according to a single-frame image corresponding to the target image, and an interactive relationship between the target object and other target objects in the single-frame image is obtained according to the single-frame image and the continuous video segment, specifically including: selecting a vehicle as a target vehicle object in the single-frame image, filtering out the corresponding target image and single-frame image according to the target vehicle object, and obtaining a continuous video segment corresponding to the single-frame image according to the single-frame image corresponding to the target vehicle object; sampling the continuous video segment through two channel paths, and analyzing the static content and dynamic content in the continuous video segment; determining whether there is only one target vehicle object in the single-frame image according to the static content and the dynamic content; if so, performing interactive aggregation according to the continuous video segment and the target image corresponding to the single-frame image, and obtaining the interactive relationship between the target vehicle object and other target objects; if not, performing interactive aggregation according to the continuous video segment and the target image corresponding to the single-frame image, and obtaining the interactive relationship between the target vehicle object and other vehicle target objects.

[0010] In one of the embodiments, the target object is subjected to behavior detection in combination with the ID information, continuous video clips and interaction relationship, the behavior information of the target object is outputted according to the detection result, and the vehicle is positioned, specifically comprising: detecting the behavior of the target vehicle object marked with the current ID according to the interaction relationship, continuous video clips and ID information to obtain the detection result; obtaining the pixel position, ID tag and vehicle behavior category of the target vehicle object according to the detection result; and obtaining the positioning information of the target vehicle object in combination with the drone height, camera angle and focal length obtained by real-time video data.

[0011] An end-to-end vehicle behavior detection system comprises: a single-frame image acquisition module, a target detection module, an image cropping module, a target tracking module, an interactive aggregation module and a behavior detection module; the single-frame image acquisition module is connected to the target detection module, the target detection module is connected to the image cropping module, the image cropping module is connected to the target tracking module, the target tracking module is connected to the interactive aggregation module, and the interactive aggregation module is connected to the behavior detection module; the target detection module is used to perform convolution operation on the target area in the single-frame image through a convolutional neural network, and classify the target object into categories through a classifier to complete target detection; the image cropping module is used to crop the image after detection to perform an area of ​​interest, obtain at least one target image, and input the target image into the target tracking module; the target tracking module is used to obtain real-time video data , and extract the feature data of the target object through a multi-task network, track the target object for all single-frame images according to the feature data, store the same ID tag for single-frame images with the same feature data, obtain ID information, transmit the continuous video segments, feature data and ID information corresponding to the single-frame images to the memory pool, and the memory pool transmits the information of the continuous video segments to the behavior detection module; the interactive aggregation module is used to screen out target images whose target objects are vehicles, obtain continuous video segments according to the single-frame images corresponding to the target images, perform interactive aggregation according to the single-frame images and continuous video segments, and obtain the interactive relationship between the target object and the remaining target objects in the single-frame images; the behavior detection module is used to perform behavior detection on the target object in combination with the ID information, continuous video segments and interactive relationship, output the behavior information of the target object according to the detection results, and locate the vehicle.

[0012] In one of the embodiments, the behavior detection module also includes a monocular vision positioning unit, which is used to obtain relevant information of the target object according to the detection results, and locate the target object according to the relevant information; the interaction aggregation module includes a V-type interaction aggregation module and an O-type interaction aggregation module, and the V-type interaction aggregation module and the O-type interaction aggregation module are second-order hybrid; the V-type interaction aggregation module is used to detect the interaction relationship between vehicles, and the input ends correspond to the target image of the vehicle target object, one of the input ends corresponds to the feature data input of the target vehicle, and the other input ends correspond to the special data input of the non-target vehicle, and the interaction relationship between the target vehicle and the non-target vehicle is obtained through the interaction aggregation operation; the O-type interaction aggregation module is used to detect the interaction relationship between the vehicle and other target objects, and the input ends correspond to the feature data of the target vehicle and the feature data of the non-vehicle object, and the interaction relationship between the vehicle and the non-vehicle object is obtained through the interaction aggregation operation.

[0013] A device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of an end-to-end vehicle behavior detection method described in the above-mentioned embodiments are implemented.

[0014] A storage medium stores a computer program, which, when executed by a processor, implements the steps of an end-to-end vehicle behavior detection method described in the above embodiments.

[0015] Compared with the prior art, the advantages and beneficial effects of the present invention are: the present invention can improve the accuracy, real-time and generalization ability of vehicle behavior detection, and has a wide range of applications, and can be used for including but not limited to traffic flow vehicle behavior recognition, vehicle behavior recognition in aerial images, and abnormal behavior detection and vehicle positioning in vehicle formations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of an end-to-end vehicle behavior detection method in an embodiment;

[0017] Figure 2 is a schematic diagram of the structure of an end-to-end vehicle behavior detection system in an embodiment;

[0018] Figure 3 A data processing architecture diagram of an end-to-end vehicle behavior detection system in one embodiment;

[0019] Figure 4 for Figure 3 Schematic diagram of the operating principle of the O-type interaction aggregation module;

[0020] Figure 5 for Figure 3 Schematic diagram of the operation principle of the V-shaped interaction aggregation module;

[0021] Figure 6 Schematic diagram of the internal structure of a device in one embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific implementation methods in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] This patent is based on the latest spatiotemporal behavior detection method AIA (Asynchronous Interaction Aggregation) to process real-time video images collected from security monitoring and unmanned system reconnaissance. It can be used for including but not limited to traffic flow vehicle behavior recognition, vehicle behavior recognition in aerial images, and detection of abnormal behavior in vehicle formations. Its vehicle behavior mainly includes car acceleration and deceleration, abnormal lane changes, car crossing the line, and abnormal driving of vehicles in car formations. In addition, the monocular visual positioning algorithm can also be used to measure the distance, locate, lock the position of the vehicle, and alarm or remind the identified abnormally driving vehicle. In the drone system, this application can take advantage of the drone's aerial flight to achieve visual positioning of the target vehicle, continuous target tracking and reconnaissance, and other functions, providing an end-to-end solution for vehicle behavior detection, target vehicle positioning and tracking.

[0024] In one embodiment, Figure 1 As shown, an end-to-end vehicle behavior detection method is provided, comprising the following steps:

[0025] Step S101, acquiring collected real-time video data, and acquiring a single frame image according to the real-time video data.

[0026] Specifically, real-time video data in roads or parking areas can be acquired through unmanned system reconnaissance, and a number of single-frame images can be split according to the real-time video data to facilitate acquisition and processing of the images.

[0027] Step S102, identifying and classifying the target object in the single frame image through a convolutional neural network.

[0028] Specifically, the convolutional neural network YOLOV5 is used to perform convolution operations on the target area in the single-frame image, and the classifier is used to classify the target object. For example, the target object can be a vehicle, pedestrian, lane mark, obstacle, etc., so as to obtain the corresponding single-frame image.

[0029] Step S103 , cropping a region of interest on a single frame image according to the target object to obtain at least one target image.

[0030] Specifically, according to the area where the target object is located, the single-frame image is cropped for the region of interest to obtain at least one target image. If there are multiple target objects, multiple target images are obtained, and the multiple target images all correspond to the single-frame image.

[0031] Specifically, when cropping, the target object in the single-frame image is framed by a detection frame to obtain the target area, and ROI (region of interest) cropping is performed according to the size of the detection frame. The cropped image is used as the target image. When there are multiple target objects, multiple target images can be obtained based on the single-frame image. The multiple target images all correspond to the single-frame image, which facilitates the subsequent search for the corresponding single-frame image based on the target image.

[0032] Step S104: adopt a multi-target tracking method to extract feature data of the target object in the target image, perform tracking ID classification on the target image according to the feature data, and obtain ID information.

[0033] Specifically, a multi-target tracking method is adopted, and a multi-task network is used to extract feature data from the cropped target image. The target objects in all target images are matched according to the feature data, the same target objects are marked with the same ID tag, the ID information is obtained, and the target images are classified according to the ID tags. The classified target objects have the behavioral characteristics of the target objects and can identify the behavior of the target image based on the target image.

[0034] Step S105, filter out the target image whose target object is a vehicle, obtain continuous video segments based on the single-frame image corresponding to the target image, perform interactive aggregation based on the single-frame image and the continuous video segments, and obtain the interactive relationship between the target object and the remaining target objects in the single-frame image.

[0035] Specifically, target images whose target objects are vehicles are screened out, one of the vehicles is taken as the target vehicle object, and combined with the ID tag, all target images of the vehicle are further obtained, and based on the single-frame image corresponding to the target image, a continuous video segment from, for example, the previous three seconds to the current moment is obtained, and the target objects in the single-frame image are interactively aggregated based on the continuous video segment to obtain the interactive relationship between the target vehicle object and the remaining target objects in the single-frame image.

[0036] Step S106, performing behavior detection on the target object in combination with the ID information, the continuous video clips and the interaction relationship, outputting the behavior information of the target object according to the detection result, and locating the vehicle.

[0037] Specifically, the target object's behavior is detected by combining ID information, continuous video clips and interactive relationships, and the detection results including the target object's behavior are obtained, and the corresponding behavior information is output. The behavior information may include the pixel position, ID tag and behavior category of the target object. The vehicle is positioned according to the pixel position in the behavior information to achieve end-to-end vehicle behavior detection.

[0038] In this embodiment, the collected real-time video data is obtained, and single-frame images are split according to the real-time video data, the target object in the single-frame image is identified and classified by a convolutional neural network, the single-frame image is cropped with a region of interest according to the target object, at least one target image is obtained, a multi-target tracking method is used to extract feature data of the target object in the target image, the target image is tracked and classified according to the feature data, ID information is obtained, the target image of the target object for the vehicle is screened out, continuous video clips are obtained according to the single-frame image corresponding to the target image, behavior detection of the target object is performed in combination with the ID information and the continuous video clips, and the behavior information of the target object is output according to the detection result, which can improve the accuracy, real-time and generalization ability of vehicle behavior detection, and has a wide range of applications, including but not limited to traffic flow vehicle behavior recognition, vehicle behavior recognition in aerial images, and detection of abnormal behavior in vehicle formations.

[0039] Among them, step S101 specifically includes: splitting the real-time video image according to the number of seconds to obtain multiple single-frame images of one second; filtering out continuous video segments corresponding to the single-frame images according to the single-frame images, and saving the continuous video segments to the memory pool.

[0040] Specifically, the acquired real-time video data is split according to the number of seconds, so as to obtain multiple single-frame images of one second. For example, if the real-time video data of a road in a certain time period is obtained for 1 minute, the real-time video data of 1 minute is split to obtain 60 single-frame images corresponding to each second; the continuous video segments corresponding to the single-frame images are screened out according to the single-frame images. For example, if the current time of the single-frame image is 12:25:25 on a certain day of a certain year, a continuous video segment from 12:25:22 to 25 seconds of the corresponding date can be obtained, and the continuous video segment is saved in the memory pool. The continuous video segment corresponds to the single-frame image one by one, which is convenient for subsequent screening of continuous video segments. Among them, the time period of the continuous video segment can be set accordingly as needed, for example, it is set to the segment from the first three seconds to the current moment, or the segment from the second before the current moment to the second after the current moment. The time period of the continuous video segment should not be set too long, and can be kept within 5 seconds, otherwise it may be too many objects, which is not conducive to the subsequent behavior detection of the target object.

[0041] In order to improve the accuracy of behavior detection, continuous video clips will be extracted from the memory pool as the basis for behavior detection. At the same time, the data and image information of the memory pool will be updated in real time to prevent excessive storage and memory consumption.

[0042] Among them, step S102 specifically includes: obtaining a number of historical video data, and splitting them into corresponding historical single-frame images, using the historical single-frame images as training sets, and the target objects corresponding to the historical single-frame images as test sets; building an initial target detection model based on the YOLOV5 convolutional neural network, training the initial target detection model according to the training set and the test set, and obtaining the target detection model after the training is completed; inputting the single-frame image into the target detection model, and outputting the corresponding target object; classifying the target objects in all single-frame images through a classifier, and storing the single-frame images by classification.

[0043] Specifically, a number of historical video data are obtained, which may be relevant video data of vehicles traveling on the road, and the historical video data are split into corresponding historical single-frame images. The historical single-frame images are used as training sets, and the corresponding target objects are used as test sets. An initial target detection model is constructed based on the YOLOV5 convolutional neural network, and the initial target detection model is trained through the training set. The generalization ability of the trained initial target detection model is judged based on the test machine. After the generalization ability reaches the preset requirements, the target detection model is obtained. The target detection model can detect vehicles and other targets related to vehicle behavior, such as behavior, lane line markings or obstacles.

[0044] The single-frame image is input into the target detection model, and the target object in the single-frame image is identified by the target detection model, such as vehicles, pedestrians, lane markings, obstacles and other target objects. According to the identified target, the single-frame image is classified and stored by the classifier, that is, the single-frame images of the same category are classified and stored for subsequent behavior detection. Of course, when performing behavior detection, only the behavior of vehicles or pedestrians is identified and detected.

[0045] Among them, step S104 specifically includes: according to the characteristic data, screening out target images with the same target object, marking the target images with the same ID tag, and obtaining corresponding ID information; classifying all target images according to the ID tag, and the classified target images have the behavioral characteristics of the target object.

[0046] Specifically, according to the extracted feature data, the feature data in all target images are matched, target images with the same target object are screened out, the same ID tag is stored for target images with the same target object, and the target images are classified according to the ID tag. The classified target images have the behavioral characteristics of the target object, and the behavioral characteristics of the corresponding target object can be identified according to the target image. For example, target image 1 has feature data 1 of vehicle 1, and all target images are matched according to feature data 1 to determine whether there are target images with feature data 1. If so, the target image is marked with ID tag 1, and all target images with ID tag 1 are stored in the same location. All target images 1 have the behavioral characteristics of vehicle 1. When the behavior detection is performed later, all target images of vehicle 1 can be obtained according to the ID tag.

[0047] Among them, step S105 specifically includes: selecting a vehicle as a target vehicle object in a single-frame image, screening out corresponding target images and single-frame images according to the target vehicle object, and obtaining continuous video segments corresponding to the single-frame image according to the single-frame image corresponding to the target vehicle object; sampling the continuous video segments through two channel paths, analyzing static content and dynamic content in the continuous video segments; determining whether there is only one target vehicle object in the single-frame image according to the static content and dynamic content; if so, performing interactive aggregation according to the continuous video segments and target images corresponding to the single-frame image, and obtaining the interactive relationship between the target vehicle object and the remaining target objects; if not, performing interactive aggregation according to the continuous video segments and target images corresponding to the single-frame image, and obtaining the interactive relationship between the target vehicle object and the remaining vehicle target objects. .

[0048] Specifically, a vehicle is selected as the target vehicle object in a single-frame image, and the corresponding target image and single-frame image are screened out according to the target vehicle object. For example, if the target vehicle object is vehicle 1, the corresponding target image and single-frame image are obtained according to vehicle 1, and the corresponding continuous video clips are obtained according to the single-frame image. The continuous video clips are sampled through two channel paths, one of which is a slow channel for analyzing the static content in the continuous video clips; and the other is a fast channel for analyzing the dynamic content in the continuous video clips. Both the slow channel and the fast channel use the 3DRestNet model, and the slowfast algorithm is used to classify and output the actions of the target object in the time series.

[0049] For example, based on the static content and dynamic content obtained through analysis, it is determined whether there is only one target vehicle object, vehicle 1, in a single frame image. If there is only one target vehicle object, the target vehicle object is interactively aggregated with the remaining target objects based on the single frame image and the continuous video clips to obtain the interactive relationship between the target vehicle object and the remaining target objects. If there are vehicles 2 and 3 in addition to vehicle 1 in a single frame image, interactive aggregation is performed based on the target images of vehicles 1, 2, and 3 corresponding to the single frame image to obtain the interactive relationship between vehicles 1, 2, and 3.

[0050] Step S106 specifically includes: detecting the behavior of the target object marked with the current ID according to the interactive relationship, continuous video clips and ID information to obtain the detection result; obtaining the pixel position, ID information and vehicle behavior category of the target object according to the detection result; and obtaining the positioning information of the target object by combining the drone height, camera angle and focal length obtained by real-time video data.

[0051] Specifically, when there is only one vehicle 1 in a single-frame image, the behavior detection model is used to perform behavior detection on the target object marked by the current ID, determine the relationship between vehicle 1 and other target objects such as pedestrians and lane line markings, and obtain the behavior detection result of vehicle 1. In addition to vehicle 1, there are also vehicles 2 and 3 in the single-frame image. After obtaining the interaction relationship between vehicles 1, 2 and 3, the corresponding relationship between vehicle 1 and vehicles 2 and 3 in terms of position, characteristics and action is determined based on the interaction relationship, continuous video clips and the ID information of vehicle 1, so as to realize the behavior judgment of vehicle 1 in the vehicle queue or cluster, that is, the behavior detection of vehicle 1 is performed, and the behavior detection result of vehicle 1 is obtained.

[0052] According to the behavior detection results of vehicle 1, the pixel position, ID information and behavior category of vehicle 1 in the single-frame image are output. Through other state parameters of the unmanned system, such as the height of the drone, the angle and focal length of the camera, etc., combined with the pixel position of vehicle 1, the longitude and latitude positioning of vehicle 1 is realized, and the longitude and latitude positioning information of vehicle 1 is obtained.

[0053] like Figure 2 As shown, an end-to-end vehicle behavior detection system 20 is provided, which is connected to an unmanned system to obtain real-time video data, including: a single-frame image acquisition module 21, a target detection module 22, an image cropping module 23, a target tracking module 24, an interactive aggregation module 25 and a behavior detection module 26; the single-frame image acquisition module 21 is connected to the target detection module 22, the target detection module 22 is connected to the image cropping module 23, the target image cropping module 23 is connected to the target tracking module 24, the target tracking module 24 is connected to the interactive aggregation module 25, and the interactive aggregation module 25 is connected to the behavior detection module 26;

[0054] The single-frame image acquisition module 21 is used to acquire the collected real-time video data and acquire a single-frame image according to the real-time video data;

[0055] The target detection module 22 is used to perform convolution operation on the target area in the single frame image through a convolutional neural network, and classify the target object into categories through a classifier to complete target detection;

[0056] An image cropping module 23 is used to crop the detected image to a region of interest, obtain at least one target image, and input the target image to the target tracking module 24;

[0057] The target tracking module 24 is used to obtain real-time video data, extract feature data of the target object through a multi-task network, track the target object for all single-frame images according to the feature data, store the same ID tag for single-frame images with the same feature data, obtain ID information, and transmit the continuous video segments, feature data and ID information corresponding to the single-frame images to the memory pool, and the memory pool transmits the information of the continuous video segments to the behavior detection module 26;

[0058] An interactive aggregation module 25 is used to filter out a target image in which the target object is a vehicle, obtain continuous video segments according to a single-frame image corresponding to the target image, and interactively aggregate the single-frame image and the continuous video segments to obtain an interactive relationship between the target object and other target objects in the single-frame image;

[0059] The behavior detection module 26 is used to perform behavior detection on the target object in combination with the ID information, the continuous video clips and the interaction relationship, output the behavior information of the target object according to the detection result, and locate the vehicle.

[0060] In one embodiment, Figure 3 As shown, the behavior detection module 26 also includes: a monocular vision positioning unit, which is used to obtain relevant information of the target object according to the detection result, and locate the target object according to the relevant information.

[0061] Specifically, after the behavior of the vehicle is detected by the behavior detection module 26, the target vehicle object is located by the monocular vision positioning unit. When the target vehicle object is located, based on the target behavior recognition, according to the monocular vision ranging principle and combined with the coordinate posture and other information of the camera, the latitude and longitude data of the vehicle or the real-time distance data is estimated to obtain the positioning information of the target vehicle, so that when there is an abnormality in the driving state of the vehicle, the vehicle and the surrounding vehicles or pedestrians can be warned according to the positioning information.

[0062] Among them, Figure 4 and Figure 5As shown, the interactive aggregation module includes a V-type interactive aggregation module and an O-type interactive aggregation module, and the V-type interactive aggregation module and the O-type interactive aggregation module are second-order hybrid; the V-type interactive aggregation module is used to detect the interactive relationship between vehicles, and the input ends correspond to the target image of the vehicle target object, one of the input ends corresponds to the feature data input of the target vehicle, and the other input ends correspond to the special data input of the non-target vehicle. Through the interactive aggregation operation, the interactive relationship between the target vehicle and the non-target vehicle is obtained; the O-type interactive aggregation module is used to detect the interactive relationship between the vehicle and other target objects, and the input ends correspond to the feature data of the target vehicle and the feature data of the non-vehicle object. Through the interactive aggregation operation, the interactive relationship between the vehicle and the non-vehicle object is obtained.

[0063] Specifically, since the behavior of the vehicle is not only related to the state of the vehicle itself, but also to the interaction with the target objects in the surrounding scene, the interaction between the vehicle and the remaining target objects can be identified through V-type and O-type interaction aggregation modules. Among them, the V-type interaction aggregation module is used to identify the action relationship between vehicles, such as going in the wrong direction, queue anomalies, and traffic accidents. The O-type interaction aggregation module is used to identify the action relationship between vehicles and other traffic signs, such as running a red light, crossing the line, hitting obstacles or pedestrians, etc. The two can be connected in series, in parallel, and mixed. The present application adopts a second-order mixed method, which enables each interaction module to receive the output of all blocks and can be aggregated through learnable weights.

[0064] Specifically, the inputs of the O-type interaction aggregation module all correspond to target images of vehicle target objects. For example, if there are three target objects in a single-frame image, the target images are input separately according to the cropped image of interest. The three target objects include a vehicle target object, and the rest are non-vehicle target objects. The feature data of the target image are operated through scale operations and softmax operations to realize the interaction between the vehicle and other target objects in the single-frame image and obtain the interaction relationship.

[0065] Specifically, the V-type interactive aggregation module is similar to the O-type interactive aggregation module, but all the inputs are inputs of vehicle target objects, among which one of the three input ends corresponds to the main vehicle object, and the other two correspond to the secondary vehicle objects. Through interactive aggregation operations, the interactive relationship between the main vehicle object and other vehicle objects in terms of position, characteristics and actions is realized, and the behavior judgment of the main vehicle in the vehicle queue and cluster is realized.

[0066] In one embodiment, a device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the device is used to store configuration templates and can also be used to store target web page data. The network interface of the device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an end-to-end vehicle behavior detection method is implemented.

[0067] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the device to which the scheme of the present application is applied. The specific device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0068] In one embodiment, a storage medium may also be provided, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method as described in the aforementioned embodiment. The computer may be part of the above-mentioned end-to-end vehicle behavior detection system.

[0069] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0070] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disk) and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0071] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. An end-to-end vehicle behavior detection method, characterized in that: The following steps are involved: Acquire the collected real-time video data, and acquire a single frame image according to the real-time video data; Recognizing and classifying the target object in the single frame image by using a convolutional neural network; Performing cropping of a region of interest on the single-frame image according to the target object to obtain at least one target image; A multi-target tracking method is used to extract feature data of the target object in the target image, and the target image is tracked and classified according to the feature data to obtain ID information; Filter out a target image whose target object is a vehicle, obtain continuous video segments according to a single-frame image corresponding to the target image, and perform interactive aggregation according to the single-frame image and the continuous video segments to obtain an interactive relationship between the target object and other target objects in the single-frame image, specifically including: selecting a vehicle as a target vehicle object in the single-frame image, filtering out corresponding target images and single-frame images according to the target vehicle object, and obtaining continuous video segments corresponding to the single-frame image according to the single-frame image corresponding to the target vehicle object; The continuous video segments are sampled through two channel paths, and the static content and dynamic content in the continuous video segments are analyzed; according to the static content and the dynamic content, it is determined whether there is only one target vehicle object in the single-frame image; if so, interactive aggregation is performed according to the continuous video segments and the target image corresponding to the single-frame image to obtain the interactive relationship between the target vehicle object and the non-vehicle target object; if not, interactive aggregation is performed according to the continuous video segments and the target image corresponding to the single-frame image to obtain the interactive relationship between the target vehicle object and the remaining vehicle target objects; The target object is subjected to behavior detection in combination with the ID information, the continuous video clips and the interaction relationship, the behavior information of the target object is output according to the detection result, and the vehicle is positioned.

2. The end-to-end vehicle behavior detection method according to claim 1, characterized in that: The acquiring of the collected real-time video data and acquiring a single frame image according to the real-time video data specifically includes: Splitting the real-time video data according to the number of seconds to obtain a plurality of single-frame images of one second; Continuous video segments corresponding to the single-frame image are screened out according to the single-frame image, and the continuous video segments are saved in a memory pool.

3. The end-to-end vehicle behavior detection method according to claim 1, characterized in that: The identifying and classifying the target object in the single frame image by using a convolutional neural network specifically includes: Acquire a number of historical video data and split them into corresponding historical single-frame images, use the historical single-frame images as training sets, and use target objects corresponding to the historical single-frame images as test sets; Building an initial target detection model based on the YOLOV5 convolutional neural network, training the initial target detection model according to the training set and the test set, and obtaining the target detection model after the training is completed; Input the single frame image into the target detection model, and output the corresponding target object; The target objects in all single-frame images are classified by the classifier, and the single-frame images are classified and stored.

4. The end-to-end vehicle behavior detection method according to claim 1, characterized in that: The method of adopting a multi-target tracking method, extracting feature data of a target object in the target image, and performing tracking ID classification on the target image according to the feature data to obtain ID information, specifically includes: Classify all target images according to the feature data, filter out target images with the same target object, store the same ID tag for the target images, and obtain corresponding ID information; All target images are classified according to the ID tags, and the classified target images have the behavior characteristics of the target objects.

5. The end-to-end vehicle behavior detection method according to claim 4, characterized in that: The step of performing behavior detection on the target object by combining the ID information, the continuous video clips and the interaction relationship, outputting behavior information of the target object according to the detection result, and locating the vehicle specifically includes: According to the interaction relationship, the continuous video clips and the ID information, the behavior of the target vehicle object marked by the current ID is detected to obtain the detection result; Acquire the pixel position, ID tag and vehicle behavior category of the target vehicle object according to the detection result; The positioning information of the target vehicle object is obtained by combining the drone altitude, camera angle and focal length that obtain real-time video data.

6. An end-to-end vehicle behavior detection system, connected to an unmanned system, to obtain real-time video data, characterized in that: include: Single-frame image acquisition module, target detection module, image cropping module, target tracking module, interaction aggregation module and behavior detection module; The single-frame image acquisition module is connected to the target detection module, the target detection module is connected to the image cropping module, the image cropping module is connected to the target tracking module, the target tracking module is connected to the interaction aggregation module, and the interaction aggregation module is connected to the behavior detection module; The target detection module is used to perform convolution operation on the target area in the single frame image through a convolutional neural network, and classify the target object into categories through a classifier to complete target detection; The image cropping module is used to crop the region of interest of the image after detection, obtain at least one target image, and input the target image into the target tracking module; The target tracking module is used to obtain real-time video data, extract feature data of the target object through a multi-task network, track the target object for all single-frame images according to the feature data, store the same ID tag for single-frame images with the same feature data, obtain ID information, and transmit the continuous video segments, feature data and ID information corresponding to the single-frame images to the memory pool, and the memory pool transmits the information of the continuous video segments to the behavior detection module; The interactive aggregation module is used to filter out a target image whose target object is a vehicle, obtain continuous video segments according to the single-frame image corresponding to the target image, and perform interactive aggregation according to the single-frame image and the continuous video segments to obtain the interactive relationship between the target object and the remaining target objects in the single-frame image, specifically including: selecting a vehicle as a target vehicle object in the single-frame image, filtering out the corresponding target image and single-frame image according to the target vehicle object, and obtaining the continuous video segments corresponding to the single-frame image according to the single-frame image corresponding to the target vehicle object; The continuous video segments are sampled through two channel paths, and the static content and dynamic content in the continuous video segments are analyzed; according to the static content and the dynamic content, it is determined whether there is only one target vehicle object in the single-frame image; if so, interactive aggregation is performed according to the continuous video segments and the target image corresponding to the single-frame image to obtain the interactive relationship between the target vehicle object and the non-vehicle target object; if not, interactive aggregation is performed according to the continuous video segments and the target image corresponding to the single-frame image to obtain the interactive relationship between the target vehicle object and the remaining vehicle target objects; The behavior detection module is used to perform behavior detection on the target object in combination with the ID information, continuous video clips and interaction relationships, output the behavior information of the target object according to the detection result, and locate the vehicle.

7. The end-to-end vehicle behavior detection system according to claim 6, characterized in that: The behavior detection module also includes a monocular visual positioning unit, which is used to obtain relevant information of the target object according to the detection result, and locate the target object according to the relevant information; The interactive aggregation module includes a V-type interactive aggregation module and an O-type interactive aggregation module, and the V-type interactive aggregation module and the O-type interactive aggregation module are second-order hybrid; The V-shaped interaction aggregation module is used to detect the interaction relationship between vehicles, and the input ends all correspond to the target image of the vehicle target object, one of the input ends corresponds to the feature data input of the target vehicle, and the other input ends correspond to the special data input of the non-target vehicle. Through the interaction aggregation operation, the interaction relationship between the target vehicle and the non-target vehicle is obtained; The O-type interaction aggregation module is used to detect the interaction relationship between the vehicle and other target objects. The input end corresponds to the feature data of the target vehicle and the feature data of the non-vehicle object. The interaction relationship between the vehicle and the non-vehicle object is obtained through interaction aggregation operation.

8. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images