Vehicle driving condition analysis method based on target detection model

Abstract

The invention discloses a vehicle driving condition analysis method based on a target detection model, and relates to the technical field of target detection, and the target detection model comprises a feature fusion branch used for detecting a small target and a detection head used for detecting the small target. The method comprises the following steps: acquiring a plurality of vehicle driving sub-images; inputting the vehicle driving sub-image into a pre-trained target detection model to obtain a target in the vehicle driving sub-image and a target position of the target; and inputting the target position into a tracking algorithm to obtain a motion track of the target and a motion speed of the target output by the tracking algorithm. The method and the device are used for solving the problem that in the prior art, target missing detection occurs when vehicle driving condition analysis is carried out by using an unmanned aerial vehicle to carry out vehicle driving video acquisition at an intersection, accurate detection of each target at the intersection is realized, and driving condition analysis of all vehicles is realized.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a vehicle driving condition analysis method based on a target detection model. Background Art

[0002] The prior art installs one or more electric police cameras (or electronic eyes) at each intersection, uses the installed electric police cameras to collect vehicle driving videos at the corresponding intersection, and then obtains the vehicle driving status at the intersection by analyzing the vehicle driving videos. However, due to the fixed installation method of the electric police camera, there is a blind spot problem when monitoring the intersection.

[0003] In order to solve the problem of blind spots in the vision of electric police cameras, drones are used to collect vehicle driving videos at intersections. However, drones are mainly used to obtain high-resolution images. If a larger visual area is collected, the target will be smaller, and the different heights of the drones will also cause different target sizes. Based on the above situation, the existing vehicle driving video analysis method will miss small targets, resulting in poor accuracy in vehicle driving status analysis. Summary of the invention

[0004] In response to the above-mentioned problems and technical needs, the applicant has proposed a vehicle driving condition analysis method based on a target detection model to solve the problem of missed targets when performing vehicle driving condition analysis using drones to collect vehicle driving videos at intersections in the prior art, to achieve accurate detection of each target at the intersection, and to realize driving condition analysis of all vehicles.

[0005] The embodiment of the present application provides a vehicle driving condition analysis method based on a target detection model, wherein the target detection model includes: a head network, a neck network and a backbone network, wherein the backbone network is connected to the neck network, and the neck network is connected to the head network;

[0006] Wherein, the neck network comprises: a plurality of feature fusion branches, wherein the feature fusion branches comprise: a feature fusion branch for detecting small targets, wherein the small targets are targets whose target size is smaller than the first predicted size;

[0007] Wherein, the head network includes: a plurality of detection heads, each detection head is used to detect fusion feature maps of different sizes, and the detection heads include: a detection head for detecting small targets;

[0008] The method comprises:

[0009] Acquire a vehicle driving image of a corresponding intersection collected by a drone, and crop the vehicle driving image based on a second preset size to obtain a plurality of vehicle driving sub-images;

[0010] The vehicle driving sub-image is input into a pre-trained target detection model, and feature extraction operations of different scales are performed on the vehicle driving sub-image through the backbone network of the target detection model to obtain a plurality of preliminary extracted features, and the preliminary extracted features are input into the neck network; the neck network fuses the preliminary extracted features based on the feature fusion branch to obtain a plurality of fused feature maps, and the fused feature maps are input into the detection head of the head network corresponding to the fusion size of the fused feature map; the head network recognizes the fused feature maps of different fusion sizes based on the detection head to obtain the target in the vehicle driving sub-image and the target position of the target;

[0011] The target detection model is trained based on vehicle driving sub-image samples with annotation information added, and the annotation information includes: target samples and target position samples of the target samples;

[0012] The target position is input into a tracking algorithm to obtain a motion trajectory of the target and a motion speed of the target output by the tracking algorithm.

[0013] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, the target position is input into a tracking algorithm to obtain a motion trajectory of the target output by the tracking algorithm, including:

[0014] Inputting the target position at the current moment into the tracking algorithm, and predicting the next predicted position and next predicted state of the target at the next moment based on Kalman filtering;

[0015] After inputting the target position at the current moment into a tracking algorithm and obtaining the motion trajectory and the motion speed of the target output by the tracking algorithm, the method further includes:

[0016] Obtaining the next actual position and next actual state of the target at the next moment;

[0017] Matching the next predicted position and the next actual position, and the next predicted state and the next actual state based on the Hungarian algorithm to obtain a matching result;

[0018] The motion trajectory corresponding to the next moment is optimized based on the matching result.

[0019] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, after obtaining the target in the vehicle driving sub-image and the target position of the target, the method further includes:

[0020] Draw a rectangular detection area corresponding to the lane at the intersection in the vehicle driving image, wherein the rectangular detection area includes: a stop line;

[0021] Based on the target and the target position, the number of targets corresponding to the last target in each rectangular detection area to the stop line is calculated.

[0022] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, after obtaining the target in the vehicle driving sub-image and the target position of the target, the method further includes:

[0023] Identifying a stop line and lanes in the vehicle driving image;

[0024] The target of each lane passing the stop line within a preset time period is obtained to obtain the target flow rate of the intersection.

[0025] According to a vehicle driving condition analysis method based on a target detection model in one embodiment of the present application, the vehicle driving image is cropped based on a second preset size to obtain a plurality of vehicle driving sub-images, including:

[0026] There is overlap of a third preset size between the multiple vehicle driving sub-images of each vehicle driving image.

[0027] According to a vehicle driving condition analysis method based on a target detection model in one embodiment of the present application, the backbone network includes: an input layer, 4 feature extraction branches and 1 pooling layer;

[0028] The four feature extraction branches include: a first feature extraction branch, a second feature extraction branch, a third feature extraction branch and a fourth feature extraction branch;

[0029] The first feature extraction branch includes: 2 convolution layers and 1 feature conversion layer; the second feature extraction branch includes: 1 convolution layer and 1 feature conversion layer; the third feature extraction branch includes: 1 convolution layer and 1 feature conversion layer; the fourth feature extraction branch includes: 1 convolution layer and 1 feature conversion layer;

[0030] Among them, the output layer is connected to the first feature extraction branch; the first feature extraction branch is respectively connected to the neck network and the second feature extraction branch; the second feature extraction branch is respectively connected to the neck network and the third feature extraction branch; the third feature extraction branch is respectively connected to the fourth feature extraction branch and the neck network; the fourth feature extraction branch is connected to the pooling layer; and the pooling layer is connected to the neck network.

[0031] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, the neck network includes: 4 feature fusion branches;

[0032] The four feature fusion branches include: a first feature fusion branch, a second feature fusion branch, a third feature fusion branch and a fourth feature fusion branch;

[0033] Among them, the first feature fusion branch includes: 1 convolution layer, 1 splicing layer and 1 feature extraction layer; the second feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the third feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the fourth feature fusion branch includes: 1 upsampling layer, 1 splicing layer and 1 feature extraction layer;

[0034] The first feature fusion branch is respectively connected to the pooling layer of the backbone network, a feature extraction layer of the second feature fusion branch, and the head network. In the first feature fusion branch, the convolution layer is connected to the splicing layer, and the splicing layer is connected to the feature extraction layer.

[0035] Among them, the second feature fusion branch is connected to the backbone network, the first feature fusion branch, the third feature fusion branch and the head network respectively. In the second feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer;

[0036] Among them, the third feature fusion branch is connected to the backbone network, the fourth feature fusion branch, the second feature fusion branch and the head network respectively; in the third feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer;

[0037] Among them, the fourth feature fusion branch is connected to the backbone network, the third feature fusion branch and the head network respectively. In the fourth feature fusion branch, the feature extraction layer is connected to the splicing layer, and the splicing layer is connected to the upsampling layer.

[0038] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, the head network includes: 4 detection heads;

[0039] Among them, the four detection heads include: a first detection head, a second detection head, a third detection head and a fourth detection head;

[0040] Among them, the first detection head is connected to the first feature fusion branch, the second detection head is connected to the second feature fusion branch, the third detection head is connected to the third feature fusion branch, and the fourth detection head is connected to the fourth feature fusion branch.

[0041] According to a vehicle driving condition analysis method based on a target detection model according to an embodiment of the present application, the training process of the target detection model includes:

[0042] Acquire a plurality of vehicle driving sub-image samples;

[0043] Adding annotation information to the target sample in the vehicle driving sub-image sample to obtain a target image sample, wherein the annotation information includes: a target position sample of the target sample;

[0044] The target image sample is input into the target detection model, and feature extraction operations of different scales are performed on the target image sample through the backbone network of the target detection model to obtain a plurality of preliminary extracted feature samples, and the preliminary extracted feature samples are input into the neck network; the neck network fuses the preliminary extracted feature samples based on the feature fusion branch to obtain a plurality of fused feature map samples, and the fused feature map samples are input into the detection head of the head network corresponding to the fusion size of the fused feature map samples; the head network recognizes fused feature map samples of different fusion sizes based on the detection head, predicts the predicted target in the target image sample and the predicted target position of the predicted target; compares the consistency between the target sample and the predicted target, and the consistency between the predicted target position and the target position sample, and when it is determined that the consistency is greater than the preset consistency, determines that the training of the target detection model is completed.

[0045] According to a vehicle driving condition analysis method based on a target detection model in one embodiment of the present application, after obtaining a target detection model through training based on vehicle driving sub-image samples with added annotation information, the method further includes:

[0046] Obtaining model parameters of the target detection model;

[0047] The model parameters are stored as different inferable deployment files based on the model deployment method.

[0048] The vehicle driving condition analysis method based on the target detection model provided by the embodiment of the present application includes: a feature fusion branch for detecting small targets and a detection head for detecting small targets. It can be seen that the target detection model of the present application can solve the problem of missing small targets when analyzing vehicle driving videos, by obtaining multiple vehicle driving sub-images; inputting the vehicle driving sub-images into a pre-trained target detection model, performing feature extraction operations of different scales on the vehicle driving sub-images through the backbone network of the target detection model to obtain multiple preliminary extracted features, and inputting the preliminary extracted features into the neck network; the neck network fuses the preliminary extracted features based on the feature fusion branch to obtain multiple fused feature maps, and inputs the fused feature maps into the fusion network. The detection head of the head network corresponds to the fusion size of the fusion feature map; the head network identifies the fusion feature maps of different fusion sizes based on the detection head, and obtains the target in the vehicle driving sub-image and the target position of the target. It can be seen that the present application uses the backbone network of the target detection model to obtain the features of different scales of the vehicle driving sub-image, uses the neck network to obtain the fusion feature maps of different fusion sizes, and uses the head network to parse the fusion feature maps of different fusion sizes, so as to accurately obtain targets of various sizes; then, the target position is input into the tracking algorithm to obtain the motion trajectory and the motion speed of the target output by the tracking algorithm. The present application tracks and infers the target based on the target position output by the target detection model, and can obtain the vehicle driving condition in real time and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is one of the structural schematic diagrams of the target detection model provided in the embodiment of the present application;

[0051] Figure 2 It is a flow chart of a vehicle driving condition analysis method based on a target detection model provided in an embodiment of the present application;

[0052] Figure 3 This is the second structural diagram of the target detection model provided in the embodiment of the present application;

[0053] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to clearly explain this application, the scenario is further described:

[0056] Low-altitude economy refers to an economic form that relies on low-altitude airspace, is dominated by the general aviation industry, and covers multiple fields such as low-altitude flight, air tourism, emergency response, logistics and transportation, and agricultural and forestry operations. With the rapid development of technologies such as drones and electric vertical take-off and landing aircraft (eVTOL), the low-altitude economy is ushering in unprecedented development opportunities and has become a new engine to promote economic transformation and upgrading. Compared with traditional manual monitoring, drone operation is simpler and more convenient. By using drones for monitoring, remote control can be performed on a shorter path to supervise and warn the work site.

[0057] An important application of low-altitude economy in the field of traffic management is to detect and monitor traffic intersections through aircraft such as drones. This monitoring method can bring revolutionary changes to urban traffic management, improve efficiency and reduce costs. Using drone technology, real-time monitoring of traffic intersections from all directions and angles can be achieved, solving the problem of blind spots in the vision of electric police cameras at each intersection. The high-definition camera of the drone can capture ground vehicle information. And through the collected drone data, combined with the data collected by the ground electric police camera for analysis and processing, it is possible to more accurately evaluate the actual operating conditions of each intersection, timely discover potential safety hazards or congestion points, and take corresponding measures.

[0058] This new monitoring system can collect, process, transmit and store data in real time, thereby achieving better economic and social benefits through more accurate supervision. In addition, with the continuous improvement of advanced hardware facilities and the continuous development of deep learning, the collection, storage and transmission of aerial images have become more convenient and faster, making the demand for accurate positioning and recognition of images increasingly greater, and the development of this field is advancing rapidly.

[0059] Based on the above scenario, the embodiment of the present application provides a vehicle driving condition analysis method based on a target detection model. The method can be applied to a smart terminal or a server. Some other descriptions in the embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application. They will not be described one by one later.

[0060] like Figure 1 As shown, the target detection model includes: a head network 101, a neck network 102 and a backbone network 103, the backbone network 103 is connected to the neck network 102, and the neck network 102 is connected to the head network 101.

[0061] The neck network 102 includes: a plurality of feature fusion branches 1022, and the feature fusion branch 1022 includes: a feature fusion branch 1022 for detecting small targets, where the small targets are targets whose size is smaller than the first predicted size.

[0062] Among them, the head network 101 includes: multiple detection heads 1011, each detection head 1011 is used to detect fused feature maps of different sizes, and the detection head 1011 includes: a detection head for detecting small targets.

[0063] Among them, Figure 1 In the figure, three feature fusion branches 1022 and four detection heads 1011 are used for illustration. This is only for illustration and is not intended to limit the scope of protection of the present application.

[0064] The specific implementation of this method is as follows Figure 2 As shown:

[0065] Step 201: obtain a vehicle driving image of a corresponding intersection collected by a drone, and crop the vehicle driving image based on a second preset size to obtain a plurality of vehicle driving sub-images.

[0066] Step 202, input the vehicle driving sub-image into a pre-trained target detection model, perform feature extraction operations of different scales on the vehicle driving sub-image through the backbone network of the target detection model, obtain multiple preliminary extracted features, and input the preliminary extracted features into the neck network; the neck network fuses the preliminary extracted features based on the feature fusion branch to obtain multiple fused feature maps, and input the fused feature maps into the detection head of the head network corresponding to the fusion size of the fused feature map; the head network identifies the fused feature maps of different fusion sizes based on the detection head to obtain the target in the vehicle driving sub-image and the target position of the target.

[0067] The target detection model is trained based on vehicle driving sub-image samples with annotation information added, and the annotation information includes: target samples and target position samples of the target samples.

[0068] Step 203, input the target position into the tracking algorithm, and obtain the target's motion trajectory and the target's motion speed output by the tracking algorithm.

[0069] The vehicle driving condition analysis method based on the target detection model provided by the embodiment of the present application includes: a feature fusion branch for detecting small targets and a detection head for detecting small targets. It can be seen that the target detection model of the present application can solve the problem of missing small targets when analyzing vehicle driving videos, by obtaining multiple vehicle driving sub-images; inputting the vehicle driving sub-images into a pre-trained target detection model, performing feature extraction operations of different scales on the vehicle driving sub-images through the backbone network of the target detection model to obtain multiple preliminary extracted features, and inputting the preliminary extracted features into the neck network; the neck network fuses the preliminary extracted features based on the feature fusion branch to obtain multiple fused feature maps, and inputs the fused feature maps into the fusion network. The detection head of the head network corresponds to the fusion size of the fusion feature map; the head network identifies the fusion feature maps of different fusion sizes based on the detection head, and obtains the target in the vehicle driving sub-image and the target position of the target. It can be seen that the present application uses the backbone network of the target detection model to obtain the features of different scales of the vehicle driving sub-image, uses the neck network to obtain the fusion feature maps of different fusion sizes, and uses the head network to parse the fusion feature maps of different fusion sizes, so as to accurately obtain targets of various sizes; then, the target position is input into the tracking algorithm to obtain the motion trajectory and the motion speed of the target output by the tracking algorithm. The present application tracks and infers the target based on the target position output by the target detection model, and can obtain the vehicle driving condition in real time and accurately.

[0070] In a specific embodiment, the backbone network 103 includes: an input layer, 4 feature extraction branches and 1 pooling layer.

[0071] Among them, the four feature extraction branches include: a first feature extraction branch, a second feature extraction branch, a third feature extraction branch and a fourth feature extraction branch.

[0072] Among them, the first feature extraction branch includes: 2 convolutional layers and 1 feature conversion layer; the second feature extraction branch includes: 1 convolutional layer and 1 feature conversion layer; the third feature extraction branch includes: 1 convolutional layer and 1 feature conversion layer; the fourth feature extraction branch includes: 1 convolutional layer and 1 feature conversion layer.

[0073] Among them, the output layer is connected to the first feature extraction branch; the first feature extraction branch is respectively connected to the neck network 102 and the second feature extraction branch; the second feature extraction branch is respectively connected to the neck network 102 and the third feature extraction branch; the third feature extraction branch is respectively connected to the fourth feature extraction branch and the neck network 102; the fourth feature extraction branch is connected to the pooling layer; and the pooling layer is connected to the neck network 102.

[0074] In a specific embodiment, the neck network 102 includes: 4 feature fusion branches 1022 .

[0075] Among them, the four feature fusion branches 1022 include: a first feature fusion branch, a second feature fusion branch, a third feature fusion branch and a fourth feature fusion branch.

[0076] Among them, the first feature fusion branch includes: 1 convolution layer, 1 splicing layer and 1 feature extraction layer; the second feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the third feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the fourth feature fusion branch includes: 1 upsampling layer, 1 splicing layer and 1 feature extraction layer.

[0077] Among them, the first feature fusion branch is respectively connected to the pooling layer of the backbone network 103, a feature extraction layer of the second feature fusion branch and the head network 101.

[0078] In the first feature fusion branch, the convolution layer is connected to the concatenation layer, and the concatenation layer is connected to the feature extraction layer.

[0079] Among them, the second feature fusion branch is connected to the backbone network 103, the first feature fusion branch, the third feature fusion branch and the head network 101 respectively.

[0080] In the second feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer.

[0081] Among them, the third feature fusion branch is connected to the backbone network 103, the fourth feature fusion branch, the second feature fusion branch and the head network 101 respectively.

[0082] In the third feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer.

[0083] Among them, the fourth feature fusion branch is connected to the backbone network 103, the third feature fusion branch and the head network 101 respectively.

[0084] In the fourth feature fusion branch, the feature extraction layer is connected to the splicing layer, and the splicing layer is connected to the upsampling layer.

[0085] In a specific embodiment, the head network 101 includes: 4 detection heads.

[0086] Among them, the four detection heads include: a first detection head, a second detection head, a third detection head and a fourth detection head.

[0087] Among them, the first detection head is connected to the first feature fusion branch, the second detection head is connected to the second feature fusion branch, the third detection head is connected to the third feature fusion branch, and the fourth detection head is connected to the fourth feature fusion branch.

[0088] For example, the detection fusion size of the first detection head is 20px*20px, the detection fusion size of the second detection head is 40px*40px, the detection fusion size of the third detection head is 80px*80px, and the detection fusion size of the fourth detection head is 160px*160px.

[0089] Specifically, through Figure 3 The structure of the target detection model is specifically illustrated.

[0090] Specifically, the target detection model of the present application includes a YOLOv8 model, specifically including a backbone network and a neck network for feature extraction, and a head network for post-processing.

[0091] The backbone network includes 5 convolutional layers, 4 feature extraction layers (c2f) and a pooling layer (SPPF) to extract multi-scale features of vehicle driving sub-images. The neck network takes each scale of the backbone network as input branch and first performs upsampling and splicing fusion, then convolution fusion of different scale features, and then inputs the multi-scale fusion feature map into the detection head of the corresponding size in the head network to finally obtain the target detection result.

[0092] Among them, the 80px×80px fusion feature map is used to detect small targets larger than 8×8, the 40px×40px fusion feature map is used to detect medium targets larger than 16×16, the 20px×20px fusion feature map is used to detect large targets larger than 32×32, and the 160px×160px feature fusion map is used to detect small targets larger than 4×4, so as to improve the model's detection ability for small targets. At the same time, considering the target size under the perspective of the drone scene, the large target detection head and the corresponding feature fusion branch are removed, which can effectively reduce the number of network parameters and improve the speed of the model to detect small targets.

[0093] In a specific embodiment, the specific implementation of the training process of the target detection model includes:

[0094] Acquire multiple vehicle driving sub-image samples; add annotation information to the target samples in the vehicle driving sub-image samples to obtain target image samples; input the target image samples into the target detection model, perform feature extraction operations of different scales on the target image samples through the backbone network of the target detection model, obtain multiple preliminary extracted feature samples, and input the preliminary extracted feature samples into the neck network; the neck network fuses the preliminary extracted feature samples based on the feature fusion branch to obtain multiple fused feature map samples, and input the fused feature map samples into the detection head of the head network corresponding to the fusion size of the fused feature map samples; the head network identifies fused feature map samples of different fusion sizes based on the detection head, predicts the predicted target in the target image sample and the predicted target position of the predicted target; compare the consistency between the target sample and the predicted target, as well as the consistency between the predicted target position and the target position sample, and when it is determined that the consistency is greater than the preset consistency, determine that the target detection model training is completed.

[0095] The annotation information includes: target position samples of target samples.

[0096] Specifically, historical video data shot by a drone is obtained, and key frame images are extracted from the historical video data to obtain vehicle driving image samples. A labeling tool (e.g., X-Anylabeling) is used to label the rotation target of the intersection scene and save it as a json file. The vehicle driving image sample is cropped based on a second preset size to obtain multiple vehicle driving sub-image samples. Then, the vehicle driving sub-image samples are divided into a training set and a validation set.

[0097] The annotation information includes: the location information of the target in the sample image (target location sample) and attribute information, wherein the target location sample includes: four point coordinates and an angle value, the point coordinates are given in the order of annotation, and the direction of the annotation box can be obtained according to the angle value. The attribute information is the category information of the target.

[0098] Then, according to the second preset size C, the vehicle driving image sample W*H is cropped into a plurality of vehicle driving sub-image samples of C*C, wherein each vehicle driving sub-image sample has an overlap of 200 pixels to ensure that the target does not obtain incomplete annotation information due to cropping. For vehicle driving sub-image samples larger than or less than C*C, C*C is obtained by filling the surroundings.

[0099] Specifically, the number of vehicle driving sub-images is obtained by taking the remainder of the division operation. The row direction is calculated based on formula (1), and the column direction is calculated based on formula (2):

[0100] numberH=(H+C–200–1) / (C–200)………………(1)

[0101] numberW=(W+C–200–1) / (C–200)………………(2)

[0102] Among them, numberH represents the number of sub-images that can be obtained from the vehicle driving image sample in the row direction, H represents the height of the vehicle driving image sample, C represents the second predicted size, numberW represents the number of sub-images that can be obtained from the vehicle driving image sample in the column direction, and W represents the width of the vehicle driving image sample.

[0103] The target detection model is trained using the training set, and the final target detection model is obtained through algorithm iteration and loss convergence. The final target detection model is converted into a deployable model structure, and the output results are inferred to obtain inference results such as the queue length at the intersection, vehicle type, and traffic flow detection results.

[0104] In addition, in addition to identifying vehicles, the target detection model can also identify people. When the target category is people, it can also infer the results of pedestrian flow detection.

[0105] In a specific embodiment, after obtaining the target detection model, the model parameters of the target detection model are obtained; and the model parameters are stored as different inferable deployment files based on the model deployment method.

[0106] Specifically, due to different model deployment methods, different inference libraries call different weight files (inferable deployment files). For example, in the local training phase, the torch library is used, and a .pt file is required. In the model deployment phase, the .pt file corresponding to the training phase needs to be converted to the weight file under the corresponding processor according to the deployment requirements of the processor.

[0107] In a specific embodiment, real-time video data of an intersection scene detected by a drone aerial camera is obtained.

[0108] Among them, it is necessary to ensure the correct installation of the drone aerial camera lens and adjust the appropriate parameters to obtain video data that meets the required range.

[0109] This application ensures the timeliness and comprehensiveness of the perception of traffic events through real-time collection by drones.

[0110] In a specific embodiment, the video stream is processed into image frames, low-quality and abnormal images are deleted, and images with similarity greater than a preset similarity are deleted to ensure the quality of vehicle driving images is reliable.

[0111] In a specific embodiment, the specific implementation of cropping the vehicle driving image based on the second preset size to obtain multiple vehicle driving sub-images includes:

[0112] There is overlap of a third preset size between the multiple vehicle driving sub-images of each vehicle driving image.

[0113] For example, if the third prediction size (C) is 200 pixels, the overlap size is 200 pixels.

[0114] In a specific embodiment, cropping a vehicle driving image to obtain multiple vehicle driving sub-images will result in more images to be processed, and the processing time will be W*H times longer than the original. In order to ensure the data processing speed, this application uses multiple threads to analyze the vehicle driving conditions. For example, one thread is used for video stream loading, four threads are used for reasoning, and one thread is used for target tracking and result analysis.

[0115] In a specific embodiment, the specific implementation of inputting the target position into the tracking algorithm and obtaining the motion trajectory of the target output by the tracking algorithm includes:

[0116] The target position at the current moment is input into the tracking algorithm, and the next predicted position and next predicted state of the target at the next moment are predicted based on the Kalman filter; the next actual position and next actual state of the target at the next moment are obtained; the next predicted position and the next actual position, as well as the next predicted state and the next actual state are matched based on the Hungarian algorithm to obtain the matching result; and the motion trajectory corresponding to the next moment is optimized based on the matching result.

[0117] Specifically, in the first frame, the current detection frame is used as the new tracked target, and the position and state of the target in the next frame are predicted through the Kalman filter KF algorithm to obtain the prediction frame of the next frame corresponding to each trajectory. In the second frame, the prediction frame and the current frame detection frame are matched with each other through the Hungarian algorithm for iou similarity calculation. If the match is successful, the trajectory of the target is updated, that is, the prediction frame is updated to the value of the target frame position. After the update, the KF prediction of the next frame will be performed; if the match is unsuccessful, two situations are considered. One is an unmatched prediction frame, which means that the prediction frame is blocked or leaves the screen, and the target is discarded; the other is an unmatched detection frame, which means that a new target appears and is added to the tracking as a new target, and the first and second frame operations are repeated.

[0118] Specifically, the image registration operation includes: converting the image coordinate system to the geographic coordinate system, using the pre-collected location points to register with the scene map taken by the drone, and using affine transformation to calculate the transformation matrix. Suppose the coordinates of a point on the image are (x, y), and the actual coordinates obtained after affine transformation are (x ′ ,y ′ ), then the affine transformation can be expressed in the following matrix form: Where (a, b) and (c, d) are the coefficients of the linear transformation that control the scaling and rotation of the image, which are constants, and (e, f) are the translation vectors that control the translation of the image, which are constants.

[0119] Specifically, the target position can be input into the tracking algorithm to obtain the vehicle speed, etc., as follows:

[0120] Vehicle speed detection: A tracking ID is assigned to each target according to the tracking algorithm. The tracking target obtains the position difference and time difference in the set interval frame image to calculate the speed of the current vehicle.

[0121] Vehicle speeding warning: Determine whether the vehicle is speeding based on vehicle speed detection and the current speed limit in the intersection area.

[0122] Vehicle stay warning: Based on the vehicle speed detection and the pre-set stay area in the current intersection area, determine whether the vehicle stays in the area for a long time.

[0123] Vehicle trajectory: According to the tracking ID, the center point coordinates of the tracking target in each frame are retained as the trajectory point. Statistical analysis is performed on the trajectory map of the intersection scene during peak hours, and variable lanes are set to improve the road release rate.

[0124] Specifically, target category detection includes: pedestrians, non-motor vehicles, small cars (sedans), medium-sized cars (vans), large cars (buses, trucks), etc.

[0125] In a specific embodiment, after obtaining the target and the target position of the target in the vehicle driving sub-image, a rectangular detection area corresponding to the lane at the intersection in the vehicle driving image is drawn; based on the target and the target position, the number of targets corresponding to the last target in each rectangular detection area to the stop line is calculated.

[0126] The rectangular detection area includes: a stop line.

[0127] Specifically, rectangular detection areas for each lane are drawn based on the intersection scenes photographed by the drone, the number of vehicles in each exit rectangular detection area is counted, and the distance from the rear of the last vehicle in the detection area to the stop line is taken as the queue length.

[0128] In a specific embodiment, after obtaining the target in the vehicle driving sub-image and the target position of the target, the stop line and each lane in the vehicle driving image are identified; the target passing the stop line in each lane within a preset time period is obtained to obtain the target flow rate at the intersection.

[0129] Specifically, the stop line is used as the dividing line in each lane. When a vehicle passes the dividing line, a counter is incremented by 1, and the number of vehicles passing through the dividing line per unit time is used as the traffic flow of the lane.

[0130] Specifically, a specific time is divided into multiple time periods, and the sum of lane traffic volume in each time period is counted in real time as the total traffic volume.

[0131] Specifically, the pedestrian flow on the sidewalk is detected in real time, the pedestrian flow data is integrated with the lane vehicle flow data, the correlation between pedestrian flow and vehicle flow in different time periods is analyzed, and the traffic lights are adaptively adjusted in a timely manner.

[0132] The target detection model of the present application can solve the problem of missing small targets when analyzing vehicle driving videos. The backbone network of the target detection model is used to obtain features of vehicle driving sub-images of different scales, the neck network is used to obtain fused feature maps of different fusion sizes, and the head network is used to analyze fused feature maps of different fusion sizes, so as to accurately obtain targets of various sizes. Furthermore, the target position output by the target detection model is used to track and infer the target, so as to obtain the vehicle driving status in real time and accurately.

[0133] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The processor 401 may call the logic instructions in the memory 403 to execute the vehicle driving condition analysis method based on the target detection model.

[0134] In addition, the logic instructions in the above-mentioned memory 403 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the vehicle driving condition analysis method based on the target detection model provided by the above-mentioned methods.

[0136] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vehicle driving condition analysis method based on the target detection model provided in the above-mentioned embodiments.

[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0139] Finally, it should be noted that the above is only the preferred implementation of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.

Claims

1. A vehicle driving condition analysis method based on a target detection model, characterized in that: The target detection model includes: a head network, a neck network and a backbone network, the backbone network is connected to the neck network, and the neck network is connected to the head network; Wherein, the neck network comprises: a plurality of feature fusion branches, wherein the feature fusion branches comprise: a feature fusion branch for detecting small targets, wherein the small targets are targets whose target size is smaller than the first predicted size; Wherein, the head network includes: a plurality of detection heads, each detection head is used to detect fusion feature maps of different sizes, and the detection heads include: a detection head for detecting small targets; The method comprises: Acquire a vehicle driving image of a corresponding intersection collected by a drone, and crop the vehicle driving image based on a second preset size to obtain a plurality of vehicle driving sub-images; The vehicle driving sub-image is input into a pre-trained target detection model, and feature extraction operations of different scales are performed on the vehicle driving sub-image through the backbone network of the target detection model to obtain a plurality of preliminary extracted features, and the preliminary extracted features are input into the neck network; the neck network fuses the preliminary extracted features based on the feature fusion branch to obtain a plurality of fused feature maps, and the fused feature maps are input into the detection head of the head network corresponding to the fusion size of the fused feature map; the head network recognizes the fused feature maps of different fusion sizes based on the detection head to obtain the target in the vehicle driving sub-image and the target position of the target; The target detection model is trained based on vehicle driving sub-image samples with annotation information added, and the annotation information includes: target samples and target position samples of the target samples; The target position is input into a tracking algorithm to obtain a motion trajectory of the target and a motion speed of the target output by the tracking algorithm.

2. The vehicle driving condition analysis method based on the target detection model according to claim 1 is characterized in that: Inputting the target position into a tracking algorithm to obtain a motion trajectory of the target output by the tracking algorithm includes: Inputting the target position at the current moment into the tracking algorithm, and predicting the next predicted position and next predicted state of the target at the next moment based on Kalman filtering; After inputting the target position at the current moment into a tracking algorithm and obtaining the motion trajectory and the motion speed of the target output by the tracking algorithm, the method further includes: Obtaining the next actual position and next actual state of the target at the next moment; Matching the next predicted position and the next actual position, and the next predicted state and the next actual state based on the Hungarian algorithm to obtain a matching result; The motion trajectory corresponding to the next moment is optimized based on the matching result.

3. The vehicle driving condition analysis method based on the target detection model according to claim 1 is characterized in that: After obtaining the target in the vehicle driving sub-image and the target position of the target, the method further includes: Draw a rectangular detection area corresponding to the lane at the intersection in the vehicle driving image, wherein the rectangular detection area includes: a stop line; Based on the target and the target position, the number of targets corresponding to the last target in each rectangular detection area to the stop line is calculated.

4. The vehicle driving condition analysis method based on the target detection model according to claim 1 is characterized in that: After obtaining the target in the vehicle driving sub-image and the target position of the target, the method further includes: Identifying a stop line and lanes in the vehicle driving image; The target of each lane passing the stop line within a preset time period is obtained to obtain the target flow rate of the intersection.

5. The vehicle driving condition analysis method based on the target detection model according to any one of claims 1 to 4, characterized in that: The vehicle driving image is cropped based on a second preset size to obtain a plurality of vehicle driving sub-images, including: There is overlap of a third preset size between the multiple vehicle driving sub-images of each vehicle driving image.

6. The vehicle driving condition analysis method based on the target detection model according to any one of claims 1 to 4, characterized in that: The backbone network includes: input layer, 4 feature extraction branches and 1 pooling layer; The four feature extraction branches include: a first feature extraction branch, a second feature extraction branch, a third feature extraction branch and a fourth feature extraction branch; The first feature extraction branch includes: 2 convolution layers and 1 feature conversion layer; the second feature extraction branch includes: 1 convolution layer and 1 feature conversion layer; the third feature extraction branch includes: 1 convolution layer and 1 feature conversion layer; the fourth feature extraction branch includes: 1 convolution layer and 1 feature conversion layer; Among them, the output layer is connected to the first feature extraction branch; the first feature extraction branch is respectively connected to the neck network and the second feature extraction branch; the second feature extraction branch is respectively connected to the neck network and the third feature extraction branch; the third feature extraction branch is respectively connected to the fourth feature extraction branch and the neck network; the fourth feature extraction branch is connected to the pooling layer; and the pooling layer is connected to the neck network.

7. The vehicle driving condition analysis method based on the target detection model according to claim 6 is characterized in that: The neck network includes: 4 feature fusion branches; The four feature fusion branches include: a first feature fusion branch, a second feature fusion branch, a third feature fusion branch and a fourth feature fusion branch; Among them, the first feature fusion branch includes: 1 convolution layer, 1 splicing layer and 1 feature extraction layer; the second feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the third feature fusion branch includes: 1 upsampling layer, 1 convolution layer, 2 splicing layers and 2 feature extraction layers; the fourth feature fusion branch includes: 1 upsampling layer, 1 splicing layer and 1 feature extraction layer; The first feature fusion branch is respectively connected to the pooling layer of the backbone network, a feature extraction layer of the second feature fusion branch, and the head network. In the first feature fusion branch, the convolution layer is connected to the splicing layer, and the splicing layer is connected to the feature extraction layer. Among them, the second feature fusion branch is connected to the backbone network, the first feature fusion branch, the third feature fusion branch and the head network respectively. In the second feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer; Among them, the third feature fusion branch is connected to the backbone network, the fourth feature fusion branch, the second feature fusion branch and the head network respectively; in the third feature fusion branch, one feature extraction layer is connected to two splicing layers respectively, one splicing layer is connected to the upsampling layer, and another splicing layer is connected to another feature extraction layer; Among them, the fourth feature fusion branch is connected to the backbone network, the third feature fusion branch and the head network respectively. In the fourth feature fusion branch, the feature extraction layer is connected to the splicing layer, and the splicing layer is connected to the upsampling layer.

8. The vehicle driving condition analysis method based on the target detection model according to claim 7 is characterized in that: The head network includes: 4 detection heads; Among them, the four detection heads include: a first detection head, a second detection head, a third detection head and a fourth detection head; Among them, the first detection head is connected to the first feature fusion branch, the second detection head is connected to the second feature fusion branch, the third detection head is connected to the third feature fusion branch, and the fourth detection head is connected to the fourth feature fusion branch.

9. The vehicle driving condition analysis method based on the target detection model according to any one of claims 1 to 4, characterized in that: The training process of the target detection model includes: Acquire a plurality of vehicle driving sub-image samples; Adding annotation information to the target sample in the vehicle driving sub-image sample to obtain a target image sample, wherein the annotation information includes: a target position sample of the target sample; The target image sample is input into the target detection model, and feature extraction operations of different scales are performed on the target image sample through the backbone network of the target detection model to obtain a plurality of preliminary extracted feature samples, and the preliminary extracted feature samples are input into the neck network; the neck network fuses the preliminary extracted feature samples based on the feature fusion branch to obtain a plurality of fused feature map samples, and the fused feature map samples are input into the detection head of the head network corresponding to the fusion size of the fused feature map samples; the head network recognizes fused feature map samples of different fusion sizes based on the detection head, predicts the predicted target in the target image sample and the predicted target position of the predicted target; compares the consistency between the target sample and the predicted target, and the consistency between the predicted target position and the target position sample, and when it is determined that the consistency is greater than the preset consistency, determines that the training of the target detection model is completed.

10. The vehicle driving condition analysis method based on the target detection model according to any one of claims 1 to 4, characterized in that: After the target detection model is obtained by training the vehicle driving sub-image samples with annotated information, it also includes: Obtaining model parameters of the target detection model; The model parameters are stored as different inferable deployment files based on the model deployment method.

Images