An occluded vehicle tail frame regression method and device based on a neural network and a storage medium

Abstract

The application discloses a kind of based on neural network's being covered vehicle tail frame regression method, device and storage medium, its method includes: acquisition contains the target image of vehicle tail frame;Based on the position information of target vehicle in target image obtained in the preset YOLOV5 target detection framework;Based on position information from target image target vehicle image is cut out, and target vehicle image is pretreated;The target vehicle image after pretreatment is input into deep neural network to obtain target vehicle tail covered information, vehicle tail feature point position information and vehicle tail midline position information;Based on vehicle tail covered information and vehicle tail feature point position information, unoccluded vehicle tail feature point is screened out;Based on unoccluded vehicle tail feature point and vehicle tail midline position information, generate vehicle tail frame;The application can improve the accuracy of vehicle motion state judgment to improve the decision performance of entire vehicle auxiliary driving system.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of neural network-based occluded vehicle tail frame regression method, device and storage medium, belong to auxiliary driving technical field. BACKGROUND

[0002] In vehicle auxiliary driving system, we need to obtain the imaging of the target vehicle in front of the road according to the vehicle camera, and then obtain the motion state of the vehicle by analyzing various attributes of the target vehicle in the image. According to the position of the target vehicle tail frame, the width of the vehicle tail in the real scene, and the imaging parameters of the camera, the position of the target vehicle in the real scene can be accurately calculated, thereby providing the basis for subsequent motion state judgment. Then one of the core is how to obtain accurate vehicle tail frame. According to the conventional process, first, the vehicle on the road is detected by the detection network, and then the detected target vehicle is finely positioned. According to the position of the feature points on the vehicle body, the width of the target vehicle and the position of the grounding point are obtained. However, when the target vehicle is occluded, the existing detection network has the problem of poor feature point positioning accuracy. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art, and provides a neural network-based occluded vehicle tail frame regression method, device and storage medium, which can improve the accuracy of the entire feature point positioning link, thereby generating a vehicle tail frame and improving the accuracy of vehicle motion state judgment to improve the decision-making performance of the entire vehicle auxiliary driving system.

[0004] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0005] In a first aspect, the present application provides a neural network-based occluded vehicle tail frame regression method, comprising:

[0006] Collecting a video segment of the vehicle in front of the vehicle, and extracting a target image containing a vehicle tail frame from the video segment;

[0007] Obtaining the position information of the target vehicle in the target image based on a preset YOLOV5 target detection framework;

[0008] Cutting out a target vehicle image from the target image based on the position information, and pre-processing the target vehicle image;

[0009] Inputting the pre-processed target vehicle image into a trained deep neural network to obtain occluded information of the tail of the target vehicle, feature point position information of the tail of the vehicle, and centerline position information of the tail of the vehicle;

[0010] Screening the vehicle tail feature points not being blocked based on the vehicle tail being blocked information and the vehicle tail feature point position information;

[0011] Generating the vehicle tail frame based on the vehicle tail feature points not being blocked and the vehicle tail centerline position information.

[0012] Optionally, the training process of the deep neural network comprises:

[0013] Obtaining historical data of the target vehicle image, and respectively labeling the vehicle tail feature points in the target vehicle image, and pre-processing the target vehicle image to generate a training set;

[0014] Inputting the training sample in the training set into the deep neural network to obtain vehicle tail feature point prediction information;

[0015] Calculating the loss of the deep neural network based on the vehicle tail feature point prediction information and the labeled information;

[0016] According to the loss of the deep neural network, iteratively propagating the network parameters of the deep neural network in reverse, until reaching a preset iteration number or loss convergence, completing the training of the deep neural network.

[0017] Optionally, the labeling of the vehicle tail feature points in the target vehicle image comprises:

[0018] The left upper corner and the right upper corner of the vehicle tail frame are labeled as feature points 6 and feature points 1 respectively;

[0019] The left lower corner and the right lower corner of the vehicle tail frame are labeled as feature points 5 and feature points 2 respectively;

[0020] The left end point of the intersection line between the vehicle left rear wheel and the ground and the right end point of the intersection line between the vehicle right rear wheel and the ground are labeled as feature points 4 and feature points 3 respectively.

[0021] Optionally, if there is a blocked feature point, the blocked feature point is labeled according to experience, and the blocked and unblocked labels of all feature points are added, and the vehicle tail centerline is labeled.

[0022] Optionally, the vehicle tail feature point prediction information comprises vehicle tail being blocked prediction information, vehicle tail feature point position prediction information, and vehicle tail centerline position prediction information;

[0023] The loss of the deep neural network based on the vehicle tail feature point prediction information and the labeled information comprises:

[0024] The loss of the target vehicle tail being blocked prediction information is calculated by a cross-entropy loss function:

[0025]

[0026] wherein sigmoid(·) is a sigmoid function, y n is the target vehicle tail occlusion information, target vehicle tail occlusion prediction information;

[0027] The loss L1_loss1 of the vehicle tail centerline position prediction information is calculated by a smooth L1 loss function:

[0028]

[0029] wherein x is the difference between the vehicle tail centerline label information and the vehicle tail centerline position prediction information;

[0030] The loss L1_loss2 of the vehicle tail feature point position prediction information is calculated by a smooth L1 loss function:

[0031] L1_loss2 = L1_loss1 * K

[0032] wherein K = 0 when the feature point is occluded, and K = 1 when the feature point is not occluded.

[0033] In a second aspect, the present application provides a neural network-based occluded vehicle tail bounding box regression device, which comprises:

[0034] An image acquisition module is configured to collect a video segment of a vehicle front road condition, and extract a target image containing a vehicle tail bounding box from the video segment;

[0035] A position acquisition module is configured to acquire position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework;

[0036] A cropping processing module is configured to crop a target vehicle image from the target image based on the position information, and pre-process the target vehicle image;

[0037] An information acquisition module is configured to input the pre-processed target vehicle image into a trained deep neural network to obtain target vehicle tail occlusion information, vehicle tail feature point position information, and vehicle tail centerline position information;

[0038] An information screening module is configured to screen out vehicle tail feature points that are not occluded based on the vehicle tail occlusion information and the vehicle tail feature point position information;

[0039] A tail bounding box generation module is configured to generate a vehicle tail bounding box based on the vehicle tail feature points that are not occluded and the vehicle tail centerline position information.

[0040] In a third aspect, the present application provides a neural network-based occluded vehicle tail bounding box regression device, comprising a processor and a storage medium.

[0041] The storage medium is configured to store instructions.

[0042] The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the above.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method according to any one of the above.

[0044] Compared with the prior art, the present application has the following beneficial effects:

[0045] The neural network-based occluded vehicle tail bounding box regression method, device and storage medium provided by the present application can obtain the following three information simultaneously by using a deep neural network: occluded information of the tail of a target vehicle, position information of feature points of the tail of the vehicle, and position information of the center line of the tail of the vehicle. The state information of the above three feature points is synchronously learned by using the feature sharing of the neural network, and such network design helps to save the computing resources of the system in this module. The reliability of the feature points is determined according to the occluded information of the tail feature points, so as to screen the feature points. A relatively accurate vehicle tail bounding box is generated according to the position of the screened feature points and the center line of the tail. When the tail is occluded, a mark indicating whether the relevant feature points are occluded is given, so that the deep neural network does not learn the position information of the occluded feature points. It can be ensured that all the feature points learned by the neural network are accurately labeled, and when the feature points of the tail of a non-occluded vehicle are jointly learned, no negative impact on the learning of the non-occluded vehicle is caused. Through the above scheme, the feature point positioning of the non-occluded vehicle can be ensured, and the relatively accurate feature point positioning of the occluded vehicle can also be obtained, and meanwhile, the related tasks are integrated into the same deep learning neural network, so that the running efficiency of the entire module is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of a neural network-based occluded vehicle tail bounding box regression method provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of labeling the feature points of the tail of a vehicle in a target vehicle image according to an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of labeling the feature points of the tail of a vehicle in a target vehicle image with occlusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0050] Example one:

[0051] As shown in the figure, the embodiment of the application provides a neural network-based occluded vehicle tail frame regression method, which includes the following steps: Figure 1

[0052] (1) Collect a video segment of the vehicle condition in front of the vehicle, and extract a target image containing a vehicle tail frame from the video segment;

[0053] In a specific implementation, a vehicle-mounted monocular camera can be installed on the vehicle window glass to collect the vehicle condition information in front of the vehicle during driving. The input size of the collected picture is 1280*720P. The image to be processed is obtained by decompressing the video image frame by frame.

[0054] (2) Obtain the position information of the target vehicle in the target image based on a preset YOLOV5 target detection framework.

[0055] (3) Crop the target vehicle image from the target image based on the position information, and pre-process the target vehicle image; the pre-processing mainly scales the target vehicle image to 128*128.

[0056] (4) Input the pre-processed target vehicle image into a trained deep neural network to obtain occluded information of the vehicle tail, position information of the vehicle tail feature points, and position information of the vehicle tail center line;

[0057] (5) Screen out the vehicle tail feature points that are not occluded based on the occluded information of the vehicle tail and the position information of the vehicle tail feature points.

[0058] (6) Generate a vehicle tail frame based on the vehicle tail feature points that are not occluded and the position information of the vehicle tail center line.

[0059] Specifically, the training process of the deep neural network includes:

[0060] Obtain historical data of the target vehicle image, and respectively label the vehicle tail feature points in the target vehicle image, and pre-process the target vehicle image to generate a training set;

[0061] Input the training samples in the training set into the deep neural network to obtain vehicle tail feature point prediction information;

[0062] Calculate the loss of the deep neural network based on the vehicle tail feature point prediction information and the labeled information.

[0063] ​The network parameters of the deep neural network are iteratively back-propagated according to a loss of the deep neural network until a preset iteration number or loss convergence is reached, and training of the deep neural network is completed.

[0064] Specifically, as shown in Figure 2 annotating vehicle tail feature points in the target vehicle image includes:

[0065] The top left corner and the top right corner of the vehicle tail frame are respectively labeled as feature point 6 and feature point 1;

[0066] The top left corner and the top right corner of the vehicle tail frame are respectively labeled as feature point 5 and feature point 2;

[0067] The left end point of the intersection line between the vehicle left rear wheel and the ground and the right end point of the intersection line between the vehicle right rear wheel and the ground are respectively labeled as feature point 4 and feature point 3.

[0068] In addition, as shown in Figure 3 If there is an occlusion in the feature points, the occluded feature points are annotated according to experience, and annotations of occluded and non-occluded are added to all feature points, and the vehicle tail centerline is annotated.

[0069] Specifically, the vehicle tail feature point prediction information includes vehicle tail occlusion prediction information, vehicle tail feature point position prediction information, and vehicle tail centerline position prediction information;

[0070] Calculating the loss of the deep neural network based on the vehicle tail feature point prediction information and the annotation information includes:

[0071] The loss loss of the target vehicle tail occlusion prediction information is calculated by a cross-entropy loss function:

[0072]

[0073] Where sigmoid(·) is a sigmoid function, y n is the target vehicle tail occlusion information, the target vehicle tail occlusion prediction information;

[0074] The loss L1_loss1 of the vehicle tail centerline position prediction information is calculated by a smooth L1 loss function:

[0075]

[0076] Where x is the difference between the vehicle tail centerline annotation information and the vehicle tail centerline position prediction information;

[0077] The loss L1_loss2 of the vehicle tail feature point position prediction information is calculated by a smooth L1 loss function:

[0078] L1_loss2 = L1_loss1 * K

[0079] Wherein, when the feature point is occluded, K = 0, when the feature point is not occluded, K = 1.

[0080] Embodiment two:

[0081] The embodiment of the application provides a neural network-based occluded vehicle tail frame regression device, which comprises:

[0082] An image acquisition module is configured to collect a video segment of a vehicle condition in front of a vehicle and extract a target image containing a vehicle tail frame from the video segment;

[0083] A position acquisition module is configured to acquire position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework;

[0084] A cropping processing module is configured to crop a target vehicle image from the target image based on the position information and pre-process the target vehicle image;

[0085] An information acquisition module is configured to input the pre-processed target vehicle image into a trained deep neural network to obtain occluded information of a tail of the target vehicle, position information of feature points of the tail of the target vehicle, and position information of a center line of the tail of the target vehicle;

[0086] An information screening module is configured to screen out vehicle tail feature points that are not occluded based on the occluded information of the tail of the target vehicle and the position information of the feature points of the tail of the target vehicle;

[0087] A tail frame generation module is configured to generate a vehicle tail frame based on the vehicle tail feature points that are not occluded and the position information of the center line of the tail of the target vehicle.

[0088] Embodiment three:

[0089] Based on the embodiment one, the embodiment of the application further provides a neural network-based occluded vehicle tail frame regression device, which comprises a processor and a storage medium;

[0090] The storage medium is configured to store instructions;

[0091] The processor is configured to operate according to the instructions to perform steps according to the above method.

[0092] Embodiment four:

[0093] Based on the embodiment one, the embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps of the above method.

[0094] The application provides a neural network-based occluded vehicle tail frame regression method and device and a storage medium, which are mainly applied to the generation of the tail frame of an occluded target vehicle in a vehicle auxiliary driving system. The target vehicle detected in a video is analyzed, and a relatively accurate vehicle tail frame is generated for the target vehicle. The method provided in the present application does not affect the generation of the tail frame of a non-occluded vehicle, and can also generate a relatively accurate tail frame of an occluded vehicle. Moreover, the same deep learning network is used to process all target vehicles, so that the overall operation amount of the system is not increased, and the running speed of the entire system is ensured.

[0095] Firstly, the vehicle-mounted camera is used to collect the image of the road scene in front of the vehicle, so that the road scene in the front view angle that needs to be processed at present is obtained. Then, the target detection method based on YOLOV5 is used to detect the target vehicle existing in the image. Then, the occluded state of all target vehicles is judged, and finally, the relatively accurate feature points and the position information of the vehicle tail center line are obtained by using the deep learning neural network, and the above information is used to generate a relatively accurate vehicle tail frame. Through the above method, the accuracy of the vehicle tail frame in the vehicle auxiliary driving system is effectively improved, so that the decision performance of the entire auxiliary driving system is effectively improved.

[0096] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0097] The present application is described with reference to the flowcharts and / or block diagrams according to the method, device (system), and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a machine that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device that realizes the function specified in one block or multiple blocks.

[0098] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0100] The above description is only preferred embodiments of the present application, it should be pointed out that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A neural network-based occluded vehicle bounding box regression method, characterized in that, The method comprises the following steps: acquire a video clip of a vehicle condition in front of a vehicle, and extract a target image containing a vehicle tail frame from the video clip; obtain position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework; crop a target vehicle image from the target image based on the position information, and pre-process the target vehicle image; input the pre-processed target vehicle image into a trained deep neural network to obtain target vehicle tail occlusion information, vehicle tail feature point position information, and vehicle tail centerline position information; screen out vehicle tail feature points that are not occluded based on the vehicle tail occlusion information and the vehicle tail feature point position information; generate a vehicle tail frame based on the vehicle tail feature points that are not occluded and the vehicle tail centerline position information; The training process of the deep neural network comprises the following steps: obtain historical data of the target vehicle image, and label the vehicle tail feature points in the target vehicle image respectively, if there are occluded feature points, label the occluded feature points according to experience, add labels of occluded and non-occluded to all feature points, label the vehicle tail centerline, and pre-process the labeled target vehicle image to generate a training set; input the training samples in the training set into the deep neural network to obtain vehicle tail feature point prediction information; The loss of the deep neural network is calculated based on vehicle tail feature point prediction information and labeled information; the vehicle tail feature point prediction information includes vehicle tail occlusion prediction information, vehicle tail feature point position prediction information, and vehicle tail centerline position prediction information; the loss of the target vehicle tail occlusion prediction information is calculated by a cross-entropy loss function ; the loss of the vehicle tail centerline position prediction information is calculated by a smooth L1 loss function ; and the loss of the vehicle tail feature point position prediction information is calculated by a smooth L1 loss function . The loss Is: ； wherein, is a function, is target vehicle tail occlusion information, target vehicle tail occlusion prediction information; The loss Is: ； wherein, is the difference between the information annotated for the vehicle tail centerline and the vehicle tail centerline position prediction information; The loss is: ； wherein, when the feature point is occluded, when the feature point is not occluded, ; iterate the network parameters of the deep neural network according to the loss line of the deep neural network by back propagation until a preset iteration number is reached or the loss converges, and complete the training of the deep neural network.

2. The neural network-based occluded vehicle bounding box regression method according to claim 1, wherein, The labeling of the vehicle tail feature points in the target vehicle image comprises the following steps: the left upper corner and the right upper corner of the vehicle tail frame are labeled as feature points 6 and feature points 1 respectively; the left lower corner and the right lower corner of the vehicle tail frame are labeled as feature points 5 and feature points 2 respectively; the left end point of the intersection line between the vehicle left rear wheel and the ground and the right end point of the intersection line between the vehicle right rear wheel and the ground are labeled as feature points 4 and feature points 3 respectively.

3. A neural network based occluded vehicle bounding box regression device, characterized in that, The device comprises: an image acquisition module for acquiring a video clip of a vehicle condition in front of a vehicle, and extracting a target image containing a vehicle tail frame from the video clip; a position acquisition module for obtaining position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework; a cropping processing module for cropping a target vehicle image from the target image based on the position information, and pre-processing the target vehicle image; an information acquisition module for inputting the pre-processed target vehicle image into a trained deep neural network to obtain target vehicle tail occlusion information, vehicle tail feature point position information, and vehicle tail centerline position information; an information screening module for screening out vehicle tail feature points that are not occluded based on the vehicle tail occlusion information and the vehicle tail feature point position information; a tail frame generation module for generating a vehicle tail frame based on the vehicle tail feature points that are not occluded and the vehicle tail centerline position information; The training process of the deep neural network comprises the following steps: The historical data of the target vehicle image is acquired, and the feature points of the vehicle tail in the target vehicle image are labeled respectively. If there is an occlusion in the feature points, the occluded feature points are labeled according to experience, and the labels of occluded and unoccluded are added to all the feature points. The center line of the vehicle tail is labeled. The labeled target vehicle image is preprocessed to generate a training set. The training sample in the training set is input into the deep neural network to obtain vehicle tail feature point prediction information. The loss of the deep neural network is calculated based on vehicle tail feature point prediction information and labeled information; the vehicle tail feature point prediction information includes vehicle tail occlusion prediction information, vehicle tail feature point position prediction information, and vehicle tail centerline position prediction information; the loss of the target vehicle tail occlusion prediction information is calculated by a cross-entropy loss function ; the loss of the vehicle tail centerline position prediction information is calculated by a smooth L1 loss function ; and the loss of the vehicle tail feature point position prediction information is calculated by a smooth L1 loss function . The loss is: ； wherein, is a function, is target vehicle tail occlusion information, target vehicle tail occlusion prediction information; The loss is: ； wherein, is the difference between the information annotated for the vehicle tail centerline and the vehicle tail centerline position prediction information; The loss Is: ； wherein, when the feature point is occluded, when the feature point is not occluded, ; The network parameters of the deep neural network are iteratively propagated in the reverse direction according to the loss of the deep neural network until a preset iteration number is reached or the loss converges, and the training of the deep neural network is completed.

4. A neural network-based occluded vehicle bounding box regression device, characterized by, comprise a processor and a storage medium; the storage medium is configured to store instructions; the processor is configured to operate according to the instructions to perform the steps of the method according to claim 1 or 2.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of claim 1 or 2.

Images