A neural network-based vehicle tail frame regression method and device and storage medium

Abstract

The application discloses a vehicle tail frame regression method and device based on a neural network and a storage medium, and the method comprises the following steps: collecting a video segment of a vehicle condition in front of a vehicle, and extracting a target image containing a vehicle tail frame from the video segment; obtaining position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework; cutting out a target vehicle image from the target image based on the position information, and pre-processing the target vehicle image; inputting the pre-processed target vehicle image into a trained deep neural network to obtain vehicle tail frame prediction information, and generating a vehicle tail frame based on the vehicle tail frame prediction information; and the application can greatly improve the regression accuracy of the vehicle tail frame.

Description

TECHNICAL FIELD

[0001] The application relates to a neural network-based vehicle tail frame regression method and device and a storage medium, and belongs to the technical field of auxiliary driving. BACKGROUND

[0002] In a vehicle auxiliary driving system, the attributes of vehicles running on a road in front of the vehicle need to be analyzed to obtain the position information of the vehicles in a real scene. According to the position of a target vehicle tail frame, the width of the tail of the target vehicle in the real scene and the imaging parameters of a camera, the position of the target vehicle in the real scene can be accurately calculated, so as to provide a basis for subsequent motion state judgment, thereby helping the system to analyze the motion states of all vehicles in the scene and provide relatively accurate auxiliary driving decisions. Then, how to obtain an accurate vehicle tail frame is a core link. There are only two common vehicle tail frame regression methods, scheme one is to directly use a detection network to detect the vehicle tail frame, and scheme two is to first use a detection network to detect a complete vehicle and then regress the tail frame of the target vehicle. Since the size of an input original image is large and there is too much redundant information in the scene, directly detecting the vehicle tail frame often results in a large deviation of the vehicle tail frame. Scheme two first detects the position of the whole vehicle to obtain the position information of the target vehicle, and then directly regresses the tail frame of the target vehicle or first regresses feature points of the tail of the vehicle and then calculates the vehicle tail regression frame according to the feature points of the tail of the vehicle. However, the above method can further improve the accuracy of the vehicle tail frame, but there is still a certain error. How to further improve the regression accuracy of the vehicle tail frame is a problem to be further explored. SUMMARY

[0003] The application aims to overcome the deficiencies in the prior art and provide a neural network-based vehicle tail frame regression method, device and storage medium, which can improve the regression accuracy of the vehicle tail frame.

[0004] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0005] In a first aspect, the application provides a neural network-based vehicle tail frame regression method, comprising:

[0006] Collecting a video segment of a vehicle condition in front of the vehicle, and extracting a target image containing a vehicle tail frame from the video segment;

[0007] Obtaining position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework;

[0008] Cutting out a target vehicle image from the target image based on the position information, and pre-processing the target vehicle image;

[0009] inputting the preprocessed target vehicle image into the trained deep neural network to obtain vehicle tail frame prediction information, and generating a vehicle tail frame based on the vehicle tail frame prediction information.

[0010] Optionally, the position information includes a center point coordinate (cx, cy) of the target vehicle image in the target image, and a width w and a height h of the target vehicle image.

[0011] The cropped target vehicle image has a top-left corner coordinate in the target image of:

[0012]

[0013] The cropped target vehicle image has a bottom-right corner coordinate in the target image of:

[0014]

[0015] Optionally, the training process of the deep neural network includes:

[0016] Obtaining historical data of target vehicle images, and respectively labeling vehicle tail frames in the target vehicle images, the labeling including adding a label frame to the vehicle tail frame, marking a label of each pixel point in the label frame as 1, marking a label of each pixel point outside the label frame as 0, and obtaining a distance of each pixel point in the label frame from a boundary of the label frame.

[0017] Preprocessing the labeled target vehicle images and generating a training set by summarizing the labeled target vehicle images;

[0018] Inputting a training sample in the training set into the deep neural network;

[0019] The classification unit of the deep neural network respectively outputs a label prediction of each pixel point in and outside the label frame, and calculates a classification unit loss based on the label prediction and the label labeling;

[0020] The regression unit of the deep neural network outputs a distance prediction of each pixel point in the label frame from the boundary of the label frame, and calculates a regression unit loss based on the distance prediction and the distance labeling.

[0021] The classification unit loss and the regression unit loss are iteratively back-propagated to network parameters of the deep neural network until a preset iteration number is reached or the loss converges, and the training of the deep neural network is completed.

[0022] Optionally, the preprocessing of the target vehicle image includes deforming the target vehicle image to 96*96 and uniformly padding black edges around to 128*128 to obtain the preprocessed target vehicle image.

[0023] Optionally, the adding of the label frame to the vehicle tail frame includes:

[0024] The bounding box of the vehicle tail frame located in front of the collection vehicle:

[0025] The vehicle roof plane is marked as the upper boundary of the bounding box;

[0026] The intersection line of the rear wheel of the vehicle and the ground is marked as the lower boundary of the bounding box;

[0027] The widest part of the left side of the vehicle tail frame is marked as the left boundary of the bounding box;

[0028] The widest part of the right side of the vehicle tail frame is marked as the right boundary of the bounding box;

[0029] The bounding box of the vehicle tail frame located in front of the collection vehicle:

[0030] The vehicle roof plane is marked as the upper boundary of the bounding box;

[0031] The intersection line of the rear wheel of the vehicle and the ground is marked as the lower boundary of the bounding box;

[0032] The widest part of the left side of the vehicle tail frame is marked as the left boundary of the bounding box;

[0033] The widest part of the right side of the vehicle tail frame is marked as the right boundary of the bounding box;

[0034] The bounding box of the vehicle tail frame located in front of the collection vehicle:

[0035] The vehicle roof plane is marked as the upper boundary of the bounding box;

[0036] The intersection line of the rear wheel of the vehicle and the ground is marked as the lower boundary of the bounding box;

[0037] The widest part of the left side of the vehicle tail frame is marked as the left boundary of the bounding box;

[0038] The widest part of the right side of the vehicle tail frame is marked as the right boundary of the bounding box.

[0039] Optionally, the classification unit loss is calculated by a cross-entropy loss function, and the classification unit loss L cls is:

[0040]

[0041] where y i ′ is the label prediction of the ith pixel point in the bounding box, I is the number of pixel points in the bounding box; y o ′ is the label prediction of the oth pixel point outside the bounding box, O is the number of pixel points outside the bounding box, and sigmoid(·) is a sigmoid function.

[0042] Optionally, the regression unit loss is calculated by a CIoU Loss loss function, and the regression unit loss Lreg is:

[0043]

[0044] wherein, L CIoUi is the regression prediction loss of the i-th pixel point in the annotation box, I is the number of pixel points in the annotation box; the regression prediction loss L CIoU of any pixel point in the annotation box is:

[0045]

[0046]

[0047]

[0048]

[0049] wherein, A is an annotation box, B is a predicted box, the annotation box A and the annotation box B are respectively marked and predicted by the distance of the pixel point to the boundary of the annotation box; p 2 (b, b gt ) is the Euclidean distance between the center point b of the annotation box A and the center point b gt of the predicted predicted box B, c is the diagonal length of the minimum circumscribed rectangle between the annotation box A and the predicted box B; w and h are the width and height of the annotation box A, respectively, w gt and h gt are the width and height of the predicted box B, respectively.

[0050] Optionally, the generating the vehicle tail box based on the vehicle tail box prediction information comprises:

[0051] According to the vehicle tail box prediction information, the label prediction y i of the i-th pixel point in the vehicle tail box is obtained, and the distances d u , d d , d l , d r of the i-th pixel point to the upper boundary, the lower boundary, the left boundary and the right boundary of the vehicle tail box are obtained, and the vehicle tail box prediction F(x, y) is obtained by calculation:

[0052] F(x, y) = (x i -d l , y i -d u , x i -d r , y i -d d )

[0053] wherein, (x i , y i) is the coordinate of the i-th pixel point in the target vehicle image in the vehicle tail frame;

[0054] The weight S(x, y) of each pixel point in the vehicle tail frame is calculated based on the preset confidence n:

[0055]

[0056] wherein sigmoid(·) is a sigmoid function;

[0057] The vehicle tail frame Reg is calculated according to the vehicle tail frame prediction F(x, y) and the weight S(x, y) of the pixel point:

[0058] Reg = ∑S(x, y) * F(x, y).

[0059] In a second aspect, the present application provides a neural network-based vehicle tail frame regression device, comprising a processor and a storage medium;

[0060] The storage medium is used for storing instructions;

[0061] The processor is used for operating according to the instructions to perform the steps of the method according to any one of the above.

[0062] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of the above.

[0063] Compared with the prior art, the present application has the following beneficial effects:

[0064] The neural network-based vehicle tail frame regression method, device and storage medium provided by the present application maximize the use of feature point information of the vehicle tail by taking all pixel points in the labeled frame as feature points without increasing the steps of manual design and labeling of feature points. Meanwhile, a confidence is learned for the tail frame generated by each pixel point in the selected region, and a tail frame of the target vehicle is generated according to the confidence and the tail frame coordinates. Through the above scheme, the steps of tail frame feature point design and labeling are optimized, and each pixel point in the vehicle tail frame is used as much as possible to improve the regression accuracy of the vehicle tail frame as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flowchart of a neural network-based vehicle tail frame regression method provided by an embodiment of the present application;

[0066] Figure 2 is a labeled frame diagram of a vehicle tail frame located in front of a vehicle collected by an embodiment of the present application;

[0067] Figure 3is a marking box schematic view of a vehicle tail frame located at the left front of a collection vehicle provided by an embodiment of the present application;

[0068] Figure 4 is a marking box schematic view of a vehicle tail frame located at the right front of a collection vehicle provided by an embodiment of the present application. DETAILED DESCRIPTION

[0069] The present application will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0070] Embodiment one:

[0071] As shown in the drawings, an embodiment of the present application provides a vehicle tail frame regression method based on a neural network, comprising the following steps: Figure 1

[0072] 1. Collect a video segment of a vehicle front road condition, and extract a target image containing a vehicle tail frame from the video segment.

[0073] 2. Obtain position information of a target vehicle in the target image based on a preset YOLOV5 target detection framework;

[0074] 2.1, the position information includes the center point coordinates (cx, cy) of the target vehicle image in the target image and the width w and height h of the target vehicle image;

[0075] Crop the top left corner coordinates of the target vehicle image in the target image as:

[0076]

[0077] Crop the right bottom corner coordinates of the target vehicle image in the target image as:

[0078]

[0079] 3. Crop the target vehicle image from the target image based on the position information, and pre-process the target vehicle image;

[0080] The pre-processing of the target vehicle image includes: deforming the target vehicle image to 96*96, and uniformly filling black edges around to 128*128 to obtain the pre-processed target vehicle image.

[0081] 4. Input the pre-processed target vehicle image into a trained deep neural network to obtain vehicle tail frame prediction information, and generate a vehicle tail frame based on the vehicle tail frame prediction information.

[0082] The training process of the deep neural network includes:

[0083] ​4.1, obtain historical data of target vehicle images, and respectively label vehicle tail frames in the target vehicle images, the labeling including adding a label frame to the vehicle tail frame, marking each pixel point in the label frame as 1 and each pixel point outside the label frame as 0, and obtaining the distance of each pixel point in the label frame from the boundary of the label frame;

[0084] 4.2, pre-process the labeled target vehicle images and generate a training set; pre-processing the target vehicle images includes deforming the target vehicle images to 96*96 and uniformly padding black edges around to 128*128 to obtain pre-processed target vehicle images.

[0085] 4.3, input the training samples in the training set into a deep neural network;

[0086] The classification unit of the deep neural network outputs the label prediction of each pixel point inside and outside the label frame respectively, and calculates the classification unit loss based on the label prediction and the label annotation;

[0087] The classification unit loss is obtained by calculating the cross-entropy loss function, and the classification unit loss L cls is:

[0088]

[0089] where y i ′ is the label prediction of the i-th pixel point inside the label frame, and I is the number of pixel points inside the label frame; y o ′ is the label prediction of the o-th pixel point outside the label frame, O is the number of pixel points outside the label frame, and sigmoid(·) is the sigmoid function.

[0090] The regression unit of the deep neural network outputs the distance prediction of each pixel point inside the label frame from the boundary of the label frame, and calculates the regression unit loss based on the distance prediction and the distance annotation;

[0091] The regression unit loss is obtained by calculating the CIoU Loss loss function, and the regression unit loss L reg is:

[0092]

[0093] where L CIoUi is the regression prediction loss of the i-th pixel point inside the label frame, and I is the number of pixel points inside the label frame; the regression prediction loss L CIoU of any pixel point inside the label frame is:

[0094]

[0095]

[0096]

[0097]

[0098] wherein A is a labeled box, B is a predicted box, the labeled box A and the labeled box B are respectively labeled by pixel-to-pixel distance and distance prediction of the labeled box boundary; p 2 (b, b gt ) is the Euclidean distance between the center point b of the labeled box A and the center point b gt of the predicted box B, c is the diagonal length of the minimum circumscribed rectangle between the labeled box A and the predicted box B; w and h are respectively the width and height of the labeled box A, w gt and h gt are respectively the width and height of the predicted box B.

[0099] 4.4, the classification unit loss and the regression unit loss are iteratively back-propagated to the network parameters of the deep neural network until a preset iteration number or loss convergence is reached, and the training of the deep neural network is completed.

[0100] 4.5, generating a vehicle tail box based on vehicle tail box prediction information includes:

[0101] According to the vehicle tail box prediction information, the label prediction y i of the i-th pixel point in the vehicle tail box is obtained, and the distances d u , d d , d l , d r of the i-th pixel point to the upper boundary, lower boundary, left boundary and right boundary of the vehicle tail box are obtained, and the vehicle tail box prediction F(x, y) is obtained by calculation:

[0102] F(x, y) = (x i -d l , y i -d u , x i -d r , y i -d d )

[0103] wherein (x i , y i ) is the coordinate of the i-th pixel point in the vehicle tail box on the target vehicle image;

[0104] Based on the preset confidence n, the weight S(x, y) of each pixel point in the vehicle tail box is calculated:

[0105]

[0106] Where sigmoid(·) is the sigmoid function; the confidence level n is generally set to 0.5;

[0107] The vehicle rear frame Reg is calculated based on the predicted vehicle rear frame F(x,y) and the pixel weights S(x,y):

[0108]

[0109] The addition of a label box to the rear frame of the vehicle includes:

[0110] like Figure 2 As shown, the label box is located at the rear of the vehicle directly in front of the data collection vehicle:

[0111] The roof plane is marked as the upper boundary of the annotation frame;

[0112] The line where the rear wheel of the vehicle intersects the ground is marked as the lower boundary of the annotation box;

[0113] The widest point on the left side of the vehicle's rear frame is marked as the left boundary of the annotation frame;

[0114] The widest point on the right side of the vehicle's rear frame is marked as the right boundary of the annotation frame;

[0115] like Figure 3 As shown, the label box is located at the rear of the vehicle on the left front of the data collection vehicle:

[0116] The roof plane is marked as the upper boundary of the annotation frame;

[0117] The intersection of the vehicle's left rear wheel and the ground is marked as the lower boundary of the annotation box;

[0118] The widest point on the left side of the vehicle's rear frame is marked as the left boundary of the annotation frame;

[0119] The widest point on the right side of the vehicle's rear frame is marked as the right boundary of the annotation frame;

[0120] like Figure 4 As shown, the label box is located at the rear of the vehicle on the right front of the data collection vehicle:

[0121] The roof plane is marked as the upper boundary of the annotation frame;

[0122] The line where the right rear wheel of the vehicle intersects with the ground is marked as the lower boundary of the annotation box;

[0123] The widest point on the left side of the vehicle's rear frame is marked as the left boundary of the annotation frame;

[0124] The widest part on the right side of the vehicle's rear frame is marked as the right boundary of the annotation frame.

[0125] Example 2:

[0126] Based on the embodiment one, the embodiment of the present application provides a vehicle tail frame regression device based on a neural network, comprising a processor and a storage medium.

[0127] The storage medium is used for storing instructions.

[0128] The processor is used for operating according to the instructions to execute steps of the above method.

[0129] Embodiment three:

[0130] Based on the embodiment one, the embodiment of the present application provides a computer readable storage medium, which has a computer program stored thereon, and the program is executed by a processor to realize steps of the above method.

[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0132] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the function specified in one block or multiple blocks.

[0133] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the function specified in one block or multiple blocks.

[0134] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or a plurality of flows and / or the functions specified in the block Figure 1 one flow or a plurality of flows and / or the functions specified in the block

[0135] The above description is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A neural network-based vehicle bounding box regression method, characterized in that, The method comprises the following steps: collecting a video clip of a vehicle in front of the vehicle, and extracting a target image containing a vehicle tail frame from the video clip; obtaining position information of the target vehicle in the target image based on a preset YOLOV5 target detection framework; cropping a target vehicle image from the target image based on the position information, and pre-processing the target vehicle image; inputting the pre-processed target vehicle image into a trained deep neural network to obtain vehicle tail frame prediction information, and generating a vehicle tail frame based on the vehicle tail frame prediction information; wherein the training process of the deep neural network comprises: obtaining historical data of the target vehicle image, and labeling the vehicle tail frame in the target vehicle image, the labeling comprising adding a label frame to the vehicle tail frame, marking each pixel point in the label frame as 1 and each pixel point outside the label frame as 0, and obtaining the distance of each pixel point in the label frame from the boundary of the label frame; pre-processing the labeled target vehicle image and generating a training set; inputting the training samples in the training set into the deep neural network; the classification unit of the deep neural network outputs the label prediction of each pixel point in and outside the label frame, and calculates the classification unit loss based on the label prediction and the label annotation; the regression unit of the deep neural network outputs the distance prediction of each pixel point in the label frame from the boundary of the label frame, and calculates the regression unit loss based on the distance prediction and the distance annotation; iteratively back-propagating the classification unit loss and the regression unit loss to the network parameters of the deep neural network until a preset number of iterations is reached or the loss converges, thereby completing the training of the deep neural network; wherein the generation of the vehicle tail frame based on the vehicle tail frame prediction information comprises: According to the vehicle tail frame prediction information, a label prediction y" of an i-th pixel point in the vehicle tail frame is obtained i and distances d of the i-th pixel point to an upper boundary, a lower boundary, a left boundary and a right boundary of the vehicle tail frame u , d , l , r A vehicle tail frame prediction F(x, y) is obtained by calculation F(x,y) = (x i -d l ,y i -d u ,x i -d r ,y i -d d ) wherein (x i , y i ) is the coordinate of the i-th pixel point in the rear frame of the vehicle on the target vehicle image; calculating the weight S(x, y) of each pixel point in the vehicle tail frame based on a preset confidence n: wherein sigmoid(·) is a sigmoid function; calculating the vehicle tail frame Reg based on the vehicle tail frame prediction F(x, y) and the weight S(x, y) of the pixel point: Reg=∑S(x,y)*F(x,y).

2. The neural network-based vehicle bounding box regression method of claim 1, wherein, The position information includes the center point coordinates (cx, cy) of the target vehicle image in the target image, and the width w and height g of the target vehicle image; The top-left corner coordinates of the cropped target vehicle image in the target image are: The bottom-right corner coordinates of the cropped target vehicle image in the target image are:

3. The neural network-based vehicle bounding box regression method of claim 1, wherein, The pre-processing of the target vehicle image comprises deforming the target vehicle image to 96*96 and uniformly padding black edges around it to 128*128 to obtain the pre-processed target vehicle image.

4. The neural network-based vehicle bounding box regression method of claim 1, wherein, The adding of the label frame to the vehicle tail frame comprises: the label frame of the vehicle tail frame located directly in front of the collecting vehicle: the roof plane is labeled as the upper boundary of the label frame; the intersection line of the rear wheel of the vehicle and the ground is labeled as the lower boundary of the label frame; the widest part of the left side of the vehicle tail frame is labeled as the left boundary of the label frame; the widest part of the right side of the vehicle tail frame is labeled as the right boundary of the label frame; the label frame of the vehicle tail frame located in the left front of the collecting vehicle: the roof plane is labeled as the upper boundary of the label frame; the intersection line of the left rear wheel of the vehicle and the ground is labeled as the lower boundary of the label frame; the widest part of the left side of the vehicle tail frame is labeled as the left boundary of the label frame; The right boundary of the annotation box is marked at the widest part of the right side of the vehicle tail frame; The annotation box of the vehicle tail frame located at the right front of the collection vehicle: The upper boundary of the annotation box is marked at the vehicle roof plane; The lower boundary of the annotation box is marked at the intersection line of the right rear wheel of the vehicle and the ground; The left boundary of the annotation box is marked at the widest part of the left side of the vehicle tail frame; The right boundary of the annotation box is marked at the widest part of the right side of the vehicle tail frame.

5. The neural network-based vehicle bounding box regression method of claim 1, wherein, The classification unit loss is obtained by a cross-entropy loss function, and the classification unit loss L cls is: Where, y′ i For the label prediction of the i-th pixel within the bounding box, I is the number of pixels within the bounding box; y′ o This is the label prediction for the o-th pixel outside the bounding box, where O is the number of pixels outside the bounding box, and sigmoid(·) is the sigmoid function.

6. The neural network-based vehicle bounding box regression method of claim 1, wherein, The regression unit loss is obtained by a CIoU Loss loss function, and the regression unit loss L reg is: wherein, L CIoUi is the regression prediction loss of the i-th pixel point in the labeled box, and I is the number of pixel points in the labeled box; the regression prediction loss L CIoU of any pixel point in the labeled box is: wherein A is a labeled box, B is a predicted box, the labeled box A and the labeled box B are respectively obtained through pixel point and pixel point distance labeled box boundary distance labeling and distance prediction; p 2 (b, b gt ) is the Euclidean distance of the center point b of the labeled box A and the center point b gt of the predicted predicted box B, c is the diagonal line length of the minimum circumscribed rectangle between the labeled box A and the predicted box B; w and h are respectively the width and height of the labeled box A, w gt and h gt are respectively the width and height of the predicted box B.

7. A neural network-based vehicle tail frame regression apparatus, characterized by, A processor and a storage medium are included; The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images