A method for judging a vehicle type based on video radar fusion perception

Abstract

The application belongs to the technical field of target identification, and discloses a method for judging vehicle type based on video radar fusion perception. A trapezoidal detection region is defined along a lane in a video picture, and coordinates in the detection region in the video picture are mapped to longitudinal distance and transverse distance of a radar. A length value and a width of a target detected by the radar are extracted, and an RGB value vector of a picture in a vehicle detection frame is extracted. The target length value, the width value and the RGB vector form a vector group representing the target. A plurality of vector group parameter samples of videos and radars form a training set, and training is performed by an SVM trainer. Real-time collected data is input into an SVM trainer target function to identify a vehicle type. The application combines the advantages of radars and videos, so that a radar-video integrated machine can perform stable and accurate vehicle type detection under all-weather and all-live conditions.

Description

Technical Field

[0001] This invention belongs to the field of target recognition technology, and in particular relates to a method for determining vehicle type based on video radar fusion perception. Background Technology

[0002] Faced with increasingly complex road traffic conditions, relevant departments of road traffic management are increasingly using radar and video sensing technologies in intelligent transportation, driven by the development of science and technology. Radar sensors measure the distance, speed, and angle of surrounding objects by emitting high-frequency electromagnetic waves and receiving echoes. Video sensors detect the type and angle of surrounding objects by monitoring video images within their lenses. However, both radar and video sensors have limitations in practical applications. For example, radar technology has limitations in two aspects: first, its resolution for environmental and obstacle details, especially in terms of angular resolution, is not high; second, it cannot identify target types. Video technology has limitations in two aspects: first, it is greatly affected by lighting conditions and environmental factors such as fog, rain, and snow; second, it cannot accurately obtain target distance and speed information. Therefore, it is essential to effectively integrate video and radar data. Summary of the Invention

[0003] This invention combines radar and video to extract feature parameters of vehicles identified by radar and vehicles identified by video. It also addresses video recognition anomalies caused by harsh environments by selecting different detection schemes and using Support Vector Machine (SVM) for calculation. Through multiple SVM binary recognitions, the invention distinguishes vehicle types.

[0004] The method for determining vehicle type based on video radar fusion perception disclosed in this invention includes the following steps:

[0005] Define a trapezoidal detection area along the lane in the video frame, and map the coordinates of the detection area in the video frame to the longitudinal and lateral distances of the radar.

[0006] Extract the length and width of the target detected by the radar, extract the RGB value vector of the image within the vehicle detection box, and combine the target length, width, and RGB vector into a vector group representing the target;

[0007] The training set is composed of vector group parameter samples from multiple videos and radars, and then trained using an SVM trainer.

[0008] The real-time collected data is input into the objective function of the SVM trainer to identify vehicle models.

[0009] Furthermore, the radar is installed at the center line of the lane being detected.

[0010] Furthermore, defining the trapezoidal detection region along the lane in the video frame includes:

[0011] Define a trapezoidal detection region along the lane in the video frame, and record the coordinates of the four vertices in the frame: (x1, y1), (x2, y2), (x3, y3), (x4, y4), where y1 = y2 and y3 = y4. Calculate the height h of the trapezoidal detection region in the video frame. v =y4-y2;

[0012] Calibrate the actual detection area width D r And the number of lanes, M;

[0013] The high h of the detection area in the video v Divide the sample into n equal parts from top to bottom, and label h. v The actual distance dh corresponding to each bisector i (i = 1, 2, ..., n, n+1);

[0014] Calculate the y-coordinate of any point within the detection area in the video frame. k (y2≤y k The actual distance H corresponding to ≤y4) k ;

[0015] A function to calculate the left and right boundary lines L1 and L2 of the lane in the video frame:

[0016]

[0017]

[0018] Calculate the coordinates (x, y) of any point within the detection area in the video frame. k ,y k The actual distance D from the center line of the lane. k The coordinates (x, y) of the detection area in the video frame k ,y k The longitudinal distance H mapped onto the radar k And lateral distance D k .

[0019] Furthermore, the H k The calculation formula and steps are as follows:

[0020]

[0021]

[0022] Furthermore, the actual distance D k The calculation formula and steps are as follows:

[0023]

[0024]

[0025]

[0026] Furthermore, the extraction of the length and width of the target detected by the radar, and the extraction of the RGB value vector of the image within the vehicle detection box, include:

[0027] Using pre-trained vehicle samples in the video, a feature classifier is used to detect vehicles within the video detection area, and the coordinates (x, y) of the detected vehicles are extracted. k ,y k ) and the width of the detection frame of the detected vehicle k and high HEIGHT k ;

[0028] Calculate the coordinates (D) of the detected vehicle in the video within the radar coordinate system. k H k );

[0029] Searching for the relative (D) detected by radar k H k The coordinates of the target with the smallest deviation are used to extract the length value C of the target detected by the radar. l and width C w ;

[0030] The image within the detection frame of the vehicle to be detected is converted into an image with width W and height H. The RGB values ​​of the image are extracted and denoted as a vector ω of length W*H*3. p .

[0031] Furthermore, the training using an SVM trainer includes:

[0032] Construct the SVM objective function:

[0033]

[0034] Where y i ∈{-1,1}, δ is a positive integer, x i For parameter ω p C L and C w The vector group formed is [C] when the video feasibility is judged to be low. L C W ], otherwise it is [ω] p C L C W ];

[0035] The training set is composed of n vector parameter samples from videos and radar. An SVM trainer is used to train the dataset to obtain the minimum ||w||, which is the minimum w. T and b.

[0036] Furthermore, the method for determining low video feasibility is as follows:

[0037] Each frame is converted to grayscale value, and the average brightness value imgL is calculated. Upper and lower limits for brightness imgL_max and imgL_min are set. If the brightness exceeds the upper limit, the image is too bright; if it is below the lower limit, the image is too dark. If the cumulative number of frames below the lower limit or above the upper limit exceeds 3000, it is considered an abnormal situation, and the video detection reliability is low.

[0038] Furthermore, the method for determining low video feasibility also includes:

[0039] The number of vehicles detected in the video detection area, C_num_v, and the number of vehicles detected by the radar in the actual detection area, C_num_r, are counted. If |C_num_v-C_num_r| is greater than C_num_r*0.2 and the cumulative time is greater than 120 seconds, then the video detection is abnormal and the feasibility of video detection is low.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This invention combines radar and video. When the video scene is suitable, a multi-parameter fusion mode of video and radar is used to achieve high-accuracy vehicle model detection. When the video detection error is large in bad weather, radar parameters are used for detection to improve detection stability.

[0042] By combining the advantages of radar and video, the radar-visual integrated machine can perform stable and accurate vehicle detection in all weather conditions. Attached Figure Description

[0043] Figure 1 A schematic diagram of a trapezoidal detection area defined along a lane in a video frame according to the present invention;

[0044] Figure 2 Flowchart for determining video credibility in this invention;

[0045] Figure 3 Flowchart of the method for determining vehicle model according to the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0047] Step 1: Unify the location of targets detected by radar and video.

[0048] 1) such as Figure 1 As shown, a trapezoidal detection region is defined along the lane in the video frame, and the coordinates of the four vertices in the frame are recorded as (x1, y1), (x2, y2), (x3, y3), and (x4, y4), where y1 = y2 and y3 = y4. The height h of the trapezoidal detection region in the video frame is calculated. v = y4 - y2 (unit: pixels in the video).

[0049] 2) Calibrate the actual detection area width D r And the number of lanes, M.

[0050] 3) High h in the detection area of ​​the video v Divide the sample into n equal parts from top to bottom, and label h. v The actual distance dh corresponding to each bisector i (i = 1, 2, ..., n, n+1).

[0051] 4) Calculate the y-coordinate of any point within the detection area in the video frame. k (y2≤y k The actual distance H corresponding to ≤y4) k The calculation formula and steps are as follows:

[0052] (i takes the form of a positive integer)

[0053]

[0054] 5) Calculate the function for the left and right boundary lines L1 and L2 of the lane in the video image.

[0055]

[0056]

[0057] 6) The radar is installed at the perpendicular line of the lane being detected, and calculates the coordinates (x, y) of any point within the detection area in the video image. k ,y k The actual distance D from the center line of the lane. k (The left side of the perpendicular bisector is negative, and the right side is positive).

[0058]

[0059]

[0060]

[0061] Therefore, the coordinates (x, y) of the detection area in the video frame can be determined.k ,y k The longitudinal distance H mapped onto the radar k And lateral distance D k .

[0062] Step 2: Radar parameter extraction and video parameter extraction

[0063] 1) Using pre-trained vehicle samples in the video, a Haar+Cascades feature classifier is used to detect vehicles within the video detection area, and the coordinates (x, y) of the detected vehicles are extracted. k ,y k ) and the width of the detection frame of the detected vehicle k and high HEIGHT k .

[0064] 2) Using the formula in step one, calculate the coordinates (D) in the radar coordinate system corresponding to the coordinates of the detected vehicle in the video. k H k ). Searching for the relative (D) detected by radar. k H k The coordinates of the target with the smallest deviation are used to extract the length value C of the target detected by the radar. l and width C w .

[0065] 3) Set a uniform width W and height H, and convert the image within the detection box of the vehicle in step 2(1) into an image with a width W and a height H. Extract the RGB values ​​of the image and denote them as a vector ω of length W*H*3. p .

[0066] 4) ω p C L and C w The data is then sent to step four for vehicle model recognition.

[0067] Step 3: Troubleshooting video processing anomalies

[0068] Because video detection is significantly affected by weather and lighting changes, the accuracy of video vehicle detection decreases during extreme weather conditions or external interference such as heavy fog, heavy rain, and strong sunlight, affecting the accuracy of the combined radar and video detection in determining vehicle type. Therefore, we need to monitor and investigate any anomalies in the video in real time, reducing the reliability of video parameters when anomalies occur. When video reliability is low, step four, vehicle type detection, will only use radar data. Figure 2 As shown, the steps are as follows:

[0069] 1) Convert each frame to grayscale value, calculate the average brightness value imgL, and set upper and lower limits for brightness imgL_max and imgL_min. If the brightness exceeds the upper limit, the image is too bright; if it is below the lower limit, the image is too dark. If the cumulative number of frames below the lower limit or above the upper limit exceeds 3000, it is considered an abnormal situation, and the video detection reliability is low.

[0070] 2) Compare the number of vehicles C_num_v detected in the video detection area and the number of vehicles C_num_r detected by the radar in the actual detection area in step 2, 1). Calculate |C_num_v - C_num_r|. If this value is greater than C_num_r * 0.2 and the cumulative time is greater than 120 seconds, then the video detection is abnormal and the feasibility of video detection is low.

[0071] Step 4: Use SVM to identify vehicle models from radar and video parameters.

[0072] 1) Construct the SVM objective function:

[0073]

[0074] Where y i ∈{-1,1}, δ is a positive integer, x i This is a vector composed of the parameters extracted in step two. When the video feasibility is determined to be low in step three, this vector is [C]. L C W ], otherwise it is [ω] p C L C W ].

[0075] Prepare n video and radar parameter samples in advance, train them using an SVM trainer, and obtain the minimum ||w||, which is the minimum w. T and b.

[0076] 2) Vehicle model recognition

[0077] Using the [ω] collected in step two p C L C WThe parameters are input into the objective function. If the result is greater than or equal to δ, the vehicle is classified as a large vehicle; otherwise, it is classified as a small vehicle. If it is necessary to distinguish multiple vehicle types, such as large, medium, and small, multiple objective functions are constructed, and multiple binary selections are performed to determine the vehicle type. For example, if it is necessary to identify large, medium, and small vehicles, two objective functions are constructed according to Formula 4 (a first objective function and a second objective function, with different δ values ​​in the first and second objective functions). The training set of the first objective function consists of small vehicle sample parameters and medium vehicle sample parameters, which are input into the first objective function for training. After training, the collected real-time vehicle image parameters are input into the first objective function. If the result is greater than or equal to δ, the vehicle is classified as a medium vehicle; otherwise, it is classified as a small vehicle. The training set of the second objective function consists of medium vehicle sample parameters and large vehicle sample parameters, which are input into the second objective function for training. After training, the collected real-time vehicle image parameters are input into the second objective function. If the result is greater than or equal to δ, the vehicle is classified as a large vehicle; otherwise, it is classified as a medium vehicle.

[0078] Compared with the prior art, the beneficial effects of the present invention are:

[0079] This invention combines radar and video. When the video scene is suitable, a multi-parameter fusion mode of video and radar is used to achieve high-accuracy vehicle model detection. When the video detection error is large in bad weather, radar parameters are used for detection to improve detection stability.

[0080] By combining the advantages of radar and video, the radar-visual integrated machine can perform stable and accurate vehicle detection in all weather conditions.

[0081] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0082] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0083] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0084] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for determining a vehicle type based on video-radar fusion perception, characterized in that, The method comprises the following steps: defining a trapezoidal detection region along the lane in the video picture, mapping the coordinates in the detection region in the video picture to the longitudinal distance and lateral distance of the radar; extracting the length value and width of the target detected by the radar, extracting the RGB value vector of the picture in the vehicle detection frame, and combining the target length value, width value and RGB vector to form a vector group representing the target; combining the vector group parameter samples of multiple videos and radars to form a training set, and training by using an SVM trainer; inputting the real-time collected data into the SVM trainer target function to identify the vehicle type; the step of defining a trapezoidal detection region along the lane in the video picture comprises: In the video picture, a trapezoidal detection region is framed along the lane, and the coordinates (xl,yl), (x2,y2), (x3,y3), (x4,y4) of the four vertices in the picture are recorded, wherein yl=y2, y3=y4, and the height h of the trapezoidal detection region in the video picture is calculated v =y4-y2; Calibrating actual detection area width D r and the number of lanes M; h v Divide the height h v The actual distance dh i (i = 1, 2,..., n, n + 1); Calculating the vertical coordinate y of any point in the detection region in the video picture k (y2≤y k ≤y4) corresponding to the actual distance H k ; calculating the functions of the left and right boundary lines L1 and L2 of the lane in the video picture: calculating the actual distance D k of any point in the detection region of the video picture corresponding to the coordinate (x k ,y k ), and mapping the coordinate (x k ,y k ) in the detection region of the video picture to the longitudinal distance H k and the lateral distance D k of the radar; The actual distance D k The formula and steps for calculation are as follows:

2. The method of claim 1, wherein the method further comprises: the radar is installed at the vertical line in the detected lane. 3.The method of claim 1, wherein, The H k The calculation formula and steps are as follows:

4. The method of claim 1, wherein the method further comprises: the step of extracting the length value and width of the target detected by the radar, and extracting the RGB value vector of the picture in the vehicle detection frame comprises: Using pre-trained vehicle samples in the video, a feature classifier is used to detect vehicles within the video detection area, and the coordinates (x, y) of the detected vehicles are extracted. k ,y k ) and the width of the detection frame of the detected vehicle k and high HEIGHT k ; coordinates in the radar coordinate system corresponding to the detected vehicle coordinates in the video are calculated (D k ,H k ) finds the coordinates of the target with the minimum relative (D k , H k ) deviation detected by the radar, and extracts the length value C l and the width C w of the target detected by the radar; The picture in the detection frame of the detected vehicle is converted into a picture with a width of W and a height of H, and the RGB values of the picture are extracted, denoted as a vector ω with a length of W*H*3 p .

5. The method of claim 1, wherein the method further comprises: the step of training by using the SVM trainer comprises: constructing an SVM target function: wherein y i ∈{-1,1}, δ is a positive integer, x i is a parameter ω p , C L and C w consist of a vector group, when judging the video feasibility is low, the vector group is [C L , C W ], otherwise [ω p , C L , C W ]; The n video and radar vector group parameter samples are combined to form a training set, which is trained by an SVM trainer to obtain the minimum ||w||, i.e., the minimum w T and b.

6. The method of claim 1, wherein the method further comprises: the method for judging whether the video is of low feasibility comprises the following steps: converting each frame of picture into a gray value, counting the average brightness imgL of the picture, setting the upper and lower limit values imgL_max and imgL_min of the picture brightness, if the picture brightness exceeds the upper limit value, the picture is too bright, if the picture brightness is lower than the lower limit value, the picture is too dark, if the number of frames lower than the lower limit brightness or higher than the upper limit brightness exceeds 3000 frames, it is an abnormal condition, and the video detection reliability is low.

7. The method of claim 1, wherein the method further comprises: the method for judging whether the video is of low feasibility further comprises: counting the number C_num_v of vehicles in the detected video detection region and the number C_num_r of vehicles detected by the radar in the actual detection region, if |C_num_v-C_num_r| is greater than C_num_r*0.2 and the accumulated time is greater than 120 seconds, the video detection is abnormal, and the feasibility of the video detection is low.

8. The method of claim 5, wherein the method further comprises: the step of inputting the real-time collected data into the SVM trainer target function to identify the vehicle type comprises: inputting the collected vehicle vector group parameters into the target function, if the result is greater than or equal to δ, the vehicle is a large vehicle, otherwise, the vehicle is a small vehicle.

Images