Nighttime vehicle detection method and device integrating radar and vehicle light detection

Abstract

The present invention provides a night vehicle detection method and device integrating radar and headlight detection, firstly obtaining an image to be processed and corresponding radar data; then processing the image to be processed based on an adaptive threshold algorithm and a pre-acquired headlight color threshold range to obtain a first image containing headlight information; determining the headlight information based on the first image and preset constraints; finally determining the target vehicle information based on the headlight information and the radar data. The present invention processes the image to be processed based on an adaptive threshold algorithm and a headlight color range to obtain relatively accurate headlight information, and further integrates it with radar data to obtain vehicle information, thereby improving the accuracy of vehicle information detection and driving safety.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a nighttime vehicle detection method and device integrating radar and vehicle light detection. Background Art

[0002] In the related art, at night, a moving vehicle usually uses an on-board camera to obtain an image of the vehicle in front, and detects the vehicle in front by extracting the vehicle taillights in the image. However, due to insufficient light at night, the light of street lamps and the light of the current vehicle will cause the accuracy of the above vehicle detection method to be low. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a night vehicle detection method and device that integrates radar and vehicle light detection, so as to improve the accuracy of vehicle information detection and enhance driving safety.

[0004] In the first aspect, an embodiment of the present invention provides a night vehicle detection method that integrates radar and headlight detection, including: obtaining an image to be processed and corresponding radar data; processing the image to be processed based on an adaptive threshold algorithm and a headlight color threshold range to obtain a first image containing headlight information; determining the headlight information based on the first image and preset constraints; and determining the target vehicle information based on the headlight information and the radar data.

[0005] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the step of processing the image to be processed based on an adaptive threshold algorithm and a headlight color threshold range to obtain a first image containing headlight information includes: processing the image to be processed based on an adaptive threshold algorithm to obtain a first preprocessed image including a first highlight area; processing the image to be processed based on the headlight color threshold range to obtain a second preprocessed image including a second highlight area; determining a third preprocessed image based on the first preprocessed image and the second preprocessed image; performing a closing operation and a median filter on the third preprocessed image to obtain a first image containing headlight information.

[0006] In combination with the first possible implementation of the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the image format of the above-mentioned image to be processed includes an RGB format; the step of processing the image to be processed based on an adaptive threshold algorithm to obtain a first preprocessed image including a first highlight area includes: converting the image format of the image to be processed to obtain a first grayscale image; calculating the mean and standard deviation of the first grayscale image; obtaining an adaptive threshold based on the mean and the standard deviation; binarizing the first grayscale image based on the adaptive threshold to obtain a first preprocessed image including a first highlight area.

[0007] In combination with the second possible implementation of the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the step of processing the image to be processed based on the headlight color threshold range to obtain a second preprocessed image including a second highlight area includes: converting the image format of the image to be processed to obtain a first preliminary image with an image format of HSV; processing the H channel of the first preliminary image based on the pre-acquired headlight color threshold range to obtain a second preliminary image; converting the image format of the second preliminary image to obtain a second grayscale image; extracting the red channel of the second grayscale image to obtain a third preliminary image; binarizing the third preliminary image based on an adaptive threshold to obtain a second preprocessed image including a second highlight area.

[0008] In combination with the first possible implementation of the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of determining the third preprocessed image based on the first preprocessed image and the second preprocessed image includes: for each second pixel in the second highlight area, judging whether the first pixel corresponding to the second pixel belongs to the first highlight area; the first pixel belongs to the first preprocessed image; the second pixel belongs to the second preprocessed image; if the corresponding first pixel belongs to the first highlight area, setting the grayscale value of the second pixel to 255; if the corresponding first pixel does not belong to the first highlight area, setting the grayscale value of the second pixel to 0; determining the second pixel and an image composed of other pixels in the second preprocessed image except the second pixel as the third preprocessed image.

[0009] In combination with the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the step of determining the headlight information based on the first image and preset constraints includes: obtaining a valid area of ​​the first image based on the shooting conditions of the image to be processed; searching for a connected domain in the valid area; the connected domain includes multiple domains; determining a valid connected domain pair based on the center distance between any two connected domains; the valid connected domain pair includes two connected domains; calculating parameter values ​​corresponding to preset parameters of the valid connected domain pair; if the parameters meet the preset constraints, determining that the valid connected domain pair includes the headlight information; obtaining the headlight information based on the valid connected domain; the headlight information includes the headlight position and the headlight brightness.

[0010] In combination with the fifth possible implementation of the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the above-mentioned headlight position includes a left headlight position and a right headlight position; the step of determining the target vehicle information based on the headlight information and radar data includes: determining the left headlight position and the middle position of the line connected by the headlight positions as target pixel points; tracking the target pixel points according to the pre-acquired predicted position to determine the effective headlight position; determining the preliminary position of the target vehicle according to the effective headlight position and a preset coordinate transformation model; determining the effective target point position according to the radar data; data-associate the preliminary position of the target vehicle with the effective target point position to obtain a matching target point; track the target point to obtain the position and speed of the target vehicle; determine the state of the target vehicle according to the headlight brightness; the state of the target vehicle includes normal driving or braking; determine the position, speed and state of the target vehicle as the target vehicle information.

[0011] In the second aspect, an embodiment of the present invention also provides a night vehicle detection device that integrates radar and headlight detection, including: a data acquisition module, used to acquire an image to be processed and corresponding radar data; an image processing module, used to process the image to be processed based on an adaptive threshold algorithm and a headlight color threshold range to obtain a first image containing headlight information; a headlight information determination module, used to determine the headlight information based on the first image and preset constraints; and a vehicle information determination module, used to determine the target vehicle information based on the headlight information and radar data.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above method.

[0013] In a fourth aspect, an embodiment of the present invention further provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above method.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] The embodiment of the present invention provides a night vehicle detection method and device integrating radar and headlight detection. First, an image to be processed and corresponding radar data are obtained; then the image to be processed is processed based on an adaptive threshold algorithm and a pre-acquired headlight color threshold range to obtain a first image containing headlight information; headlight information is determined based on the first image and preset constraints; finally, target vehicle information is determined based on the headlight information and radar data. This method processes the image to be processed through an adaptive threshold algorithm and a headlight color range to obtain relatively accurate headlight information, and further integrates it with radar detection data to obtain vehicle information, thereby improving the accuracy of vehicle information detection and driving safety.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flow chart of a night vehicle detection method integrating radar and vehicle light detection provided by an embodiment of the present invention;

[0020] Figure 2 A flow chart of another night vehicle detection method integrating radar and vehicle light detection provided by an embodiment of the present invention;

[0021] Figure 3 A flow chart of a night vehicle detection method based on the fusion of millimeter-wave radar and vehicle light detection provided by an embodiment of the present invention;

[0022] Figure 4 A flowchart of a data preprocessing process provided by an embodiment of the present invention;

[0023] Figure 5 A flow chart of a vehicle light detection process provided by an embodiment of the present invention;

[0024] Figure 6A flowchart of probability calculation in a vehicle light area matching process provided by an embodiment of the present invention;

[0025] Figure 7 A flowchart of a pixel tracking process provided by an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of a vehicle position calculation process provided by an embodiment of the present invention;

[0027] Fig. 9 A schematic diagram of a radar target point determination process provided by an embodiment of the present invention;

[0028] Fig.10 A flowchart of a comparison process for determining a radar point from a radar target point provided by an embodiment of the present invention;

[0029] Fig.11 A schematic diagram of the structure of a night vehicle detection device integrating radar and vehicle light detection provided by an embodiment of the present invention;

[0030] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] With the rapid development of the autonomous driving industry, its safety issues are becoming more and more prominent. The safety of smart cars depends largely on the development of environmental perception technology. At present, the development of smart car environmental perception technology mainly includes smart car environmental perception technology based on lane electronic marking recognition, smart car environmental perception technology based on a single sensor, and smart car environmental perception technology based on multi-sensor information fusion. The combination of sensors such as cameras, millimeter wave radars, lidars, and ultrasonic radars has become the basic configuration for smart cars to perceive the external environment.

[0033] Millimeter-wave radar is mostly used to sense obstacles ahead, but it cannot identify obstacles. Camera vision sensors are mostly used for obstacle detection and classification, but they are affected by light, climate conditions, etc., and cannot work around the clock. In particular, the difficulty of detecting vehicles at night will increase significantly. LiDAR has high measurement accuracy and a wide range of applications. It can perform target segmentation and tracking methods, and can identify and track obstacles. The disadvantage is that the cost is too high and it cannot be applied to mass-produced models. Therefore, environmental perception technology solutions that rely on one sensor have great limitations. Vehicle detection using multi-sensor fusion is the mainstream trend of current technology.

[0034] The detection of the vehicle ahead by the on-board camera depends largely on different lighting environments. Insufficient lighting conditions are an important difficulty in nighttime vehicle detection research. In the nighttime camera images, the vehicle headlights are relatively obvious. However, due to the high brightness of the current vehicle headlights, it is easy to cause image overexposure. In actual scenes, there is often interference from isolation guardrails and road lights between the left and right lanes, which makes it more difficult to extract the headlights.

[0035] At present, some research has been done on the recognition of vehicle taillights. The related art proposes an image-based taillight detection and recognition method. The taillights are extracted using image sharpening and segmentation technology, and the SVM method is used to classify the taillights. In order to speed up the detection speed, this method only detects the red component of the night image, uses the symmetry of the taillight detection, and sets different segmentation thresholds according to the distance to achieve the extraction of the lights and vehicle recognition.

[0036] Related technologies also propose a vehicle recognition method in nighttime high-speed scenarios. The default shape of the headlights is circular. The reflectors are extracted and the lane boundaries are identified through the brightness and area information of the headlights. The reflectors in the image are removed and the circumference of the red area around the white light spot is calculated. If most of it is red, it is considered to be a pair of taillights. Finally, the vehicle is recognized based on the pair of taillights.

[0037] An analysis of the above-mentioned detection method based on car lights shows that the current detection method is still imperfect in the following aspects: the taillight extraction threshold has poor adaptability to the environment and basically adopts a fixed threshold; the shapes of the car lights are not uniform and the detection algorithm is not robust enough; the taillight extraction is easily interfered by other light sources; and the accuracy of vehicle detection is seriously reduced when vehicles block each other.

[0038] Based on this, the embodiments of the present invention provide a night vehicle detection method, device and electronic equipment that integrates radar and vehicle light detection, which can be applied to vehicle detection in various driving scenarios, such as at night and on cloudy days.

[0039] To facilitate understanding of this embodiment, a night vehicle detection method integrating radar and vehicle light detection disclosed in an embodiment of the present invention is first introduced in detail.

[0040] The embodiment of the present invention provides a night vehicle detection method integrating radar and vehicle light detection. Figure 1 As shown, the method includes:

[0041] Step S100, obtaining an image to be processed and corresponding radar data.

[0042] The above-mentioned image to be processed can be obtained by taking pictures by a vehicle-mounted camera, and its image format is usually RGB (red green blue) format. The radar data corresponding to the above-mentioned image to be processed can be obtained by a millimeter wave radar at the same time as the image to be processed is taken.

[0043] Step S102 : processing the image to be processed based on the adaptive threshold algorithm and the pre-acquired vehicle light color threshold range to obtain a first image containing vehicle light information.

[0044] The above adaptive threshold algorithm is a method of using the local threshold of the image to replace the global threshold for image calculation. When used, it is specifically aimed at pictures with large changes in light and shadow, or pictures with less obvious color differences within the range. Adaptation means ensuring that the computer can iterate by determining and calculating the average threshold of the image area. The above headlight color threshold range can be obtained through statistics, and the pixels in the image to be processed that are within the headlight color threshold range can be retained.

[0045] Step S104, determining vehicle light information according to the first image and preset constraints.

[0046] The above-mentioned preset constraint conditions can be geometric conditions, such as the shape of the headlights, the distance between two headlights, etc.; they can also be pixel grayscale conditions, such as determining pixels whose pixel grayscale exceeds a certain threshold as part of the headlights. When the first image already contains the headlight information, the headlight information can be extracted from the first image based on the preset constraint conditions. The headlight information can include the position of the headlights and the brightness of the headlights.

[0047] Step S106, determining target vehicle information based on vehicle light information and radar data.

[0048] The above-mentioned target vehicle information may include position information and driving status information. After the headlight information is determined, the position information of the target vehicle can be calculated based on the position of the headlight. However, since the position information is obtained through images, and the images have certain distortions, the position information is inaccurate. Radar data is obtained at the same time as the image to be processed. The vehicle information obtained from the radar data contains the position information of the current target vehicle. The position information obtained from the image to be processed is associated with the position information obtained from the radar data to obtain more accurate target vehicle information.

[0049] The embodiment of the present invention provides a night vehicle detection method integrating radar and headlight detection, firstly obtaining an image to be processed and corresponding radar data; then processing the image to be processed based on an adaptive threshold algorithm and a pre-acquired headlight color threshold range to obtain a first image containing headlight information; determining the headlight information based on the first image and preset constraints; finally determining the target vehicle information based on the headlight information and radar data. The method processes the image to be processed using an adaptive threshold algorithm and a headlight color range to obtain relatively accurate headlight information, and further integrates it with radar data to obtain vehicle information, thereby improving the accuracy of vehicle information detection and driving safety.

[0050] The embodiment of the present invention also provides another night vehicle detection method integrating radar and vehicle light detection. Figure 1 The method shown is implemented based on Figure 2 As shown, the method comprises the following steps:

[0051] Step S200, obtaining an image to be processed and corresponding radar data.

[0052] Step S202: Process the image to be processed based on an adaptive threshold algorithm to obtain a first pre-processed image including a first highlight area.

[0053] Specifically, the image format of the above-mentioned image to be processed includes RGB format; the image format of the image to be processed can be converted first to obtain a first grayscale image; then the mean and standard deviation of the first grayscale image are calculated; an adaptive threshold is obtained based on the mean and standard deviation; and the first grayscale image is binarized based on the adaptive threshold to obtain a first preprocessed image including a first highlight area.

[0054] Step S204: Process the image to be processed based on the vehicle light color threshold range to obtain a second pre-processed image including a second highlight area.

[0055] The above steps can be implemented based on the HSV (Hue, Saturation, Value) format. First, the image format of the image to be processed is converted to obtain a first preliminary image with an image format of HSV; then, based on the pre-acquired headlight color threshold range, the H channel of the first preliminary image is processed to obtain a second preliminary image, specifically, the H channel threshold is set, and the H channel values ​​within the threshold range are retained, and the H values ​​outside the range are directly set to 0; then the image format of the second preliminary image is converted to obtain a second grayscale image; the red channel of the second grayscale image is further extracted to obtain a third preliminary image; finally, the third preliminary image is binarized based on an adaptive threshold to obtain a second preprocessed image including a second highlight area. The above-mentioned headlight color threshold range mainly includes the color ranges of yellow, purple, and red.

[0056] Step S206: determining a third preprocessed image according to the first preprocessed image and the second preprocessed image.

[0057] Normally, the highlighted parts in both the first preprocessed image and the second preprocessed image need to be retained as highlighted, and the rest are set to grayscale 255. In specific operations, for each second pixel in the second highlighted area, it can be determined whether the first pixel corresponding to the second pixel belongs to the first highlighted area; the first pixel belongs to the first preprocessed image; the second pixel belongs to the second preprocessed image; if the corresponding first pixel belongs to the first highlighted area, the grayscale value of the second pixel is set to 255; if the corresponding first pixel does not belong to the first highlighted area, the grayscale value of the second pixel is set to 0; the image composed of the second pixel and other pixels in the second preprocessed image except the second pixel is determined as the third preprocessed image.

[0058] Step S208, performing a closing operation and a median filter process on the third pre-processed image to obtain a first image containing vehicle light information.

[0059] Step S210, based on the shooting conditions of the image to be processed, obtain the effective area of ​​the first image. The above shooting conditions mainly refer to the height of the vehicle camera when the vehicle camera shoots the image to be processed; this condition will affect the part of the image to be processed that includes the headlight information. Generally speaking, the effective search area (i.e. the above effective area) is limited to the upper half of the image, and the front part of the vehicle at the lower edge is removed. The front of the vehicle often has reflections from the headlights of the preceding vehicle, which can cause false detection, so it also needs to be filtered out. The front of the vehicle is approximately 0.18 times the height of the image.

[0060] Step S212, searching for a connected domain in the valid area; the connected domain includes multiple connected domains.

[0061] The above-mentioned connected component generally refers to an image region (Region, Blob) composed of foreground pixels with the same pixel value and adjacent positions in the image. The size of the connected domain is usually set between 100 and 1000 pixels.

[0062] Step S214, determining a valid connected domain pair according to the center distances between any two connected domains; the valid connected domain pair includes two connected domains.

[0063] Step S216, calculating parameter values ​​corresponding to preset parameters of the valid connected domain pair, which may include horizontal distance, vertical distance, overlap, height similarity, width similarity, body-to-light width ratio, body-to-light height ratio, etc. of two connected domains.

[0064] Step S218, if the parameters meet the preset constraints, determine whether the valid connected domain pair includes the headlight information; the above preset constraints can be statistically obtained empirical thresholds. If the parameters fall within the valid range interval composed of the empirical thresholds, the probability is increased, and if they do not meet the valid range, the probability is reduced; then based on the preset probability threshold, determine whether the valid connected domain pair includes the headlight information.

[0065] Step S220, based on the valid connected domain, obtain the headlight information; the headlight information includes the headlight position and the headlight brightness; in this process, duplicate headlights will be removed.

[0066] Step S222, determining the target vehicle information based on the vehicle light information and radar data.

[0067] Specifically, the above-mentioned headlight positions usually include the left headlight position and the right headlight position; when determining the target vehicle information, the left headlight position and the middle position of the line connected by the headlight positions can be determined as the target pixel point; the target pixel point is tracked according to the pre-acquired predicted position to determine the effective headlight position; the preliminary position of the target vehicle is determined according to the effective headlight position and the preset coordinate transformation model; the effective target point position is determined according to the radar data; the preliminary position of the target vehicle is data-associated with the effective target point position to obtain a matching target point; the target point is tracked to obtain the position and speed of the target vehicle; the state of the target vehicle is determined according to the headlight brightness; the state of the target vehicle includes normal driving or braking; the position, speed and state of the target vehicle are determined as the target vehicle information.

[0068] This method combines the image information captured by the vehicle-mounted camera with the millimeter-wave radar information to determine the vehicle information, improve detection accuracy, and enhance driving safety.

[0069] An embodiment of the present invention also proposes a night vehicle detection method based on the fusion of millimeter-wave radar and headlight detection (equivalent to the night vehicle detection method based on the fusion of radar and headlight detection mentioned above), which can effectively improve the accuracy of night vehicle recognition, while improving the accuracy of night vehicle headlight recognition, and effectively realize the headlight classification function.

[0070] like Figure 3 As shown, the method comprises the following steps:

[0071] (1) Camera data acquisition.

[0072] (2) Data preprocessing: Specifically, using the RGB image captured by the camera, an adaptive threshold algorithm is applied to the grayscale image to extract the highlight area, and the red and yellow areas are extracted based on the HSV channel and the color statistics of the car lights, and the intersection is taken to obtain a binary image. The obtained binary image is morphologically processed, that is, closed operation and median filtering are performed, and the processed image containing only the area of ​​interest is output.

[0073] (3) Headlight detection: In the valid area of ​​the preprocessed image, search for a connected domain of appropriate size, and then calculate the matching probability of two possible headlight areas based on geometric constraints. Filter out the headlight pair with the highest probability, remove duplicate headlights, and obtain the detected headlight pair.

[0074] (4) Pixel tracking: Based on the matched pair of headlights, the midpoint coordinates of the headlights, i.e., the pixel coordinates of the target point, are obtained, and pixel tracking is performed to determine the valid headlight pair.

[0075] (5) Vehicle status classification: Based on the brightness value of the valid headlight pair, it is further classified as brake lights or tail lights to obtain the status of the target vehicle.

[0076] (6) Coordinate transformation: Based on the effective light pair, the camera calibration parameters and coordinate transformation are used to obtain the target position in the world coordinate system.

[0077] (7) Millimeter-wave radar data acquisition: This process should be carried out simultaneously with camera data acquisition.

[0078] (8) Target point preprocessing: Obtain the target position and velocity information output by the millimeter-wave radar, remove interference noise points, and screen out valid target points.

[0079] (9) Target matching: Based on the target position in step (6), data is associated with the effective target points of the millimeter-wave radar within a certain range using matching criteria.

[0080] (10) Target tracking: Use the Kalman filter algorithm to track the target point matched in step (9) to obtain the final target vehicle position, speed and state information as the target vehicle information.

[0081] like Figure 4 As shown, the specific implementation process of the above step (2) is as follows:

[0082] a. Input RGB image: The input image is each frame of image data collected by the camera, and the image data is in RGB image format.

[0083] b. Extract highlight areas using adaptive thresholds: Convert image data in RGB format into grayscale images, and calculate the mean and standard deviation of the grayscale images. Obtain an adaptive threshold based on the mean and standard deviation, and use the threshold to binarize the image. The portion greater than the threshold is set as a highlight area, and the portion less than the threshold is set as a non-highlight area. Thus, the highlight area of ​​the image is extracted using the adaptive threshold method.

[0084] c. Extract the red and yellow areas based on the HSV channel according to the statistical data: Convert the original RGB format image data into HSV format. Perform statistics on the HSV image of the headlights. Since the color of the headlights at night is not uniform, it is mainly distributed in the color range of yellow, purple, and red. Based on the color distribution of the headlights, process the HSV image and set the H channel threshold. The H channel values ​​within the threshold range are retained, and the H values ​​outside the range are directly set to 0. After the HSV image is processed, it is converted into a grayscale image, and then the red channel is extracted from the grayscale image. The above-mentioned adaptive threshold method is used to extract the highlight area of ​​the red channel.

[0085] d. Take the intersection and get the binary image: According to the above image processing results, take the intersection of the two images. That is, the highlighted areas of the two binary images retain the highlighted features, otherwise they are set as non-highlighted areas.

[0086] e. Closing operation, median filtering: The image is further closed and median filtered, and the final output is a processed image containing only the region of interest, that is, the output preprocessing result.

[0087] like Figure 5 As shown, the specific implementation process of the above step (3) is as follows:

[0088] a. Input preprocessed image.

[0089] b. Find a connected area of ​​appropriate size: limit the effective search area to the upper half of the image, and remove the front part of the vehicle at the lower edge. The front of the vehicle often has reflections from the headlights of the preceding vehicle, which can cause false detection, so it also needs to be filtered out. The front of the vehicle is about 0.18 times the height of the image. Search for connected domains in the effective area, only save areas of 100 to 1000 pixels in size, and filter out areas that are too large or too small.

[0090] c. Using multiple geometric constraints, find the matching probability of the two headlight regions: For each connected region, calculate the matching probability of other connected regions within 200 pixels of its center. The probability calculation process is as follows: Figure 6 As shown. Set the initial probability value to 0, and calculate the parameters of connected domain 1 and connected domain 2, including horizontal distance, vertical distance, overlap, height similarity, width similarity, body-to-light width ratio, and body-to-light height ratio. Use the statistical empirical threshold to increase the probability (equivalent to a positive effect) of the lights whose parameters fall within the valid range (expressed as range 1), and reduce the probability (equivalent to a negative effect) of lights that do not meet the valid range (falling in range 2). Consider the constraints of multiple geometric parameters and get the final matching probability.

[0091] Its parameters, ranges and functional relationships are shown in Table 1:

[0092] Table 1

[0093]

[0094] Among them, "+" means increasing the probability, and "-" means decreasing the probability.

[0095] d. Find the headlight pair with the highest matching probability: Find other headlights with the highest matching probability for each headlight, and ensure that this probability value is greater than 1, then the match is successful, otherwise the match fails.

[0096] e. Remove duplicate lights: If the vehicles in adjacent lanes are too close, it is very likely that the matched lights will contain the same light in two pairs. In this case, remove duplicates. Keep the pair of lights with a larger matching probability value and remove the pair of lights with a smaller probability to increase the detection accuracy. Finally, output the matched light pairs. Step (3) can also use other similar solutions such as the symmetry of the lights and the size of the lights to match the light pairs.

[0097] like Figure 7 As shown, the specific implementation process of the above step (4) is as follows:

[0098] a. Cache 3 frames: cache the pixel position of the target point for 3 frames.

[0099] b. Inter-frame association.

[0100] c. The predicted value at time K is obtained through the optimal estimate at time K-1: the predicted value is obtained through Kalman filtering.

[0101] d. Target matching: Use Kalman filtering to obtain the predicted value of the current frame and match it with the observed value of the current frame for target matching.

[0102] e. Update status: If the match is successful, the status of the target point is updated and confirmed as a tracked target.

[0103] f. If the target is not matched successfully, determine whether it is a new target.

[0104] g. If it is a new target, create a new tracking target.

[0105] h. If not, the number of lost frames increases by 1.

[0106] i. If the number of lost frames is greater than the maximum acceptable number of lost frames, the target is deleted. If it is not greater than, the predicted value is output as the optimal value.

[0107] For the target points obtained by tracking, the lights are further divided into brake lights and tail lights according to the brightness of the lights and the threshold value obtained by statistics. The vehicle status information is obtained, that is, whether it is in the braking state or the normal driving state; in actual use, the solution of light classification can also be realized by other methods to realize the light status judgment.

[0108] The specific implementation process of the above step (6) is as follows: Based on the pixel positions of the headlight pair, the pixel positions of the symmetric points of the two headlights can be calculated. The vehicle and the ground are segmented, and the symmetric points are projected onto the pixel positions of the ground (such as Figure 8 (as shown in the figure at point A).

[0109] According to the center axis position of the vehicle, the pixel position from point A to the center axis can be calculated. The pixel position of AB is p(AB), and the pixel position of bc is p(BC).

[0110] According to the camera calibration model, p(AB) and p(BC) can be converted into actual distances S(AB) and S(BC).

[0111] The blind spot distance D3 of the camera of the ego vehicle and the distance D2 from the camera to the front end of the ego vehicle can be measured. Therefore, the relative position of the front vehicle relative to the front end of the ego vehicle can be obtained, where the longitudinal distance is D4 = S (BC) + D3-D2, and the lateral distance is D5 = S (AB). Similarly, the lateral distance and longitudinal distance of other vehicles relative to the ego vehicle can be obtained. With the driving direction of the ego vehicle as the X-axis, the coordinates of the front vehicle are (D4, D5).

[0112] The specific implementation process of the above step (8) is as follows: obtain the target position and speed information output by the millimeter wave radar, remove the interference noise points, and filter out the valid target point position to obtain the vehicle distance information. According to the target position in step (4), data is associated with the millimeter wave radar target point within a certain range using the matching criterion.

[0113] The radar has a wide detection range and will naturally obtain more target information, so it is necessary to divide the radar into effective target areas. When the vehicle is driving, the area with a width of one lane on the left and right, centered on the current expected driving trajectory of the vehicle, can be regarded as the effective target area of ​​the radar. Targets in the left and right adjacent lanes of the vehicle and targets in the same lane are the objects to be detected.

[0114] Filter out invalid targets and empty signals detected by radar, so as to determine the effective target point position of radar.

[0115] The specific implementation process of the above step (9) is as follows: After the vehicle light detection result is transformed into coordinates, the position of the target in the world coordinate system will have a large error, so it is necessary to perform data fusion with the millimeter wave target point. The method is as follows Fig. 9 As shown, for each target vehicle position (x0, y0) obtained in step (4), the search range is [x0-5 m, x0+5m] and [y0-2.5m, y0+2.5m] for the millimeter wave radar target point in the rectangular box area. The 5m and 2.5m here are taken from the maximum vehicle length and width.

[0116] All radar points in the area are compared in pairs to determine whether they are points of the same target vehicle. Suppose the two radar coordinate points to be compared are (x1, y1) and (x2, y2), and the comparison criteria are as follows: Fig.10 As shown in FIG, when the horizontal distance or the vertical distance is greater than the threshold, they are considered to be different targets, otherwise they are the same target and the target point with a closer distance is retained.

[0117] The tracking method of step (10) is consistent with that of step (4) and will not be described in detail. The difference is that in step (10), the target point position after the fusion of the millimeter wave radar target point and the headlight detection target point is tracked in the world coordinate system. Finally, more accurate target vehicle position, speed and status information are output.

[0118] The above method applies an adaptive threshold algorithm to the grayscale image to extract highlight areas, which can better adapt to the environment and has high robustness; and based on the HSV channel and the statistical information of the headlight color, it extracts areas that may be the headlight color, not just the red area, covering more headlights of different colors and improving the accuracy of headlight position extraction; based on the geometric constraints, the matching probability of two possible headlight areas is calculated, and the probability value is used to remove duplicate headlights. The headlight detection algorithm based on matching probability calculation and deduplication operation adopts multiple geometric constraints and has a low false detection rate and headlight positioning accuracy; after determining the valid headlight pair, it is further classified as brake lights or tail lights according to the brightness value of the headlights, and the status of the target vehicle can be obtained; the data fusion of the headlight detection target and the millimeter-wave radar target can better assist the judgment of the self-driving car; according to the target position of the headlight detection, data association is performed with the millimeter-wave radar target point within a certain range, which improves the accuracy of target vehicle positioning and makes up for the shortcomings of the existing methods.

[0119] Corresponding to the above method embodiment, the embodiment of the present invention also provides a night vehicle detection device integrating radar and vehicle light detection, such as Fig.11 As shown, the device comprises:

[0120] The data acquisition module 1100 is used to acquire the image to be processed and the corresponding radar data;

[0121] An image processing module 1102 is used to process the image to be processed based on an adaptive threshold algorithm and a vehicle light color threshold range to obtain a first image containing vehicle light information;

[0122] A vehicle light information determination module 1104, configured to determine vehicle light information according to the first image and preset constraints;

[0123] The vehicle information determination module 1106 is used to determine the target vehicle information based on the vehicle light information and radar data.

[0124] The night vehicle detection device integrating radar and headlight detection provided in the embodiment of the present invention has the same technical features as the night vehicle detection method integrating radar and headlight detection provided in the above-mentioned embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0125] The embodiment of the present invention further provides an electronic device, see Fig.12 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the night vehicle detection method integrating radar and vehicle light detection.

[0126] Further, Fig.12The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0127] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0128] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0129] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the night vehicle detection method that integrates the above-mentioned radar and vehicle light detection. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0130] The computer program product of the night vehicle detection method, device and electronic device that integrates radar and headlight detection provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0131] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a gateway electronic device, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nighttime vehicle detection method integrating radar and vehicle light detection, characterized in that: include: Obtain the image to be processed and the corresponding radar data; Processing the image to be processed based on an adaptive threshold algorithm and a pre-acquired vehicle light color threshold range to obtain a first image containing vehicle light information; Determining vehicle light information according to the first image and preset constraints; Determining target vehicle information according to the vehicle light information and the radar data; The step of determining the vehicle light information according to the first image and the preset constraint conditions comprises: Based on the shooting conditions of the image to be processed, obtaining a valid area of ​​the first image; Searching for a connected domain in the valid area; the connected domain includes a plurality of connected domains; Determine a valid connected domain pair according to the center distance between any two connected domains; the valid connected domain pair includes two connected domains; Calculating parameter values ​​corresponding to preset parameters of the effective connected domain pair; If the parameters satisfy the preset constraint conditions, determining that the valid connected domain pair includes vehicle light information; Based on the effective connected domain, obtain vehicle light information; the vehicle light information includes vehicle light position and vehicle light brightness; The vehicle light position includes a left vehicle light position and a right vehicle light position; The step of determining target vehicle information according to the vehicle light information and the radar data comprises: The left headlight position and the middle position of the line connecting the headlight positions are determined as the target pixel point; Tracking the target pixel point according to the predicted position obtained in advance to determine the effective vehicle light position; Determine the initial position of the target vehicle according to the effective vehicle light position and a preset coordinate transformation model; Determine the effective target point position according to the radar data; Associating the preliminary position of the target vehicle with the position of the valid target point to obtain a matched target point; Tracking the target point to obtain the position and speed of the target vehicle; Determine the state of the target vehicle according to the brightness of the vehicle lights; the state of the target vehicle includes normal driving or braking; The position, speed and state of the target vehicle are determined as target vehicle information.

2. The method according to claim 1, characterized in that: The step of processing the image to be processed based on an adaptive threshold algorithm and a pre-acquired vehicle light color threshold range to obtain a first image containing vehicle light information includes: Processing the image to be processed based on the adaptive threshold algorithm to obtain a first pre-processed image including a first highlight area; Processing the image to be processed based on a pre-acquired vehicle light color threshold range to obtain a second pre-processed image including a second highlight area; Determining a third preprocessed image according to the first preprocessed image and the second preprocessed image; The third pre-processed image is subjected to a closing operation and a median filter process to obtain a first image containing vehicle light information.

3. The method according to claim 2, characterized in that The image format of the image to be processed includes RGB format; The step of processing the image to be processed based on the adaptive threshold algorithm to obtain a first pre-processed image including a first highlight area includes: Converting the image format of the image to be processed to obtain a first grayscale image; Calculating the mean and standard deviation of the first grayscale image; Get an adaptive threshold based on the mean and standard deviation; The first grayscale image is binarized based on the adaptive threshold to obtain a first preprocessed image including a first highlight area.

4. The method according to claim 3, characterized in that The step of processing the image to be processed based on the vehicle light color threshold range to obtain a second pre-processed image including a second highlight area comprises: Converting the image format of the image to be processed to obtain a first preliminary image in HSV format; Based on the pre-acquired vehicle light color threshold range, processing the H channel of the first preliminary image to obtain a second preliminary image; Converting the image format of the second preliminary image to obtain a second grayscale image; Extracting the red channel of the second grayscale image to obtain a third preliminary image; The third preliminary image is binarized based on the adaptive threshold to obtain a second pre-processed image including a second highlight area.

5. The method according to claim 2, characterized in that: The step of determining a third preprocessed image according to the first preprocessed image and the second preprocessed image comprises: For each second pixel in the second highlight area, determine whether a first pixel corresponding to the second pixel belongs to the first highlight area; the first pixel belongs to the first pre-processed image; and the second pixel belongs to the second pre-processed image; If the corresponding first pixel belongs to the first highlight area, the grayscale value of the second pixel is set to 255; If the corresponding first pixel does not belong to the first highlight area, setting the grayscale value of the second pixel to 0; The second pixel and an image consisting of other pixels in the second preprocessed image except the second pixel are determined as a third preprocessed image.

6. A night vehicle detection device integrating radar and vehicle light detection, characterized in that: include: A data acquisition module, used to acquire the image to be processed and the corresponding radar data; An image processing module, used for processing the image to be processed based on an adaptive threshold algorithm and a vehicle light color threshold range to obtain a first image containing vehicle light information; A vehicle light information determination module, configured to determine the vehicle light information according to the first image and preset constraints; A vehicle information determination module, used to determine target vehicle information based on the vehicle light information and the radar data; A vehicle light information determination module is used to obtain a valid area of ​​the first image based on the shooting conditions of the image to be processed; search for a connected domain in the valid area; the connected domain includes a plurality of connected domains; determine a valid connected domain pair according to the center distance between any two connected domains; the valid connected domain pair includes two connected domains; calculate parameter values ​​corresponding to preset parameters of the valid connected domain pair; if the parameters meet preset constraints, determine that the valid connected domain pair includes vehicle light information; obtain vehicle light information based on the valid connected domain; the vehicle light information includes vehicle light position and vehicle light brightness; The vehicle light position includes a left vehicle light position and a right vehicle light position; The vehicle information determination module is used to determine the left headlight position and the middle position of the line connected by the headlight positions as the target pixel point; track the target pixel point according to the predicted position obtained in advance to determine the effective headlight position; determine the preliminary position of the target vehicle according to the effective headlight position and a preset coordinate conversion model; determine the effective target point position according to the radar data; perform data association between the preliminary position of the target vehicle and the effective target point position to obtain a matching target point; The target point is tracked to obtain the position and speed of the target vehicle; the state of the target vehicle is determined according to the brightness of the headlights; the state of the target vehicle includes normal driving or braking; the position, speed and state of the target vehicle are determined as the target vehicle information.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images