A vehicle automatic emergency braking system and method based on depth images

Abstract

The application relates to the technical field of driving assistance, and discloses a vehicle automatic emergency braking system and method based on a depth image, a vehicle automatic emergency braking system based on a depth image, which comprises a depth camera, a steering torque sensor, a vehicle-mounted controller and a robot operating system. The application further discloses an implementation method of the vehicle automatic emergency braking system based on the depth image, which comprises the following steps: S1, receiving a color image and a depth image and aligning the images; S2, generating a depth mask according to a vehicle state; S3, performing image processing on the depth image; and S4, determining a target distance according to the processed depth image. The application can realize nonlinear change of a depth threshold line and multi-scene adaptation, can compensate for defects of an ordinary camera imaging principle, can realize accurate target object detection, can have good robustness for target distance detection, and can further realize precise emergency braking.

Description

Technical Field

[0001] This invention relates to the field of driver assistance technology, specifically to an automatic emergency braking system and method for vehicles based on depth images. Background Technology

[0002] Advanced Driver Assistance Systems (ADAS) are one of the important development directions in the automotive industry; and computer vision technology is one of the core technologies of ADAS systems. It can help vehicles identify obstacles such as traffic signs, vehicles, and pedestrians on the road, and generate high-precision maps and location information. In order to further improve the safety performance of ADAS systems, automakers are actively introducing Automatic Emergency Braking (AEB) functions. Using depth cameras to implement AEB functions can effectively improve the performance and accuracy of AEB systems, because depth cameras can provide more accurate depth information, can more accurately identify the distance and size of obstacles, and thus more precisely control the vehicle's braking system.

[0003] Depth cameras can detect the depth of field in a shooting space, which is the biggest difference from ordinary cameras. Ordinary color cameras can only capture and record all objects within the camera's field of view, but the recorded data does not include the distance of these objects from the camera. Semantic analysis of images can only make a simple judgment about which objects are far away and which are close, but it does not provide precise data. Depth cameras, on the other hand, solve this problem. With the data obtained by a depth camera, we can accurately know the distance of each point in the image from the camera. By adding the (x, y) coordinates of that point in the 2D image, we can obtain the three-dimensional spatial coordinates of each point in the image, enabling applications such as scene modeling.

[0004] However, current automatic emergency braking functions based on depth cameras simply utilize depth information and cannot accurately trigger braking based on different vehicle driving conditions, which significantly impacts the effectiveness of driver assistance.

[0005] A Chinese patent discloses an intelligent emergency braking control method based on vehicle-to-everything (V2X) (publication number CN104361760B). This patented technology ensures efficient and reliable safety warnings while using less communication cost and computing overhead, reducing the energy consumption of the on-board unit. It provides a simple and effective intelligent emergency braking control method for the vehicle-to-everything (V2X) auxiliary driving safety mechanism. However, its accuracy in recognizing road conditions is not high, and it cannot effectively achieve precise emergency braking. Summary of the Invention

[0006] The purpose of this invention is to provide a vehicle automatic emergency braking system and method based on depth images to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An automatic emergency braking system for vehicles based on depth images includes a depth camera, a steering torque sensor, an onboard controller, and a robot operating system. The depth camera, steering torque sensor, and onboard controller are all connected via a USB Type-C data cable. The onboard controller receives information from the depth camera and steering torque sensor and converts the input information into signals that the robot operating system can receive. The robot operating system receives signals from the onboard controller and processes and analyzes the signals.

[0009] A method for implementing an automatic emergency braking system for vehicles based on depth images includes the following steps:

[0010] S1: Receive and align color and depth images: First, use a depth camera to acquire the original color and depth images of the vehicle's driving direction. After processing by the onboard controller, the images are sent to the robot operating system. Then, the robot operating system aligns the depth images to the color images.

[0011] S2: Generate depth mask based on vehicle status: An adaptive depth determination method based on vehicle driving status monitors the vehicle's steering angle in real time through a steering torque sensor, and classifies the vehicle into three states: straight, left turn, and right turn, and generates corresponding depth masks.

[0012] S3: Image processing of the depth image: First, a depth image hole-filling method based on logical operations is used to pre-fill the black holes in the depth image before processing, resulting in a depth image with holes filled. Next, the Sobel operator is used to calculate the image gradient of the depth image, filtering out parts with subtle gradient changes. Then, the depth mask generated in step S2 is treated as a binary mask and a bitwise AND operation is performed between it and the depth image. In the resulting new image, only pixels with depth values ​​within a specified threshold range are retained, and all other pixels are removed, thus achieving depth mask coverage of the depth image.

[0013] S4: Determine the target distance based on the processed depth image: First, perform image opening operation on the depth image after covering the depth mask. Then, adaptively select the corresponding depth threshold according to the vehicle status to determine the target distance and trigger the automatic emergency braking function, thereby deciding whether to drive at the original speed or brake urgently to achieve emergency braking for obstacles at different distances.

[0014] As a further aspect of the present invention: in step S1, the method for aligning the depth image to the color image includes the following steps:

[0015] S11, Set the color pixel coordinates to pRGB (x, y), depth pixel coordinates are p d (x,y) converts the color pixel coordinates and depth pixel coordinates into color image coordinates p. RGB and depth image coordinates p d The calculation formula is as follows:

[0016] p RGB =M RGB p RGB (x,y) (1)

[0017] p d =M d p d (x,y) (2)

[0018] In equations (1) and (2) above, M RGB M is the intrinsic parameter matrix of the color image. d This is the intrinsic parameter matrix of the depth image;

[0019] S12. Associate depth image coordinates p using rotation matrix R and translation matrix T. d and color image coordinates p RGB The calculation formula is as follows:

[0020] p RGB =Rp d +T (3)

[0021] S13, via color image coordinates p RGB and depth image coordinates p d By calculating the corresponding point coordinates p in the world coordinate system, the depth image can be aligned to the color image; where the corresponding point coordinates p and p in the world coordinate system are... RGB and p d The formula for calculating the relationship between them is as follows:

[0022] p RGB =R RGB p+T RGB (4)

[0023] p d =R dp +T d (5)

[0024] In equations (4) and (5) above, R RGB and R d T represents the rotation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively; RGB and T d These are the translation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively.

[0025] As a further aspect of the present invention: in step S2, the method for generating the mask includes the following steps:

[0026] S21. Generate the corresponding center point according to the vehicle status, and divide the depth image into two regions accordingly. Use the front and side depth thresholds to calculate the corresponding scaling factors and establish the threshold curve.

[0027] S22. Model the threshold curve as two quadratic equations, and use the coordinates of the center point as the vertex of the quadratic function;

[0028] S23. Calculate the coefficients of the quadratic function based on the thresholds on the left and right sides and the center point threshold, that is, convert the threshold curve from a linear form to a non-linear form; after expanding the depth threshold line to an image of the same size as the depth image, the generation of the adaptive depth determination mask is completed.

[0029] As a further aspect of the present invention: in step S3, the logical operation method for filling in the holes in the depth image is as follows:

[0030] S31. Use logical operations to set the depth values ​​of all black holes to the farthest distance, i.e., the maximum depth value of the black holes. The formula for the logical operation is as follows:

[0031] T=S×([(S==0)+0]*MAX_DIST) (6)

[0032] In the above formula (6), S is the original depth image, T is the depth image, and MAX_DIST is the maximum depth value of the black hole;

[0033] S32. Determine the pixel values ​​in the original depth image S. If the pixel value is 0, set it to 0; if the pixel value is not 0, set it to 1. This will result in a mask M.

[0034] S33. Multiply the mask M by the maximum depth value MAX_DIST of the black hole to update the value in the mask M; finally, add the mask M to the original depth image S to obtain the depth image T with the hole filled in.

[0035] As a further aspect of the present invention: in step S3, when using the Sobel operator to calculate the image gradient, the convolution kernel size is set to 3*3, the gradient in the y-direction is solved, and the image depth is set to cv2.CV_64F.

[0036] As a further aspect of the present invention: in step S4, the method for determining the target distance and triggering the automatic emergency braking function is as follows:

[0037] S41. Extract the contour from the depth image after covering the depth mask, calculate the number of pixels in the contour, and if it meets the target definition, it is considered to be the target to be detected.

[0038] S42. Detect the minimum distance within the contour and determine the triggering of the automatic emergency braking function accordingly. If the minimum distance is less than the set depth threshold and within the threshold mask range, then perform emergency braking; if the minimum distance is less than the set depth threshold but not within the threshold mask range, then continue driving at the original speed; if the minimum distance is greater than the set depth threshold, then continue driving at the original speed.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] This invention monitors the vehicle's steering angle in real time, inputting different vehicle center points to divide the depth image into two regions. Using front and side depth thresholds, it calculates corresponding scaling factors to generate depth masks for different vehicle driving states, thereby achieving non-linear changes in the depth threshold line and multi-scene adaptation. Image logic operations are used to fill in holes in the depth image, compensating for the deficiencies of ordinary camera imaging principles and achieving accurate target object detection. The Sobel operator is used to calculate the gradient of the depth image, filtering out subtle gradient changes, providing good robustness for target distance detection, and ultimately enabling precise emergency braking. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a method for implementing an automatic emergency braking system for vehicles based on depth images.

[0042] Figure 2 This is a flowchart illustrating step S1 in a method for implementing an automatic emergency braking system for vehicles based on depth images.

[0043] Figure 3 This is a flowchart illustrating step S2 in a method for implementing an automatic emergency braking system for vehicles based on depth images.

[0044] Figure 4 This is a flowchart illustrating step S3 in a method for implementing an automatic emergency braking system for vehicles based on depth images.

[0045] Figure 5 This is a flowchart illustrating step S4 of a depth-image-based automatic emergency braking system and method for vehicles. Detailed Implementation

[0046] In this embodiment of the invention, a vehicle automatic emergency braking system based on depth images includes a depth camera, a steering torque sensor, an onboard controller, and a robot operating system. The depth camera, steering torque sensor, and onboard controller are all connected via a USB Type-C data cable. The onboard controller receives information input from the depth camera and steering torque sensor and converts the input information into signals that the robot operating system can receive. The robot operating system receives signals input from the onboard controller and processes and analyzes the signals.

[0047] Please see Figures 1-5 In this embodiment of the invention, a method for implementing a vehicle automatic emergency braking system based on depth images includes the following steps:

[0048] S1: Receive and align the color and depth images: e.g. Figure 2 As shown, firstly, the original color image and depth image of the vehicle's driving direction are acquired using a depth camera. After being processed by the onboard controller, they are sent to the robot operating system, which then aligns the depth image to the color image.

[0049] S2: Generate a depth mask based on the vehicle status: e.g. Figure 3 As shown, the adaptive depth determination method based on vehicle driving status monitors the vehicle's steering angle in real time through a steering torque sensor, and classifies the vehicle into three states: straight, left turn, and right turn, and generates corresponding depth masks.

[0050] S3: Image processing of depth images: such as... Figure 4 As shown, firstly, the depth image hole-filling method based on logical operations fills in the black holes in the depth image before processing it, resulting in a hole-filled depth image. Next, the Sobel operator is used to calculate the image gradient of the depth image, filtering out parts with subtle gradient changes. Then, the depth mask generated in step S2 is treated as a binary mask and a bitwise AND operation is performed between it and the depth image. In the resulting new image, only pixels with depth values ​​within a specified threshold range are retained, and all other pixels are removed, thus achieving depth mask coverage of the depth image.

[0051] S4: Determine the target distance based on the processed depth image: e.g. Figure 5 As shown, firstly, an image opening operation is performed on the depth image after the depth mask is covered. Then, the corresponding depth threshold is adaptively selected according to the vehicle status to determine the target distance and the triggering of the automatic emergency braking function, thereby deciding whether to drive at the original speed or to brake urgently, so as to realize emergency braking of obstacles at different distances.

[0052] Preferably, in step S1, the method for aligning the depth image to the color image includes the following steps:

[0053] S11, Set the color pixel coordinates to p RGB (x, y), depth pixel coordinates are p d (x,y) converts the color pixel coordinates and depth pixel coordinates into color image coordinates p. RGB and depth image coordinates p d The calculation formula is as follows:

[0054] p RGB =M RGB p RGB (x,y) (1)

[0055] p d =M d p d (x,y) (2)

[0056] In equations (1) and (2) above, M RGB M is the intrinsic parameter matrix of the color image. d This is the intrinsic parameter matrix of the depth image;

[0057] S12. Associate depth image coordinates p using rotation matrix R and translation matrix T. d and color image coordinates p RGB The calculation formula is as follows:

[0058] p RGB =Rp d +T (3)

[0059] S13, via color image coordinates p RGB and depth image coordinates p d By calculating the corresponding point coordinates p in the world coordinate system, the depth image can be aligned to the color image; where the corresponding point coordinates p and p in the world coordinate system are... RGB and p d The formula for calculating the relationship between them is as follows:

[0060] p RGB =R RGB p+T RGB (4)

[0061] p d =R d p+T d (5)

[0062] In equations (4) and (5) above, R RGB and R d T represents the rotation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively; RGB and T dThese are the translation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively.

[0063] Preferably, in step S2, the method for generating the mask includes the following steps:

[0064] S21. Generate the corresponding center point according to the vehicle status, and divide the depth image into two regions accordingly. Use the front and side depth thresholds to calculate the corresponding scaling factors and establish the threshold curve.

[0065] S22. Model the threshold curve as two quadratic equations, and use the coordinates of the center point as the vertex of the quadratic function;

[0066] S23. Calculate the coefficients of the quadratic function based on the thresholds on the left and right sides and the center point threshold, that is, convert the threshold curve from a linear form to a non-linear form; after expanding the depth threshold line to an image of the same size as the depth image, the generation of the adaptive depth determination mask is completed, thereby realizing the non-linear change of the depth threshold line and multi-scene adaptation.

[0067] Preferably, in step S3, the logical operation method for filling in the holes in the depth image is as follows:

[0068] S31. Due to the special imaging principle of depth cameras, the depth images acquired by depth cameras may contain black holes of varying sizes. To eliminate their influence on the results, logical operations are used to set the depth values ​​of all black holes to the farthest distance, i.e., the maximum depth value of the black holes. The formula for the logical operation is as follows:

[0069] T=S×([(S==0)+0]*MAX-DIST) (6)

[0070] In the above formula (6), S is the original depth image, T is the depth image, and MAX_DIST is the maximum depth value of the black hole;

[0071] S32. Determine the pixel values ​​in the original depth image S. If the pixel value is 0, set it to 0; if the pixel value is not 0, set it to 1. This will result in a mask M.

[0072] S33. Multiply the mask M by the maximum depth value MAX_DIST of the black hole to update the value in the mask M; finally, add the mask M to the original depth image S to obtain the depth image T with the hole filled in.

[0073] Preferably, in step S3, when using the Sobel operator to calculate the image gradient, the convolution kernel size is set to 3*3. In order to separate foreground and background targets, the gradient is only calculated in the y direction, and the image depth is set to cv2.CV_64F. This is because depth images are different from general 8-bit images, and there will be parts less than 0 when calculating the gradient. If it is set to uint8, the parts less than 0 will automatically become 0. Therefore, the image depth should be set to cv2.CV_64F.

[0074] Preferably, in step S4, the methods for determining the target distance and triggering the automatic emergency braking function are as follows:

[0075] S41. Extract the contour from the depth image after covering the depth mask, calculate the number of pixels in the contour, and if it meets the target definition, it is considered to be the target to be detected.

[0076] S42. Detect the minimum distance within the contour and determine the triggering of the automatic emergency braking function accordingly. If the minimum distance is less than the set depth threshold and within the threshold mask range, then perform emergency braking; if the minimum distance is less than the set depth threshold but not within the threshold mask range, then continue driving at the original speed; if the minimum distance is greater than the set depth threshold, then continue driving at the original speed.

[0077] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for implementing an automatic emergency braking system for vehicles based on depth images, characterized in that, Includes the following steps: S1: Receive and align color and depth images: First, use a depth camera to acquire the original color and depth images of the vehicle's driving direction. After processing by the onboard controller, the images are sent to the robot operating system. Then, the robot operating system aligns the depth images to the color images. S2: Generate depth mask based on vehicle status: An adaptive depth determination method based on vehicle driving status monitors the vehicle's steering angle in real time through a steering torque sensor, and classifies the vehicle into three states: straight, left turn, and right turn, and generates corresponding depth masks. S3: Image processing of the depth image: First, a depth image hole-filling method based on logical operations is used to pre-fill the black holes in the depth image before processing, resulting in a depth image with holes filled. Next, the Sobel operator is used to calculate the image gradient of the depth image, filtering out parts with subtle gradient changes. Then, the depth mask generated in step S2 is treated as a binary mask and a bitwise AND operation is performed between it and the depth image. In the resulting new image, only pixels with depth values ​​within a specified threshold range are retained, and all other pixels are removed, thus achieving depth mask coverage of the depth image. S4: Determine the target distance based on the processed depth image: First, perform image opening operation on the depth image after covering the depth mask. Then, adaptively select the corresponding depth threshold according to the vehicle status to determine the target distance and trigger the automatic emergency braking function, thereby deciding whether to drive at the original speed or brake urgently to achieve emergency braking for obstacles at different distances.

2. The method for implementing a vehicle automatic emergency braking system based on depth images according to claim 1, characterized in that, In step S1, the method for aligning a depth image to a color image includes the following steps: S11, Set the color pixel coordinates as Depth pixel coordinates are Convert color pixel coordinates and depth pixel coordinates into color image coordinates. and depth image coordinates The calculation formula is as follows: In equations (1) and (2) above, The intrinsic parameter matrix of the color image. This is the intrinsic parameter matrix of the depth image; S12, via rotation matrix Translation matrix Related and The calculation formula is as follows: S13, Through and Calculate the coordinates of the corresponding point in the world coordinate system. This allows for the alignment of a depth image to a color image; where the coordinates of the corresponding point in the world coordinate system are... and and The formula for calculating the relationship between them is as follows: In equations (4) and (5) above, and These are the rotation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively; and These are the translation matrices from the world coordinate system to the color image coordinate system and the depth image coordinate system, respectively.

3. The method for implementing a vehicle automatic emergency braking system based on depth images according to claim 1, characterized in that, In step S2, the method for generating the mask includes the following steps: S21. Generate the corresponding center point according to the vehicle status, and divide the depth image into two regions accordingly. Use the front and side depth thresholds to calculate the corresponding scaling factors and establish the threshold curve. S22. Model the threshold curve as two quadratic equations, and use the coordinates of the center point as the vertex of the quadratic function; S23. Calculate the coefficients of the quadratic function based on the thresholds on the left and right sides and the center point threshold, that is, convert the threshold curve from a linear form to a non-linear form; after expanding the depth threshold line to an image of the same size as the depth image, the generation of the adaptive depth determination mask is completed.

4. The method for implementing a vehicle automatic emergency braking system based on depth images according to claim 1, characterized in that, In step S3, the logical operation method for filling in the holes in the depth image is as follows: S31. Use logical operations to set the depth values ​​of all black holes to the farthest distance, i.e., the maximum depth value of the black holes. The formula for the logical operation is as follows: In the above formula (6), For the original depth image, For depth images, This represents the maximum depth of the black hole. S32. For the original depth image The pixel value is evaluated; if the pixel value is 0, it is set to 0; otherwise, it is set to 1, resulting in a mask. ; S33, Mask Maximum depth of the black hole Multiply and update the mask. The value in the middle; finally, the mask. Adding it to the original depth image yields a depth image with holes filled in. .

5. The method for implementing a vehicle automatic emergency braking system based on depth images according to claim 1, characterized in that, In step S3, using When the operator calculates the image gradient, the convolution kernel size is set to 3*3, and... The gradient is calculated in the direction and the image depth is set. .

6. The method for implementing a vehicle automatic emergency braking system based on depth images according to claim 1, characterized in that, In step S4, the methods for determining the target distance and triggering the automatic emergency braking function are as follows: S41. Extract the contour from the depth image after covering the depth mask, calculate the number of pixels in the contour, and if it meets the target definition, it is considered to be the target to be detected. S42. Detect the minimum distance within the contour and determine the triggering of the automatic emergency braking function accordingly. If the minimum distance is less than the set depth threshold and within the threshold mask range, then perform emergency braking; if the minimum distance is less than the set depth threshold but not within the threshold mask range, then continue driving at the original speed. If the minimum distance is greater than the set depth threshold, then continue at the original speed.

Images