Driving assistance system data visualization method

Abstract

The invention relates to the technical field of vehicle advanced driving assistance systems, and particularly discloses a driving assistance system data visualization method, which comprises the following steps of: receiving and analyzing a sensing signal required by a vehicle CAN (Controller Area Network) bus by adopting a data receiving and analyzing module; the coordinate conversion module converts the coordinate information of the lane lines and the obstacles from vehicle coordinates to pixel coordinates; the data marking module marks the sensor sensing data in a video image so as to compare and observe the distance error between the sensor detection data and the real object; according to the driving assistance system data visualization method designed by the invention, the sensor sensing data is marked in the video through coordinate conversion, and the sensor sensing data and the real object in the video are correspondingly displayed, so that the sensing recognition effect can be intuitively observed, and driving assistance product development and testing personnel can more easily understand and sense the signal content; the data observation efficiency is effectively improved, problems of products can be found more easily, and the product quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of advanced vehicle driver assistance systems, and in particular to a driver assistance system data visualization method. Background Art

[0002] At present, the vehicle's advanced driving assistance system collects camera, radar and video signals simultaneously, and observes the signal curve and compares it with the video image to observe whether the sensor recognition effect is normal, such as Figure 1 As shown in the figure, the process is as follows: 1) Receive and parse the target information detected by the camera and radar on the CAN bus; 2) Observe the sensor detection data of some targets through the data curve; 3) Compare the data curve with the synchronized video to evaluate the sensor detection effect.

[0003] The sensors of the vehicle driving assistance system perceive a large amount of information. If we only observe the position information of the target ahead detected by the sensor, the number of signals may be dozens or even hundreds. Using this solution to observe the signals is not intuitive enough, and it is difficult to associate and understand the curve signals with the actual targets, which is not convenient for real-time observation of the sensor perception performance.

[0004] Therefore, there is an urgent need to design a data visualization method for a driving assistance system to solve the problems of large amounts of sensor perception data and non-intuitive observations in the current development process of vehicle driving assistance systems. Summary of the invention

[0005] In view of the problems existing in the prior art, an object of the present invention is to provide a driving assistance system data visualization method.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a driving assistance system data visualization method, comprising the following steps: S1. Data reception and analysis: The data reception and analysis module receives calibrated sensor perception data from the camera and millimeter-wave radar CAN network, analyzes the lane line information detected by the camera, the position information of the vehicle in front, and the target information in front detected by the millimeter-wave radar, and receives the camera video signal for image marking processing, and receives and processes the key information that needs to be displayed on the video; S2, coordinate conversion: The coordinate conversion module obtains the parameter information of the camera focal length, height from the ground, pitch angle, and image size, calculates and performs coordinate conversion, and obtains the specific position of the lane curve and the perceived target position in the pixel coordinate system; S3. Data marking: The data marking module is responsible for marking lane lines, camera detection targets, and radar detection targets on the video image in the form of curves, boxes whose size changes with distance, and dots, and displaying them in correspondence with the real objects in the video. It also marks the key information that needs to be observed at the appropriate position in the video, and compares the real-time gap between the sensor detection data and the real objects in the video.

[0007] Preferably, the camera and millimeter-wave radar in S1 detect the forward perception information through internal algorithms and send it to the CAN bus. The processing unit is connected to the camera and millimeter-wave radar through a CAN bus device, and the CAN message signal is parsed in the processing unit to obtain lane line parameters, camera target position information, and radar target position information.

[0008] Preferably, the camera target position information or radar target position information includes the longitudinal distance and lateral distance of the target relative to the origin of the vehicle coordinate system, and the lane line parameters include the lateral distance c0 between the origin of the vehicle coordinate system and the lane line, the tangent value c1 of the angle between the lane line and the center line of the longitudinal axis of the vehicle, the lane line curvature c2, the lane line curvature change rate c3, and the lane line length. Let x be the longitudinal distance, y(x) be the lateral distance of the lane line at the longitudinal distance x, then the lane line parameter formula y(x) can be expressed as: y(x)=c0+c1×x+1 / 2×c2×x^2+1 / 6×c3×x^3.

[0009] Preferably, the reference coordinate system of the perception information output by the calibrated sensor in the coordinate conversion in S2 is the vehicle coordinate system, whose origin is the center of the front face of the vehicle. During the coordinate conversion process, it is necessary to first convert the perception information in the vehicle coordinate system into camera coordinates, and then convert it from camera coordinates into pixel coordinates, mark the perception information in pixel coordinates, and convert the coordinate units of the perception information in the vehicle coordinate system into millimeters to keep consistent with the camera coordinate system.

[0010] Preferably, the coordinates of the target in the coordinate conversion process in the vehicle coordinate system are (Obj_x, Obj_y, Obj_z), Obj_x is the longitudinal distance, Obj_y is the lateral distance, Obj_z is the height from the ground, and the coordinates in the camera coordinate system are (Obj_cam_x, Obj_cam_y, Obj_cam_z), the camera height is cam_height, and the camera pitch angle is cam_theta. Then, the target position in the camera coordinate system is expressed as: Obj_cam_x=-Obj_x×sin(-cam_theta)+ cam_height×cos(cam_theta)- Obj_z× cos(cam_theta); Obj_cam_y=-Obj_y; Obj_cam_z= Obj_x×cos(cam_theta) + cam_height×sin(-cam_theta); After converting the target coordinates to camera coordinates, you also need to convert the camera coordinates to image coordinates and pixel coordinates.

[0011] Preferably, in the coordinate conversion process, the coordinates of the target point in the pixel coordinate system are (px_y, pic_x), the coordinates in the image coordinate system are (pic_u, pic_v), the image width is pic_w, the image width is pic_h, the camera focal length is cam_f, the horizontal scaling factor is α, and the vertical scaling factor is β, then the image coordinates of the target are expressed as: px_y=α×cam_f×Obj_cam_y / Obj_cam_z; px_x=β×cam_f×Obj_cam_x / Obj_cam_z; Pixel coordinates are expressed as: pic_u = px_y + 0.5 × pic_w; pic_v = px_x +0.5×pic_h.

[0012] Preferably, the data marking in S3 marks the processed lane line signal, camera detection target signal, and radar detection target signal in a pixel coordinate system, and the data marking is divided into lane line marking, camera detection target marking, and radar detection target marking.

[0013] Preferably, the lane line marking discretizes the curve into a finite number of points, and then marks these points within the lane line length on the image after coordinate transformation. The camera detects targets including vehicles and pedestrians, and uses a box to mark the entire target object. The radar detects targets using dots.

[0014] The present invention has the following beneficial effects: The driving assistance system data visualization method designed by the present invention can intuitively observe the perception and recognition effect by marking the sensor perception data in the video after coordinate conversion, and displaying the data corresponding to the real object in the video, so that driving assistance product development and testing personnel can more easily understand the perception signal content, effectively improve data observation efficiency, and more easily discover product problems and improve product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the solution in the prior art.

[0016] Figure 2It is a flow chart of the scheme of the data visualization method of the driver assistance system.

[0017] Figure 3 It is a structural diagram of the vehicle coordinate system.

[0018] Figure 4 It is a schematic diagram of the structure of the camera coordinate system.

[0019] Figure 5 It is a schematic diagram of the conversion structure between the camera coordinate system and the image coordinate system.

[0020] Figure 6 It is a schematic diagram of the conversion structure between the image coordinate system and the pixel coordinate system. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely further describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0022] The key terms of the present invention are described as follows: Driver assistance system: A general term for various systems that use sensors, communications, decision-making and execution devices installed on the vehicle to monitor the driver, vehicle and its driving environment and assist the driver in performing driving tasks or actively avoiding / mitigating collision hazards through images, lights, sounds, tactile prompts / warnings or controls.

[0023] Smart camera: An on-board camera that uses software algorithms to detect lane lines ahead and the position and speed of vehicles and pedestrians.

[0024] Millimeter-wave radar: 77G vehicle-mounted millimeter-wave radar that detects the position and speed of obstacles ahead through software algorithms.

[0025] CAN bus: A serial data communication protocol for data exchange between control and test instruments in a car.

[0026] like Figure 1-Figure 6 As shown, a method for visualizing data of a driving assistance system includes a data receiving and parsing module, a coordinate conversion module and a data marking module. The data receiving and parsing module receives calibrated sensor perception data on the camera and millimeter-wave radar CAN network, and parses the lane line information detected by the camera, the position information of the vehicle in front and the target information in front detected by the millimeter-wave radar, and receives the camera video signal for image marking processing, and receives and processes the key information that needs to be displayed on the video (such as real-time vehicle speed, system operation status and other signals).

[0027] The coordinate conversion module calculates and performs coordinate conversion by obtaining parameter information such as camera focal length, height from the ground, pitch angle, and image size. The reference coordinate system of the perception information output by the calibrated sensor is the vehicle coordinate system, and its origin is the center of the vehicle's front face. During the coordinate conversion process, the perception information in the vehicle coordinate system must first be converted into camera coordinates, and then converted from camera coordinates into pixel coordinates. The perception information is marked in pixel coordinates to obtain the specific positions of the lane line curve and the perception target position in the pixel coordinate system.

[0028] The data marking module is responsible for marking lane lines, camera detection targets, and radar detection targets on the video image in the form of curves, boxes whose size changes with distance, and dots, and displaying them in correspondence with the real objects in the video. It also marks the key information that needs to be observed at the appropriate position in the video, and compares the real-time gap between the observation sensor detection data and the real objects in the video.

[0029] A method for visualizing data of a driving assistance system, comprising the following steps: 1. Data reception and analysis: The camera and millimeter-wave radar detect the forward perception information through internal algorithms and send it to the CAN bus. The processing unit is connected to the camera and millimeter-wave radar through the CAN bus device, and the CAN message signal is analyzed in the processing unit to obtain the lane line parameters, camera target position information, and radar target position information.

[0030] The camera target position information or radar target position information includes the longitudinal distance and lateral distance of the target relative to the origin of the vehicle coordinate system. The lane line parameters include the lateral distance c0 between the origin of the vehicle coordinate system and the lane line, the tangent value c1 of the angle between the lane line and the center line of the vehicle longitudinal axis, the lane line curvature c2, the lane line curvature change rate c3, and the lane line length. Let x be the longitudinal distance, y(x) be the lateral distance of the lane line at the longitudinal distance x, then the lane line parameter formula y(x) can be expressed as: y(x)=c0+c1×x+1 / 2×c2×x^2+1 / 6×c3×x^3.

[0031] 2. Coordinate conversion: First, convert the coordinate units of the perception information in the vehicle coordinate system into millimeters to keep it consistent with the camera coordinate system.

[0032] During the coordinate conversion process, the coordinates of the target in the vehicle coordinate system are (Obj_x, Obj_y, Obj_z), Obj_x is the longitudinal distance, Obj_y is the lateral distance, Obj_z is the height from the ground, the coordinates in the camera coordinate system are (Obj_cam_x, Obj_cam_y, Obj_cam_z), the camera height is cam_height, and the camera pitch angle is cam_theta. The target position in the camera coordinate system can be expressed as: Obj_cam_x=-Obj_x×sin(-cam_theta)+ cam_height×cos(cam_theta)- Obj_z× cos(cam_theta); Obj_cam_y=-Obj_y; Obj_cam_z= Obj_x×cos(cam_theta) + cam_height×sin(-cam_theta); After converting the target coordinates to camera coordinates, you also need to convert the camera coordinates to image coordinates and pixel coordinates.

[0033] In the coordinate conversion process, the coordinates of the target point in the pixel coordinate system are (px_y, pic_x), the coordinates in the image coordinate system are (pic_u, pic_v), the image width is pic_w, the image width is pic_h, the camera focal length is cam_f, the horizontal scaling factor is α, and the vertical scaling factor is β. The image coordinates of the target can be expressed as: px_y=α×cam_f×Obj_cam_y / Obj_cam_z; px_x=β×cam_f×Obj_cam_x / Obj_cam_z; Pixel coordinates can be expressed as: pic_u = px_y + 0.5 × pic_w; pic_v = px_x +0.5×pic_h.

[0034] 3. Data marking: Data marking marks the processed lane line signals, camera detection target signals, and radar detection target signals in the pixel coordinate system. Data marking can be divided into lane line marking, camera detection target marking, and radar detection target marking.

[0035] Lane marking discretizes the curve into a finite number of points, and then marks these points within the length of the lane line on the image after coordinate conversion. Since the camera detection targets only include vehicles and pedestrians, the targets are relatively sparse, so it is suitable to use a box to mark the entire target. Radar detection is more sensitive to metal targets, and radar detection targets are relatively dense, so dot marks are used to distinguish them from camera detection targets. For key signals such as real-time vehicle speed signals and working status, they can be displayed synchronously in the video to observe the system operation status in real time.

[0036] The present invention is not limited to the above-mentioned implementation modes, and anyone should be aware of the structural changes made under the enlightenment of the present invention, and all those having the same or similar technical solutions as the present invention fall within the protection scope of the present invention.

[0037] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A method for visualizing data of a driving assistance system, characterized in that: The following steps are involved: S1. Data reception and analysis: The data reception and analysis module receives the calibrated sensor perception data on the camera and millimeter-wave radar CAN network, analyzes the lane line information detected by the camera, the position information of the vehicle in front, and the target information in front detected by the millimeter-wave radar, and receives the camera video signal for image marking processing. At the same time, it receives and processes the key information that needs to be displayed on the video; The camera and millimeter-wave radar detect the forward perception information through internal algorithms and send it to the CAN bus. The processing unit is connected to the camera and millimeter-wave radar through the CAN bus device, and the CAN message signal is analyzed in the processing unit to obtain the lane line parameters, camera target position information, and radar target position information; The camera target position information or radar target position information includes the longitudinal distance and lateral distance of the target relative to the origin of the vehicle coordinate system. The lane line parameters include the lateral distance c0 between the origin of the vehicle coordinate system and the lane line, the tangent value c1 of the angle between the lane line and the center line of the vehicle longitudinal axis, the lane line curvature c2, the lane line curvature change rate c3, and the lane line length. Let x be the longitudinal distance, y(x) be the lateral distance of the lane line at the longitudinal distance x, then the lane line parameter formula y(x) can be expressed as: y(x)=c0+c1×x+1 / 2×c2×x^2+1 / 6×c3×x^3; S2. Coordinate conversion: The coordinate conversion module obtains the parameter information of the camera focal length, height from the ground, pitch angle, and image size, calculates and performs coordinate conversion, and obtains the specific position of the lane curve and the perceived target position in the pixel coordinate system; The reference coordinate system of the perception information output by the calibrated sensor is the vehicle coordinate system, and its origin is the center of the front face of the vehicle. In the coordinate conversion process, the perception information in the vehicle coordinate system needs to be converted into camera coordinates first, and then converted from camera coordinates into pixel coordinates, and the perception information is marked in pixel coordinates. First, the coordinate unit of the perception information in the vehicle coordinate system is converted into millimeters to keep consistent with the camera coordinate system; S3. Data marking: The data marking module is responsible for marking lane lines, camera detection targets, and radar detection targets on the video image in the form of curves, boxes whose size changes with distance, and dots, and displaying them in correspondence with the real objects in the video. It also marks the key information that needs to be observed at the appropriate position in the video, and compares the real-time gap between the sensor detection data and the real objects in the video.

2. The method for visualizing driving assistance system data according to claim 1, characterized in that: The coordinates of the target in the vehicle coordinate system during the coordinate conversion process are (Obj_x, Obj_y, Obj_z), Obj_x is the longitudinal distance, Obj_y is the lateral distance, Obj_z is the height from the ground, and the coordinates in the camera coordinate system are (Obj_cam_x, Obj_cam_y, Obj_cam_z), the camera height is cam_height, and the camera pitch angle is cam_theta. The target position in the camera coordinate system is expressed as: Obj_cam_x=-Obj_x×sin(-cam_theta)+ cam_height×cos(cam_theta)- Obj_z×cos(cam_theta); Obj_cam_y=-Obj_y; Obj_cam_z= Obj_x× cos(cam_theta) + cam_height*×sin(-cam_theta); After converting the target coordinates to camera coordinates, you also need to convert the camera coordinates to image coordinates and pixel coordinates.

3. The method for visualizing driving assistance system data according to claim 2, characterized in that: The coordinates of the target point in the coordinate conversion process in the pixel coordinate system are (px_y, pic_x), the coordinates in the image coordinate system are (pic_u, pic_v), the image width is pic_w, the image width is pic_h, the camera focal length is cam_f, the horizontal scaling factor is α, and the vertical scaling factor is β. The image coordinates of the target are expressed as: px_y=α×cam_f×Obj_cam_y / Obj_cam_z; px_x=β×cam_f×Obj_cam_x / Obj_cam_z; Pixel coordinates are expressed as: pic_u = px_y + 0.5 × pic_w; pic_v = px_x +0.5×pic_h.

4. The method for visualizing driving assistance system data according to claim 1, characterized in that: The data marking in S3 marks the processed lane line signal, camera detection target signal, and radar detection target signal in a pixel coordinate system, and the data marking is divided into lane line marking, camera detection target marking, and radar detection target marking.

5. The method for visualizing driving assistance system data according to claim 4, characterized in that: The lane line marking discretizes the curve into a finite number of points, and marks these points within the lane line length on the image after coordinate transformation. The camera detects targets including vehicles and pedestrians, and uses a box to mark the entire target object. The radar detects targets using dots.

Images