Truck driving assistance method and system based on laser radar visualization technology

Abstract

The invention discloses a truck driving assistance method and system based on a laser radar visualization technology, and the method comprises the following steps: firstly, extracting a signal from a laser radar sensor installed on a vehicle; secondly, preprocessing the collected point cloud data to remove noise and optimize data quality; then, objects in the point cloud are classified based on reflectivity and distance information, and potential obstacles or other traffic participants are identified; then, danger early warning information for the driver is generated according to the classification result; in addition, a three-dimensional environment model is constructed, and the model and early warning information are presented to a driver in a visual form; and finally, important information is projected to a windshield by means of a head-up display technology, so that the driver can quickly and accurately obtain the surrounding environment condition, and meanwhile, the sight is kept not away from the front road. According to the method, the view limitation problem caused by the blind area can be reduced, and the safety of the truck in the running process can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation and assisted driving technology, and in particular to a driving assistance method and system that utilizes a laser radar (LiDAR) to obtain environmental information and displays point cloud data in a visual form in real time on a truck head-up display (HUD). Background Art

[0002] As traffic volume continues to grow, the incidence of truck accidents has also increased accordingly, which not only poses a threat to people's lives, but also brings huge economic losses to society. In order to improve the safety and efficiency of truck driving, driver assistance systems have gradually become a research hotspot. These systems provide drivers with real-time information feedback by integrating advanced sensor technology and display technology, helping them make more accurate judgments and operations.

[0003] LiDAR (Light Detection And Ranging) is a remote sensing technology that determines the distance of an object by emitting a laser beam and measuring the time difference of the reflected beam, thereby constructing a three-dimensional point cloud model of the surrounding environment. Compared with traditional radar, LiDAR has higher resolution and accuracy, and can accurately identify key information such as obstacles and pedestrians in complex and changing road environments. In recent years, with the reduction of costs and the advancement of technology, LiDAR has been increasingly used in the field of autonomous driving.

[0004] Three.js is a JavaScript 3D library based on WebGL, which greatly simplifies the process of creating 3D content. In this patent, Three.js is used to convert the data collected by the lidar into a 3D visualization interface that is easy to understand and operate in real time. Through Three.js, complex point cloud data can be rendered into intuitive images to help drivers better perceive the environmental conditions around the vehicle. In addition, Three.js also supports a variety of lighting effects and material settings, which can enhance the realism of visual effects, thereby further improving the user experience of the driver assistance system.

[0005] Head-Up Display (HUD) technology was first used in the aviation field and later gradually expanded to the automotive industry. HUD projects important driving information directly onto the windshield in front of the driver, allowing the driver to obtain key information such as speed and navigation without having to look down at the dashboard, which helps reduce distraction caused by line of sight and improve driving safety. In this patent, combined with the 3D visualization results of LiDAR, this information is presented to the driver in the most intuitive way through HUD technology, achieving instant driving assistance. Summary of the invention

[0006] Purpose of the invention: To solve the problem of the lack of driving assistance technology in traditional trucks, the present invention proposes a truck driving assistance method and system based on LiDAR visualization technology, aiming to improve the safety of truck driving by using advanced sensing technology and human-computer interaction. This method combines the data collected by LiDAR with Three.js for 3D visualization, and presents the relevant information to the driver through HUD technology, which can not only provide more accurate environmental perception, but also effectively reduce the driver's workload and reduce the occurrence of major traffic accidents.

[0007] Technical solution:

[0008] A truck driving assistance method based on laser radar visualization technology comprises the following steps:

[0009] Step 1: Extract the lidar signal.

[0010] The laser radar is installed on the front or around the vehicle body to collect 3D point cloud data of the surrounding environment. The laser radar is connected to the information processing unit of the driving assistance system. The laser radar scans the surrounding traffic environment in real time while the truck is driving and sends it to the information processing unit.

[0011] Step 2: Point cloud data preprocessing.

[0012] Furthermore, the processing unit preprocesses the received point cloud data, including extraction, denoising, filtering, coordinate conversion and other operations to ensure data quality.

[0013] Specifically, the preprocessing of point cloud data is a multi-step process aimed at improving the quality of the data for subsequent analysis or modeling. First, extraction refers to selecting representative parts from a large amount of point cloud data, which can be achieved through methods such as uniform sampling, grid sampling, or feature-based sampling. Secondly, denoising is to remove outliers or noise points in the point cloud. Common methods include statistical denoising, denoising based on local plane fitting, etc. Next, the filtering step is used to further clean up the data, such as removing unnecessary background information or separating specific objects. Common filtering techniques include voxel grid filtering, pass-through filtering, and conditional filtering. Finally, coordinate transformation is to align the point cloud data to a common coordinate system, which is particularly important for multi-source data fusion, and usually involves transformation operations such as translation and rotation. Through these preprocessing steps, the accuracy and availability of point cloud data can be significantly improved, laying a solid foundation for subsequent processing steps.

[0014] Step 3: Classify the point cloud based on its reflectivity and distance.

[0015] Furthermore, classification processing based on the reflectivity and distance of the lidar point cloud includes calculating the distance between each point and the truck, deleting points that are far away from the truck, dividing the point cloud into different instances according to the reflectivity, and using a 3D image target detection algorithm to confirm the category of each instance.

[0016] Furthermore, the distance between each point in the point cloud and the coordinate origin (truck) is calculated using the following formula:

[0017]

[0018] Among them, x is the horizontal coordinate of a point, and y is the vertical coordinate of the point.

[0019] Furthermore, points that are far from the center of the truck are eliminated. The scanning range of common laser radars on the market can range from tens of meters to hundreds of meters. Since the purpose of this method is to improve the truck driver's perception of the surrounding driving environment and in order to more effectively reduce the amount of radar point cloud data, this method will eliminate all points that exceed the distance threshold.

[0020] Furthermore, the points retained in the point cloud are divided into different instances according to their reflectivity. Reflectivity refers to the proportion of laser energy reflected by the surface of an object. Different materials and surface properties will result in different reflectivities. Knowing the reflectivity range of various objects can roughly distinguish different objects. The categories focused on in this method are pedestrians, vehicles, and road infrastructure.

[0021] The reflection ranges of various objects used in this method are shown in the following table.

[0022] Table 1 Reflectivity range of common objects

[0023] category Snow and ice water Metal coating human body Reflectivity range 0.85-0.95 0.02-0.05 0.7-0.9 0.8-0.9 0.1-0.3

[0024] According to the reflectivity and distance of each point in the point cloud, the K-Means clustering method can be used to roughly divide the instances into different categories, and then calculate and save the center coordinates of each instance. This algorithm has low requirements for computing power and is a necessary processing step in this method. The target detection algorithm in the next step is an enhancement of this step and has high requirements for computing power. If computing power is insufficient, the target detection algorithm can be skipped.

[0025] Furthermore, the 3D image target detection algorithm is used to confirm the category of each instance, and the detection categories are still pedestrians, vehicles and road infrastructure. This method uses target detection algorithms such as YOLOv11 to mark each instance with a box, and combines the center coordinates generated by clustering to generate more accurate instance center coordinates.

[0026] Step 4: Generate hazard warning information.

[0027] Furthermore, the danger information is warned according to the current speed of the truck. The information processing unit used in this method is not only connected to the laser radar installed on the vehicle body, but also connected to the central control system of the truck, so that the real-time speed information of the truck can be obtained in real time. Since the driver cannot act immediately after receiving the emergency stop signal, but it takes T1 seconds to realize that emergency braking should be performed and start moving the right foot, and then it takes T2 seconds to start stepping on the brake pedal, so the driver's reaction time T is:

[0028] T=T1+T2

[0029] Generally, the driver's reaction time is between 0.4s and 1s. For safety reasons, the time threshold t0 used in this method is 1s. Therefore, the calculation formula for the danger warning distance threshold d0 is:

[0030] d0=v*t0 / 3.6

[0031] Wherein, v represents the real-time speed of the truck (unit: km / h).

[0032] When the distance between an instance and the truck is less than or equal to the danger warning distance threshold, the information processing unit will generate danger warning information, including broadcasting warning audio through the truck speaker and the corresponding instance flashing continuously in the displayed 3D scene.

[0033] Step 5: Visualize the 3D scene.

[0034] Furthermore, Three.js is used to perform 3D visualization on the point cloud obtained from step 3, aiming to provide users with intuitive and accurate environmental perception. The following is a detailed technical implementation process:

[0035] 1.Three.js environment setup: Initialize the Three.js renderer, scene, and camera. Set appropriate renderer parameters, such as background color, size, etc., create a 3D scene object, and configure the camera's position and perspective to match the global observation perspective proposed in this method.

[0036] 2. Loading and converting point cloud data: Due to the large amount of point cloud data, it is necessary to efficiently convert this data into a geometric form that can be recognized by Three.js. You can store point cloud information by creating a THREE.Geometry or THREE.BufferGeometry object, in which the position, color and other properties of each point need to be set correctly.

[0037] 3. Point cloud rendering: Use the materials (such as THREE.PointsMaterial) and shaders provided by Three.js to define the appearance characteristics of the point cloud, such as point size, color, etc. Then, combine the geometry object with the material to form the final point cloud model, and add it to the scene for rendering. In order to highlight objects that are less than the danger warning distance threshold from the truck, a flashing effect can be achieved by changing the material properties of these specific points. This usually involves dynamically adjusting the color or transparency of these points when each frame is updated. This function can be achieved by listening to the update event in the scene, or using the animation loop of Three.js.

[0038] 4. Implementation of interactive functions: In order to enhance the user experience, a variety of interactive functions can be added to the 3D visualization interface, such as zooming, rotating, and panning, so that users can view point cloud data from different angles.

[0039] Step 6: HUD display screen.

[0040] Furthermore, the system will adjust the display parameters according to the light intensity in the environment to ensure the best visual effect and user experience. This process specifically includes the following aspects:

[0041] 1. Light detection: First, the system monitors the light level of the vehicle's surroundings in real time through built-in light sensors. These light sensors can be installed in different locations of the vehicle, such as behind the front windshield, to accurately capture the current lighting conditions in the environment.

[0042] 2. Brightness adjustment: Based on the detected light intensity data, the system automatically adjusts the display brightness of the HUD. In strong light environments, such as direct sunlight during the day, the HUD will increase its brightness output to ensure that the displayed content is clearly visible and not interfered with by external light. In low light conditions, such as when driving at night, the brightness will be appropriately reduced to avoid excessively bright displays that cause discomfort or distraction to the driver's vision.

[0043] 3. Contrast optimization: In addition to brightness, the system will dynamically adjust the contrast of the displayed content according to the lighting conditions. High contrast helps to highlight information in bright environments, while low contrast can provide a softer and more comfortable visual experience in darker environments.

[0044] 4. Color correction: In order to adapt to different lighting conditions, the system may also correct the displayed color. For example, during periods of time when the light changes greatly, such as dusk or dawn, proper color temperature adjustment can make the image look more natural and reduce visual fatigue.

[0045] 5. Content Adaptation: The system can choose the most appropriate type of information to display based on the current lighting conditions. For example, in very bright conditions, black and white line drawings or high-contrast text information may be displayed first; in darker environments, more colors and graphic elements can be used to enrich the visual presentation.

[0046] The present invention also provides a truck driving assistance system based on laser radar visualization technology, comprising a laser radar installed on a truck, an information processing unit, and a head-up display; the information processing unit runs a Three.js framework and communicates with the truck central control system through a CAN bus, the laser radar sends a laser radar point cloud to the information processing unit through a CAN bus or a USB interface, and the information processing unit outputs two image signals, one for display on the central control screen and the other for display on the head-up display;

[0047] The laser radar is used to collect three-dimensional point cloud data of the surrounding environment and send it to the information processing unit;

[0048] The information processing unit includes: pre-processing the 3D point cloud data; identifying, locating and marking each instance in the point cloud based on the reflectivity of each point in the point cloud; obtaining the real-time speed information of the truck from the central control system of the truck in real time to generate danger warning information; using Three.js to render the 3D point cloud processed in step 3, visualize the 3D point cloud into a 2D display screen, and send it to the head-up display;

[0049] The head-up display is used to project 2D display images and hazard warning information onto the windshield of the truck.

[0050] Beneficial effects: The present invention proposes a truck driving assistance method based on laser radar visualization technology. The method can reduce the front, back, left, and right blind spots of truck drivers based on environmental information obtained by multiple laser radars, and reduce the incidence of truck traffic accidents. The present invention uses reflectivity to identify objects, with more continuous edges and less computational complexity.

[0051] The present invention constructs a three-dimensional environmental model and presents the model and warning information to the driver in an intuitive form. It also projects important information onto the windshield with the help of head-up display technology, ensuring that the driver can quickly and accurately obtain the surrounding environment while keeping his eyes on the road ahead. This method not only helps to reduce the problem of limited vision caused by blind spots, but also effectively improves the safety of trucks during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flowchart of the method.

[0053] Figure 2This is a structural diagram of the truck driving assistance system provided in the implementation of the present invention, wherein 1 and 2 are body laser radars, 3 is the central control large screen, and 4 is the HUD display.

[0054] Figure 3 This is a schematic diagram of the HUD provided in the implementation of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further explained below in conjunction with the accompanying drawings and a specific embodiment.

[0056] like Figure 1 As shown, the method of the present invention comprises the following steps: Step 1, extracting laser radar signals. Step 2, preprocessing laser radar point cloud data. Step 3, classifying and processing based on reflectivity and distance of laser radar point cloud. Step 4, generating danger warning information. Step 5, visualizing 3D scene. Step 6, HUD display screen.

[0057] The structural diagram of the truck auxiliary driving system is as follows Figure 2 shown.

[0058] Example: Driving assistance system for a certain brand of semi-trailer truck

[0059] Step 1: Install a laser radar in the center of the roof of the truck compartment and connect it to the driver assistance system through the CAN bus. The scanning and transmission rates are both 10 Hz.

[0060] Step 2: Use the velodyne_driver package in ROS to drive the lidar and collect raw point cloud data. Then use the velodyne_pointcloud package to perform preliminary processing on the point cloud data, including denoising, filtering, and coordinate conversion to ensure the quality and accuracy of the point cloud data. Finally, use the pcl_ros package to further process the point cloud data, such as ground segmentation, cluster analysis, etc., to extract important environmental features such as pedestrians, vehicles, and road signs.

[0061] Step 3: Use Three.js to convert the processed point cloud data into a 3D model. Generate a 2D or 3D environment image based on the model, using different colors and icons to represent different object types. For example, pedestrians are represented by red circles, vehicles are represented by blue rectangles, and road signs are represented by green triangles. The generated image data is passed to the HUD display device through ROS messages.

[0062] Step 4: The truck driving assistance system obtains the real-time speed information of the truck from the central control system of the truck and calculates the danger warning distance threshold. If the distance between an instance in the point cloud and the truck is less than the threshold, the instance will flash on the display screen, and the closer the distance, the higher the flashing frequency. The speakers in the truck will also broadcast warning audio to remind the truck driver to make a timely decision.

[0063] Step 5: The HUD display device receives the image data from the information processing unit and projects it onto the front windshield. The projection position and size can be dynamically adjusted according to the driver's line of sight height and distance to ensure that the image is clearly visible and does not block the driver's field of vision. This system provides multiple display modes, such as day mode, night mode, and rain and fog mode, to adapt to different driving environments.

[0064] Step 6: The truck driving assistance system has a touch screen on the center console that provides a user interface. Truck drivers can use the touch screen to select the content to be displayed, such as whether to display pedestrians, vehicles, road signs, etc. Users can also adjust the color mode, transparency, and display ratio of the image to meet personal preferences.

[0065] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A truck driving assistance method based on laser radar visualization technology, characterized in that: The steps include: Step 1: Collect 3D point cloud data of the surrounding environment from the laser radar installed on the truck; Step 2: preprocess the 3D point cloud data to remove noise; Step 3: Based on the reflectivity of each point in the point cloud, the point cloud data is divided into different blocks using a clustering algorithm. Each block represents a potential traffic participant instance. The target detection algorithm is then used to identify, locate, and annotate each instance. Step 4: When the distance between a certain instance and the truck is less than or equal to the danger warning distance threshold, a danger warning message is generated; Step 5: Use Three.js to render the 3D point cloud processed in step 3 and visualize the 3D point cloud as a 2D display screen; Step 6: Project the 2D display image and hazard warning information onto the truck windshield via the head-up display installed on the truck.

2. The truck driving assistance method based on laser radar visualization technology according to claim 1 is characterized in that: The preprocessing in step 2 specifically includes extraction, denoising, filtering, and coordinate conversion of the three-dimensional point cloud data.

3. The truck driving assistance method based on laser radar visualization technology according to claim 1 is characterized in that: The clustering algorithm in step 3 is the K-Means clustering algorithm, and the target detection algorithm is the YOLOv11 algorithm.

4. The truck driving assistance method based on laser radar visualization technology according to claim 1 is characterized in that: The calculation formula of the danger warning distance threshold d0 in step 4 is: d0=v*t0 / 3.6 Among them, v represents the real-time speed of the truck, and t0 represents the reaction time of the truck driver.

5. The truck driving assistance method based on laser radar visualization technology according to claim 1 is characterized by: Step 5 specifically includes: Step 5.1, Three.js environment setup: Initialize the Three.js renderer, scene, and camera; set appropriate renderer parameters, create a 3D scene object, and configure the camera's position and viewing angle; Step 5.2, point cloud data loading and conversion: Convert the point cloud data into a geometric form recognizable by Three.js, and store the point cloud information by creating a THREE.Geometry or THREE.BufferGeometry object; Step 5.3, point cloud rendering: Use the materials and shaders provided by Three.js to define the appearance characteristics of the point cloud; then, combine the geometric objects with the materials to form the final point cloud model and add it to the scene for rendering; Step 5.4, interactive function implementation: In order to enhance the user experience, add a variety of interactive functions to the 3D visualization interface, including zooming, rotating, and panning, allowing users to view point cloud data from different angles.

6. The truck driving assistance method based on laser radar visualization technology according to claim 5 is characterized by: In step 5.3, in order to highlight the objects whose distance from the truck is less than the danger warning distance threshold, the flashing effect is achieved by changing the material properties of these specific points, and dynamically adjusting the color or transparency of these points when each frame is updated. This function can be achieved by listening to the update events in the scene or using the animation loop of Three.js.

7. The truck driving assistance method based on laser radar visualization technology according to claim 1 is characterized by: Step 6 includes adjusting the display parameters of the 2D display image according to the light intensity in the environment, specifically including: Step 6.1, light detection: monitor the light intensity of the vehicle's surrounding environment in real time through the built-in light sensor; Step 6.2, brightness adjustment: adjusting the display brightness of the head-up display based on the detected light intensity data; Step 6.3, contrast optimization: dynamically adjust the contrast of the displayed content according to the lighting conditions; Step 6.4, color correction: correct the displayed color; Step 6.5: Content adaptive adjustment: Select the most suitable information type to be displayed according to the current lighting conditions.

8. A truck driving assistance system based on laser radar visualization technology based on the method of any one of claims 1 to 7, characterized in that: It includes a laser radar, an information processing unit, and a head-up display installed on a truck; the information processing unit runs the Three.js framework and communicates with the truck central control system through the CAN bus. The laser radar sends the laser radar point cloud to the information processing unit through the CAN bus or the USB interface. The information processing unit outputs two image signals, one for display on the central control screen and the other for display on the head-up display; The laser radar is used to collect three-dimensional point cloud data of the surrounding environment and send it to the information processing unit; The information processing unit includes: pre-processing the 3D point cloud data; identifying, locating and marking each instance in the point cloud based on the reflectivity of each point in the point cloud; obtaining the real-time speed information of the truck from the central control system of the truck in real time to generate danger warning information; using Three.js to render the 3D point cloud processed in step 3, visualize the 3D point cloud into a 2D display screen, and send it to the head-up display; The head-up display is used to project 2D display images and hazard warning information onto the windshield of the truck.

Images