Multi-sensor information visualization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of automotive electronics, and discloses a multi-sensor information visualization method and device, electronic equipment and a storage medium. On one hand, visual data of multi-source sensing information such as a high-precision map, a laser radar point cloud, a laser radar obstacle, a millimeter-wave radar obstacle and a camera obstacle in a vehicle coordinate system can be obtained, and the visual data are integrated and then transmitted, so that the advantages of different sensors can be fully fused; and a rich and three-dimensional perception basis is provided for an intelligent driving system. And on the other hand, the multi-source visual information is uniformly transmitted to the visual engineering end to be graphically displayed, so that the accuracy of the information sensed by each sensor can be visually verified, an intelligent driving developer can conveniently and quickly find the problems of matching, errors and the like among the data of the multiple sensors, and the intelligent driving development efficiency is improved. The development and debugging efficiency of the intelligent driving system is improved, and the reliability of the system sensing result is enhanced.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics technology, and in particular to a method, apparatus, electronic device, and storage medium for visualizing multi-sensor information. Background Technology

[0002] Perceptual information is at the forefront of intelligent driving systems. The accuracy of perceptual information affects the safety and comfort of the entire intelligent driving system. Visualization is the most intuitive means of verifying the accuracy of perceptual information. It can simulate real-world scenarios and map sensor information onto visualization tools to help intelligent driving developers discover and solve problems.

[0003] Patent [Intelligent Driving Visual Debugging Method, Device, Equipment and Storage Medium CN202211420854.6] discloses an intelligent driving visual debugging method, device, equipment and storage medium. The method reads the BLF data files of each sensor through Matlab, outputs the results to the visualization interface and dynamically presents them with adjustable playback speed, which can realize visual analysis; however, the method has great limitations on data format and does not have real-time performance.

[0004] Patent [Intelligent Driving Data Processing Method, Device, Computer Equipment, Readable Storage Medium and Program Product CN202411160585.3] relates to an intelligent driving data processing method, device and computer equipment. It adopts a visualization processing strategy that matches the target data format to perform visualization processing on the data frame to obtain visualized data, and displays the visualized data through a browser page. However, this method requires first converting quaternions into Euler angles and then performing coordinate transformation. During this process, there is a certain probability that due to the 180° error in the Euler angle transformation, the transformed coordinates will have completely opposite results in a certain direction, which will result in a large deviation in the visualization results.

[0005] Patent [Intelligent Driving Computing Platform Functional Software Reconstruction and Visualization Method and Apparatus CN202410032227.8] relates to an intelligent driving computing platform functional software reconstruction and visualization method and apparatus. When the execution function of the target intelligent driving source code is consistent with that of the target computer language code, the target computer language code is subjected to front-end integrated graphical processing to generate visualized target computer language code, and then deployed to the corresponding hardware platform. The process of this method is relatively complex and unpredictable errors are prone to occur when reconstructing the intelligent driving source code. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device, electronic device and storage medium for visualizing multi-sensor information, which is applicable to various development environments, realizes the visualization of information and helps to improve the development efficiency of intelligent driving systems.

[0007] To address the aforementioned technical problems, in a first aspect, the present invention provides a multi-sensor information visualization method, comprising at least:

[0008] Determine the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information;

[0009] Determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system to obtain the second visualization information;

[0010] Determine the first laser obstacle contour coordinates of the lidar obstacle information in the vehicle coordinate system to obtain third visualization information;

[0011] Determine the millisecond-wave obstacle contour coordinates of the millisecond-wave radar obstacle information in the vehicle coordinate system to obtain fourth visualization information;

[0012] Determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information;

[0013] At least the first visualization information, the second visualization information, the third visualization information, the fourth visualization information, and the fifth visualization information are uniformly transmitted to the visualization engineering terminal to achieve at least the graphical display of multi-sensor information.

[0014] Optionally, determining the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information specifically includes:

[0015] Determine vehicle positioning information and vehicle attitude information, as well as the coordinates of the second high-precision map point in the high-precision map under the world coordinate system;

[0016] The vehicle translation vector is obtained based on the vehicle positioning information, and the vehicle rotation matrix is ​​determined based on the vehicle attitude information.

[0017] The coordinates of the first high-precision map point are determined based on the vehicle rotation matrix, the vehicle translation vector, and the coordinates of the second high-precision map point.

[0018] The first visualization information is obtained based on the coordinates of the first high-precision map point.

[0019] Optionally, determining the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system to obtain the second visualization information specifically includes:

[0020] Determine the second point cloud coordinates of the lidar point cloud information in the lidar coordinate system;

[0021] The three-dimensional relative position between the lidar and the center of the vehicle's rear axle is set as the lidar translation vector;

[0022] The coordinates of the first point cloud are determined based on the vehicle rotation matrix, the radar translation vector, and the second point cloud coordinates.

[0023] The second visualization information is obtained based on the cloud coordinates of the first point.

[0024] Optionally, determining the first laser obstacle contour coordinates in the vehicle coordinate system to obtain third visualization information specifically includes:

[0025] Determine the second laser obstacle contour coordinates in the laser radar coordinate system to define the laser radar obstacle information.

[0026] The first laser obstacle contour coordinates are determined based on the vehicle rotation matrix, the radar translation vector, and the second laser obstacle contour coordinates.

[0027] The third visualization information is obtained based on the outline coordinates of the first laser obstacle.

[0028] Optionally, determining the millisecond-wave obstacle contour coordinates in the vehicle coordinate system to obtain fourth visualization information specifically includes:

[0029] Determine the first millisecond-wave obstacle coordinates in the millisecond-wave radar coordinate system for the millisecond-wave radar obstacle information;

[0030] The three-dimensional relative position between the millisecond-wave radar and the center of the vehicle's rear axle is set as the millisecond radar translation vector.

[0031] The coordinates of the second millisecond wave obstacle are determined based on the vehicle rotation matrix, the millisecond radar translation vector, and the coordinates of the first millisecond wave obstacle.

[0032] The first obstacle contour point coordinate matrix is ​​determined based on the second millisecond wave obstacle coordinates to obtain the millisecond wave obstacle contour coordinates and acquire the fourth visualization information.

[0033] Optionally, determining the camera obstacle contour coordinates in the vehicle coordinate system to obtain the fifth visualization information specifically includes:

[0034] The coordinates of the first camera obstacle are determined in the camera coordinate system to define the camera obstacle information.

[0035] The three-dimensional relative position between the detection camera and the center of the vehicle's rear axle is set as the camera translation vector;

[0036] The coordinates of the second camera obstacle are determined based on the vehicle rotation matrix, the camera translation vector, and the first camera obstacle coordinates.

[0037] Based on the obstacle coordinates of the second camera, the coordinate matrix of the second obstacle contour points is determined to obtain the camera obstacle contour coordinates and acquire the fifth visualization information.

[0038] Secondly, the present invention also provides a multi-sensor information visualization device, comprising at least:

[0039] The first acquisition module is used to determine the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information.

[0040] The second acquisition module is used to determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system in order to obtain the second visualization information.

[0041] The third acquisition module is used to determine the first laser obstacle contour coordinates of the laser radar obstacle information in the vehicle coordinate system, so as to obtain the third visualization information.

[0042] The fourth acquisition module is used to determine the millisecond wave obstacle contour coordinates of the millisecond wave radar obstacle information in the vehicle coordinate system, so as to obtain the fourth visualization information;

[0043] The fifth acquisition module is used to determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information;

[0044] The graphics display module is used to transmit at least the first visualization information, the second visualization information, the third visualization information, the fourth visualization information, and the fifth visualization information to the visualization engineering terminal in order to at least realize the graphics display of multi-sensor information.

[0045] The first acquisition module is specifically used for:

[0046] The system determines vehicle positioning information, vehicle attitude information, and the coordinates of a second high-precision map point in a high-precision map under a world coordinate system; it also obtains a vehicle translation vector based on the vehicle positioning information and determines a vehicle rotation matrix based on the vehicle attitude information; it further determines the coordinates of a first high-precision map point based on the vehicle rotation matrix, the vehicle translation vector, and the coordinates of the second high-precision map point; and it obtains the first visualization information based on the coordinates of the first high-precision map point.

[0047] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that the processor, when executing the program, implements the steps of the multi-sensor information visualization method according to any one of the first aspects.

[0048] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the multi-sensor information visualization method according to any one of the first aspects.

[0049] The technical solution provided by this invention first determines the coordinates of a first high-precision map point in the vehicle coordinate system to obtain first visualization information; secondly, it determines the coordinates of a first point cloud of LiDAR point cloud information in the vehicle coordinate system to obtain second visualization information; thirdly, it determines the coordinates of a first LiDAR obstacle outline in the vehicle coordinate system to obtain third visualization information; fourthly, it determines the coordinates of a millisecond-wave obstacle outline in the vehicle coordinate system to obtain fourth visualization information; then, it determines the coordinates of a camera obstacle outline in the vehicle coordinate system to obtain fifth visualization information; finally, it transmits at least the first, second, third, fourth, and fifth visualization information to a visualization engineering terminal to achieve at least the graphical display of multi-sensor information.

[0050] Therefore, the embodiments of the present invention, on the one hand, can acquire visualized data from multiple sources of sensor information, such as high-precision maps, LiDAR point clouds, LiDAR obstacles, millimeter-wave radar obstacles, and camera obstacles, in the vehicle coordinate system, and transmit this data after integration. This fully integrates the advantages of different sensors, providing rich and comprehensive perception data for the intelligent driving system. On the other hand, by uniformly transmitting multi-source visualized information to the visualization engineering end for graphical display, the embodiments of the present invention can intuitively verify the accuracy of the perception information from each sensor. This facilitates intelligent driving developers in quickly identifying issues such as matching and errors between multi-sensor data, improving the efficiency of intelligent driving system development and debugging, and enhancing the reliability of system perception results. Furthermore, after acquiring various types of visualized information, the embodiments of the present invention uniformly transmit and display multi-source data, which helps to standardize the processing logic of multi-sensor information and improve the orderliness and efficiency of data processing. Attached Figure Description

[0051] Figure 1 This is a flowchart of a multi-sensor information visualization method provided in an embodiment of the present invention;

[0052] Figure 2 This is a sub-flowchart of a multi-sensor information visualization method provided in an embodiment of the present invention;

[0053] Figure 3 This is a sub-flowchart of another multi-sensor information visualization method provided in an embodiment of the present invention;

[0054] Figure 4This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention;

[0055] Figure 5 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention;

[0056] Figure 6 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention;

[0057] Figure 7 This is a schematic diagram of the structure of a multi-sensor information visualization device provided in an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0061] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0062] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0063] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0064] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0065] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0066] Figure 1 This is a flowchart of a multi-sensor information visualization method provided by an embodiment of the present invention. This embodiment is at least applicable to the visualization of sensor data during the development of vehicle intelligent driving systems. This multi-sensor information visualization method can be, but is not limited to, executed by the multi-sensor information visualization device of this embodiment, which can be implemented in software and / or hardware. Figure 1 As shown, this multi-sensor information visualization method includes at least the following steps:

[0067] S1. Determine the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information.

[0068] High-precision maps (HPPs) are specialized maps for autonomous driving, representing elements such as roads, lanes, and traffic signs with centimeter-level precision. They provide prior data support for vehicle positioning, planning, and control. It is known that the information on a HPP consists of points on the boundaries of the drivable area. HPP defines the drivable area for the vehicle and is crucial input information for vehicle path planning. While HPP information is based on the world coordinate system, visualization engineering is performed in the vehicle coordinate system. This makes the location of each information point more intuitive. Therefore, coordinate transformation of each point on the HPP is necessary to determine the coordinates of the first HPP point in the vehicle coordinate system.

[0069] Understandably, although a high-precision map is composed of individual points, it actually represents the boundary of the drivable area. Therefore, during visualization, the boundary lines connecting these points should be displayed to show the drivable area more intuitively. Thus, the points on the map need to be sorted, and then connected sequentially to form the high-precision map (i.e., the first visualization information).

[0070] S2. Determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system to obtain the second visualization information.

[0071] The first visualization information can be the raw LiDAR data; dense LiDAR point clouds can reflect the contours of the environment and are an important basis for obtaining obstacle information. The second visualization information can be the contour of the LiDAR point cloud data in the vehicle coordinate system.

[0072] S3. Determine the first laser obstacle contour coordinates in the vehicle coordinate system to obtain the third visualization information.

[0073] Among them, the obstacle information of lidar can be the result of point cloud information clustering, which is the final output information of lidar sensing.

[0074] S4. Determine the millisecond-wave obstacle contour coordinates of the millisecond-wave radar obstacle information in the vehicle coordinate system to obtain the fourth visualization information.

[0075] Among them, obstacle information from millimeter-wave radar can be the final output information of millimeter-wave sensing.

[0076] S5. Determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information.

[0077] Among them, obstacle information from the camera can be the final output information perceived by the intelligent camera.

[0078] S6. At least the first, second, third, fourth, and fifth visualization information are uniformly transmitted to the visualization engineering end to achieve at least the graphical display of multi-sensor information.

[0079] The visualized information can be transmitted from the intelligent driving software to the visualization engineering end through the ZMQ toolkit. Once the visualization engineering end obtains the visualized data, it can display the sensor data graphically through the RVIZ software.

[0080] The technical solution provided in this embodiment first determines the coordinates of a first high-precision map point in the vehicle coordinate system to obtain first visualization information; second, it determines the coordinates of a first point cloud of LiDAR point cloud information in the vehicle coordinate system to obtain second visualization information; third, it determines the coordinates of a first LiDAR obstacle outline in the vehicle coordinate system to obtain third visualization information; fourth, it determines the coordinates of a millisecond wave obstacle outline in the vehicle coordinate system to obtain fourth visualization information; then, it determines the coordinates of a camera obstacle outline in the vehicle coordinate system to obtain fifth visualization information; finally, it transmits at least the first, second, third, fourth, and fifth visualization information to the visualization engineering terminal to achieve at least the graphical display of multi-sensor information.

[0081] Therefore, this embodiment can, on the one hand, acquire visualized data from multiple sensor sources, including high-precision maps, LiDAR point clouds, LiDAR obstacles, millimeter-wave radar obstacles, and camera obstacles, within the vehicle coordinate system, and integrate and transmit this data. This fully leverages the advantages of different sensors, providing rich and comprehensive perceptual data for the intelligent driving system. On the other hand, by uniformly transmitting multi-source visualized information to the visualization engineering end for graphical display, this embodiment can intuitively verify the accuracy of the perceptual information from each sensor. This facilitates intelligent driving developers in quickly identifying issues such as matching and errors between multi-sensor data, improving the efficiency of intelligent driving system development and debugging, and enhancing the reliability of system perception results. Furthermore, this embodiment, after acquiring various types of visualized information separately, performs unified transmission and display processing of multi-source data, which helps to standardize the processing logic of multi-sensor information and improve the orderliness and efficiency of data processing.

[0082] Based on the above embodiments or implementation methods Figure 2 This is a sub-flowchart of a multi-sensor information visualization method provided in an embodiment of the present invention, used to refine step S1. For example... Figure 2 As shown, step S1 includes at least the following steps:

[0083] S11. Determine the vehicle positioning information, vehicle attitude information, and the coordinates of the second high-precision map point in the high-precision map under the world coordinate system.

[0084] Vehicle positioning information and vehicle attitude information can be obtained by an inertial navigation system. Vehicle attitude information can refer to vehicle pitch, roll, and yaw angles.

[0085] S12. Obtain the vehicle translation vector based on vehicle positioning information, and determine the vehicle rotation matrix based on vehicle attitude information.

[0086] The vehicle translation vector T and the vehicle rotation matrix R can be directly determined by the vehicle positioning information and the vehicle attitude information, respectively, and will not be elaborated here.

[0087] S13. Determine the coordinates of the first high-precision map point based on the vehicle rotation matrix, the vehicle translation vector, and the coordinates of the second high-precision map point.

[0088] The method for determining the coordinates of the first high-precision map point can be at least:

[0089] P vehicle =R*(P world –T);

[0090] In the formula, P vehicle P represents the coordinates of the first high-precision map point. world This indicates the coordinates of the second high-precision map point.

[0091] S14. Obtain the first visualization information based on the coordinates of the first high-precision map point.

[0092] Based on the above embodiments or implementation methods Figure 3 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention, used to refine step S2. For example... Figure 3 As shown, step S2 includes at least the following steps:

[0093] S21. Determine the coordinates of the second point cloud of the lidar point cloud information in the lidar coordinate system.

[0094] S22. Set the three-dimensional relative position between the lidar and the center of the vehicle's rear axle as the lidar translation vector.

[0095] S23. Determine the coordinates of the first point cloud based on the vehicle rotation matrix, radar translation vector, and the second point cloud coordinates.

[0096] The coordinates of the first point cloud can be determined in at least the following ways:

[0097] P vehicle1 =R*(P lidar1 –T lidar );

[0098] In the formula, T lidar P represents the translation vector of the lidar. vehicle1 P represents the coordinates of the first point cloud. lidar1 This represents the cloud coordinates of the second point.

[0099] S24. Obtain the second visualization information based on the coordinates of the first point cloud.

[0100] The second visualization information can also display the reflection intensity of the laser point cloud. For example, the reflection intensity can be mapped to a grayscale level, with 0 representing pure black and 255 representing pure white, to reflect the reflection intensity of each point cloud.

[0101] Based on the above embodiments or implementation methods Figure 4 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention, used to refine step S3. For example... Figure 4 As shown, step S3 includes at least the following steps:

[0102] S31. Determine the second laser obstacle contour coordinates in the laser radar coordinate system to obtain the laser radar obstacle information.

[0103] S32. Determine the first laser obstacle contour coordinates based on the vehicle rotation matrix, radar translation vector, and second laser obstacle contour coordinates.

[0104] Among these, due to the density of the laser point cloud, the contour point coordinates P of the lidar obstacle can be obtained. lidar2 The converted coordinates of the obstacle outline points of the lidar are P. vehicle2 To obtain the first laser obstacle contour coordinates in the vehicle coordinate system. The determination of the first laser obstacle contour coordinates can be achieved in at least the following ways:

[0105] P vehicle2 =R*(P lidar2 -T lidar );

[0106] In the formula, P vehicle2 P represents the coordinates of the first laser obstacle outline. lidar2 This represents the coordinates of the second laser obstacle outline.

[0107] In one scenario, the coordinates of the LiDAR obstacle in the laser coordinate system can be directly converted to the coordinates of the LiDAR obstacle in the vehicle coordinate system. This allows the precise coordinates of the obstacle to be displayed in text form on the visualization interface, making the obstacle more intuitive.

[0108] S33. Obtain third visualization information based on the first laser obstacle contour coordinates.

[0109] By connecting the contour points of each obstacle in the LiDAR system, a complete outline of the obstacle can be obtained, providing a more intuitive visualization of the obstacle information, which is known as the third visualization information.

[0110] Based on the above embodiments or implementation methods Figure 5 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention, used to refine step S4. For example... Figure 5As shown, this multi-sensor information visualization method includes at least the following steps:

[0111] S41. Determine the first millisecond wave obstacle coordinates of the millisecond wave radar obstacle information in the millisecond wave radar coordinate system.

[0112] S42. Set the three-dimensional relative position between the millisecond wave radar and the center of the vehicle's rear axle as the millisecond radar translation vector.

[0113] The millisecond radar translation vector can be the translation vector used to transform the millimeter-wave radar coordinate system to the vehicle coordinate system.

[0114] S43. Determine the coordinates of the second millisecond wave obstacle based on the vehicle rotation matrix, the millisecond radar translation vector, and the coordinates of the first millisecond wave obstacle.

[0115] The determination of the second millisecond wave obstacle coordinates can be achieved in at least the following ways:

[0116] P vehicle3 =R*(P lidar3 -T lidar1 );

[0117] In the formula, P vehicle3 P represents the coordinates of the obstacle in the second millisecond wave. lidar3 T represents the coordinates of the obstacle in the first millisecond wave. lidar1 This represents the millisecond radar translation vector.

[0118] S44. Determine the coordinate matrix of the first obstacle contour point based on the second millisecond wave obstacle coordinates to obtain the millisecond wave obstacle contour coordinates and acquire the fourth visualization information.

[0119] As is known, millimeter-wave radar cannot obtain obstacle contour point information like lidar. Therefore, it needs to rely on the obstacle size to indirectly obtain the obstacle contour.

[0120] For example, if the obstacle has a length of Length, a width of Width, and a height of Height, the coordinate matrix of the first obstacle contour points can be obtained based on the second millisecond wave obstacle coordinates:

[0121] [(P vehicle. X-0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z-0.5*Height),

[0122] (P) vehicle .X-0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z+0.5*Height),

[0123] (P) vehicle .X-0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z-0.5*Height),

[0124] (P) vehicle .X-0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z+0.5*Height),

[0125] (P) vehicle .X+0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z-0.5*Height),

[0126] (P) vehicle .X+0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z+0.5*Height),

[0127] (P) vehicle .X+0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z-0.5*Height),

[0128] (P) vehicle .X+0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z+0.5*Height)

[0129] The millisecond wave obstacle contour coordinates can be obtained through the first obstacle contour point coordinate matrix. By connecting the coordinates of each contour point of the millisecond wave obstacle, the complete millimeter wave radar obstacle contour can be obtained, providing a more intuitive visualization of the millimeter wave radar obstacle information (i.e., the fourth visualization information).

[0130] Based on the above embodiments or implementation methods Figure 6 This is a sub-flowchart of another multi-sensor information visualization method provided in this embodiment of the invention, used to refine step S5. For example... Figure 6 As shown, this multi-sensor information visualization method includes at least the following steps:

[0131] S51. Determine the first camera obstacle coordinates in the camera coordinate system to obtain camera obstacle information.

[0132] S52. Set the three-dimensional relative position between the detection camera and the center of the vehicle's rear axle as the camera translation vector.

[0133] S53. Determine the coordinates of the second camera obstacle based on the vehicle rotation matrix, the camera translation vector, and the first camera obstacle coordinates.

[0134] The method for determining the coordinates of the obstacle for the second camera can be at least:

[0135] P vehicle4 =R*(P lidar4 -T lidar2 );

[0136] In the formula, P vehicle4 P represents the coordinates of the obstacle for the second camera. lidar4 T represents the coordinates of the obstacle in the first camera view. lidar2 This represents the camera translation vector.

[0137] S54. Determine the coordinate matrix of the second obstacle outline points based on the obstacle coordinates of the second camera to obtain the camera obstacle outline coordinates and acquire the fifth visualization information.

[0138] This is understandable. Like millimeter-wave radar, smart cameras need to indirectly obtain the outline of obstacles by relying on their size.

[0139] For example, if the obstacle has a length of Length, a width of Width, and a height of Height, the coordinate matrix of the second obstacle contour points can be obtained based on the second millisecond wave obstacle coordinates:

[0140] [(P vehicle. X-0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z-0.5*Height),

[0141] (P) vehicle .X-0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z+0.5*Height),

[0142] (P) vehicle .X-0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z-0.5*Height),

[0143] (P) vehicle .X-0.5*Length,P vehicle .Y+0.5*Width,Pvehicle .Z+0.5*Height),

[0144] (P) vehicle .X+0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z-0.5*Height),

[0145] (P) vehicle .X+0.5*Length,P vehicle .Y-0.5*Width,P vehicle .Z+0.5*Height),

[0146] (P) vehicle .X+0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z-0.5*Height),

[0147] (P) vehicle .X+0.5*Length,P vehicle .Y+0.5*Width,P vehicle .Z+0.5*Height)

[0148] The camera obstacle outline coordinates can be obtained through the second obstacle outline point coordinate matrix. By connecting the camera obstacle outline coordinates, the complete camera obstacle outline can be obtained, which can more intuitively visualize the millimeter-wave radar obstacle information (i.e., the fifth visualization information).

[0149] Therefore, this embodiment provides a visualization method for multi-sensor information, which can realize the visualization of sensor information both online and offline, and is also applicable to the visualization of remote scenes. The method is highly portable and can be applied to various development environments. The information can be visualized simply by converting the sensor information to a specific format and sending it to the visualization engineering end, which can improve the development efficiency of intelligent driving systems.

[0150] Figure 7 This is a schematic diagram of a multi-sensor information visualization device provided in an embodiment of the present invention. This embodiment is at least applicable to the visualization of sensor data during the development of vehicle intelligent driving systems. This multi-sensor information visualization device can be implemented in software and / or hardware. Figure 7 As shown, the multi-sensor information visualization device includes at least:

[0151] The first acquisition module 110 is used to determine the coordinates of the first high-precision map point in the high-precision map under the vehicle coordinate system in order to obtain the first visualization information.

[0152] The second acquisition module 120 is used to determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system in order to obtain the second visualization information.

[0153] The third acquisition module 130 is used to determine the first laser obstacle contour coordinates of the lidar obstacle information in the vehicle coordinate system in order to obtain the third visualization information.

[0154] The fourth acquisition module 140 is used to determine the millisecond wave obstacle contour coordinates of the millisecond wave radar obstacle information in the vehicle coordinate system in order to obtain the fourth visualization information.

[0155] The fifth acquisition module 150 is used to determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information.

[0156] The graphics display module 160 is used to uniformly transmit at least the first, second, third, fourth, and fifth visualization information to the visualization engineering end, so as to realize the graphics display of multi-sensor information.

[0157] Optionally, the first acquisition module is specifically used for:

[0158] The system determines vehicle positioning information, vehicle attitude information, and the coordinates of a second high-precision map point in the world coordinate system; it also obtains a vehicle translation vector based on the vehicle positioning information and determines a vehicle rotation matrix based on the vehicle attitude information; it further determines the coordinates of a first high-precision map point based on the vehicle rotation matrix, the vehicle translation vector, and the second high-precision map point coordinates; and it acquires first visualization information based on the first high-precision map point coordinates.

[0159] Optionally, the second acquisition module is specifically used for:

[0160] The system determines the second point cloud coordinates of the lidar point cloud information in the lidar coordinate system; sets the three-dimensional relative position between the lidar and the rear axle center of the vehicle as the lidar translation vector; determines the first point cloud coordinates based on the vehicle rotation matrix, the lidar translation vector, and the second point cloud coordinates; and obtains the second visualization information based on the first point cloud coordinates.

[0161] Optionally, the third acquisition module is specifically used for:

[0162] The system determines the second laser obstacle contour coordinates in the lidar coordinate system to identify the lidar obstacle information; and determines the first laser obstacle contour coordinates based on the vehicle rotation matrix, the radar translation vector, and the second laser obstacle contour coordinates; and obtains third visualization information based on the first laser obstacle contour coordinates.

[0163] Optionally, the fourth acquisition module is specifically used for:

[0164] The system determines the first millisecond-wave obstacle coordinates in the millisecond-wave radar coordinate system; sets the three-dimensional relative position between the millisecond-wave radar and the rear axle center of the vehicle as the millisecond-wave radar translation vector; determines the second millisecond-wave obstacle coordinates based on the vehicle rotation matrix, the millisecond-wave radar translation vector, and the first millisecond-wave obstacle coordinates; and determines the first obstacle contour point coordinate matrix based on the second millisecond-wave obstacle coordinates to obtain the millisecond-wave obstacle contour coordinates and acquire the fourth visualization information.

[0165] Optionally, the fifth acquisition module is specifically used for:

[0166] The system determines the first camera obstacle coordinates in the camera coordinate system; sets the three-dimensional relative position between the detection camera and the rear axle center of the vehicle as the camera translation vector; determines the second camera obstacle coordinates based on the vehicle rotation matrix, the camera translation vector, and the first camera obstacle coordinates; and determines the second obstacle contour point coordinate matrix based on the second camera obstacle coordinates to obtain the camera obstacle contour coordinates and acquire the fifth visualization information.

[0167] The technical solution provided in this embodiment firstly obtains first visualization information by determining the coordinates of a first high-precision map point in the vehicle coordinate system using a first acquisition module. Further, it obtains second visualization information by determining the coordinates of a first point cloud of LiDAR point cloud information in the vehicle coordinate system using a second acquisition module. Further, it obtains third visualization information by determining the coordinates of a first laser obstacle contour of LiDAR obstacle information in the vehicle coordinate system using a third acquisition module. Further, it obtains fourth visualization information by determining the coordinates of a millisecond-wave obstacle contour of millisecond-wave radar obstacle information in the vehicle coordinate system using a fourth acquisition module. Further, it obtains fifth visualization information by determining the coordinates of a camera obstacle contour of camera obstacle information in the vehicle coordinate system using a fifth acquisition module. Finally, it transmits the first, second, third, fourth, and fifth visualization information to a visualization engineering terminal via a graphics display module to achieve at least the graphical display of multi-sensor information.

[0168] Therefore, the embodiments of the present invention, on the one hand, can acquire visualized data from multiple sources of sensor information, such as high-precision maps, LiDAR point clouds, LiDAR obstacles, millimeter-wave radar obstacles, and camera obstacles, in the vehicle coordinate system, and transmit this data after integration. This fully integrates the advantages of different sensors, providing rich and comprehensive perception data for the intelligent driving system. On the other hand, by uniformly transmitting multi-source visualized information to the visualization engineering end for graphical display, the embodiments of the present invention can intuitively verify the accuracy of the perception information from each sensor. This facilitates intelligent driving developers in quickly identifying issues such as matching and errors between multi-sensor data, improving the efficiency of intelligent driving system development and debugging, and enhancing the reliability of system perception results. Furthermore, after acquiring various types of visualized information, the embodiments of the present invention uniformly transmit and display multi-source data, which helps to standardize the processing logic of multi-sensor information and improve the orderliness and efficiency of data processing.

[0169] This embodiment provides an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. See also: Figure 8 The electronic device 1000 includes a processor 1001 and a memory 1002. The memory 1002 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 1001, the steps in any of the above multi-sensor information visualization methods are performed. Through the above technical solution, the processor 1001 and the memory 1002 are interconnected and communicate with each other through a communication bus and / or other forms of connection mechanism (not shown). The memory 1002 stores a computer program executable by the processor. When the electronic device 1000 is running, the processor 1001 executes the computer program to execute the multi-sensor information visualization method in any optional implementation of the above embodiments, so as to at least achieve the following functions: determine the coordinates of a first high-precision map point in the high-precision map under the vehicle coordinate system to obtain first visualization information; determine the coordinates of a first point cloud of LiDAR point cloud information under the vehicle coordinate system to obtain second visualization information; determine the coordinates of a first laser obstacle contour of LiDAR obstacle information under the vehicle coordinate system to obtain third visualization information; determine the coordinates of a millisecond wave obstacle contour of millisecond wave radar obstacle information under the vehicle coordinate system to obtain fourth visualization information; determine the coordinates of a camera obstacle contour of camera obstacle information under the vehicle coordinate system to obtain fifth visualization information; and at least transmit the first, second, third, fourth, and fifth visualization information to the visualization engineering end to at least achieve the graphical display of multi-sensor information.

[0170] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the multi-sensor information visualization method provided in all embodiments of this application: determining the coordinates of a first high-precision map point in a high-precision map under a vehicle coordinate system to obtain first visualization information; determining the coordinates of a first point cloud of LiDAR point cloud information under a vehicle coordinate system to obtain second visualization information; determining the coordinates of a first laser obstacle contour of LiDAR obstacle information under a vehicle coordinate system to obtain third visualization information; determining the coordinates of a millisecond-wave obstacle contour of millisecond-wave radar obstacle information under a vehicle coordinate system to obtain fourth visualization information; determining the coordinates of a camera obstacle contour of camera obstacle information under a vehicle coordinate system to obtain fifth visualization information; and transmitting at least the first, second, third, fourth, and fifth visualization information to a visualization engineering terminal to at least achieve graphical display of multi-sensor information.

[0171] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0172] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0173] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0174] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visualizing multi-sensor information, characterized in that, At least including: Determine the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information; Determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system to obtain the second visualization information; Determine the first laser obstacle contour coordinates of the lidar obstacle information in the vehicle coordinate system to obtain third visualization information; Determine the millisecond-wave obstacle contour coordinates of the millisecond-wave radar obstacle information in the vehicle coordinate system to obtain fourth visualization information; Determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information; At least the first visualization information, the second visualization information, the third visualization information, the fourth visualization information, and the fifth visualization information are uniformly transmitted to the visualization engineering terminal to achieve at least the graphical display of multi-sensor information.

2. The multi-sensor information visualization method according to claim 1, characterized in that, The process of determining the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information specifically includes: Determine vehicle positioning information and vehicle attitude information, as well as the coordinates of the second high-precision map point in the high-precision map under the world coordinate system; The vehicle translation vector is obtained based on the vehicle positioning information, and the vehicle rotation matrix is ​​determined based on the vehicle attitude information. The coordinates of the first high-precision map point are determined based on the vehicle rotation matrix, the vehicle translation vector, and the coordinates of the second high-precision map point. The first visualization information is obtained based on the coordinates of the first high-precision map point.

3. The multi-sensor information visualization method according to claim 2, characterized in that, The process of determining the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system to obtain the second visualization information specifically includes: Determine the second point cloud coordinates of the lidar point cloud information in the lidar coordinate system; The three-dimensional relative position between the lidar and the center of the vehicle's rear axle is set as the lidar translation vector; The coordinates of the first point cloud are determined based on the vehicle rotation matrix, the radar translation vector, and the second point cloud coordinates. The second visualization information is obtained based on the cloud coordinates of the first point.

4. The multi-sensor information visualization method according to claim 3, characterized in that, The process of determining the first laser obstacle contour coordinates in the vehicle coordinate system to obtain third visualization information specifically includes: Determine the second laser obstacle contour coordinates in the laser radar coordinate system to define the laser radar obstacle information. The first laser obstacle contour coordinates are determined based on the vehicle rotation matrix, the radar translation vector, and the second laser obstacle contour coordinates. The third visualization information is obtained based on the outline coordinates of the first laser obstacle.

5. The multi-sensor information visualization method according to claim 2, characterized in that, The step of determining the millisecond-wave obstacle contour coordinates in the vehicle coordinate system to obtain fourth visualization information specifically includes: Determine the first millisecond-wave obstacle coordinates in the millisecond-wave radar coordinate system for the millisecond-wave radar obstacle information; The three-dimensional relative position between the millisecond-wave radar and the center of the vehicle's rear axle is set as the millisecond radar translation vector. The coordinates of the second millisecond wave obstacle are determined based on the vehicle rotation matrix, the millisecond radar translation vector, and the coordinates of the first millisecond wave obstacle. The first obstacle contour point coordinate matrix is ​​determined based on the second millisecond wave obstacle coordinates to obtain the millisecond wave obstacle contour coordinates and acquire the fourth visualization information.

6. The multi-sensor information visualization method according to claim 2, characterized in that, The step of determining the camera obstacle contour coordinates in the vehicle coordinate system to obtain the fifth visualization information specifically includes: The coordinates of the first camera obstacle are determined in the camera coordinate system to define the camera obstacle information. The three-dimensional relative position between the detection camera and the center of the vehicle's rear axle is set as the camera translation vector; The coordinates of the second camera obstacle are determined based on the vehicle rotation matrix, the camera translation vector, and the first camera obstacle coordinates. Based on the obstacle coordinates of the second camera, the coordinate matrix of the second obstacle contour points is determined to obtain the camera obstacle contour coordinates and acquire the fifth visualization information.

7. A multi-sensor information visualization device, characterized in that, At least including: The first acquisition module is used to determine the coordinates of the first high-precision map point in the vehicle coordinate system to obtain the first visualization information. The second acquisition module is used to determine the first point cloud coordinates of the lidar point cloud information in the vehicle coordinate system in order to obtain the second visualization information. The third acquisition module is used to determine the first laser obstacle contour coordinates of the laser radar obstacle information in the vehicle coordinate system, so as to obtain the third visualization information. The fourth acquisition module is used to determine the millisecond wave obstacle contour coordinates of the millisecond wave radar obstacle information in the vehicle coordinate system, so as to obtain the fourth visualization information; The fifth acquisition module is used to determine the camera obstacle outline coordinates in the vehicle coordinate system to obtain the fifth visualization information; The graphics display module is used to transmit at least the first visualization information, the second visualization information, the third visualization information, the fourth visualization information, and the fifth visualization information to the visualization engineering terminal in order to at least realize the graphics display of multi-sensor information.

8. The multi-sensor information visualization device according to claim 7, characterized in that, The first acquisition module is specifically used for: The system determines vehicle positioning information, vehicle attitude information, and the coordinates of a second high-precision map point in a high-precision map under the world coordinate system; and obtains a vehicle translation vector based on the vehicle positioning information and a vehicle rotation matrix based on the vehicle attitude information. And, the coordinates of the first high-precision map point are determined based on the vehicle rotation matrix, the vehicle translation vector, and the coordinates of the second high-precision map point; In addition, the first visualization information is obtained based on the coordinates of the first high-precision map point.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multi-sensor information visualization method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-sensor information visualization method according to any one of claims 1 to 6.

Images