A point cloud fusion method and device for collecting data and electronic equipment

Abstract

The application provides a point cloud fusion method and device for collecting data and electronic equipment. The method is suitable for fusing point clouds detected by a radar on a target vehicle. According to radar calibration data, first calibration data, second calibration data, first data and second data of different regions of the target vehicle, the first point cloud in the first radar coordinate system is transformed into a target coordinate system with the second vehicle body as a reference to obtain the first target point cloud. The second point cloud in the second radar coordinate system is also transformed into the target coordinate system to obtain the second target point cloud. Finally, the fusion point cloud is determined according to the first target point cloud and the second target point cloud. According to the data measured by the sensors of different vehicle bodies of the target vehicle and the calibration data, the pose changes of different vehicle bodies are fully considered, and then the point clouds measured by the radars of different vehicle bodies are directly fused into the same coordinate system based on the changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of point cloud data processing, and in particular to a point cloud fusion method and device for collecting data and an electronic device. BACKGROUND

[0002] With the rapid development of automatic driving technology, in the scheme of data collection of an advanced driving assistance system or an automatic driving system, the required processing can be completed based on the data collected by sensors of each vehicle, for example, the data collected by sensors of each vehicle can be presented in real time or played back afterwards by using a relevant interface of software, and for another example, model training can also be performed based on the data collected by sensors.

[0003] However, in a vehicle such as a tractor-trailer, the tractor and the trailer are connected by a drawbar or a hitch, and the relative poses of the tractor and the trailer change dynamically with the driving of the vehicle. At this time, the point cloud data collected by the radar of the tractor and the radar of the trailer cannot be directly calibrated in two coordinate systems, and thus the point cloud data collected by the radar of the tractor and the radar of the trailer cannot be directly fused in the same coordinate system.

[0004] Therefore, it is necessary to provide a point cloud fusion method to solve the technical problem that the point cloud data collected by the radar of the tractor and the radar of the trailer in a vehicle such as a tractor-trailer cannot be directly fused in the same coordinate system. SUMMARY

[0005] The present application provides a point cloud fusion method and device for collecting data and an electronic device to solve the technical problem that the point cloud data collected by the radar of the tractor and the radar of the trailer in a vehicle such as a tractor-trailer cannot be directly fused in the same coordinate system.

[0006] To solve the above technical problem, the present application provides the following technical solution:

[0007] The present application provides a point cloud fusion method for collecting data, which is suitable for fusing the point cloud detected by the radar on a target vehicle, and the method comprises the following steps:

[0008] acquire a first point cloud measured by a first radar arranged on a first vehicle body of the target vehicle, first radar calibration data of the first radar, first data measured by a first sensor arranged on the first vehicle body, first calibration data of the first sensor, and a second point cloud measured by a second radar arranged on a second vehicle body of the target vehicle, second radar calibration data of the second radar, second data measured by a second sensor arranged on the second vehicle body, and second calibration data of the second sensor; wherein the first point cloud is in a first radar coordinate system, the first radar coordinate system taking the first radar as a reference; the second point cloud is in a second radar coordinate system, the second radar coordinate system taking the second radar as a reference; the first data includes position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data includes position data and / or attitude data of the second vehicle body measured by the second sensor;

[0009] transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud; wherein the target coordinate system takes the second vehicle body as a reference;

[0010] transform the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud;

[0011] determine a fused point cloud according to the first target point cloud and the second target point cloud.

[0012] Meanwhile, the application also provides a point cloud fusion device for collecting data, which is suitable for fusing point clouds detected by radars on a target vehicle, and the device comprises:

[0013] a data acquisition module, configured to acquire a first point cloud measured by a first radar arranged on a first vehicle body of the target vehicle, first radar calibration data of the first radar, first data measured by a first sensor arranged on the first vehicle body, first calibration data of the first sensor, and a second point cloud measured by a second radar arranged on a second vehicle body of the target vehicle, second radar calibration data of the second radar, second data measured by a second sensor arranged on the second vehicle body, and second calibration data of the second sensor; wherein the first point cloud is in a first radar coordinate system, the first radar coordinate system taking the first radar as a reference; the second point cloud is in a second radar coordinate system, the second radar coordinate system taking the second radar as a reference; the first data includes position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data includes position data and / or attitude data of the second vehicle body measured by the second sensor;

[0014] a first transformation module configured to transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud, wherein the target coordinate system is based on the second vehicle body;

[0015] a second transformation module configured to transform the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud;

[0016] a fusion module configured to determine a fusion point cloud according to the first target point cloud and the second target point cloud.

[0017] Meanwhile, the present application provides an electronic device, which comprises a processor and a memory, the memory is used to store a computer program, and the processor is used to run the computer program in the memory to execute the steps in the point cloud fusion method for collecting data.

[0018] In addition, the present application also provides a computer readable storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the point cloud fusion method for collecting data.

[0019] Beneficial effects: the present application provides a point cloud fusion method, device and electronic equipment for collecting data, which is suitable for fusing the point cloud detected by the radar on the target vehicle. Specifically, the method first acquires the first point cloud measured by the first radar arranged on the first vehicle body of the target vehicle, the first radar calibration data of the first radar, the first data measured by the first sensor arranged on the first vehicle body, the first calibration data of the first sensor, the second point cloud measured by the second radar arranged on the second vehicle body of the target vehicle, the second radar calibration data of the second radar, the second data measured by the second sensor arranged on the second vehicle body, and the second calibration data of the second sensor, wherein the first point cloud is in the first radar coordinate system, the first radar coordinate system is based on the first radar, the second point cloud is in the second radar coordinate system, and the second radar coordinate system is based on the second radar. Then, according to the acquired first radar calibration data, first calibration data, first data, second data and second calibration data, the first point cloud is transformed to the target coordinate system to obtain the first target point cloud, and simultaneously, according to the acquired second radar calibration data, the second point cloud is transformed to the target coordinate system to obtain the second target point cloud. Finally, according to the first target point cloud and the second target point cloud, the fusion point cloud is determined. The method fully describes the pose change of different vehicle bodies when moving with the target vehicle through the data and calibration data measured by the sensors of different vehicle bodies of the target vehicle, and based on this change, the first point cloud in the first radar coordinate system based on the first radar of the first vehicle body is moved to the target coordinate system based on the second vehicle body, and at the same time, the second point cloud in the second radar coordinate system based on the second radar of the second vehicle body is also moved to the target coordinate system, so as to realize the purpose of directly fusing the point cloud data of different regions to the same coordinate system for presentation. BRIEF DESCRIPTION OF DRAWINGS

[0020] The technical solutions and other beneficial effects of the present application will become apparent from the following detailed description of the specific embodiments of the present application, combined with the accompanying drawings.

[0021] Figure 1 is a scene schematic diagram of the point cloud fusion system for collecting data provided by the embodiments of the present application.

[0022] Figure 2 is a flowchart of the point cloud fusion method provided by the embodiments of the present application.

[0023] Figure 3 is a schematic diagram of the sensors and radars of the target vehicle provided by the embodiments of the present application.

[0024] Figure 4 is a schematic diagram of each coordinate system provided by the embodiments of the present application.

[0025] Figure 5 is a process of collecting data point cloud fusion provided by the embodiment of the application.

[0026] Figure 6a is an effect diagram before point cloud fusion provided by the embodiment of the application.

[0027] Figure 6b is an effect diagram after point cloud fusion provided by the embodiment of the application.

[0028] Figure 7 is a structure diagram of a point cloud fusion device for collecting data provided by the embodiment of the application.

[0029] Figure 8 is a structure diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the application.

[0031] The terms “include” and “have” and any variations thereof in the specification and claims of the application are intended to cover non-exclusive inclusion; and the division of modules in the application is only a logical division, and there can be another division manner in actual application, for example, a plurality of modules can be combined or integrated in another system, or some features can be ignored or not executed.

[0032] In the application, the target carrier includes large vehicles such as large trucks, small trains, and bullet trains; and the point cloud detected by the radar on the target carrier refers to a point data set in a certain coordinate system, which contains rich information including three-dimensional coordinates X, Y, and Z, and can also include color, classification value, intensity value, time, etc.

[0033] In the application, the first vehicle body and the second vehicle body refer to two relatively independent vehicle bodies separated by the third vehicle body. For example, the trailer part of a large truck such as a trailer truck can be regarded as the first vehicle body, and the vehicle head can be regarded as the second vehicle body, and the connecting device such as the traction rod or the traction seat connecting the vehicle head and the trailer part can be regarded as the third vehicle body separating the vehicle head and the trailer part. It should be noted that in the application, the vehicle head can also be regarded as the first vehicle body, and the trailer part can also be regarded as the second vehicle body.

[0034] In the present application, the first vehicle body and the second vehicle body each include three radars and one sensor (for example, an inertial measurement unit, IMU). Among them, the three radars and the one sensor (for example, the inertial measurement unit, IMU) in the first vehicle body each correspond to its own coordinate system, and these coordinate systems share one origin; similarly, the three radars and the one sensor (for example, the inertial measurement unit, IMU) in the second vehicle body each correspond to its own coordinate system, and these coordinate systems share one origin.

[0035] In the present application, the first point cloud and the second point cloud are respectively real-time and dynamic radar point cloud data collected by the radars of the first vehicle body and the radars of the second vehicle body; the first radar calibration data is the position of the radar of the first vehicle body in the first vehicle body, that is, the relative pose of the coordinate system of the first radar (the first radar coordinate system) relative to the coordinate system of the first vehicle body (the first coordinate system) from the data level; the second radar calibration data is the position of the radar of the second vehicle body in the second vehicle body, that is, the relative pose of the coordinate system of the second radar (the second radar coordinate system) relative to the coordinate system of the second vehicle body (the second coordinate system) from the data level; similarly, the first calibration data is the position of the first sensor of the first vehicle body in the first vehicle body, that is, the relative pose of the coordinate system of the first sensor (the first pose coordinate system) relative to the coordinate system of the first vehicle body (the first coordinate system) from the data level; the second calibration data is the position of the second sensor of the second vehicle body in the second vehicle body, that is, the relative pose of the coordinate system of the second sensor (the second pose coordinate system) relative to the coordinate system of the second vehicle body (the second coordinate system) from the data level; the first data and the second data are respectively real-time position information collected by the first sensor and the second sensor, including position data and / or attitude data, and the real-time position information is position information corresponding to the time stamp of the current frame of the point cloud.

[0036] The present application provides a point cloud fusion method and device for collecting data and electronic equipment.

[0037] Please refer to Figure 1 , Figure 1 is a scene schematic diagram of a point cloud fusion system for collecting data provided by the embodiments of the present application, as Figure 1 shown, the point cloud fusion system for collecting data at least includes a software terminal 101, a data server 102 and a target carrier 103, wherein:

[0038] A communication link is provided between the software terminal 101, the data server 102 and the target carrier 103 to realize information interaction. The type of communication link can include wired, wireless communication link or optical fiber cable, etc., which is not limited in the present application.

[0039] The software terminal 101 is mainly used to provide relevant software and present the collected advanced driving assistance system data of the target vehicle 103 in real time or after playback based on the software.

[0040] The data server 102 is mainly used to store various sensor data collected by the target vehicle and provide the data to the software terminal so that the software terminal 101 can process the data based on the data. The data server 102 can be a stand-alone server, a server network or a server cluster. For example, the server described in the present application includes but is not limited to a computer, a network host, a database server, an application server or a cloud server composed of multiple servers, wherein the cloud server is composed of a large number of computers or network servers based on cloud computing.

[0041] The target vehicle 103 includes large vehicles such as large trucks, small trains and bullet trains. The target vehicle 103 is equipped with an advanced driving assistance system and various sensors such as millimeter wave radar, laser radar, monocular / dual camera and satellite navigation, which are mainly used to sense the surrounding environment during the movement (such as driving) of the target vehicle, collect various sensor data, and transmit the sensor data to the software terminal for static and dynamic object recognition, detection and tracking, and combined with navigation map data, system operation and analysis.

[0042] The embodiment of the application provides a point cloud fusion system for collecting data, which comprises a software terminal, a data server and a target vehicle, and is suitable for fusing point clouds detected by radars on the target vehicle. Specifically, taking the target vehicle as a trailer, the first vehicle body as the trailer part of the trailer, and the second vehicle body as the head of the trailer as examples, first, the software terminal records the absolute positions and Euler angles of the head radar, the trailer part radar, the head sensor and the trailer part sensor of the trailer relative to the head and the trailer part, and generates respective calibration data based on the data and stores the calibration data in the memory (including first radar calibration data of the head radar, second radar calibration data of the trailer part radar, first calibration data of the head sensor, and second calibration data of the trailer part sensor), during the movement of the trailer, the trailer sends the first point cloud and the second point cloud collected by the head trailer part radars and the first data and the second data collected by the head trailer part sensors to the software terminal, and the software terminal processes the data into specific files and stores the files in the data server; then the software terminal obtains and parses the corresponding files from the data server to obtain corresponding data, and according to the first radar calibration data, the first calibration data, the first data, the second data and the second calibration data, the first point cloud measured by the trailer part radar is converted to the target coordinate system with the head as the reference to obtain the first target point cloud, and at the same time, the second point cloud measured by the head radar is also converted to the target coordinate system with the head as the reference according to the second radar calibration data to obtain the second target point cloud, and finally the software terminal processes the first target point cloud and the second target point cloud to obtain the fusion point cloud in the same coordinate system.

[0043] In the above point cloud dynamic fusion process, the position data and / or attitude data collected by the head and trailer part sensors of the trailer are fully considered for the changes in the positions and postures of the head and the trailer part of the trailer, and based on the changes, the point cloud measured by the trailer part radar and the point cloud measured by the head radar are respectively converted to the target coordinate system, so that the point cloud measured by the trailer part radar and the point cloud measured by the head radar are presented in the same coordinate system, and the point clouds measured by the head and the trailer part radars can be effectively and accurately fused together.

[0044] It should be noted that, Figure 1 The scene diagram shown is only an example, and the server, terminal and scene described in the embodiment of the application are used to more clearly illustrate the technical scheme of the embodiment of the application, and do not constitute a limitation on the technical scheme provided by the embodiment of the application. Those skilled in the art can know that the technical scheme provided by the embodiment of the application is also applicable to similar technical problems as the system evolves and new business scenarios appear. The following will be described in detail. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0045] In view of the above, the scene of the point cloud fusion system for collecting data will be described below. Please refer to Figure 2 as shown, Figure 2 is a flowchart of the point cloud fusion method for collecting data provided by the embodiments of the present application. The method is suitable for fusing the point cloud detected by the radar on the target vehicle. The method comprises at least the following steps:

[0046] S201: Obtain the first point cloud measured by the first radar arranged on the first vehicle body of the target vehicle, the first radar calibration data of the first radar, the first data measured by the first sensor arranged on the first vehicle body, the first calibration data of the first sensor, and the second point cloud measured by the second radar arranged on the second vehicle body of the target vehicle, the second radar calibration data of the second radar, the second data measured by the second sensor arranged on the second vehicle body, and the second calibration data of the second sensor; wherein the first point cloud is in the first radar coordinate system, and the first radar coordinate system is based on the first radar; the second point cloud is in the second radar coordinate system, and the second radar coordinate system is based on the second radar; the first data comprises the position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data comprises the position data and / or attitude data of the second vehicle body measured by the second sensor.

[0047] The target vehicle is a trailer vehicle comprising a vehicle head and a trailer part, and the pose of the trailer part relative to the vehicle head is variable. In addition, the target vehicle can also include large vehicles such as trucks, small trains, and motor trains. As Figure 3 as shown, Figure 3 is a schematic diagram of the sensor and radar layout of the target vehicle provided by the embodiments of the present application. Figure 3 The target vehicle is divided into a first vehicle body, a second vehicle body, and a third vehicle body, and the first vehicle body and the second vehicle body are separated into two independent areas by the third vehicle body. Therefore, when the first vehicle body is the vehicle head of the target vehicle, the second vehicle body is the trailer part of the target vehicle; when the first vehicle body is the trailer part of the target vehicle, the first vehicle body is the vehicle head of the target vehicle. For example, the trailer part of a large truck such as a trailer vehicle can be regarded as the first vehicle body, and the vehicle head can be regarded as the second vehicle body. The connecting device such as the towing rod or the towing seat connecting the vehicle head and the trailer part can be regarded as the third vehicle body separating the vehicle head and the trailer part. In addition, the vehicle head can also be regarded as the first vehicle body, and the trailer part can also be regarded as the second vehicle body. The sensors of the first vehicle body comprise three first radars and one first sensor (such as an inertial measurement unit IMU), and similarly, the sensors of the second vehicle body also comprise three second radars and one second sensor (such as an inertial measurement unit IMU).

[0048] For the convenience of description, the target vehicle is exemplified by a trailer in the following description, and the first vehicle body is exemplified by the trailer part of the trailer, and the second vehicle body is exemplified by the vehicle head of the trailer.

[0049] It should be noted that the three second radars and the second sensor of the vehicle head each correspond to a coordinate system sharing one origin point (which can be referred to as the vehicle head origin point), and the three first radars and the first sensor of the trailer part each correspond to a coordinate system sharing one origin point (which can be referred to as the trailer part origin point).

[0050] Because the coordinate system corresponding to the vehicle head radar and the coordinate system corresponding to the trailer part radar have different origin points at this time, that is, the vehicle head radar and the trailer part radar are in two coordinate systems, and cannot be directly calibrated, thereby causing the point clouds detected by each radar to be unable to be directly fused in the same coordinate system. Therefore, according to the following processing process, the real-time change between the vehicle head and the trailer part is first detected, and then the point clouds are dynamically fused based on the transformation.

[0051] In an embodiment, the first point cloud, the second point cloud, the first data, and the second data are all collected by the radars and sensors in the trailer, and the software terminal generates specific files from these data and stores them in the data server. In use, the software terminal obtains these files from the data server and parses them to obtain the required data, and the specific steps include: reading the radar data file, the sensor calibration file, and the sensor data file; parsing the radar data file to obtain the first point cloud collected by the first radar and the second point cloud collected by the second radar at the current time; parsing the sensor calibration file to obtain the first radar calibration data of the first radar and the first calibration data of the first sensor, and the second radar calibration data of the second radar and the second calibration data of the second sensor; and parsing the sensor data file to obtain the first data measured by the first sensor and the second data measured by the second sensor at the current time.

[0052] The radar data file includes a pcap file generated according to the point cloud data collected by the first radar arranged on the trailer part and the second radar arranged on the vehicle head. The file describes the information of some point clouds, including three-dimensional coordinates (x, y, z coordinates), laser reflection intensity (Inrensity), etc. The sensor calibration file includes a json calibration file generated according to the absolute positions and Euler angles of the first radar and the first sensor arranged on the trailer part relative to the trailer part, and the absolute positions and Euler angles of the second radar and the second sensor arranged on the vehicle head relative to the vehicle head. The file records some initial positions. The sensor data file includes a pb file generated according to the real-time motion trajectories of the first sensor arranged on the trailer part and the second sensor arranged on the vehicle head during the driving of the trailer. The file records the GPS coordinates and Euler angles of the sensors.

[0053] It should be noted that each sensor (including radar and sensor) arranged in the trailer sends each data collected during the vehicle driving to the software terminal, and the software terminal processes the received data accordingly to generate the aforementioned various files and store them in the data server. When performing point cloud fusion processing, these files are imported / bound into the processing software. Specifically, taking the use of ADAS View software to process the point cloud as an example, first, the software personnel need to add the first point cloud collected by the first radar arranged in the trailer and the second point cloud collected by the second radar arranged in the vehicle head which need to be fused in the ADAS View software, and then obtain the aforementioned files from the data server, and bind the radar data file, import the sensor calibration file, and bind the sensor data file, so as to subsequently fuse the first point cloud and the second point cloud according to the data in these files.

[0054] Specifically, the radar data file is parsed to obtain the first point cloud of the trailer (i.e., the point cloud measured by the first radar arranged in the trailer) and the second point cloud of the vehicle head (i.e., the point cloud measured by the second radar arranged in the vehicle head) at the current time; the sensor calibration file is parsed to obtain the first radar calibration data of the first radar (i.e., the absolute position and Euler angle of the first radar relative to the trailer) and the second radar calibration data of the second radar (i.e., the absolute position and Euler angle of the second radar relative to the vehicle head); the sensor data file is parsed to obtain the first data measured by the first sensor and the second data measured by the second sensor according to the timestamp corresponding to the current frame of the point cloud.

[0055] S202: Transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data to obtain a first target point cloud; wherein the target coordinate system takes the second vehicle body as a reference.

[0056] In one embodiment, the first point cloud needs to be transformed according to the data obtained above to obtain the first target point cloud of the first point cloud in the target coordinate system, and the specific steps include: determining a point cloud fusion matrix according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data; transforming the first point cloud to the target coordinate system according to the point cloud fusion matrix to obtain the first target point cloud.

[0057] Specifically, by using the first radar calibration data of the first radar, the first calibration data of the first sensor, the first data measured by the first sensor, the second data measured by the second sensor, and the second calibration data of the second sensor, a point cloud fusion matrix T is generated, and the first point cloud is denoted as P1. Then, the calculation is performed by T*P1, that is, the first point cloud is transformed to the target coordinate system, and the calculation result is the first target point cloud.

[0058] In one embodiment, the fusion matrix T can be calculated first. The calculation process includes: performing an inversion operation on the first calibration data to obtain a first transformation matrix; obtaining the preset pose coordinate system of the target vehicle; generating a second transformation matrix based on the relative pose relationship between the first data and the preset pose coordinate system; generating a third transformation matrix based on the relative pose relationship between the second data and the preset pose coordinate system; and generating a point cloud fusion matrix based on the first radar calibration data, the first transformation matrix, the second transformation matrix, the third transformation matrix, and the second calibration data. The calibration data itself embodies the transformation matrix. For example, the calibration data of the first radar reflects the position of the first radar on the first vehicle body. From a data perspective, it is the relative pose of the coordinate system of the first radar (i.e., the coordinate system used for the data measured by the first radar, denoted here as the Ord1 coordinate system) relative to the coordinate system of the first vehicle body (i.e., the coordinate system based on the first vehicle body, which is artificially defined, denoted here as the O1 coordinate system). This relative pose is the transformation matrix for converting data from the Ord1 coordinate system to the O1 coordinate system. The preset pose coordinate system refers to the pre-set coordinate system (denoted here as the Oimu coordinate system) used to convert the data measured by the first sensor and the second sensor to the same coordinate system.

[0059] Specifically, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the various coordinate systems provided in the embodiments of this application. To transform the first point cloud measured by the first radar of the trailer to a coordinate system based on the vehicle's front end, a fusion matrix T can be calculated first. The fusion matrix T consists of the transformation matrix Ta (lidar_to_rear_base) from the first radar coordinate system (Orad1 coordinate system) to the first coordinate system (O1 coordinate system), the transformation matrix Tb (imu_to_rear_base.inverse) from the first coordinate system (O1 coordinate system) to the first pose coordinate system (i.e., the coordinate system used by the data measured by the first sensor, denoted here as Oimu1 coordinate system), and the transformation matrix Tb (imu_to_rear_base.inverse) from the first pose coordinate system (Oimu1 coordinate system) to the preset pose. The system consists of the transformation matrix Tc (rear_imu_pose) of the Oimu coordinate system, the transformation matrix Td (front_imu_pose.inverse) of the preset pose coordinate system (Oimu coordinate system) to the second pose coordinate system (i.e., the coordinate system used for the data measured by the second sensor, denoted here as Oimu2 coordinate system), and the transformation matrix Te (front_imu_to_front_base) of the second pose coordinate system (Oimu2 coordinate system) to the second coordinate system (i.e., the coordinate system based on the second vehicle body, which is also artificially defined, denoted here as O2 coordinate system).

[0060] Therefore, the conversion matrix Ta (lidar_to_rear_base) can be represented by the first radar calibration data, the conversion matrix Te (front_imu_to_front_base) can be represented by the second calibration data, and the conversion matrix Tb (imu_to_rear_base.inverse) refers to the transformation from the O1 coordinate system to the Oimu1 coordinate system, while the first calibration data represents the transformation from the Oimu1 coordinate system to the O1 coordinate system. Therefore, the first calibration data needs to be inversely operated to obtain the conversion matrix Tb. It should be noted that the aforementioned calibration data can be calibrated after the first coordinate system with the first vehicle body as the reference and the second coordinate system with the second vehicle body as the reference are defined, that is, the calibration data is fixed and does not need to be calculated every time.

[0061] However, the conversion matrix Tc (rear_imu_pose) and the conversion matrix Td (front_imu_pose.inverse) change with the pose of the first vehicle body and the second vehicle body, and therefore need to be calculated every time. Specifically, the relative pose of the Oimu1 coordinate system in which the first data is located relative to the Oimu coordinate system is calculated by a matrix algorithm, and the conversion matrix Tc (rear_imu_pose) is obtained according to the relative pose. Similarly, the relative pose of the Oimu2 coordinate system in which the second data is located relative to the Oimu coordinate system is calculated by a matrix algorithm, and the conversion matrix Td (front_imu_pose.inverse) is obtained by inversely operating the calculated data.

[0062] Finally, the conversion matrices determined or calculated above are multiplied in sequence according to the fusion order to obtain a point cloud fusion matrix (T, that is, mix_martix), which is represented by the aforementioned matrices as follows:

[0063] T = Te * Td * Tc * Tb * Ta (that is, mix_martix = front_imu_to_front_base * front_imu_pose.inverse * rear_imu_pose * imu_to_rear_base.inverse * lidar_to_rear_base).

[0064] In an embodiment, the point cloud can also be sequentially transformed to the corresponding coordinate system in sequence, and the specific steps include: transforming the first point cloud to the first coordinate system according to the first radar calibration data to obtain the first converted point cloud; the first coordinate system takes the first vehicle body as a reference; performing inverse operation on the first calibration data to obtain the first conversion matrix, and transforming the first converted point cloud to the first pose coordinate system according to the first conversion matrix to obtain the second converted point cloud; the first pose coordinate system takes the first sensor as a reference; obtaining a preset pose coordinate system of the target vehicle; generating a second conversion matrix according to the relative pose relationship between the first data and the preset pose coordinate system, and transforming the second converted point cloud to the preset pose coordinate system according to the second conversion matrix to obtain a third converted point cloud; generating a third conversion matrix according to the relative pose relationship between the second data and the preset pose coordinate system, and transforming the third converted point cloud to the second pose coordinate system according to the third conversion matrix to obtain a fourth converted point cloud; the second pose coordinate system takes the second sensor as a reference; transforming the fourth converted point cloud to the target coordinate system according to the second calibration data to obtain the first target point cloud; the target coordinate system takes the second vehicle body as a reference.

[0065] As shown in Figure 5 , Figure 5 The process of collecting data and fusing point clouds provided by the embodiments of the present application is illustrated in detail. The foregoing has described the various coordinate systems, which will not be described again here. Let the first point cloud be P1, first transform the first point cloud P1 from the Orad1 coordinate system to the O1 coordinate system according to the conversion matrix Ta determined according to the first radar calibration data to obtain the first converted point cloud (i.e. Ta*P1); then, according to the description of the foregoing content, obtain the first conversion matrix Tb according to the first calibration data, and then transform the first converted point cloud from the O1 coordinate system to the Oimu1 coordinate system according to the first conversion matrix Tb to obtain the second converted point cloud (i.e. Tb*Ta*P1); then, according to the description of the foregoing content, transform the second converted point cloud from the Oimu1 coordinate system to the Oimu coordinate system according to the second conversion matrix Tc calculated to obtain the third converted point cloud (i.e. Tc*Tb*Ta*P1); then, according to the description of the foregoing content, transform the third converted point cloud from the Oimu coordinate system to the Oimu2 coordinate system according to the third conversion matrix Td calculated to obtain the fourth converted point cloud (i.e. Td*Tc*Tb*Ta*P1); finally, transform the fourth converted point cloud from the Oimu2 coordinate system to the O2 coordinate system according to the conversion matrix Te determined according to the second calibration data to obtain the first target point cloud, at this time the O2 coordinate system is the target coordinate system, and the first target point cloud is the point cloud in the target coordinate system (O2 coordinate system).

[0066] S203: Transform the second point cloud to the target coordinate system according to the second radar calibration data to obtain the second target point cloud.

[0067] According to step S202, the first point cloud measured by the first radar located on the trailer is moved to the front of the vehicle. In order to merge the point cloud measured by the first radar and the point cloud measured by the second radar, the point cloud measured by the second radar also needs to be moved to the target coordinate system.

[0068] like Figure 5 As shown, since the calibration data of the second radar reflects the position of the second radar on the second vehicle body, from the data level, it is the relative pose of the coordinate system of the second radar (i.e., the coordinate system used for the data measured by the second radar, denoted here as the Orad2 coordinate system) relative to the coordinate system of the second vehicle body (i.e., the coordinate system based on the second vehicle body, which is artificially defined, denoted here as the O2 coordinate system). This relative pose is the transformation matrix from the Orad2 coordinate system to the O2 coordinate system, i.e., the second radar transformation matrix Tf(lidar_to_front_base).

[0069] Therefore, if the second point cloud is denoted as P2, then the calculation is performed using Tf*P2, which transforms the second point cloud to the target coordinate system. The result is the second target point cloud. This transformation method achieves the goal of moving the second radar located at the front of the vehicle to a target coordinate system based on the front of the vehicle.

[0070] This application fully utilizes the data measured by the front-end sensor and the trailer sensor, and takes into account the different pose changes of the front-end and the trailer based on the data. Based on this, the coordinate system of the point cloud data collected by the front-end radar and the trailer radar is transformed, and the point cloud data collected by different vehicle radars are directly fused into the same coordinate system, thereby improving the fusion accuracy.

[0071] S204: Determine the fused point cloud based on the first target point cloud and the second target point cloud.

[0072] Based on the aforementioned steps, the first point cloud measured by the first radar located on the trailer is moved to the target coordinate system with the vehicle's front as the reference, thus obtaining the first target point cloud; simultaneously, the second point cloud measured by the second radar located on the vehicle's front is moved to the same target coordinate system, thus obtaining the second target point cloud. In this way, the point clouds measured by the first and second radars located in different areas of the same target vehicle are fused into the same coordinate system for presentation. Figure 6a As shown, Figure 6a This is a schematic diagram illustrating the effect of point cloud fusion before it is performed, using a trailer as an example. It shows the effect of the point clouds measured by the second radar located at the front of the vehicle and the first radar located at the trailer being in two different coordinate systems. Figure 6b As shown, Figure 6b This is a schematic diagram illustrating the effect of point cloud fusion provided in an embodiment of this application. Figure 6aThe point cloud fusion based on the illustrated diagram shows the effect of fusing the point clouds measured by the second radar arranged at the vehicle head and the first radar arranged at the trailer into the same coordinate system. It should be noted that after the point cloud fusion, part of the image recognition results can also be fused into the point cloud, for example, the point clouds of vehicles, pedestrians and specific obstacles are represented in different colors.

[0073] In an embodiment, both the first point cloud and the second point cloud are moved to the target coordinate system through the foregoing steps, but the first target point cloud and the second target point cloud still need to be processed accordingly to obtain the fusion point cloud of the large target object, and the specific steps include: obtaining a preset filtering condition; filtering the first target point cloud and the second target point cloud according to the preset filtering condition to obtain the fusion point cloud.

[0074] Specifically, the first target point cloud and the second target point cloud are both moved to the same coordinate system for presentation, at this time, the point clouds described by the first target point cloud and the second target point cloud may have a common dense distribution part, which can be regarded as an overlapping area, and therefore it is necessary to filter these point clouds according to the preset filtering condition to obtain the fusion point cloud.

[0075] In an embodiment, the filtering standard can be set according to the distance between the first target point cloud and the second target point cloud, so as to filter part of the point clouds in the overlapping part and avoid the case that the number of point clouds in the overlapping part is too large, and the specific steps include: traversing L points in the first target point cloud; when any Mth point in the L points is traversed, taking the Mth point as a reference point and a point in the second target point cloud that has not been removed as a to-be-processed point, calculating the distance between the reference point and the to-be-processed point, and performing Mth round filtering on the second target point cloud based on a preset distance threshold to remove the points in the to-be-processed point whose distance from the reference point is less than the preset distance threshold, to obtain the points in the second target point cloud that have not been removed after the Mth round filtering is completed; after the Lth round filtering of the second target point cloud is completed, obtaining the fusion point cloud according to the points in the second target point cloud that have not been removed and the first target point cloud.

[0076] Specifically, taking the first target point cloud as an example, for the L points in the first target point cloud, the L points are traversed, when any Mth point in the L points is traversed, the Mth point is taken as a reference point, and the points in the second target point cloud that have not been removed are taken as to-be-processed points, then the distance between the reference point and the to-be-processed points is calculated, assuming that there are 10 to-be-processed points in the second target point cloud whose distance from the reference point is less than the preset distance threshold, then the second target point cloud is filtered for the Mth time, and the 10 points in the second target point cloud are filtered out, to obtain the points in the second target point cloud that have not been removed after the Mth time of filtering; the L points in the first target point cloud are all traversed and valued, each point is taken as a reference point, and the points in the second target point cloud that have not been removed are taken as to-be-processed points, the distance between the reference point and the to-be-processed points is calculated, and the second target point cloud is filtered based on the preset distance threshold, until the L points in the first target point cloud are processed, that is, the Lth time of filtering of the second target point cloud is completed, to obtain the points in the second target point cloud that have not been removed, and the points in the second target point cloud that have not been removed and the first target point cloud are the fusion point cloud. It should be noted that the second target point cloud can also be taken as a reference, and the first target point cloud that is less than the preset distance threshold from the second target point cloud is filtered, and the remaining point cloud is the fusion point cloud.

[0077] In an embodiment, a certain framing condition can also be set to frame the overlapping part of the first target point cloud and the second target point cloud, and then the point cloud in the framed area is filtered, and the specific steps include: framing the overlapping area of the first target point cloud and the second target point cloud; filtering at least part of the first target point cloud or at least part of the second target point cloud in the overlapping area to obtain the remaining target point cloud in the overlapping area; combining the remaining target point cloud in the overlapping area, the first target point cloud outside the overlapping area, and the second target point cloud outside the overlapping area to obtain the fusion point cloud.

[0078] In an embodiment, the framing of the overlapping area can be performed according to the ratio of the point cloud quantity of each space unit, and the specific steps include: dividing the space range where the first target point cloud and the second target point cloud are located into a plurality of space units; calculating the total point cloud quantity in each space unit; determining target space units in the plurality of space units according to a preset quantity threshold and the total point cloud quantity of each space unit; calculating the point cloud statistical information of each target space unit, the point cloud statistical information representing at least one of the following: a first proportion of the first target point cloud in the total point cloud quantity of the target space unit, a second proportion of the second target point cloud in the total point cloud quantity of the target space unit, and a ratio of the first proportion to the second proportion; and taking a set of target space units in which the point cloud statistical information of the target space units satisfies a preset overlapping condition as the overlapping area.

[0079] Specifically, the space where the first target point cloud and the second target point cloud are located is divided into a plurality of space units, for example, the three-dimensional space of a large cube can be regarded as being divided into a plurality of small cube space units, then the number of point clouds in each space unit is calculated, the space units with a number of point clouds greater than a number threshold are found, and they are taken as target space units, then the point cloud statistical information of these target space units is calculated respectively, including at least one of the following: the first proportion of the first target point cloud in the total number of point clouds in the target space unit, the second proportion of the second target point cloud in the total number of point clouds in the target space unit, the ratio of the first proportion and the second proportion, then according to the calculated point cloud statistical information, it is judged whether the corresponding space unit has a more obvious point cloud overlap, if the point cloud statistical information is within a specified numerical range, it can be determined that there is a significant point cloud overlap, finally, all the space units determined to have a significant point cloud overlap are combined together to frame the overlap region. Through this calculation method, the overlapped point clouds are filtered, which lays a foundation for subsequent processing and reduces the difficulty of subsequent processing.

[0080] In addition, the preset framing condition can also be to divide the space where the first target point cloud and the second target point cloud are located into a plurality of space units, for example, the three-dimensional space of a large cube can be regarded as being divided into a plurality of small cube space units, then the first proportion of the first target point cloud in the total number of point clouds in each space unit (or the second proportion of the second target point cloud in the total number of point clouds in each space unit) is calculated, forming a three-dimensional matrix of proportion data; inputting the three-dimensional matrix into a trained neural network (such as a CNN convolutional neural network), the neural network can be used to find space units with significant point cloud overlap, and finally, all the space units determined to have a significant point cloud overlap are combined together to frame the overlap region. This way of framing the overlap region through the neural network improves the processing speed.

[0081] After the overlap region of the first target point cloud and the second target point cloud is framed, the first target point cloud or the second target point cloud in part or all of the overlap region is screened out, and the point cloud left in the overlap region is the target point cloud of the overlap region, and finally, the target point cloud of the overlap region and the point cloud outside the overlap region are combined, and the fusion point cloud of the first target point cloud and the second target point cloud is obtained.

[0082] It should be noted that the above-mentioned overlap region can be a complete continuous region, or a collection of a plurality of scattered sub-regions.

[0083] In the above scheme, in the process of filtering at least part of the first target point cloud or at least part of the second target point cloud in the overlapping region to obtain the remaining target point cloud in the overlapping region, because there is an OR relationship between the "at least part of the first target point cloud" and the "at least part of the second target point cloud", two cases are included, one is to filter out part or all of the second target point cloud while retaining all the first target point cloud, and the other is to filter out part or all of the first target point cloud while retaining all the second target point cloud.

[0084] Further, in order to completely and comprehensively depict the environment in which the target vehicle is located, even if the first target point cloud and the second target point cloud are fused together, there may be certain differences, for example, the point cloud part of a certain object (such as a person) in the first target point cloud and the point cloud part of the object in the second target point cloud may have a certain aliasing, which will make the subsequent object fusion, recognition, display and other processing extremely complex, and increase the difficulty of subsequent processing. In the above scheme, by selectively retaining the first target point cloud or the second target point cloud and filtering the other target point cloud, it can help to avoid or reduce the prominence of such aliasing in the fused point cloud.

[0085] Further, in the example, the average distribution density of the points in the first target point cloud in the overlapping region can be compared with the average distribution density of the points in the second target point cloud, and based on the average distribution density, it is determined whether the first target point cloud or the second target point cloud needs to be retained in its entirety; if the average distribution density corresponding to the first target point cloud is higher, then part or all of the second target point cloud is filtered out while retaining all the first target point cloud; if the average distribution density corresponding to the second target point cloud is higher, then part or all of the first target point cloud is filtered out while retaining all the second target point cloud.

[0086] In the above example, based on the average distribution density, the target point cloud to be retained can be selected, so that even if filtering occurs, there are still relatively more points available for depicting the environment.

[0087] In another example, the target object detection result in the overlapping region can also be further combined to determine whether the first target point cloud or the second target point cloud needs to be retained.

[0088] In the specific implementation process, the detected point cloud is usually subjected to target object recognition, for example, when the point cloud is played back afterwards, the point cloud of different target objects can be identified, and then corresponding colors can be configured, for example, the point cloud part of a detected person can be configured as yellow, and the point cloud part of a detected vehicle can be configured as blue. At this time, the detected point cloud needs to be subjected to target object recognition.

[0089] Further, in the scheme of identifying the target object in the target point cloud, the number of the target objects detected in the first target point cloud in the overlapping region can be compared with the number of the target objects detected in the second target point cloud.

[0090] If the number of the target objects detected in the first target point cloud is larger, and the difference is greater than a threshold, the first target point cloud is selected to be reserved, and part or all of the second target point cloud is filtered out.

[0091] If the number of the target objects detected in the second target point cloud is larger, and the difference is greater than the threshold, the second target point cloud is selected to be reserved, and part or all of the first target point cloud is filtered out.

[0092] If the difference is less than the threshold, whether the first target point cloud or the second target point cloud needs to be completely reserved is selected based on the average distribution density mentioned above.

[0093] Further, in the above example, even if filtering occurs, a relatively large number of target objects can still be depicted.

[0094] The objects can refer to all identifiable objects, including cars, people, obstacles, trees, road signs, and signs; further, the target objects can be part or all of them.

[0095] In another example, the target objects can refer to specific objects among all objects, and the specific objects can be adaptively changed according to the road section where the target vehicle is located, that is, the method further includes: obtaining actual road information of the road where the target vehicle is located, and determining the target objects based on the actual road information.

[0096] For example, if the actual road information indicates that the target vehicle is on a highway, the target objects can include cars, signs, and the like, but not people; if the actual road information indicates that the target vehicle is near a pedestrian street, the target objects can include people, road signs, and the like, but not cars; if the actual road information indicates that the target vehicle is on a wild road, the target objects can include trees.

[0097] The above is only an example, and in an actual scheme, the target objects can be determined according to a plurality of preset mapping relationships between road information and objects (one-to-many, many-to-one, or many-to-many), and the actual road information of the target vehicle.

[0098] Further, in the above scheme, the selected point cloud can be effectively ensured to be suitable for accurately depicting the real environment where the target vehicle is located.

[0099] In addition, in the process of reserving the first target point cloud and filtering out part or all of the second target point cloud, all of the second target point cloud can be filtered out, or a specified proportion of the second target point cloud can be filtered out, for example, a specified proportion of the second target point cloud can be uniformly filtered out. Similarly, in the process of reserving the first target point cloud and filtering out part or all of the second target point cloud, all of the second target point cloud can be filtered out, or a specified proportion of the second target point cloud can be filtered out, for example, a specified proportion of the second target point cloud can be uniformly filtered out.

[0100] In one embodiment, in the process of dividing the space range where the first target point cloud and the second target point cloud are located into a plurality of space units, a target bounding box can be determined for the first target point cloud and the second target point cloud, which is used as the space range. The first target point cloud and the second target point cloud are both in the target bounding box. In one example, the target bounding box can be a cuboid. In other examples, the target bounding box can also be spherical or other regular or irregular shapes. In one example, a first bounding box (for example, a cuboid, a sphere, or other regular or irregular shape) containing all the first target point clouds can be formed first, and a second bounding box (for example, a cuboid, a sphere, or other regular or irregular shape) containing all the second target point clouds can be formed. Then, the first bounding box and the second bounding box are combined to form the target bounding box. Further, when the target bounding box is used as the space range, it can better match the distribution of the first target point cloud and the second target point cloud. Furthermore, when the first bounding box, the second bounding box, and the target bounding box are all cuboids, the edges of the cuboids are parallel to the coordinate axes of the target coordinate system.

[0101] It should be noted that the above examples all take the first vehicle body as the trailer and the second vehicle body as the vehicle head, thereby describing the movement of the first point cloud measured by the first radar arranged on the trailer to the target coordinate system with the vehicle head as the reference, and the movement of the second point cloud measured by the second radar arranged on the vehicle head to the target coordinate system with the vehicle head as the reference, thereby realizing point cloud fusion. It can also be that the first vehicle body is regarded as the vehicle head and the second vehicle body is regarded as the trailer. According to the above scheme, the point cloud measured by the radar arranged on the vehicle head is moved to the target coordinate system with the trailer as the reference, and the point cloud measured by the radar arranged on the trailer is also moved to the target coordinate system with the trailer as the reference, thereby realizing point cloud fusion.

[0102] In the fusion of the point clouds measured by the radar arranged on the first vehicle body and the radar arranged on the second vehicle body, the data (including position data and / or attitude data) measured by the sensors arranged on the first vehicle body and the second vehicle body are used, the pose changes of the first vehicle body and the second vehicle body during driving of the target vehicle are fully considered, and the point cloud fusion is performed based on the changes, thereby ensuring that the point clouds measured by the radars can be effectively and directly fused together.

[0103] Based on the content of the above embodiments, the embodiment of the present application provides a point cloud fusion device for collecting data, which is suitable for fusing the point cloud detected by the radar on the target vehicle. The point cloud dynamic fusion device is used to execute the point cloud fusion method for collecting data provided in the above method embodiment. Specifically, please refer to Figure 7 The device comprises:

[0104] The data acquisition module 701 is configured to acquire the first point cloud measured by the first radar arranged on the first vehicle body of the target vehicle, the first radar calibration data of the first radar, the first data measured by the first sensor arranged on the first vehicle body, the first calibration data of the first sensor, the second point cloud measured by the second radar arranged on the second vehicle body, the second radar calibration data of the second radar, the second data measured by the second sensor arranged on the second vehicle body of the target vehicle, and the second calibration data of the second sensor. The first point cloud is in a first radar coordinate system, and the first radar coordinate system is based on the first radar. The second point cloud is in a second radar coordinate system, and the second radar coordinate system is based on the second radar. The first data comprises the position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data comprises the position data and / or attitude data of the second vehicle body measured by the second sensor.

[0105] The first transformation module 702 is configured to transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud. The target coordinate system is based on the second vehicle body.

[0106] The second transformation module 703 is configured to transform the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud.

[0107] The fusion module 704 is configured to determine a fused point cloud according to the first target point cloud and the second target point cloud.

[0108] In an embodiment, the target vehicle is a trailer vehicle comprising a vehicle head and a trailer part, and the pose of the trailer part relative to the vehicle head is variable.

[0109] If the first vehicle body is the vehicle head, the second vehicle body is the trailer part.

[0110] If the first vehicle body is the trailer part, the second vehicle body is the vehicle head.

[0111] In an embodiment, the first transformation module 702 comprises:

[0112] a fusion matrix determination module, configured to determine a point cloud fusion matrix according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data;

[0113] a first transformation submodule, configured to transform the first point cloud to the target coordinate system according to the point cloud fusion matrix to obtain the first target point cloud.

[0114] In an embodiment, the fusion matrix determination module comprises:

[0115] a first conversion matrix generation module, configured to perform inverse operation on the first calibration data to obtain a first conversion matrix;

[0116] a coordinate system acquisition module, configured to acquire a preset pose coordinate system of the target vehicle;

[0117] a second conversion matrix generation module, configured to generate a second conversion matrix according to a relative pose relationship between the first data and the preset pose coordinate system;

[0118] a third conversion matrix generation module, configured to generate a third conversion matrix according to a relative pose relationship between the second data and the preset pose coordinate system;

[0119] a fusion module, configured to generate the point cloud fusion matrix according to the first radar calibration data, the first conversion matrix, the second conversion matrix, the third conversion matrix, and the second calibration data.

[0120] In an embodiment, the first transformation module 702 further comprises:

[0121] a first transformation module, configured to transform the first point cloud to a first coordinate system according to the first radar calibration data to obtain a first conversion point cloud; the first coordinate system is based on the first vehicle body;

[0122] a second transformation module, configured to perform inverse operation on the first calibration data to obtain a first conversion matrix, and transform the first conversion point cloud to a first pose coordinate system according to the first conversion matrix to obtain a second conversion point cloud; the first pose coordinate system is based on the first sensor;

[0123] a coordinate system acquisition module, configured to acquire a preset pose coordinate system of the target vehicle;

[0124] a third transformation module, configured to generate a second conversion matrix according to a relative pose relationship between the first data and the preset pose coordinate system, and transform the second conversion point cloud to the preset pose coordinate system according to the second conversion matrix to obtain a third conversion point cloud.

[0125] a fourth transformation module, configured to generate a third conversion matrix according to the second data and a relative pose relationship of the preset pose coordinate system, and transform the third converted point cloud to a second pose coordinate system according to the third conversion matrix to obtain a fourth converted point cloud; the second pose coordinate system is based on the second sensor;

[0126] a target transformation module, configured to transform the fourth converted point cloud to a target coordinate system according to the second calibration data to obtain a first target point cloud; the target coordinate system is based on the second vehicle body.

[0127] In an embodiment, the fusion module 704 includes:

[0128] a first filtering module, configured to traverse L points in the first target point cloud; when traversing to any Mth point in the L points, taking the Mth point as a reference point and a point in the second target point cloud that has not been removed as a to-be-processed point, calculating a distance between the reference point and the to-be-processed point, and performing Mth round filtering on the second target point cloud based on a preset distance threshold to remove points in the to-be-processed point that are less than the preset distance threshold from the reference point, to obtain points in the second target point cloud that have not been removed after the Mth round filtering is completed;

[0129] a fusion point cloud determination module, configured to obtain a fusion point cloud according to the points in the second target point cloud that have not been removed and the first target point cloud after Lth round filtering of the second target point cloud is completed.

[0130] In an embodiment, the fusion module 704 further includes:

[0131] a region framing module, configured to frame an overlapping region of the first target point cloud and the second target point cloud;

[0132] a target point cloud determination module, configured to filter at least part of the first target point cloud or at least part of the second target point cloud in the overlapping region to obtain a remaining target point cloud in the overlapping region;

[0133] a point cloud combination module, configured to combine the remaining target point cloud in the overlapping region, the first target point cloud and the second target point cloud outside the overlapping region to obtain a fusion point cloud.

[0134] In an embodiment, the region framing module includes:

[0135] a space division module, configured to divide a space range in which the first target point cloud and the second target point cloud are located into a plurality of space units;

[0136] a quantity calculation module, configured to calculate a total point cloud quantity in each space unit respectively.

[0137] a unit determination module configured to determine target spatial units from the plurality of spatial units according to a preset quantity threshold and a total point cloud quantity of each spatial unit;

[0138] a ratio calculation module configured to calculate point cloud statistical information of each target spatial unit, the point cloud statistical information representing at least one of a first proportion of the first target point cloud in the total point cloud quantity of the target spatial unit, a second proportion of the second target point cloud in the total point cloud quantity of the target spatial unit, and a ratio of the first proportion to the second proportion;

[0139] a coincidence region determination module configured to determine a set of target spatial units in which the point cloud statistical information satisfies a preset coincidence condition as the coincidence region.

[0140] The point cloud dynamic fusion device provided in the embodiments of the present application can be used to execute the technical solutions of the foregoing method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0141] Different from the prior art, the point cloud dynamic fusion device provided in the present application is provided with a first transformation module, a second transformation module, and a fusion module. The first point cloud of the first vehicle body is moved to the second origin of the second vehicle body by the first transformation module to obtain the first target point cloud. The second point cloud of the second vehicle body is also moved to the second origin by the second transformation module to obtain the second target point cloud. The first target point cloud and the second target point cloud are fused by the fusion module, so as to achieve the purpose of directly fusing the point cloud data of different regions into the same coordinate system for presentation.

[0142] Correspondingly, the embodiments of the present application also provide an electronic device, such as Figure 8 As shown in the figure, the electronic device can include a processor 801 having one or more processing cores, a wireless (WiFi, Wireless Fidelity) module 802, a memory 803 having one or more computer readable storage media, an audio circuit 804, a display unit 805, an input unit 806, a sensor 807, a power supply 808, and a radio frequency (RF, Radio Frequency) circuit 809, and the like. Those skilled in the art can understand that Figure 8 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. Among them:

[0143] The processor 801 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines, executing various functions of the electronic device and processing data by running or executing software programs and / or modules stored in the memory 803 and calling data stored in the memory 803, thereby monitoring the entire electronic device. In an embodiment, the processor 801 can include one or more processing cores; preferably, the processor 801 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 801.

[0144] WiFi belongs to short-distance wireless transmission technology, and the electronic device can help users to send and receive emails, browse web pages, and access streaming media through the wireless module 802, which provides users with wireless broadband Internet access. Although Figure 8 The wireless module 802 is shown, but it can be understood that it does not belong to the necessary components of the terminal, and can be omitted as needed without changing the essence of the application.

[0145] The memory 803 can be used to store software programs and modules, and the processor 801 executes various functions and data processing by running the computer programs and modules stored in the memory 803. The memory 803 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one application program required by the function (such as sound playing function, image playing function, etc.), etc.; the data storage area can store the data created according to the use of the terminal (such as audio data, phone book, etc.), etc. In addition, the memory 803 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 803 can also include a memory controller to provide access for the processor 801 and the input unit 806 to the memory 803.

[0146] The audio circuit 804 includes a speaker, which can provide an audio interface between the user and the electronic device. The audio circuit 804 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal output; on the other hand, the speaker converts the collected sound signal into an electrical signal, which is received by the audio circuit 804 and converted into audio data, which is output to the processor 801 for processing, and then transmitted to another electronic device through the radio frequency circuit 809, or output to the memory 803 for further processing.

[0147] The display unit 805 can be used to display information input by a user or information provided to a user, as well as various graphical user interfaces of the terminal, which can be composed of graphics, text, icons, video, and any combination thereof. The display unit 805 can include a display panel, which in one embodiment can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, a touch-sensitive surface can cover the display panel, which, when detecting a touch operation thereon or proximate thereto, transmits to the processor 801 to determine the type of touch event, and then the processor 801 provides corresponding visual output on the display panel according to the type of touch event.

[0148] The input unit 806 can be used to receive inputted digital or character information, as well as to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls. Specifically, in one specific embodiment, the input unit 806 can include a touch-sensitive surface as well as other input devices. The touch-sensitive surface, also known as a touch display screen or touchpad, can collect touch operations of a user thereon or proximate thereto (such as operations of a user using a finger, a stylus, or any suitable object or accessory on or proximate to the touch-sensitive surface), and drive corresponding connections according to a pre-set program. In one embodiment, the touch-sensitive surface can include two parts, a touch detection device and a touch controller. The touch detection device detects the touch position of a user and detects signals resulting from touch operations, and transmits the signals to the touch controller; the touch controller receives touch information from the touch detection device and converts it into touch coordinates, and sends it to the processor 801, and can receive commands from the processor 801 and execute them. In addition, the touch-sensitive surface can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 806 can also include other input devices. Specifically, the other input devices can include one or more of a physical keyboard, function keys (such as volume control keys, on / off keys, etc.), a trackball, a mouse, a joystick, and the like.

[0149] The electronic device can further include at least one sensor 807, such as an inductive coil, a single / dual camera, a satellite navigation, a millimeter wave radar, a laser radar, a GPS module, an inertial measurement unit (IMU), through cooperation of which the surrounding environment can be sensed at any time during driving of the target, so as to collect various data. Specifically, the laser radar can detect the position, speed, and other characteristic quantities of the target through emission of a laser beam, so as to collect radar point cloud data; the main components of the inertial measurement unit (IMU) include a gyroscope, which is mainly used for measuring the three-axis attitude angle of the target, and in combination with the GPS module, the position and pose of the target can be measured, so as to collect real-time motion trajectories of the sensor and the like; as for other sensors that the electronic device can further be configured with, such as a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, no further description is given here.

[0150] The electronic device further includes a power supply 808 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 801 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 808 can further include one or more than one direct or alternating current power supply, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, and the like.

[0151] The radio frequency circuit 809 can be used for receiving and sending signals in the process of information or communication, in particular, receiving the downlink information of the base station and handing it over to one or more than one processor 801 for processing; in addition, sending data related to the uplink to the base station. Generally, the radio frequency circuit 809 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more than one oscillator, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like. In addition, the radio frequency circuit 809 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to the global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), electronic mail, short message service (SMS), and the like.

[0152] Although not shown, the electronic device can further include a Bluetooth module and the like, which will not be described herein. Specifically, in the present embodiment, the processor 801 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 803 according to the following instructions, and run the application program stored in the memory 803 by the processor 801, thereby realizing the following functions:

[0153] obtain a first point cloud measured by a first radar arranged on a first vehicle body of the target vehicle, first radar calibration data of the first radar, first data measured by a first sensor arranged on the first vehicle body, first calibration data of the first sensor, and a second point cloud measured by a second radar arranged on a second vehicle body of the target vehicle, second radar calibration data of the second radar, second data measured by a second sensor arranged on the second vehicle body, and second calibration data of the second sensor; wherein the first point cloud is in a first radar coordinate system, and the first radar coordinate system is based on the first radar; the second point cloud is in a second radar coordinate system, and the second radar coordinate system is based on the second radar; the first data includes position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data includes position data and / or attitude data of the second vehicle body measured by the second sensor;

[0154] transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud; wherein the target coordinate system is based on the second vehicle body;

[0155] transform the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud;

[0156] determine a fused point cloud according to the first target point cloud and the second target point cloud.

[0157] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling relevant hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0158] To this end, an embodiment of the present application provides a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to realize the functions of the above-mentioned data acquisition point cloud fusion method.

[0159] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0160] The above describes the point cloud fusion method and device for collecting data and the electronic device provided by the embodiments of the present application in detail. The principles and implementation manners of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A point cloud fusion method for collecting data, characterized in that, The method is suitable for fusing point clouds detected by radars on a target vehicle, and the method comprises the following steps: obtaining a first point cloud measured by a first radar arranged on a first vehicle body of the target vehicle, first radar calibration data of the first radar, first data measured by a first sensor arranged on the first vehicle body, first calibration data of the first sensor, a second point cloud measured by a second radar arranged on a second vehicle body of the target vehicle, second radar calibration data of the second radar, second data measured by a second sensor arranged on the second vehicle body, and second calibration data of the second sensor; wherein the first point cloud is in a first radar coordinate system, and the first radar coordinate system takes the first radar as a reference; the second point cloud is in a second radar coordinate system, and the second radar coordinate system takes the second radar as a reference; the first data comprises position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data comprises position data and / or attitude data of the second vehicle body measured by the second sensor; transforming the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud; wherein the target coordinate system takes the second vehicle body as a reference; transforming the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud; determining a fused point cloud according to the first target point cloud and the second target point cloud; wherein the target vehicle is a trailer vehicle comprising a vehicle head and a trailer part; the pose of the trailer part relative to the vehicle head is variable; if the first vehicle body is the vehicle head, the second vehicle body is the trailer part; if the first vehicle body is the trailer part, the second vehicle body is the vehicle head.

2. The method of claim 1, wherein, The step of transforming the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud, comprises the following steps: determining a point cloud fusion matrix according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data; transforming the first point cloud to the target coordinate system according to the point cloud fusion matrix, to obtain the first target point cloud.

3. The method of claim 2, wherein, The step of determining a point cloud fusion matrix according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, comprises the following steps: performing an inverse operation on the first calibration data to obtain a first conversion matrix; obtaining a preset pose coordinate system of the target vehicle; generating a second conversion matrix according to a relative pose relationship between the first data and the preset pose coordinate system; generating a third conversion matrix according to a relative pose relationship between the second data and the preset pose coordinate system; According to the first radar calibration data, the first conversion matrix, the second conversion matrix, the third conversion matrix, and the second calibration data, a point cloud fusion matrix is generated.

4. The method of claim 1, wherein, The step of transforming the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data to obtain a first target point cloud comprises: According to the first radar calibration data, the first point cloud is transformed to a first coordinate system to obtain a first conversion point cloud; the first coordinate system takes the first vehicle body as a reference; The first calibration data is subjected to an inverse operation to obtain a first conversion matrix, and the first conversion point cloud is transformed to a first pose coordinate system according to the first conversion matrix to obtain a second conversion point cloud; the first pose coordinate system takes the first sensor as a reference; A preset pose coordinate system of the target vehicle is obtained; According to the relative pose relationship between the first data and the preset pose coordinate system, a second conversion matrix is generated, and the second conversion point cloud is transformed to the preset pose coordinate system according to the second conversion matrix to obtain a third conversion point cloud; According to the relative pose relationship between the second data and the preset pose coordinate system, a third conversion matrix is generated, and the third conversion point cloud is transformed to a second pose coordinate system according to the third conversion matrix to obtain a fourth conversion point cloud; the second pose coordinate system takes the second sensor as a reference; According to the second calibration data, the fourth conversion point cloud is transformed to a target coordinate system to obtain a first target point cloud; the target coordinate system takes the second vehicle body as a reference.

5. The method of claim 1 to 4, wherein, The step of determining a fusion point cloud according to the first target point cloud and the second target point cloud comprises: L points in the first target point cloud are traversed; when any Mth point in the L points is traversed, the Mth point is taken as a reference point, and a point in the second target point cloud that has not been removed is taken as a to-be-processed point, a distance between the reference point and the to-be-processed point is calculated, and the second target point cloud is subjected to Mth round filtering based on a preset distance threshold to remove points in the to-be-processed point that are less than the preset distance threshold from the reference point, to obtain points in the second target point cloud that have not been removed after the Mth round filtering is completed; After Lth round filtering of the second target point cloud is completed, a fusion point cloud is obtained according to the points in the second target point cloud that have not been removed and the first target point cloud.

6. The method of claim 1 to 4, wherein, The step of determining a fusion point cloud according to the first target point cloud and the second target point cloud comprises: The overlapping region of the first target point cloud and the second target point cloud is framed; At least part of the first target point cloud or at least part of the second target point cloud in the overlapping region is filtered to obtain a remaining target point cloud in the overlapping region; The remaining target point cloud in the overlapping region, the first target point cloud, and the second target point cloud outside the overlapping region are combined to obtain a fusion point cloud.

7. The method of claim 6, wherein, The step of framing the overlapping region of the first target point cloud and the second target point cloud comprises: Divide a space range where the first target point cloud and the second target point cloud are located into a plurality of space units; Calculate a total point cloud quantity in each space unit respectively; Determine a target space unit from the plurality of space units according to a preset quantity threshold and the total point cloud quantity of each space unit; Calculate point cloud statistical information of each target space unit respectively, the point cloud statistical information representing at least one of the following: a first proportion of the first target point cloud in the total point cloud quantity of the target space unit, a second proportion of the second target point cloud in the total point cloud quantity of the target space unit, a ratio of the first proportion to the second proportion; Take a set of target space units in which the point cloud statistical information of the target space units satisfies a preset coincidence condition as the coincidence region.

8. A point cloud fusion apparatus for collecting data, characterized by, The device is suitable for fusing point clouds detected by radars on a target vehicle, and the device comprises: a data acquisition module, configured to acquire a first point cloud measured by a first radar arranged on a first vehicle body of the target vehicle, first radar calibration data of the first radar, first data measured by a first sensor arranged on the first vehicle body, first calibration data of the first sensor, a second point cloud measured by a second radar arranged on a second vehicle body of the target vehicle, second radar calibration data of the second radar, second data measured by a second sensor arranged on the second vehicle body, and second calibration data of the second sensor; wherein the first point cloud is in a first radar coordinate system, and the first radar coordinate system takes the first radar as a reference; the second point cloud is in a second radar coordinate system, and the second radar coordinate system takes the second radar as a reference; the first data comprises position data and / or attitude data of the first vehicle body measured by the first sensor, and the second data comprises position data and / or attitude data of the second vehicle body measured by the second sensor; a first transformation module, configured to transform the first point cloud to a target coordinate system according to the first radar calibration data, the first calibration data, the first data, the second data, and the second calibration data, to obtain a first target point cloud; wherein the target coordinate system takes the second vehicle body as a reference; a second transformation module, configured to transform the second point cloud to the target coordinate system according to the second radar calibration data, to obtain a second target point cloud; a fusion module, configured to determine a fused point cloud according to the first target point cloud and the second target point cloud. The target vehicle is a trailer comprising a trailer head and a trailer part, and a pose of the trailer part relative to the trailer head is variable. If the first vehicle body is the trailer head, the second vehicle body is the trailer part. If the first vehicle body is the trailer part, the second vehicle body is the trailer head.

9. An electronic device, comprising: The device comprises a processor and a memory, the memory is configured to store a computer program, and the processor is configured to run the computer program in the memory to execute the steps in the point cloud fusion method for collecting data according to any one of claims 1 to 7.

Images