Multi-sensor fusion method and device, electronic equipment and storage medium
By synchronizing clock source and time for lidar, millimeter-wave radar and binocular cameras, the information error problem caused by inconsistent sampling frequency in multi-sensor monitoring is solved, and the reliability and data accuracy of monitoring are improved.
Patent Information
- Application Number
- CN202411091476.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
AI Technical Summary
When using multiple sensing technologies such as lidar, millimeter-wave radar and binocular cameras for multi-source heterogeneous fusion monitoring, due to inconsistent sampling frequency, the information fusion system provides incorrect information, reducing the reliability of multi-sensor monitoring .
By synchronizing the clock source of the lidar, millimeter-wave radar and binocular cameras, and performing time based on the data collected by lidar and millimeter-wave radar at the same time frame, the point cloud acquisition interval of lidar and the image acquisition interval of binocular cameras Synchronization ensures that the clock frequency and phase of the sensor are consistent, thereby achieving time synchronization.
It effectively avoids information errors caused by inconsistent sensor sampling conditions, and improves the reliability and data accuracy of multi-sensor monitoring.
Smart Images

Figure CN119087422A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a multi-sensor fusion method, device, electronic device and storage medium. Background Art
[0002] According to the characteristics of the key sections of the "three spans" of the transmission lines, the application of new technologies such as lidar, millimeter-wave radar, binocular cameras, etc., to realize the identification and screening of external hidden dangers, tree obstacles in the channel, cross-crossing and other information, and safety warning is a new technical approach. It can effectively prevent accidents such as tower collapse, line breakage, and string drop in the "three spans", and put an end to major public safety and power grid safety incidents caused by the "three spans".
[0003] When using multiple sensing technologies such as laser radar, millimeter wave radar, and binocular camera for multi-source heterogeneous fusion monitoring, the sampling frequencies of millimeter wave radar, laser radar, and binocular camera are different. The sampling frequency of laser radar is generally 10Hz or 20Hz, and the sampling timestamp is based on the time when the laser radar is started as the zero point. The sampling frequency of millimeter wave radar is 30-50Hz, and the sampling frequency of binocular camera is about 20-30Hz. The sampling timestamp is based on the current hardware timestamp at the time of sampling as the zero point. If the sampling conditions of these three sensors are inconsistent at a certain moment, it will provide wrong information to the information fusion system, resulting in a large error, that is, the reliability of the current multi-sensor monitoring solution is low. Summary of the invention
[0004] The embodiments of the present application provide a multi-sensor fusion system, method, electronic device, and storage medium, which can fuse data from multiple sensors to improve the reliability of multi-sensor monitoring.
[0005] In a first aspect, an embodiment of the present application provides a multi-sensor fusion method, including:
[0006] Determine the LiDAR, millimeter wave radar, and binocular camera installed on the same physical architecture;
[0007] Synchronizing the clock sources of the laser radar, millimeter wave radar and binocular camera;
[0008] Performing time synchronization on the laser radar and the millimeter wave radar according to data collected by the laser radar and the millimeter wave radar in the same time frame;
[0009] Based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera, time synchronization is performed on the laser radar and the binocular camera;
[0010] The laser radar, the millimeter-wave radar and the binocular camera are time-synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
[0011] Optionally, in some embodiments of the present application, the step of synchronizing the clock sources of the laser radar, the millimeter wave radar, and the binocular camera includes:
[0012] Sending first detection information to the laser radar, millimeter wave radar and binocular camera respectively;
[0013] After receiving the second detection information sent by the laser radar, the millimeter-wave radar and the binocular camera, sending third detection information to the laser radar, the millimeter-wave radar and the binocular camera respectively;
[0014] The laser radar, millimeter-wave radar and binocular camera are synchronized with their clock sources according to the time when the laser radar, millimeter-wave radar and binocular camera receive the first detection information and the time when the laser radar, millimeter-wave radar and binocular camera receive the third detection information.
[0015] Optionally, in some embodiments of the present application, the time synchronization of the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame includes:
[0016] Acquire data collected by the laser radar and the millimeter wave radar in the same time frame;
[0017] Obtaining network delay, clock deviation of the laser radar, and clock deviation of the millimeter wave radar from unified clock source information;
[0018] The laser radar and the millimeter-wave radar are time synchronized according to the data collected by the laser radar and the millimeter-wave radar in the same time frame, the network delay, the clock deviation of the laser radar and the clock deviation of the millimeter-wave radar.
[0019] Optionally, in some embodiments of the present application, the time synchronization of the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame, the network delay, the clock deviation of the laser radar, and the clock deviation of the millimeter wave radar includes:
[0020] Determine the first frame data corresponding to the laser radar according to the data collected by the laser radar in the Nth frame time frame, the network delay and the clock deviation of the laser radar, wherein N is a positive integer greater than 0;
[0021] Determine a second frame of data corresponding to the laser radar according to the data collected by the millimeter-wave radar in the Nth frame time frame, the network delay, and the clock deviation of the millimeter-wave radar;
[0022] Based on the first frame data and the second frame data, the laser radar and the millimeter wave radar are time synchronized.
[0023] Optionally, in some embodiments of the present application, the step of performing time synchronization on the laser radar and the binocular camera based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera includes:
[0024] Obtaining the point cloud acquisition interval of the laser radar, and;
[0025] Obtaining an image acquisition interval of the binocular camera;
[0026] In the same sampling time, the laser radar and the binocular camera are time synchronized according to the point cloud acquisition interval and the image acquisition interval.
[0027] Optionally, in some embodiments of the present application, the laser radar and the binocular camera are time synchronized according to the point cloud acquisition interval and the image acquisition interval within the same sampling time, including:
[0028] Determine the data acquired by the laser radar within the same sampling time according to the point cloud acquisition interval, and;
[0029] Determining the data acquired by the binocular camera within the same sampling time according to the image acquisition interval;
[0030] Based on the data acquired by the laser radar within the same sampling time and the data acquired by the binocular camera within the same sampling time, the laser radar and the binocular camera are time synchronized.
[0031] Optionally, in some embodiments of the present application, the step of performing time synchronization on the laser radar, the millimeter-wave radar, and the binocular camera according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera includes:
[0032] Taking the laser radar as a reference, the laser radar, the millimeter-wave radar and the binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
[0033] In a second aspect, an embodiment of the present application provides a multi-sensor fusion device, including:
[0034] A determination module, used to determine the laser radar, millimeter wave radar and binocular camera installed on the same physical architecture;
[0035] A first synchronization module, used for synchronizing the clock sources of the laser radar, the millimeter wave radar and the binocular camera;
[0036] A second synchronization module is used to perform time synchronization on the laser radar and the millimeter wave radar according to data collected by the laser radar and the millimeter wave radar in the same time frame;
[0037] A third synchronization module is used to perform time synchronization on the laser radar and the binocular camera based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera;
[0038] The fourth synchronization module is used to perform time synchronization on the laser radar, the millimeter wave radar and the binocular camera according to the time synchronization information of the laser radar and the millimeter wave radar and the time synchronization information of the laser radar and the binocular camera.
[0039] Correspondingly, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of any of the above methods when executing the program.
[0040] The present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements any of the above methods when the computer program is executed by a processor.
[0041] The present application provides a multi-sensor fusion method, device, electronic device and storage medium. After determining a laser radar, a millimeter-wave radar and a binocular camera installed on the same physical architecture, the clock source of the laser radar, the millimeter-wave radar and the binocular camera are synchronized. Then, according to the data collected by the laser radar and the millimeter-wave radar in the same time frame, the laser radar and the millimeter-wave radar are time-synchronized. Then, based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera, the laser radar and the binocular camera are time-synchronized. Finally, according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera, the laser radar, the millimeter-wave radar and the binocular camera are time-synchronized. In the multi-sensor fusion solution provided in the present application, first, the clock source synchronization of the laser radar, millimeter-wave radar and binocular camera is performed to ensure that the clock frequency and phase of the laser radar, millimeter-wave radar and binocular camera are consistent, and then, the laser radar and millimeter-wave radar are time synchronized, and the laser radar and binocular camera are synchronized. After synchronization, the laser radar, millimeter-wave radar and binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In this way, the situation where the sampling conditions of the three sensors are inconsistent at a certain moment, causing the information fusion system to provide erroneous information, can be avoided, thereby improving the reliability of multi-sensor monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0043] Figure 1 is a flow chart of a multi-sensor fusion method provided in an embodiment of the present application;
[0044] Figure 2 It is a structural diagram of the physical architecture of the multi-sensor fusion method provided in the embodiment of the present application;
[0045] Figure 3 is a schematic diagram of the structure of a multi-sensor fusion device provided in an embodiment of the present application;
[0046] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with the examples detailed in the attached claims, or with some aspects of the present application.
[0048] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.
[0049] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0051] Embodiments of the present application provide a multi-sensor fusion method, device, electronic device, and storage medium.
[0052] Among them, the multi-sensor fusion device can be deployed in a terminal, and the terminal can be a smart phone, a tablet, an intelligent audio and video interactive device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a wearable intelligent device, a medical device, etc. As mentioned above, the terminal 102 is often configured with a display device, and the display device can also be a display, a display screen, a touch screen, etc. The touch screen can also be a touch screen, a touch panel, etc., but is not limited to this. The terminal can communicate with the server in a wired or wireless manner. The server can include an independently running server or a distributed server, or a server cluster composed of multiple servers. The terminal can include a mobile phone, a tablet computer or a personal computer (PC, Personal Computer).
[0053] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0054] The present application provides a multi-sensor fusion method, including: determining a laser radar, a millimeter-wave radar, and a binocular camera installed on the same physical architecture; synchronizing the clock sources of the laser radar, the millimeter-wave radar, and the binocular camera; synchronizing the laser radar and the millimeter-wave radar in time according to data collected by the laser radar and the millimeter-wave radar in the same time frame; synchronizing the laser radar and the binocular camera in time based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera; synchronizing the laser radar, the millimeter-wave radar, and the binocular camera in time according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
[0055] See also Figure 1 , Figure 1 : is a flow chart of the multi-sensor fusion method provided by the present application, and the multi-sensor fusion method specifically includes the following process:
[0056] 101. Determine the laser radar, millimeter wave radar and binocular camera installed on the same physical architecture.
[0057] The physical architecture refers to the specific physical structure or device used to support and fix the lidar, millimeter wave radar and stereo camera. This physical architecture enables these sensors to maintain a relatively fixed position and orientation in space, thereby realizing their spatial coordinate system. The sensors (i.e. lidar, millimeter wave radar, stereo camera and other sensors) are fixed on the same platform, which can be a tripod, bracket, vehicle or other structure to keep the sensors stable and determine their spatial relationship. By mounting all sensors on the same physical architecture, it is ensured that they share a unified spatial coordinate system. This means that the data collected by the sensors can be aligned and fused based on the same reference point and orientation. The physical architecture allows the position and orientation of each sensor relative to other sensors to be precisely defined, which is critical for subsequent data fusion and analysis. The physical architecture needs to be stable enough to prevent sensor movement caused by wind, vibration or other environmental factors, which may affect the measurement accuracy.
[0058] LIDAR (Light Detection and Ranging) is a remote sensing technology that uses lasers to measure distance and speed. It is widely used in many fields, including but not limited to geographic information systems (GIS), environmental monitoring, self-driving cars, aerial surveying, robot navigation, etc. LIDAR measures distance by emitting laser pulses and receiving the reflected light beams. The laser pulses emitted by the transmitter are reflected after encountering an object, and the receiver receives these reflected lights and records the time difference. Based on the speed of light and the recorded time difference, the round-trip distance of the light pulse can be calculated.
[0059] Millimeter Wave Radar (MMW Radar) is a radar system that uses millimeter wave band electromagnetic waves for detection and measurement. Millimeter waves generally refer to electromagnetic waves with a frequency range of 30 to 300 GHz and a wavelength between 1 mm and 10 mm.
[0060] A binocular camera, also known as a stereo camera or a binocular vision system, is a device that uses two cameras to simulate human binocular vision. It achieves depth perception and three-dimensional space measurement by imitating the human eyes to obtain stereo images. A binocular camera consists of two cameras arranged side by side or horizontally, which capture images of the same scene from slightly different angles. By comparing the difference between the images captured by the two cameras (i.e., parallax), the distance and three-dimensional structure of objects in the scene can be calculated.
[0061] Optionally, in some embodiments of the present application, laser radar, millimeter wave radar and binocular camera are used as sensors for data collection and are connected to the edge computing main control terminal through the main port, such as Figure 2 shown.
[0062] 102. Synchronize the clock sources of the laser radar, millimeter wave radar and binocular camera.
[0063] A clock source is a device or system that provides a time reference signal, which ensures the accuracy and consistency of time information. In various technical applications, clock sources are the basis for synchronous operations and time-sensitive tasks. A clock source is a device or system that provides a time reference signal, which ensures the accuracy and consistency of time information. In various technical applications, clock sources are the basis for synchronous operations and time-sensitive tasks. Clock source synchronization for LiDAR, millimeter-wave radar, and binocular cameras is to ensure that these sensors can operate under the same time reference, so that the data they collect can be accurately aligned and fused.
[0064] For example, a synchronization signal can be sent to each sensor, and the time when the sensor receives the synchronization signal can be recorded to calibrate and synchronize the hardware clock source of each sensor. Specifically, the edge computing master terminal (hereinafter referred to as the master terminal) can send synchronization signals to each sensor, and record the time when the sensor receives the synchronization signal, so as to calibrate and synchronize the hardware clock source of each sensor. The edge computing master terminal is the central node of the synchronization process and is responsible for sending synchronization signals to each connected sensor. The master terminal sends a synchronization signal containing time information to the ports of sensors such as laser radar, millimeter wave radar and binocular camera through its main port. After each sensor receives the synchronization signal, it will record the timestamp of the received signal. This timestamp reflects the time required from the master terminal sending the signal to the sensor receiving the signal. By comparing the time F1 when the master terminal sends the signal and the time F2 when the sensor receives the signal, the network delay can be calculated; further, by analyzing the timestamp of the sensor receiving the signal and the time of the master terminal, the clock deviation of the sensor can be determined. Finally, according to the calculated network delay and clock deviation, the clock of the sensor is calibrated to ensure that they are synchronized with the clock of the master terminal.
[0065] Optionally, in some embodiments of the present application, the step of “synchronizing the clock sources of the laser radar, the millimeter wave radar, and the binocular camera” may specifically include:
[0066] Sending first detection information to the laser radar, the millimeter wave radar, and the binocular camera respectively;
[0067] After receiving the second detection information sent by the laser radar, the millimeter-wave radar and the binocular camera, sending third detection information to the laser radar, the millimeter-wave radar and the binocular camera respectively;
[0068] The laser radar, millimeter-wave radar and binocular camera are synchronized with their clock sources according to the time when they receive the first detection information and the time when they receive the third detection information.
[0069] The main port of the edge computing master terminal sends information to each sensor port at a sending time of F1, and simultaneously records the time S when each sensor of the laser radar, millimeter wave radar, and binocular camera receives the information i (i=1,2,3).
[0070] The master port puts the F1 time parameter into the information packet and sends it to the receiving port of each sensor. After receiving the information packet, each sensor can parse F1 and obtain:
[0071] F1+Δt i +τi =S i
[0072] Δt i is the network delay of each sensor; τ i is the clock deviation of each sensor; the master port is ahead / lags behind the receiving port by τ i .
[0073] The receiving port of each sensor sends information to the main port, and the sending time F3 is recorded synchronously. The time when the main port receives the message is F4. Then the main port puts the F4 time parameter into the information packet and sends it to the receiving port of the sensor. After receiving it, the receiving port can parse F4 and get:
[0074] F3+Δt i -τ i =F4
[0075] Applying mathematical knowledge can solve:
[0076]
[0077] When obtaining the network delay Δt of each of the above sensors i , the clock deviation of each sensor τ i Finally, the clock source synchronization of multiple sensors is achieved through corresponding correction.
[0078] For example, in a security monitoring scenario, the relevant initial parameter conditions can be set as follows:
[0079] Δt1=1ms, Δt2=1ms, Δt3=2ms. τ1=3ms, τ2=2ms, τ3=5ms.
[0080] S1=9ms, S2=10ms, S3=15ms. F1=1ms. F3=2ms. F4=2ms.
[0081] After time T, the laser radar obtains a=15 sets of data, and the millimeter wave radar obtains b=12 sets of data.
[0082] After a certain period of time, the laser radar obtained j=50 sets of point cloud data, and the millimeter wave radar obtained k=35 sets of point cloud data.
[0083] The process of multi-sensor clock source synchronization is as follows:
[0084] F1+Δt i +τ i =S i
[0085] F3+Δt i -τ i=F4
[0086] Applying mathematical knowledge can solve:
[0087]
[0088] Δt2=5ms
[0089] Δt3=7.5ms
[0090]
[0091] τ2=4.5ms
[0092] τ3=7ms
[0093] 103. Time synchronization is performed on the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame.
[0094] Time synchronization is the process of ensuring that multiple devices or systems are aligned in time. In multi-sensor systems, such as LiDAR, mmWave radar, and stereo cameras, time synchronization is especially important because it allows the system to accurately correlate data points from different sensors, thereby improving data accuracy and reliability.
[0095] Time synchronization refers to ensuring the consistency of time measurement in multiple devices or systems. It focuses on whether the timestamps recorded by the devices at a specific moment are consistent. Clock source synchronization, on the other hand, refers to adjusting the clock sources of multiple devices to the same state to ensure that their clock frequencies and phases are consistent. Clock source synchronization is usually the basis of time synchronization. Only when the clock sources of the devices are synchronized can accurate time synchronization be achieved. For example, in a multi-sensor system, it is first necessary to ensure that the clock sources of all sensors are synchronized, and then time synchronization can be used to ensure that the data they collect at a specific moment has a consistent timestamp.
[0096] For example, the main port of the edge computing master terminal sends synchronization signals to the lidar and millimeter wave radar, and records the time when the sensor receives the signal. In this way, the network delay and clock deviation are calculated to calibrate and synchronize the hardware clock sources of each sensor.
[0097] Optionally, in some embodiments of the present application, the step of “synchronizing the laser radar and the millimeter wave radar in time according to the data collected by the laser radar and the millimeter wave radar in the same time frame” may specifically include:
[0098] Get the data collected by LiDAR and MMW radar in the same time frame;
[0099] Obtain network delay, LiDAR clock deviation, and millimeter wave radar clock deviation from the unified clock source information;
[0100] The laser radar and the millimeter-wave radar are time synchronized according to the data collected by the laser radar and the millimeter-wave radar in the same time frame, the network delay, the clock deviation of the laser radar and the clock deviation of the millimeter-wave radar.
[0101] For example, the edge computing master terminal is used as the central node of synchronization, which is responsible for sending synchronization signals and collecting timestamp data from sensors. Then, mathematical models and algorithms are used to process the collected data, calculate network delays and clock deviations, and finally, adjust the sensor clock based on the calculation results to achieve accurate time synchronization.
[0102] Specifically, the master terminal sends a synchronization signal F1 to the sensor and records the sending time. The sensor receives the synchronization signal and records the receiving time F2. Network delay is the propagation time of the signal from the master terminal to the sensor, and the calculation formula is △t=F2-F1. According to the time F2 when the sensor receives the synchronization signal and the time F1 when the master terminal sends the signal, the clock deviation of the lidar and millimeter-wave radar is calculated. Finally, the calculated clock deviation is used to adjust the internal clock of the lidar and millimeter-wave radar, and according to the adjusted clock, ensure that the lidar and millimeter-wave radar collect data in the same time frame.
[0103] Optionally, in some embodiments of the present application, the step of “synchronizing the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame, the network delay, the clock deviation of the laser radar, and the clock deviation of the millimeter wave radar” may specifically include:
[0104] Determine the first frame data corresponding to the laser radar according to the data collected by the laser radar in the Nth frame time frame, the network delay and the clock deviation of the laser radar, wherein N is a positive integer greater than 0;
[0105] Determine a second frame of data corresponding to the laser radar according to the data collected by the millimeter-wave radar in the Nth frame time frame, the network delay, and the clock deviation of the millimeter-wave radar;
[0106] Based on the first frame data and the second frame data, the laser radar and the millimeter wave radar are time synchronized.
[0107] For example, the laser radar collects data at the Nth frame time frame, where N is a positive integer greater than 0. Determine the network delay when the laser radar receives the synchronization signal, and determine the clock deviation of the laser radar, and then adjust the timestamp of the laser radar according to the network delay and clock deviation to determine the exact time of the first frame of data. Similarly, the millimeter wave radar collects data at the same Nth frame time frame, determine the network delay when the millimeter wave radar receives the synchronization signal, and determine the clock deviation of the millimeter wave radar, and then adjust the timestamp of the millimeter wave radar according to the network delay and clock deviation to determine the exact time of the second frame of data. Finally, compare the first frame data time of the laser radar and the second frame data time of the millimeter wave radar. If the two timestamps do not match, adjust them according to the difference to align the data frame times of the two sensors.
[0108] Specifically, the laser radar acquires group a data after time T, and the millimeter wave radar acquires group b data. At this time, the actual time for the laser radar to collect data is T-Δt1-τ1, and the laser radar acquires group j data. The actual time for the millimeter wave radar to collect data is T-Δt2-τ2, and the acquired data is:
[0109]
[0110] After another time T, the lidar obtains the next j sets of data.
[0111] Since the clock deviation is constant, it only needs to be calculated when processing data for the first time, and there is no need to accumulate it repeatedly. However, the network deviation exists every time data is received, so the next set of data still needs to be calculated.
[0112] The next set of data for millimeter wave radar is:
[0113]
[0114] The above [] indicates rounding.
[0115]
[0116] j / / k+1,else
[0117] Align the jth group of point lidar cloud data with the kth group of millimeter-wave radar data. Otherwise, align the jth group of lidar data with the k+1th group of millimeter-wave radar data.
[0118] 104. Based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera, the laser radar and the binocular camera are time synchronized.
[0119] Determine the laser radar point cloud data collection interval p, that is, the time interval between two consecutive sets of point cloud data collection. Determine the binocular camera image data collection interval q, that is, the time interval between two consecutive frames of image collection.
[0120] Under the unified time reference of the main control terminal, record the time when the laser radar and binocular camera start collecting data. Use the synchronization signal of the main control terminal to trigger the first data collection of the laser radar and binocular camera. According to the acquisition interval p and the initial synchronization time, calculate the timestamp of each group of laser radar point cloud data. According to the acquisition interval and the initial synchronization time, calculate the timestamp of each frame of the binocular camera. Determine the lowest common multiple of the acquisition cycles of the laser radar and the binocular camera as the synchronization cycle. Adjust the acquisition time of the laser radar and the binocular camera according to the lowest common multiple to ensure that they collect data at the same time at the end of the synchronization cycle. Optionally, in some embodiments of the present application, the time synchronization status of the laser radar and the binocular camera can be continuously monitored to ensure that they remain synchronized. At the end of each synchronization cycle, periodic time adjustments are performed as needed to compensate for possible clock drift or acquisition delays.
[0121] Optionally, in some embodiments of the present application, the step of “synchronizing the laser radar and the binocular camera in time based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera” may specifically include:
[0122] Get the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera;
[0123] In the same sampling time, the laser radar and binocular camera are synchronized according to the point cloud acquisition interval and the image acquisition interval.
[0124] Calculate the least common multiple LCM(p,q) of the point cloud acquisition interval and the image acquisition interval, which will be the synchronization period. Initialize and synchronize the start acquisition time of the LiDAR and the binocular camera under the unified time reference of the master terminal. At the beginning of each synchronization period, send a synchronization signal through the master terminal to trigger the LiDAR and binocular camera to collect data simultaneously.
[0125] Optionally, in some embodiments of the present application, the step of “synchronizing the laser radar and the binocular camera in time according to the point cloud acquisition interval and the image acquisition interval within the same sampling time” may specifically include:
[0126] According to the point cloud acquisition interval, the data acquired by the laser radar within the same sampling time is determined, and according to the image acquisition interval, the data acquired by the binocular camera within the same sampling time is determined;
[0127] Based on the data acquired by the laser radar within the same sampling time and the data acquired by the stereo camera within the same sampling time, the laser radar and the stereo camera are time synchronized.
[0128] For example, specifically, during the calculation process, assume that the interval between each set of point cloud data of the radar is p, and the interval between each set of images of the binocular camera is q. After a period of time T, the laser radar obtains m sets of point cloud data, and the binocular camera obtains n sets of images.
[0129] When the laser radar obtains m sets of data, the data obtained by the binocular camera is:
[0130]
[0131] The next set of images are:
[0132]
[0133] After a certain period of time, when the radar acquires m groups, the camera acquires n groups of images.
[0134] at this time
[0135] 105. According to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera, the laser radar, the millimeter-wave radar and the binocular camera are time synchronized.
[0136] After achieving time synchronization between two sensors, the timestamps of the time synchronization between two sensors are aligned by the least common multiple. The sampling data of the lidar is set as the reference standard, and the timestamps of the sampling data of the millimeter wave radar and the binocular camera are aligned by the least common multiple.
[0137] Optionally, in some embodiments of the present application, the step of “synchronizing the laser radar, the millimeter-wave radar, and the binocular camera according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera” may specifically include:
[0138] Taking the laser radar as a reference, the laser radar, the millimeter wave radar and the binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter wave radar and the time synchronization information of the laser radar and the binocular camera.
[0139] After achieving time synchronization between two sensors, the time stamps of the time synchronization between two sensors are aligned by the least common multiple. That is, after time 2jT, the sampling data of the lidar is set as the reference standard, and the timestamps of the sampling data of the millimeter wave radar and the binocular camera are aligned by the least common multiple.
[0140] Align the jmth group of point lidar cloud data with the km / km+1th group of millimeter-wave radar data; align the jmth group of point radar cloud data with the jn / jn+1th group of images. That is:
[0141]
[0142]
[0143] Through the above steps, the joint time synchronization of the three sensors, namely, lidar, millimeter-wave radar and binocular radar, can be completed through adaptive matching.
[0144] The above is the multi-sensor fusion process provided in the embodiment of the present application.
[0145] In summary, the embodiment of the present application, after determining the laser radar, millimeter-wave radar and binocular camera installed on the same physical architecture, performs clock source synchronization on the laser radar, millimeter-wave radar and binocular camera, then performs time synchronization on the laser radar and the millimeter-wave radar according to the data collected by the laser radar and the millimeter-wave radar in the same time frame, and then performs time synchronization on the laser radar and the binocular camera based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera. Finally, the laser radar, millimeter-wave radar and binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In the multi-sensor fusion solution provided in the present application, first, the clock source synchronization of the laser radar, millimeter-wave radar and binocular camera is performed to ensure that the clock frequency and phase of the laser radar, millimeter-wave radar and binocular camera are consistent, and then, the laser radar and millimeter-wave radar are time synchronized, and the laser radar and binocular camera are synchronized. After synchronization, the laser radar, millimeter-wave radar and binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In this way, the situation where the sampling conditions of the three sensors are inconsistent at a certain moment, causing the information fusion system to provide erroneous information, can be avoided, thereby improving the reliability of multi-sensor monitoring.
[0146] In order to better implement the multi-sensor fusion method of the embodiment of the present application, the embodiment of the present application also provides a multi-sensor fusion device, wherein the meanings of the terms are the same as those in the multi-sensor fusion system described above, and the specific implementation details can refer to the description in the system embodiment.
[0147] See also Figure 3 , Figure 3A schematic diagram of the structure of a multi-sensor fusion device provided in an embodiment of the present application, wherein the multi-sensor fusion device may specifically include a determination module 201, a first synchronization module 202, a second synchronization module 203, a third synchronization module 204, and a fourth synchronization module 205, which may be specifically as follows:
[0148] A determination module 201 is used to determine the laser radar, millimeter wave radar and binocular camera installed on the same physical architecture;
[0149] The first synchronization module 202 is used to synchronize the clock sources of the laser radar, the millimeter wave radar and the binocular camera.
[0150] The second synchronization module 203 is used to perform time synchronization on the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame.
[0151] The third synchronization module 204 is used to perform time synchronization on the laser radar and the binocular camera based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera.
[0152] The fourth synchronization module 205 is used to perform time synchronization on the laser radar, the millimeter wave radar and the binocular camera according to the time synchronization information of the laser radar and the millimeter wave radar and the time synchronization information of the laser radar and the binocular camera.
[0153] An embodiment of the present application provides a multi-sensor fusion device. After the determination module 201 determines that the laser radar, millimeter-wave radar and binocular camera are installed on the same physical architecture, the first synchronization module 202 synchronizes the clock sources of the laser radar, millimeter-wave radar and binocular camera. Then, the second synchronization module 203 synchronizes the laser radar and the millimeter-wave radar according to the data collected by the laser radar and the millimeter-wave radar in the same time frame. Then, the third synchronization module 204 synchronizes the laser radar and the binocular camera based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera. Finally, the fourth synchronization module 205 synchronizes the laser radar, the millimeter-wave radar and the binocular camera according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In the multi-sensor fusion solution provided in the present application, first, the clock source synchronization of the laser radar, millimeter-wave radar and binocular camera is performed to ensure that the clock frequency and phase of the laser radar, millimeter-wave radar and binocular camera are consistent, and then, the laser radar and millimeter-wave radar are time synchronized, and the laser radar and binocular camera are synchronized. After synchronization, the laser radar, millimeter-wave radar and binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In this way, the situation where the sampling conditions of the three sensors are inconsistent at a certain moment, causing the information fusion system to provide erroneous information, can be avoided, thereby improving the reliability of multi-sensor monitoring.
[0154] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0155] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0156] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0157] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and multi-sensor fusion methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0158] The electronic device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 303 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0159] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0160] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:
[0161] Determine the laser radar, millimeter-wave radar and binocular camera installed on the same physical architecture; synchronize the clock sources of the laser radar, millimeter-wave radar and binocular camera; synchronize the laser radar and millimeter-wave radar according to the data collected by the laser radar and the millimeter-wave radar in the same time frame; synchronize the laser radar and the binocular camera based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera; synchronize the laser radar, millimeter-wave radar and binocular camera according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
[0162] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0163] After determining that the laser radar, millimeter-wave radar and binocular camera are installed on the same physical architecture, the embodiment of the present application synchronizes the clock sources of the laser radar, millimeter-wave radar and binocular camera. Then, based on the data collected by the laser radar and the millimeter-wave radar in the same time frame, the laser radar and the millimeter-wave radar are time synchronized. Then, based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera, the laser radar and the binocular camera are time synchronized. Finally, based on the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera, the laser radar, millimeter-wave radar and the binocular camera are time synchronized. In the multi-sensor fusion solution provided in the present application, first, the clock source synchronization of the laser radar, millimeter-wave radar and binocular camera is performed to ensure that the clock frequency and phase of the laser radar, millimeter-wave radar and binocular camera are consistent, and then, the laser radar and millimeter-wave radar are time synchronized, and the laser radar and binocular camera are synchronized. After synchronization, the laser radar, millimeter-wave radar and binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera. In this way, the situation where the sampling conditions of the three sensors are inconsistent at a certain moment, causing the information fusion system to provide erroneous information, can be avoided, thereby improving the reliability of multi-sensor monitoring.
[0164] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0165] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the multi-sensor fusion methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0166] Determine the laser radar, millimeter-wave radar and binocular camera installed on the same physical architecture; synchronize the clock sources of the laser radar, millimeter-wave radar and binocular camera; synchronize the laser radar and millimeter-wave radar according to the data collected by the laser radar and the millimeter-wave radar in the same time frame; synchronize the laser radar and the binocular camera based on the point cloud collection interval of the laser radar and the image collection interval of the binocular camera; synchronize the laser radar, millimeter-wave radar and binocular camera according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
[0167] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0168] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0169] Since the instructions stored in the storage medium can execute the steps in any one of the multi-sensor fusion methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the multi-sensor fusion methods provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.
[0170] The above is a detailed introduction to a multi-sensor fusion method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A multi-sensor fusion method, characterized in that: include: Determine the LiDAR, millimeter wave radar, and binocular camera installed on the same physical architecture; Synchronizing the clock sources of the laser radar, millimeter wave radar and binocular camera; Performing time synchronization on the laser radar and the millimeter wave radar according to data collected by the laser radar and the millimeter wave radar in the same time frame; Based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera, time synchronization is performed on the laser radar and the binocular camera; The laser radar, the millimeter-wave radar and the binocular camera are time-synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
2. The multi-sensor fusion method according to claim 1, characterized in that: The clock source synchronization of the laser radar, the millimeter wave radar and the binocular camera includes: Sending first detection information to the laser radar, millimeter wave radar and binocular camera respectively; After receiving the second detection information sent by the laser radar, the millimeter-wave radar and the binocular camera, sending third detection information to the laser radar, the millimeter-wave radar and the binocular camera respectively; The laser radar, millimeter-wave radar and binocular camera are synchronized with their clock sources according to the time when the laser radar, millimeter-wave radar and binocular camera receive the first detection information and the time when the laser radar, millimeter-wave radar and binocular camera receive the third detection information.
3. The multi-sensor fusion method according to claim 1, characterized in that: The step of performing time synchronization on the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame includes: Acquire data collected by the laser radar and the millimeter wave radar in the same time frame; Obtaining network delay, clock deviation of the laser radar, and clock deviation of the millimeter wave radar from unified clock source information; The laser radar and the millimeter-wave radar are time synchronized according to the data collected by the laser radar and the millimeter-wave radar in the same time frame, the network delay, the clock deviation of the laser radar and the clock deviation of the millimeter-wave radar.
4. The multi-sensor fusion method according to claim 3, characterized in that: The time synchronization of the laser radar and the millimeter wave radar according to the data collected by the laser radar and the millimeter wave radar in the same time frame, the network delay, the clock deviation of the laser radar and the clock deviation of the millimeter wave radar includes: Determine the first frame data corresponding to the laser radar according to the data collected by the laser radar in the Nth frame time frame, the network delay and the clock deviation of the laser radar, wherein N is a positive integer greater than 0; Determine a second frame of data corresponding to the laser radar according to the data collected by the millimeter-wave radar in the Nth frame time frame, the network delay, and the clock deviation of the millimeter-wave radar; Based on the first frame data and the second frame data, the laser radar and the millimeter wave radar are time synchronized.
5. The multi-sensor fusion method according to claim 1, characterized in that: The step of performing time synchronization on the laser radar and the binocular camera based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera includes: Obtaining the point cloud acquisition interval of the laser radar, and; Obtaining an image acquisition interval of the binocular camera; In the same sampling time, the laser radar and the binocular camera are time synchronized according to the point cloud acquisition interval and the image acquisition interval.
6. The multi-sensor fusion method according to claim 5, characterized in that: The step of performing time synchronization on the laser radar and the binocular camera according to the point cloud acquisition interval and the image acquisition interval within the same sampling time includes: Determine the data acquired by the laser radar within the same sampling time according to the point cloud acquisition interval, and; Determining the data acquired by the binocular camera within the same sampling time according to the image acquisition interval; Based on the data acquired by the laser radar within the same sampling time and the data acquired by the binocular camera within the same sampling time, the laser radar and the binocular camera are time synchronized.
7. The multi-sensor fusion method according to claim 1, characterized in that: The step of performing time synchronization on the laser radar, the millimeter wave radar, and the binocular camera according to the time synchronization information of the laser radar and the millimeter wave radar and the time synchronization information of the laser radar and the binocular camera includes: Taking the laser radar as a reference, the laser radar, the millimeter-wave radar and the binocular camera are time synchronized according to the time synchronization information of the laser radar and the millimeter-wave radar and the time synchronization information of the laser radar and the binocular camera.
8. A multi-sensor fusion device, characterized in that: include: A determination module, used to determine the laser radar, millimeter wave radar and binocular camera installed on the same physical architecture; A first synchronization module, used for synchronizing the clock sources of the laser radar, the millimeter wave radar and the binocular camera; A second synchronization module is used to perform time synchronization on the laser radar and the millimeter wave radar according to data collected by the laser radar and the millimeter wave radar in the same time frame; A third synchronization module is used to perform time synchronization on the laser radar and the binocular camera based on the point cloud acquisition interval of the laser radar and the image acquisition interval of the binocular camera; The fourth synchronization module is used to perform time synchronization on the laser radar, the millimeter wave radar and the binocular camera according to the time synchronization information of the laser radar and the millimeter wave radar and the time synchronization information of the laser radar and the binocular camera.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program according to the steps of the multi-sensor fusion method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the multi-sensor fusion method according to any one of claims 1 to 7.
