Method, apparatus, and vehicle for synchronization

Abstract

This application provides a synchronization method, apparatus, and vehicle. The method includes determining a synchronization mode for a LiDAR and a camera device based on the resource status of a Complex Programmable Logic Device (CPLD) or a Field Programmable Gate Array (FPGA), wherein the synchronization mode for the LiDAR and camera device includes a first synchronization mode or a second synchronization mode; and synchronizing the LiDAR and camera device according to the synchronization mode. This method allows for the determination of the synchronization mode between the LiDAR and camera based on the resource status of the CPLD or FPGA, and enables time or spatiotemporal synchronization of the LiDAR and camera device according to the determined synchronization mode, thereby improving the accuracy and reliability of data fusion between the radar and camera devices.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and more specifically, to a synchronization method, apparatus, and vehicle. Background Technology

[0002] As vehicles become increasingly prevalent in daily life, driving safety is receiving more and more attention. Modern vehicles, especially intelligent vehicles, can be equipped with numerous sensors, and different types of sensors need to use the same system to collect and process data. During data collection and processing, it is necessary to unify the coordinate systems and clocks of these sensors, with the aim of ensuring that the same target appears at the same world coordinates across different types of sensors at the same time.

[0003] Among numerous sensors, time synchronization between LiDAR and cameras is particularly important. Currently, existing technologies use the timestamps of their data to determine whether they are synchronized. However, since LiDAR uses mechanical rotating scanning while cameras use instantaneous exposure, the number of frames that overlap in time is extremely small. This results in poor accuracy and reliability of data fusion between different sensors, making it difficult to achieve spatiotemporal synchronization between multiple sensors. Summary of the Invention

[0004] This application provides a synchronization method, apparatus, and vehicle. When there are sufficient resources of complex programmable logic devices (CPLDs) or field-programmable gate arrays (FPGAs), the initial azimuth angle of the lidar rotor can be controlled, thereby controlling the scanning angle of the lidar rotor to trigger the camera device to capture images within a corresponding range, achieving time synchronization and / or spatiotemporal synchronization between the lidar and the camera device. When CPLD or FPGA resources are insufficient, spatiotemporal matching between the lidar and the camera device can be achieved as much as possible based on the periodic matching of the lidar and the camera device. In this way, the sampling frequency and phase of the lidar and the camera device can be synchronized, improving the accuracy and reliability of data fusion between the lidar and the camera device.

[0005] In a first aspect, a time synchronization method is provided, comprising: determining a synchronization mode of a lidar and a camera device based on the resource status of a complex programmable logic device (CPLD) or a field-programmable gate array (FPGA), wherein the synchronization mode of the lidar and the camera device includes a first synchronization mode or a second synchronization mode; and synchronizing the lidar and the camera device according to the synchronization mode of the lidar and the camera device.

[0006] The camera device may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device may include still images or video stream information.

[0007] The synchronization mode between the LiDAR and the camera is determined based on the resource status of the CPLD or FPGA. Specifically, when CPLD or FPGA resources are insufficient, a first-time synchronization mode is used; when CPLD or FPGA resources are sufficient, a second-time synchronization mode is used. The first synchronization mode can refer to the camera being triggered by the CPLD or FPGA at its original frequency. The second synchronization mode can refer to the LiDAR's internal rotor rotating to a certain angle or several angles, triggering the CPLD or FPGA, thereby triggering the camera to expose.

[0008] It should be understood that sufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate more resources to process tasks, that is, the CPLD or FPGA has strong computing power and current idle capacity. Insufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate relatively few resources when processing tasks, that is, the computing power and current idle capacity are weak.

[0009] The fundamental principle of sensor data fusion is to perform multi-level and multi-spatial information complementarity and optimized combination processing on various sensors, ultimately producing a consistent interpretation of the observation environment. This process requires the full and rational allocation and use of multi-source data, and the ultimate goal of information fusion is to derive more useful information based on the separate observation information obtained from each sensor through multi-level and multi-faceted combination of information. However, data fusion between lidar and cameras is relatively difficult in the sensor data fusion process. Specifically, compared to cameras, lidar is a slow-scanning device, while cameras provide instantaneous exposure. In low-speed vehicle scenarios, achieving time synchronization between lidar and cameras is already very difficult. For high-speed vehicle scenarios, not only is improved accuracy and reliability of data fusion required from lidar and cameras, but spatiotemporal synchronization is also essential.

[0010] Traditional time synchronization methods simply mechanically timestamp the data collected by radar and cameras based on the same time reference. Further, motion compensation can be applied to the data collected by radar and cameras. However, since traditional methods only hand the data over to the algorithm to determine the time difference, they do not achieve synchronization of the sampling frequency and phase between radar and camera, and therefore cannot achieve spatiotemporal synchronization, thus failing to improve vehicle driving safety.

[0011] In this embodiment, the synchronization mode of the LiDAR and camera device can be determined based on the resource status of the CPLD or FPGA. When CPLD or FPGA resources are sufficient, a second synchronization mode can be used to control the initial azimuth angle of the LiDAR rotor, thereby controlling the scanning angle of the LiDAR rotor to trigger the camera device to capture images within the corresponding range, achieving time synchronization and / or spatiotemporal synchronization between the radar and camera device. When CPLD or FPGA resources are insufficient, a first synchronization mode is used, which matches the period of the LiDAR and camera device to achieve spatiotemporal matching between them as much as possible. In this way, the sampling frequency and phase between the radar and camera device can be synchronized, improving the accuracy and reliability of data fusion between the radar and camera device.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: dividing the lidar and camera device into sensor groups.

[0013] Specifically, when dividing sensor groups, any number of lidar and camera devices located at any position on the vehicle can be grouped into the same sensor group based on their layout on the vehicle. This allows for time synchronization or spatiotemporal synchronization of the lidar and camera devices located within the same sensor group.

[0014] In this embodiment, the lidar and camera devices can be divided into sensor groups according to the layout of the sensors on the vehicle and the needs of actual applications. The lidar and camera devices located in the same sensor group can be synchronized in time or in space, thereby improving the efficiency of data fusion between the lidar and camera devices.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the synchronization mode of the lidar and the camera device is the first synchronization mode. The method further includes: determining the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices; triggering the first camera device to expose based on the exposure frequency of the first camera device to obtain first data, wherein the camera device includes the first camera device; acquiring second data collected by the lidar when the first camera device is exposed; and performing synchronization processing on the first data and the second data if the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold.

[0016] The first threshold can be a pre-set value, determined based on the actual application of the LiDAR and camera synchronization method. For example, if high accuracy is required for the synchronization of the LiDAR and camera, the first threshold can be set to a larger value; if low accuracy is required, the first threshold can be set to a smaller value.

[0017] Alternatively, the exposure frequency of the camera device can be determined according to the following formula:

[0018]

[0019] Where fc represents the exposure frequency of each camera device, n represents the number of one or more camera devices and n is a positive integer, T L This indicates the scanning period of the lidar.

[0020] In this embodiment, the exposure frequency of the camera devices can be determined based on the scanning cycle of the LiDAR and the number of camera devices, and the camera devices are triggered to expose themselves according to the exposure frequency. After exposure, the LiDAR and camera devices are judged to meet the time synchronization or spatiotemporal synchronization requirements by determining whether the difference between the timestamps of the first and second data collected by the LiDAR and camera devices is less than or equal to a first threshold. Only when the time synchronization or spatiotemporal synchronization requirements are met is the result output to the algorithm application for processing. In this way, the LiDAR and camera devices can match more frame data in time, reducing the workload of the algorithm application and improving the accuracy and reliability of data fusion between the LiDAR and camera devices.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the rotation period of the lidar is the maximum value within a preset range.

[0022] The preset range refers to the predetermined range within which the radar's scanning cycle can be achieved. The value of the preset range can be determined based on the inherent properties of the radar. For example, if the radar's inherent scanning cycle has two settings, 50ms and 100ms, then the preset range is 50ms-100ms. Setting the LiDAR's scanning cycle to the maximum value of the preset range, as in the example above, would be setting the LiDAR's scanning cycle to 100ms.

[0023] In this embodiment, the LiDAR can be set to the slowest operating mode suitable for the work scenario, and the exposure frequency of the camera devices can be determined based on the LiDAR's scanning cycle and the number of camera devices in the sensor group, thereby triggering the camera devices to expose. This method allows the LiDAR to better coordinate with the exposure characteristics of the camera devices, enabling the LiDAR and camera to match more frames of data in time, further improving the accuracy and reliability of data fusion between the radar and camera devices.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the synchronization mode of the lidar and the camera device is the second synchronization mode, and the method further includes: setting the initial azimuth angle of the lidar to a first azimuth angle at a first moment, and arranging a first camera device in the direction of the first azimuth angle, the camera device including the first camera device; acquiring first data collected by the lidar and second data collected by the first camera device at the first moment; and performing synchronization processing on the first data and the second data.

[0025] In this embodiment of the application, by setting the initial azimuth angles of several lidars to be consistent, and by setting the initial azimuth angles of several lidars within the azimuth angle range of the first camera device in the camera device group, the several lidars and the first camera device are triggered synchronously, ensuring time synchronization or spatiotemporal synchronization between the several lidars and the first camera device.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining the exposure time of the second camera device as a second moment based on the positional relationship between the lidar, the first camera device, and the second camera device, wherein the camera device includes the second camera device; acquiring third data collected by the lidar and fourth data collected by the second camera device at the second moment; and performing synchronous processing on the third data and the fourth data.

[0027] Specifically, the exposure time of the second camera device can be determined as the second moment based on the angle between the line connecting the position of the lidar in the camera device group and the position of the first camera device, and the line connecting the position of the lidar and the position of the second camera device, and the exposure of the second camera device can be triggered at the second moment.

[0028] It should be understood that in the embodiments of this application, the second camera device may refer to a camera device located behind the first camera device within the camera device group that performs exposure, or it may refer to multiple camera devices located behind the first camera device that perform exposure.

[0029] In this embodiment, several lidar units can be set to the same initial azimuth angle. Time or spatiotemporal synchronization between multiple lidar units and camera devices is achieved through synchronous triggering of the first camera within the lidar and camera unit group, and triggering of the lidar and second camera according to a calculated sequence of times. This ensures the high requirements for spatiotemporal synchronization between the lidar and camera devices in high-speed vehicle driving scenarios. Specifically, the CPLD only performs angle detection once; that is, the CPLD only detects the angle between the first camera and the lidar. Subsequent detections are based on the set exposure sequence times, reducing the CPLD's resource consumption.

[0030] It should be understood that the triggering of the lidar and the second camera device according to the calculated sequence of time can mean that the second camera device triggers sequentially from the start of time to the end of time according to the calculated time order.

[0031] Secondly, a synchronization device is provided, comprising: a processing unit; the processing unit is configured to determine a synchronization mode of a lidar and a camera device based on the resource status of a CPLD or FPGA, wherein the synchronization mode of the lidar and the camera device includes a first synchronization mode or a second synchronization mode; the processing unit is further configured to synchronize the lidar and the camera device according to the synchronization mode.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to divide the lidar and camera device into sensor groups.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the synchronization mode of the lidar and the camera device includes a first synchronization mode. Specifically, the processing unit is used to determine the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices. The processing unit is also used to trigger the first camera device to expose based on its exposure frequency, thereby obtaining first data. The camera device includes the first camera device. The device further includes an acquisition unit used to acquire second data collected by the lidar when the first camera device is exposed. If the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold, the processing unit is used to perform synchronization processing on the first data and the second data.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to determine the exposure frequency of the camera device according to the following formula:

[0035]

[0036] Where fc represents the exposure frequency of each camera device, n represents the number of one or more camera devices and n is a positive integer, T L This indicates the scanning period of the lidar.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the synchronization mode of the lidar and the camera device is the second synchronization mode.

[0038] The processing unit is specifically configured to set the initial azimuth angle of the lidar to a first azimuth angle at a first moment, and to set a first camera device in the direction of the first azimuth angle, the camera device including the first camera device; the device further includes an acquisition unit, the acquisition unit being configured to acquire first data collected by the lidar and second data collected by the first camera device at the first moment; the processing unit is also configured to perform synchronous processing on the first data and the second data.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to determine the exposure time of the second camera device as the second moment based on the positional relationship between the lidar, the first camera device, and the second camera device, wherein the camera device includes the second camera device; the acquisition unit is further configured to acquire third data collected by the lidar and fourth data collected by the second camera device at the second moment; the processing unit is further configured to perform synchronous processing on the third data and the fourth data.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to determine the exposure time of the second camera device as the second moment based on the angle between the line connecting the position of the lidar and the position of the first camera device and the line connecting the position of the lidar and the position of the second camera device.

[0041] Thirdly, a synchronization device is provided, comprising: at least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory, the device being used to perform the methods in the above aspects.

[0042] Fourthly, a computer-readable medium is provided that stores program code, which, when run on a computer, causes the computer to perform the methods described in the preceding aspects.

[0043] Fifthly, a chip is provided, comprising: at least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory, the means being used to perform the methods in the foregoing aspects.

[0044] In a sixth aspect, a vehicle is provided, the vehicle comprising: at least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory, the processor in the vehicle being used to perform the methods in the foregoing aspects. Attached Figure Description

[0045] Figure 1This is a functional schematic diagram of a vehicle provided in an embodiment of this application.

[0046] Figure 2 This is a schematic diagram of radar and camera synchronization provided in an embodiment of this application.

[0047] Figure 3 This application provides a system architecture for synchronizing radar and camera devices.

[0048] Figure 4 This is another system architecture for synchronizing radar and camera devices provided in the embodiments of this application.

[0049] Figure 5 This is a schematic diagram of the sensor group division provided in the embodiments of this application.

[0050] Figure 6 This is the radar and camera device synchronization method 600 provided in the embodiments of this application.

[0051] Figure 7 This is the radar and camera synchronization method 700 provided in the first synchronization mode of the embodiments of this application.

[0052] Figure 8 This is the radar and camera synchronization method 800 provided in the second synchronization mode of the embodiments of this application.

[0053] Figure 9 This is the radar and camera synchronization device 900 provided in the embodiments of this application.

[0054] Figure 10 This is the radar and camera synchronization device 1000 provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0056] For ease of understanding, the following text will combine... Figure 1 Taking intelligent driving as an example, this application will introduce an example scenario to which the embodiments are applicable.

[0057] Figure 1 This is a functional schematic diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to fully or partially autonomous driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an autonomous driving strategy based on the analysis of the surrounding environmental information to achieve fully autonomous driving, or present the analysis results to the user to achieve partial autonomous driving.

[0058] Vehicle 100 may include various subsystems, such as infotainment system 110, sensing system 120, computing platform 130, and display device 140. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0059] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0060] Communication system 111 may include a wireless communication system that can communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system may utilize Wi-Fi and a wireless local area network (WLAN) to communicate. In some embodiments, the wireless communication system may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system may include one or more dedicated short range communications (DSRC) devices that can enable public and / or private data communication between vehicles and / or roadside stations.

[0061] The entertainment system 112 may include a central control screen, microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate it via touch. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0062] The navigation system 113 may include map services provided by a map provider to provide navigation for the vehicle 100. The navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map services provided by the map provider may be two-dimensional maps or high-precision maps.

[0063] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a positioning system 121, which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system; an inertial measurement unit (IMU) 122; a lidar 123; a millimeter-wave radar 124; an ultrasonic radar 126; and a camera device 126, or more thereof. The perception system 120 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a critical function for the safe operation of the vehicle 100.

[0064] The positioning system 121 can be used to estimate the geographical location of the vehicle 100.

[0065] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0066] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0067] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of the objects.

[0068] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0069] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still images or video stream information.

[0070] Some or all of the functions of vehicle 100 can be controlled by computing platform 130. Computing platform 130 may include processors 131 to 13n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. In addition, it can also be hardware circuitry designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. Furthermore, the computing platform 130 may also include a memory for storing instructions. Some or all of the processors 131 to 13n can call the instructions in the memory to execute them and achieve the corresponding functions.

[0071] The computing platform 130 can control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the sensing system 120). In some embodiments, the computing platform 130 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0072] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1 This should not be construed as a limitation on the embodiments of this application.

[0073] Autonomous vehicles traveling on roads, such as vehicle 100 above, can identify objects in their surrounding environment to determine adjustments to their current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the autonomous vehicle can be determined.

[0074] Optionally, vehicle 100 or its associated perception and computing devices (e.g., computing platform 130) can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. Vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of vehicle 100, such as the lateral position of vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc.

[0075] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., cars in adjacent lanes on the road).

[0076] The aforementioned vehicle 100 can be a car, truck, motorcycle, public vehicle, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, etc., and this application embodiment does not impose any special limitations.

[0077] To facilitate understanding of the embodiments of this application, the terminology used in the embodiments of this application will be introduced below.

[0078] (1) Complex programmable logic device (CPLD): CPLD is suitable for implementing various operations and combinational logic. A single CPLD contains several programmable array logics. The interconnections between logic blocks can also be programmed and recorded. By using this integrated approach, a single CPLD can implement circuits that would otherwise require thousands or even hundreds of thousands of logic gates.

[0079] (2) Field-programmable gate array (FPGA) is developed based on programmable logic devices. As a semi-custom circuit in special application integrated circuits, it not only makes up for the shortcomings of fully custom circuits, but also overcomes the limitation of the limited number of gate circuits in the original programmable logic controllers. Moreover, its internal logic can be repeatedly modified by the designer to correct errors in the program.

[0080] (3) Micro control unit (MCU), also known as processing unit, control unit or single chip microcomputer (SCM), refers to a computer whose CPU, RAM, ROM, timer and various I / O interfaces are integrated on a single chip with the emergence and development of large-scale integrated circuits, forming a chip-level computer, which can perform different combination control for different application scenarios.

[0081] (4) Internet service provider (ISP), also known as Internet service provider, Internet service provider, or network service provider, refers to a company that provides Internet access services.

[0082] (5) (gigabit multimedia serial links, GMSL) is a high-speed serial interface suitable for the transmission of video, audio and control signals.

[0083] (6) Local area network switch (LANS): refers to a device used for data exchange within a switched local area network.

[0084] (7) Pulse per second (PPS): An abbreviation for pulses per second, used in the communications industry.

[0085] (8) Motion compensation: is a method to describe the difference between adjacent frames. Specifically, it describes how each small block in the previous frame moves to a certain position in the current frame.

[0086] (9) LiDAR: Also known as light detection and ranging (LDR), it is a sensing technology that uses a light source and a receiver to detect and range remotely. In the automotive field, LiDAR is used for detecting and modeling obstacles around vehicles.

[0087] (10) Time synchronization: A unified host provides a reference time to each sensor. Each sensor adds a timestamp to its independently collected data based on its own calibrated time. This can achieve timestamp synchronization of all sensors. However, since the collection cycles of each sensor are independent, it may not be possible to guarantee that the same information is collected at the same time.

[0088] (11) Spatiotemporal synchronization: Transform the measured values ​​of different sensor coordinate systems into the same coordinate system. When the laser sensor is moving at high speed, the intra-frame displacement calibration at the current speed needs to be considered.

[0089] (12) Hard synchronization: Using the same hardware to issue trigger acquisition commands simultaneously to achieve time synchronization of acquisition and measurement by various sensors. This ensures that the same information is acquired at the same time.

[0090] Figure 2 This is a schematic diagram of radar and camera synchronization provided in an embodiment of this application. Figure 2 The radar and camera synchronization method in this paper can be applied to... Figure 1 During the driving of vehicle 100.

[0091] The fundamental principle of sensor data fusion is to perform multi-level and multi-spatial information complementarity and optimized combination processing from various sensors, ultimately producing a consistent interpretation of the observed environment. This process requires the full utilization and rational allocation of multi-source data. The ultimate goal of information fusion is to derive more useful information based on the separate observation information obtained from each sensor, through multi-level and multi-faceted combination of information. This not only leverages the advantages of multiple sensors operating collaboratively but also comprehensively processes data from other information sources to enhance the intelligence of the entire sensor system.

[0092] However, data fusion between LiDAR and cameras is relatively difficult in the sensor data fusion process. Specifically, compared to cameras, LiDAR is a slow scanning device, while cameras provide instantaneous exposure. In scenarios where vehicles are traveling at low speeds, achieving time synchronization between LiDAR and cameras is already very difficult. For scenarios where vehicles are traveling at high speeds, not only is it required that LiDAR and cameras improve the accuracy and reliability of data fusion, but also achieve spatiotemporal synchronization.

[0093] like Figure 2As shown in (a), in scenarios where vehicles are traveling at high speeds, if the radar and camera cannot achieve spatiotemporal synchronization, the point cloud scanned by the radar will deviate from the actual object to be photographed, resulting in a certain error between the radar scan results and the actual situation, which may endanger the driving safety of the vehicle. Traditional time synchronization methods simply mechanically timestamp the data collected by the radar and camera based on the same time reference. Furthermore, motion compensation can be applied to the data collected by the radar and camera. However, since traditional methods only hand the data over to the algorithm to determine the time difference, they do not achieve synchronization of the sampling frequency and phase between the radar and camera, and therefore cannot achieve spatiotemporal synchronization, thus failing to improve the driving safety of the vehicle.

[0094] This application provides a radar and camera synchronization method. When CPLD resources are sufficient, the initial azimuth angle of the lidar rotor can be controlled, thereby controlling the scanning angle of the lidar rotor to trigger camera capture within a corresponding range, achieving time synchronization and / or spatiotemporal synchronization between the radar and camera. When CPLD resources are insufficient, spatiotemporal matching between the radar and camera can be achieved as much as possible based on the periodic matching of the lidar and camera. Figure 2 As shown in (b), after the lidar and camera are triggered synchronously, the point cloud scanned by the lidar can completely cover the actual object to be photographed, thus achieving spatiotemporal synchronization between the lidar and the camera.

[0095] It should be understood that sufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate more resources to process tasks, that is, the CPLD or FPGA has strong computing power and current idle capacity. Insufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate relatively few resources when processing tasks, that is, the computing power and current idle capacity are weak.

[0096] For example, in an autonomous driving system, after the LiDAR is powered on, it will rotate periodically according to its own operating mode, such as rotating 360° periodically. The camera needs to trigger a trigger signal from the CPLD / FPGA in the domain controller according to the camera's original operating frequency to trigger the camera module's exposure. This can be achieved in the following two ways:

[0097] (1) The camera is triggered by the CPLD / FPGA at the original frequency. This method can be called the first synchronization mode or the soft synchronization mode.

[0098] (2) When the internal rotor of the lidar rotates to a certain angle or several angles, it triggers the CPLD / FPGA to trigger the camera exposure. This method can be called the second synchronization mode or the hard synchronization mode.

[0099] Figure 3This application provides a system architecture for synchronizing radar and camera devices. Figure 3 The system architecture that synchronizes radar and camera devices can be applied to Figure 1 Of the 100 vehicles. Figure 3 Specifically, this may include the following steps:

[0100] S301, configured sensor layout and synchronization architecture.

[0101] Specifically, based on the layout of the radar and camera sensors on the vehicle, different sensors can be divided into different sensor groups, and then time synchronization between the radar and camera devices can be achieved according to the divided sensor groups. When dividing the sensor groups, any number of sensors in any position can be selected as sensor groups according to actual needs.

[0102] The camera device may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device may include still images or video stream information.

[0103] S302, set the synchronization mode for multiple sensors.

[0104] Specifically, the multi-sensor synchronization mode can be set according to the resource availability of the CPLD or FPGA. Hard synchronization mode is used when CPLD or FPGA resources are sufficient, while soft synchronization mode is used when CPLD or FPGA resources are insufficient. The specific setting method can be to configure the time synchronization mode of multiple sensors using a switch on the domain controller.

[0105] In soft synchronization mode, the camera device is triggered by the CPLD / FPGA at its original frequency. In hard synchronization mode, the internal rotor of the LiDAR rotates to a certain angle or several angles, triggering the CPLD / FPGA and thus the camera device to expose itself.

[0106] S303 determines whether the current mode is hard synchronization mode or soft synchronization mode.

[0107] Specifically, the determination can be made based on the result of setting the synchronization mode in step S302. If the synchronization mode is set to hard synchronization mode through the domain controller switch in step S302, then the synchronization mode determined here is hard synchronization mode; if the synchronization mode is set to soft synchronization mode through the domain controller switch in step S302, then the synchronization mode determined here is soft synchronization mode.

[0108] S304, align the initial azimuth angles of m lidars.

[0109] Specifically, if step S303 determines that the synchronization mode is hard synchronization mode, then in this step, the initial azimuth angles of m lidars are aligned, where m is a positive integer. Aligning the initial azimuth angles of the lidars can mean setting the rotor angles of different lidars to the same value.

[0110] S305, when the rotor of the lidar reaches the first azimuth angle range of the camera device group, the CPLD sends an exposure signal to trigger the camera device and records time t1.

[0111] Specifically, when the LiDAR rotor reaches the first azimuth angle range of the first camera in the camera assembly, the LiDAR sends a signal to the CPLD. Upon receiving the LiDAR signal, the CPLD sends an exposure signal to the first camera to trigger exposure, and records the exposure time t1. When the first camera exposes, it indicates that the first camera and the LiDAR have synchronized. The first azimuth angle range of the camera assembly can be preset according to the parameters of the camera or the needs of the actual application.

[0112] S306, the CPLD sets the time t2 to tn corresponding to the azimuth angle range of the 2nd to nth camera devices according to the sensor layout and time synchronization architecture. n .

[0113] Specifically, the CPLD can set the time t2 to tn when the lidar rotor arrives at the azimuth range of the 2nd to nth camera in the camera group, based on the sensor layout and time synchronization architecture. n , where n is an integer greater than 2.

[0114] S307, CPLD according to time t2 to t n Exposure signals are sent sequentially to trigger the camera device to expose.

[0115] Specifically, the CPLD sets the time from t2 to t according to step S306. n Exposure signals are sent sequentially to the camera device to trigger its exposure. In this way, the lidar and the second to nth camera devices in the camera device group can be synchronized in time or in space.

[0116] S308, determine whether to exit autonomous driving mode.

[0117] Specifically, if time synchronization or spatiotemporal synchronization of the radar and camera is achieved in step S307, then the system is determined to exit the automatic driving mode and the synchronization process ends in this step. If time synchronization or spatiotemporal synchronization of the radar and camera is not achieved in step S307, then step S305 is executed again to restart the synchronization process of the radar and camera.

[0118] In this embodiment, the sampling frequency and phase between the radar and camera are synchronized by triggering the radar and camera simultaneously, thereby improving the accuracy and reliability of data fusion between the radar and camera.

[0119] Figure 4 This is another system architecture for synchronizing radar and camera devices provided in the embodiments of this application. Figure 4 The system architecture that synchronizes radar and camera devices can be applied to Figure 1 Of the 100 vehicles. Figure 4 Specifically, this may include the following steps.

[0120] S401, configuring sensor layout and synchronization architecture.

[0121] Specifically, based on the layout of the sensors on the vehicle, different radar and camera devices can be divided into different sensor groups, and then synchronization between the radar and camera devices can be achieved according to the divided sensor groups. When dividing the sensor groups, any number of sensors in any position can be selected as sensor groups according to actual needs.

[0122] The camera device may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device may include still images or video stream information.

[0123] S402, set the time synchronization mode for multiple sensors.

[0124] Specifically, the multi-sensor synchronization mode can be set according to the resource availability of the CPLD or FPGA. Hard synchronization mode is used when CPLD or FPGA resources are sufficient, while soft synchronization mode is used when CPLD or FPGA resources are insufficient. The specific setting method can be to configure the time synchronization mode of multiple sensors using a switch on the domain controller.

[0125] In soft synchronization mode, the camera device is triggered by the CPLD / FPGA at its original frequency. In hard synchronization mode, the internal rotor of the LiDAR rotates to a certain angle or several angles, triggering the CPLD / FPGA and thus the camera device to expose itself.

[0126] S403 determines whether the current mode is hard synchronization mode or soft synchronization mode.

[0127] Specifically, the determination can be made based on the result of setting the time synchronization mode in step S402. If the synchronization mode is set to hard synchronization mode through the domain controller switch in step S402, then the synchronization mode determined here is hard synchronization mode; if the synchronization mode is set to soft synchronization mode through the domain controller switch in step S402, then the synchronization mode determined here is soft synchronization mode.

[0128] S404 sets the radar frequency to the lowest setting.

[0129] Specifically, if step S403 determines that the time synchronization mode is soft synchronization mode, then in this step the radar frequency is set to the lowest level, that is, the radar working cycle is the slowest. Preferably, the radar scanning cycle is set to 100ms.

[0130] The slowest working cycle of a radar can refer to the longest scanning cycle among the selectable working modes. For example, a lidar is set to two modes by the factory: 100ms and 50ms. The mode with the longest cycle is 100ms. Therefore, when the scanning cycle of a lidar is 100ms, the radar's working cycle can be called the "slowest".

[0131] S405 calculates the frequency of the camera device based on the period of the lidar and the layout of the multiple sensors.

[0132] Specifically, the frequency fc of the camera device can be calculated using the following formula.

[0133]

[0134] Where n represents the number of camera devices, and its value is a positive integer, T L This represents the scanning cycle of the lidar. For example, if one lidar corresponds to a layout of 6 camera devices, and the lidar's selection cycle is 100ms, then the frequency of the camera devices is fc = 6 / 0.1s = 60Hz.

[0135] S406, the lidar is rotating normally, and the CPLD triggers the camera device to expose according to the PPS of the fc frequency multiple.

[0136] Specifically, the exposure frequency fc of the camera device is calculated using the formula in step S406, and the CPLD triggers the camera device to expose using PPS multiples of fc. Here, the value of fc multiples is a positive integer. The frequency multiples mentioned in this application embodiment are also known as frequency multipliers.

[0137] S407, determine whether the data timestamps of the lidar and camera device are less than or equal to the first threshold.

[0138] Specifically, the system obtains the timestamps of the LiDAR and camera data when the camera device is exposed, determines whether the difference between the timestamps of the LiDAR and camera device is less than or equal to a first threshold, and if the condition is met, it indicates that the requirements for time synchronization or spatiotemporal synchronization between the radar and camera device are met. The data collected by the radar and camera device is then output to the algorithm application for processing.

[0139] S408, determine whether to exit autonomous driving mode.

[0140] Specifically, if in step S407 the timestamps of the LiDAR and the camera are less than or equal to the first threshold, meeting the requirements for time synchronization or spatiotemporal synchronization between the LiDAR and the camera, then in this step, it is determined that the automatic driving mode should be exited and the synchronization process should be terminated. If in step S407 the requirements for time synchronization between the LiDAR and the camera are not met, then step S406 is re-executed, and the CPLD triggers the camera to perform exposure again according to the PPS of the fc frequency multiple.

[0141] In this embodiment, the lidar can be set to the slowest working mode suitable for the work scenario, and the camera device can be set to a corresponding faster working mode. This enables the lidar and camera device to match more frames of data in time when hard synchronization cannot be achieved due to resource limitations, thereby improving the accuracy and reliability of data fusion between the lidar and camera device.

[0142] Figure 5 This is a schematic diagram of the sensor group provided in the embodiments of this application. Figure 5 The sensor group in the middle can be applied to Figure 3 or Figure 4 In a system architecture that synchronizes radar and camera devices.

[0143] As an illustrative example, such as Figure 5 As shown, two radars and seven camera devices can be deployed on a vehicle. ① in the diagram represents the radar, and ② represents the camera devices. When dividing the sensor groups, any number of sensors from any location can be selected based on actual needs. For example, as... Figure 5 As shown in (a), a sensor group may include radar and camera devices. For example, sensor group A includes all the radars and all the camera devices shown in the figure, thus synchronizing the two radars and seven camera devices in time. Another example is... Figure 5 As shown in (b), a radar and a camera located at the front of the vehicle can be classified as sensor group A, and a radar and six cameras located on the top of the vehicle can be classified as sensor group B. In this way, the radar and camera devices located in synchronization groups A and B can perform time synchronization within their respective sensor groups. The sensor groups described in this application embodiment can also be called synchronization groups, and this application does not make that distinction.

[0144] Specifically, when dividing different sensor groups, the radar and camera devices can be divided into synchronization groups using graphical configuration software, and the division results can be sent to the time synchronization module of the sensors.

[0145] In this embodiment, the radar and camera devices can be divided into sensor groups according to the layout of the sensors on the vehicle and the needs of actual applications. The radar and camera devices located in the same sensor group can be hard synchronized or soft synchronized, which can improve the efficiency of data fusion between the radar and camera devices.

[0146] The following is combined with Figure 6 The process shown describes the synchronization method for radar and camera devices.

[0147] Figure 6 This application provides a synchronization method 600 for radar and camera devices. This synchronization method 600 can be applied to... Figure 1 In vehicle 100. Method 600 may include the following steps.

[0148] S601, determine the time synchronization mode of the lidar and camera device.

[0149] The synchronization mode between the LiDAR and the camera device can be determined based on the resource status of the CPLD or FPGA. A first-time synchronization mode is used when CPLD or FPGA resources are insufficient, and a second-time synchronization mode is used when CPLD or FPGA resources are sufficient. The first synchronization mode refers to the camera device being triggered by the CPLD or FPGA at its original frequency. The second synchronization mode refers to the LiDAR's internal rotor rotating to a certain angle or several angles, triggering the CPLD or FPGA and thus triggering the camera device to expose itself.

[0150] It should be understood that sufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate more resources to process tasks, that is, the CPLD or FPGA has strong computing power and current idle capacity. Insufficient CPLD or FPGA resources can mean that the CPLD or FPGA can allocate relatively few resources when processing tasks, that is, the computing power and current idle capacity are weak.

[0151] In one possible implementation, the synchronization mode of the radar and camera device can be skipped, and the lidar and camera device can be synchronized directly according to the first synchronization mode or the second synchronization mode.

[0152] In one possible implementation, before performing step S601, the method further includes: assigning the lidar and camera imaging devices to a sensor group.

[0153] Specifically, when grouping sensor devices, the layout of the lidar and camera devices on the vehicle can be used to group any number of lidar and camera devices located in any position on the vehicle into the same sensor group. This allows for time synchronization or spatiotemporal synchronization of the lidar and cameras located within the same sensor group. For example, it can be done according to... Figure 5 The method described is used to divide different sensor groups.

[0154] In this embodiment, the lidar and camera devices can be divided into sensor groups according to the layout of the sensors on the vehicle and the needs of actual applications. The lidar and camera devices located in the same sensor group can be synchronized in time or in space, thereby improving the efficiency of data fusion between the lidar and camera devices.

[0155] S602 synchronizes the lidar and camera device according to the synchronization mode of the lidar and camera device.

[0156] In one possible implementation, after determining the synchronization mode of the lidar and the camera device as the first synchronization mode in step S601, the synchronization of the lidar and the camera device in step S602 includes: determining the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices; triggering the first camera device to expose according to the exposure frequency of the first camera device to obtain first data, wherein the camera device includes the first camera device; acquiring second data collected by the lidar when the first camera device is exposed; and if the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold, then synchronizing the first data and the second data.

[0157] Specifically, the frequency fc of the camera device can be calculated using the following formula:

[0158]

[0159] Where n represents the number of camera devices, and its value is a positive integer, T L This represents the scanning cycle of the lidar. For example, if a lidar in the same resource group corresponds to 6 camera devices, and the minimum scanning cycle of the camera device within a preset range is 100ms, then the frequency of the camera device is fc = 6 / 0.1s = 60Hz.

[0160] After determining the exposure frequency of the camera device, the CPLD triggers the camera device to expose itself according to the exposure frequency, and acquires the first and second data collected by the LiDAR and the camera device respectively at the exposure time. Then, the CPLD calculates the difference between the timestamps of the acquired first and second data and compares it with a first threshold, thereby performing different processing on the data collected by the LiDAR and the camera device.

[0161] The first threshold can be a pre-set value, determined based on the actual application of the time synchronization method. For example, if high accuracy is required for the synchronization of the LiDAR and camera, the first threshold can be set larger; if low accuracy is required, it can be set smaller. If the difference between the timestamps of the first and second data is less than or equal to the first threshold, it indicates that the LiDAR and camera meet the synchronization requirements, and the first and second data can be output to the algorithm application for processing. If the difference between the timestamps of the first and second data is greater than the first threshold, it indicates that the LiDAR and camera have not met the synchronization requirements, and the first and second data must be discarded.

[0162] In one possible implementation, the rotation period of the lidar is the maximum value within a preset range.

[0163] The preset range refers to the predetermined range within which the radar's scanning cycle can be achieved. The value of the preset range can be determined based on the inherent properties of the radar. For example, if the radar's inherent scanning cycle has two settings, 50ms and 100ms, then the preset range is 50ms-100ms. Setting the LiDAR's scanning cycle to the maximum value of the preset range, as in the example above, would be setting the LiDAR's scanning cycle to 100ms.

[0164] In this embodiment, the LiDAR can be set to the slowest operating mode suitable for the work scenario. The exposure frequency of the camera devices is determined based on the LiDAR's scanning cycle and the number of cameras corresponding to the LiDAR, and the camera exposure is triggered according to the exposure frequency. After the camera exposure, the time synchronization requirement between the LiDAR and the camera devices is determined by whether the difference between the data timestamps of the LiDAR and the camera devices is less than or equal to a first threshold. Only when the time synchronization requirement is met is the result output to the algorithm application for processing. In this way, the LiDAR and camera devices can match more frames of data in time, reducing the workload of the algorithm application and improving the accuracy and reliability of data fusion between the LiDAR and the camera devices.

[0165] In one possible implementation, in step S601, the synchronization mode of the lidar and the camera device is determined to be the second synchronization mode. In step S602, synchronizing the lidar and the camera device includes: at a first moment, setting the initial azimuth angle of the lidar to a first azimuth angle, and setting a first camera device in the direction of the first azimuth angle, the camera device including the first camera device; acquiring first data collected by the lidar and second data collected by the first camera device at the first moment; and performing synchronization processing on the first data and the second data.

[0166] Specifically, the initial azimuth angles of m lidars can be set to be equal, where m is an integer greater than 0. For example, the initial azimuth angles of all m lidars can be set to zero degrees, and the phase angles of the m lidars with the same initial azimuth angle are within the azimuth angle range of the first camera device in the camera device group. At this time, the m lidars and the first camera device in the camera device group are triggered synchronously, the moment of synchronous triggering of the lidars and the camera device is recorded as the first moment, and the first data and second data collected by the lidars and the first camera device at the first moment are processed synchronously.

[0167] In this embodiment of the application, by setting the initial azimuth angles of several lidars to be equal, and setting the initial azimuth angles of several lidars within the azimuth angle range of the first camera device in the camera device group, the several lidars and the first camera device are triggered synchronously, ensuring that the several lidars and the first camera device are synchronized in time or in space.

[0168] In one possible implementation, step S602 may further include: determining the exposure time of the second camera device as a second moment based on the positional relationship between the lidar, the first camera device, and the second camera device, wherein the camera device includes the second camera device; acquiring third data collected by the lidar and fourth data collected by the second camera device at the second moment; and performing synchronous processing on the third data and the fourth data.

[0169] Specifically, the exposure time of the second camera can be determined as the second moment based on the line connecting the position of the lidar in the camera assembly to the position of the first camera and the angle between the position of the lidar and the position of the second camera, and the camera exposure can be triggered at the second moment.

[0170] It should be understood that in the embodiments of this application, the second camera device may refer to a camera device located behind the first camera device within the camera device group that performs exposure, or it may refer to multiple camera devices located behind the first camera device that perform exposure.

[0171] The second moment of exposure by the second camera can be calculated using the following formula:

[0172]

[0173] Among them, T L θn represents the scanning period of the lidar, and θn represents the angle between the nth camera and the initial azimuth of the lidar. For example, if the lidar's scanning period is 100ms and the angle between the second camera and the lidar is π / 3, then the trigger time of the second camera, t2, is 100 / 2π × π / 3 = 16.7ms. When the time reaches 16.7ms, the CPLD triggers the second camera to expose via the ISP. During camera exposure, the lidar collects the third data, and the second camera collects the fourth data; the CPLD performs synchronous processing on the third and fourth data.

[0174] In this embodiment, several lidar units can be set to the same initial azimuth angle. Synchronization between multiple lidar units and cameras is achieved through synchronous triggering of the lidar and the first camera unit within the camera unit group, and triggering of the lidar and the second camera unit according to the calculated sequence time. This ensures the high requirements for spatiotemporal synchronization of lidar and camera units in high-speed vehicle driving scenarios. The CPLD only performs angle detection once; that is, the CPLD only detects the angle between the first camera unit and the lidar unit. Subsequent detections are based on the set exposure sequence time, which reduces the CPLD's resource consumption.

[0175] The following is combined with Figure 7 and Figure 8 The process shown introduces the synchronization method between radar and camera.

[0176] Figure 7 This is the radar and camera synchronization method 700 provided in the first synchronization mode of the embodiments of this application. Figure 7 The radar-camera time synchronization method 700 in the first synchronization mode can be applied to... Figure 1 In vehicle 100. Method 700 may include the following steps.

[0177] S701, set the LiDAR frequency to the lowest setting, i.e., the slowest working cycle.

[0178] For example, the inherent scanning period of a radar has two settings: 50ms and 100ms. Setting the frequency of the lidar to the lowest setting sets the scanning period of the lidar to 100ms. Here, 100ms is the preferred setting for the radar scanning period in this embodiment of the application.

[0179] S702 calculates the camera frequency based on the LiDAR scanning cycle and the multi-sensor layout.

[0180] Specifically, the camera frequency fc can be calculated using the following formula:

[0181]

[0182] Where n represents the number of cameras, and its value is a positive integer, T L This represents the scanning cycle of the LiDAR. For example, if a LiDAR in the same resource group corresponds to 6 cameras and the scanning cycle of the cameras is 100ms, then the camera frequency fc = 6 / 0.1s = 60Hz.

[0183] S703, the radar is rotating normally, and the CPLD triggers the camera exposure according to the fc frequency multiplication PPS.

[0184] Specifically, after calculating the camera frequency fc in step S702, the CPLD triggers the camera exposure using PPS multiples of the fc frequency. The value of the fc frequency multiple is a positive integer.

[0185] S704: If the timestamps of the LiDAR and camera data are less than or equal to the first threshold, the time synchronization requirement is met, and the data is output to the algorithm for processing.

[0186] Specifically, the algorithm checks whether the timestamps of the LiDAR and camera are less than or equal to a first threshold. If the condition is met, it indicates that the time synchronization requirement between the LiDAR and camera is satisfied, and the result is output to the algorithm application for processing. Before outputting the data to the algorithm application, the obtained data can be corrected to improve its accuracy.

[0187] In this embodiment, the lidar can be set to the slowest working mode suitable for the needs of the working scenario, and the camera can be set to a corresponding faster working mode. This enables the lidar and camera to match more frames of data in time when hard synchronization cannot be achieved due to resource limitations, thereby improving the accuracy and reliability of data fusion between the lidar and the camera.

[0188] Figure 8 This application provides a radar and camera synchronization method 800 in a second synchronization mode. The radar and camera synchronization method 800 in the second synchronization mode can be applied to... Figure 1 In vehicle 100. Method 800 may include the following steps.

[0189] S801, the initial azimuth angle of m lidars is set to 0 degrees.

[0190] Specifically, such as Figure 8 As shown in (a), the initial azimuth angles of the m lidars are set to 0 degrees to ensure that the m lidars can be triggered synchronously, and that the initial azimuth angles of the m lidars are within the range of the azimuth angle of the first camera in the camera group. Here, m is a positive integer.

[0191] S802, the CPLD triggers camera exposure via the ISP, time t1 = 0.

[0192] Specifically, such as Figure 8 As shown in (a), the information that the initial azimuth angles of m lidars are within the range of the first camera azimuth angle of the camera group is fed back to the CPLD. The CPLD triggers the camera exposure through the ISP and records the camera trigger time t1 = 0 at this time.

[0193] S803, the CPLD sets the time t2 to tn corresponding to the azimuth angle range of the 2nd to nth cameras based on the sensor layout and time synchronization architecture. n .

[0194] like Figure 8 (b) and Figure 8 As shown in (c), after deploying radar and cameras at different locations on the vehicle, the angle between the radar and each camera in the camera group can be measured. Once the angle is determined, the exposure time t2 to t3 of each camera in the camera group can be calculated. n Specifically, it can be calculated using the following formula:

[0195]

[0196] Among them, T L θn represents the scanning period of the lidar, and θn represents the angle between the nth camera and the initial azimuth of the lidar. For example, if the lidar's scanning period is 100ms and the angle between the second camera and the lidar is π / 3, then the trigger time of the second camera is t2 = 100 / 2π × π / 3 = 16.7ms. When the time reaches 16.7ms, the CPLD triggers the second camera to expose via the ISP. Similarly, the trigger times t3 to tn of the third to nth cameras can be calculated. n .

[0197] S804, CPLD according to t2 to t n The camera exposure is triggered sequentially via the ISP.

[0198] Specifically, the CPLD is based on the set t2 to t n The exposure of the 2nd to nth cameras is triggered sequentially through the ISP.

[0199] In this embodiment, the initial azimuth angle of several lidars can be set to 0 degrees. Synchronization between multiple lidars and cameras is achieved by synchronously triggering the lidars and the first camera in the camera group, and sequentially triggering the lidars and the second to nth cameras according to the calculated sequence time. This ensures the high requirements for spatiotemporal synchronization of lidars and cameras in high-speed vehicle driving scenarios. The CPLD only performs angle detection once, that is, it only detects the angle between the first camera and the lidar, and subsequent detections are based on the set sequence, which reduces the resource consumption of the CPLD.

[0200] Figure 9 This is a schematic diagram of a radar and camera synchronization device 900 provided in an embodiment of this application. This device 900 can be applied to... Figure 1 Of the 100 vehicles.

[0201] The device 900 may include an acquisition unit 910, a storage unit 920, and a processing unit 930. The acquisition unit 910 can implement corresponding communication functions and may also be referred to as a communication interface or communication unit for acquiring data. The storage unit 920 can be used to store corresponding instructions and / or data, and the processing unit 930 is used to perform data processing. The processing unit 930 can read instructions and / or data from the storage unit to enable the device to implement the aforementioned method embodiments.

[0202] The radar and camera synchronization device includes a processing unit 930; the processing unit 930 is used to determine the synchronization mode of the lidar and the camera device according to the resource status of the CPLD or FPGA, the synchronization mode of the lidar and the camera device includes a first synchronization mode or a second synchronization mode; the processing unit 930 is also used to synchronize the lidar and the camera device according to the synchronization mode of the lidar and the camera device.

[0203] In one possible implementation, the processing unit 930 is further configured to divide the lidar and camera imaging device into sensor groups.

[0204] In one possible implementation, the synchronization mode of the lidar and the camera device is the first synchronization mode. The processing unit 930 is specifically used to determine the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices, wherein the camera device includes the first camera device. The processing unit 930 is also used to trigger the first camera device to expose according to the exposure frequency of the first camera device to obtain first data, wherein the camera device includes the first camera device. The device further includes an acquisition unit 910, used to acquire second data collected by the lidar when the first camera device is exposed. If the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold, the processing unit 930 is used to perform synchronization processing on the first data and the second data.

[0205] In one possible implementation, the processing unit 930 is specifically configured to determine the exposure frequency of the camera device according to the following formula:

[0206]

[0207] Where fc represents the exposure frequency of the camera device, n represents the number of one or more camera devices and n is a positive integer, T L This indicates the scanning period of the lidar.

[0208] In one possible implementation, the synchronization mode of the lidar and the camera device is the second synchronization mode. The processing unit 930 is specifically used to set the initial azimuth angle of the lidar to a first azimuth angle at a first moment, and a first camera device is arranged in the direction of the first azimuth angle. The camera device includes the first camera device. The device also includes an acquisition unit 910, which is used to acquire first data collected by the lidar and second data collected by the first camera device at the first moment. The processing unit 930 is also used to perform synchronous processing on the first data and the second data.

[0209] In one possible implementation, the processing unit 930 is further configured to determine the exposure time of the second camera device as the second moment based on the positional relationship between the lidar, the first camera device, and the second camera device; the acquisition unit 910 is further configured to acquire the third data collected by the lidar and the fourth data collected by the second camera device at the second moment; and the processing unit 930 is further configured to perform synchronous processing on the third data and the fourth data.

[0210] In one possible implementation, the processing unit 930 is specifically used to determine the exposure time of the second camera device as the second moment based on the angle between the line connecting the position of the lidar and the position of the first camera device and the line connecting the position of the lidar and the position of the second camera device.

[0211] Figure 10 This is a schematic diagram of a radar and camera synchronization device 1000 provided in an embodiment of this application. This device 1000 can be applied to... Figure 1 Of the 100 vehicles.

[0212] The radar and time synchronization device includes a memory 1010, a processor 1020, and a communication interface 1030. The memory 1010, processor 1020, and communication interface 1030 are connected via an internal connection path. The memory 1010 stores instructions, and the processor 1020 executes the instructions stored in the memory 1020 to control the input / output interface 1030 to receive / transmit at least some parameters of the second channel model. Optionally, the memory 1010 can be coupled to the processor 1020 via an interface, or it can be integrated with the processor 1020.

[0213] It should be noted that the aforementioned communication interface 1030 uses a transceiver device, such as, but not limited to, a transceiver, to realize communication between the communication device 1000 and other devices or communication networks. The aforementioned communication interface 1030 may also include an input / output interface.

[0214] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1020 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1010, and the processor 1020 reads the information in memory 1010 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0215] This application also provides a computer-readable medium storing program code that, when executed on a computer, causes the computer to perform the above-described actions. Figures 6 to 8 Any of the methods mentioned above.

[0216] This application also provides a chip, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figures 6 to 8 Any of the methods mentioned above.

[0217] This application also provides a vehicle, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figures 6 to 8 Any of the methods mentioned above.

[0218] This application also provides a vehicle, including... Figure 9 or Figure 10 Any type of radar and camera device timing device.

[0219] It should be understood that in the embodiments of this application, the processor mentioned above can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0220] It should also be understood that, in embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0221] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0222] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0223] As used in this specification, the terms "component," "module," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0224] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0225] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0229] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A synchronization method, characterized in that, The method includes: The synchronization mode of the lidar and camera device is determined based on the resource status of the complex programmable logic device CPLD or the field programmable gate array FPGA. The synchronization mode of the lidar and camera device includes a first synchronization mode or a second synchronization mode. When the resources of the CPLD or the FPGA are insufficient, the lidar and the camera device are synchronized according to the first synchronization mode, or When the resources of the CPLD or the FPGA are sufficient, the lidar and the camera device are synchronized according to the second synchronization mode.

2. The method as described in claim 1, characterized in that, The method further includes: The lidar and the camera device are grouped into a sensor group.

3. The method as described in claim 1 or 2, characterized in that, The synchronization mode of the lidar and the camera device is the first synchronization mode, and the method further includes: The exposure frequency of the camera device is determined based on the scanning cycle of the lidar and the number of camera devices. Based on the exposure frequency of the first camera device, the first camera device is triggered to expose, and first data is obtained. The camera device includes the first camera device. Acquire the second data collected by the lidar when the first camera device is exposed; If the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold, then the first data and the second data are synchronized.

4. The method as described in claim 3, characterized in that, The scanning period of the lidar is the maximum value within a preset range.

5. The method as described in claim 3, characterized in that, Determining the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices includes: determining the exposure frequency of the camera device according to the following formula: Where fc represents the exposure frequency of the camera device, n represents the number of one or more camera devices and n is a positive integer, T L This indicates the scanning period of the lidar.

6. The method as described in claim 1 or 2, characterized in that, The synchronization mode of the lidar and the camera device is the second synchronization mode, and the method further includes: At the first moment, the initial azimuth angle of the lidar is set to the first azimuth angle, and a first camera device is set in the direction of the first azimuth angle, the camera device including the first camera device; At the first moment, acquire the first data collected by the lidar and the second data collected by the first camera device; The first data and the second data are synchronized.

7. The method as described in claim 6, characterized in that, The method further includes: Based on the positional relationship between the lidar, the first camera device, and the second camera device, the exposure time of the second camera device is determined as the second moment, and the camera device includes the second camera device; At the second moment, acquire the third data collected by the lidar and the fourth data collected by the second camera device; The third and fourth data are processed simultaneously.

8. The method as described in claim 7, characterized in that, The step of determining the exposure time of the second camera device as the second moment based on the positional relationship between the lidar, the first camera device, and the second camera device includes: The exposure time of the second camera device is determined as the second moment based on the angle between the line connecting the position of the lidar and the position of the first camera device and the line connecting the position of the lidar and the position of the second camera device.

9. A synchronization device, characterized in that, The device includes: a processing unit; The processing unit is used to determine the synchronization mode of the lidar and the camera device based on the resource status of the complex programmable logic device CPLD or the field programmable gate array FPGA. The synchronization mode of the lidar and the camera device includes a first synchronization mode or a second synchronization mode. The processing unit is further configured to synchronize the lidar and the camera device according to the first synchronization mode when the resources of the CPLD or the FPGA are insufficient, or When the resources of the CPLD or the FPGA are sufficient, the lidar and the camera device are synchronized according to the second synchronization mode.

10. The apparatus as claimed in claim 9, characterized in that, The processing unit is also used to divide the lidar and the camera device into sensor groups.

11. The apparatus as claimed in claim 9 or 10, characterized in that, The synchronization mode of the lidar and the camera device is the first synchronization mode. The processing unit is specifically used to determine the exposure frequency of the camera device based on the scanning cycle of the lidar and the number of camera devices; The processing unit is further configured to trigger the first camera device to expose according to the exposure frequency of the first camera device, and obtain first data, wherein the camera device includes the first camera device; The device further includes an acquisition unit for acquiring second data collected by the lidar when the first camera device is exposed. If the difference between the timestamp of the first data and the timestamp of the second data is less than or equal to a first threshold, the processing unit is used to perform synchronization processing on the first data and the second data.

12. The apparatus as claimed in claim 11, characterized in that, The scanning cycle of the lidar is the maximum value within a preset range.

13. The apparatus as claimed in claim 11, characterized in that, The processing unit is specifically used to determine the exposure frequency of the camera device according to the following formula: Where fc represents the exposure frequency of the camera device, n represents the number of one or more camera devices and n is a positive integer, T L This indicates the scanning period of the lidar.

14. The apparatus as claimed in claim 9 or 10, characterized in that, The synchronization mode of the lidar and camera device is the second synchronization mode. The processing unit is specifically used to set the initial azimuth angle of the lidar to a first azimuth angle at a first moment, and to set a first camera device in the direction of the first azimuth angle, wherein the camera device includes the first camera device. The device further includes an acquisition unit, which is used to acquire first data collected by the lidar and second data collected by the first camera device at the first moment. The processing unit is also used to perform synchronous processing on the first data and the second data.

15. The apparatus as claimed in claim 14, characterized in that, The processing unit is further configured to determine the exposure time of the second camera device as the second moment based on the positional relationship between the lidar, the first camera device, and the second camera device, wherein the camera device includes the second camera device; The acquisition unit is further configured to acquire the third data collected by the lidar and the fourth data collected by the second camera device at the second time. The processing unit is also used to perform synchronous processing on the third data and the fourth data.

16. The apparatus as claimed in claim 15, characterized in that, The processing unit is specifically used to determine the exposure time of the second camera device as the second moment based on the angle between the line connecting the position of the lidar and the position of the first camera device and the line connecting the position of the lidar and the position of the second camera device.

17. A synchronization device, characterized in that, include: At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform the method as claimed in any one of claims 1 to 8.

18. A computer-readable medium, characterized in that, The computer-readable medium stores program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

19. A chip, characterized in that, include: At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform the method as claimed in any one of claims 1 to 8.

20. A vehicle, characterized in that, include: At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform the method as claimed in any one of claims 1 to 8.

Images