Method, system, electronic device and storage medium for multi-sensor perception synchronization

Abstract

The application provides a multi-sensor perception synchronization method, comprising: acquiring a first detection time; controlling a first sensor to perform detection perception at the first detection time according to the first detection time; acquiring a second detection time; and controlling a second sensor to perform detection perception at the second detection time according to the second detection time; wherein the first detection time and the second detection time are configured such that, when the second sensor and the first sensor perform detection perception at the second detection time and the first detection time respectively, a time window of the detection perception of the second sensor is aligned with a time window of the detection perception of the first sensor. This technical solution solves the problem of how to make the detection perception actions of the sensors occur almost simultaneously (the time windows of the sensors are aligned and the time difference of the detection perception is much smaller than the time difference that can be achieved by the prior art), so as to achieve the effect that the detection information collected by the sensors reflects the same detection scene.

Description

Technical Field

[0001] This invention relates to the field of semiconductor devices, and more particularly to a method, system, electronic device, and storage medium for multi-sensor sensing synchronization. Background Technology

[0002] In the fields of automotive driving and robotics, various types of sensors are often needed to perceive the surrounding environment. Typical sensors include cameras, millimeter-wave radar, and lidar. Fusion of data from multiple sensors can lead to more accurate environmental perception and improve driving safety. Achieving time alignment of data from multiple sensors is a prerequisite for data fusion and is crucial to its effectiveness.

[0003] To achieve time alignment between sensor-acquired data, existing technologies typically record the timestamp of the moment the data is received from a particular sensor when the data processing device (such as a central processing unit) receives that data. When fusing data from multiple sensors, the data processing device reads sensor data with similar timestamps, thereby achieving the maximum possible time alignment.

[0004] However, this time-alignment method does not actually achieve the important functional requirement of all sensors acquiring sensing data at the same time. For example, the camera's exposure and the millimeter-wave radar's emission of electromagnetic waves for sensing must occur simultaneously. Therefore, developing a multi-sensor sensing synchronization system that aligns the time of camera exposure and millimeter-wave radar detection has become a key technical challenge that urgently needs to be addressed by those skilled in the art. Summary of the Invention

[0005] This invention provides a method, system, electronic device, and storage medium for multi-sensor perception synchronization to solve the problem of how to make the detection and sensing actions of each sensor occur almost simultaneously (the time windows of each sensor are aligned and the time difference between detection and sensing is much smaller than the time difference that can be achieved by existing technologies).

[0006] According to a first aspect of the present invention, a method for synchronizing perception across multiple sensors is provided, comprising:

[0007] The first sensor is controlled to perform detection and sensing at a first detection moment; the first detection moment represents the arbitrarily set moment when the first sensor performs detection and sensing.

[0008] Obtain the second detection time;

[0009] Based on the second detection time, control the second sensor to perform detection and sensing at the second detection time;

[0010] The first detection time and the second detection time are configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor detection and sensing is aligned with the time window of the first sensor detection and sensing.

[0011] Optionally, the second detection time is configured as follows:

[0012] The time difference between the second detection time and the first detection time is half of the first detection time window; the first detection time window represents the largest detection and sensing time window of the first sensor and the second sensor.

[0013] Optionally, the second detection time is configured to be the same as the first detection time, so that the time window for the second sensor to detect and perceive is aligned with the time window for the first sensor to detect and perceive.

[0014] Optionally, the second detection time is configured such that the time difference between the second detection time and the first detection time is equal to the first detection time window.

[0015] Optionally, the first sensor includes a radar sensor or a camera; the second sensor includes the radar sensor or the camera.

[0016] Optionally, the data acquisition timing of both the first sensor and the second sensor can be controlled, or the data acquisition timing of both the first sensor and the second sensor can be uncontrollable.

[0017] Optionally, when the first sensor is a radar sensor and the second sensor is a camera, the method further includes acquiring the first detection time before controlling the first sensor to perform detection and perception at the first detection time.

[0018] Optionally, the first detection time window is a pre-obtained detection time window of the radar sensor, and obtaining the second detection time specifically includes:

[0019] The first detection interval is calculated based on the first detection time window; the first detection interval represents the time difference between the detection and sensing by the first sensor and the second sensor;

[0020] The second detection time is calculated based on the first detection interval and the first detection time.

[0021] Optionally, the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable.

[0022] Optionally, when the first sensor is a camera and the second sensor is a radar sensor, then: obtaining the second detection time specifically includes:

[0023] Acquire first detection information; the first detection information represents the detection information collected by the camera; and the first detection information includes the first detection time;

[0024] The first moment is obtained; the first moment represents the time when the first detection information was obtained.

[0025] A first delay is calculated based on the first time and the first detection time; the first delay represents the difference between the first detection time and the first time.

[0026] The second detection time is calculated based on the first delay, the first inter-frame exposure interval, the first detection interval, and the first time. The first inter-frame exposure interval represents the inter-frame exposure interval of the camera. The first detection interval represents the time difference between the detection and sensing by the first sensor and the second sensor.

[0027] Optionally, when the first sensor is a radar sensor and the second sensor is a camera, then: obtaining the second detection time specifically includes:

[0028] Acquire second detection information; the second detection information characterizes the detection information collected by the radar sensor; and the second detection information includes the distance and radial velocity of the rotating reflector relative to the radar sensor at the midpoint of the detection time window; acquire a second moment; the second moment characterizes the moment when the second detection information was acquired;

[0029] The second delay is calculated based on the second detection information, the first relationship table, and the second time. The first relationship table represents the correspondence between the distance, radial velocity, and nominal time characteristics of the rotating reflector relative to the radar sensor, which was obtained in advance. The second delay represents the difference between the midpoint of the detection time window of the radar sensor and the second time.

[0030] The second detection time is calculated based on the second delay, the first inter-frame detection interval, and the second time.

[0031] According to a second aspect of the present invention, a system for synchronizing perception across multiple sensors is provided, comprising:

[0032] The system comprises a first sensor, a second sensor, and a control module; the control module is connected to the first sensor and the second sensor.

[0033] The control module is used to obtain a first detection time and control the first sensor to perform detection and sensing at the first detection time; and to obtain a second detection time and control the second sensor to perform detection and sensing at the second detection time.

[0034] The first detection time and the second detection time are configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor detection and sensing is aligned with the time window of the first sensor detection and sensing.

[0035] Optionally, the control module includes: a control unit, a storage unit, a timing unit, and a computing unit;

[0036] The control unit is used to control the first sensor to perform detection and sensing at the first detection time, or / and to control the second sensor to perform detection and sensing at the second detection time.

[0037] The storage unit is used to store the first detection time window and the first inter-frame exposure interval.

[0038] The timing unit is used to acquire the first detection time and the second detection time, and when the first detection time and the second detection time are acquired, to trigger the control unit to control the first sensor to perform detection and sensing at the first detection time, or to trigger the control unit to control the second sensor to perform detection and sensing at the second detection time.

[0039] The calculation unit is used to calculate the second detection time based on the first detection time, the first detection time window, and the first inter-frame exposure interval, and send the second detection time to the timing unit.

[0040] Optional, multi-sensor sensing synchronization systems also include:

[0041] A first control line; the control module is connected to the second sensor through the first control line; so that the control module performs data acquisition timing on the second sensor through the first control line; the data acquisition timing on the second sensor indicates that at the second detection time, the control module transmits a first trigger signal to the second sensor through the first control line, triggering the second sensor to perform detection and sensing at the second detection time.

[0042] Optional, multi-sensor sensing synchronization systems also include:

[0043] First control line; used for timing data acquisition from the second sensor;

[0044] The second control line; the control module is connected to the first sensor through the second control line so that the control module performs data acquisition timing on the first sensor through the second control line; the data acquisition timing on the first sensor indicates that at the first detection time, the control module transmits a second trigger signal to the first sensor through the second control line, triggering the first sensor to perform detection and sensing at the first detection time.

[0045] Optional, multi-sensor sensing synchronization systems also include:

[0046] First reset / programming line and second reset / programming line;

[0047] The first reset / programming line is connected to the first sensor; the first reset / programming line is used to transmit the first detection time to the first sensor, so that the controller controls the first sensor to perform detection and sensing at the first detection time.

[0048] The control module is connected to the second sensor via the second reset / programming line; the second reset / programming line is used to transmit the second detection time to the second sensor, so that the controller controls the second sensor to perform detection and sensing at the second detection time.

[0049] According to a third aspect of the present invention, an electronic device is provided, including a processor and a memory, the memory being used to store code;

[0050] The processor is configured to execute code in the memory to implement the multi-sensor perception synchronization method described in the first aspect of the present invention.

[0051] According to a fourth aspect of the invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for multi-sensor sensing synchronization as described in any of the first aspects of the invention.

[0052] This invention provides a method for multi-sensor perception synchronization. It first controls a first sensor to perform detection and perception at a first detection moment, and then controls a second sensor to perform detection and perception at a specially designed second detection moment. This aligns the detection and perception time window of the second sensor with that of the first sensor, making the time difference between the detection and perception of each sensor much smaller than that achievable in existing technologies. This enables each sensor to collect perception data at almost the same time, thereby achieving joint detection and perception of the same scene by all sensors. This improves the accuracy and reliability of detection and perception, and enhances the effect of data fusion.

[0053] Furthermore, when the first sensor lacks a data acquisition timing function, at a known time: the first detection time, the first sensor is controlled to perform detection and sensing to acquire detection and sensing information, which includes the timestamp of the first detection time. Then, the second detection time is calculated, and the second sensor is controlled to perform detection and sensing at the second detection time, so that the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing. This enables each sensor to acquire sensing data at almost the same time, even when there are sensors among the sensors that lack a data acquisition timing function. This allows all sensors to jointly detect and sense the same scene, improving the accuracy and reliability of detection and sensing, and enhancing the effect of data fusion. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a method for multi-sensor sensing synchronization according to an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram showing the alignment of the second detection time with the first detection time when the timing of data acquisition for both camera exposure imaging and radar sensor detection is controllable, according to one embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram showing the alignment of the second detection time with the first detection time when the timing of camera exposure imaging is uncontrollable and the timing of radar sensor detection and perception is controllable, provided in one embodiment of the present invention.

[0058] Figure 4This is a schematic diagram showing the alignment of the second detection time with the first detection time when the timing of camera exposure imaging is controllable and the timing of radar sensor detection and perception is uncontrollable, according to one embodiment of the present invention.

[0059] Figure 5 This is a schematic diagram showing the alignment of the second detection time with the first detection time when the timing of data acquisition for both camera exposure imaging and radar sensor detection is uncontrollable, according to one embodiment of the present invention.

[0060] Figure 6 This invention provides a method for estimating the delay amount in a radar sensor detection and sensing rotating reflector, as provided in one embodiment of the invention.

[0061] Figure 7 This is a first relation table provided in one embodiment of the present invention;

[0062] Figure 8 This is a schematic diagram of the specific structure of a system for multi-sensor perception synchronization when the timing of data acquisition for camera exposure imaging and radar sensor detection is controllable, provided in a specific embodiment of the present invention.

[0063] Figure 9 This is a schematic diagram of the specific structure of a system for synchronizing the perception of multiple sensors when the timing of camera exposure imaging is uncontrollable and the timing of radar sensor detection and perception is controllable, provided in a specific embodiment of the present invention.

[0064] Figure 10 This is a schematic diagram of the specific structure of a system for synchronizing the perception of multiple sensors when the timing of camera exposure imaging is controllable and the timing of radar sensor detection and perception is uncontrollable, provided in a specific embodiment of the present invention.

[0065] Figure 11 This is a schematic diagram of the specific structure of a multi-sensor perception synchronization system provided in a specific embodiment of the present invention when the timing of data acquisition for both camera exposure imaging and radar sensor detection perception is uncontrollable.

[0066] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0068] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] In the fields of automotive driving or robotics that utilize various sensors, traditional time alignment methods fail to meet the crucial requirement of simultaneous data acquisition by each sensor. For example, the actions of camera exposure and millimeter-wave radar emission for sensing must occur simultaneously, resulting in a significant time lag between the detection information collected by each sensor. This makes it impossible for the detection information collected by each sensor to reflect the same scene.

[0070] In view of this, the inventors of this application creatively propose a method for multi-sensor perception synchronization: Based on the different functions of sensors in a practical system, this invention proposes four modes and corresponding design methods. Specifically, for sensors lacking data acquisition timing functionality, the acquisition time of the next data acquisition is predicted based on the time when the acquired data arrives at the controller, combined with the data acquisition cycle or a predefined rule. Furthermore, the timing alignment of other sensors is controlled, achieving time alignment and perception synchronization of data acquisition from multiple sensors, thereby ensuring data homogeneity across multiple sensors.

[0071] Specifically, when the data acquisition timings of both the camera and radar sensors are controllable, the controller controls the data acquisition timing of each sensor separately. When the camera's data acquisition timing is uncontrollable but the radar sensor's data acquisition timing is controllable, the controller predicts the next data acquisition time based on the arrival time of the camera's acquired data and controls the radar sensor's data acquisition timing accordingly. When the camera's data acquisition timing is controllable but the radar sensor's data acquisition timing is uncontrollable, the controller predicts the next data acquisition time based on the arrival time of the radar sensor's acquired data and controls the camera's data acquisition timing accordingly. When the data acquisition timings of both the camera and radar sensors are uncontrollable, the controller determines the start time of the time window for camera exposure imaging and radar sensor detection sensing respectively through a reset / programming method.

[0072] As can be seen, the multi-sensor perception synchronization method provided in this application can achieve time window alignment of various sensors such as camera exposure and millimeter-wave radar detection and perception. That is, the detection time window of the sensor with the largest detection and perception time window among all sensors includes the detection and perception time windows of the other sensors. This enables the sensors to collect perception data at almost the same time, achieving the effect of all sensors jointly detecting and perceiving the same scene, which can improve the accuracy and reliability of detection and perception.

[0073] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0074] Please refer to Figures 1-12 According to an embodiment of the present invention, a method for synchronizing perception across multiple sensors is provided, comprising:

[0075] S11: Control the first sensor to perform detection and sensing at the first detection moment; the first detection moment represents the arbitrarily set moment when the first sensor performs detection and sensing.

[0076] S12: Obtain the second detection time;

[0077] S13: Based on the second detection time, control the second sensor to perform detection and sensing at the second detection time;

[0078] The first detection time and the second detection time are configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor detection and sensing is aligned with the time window of the first sensor detection and sensing.

[0079] Wherein, the alignment of the time window for the second sensor detection and perception with the time window for the first sensor detection and perception means that the time window for the first sensor detection and perception includes the time window for the second sensor detection and perception; or the time window for the second sensor detection and perception includes the time window for the first sensor detection and perception.

[0080] This invention provides a method for multi-sensor perception synchronization. It first controls a first sensor to detect and sense at a first detection moment, and then controls a second sensor to detect and sense at a specially designed second detection moment. This aligns the time window of the second sensor's detection with that of the first sensor, enabling all sensors to collect sensing data at almost the same time. This allows all sensors to jointly detect and sense the same scene, improving the accuracy and reliability of the detection and sensing.

[0081] The following are some embodiments of aligning the time window of the second sensor with the time window of the first sensor:

[0082] In one embodiment, the second detection time is configured as follows:

[0083] The time difference between the second detection time and the first detection time is half of the first detection time window; the first detection time window represents the largest detection and sensing time window of the first sensor and the second sensor.

[0084] In one embodiment, the second detection time is configured to be the same as the first detection time, so that the time window for the second sensor to detect and perceive is aligned with the time window for the first sensor to detect and perceive.

[0085] In one embodiment, the second detection time is configured such that the time difference between the second detection time and the first detection time is equal to the first detection time window.

[0086] The following section uses radar sensors and cameras as examples to illustrate the method of multi-sensor perception synchronization:

[0087] In one embodiment, the first sensor includes a radar sensor or a camera; the second sensor includes the radar sensor or the camera.

[0088] In one embodiment, the data acquisition timing of both the first sensor and the second sensor is controllable, or the data acquisition timing of both the first sensor and the second sensor is uncontrollable.

[0089] In one embodiment, when the first sensor is a radar sensor and the second sensor is a camera, the method further includes acquiring the first detection time before controlling the first sensor to perform detection and perception at the first detection time.

[0090] In one embodiment, the first detection time window is a pre-obtained detection time window of the radar sensor, and obtaining the second detection time specifically includes:

[0091] The first detection interval is calculated based on the first detection time window; the first detection interval represents the time difference between the detection and sensing by the first sensor and the second sensor;

[0092] The second detection time is calculated based on the first detection interval and the first detection time.

[0093] In one specific implementation, the data acquisition timing of both the first sensor and the second sensor is controllable; that is, the data acquisition timing of the camera's exposure imaging is controllable, and the data acquisition timing of the radar sensor's detection and sensing is controllable. The goal of this mode is to achieve synchronous sensing between the camera and the radar sensor, which is relatively easy to implement. Taking the time difference between the second detection time and the first detection time as half of the first detection time window for time alignment as an example, the calculation process of the second detection time is explained in detail; the specific calculation method for obtaining the second detection time includes: (wherein, the first detection time window is denoted by T). R The first detection time is represented by t1; the first detection interval is represented by τ. enable The second detection time is denoted by t2. It should be understood that aligning the time windows of the first and second sensors means that the camera's detection time falls within the radar sensor's first detection time window T. R Within.

[0094] The first detection interval is calculated based on the first detection time window; the formula for calculating the first detection interval is shown below:

[0095]

[0096] The second detection time is calculated based on the first detection interval and the first detection time; the formula for calculating the second detection time is as follows:

[0097]

[0098] In summary, performing detection and sensing at the second detection moment ensures that the camera is exposed at the midpoint of the radar sensor's detection and sensing time window, thus achieving synchronization between camera exposure imaging and radar sensor detection. Figure 2 As shown.

[0099] The following example illustrates the calculation process for the second detection time in this embodiment: Assuming the frame rate of the camera exposure imaging is 25 frames per second, the inter-frame exposure interval is T. C =40ms, the first detection time window T of the radar sensor R = 5ms, and the period of each linear frequency modulation is 1ms. The timing unit obtains the timestamp of time t1 and triggers the control unit to control the radar sensor to start detection and sensing at time t1. The calculation unit calculates according to the T data pre-stored in the storage unit. R calculate After delay τ enable The timing unit triggers the control unit to control the camera at a specific time. Exposure imaging is performed.

[0100] In another specific implementation, the data acquisition timing of both the first and second sensors is uncontrollable; that is, the acquisition timing of the data from the camera's exposure imaging and the data from the radar sensor's detection and sensing is uncontrollable. The difference between the multi-sensor perception synchronization method and the aforementioned specific implementation lies in the use of a reset / programming function instead of a timing unit triggering the control unit. This controls the camera and radar sensor to perform detection and sensing at the first and second detection times respectively, thereby achieving perception synchronization among the sensors. A schematic diagram of the time alignment for sensor perception synchronization is shown below. Figure 5 As shown. The reset / programming function is described in detail in the later embodiments of this application, and will not be repeated here.

[0101] In one embodiment, the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable.

[0102] In one embodiment, when the first sensor is a camera and the second sensor is a radar sensor, then: obtaining the second detection time specifically includes:

[0103] Acquire first detection information; the first detection information represents the detection information collected by the camera; and the first detection information includes the first detection time;

[0104] The first moment is obtained; the first moment represents the time when the first detection information was obtained.

[0105] A first delay is calculated based on the first time and the first detection time; the first delay represents the difference between the first detection time and the first time.

[0106] The second detection time is calculated based on the first delay, the first inter-frame exposure interval, the first detection interval, and the first time. The first inter-frame exposure interval represents the inter-frame exposure interval of the camera. The first detection interval represents the time difference between the detection and sensing by the first sensor and the second sensor.

[0107] The technical solution provided by this invention, when the first sensor does not have a data acquisition timing function, controls the first sensor to perform detection and sensing at a known time: a first detection time to acquire detection and sensing information, and this information includes a timestamp of the first detection time. Then, a second detection time is calculated, and the second sensor is controlled to perform detection and sensing at the second detection time. The second detection time is configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing, thereby realizing the function of each sensor acquiring sensing data at almost the same time, and thus realizing the joint detection and sensing of the same scene by each sensor, which can improve the accuracy and reliability of detection and sensing.

[0108] The challenge and objective of this mode is to estimate the delay in acquiring the first detection information from the camera's exposure imaging. This estimation can be achieved before system deployment using a certain method. If necessary, the detection and sensing time can be adjusted at a specific moment for each sensor, or it can be adjusted for each detection and sensing operation to better achieve time alignment. The underlying idea is to embed obtainable timestamp information into the data. The following section details the calculation process for the second detection moment, using a mode where the time difference between the second and first detection moments is half of the first detection time window, the timing of data acquisition for camera exposure imaging is uncontrollable, but the timing of data acquisition for radar sensor detection and sensing is controllable, as an example. The specific calculation method for obtaining the second detection moment includes: (where the first delay is represented by τ; the time difference between the radar sensor and the camera's detection and sensing is represented by τ). enable Indicates; the exposure interval between the first frames is represented by T. C Indicated by T; the first detection time window is represented by T. R express)

[0109] Obtain first detection information; the first detection information includes the first detection time t1;

[0110] Get the first time step t2;

[0111] Based on the first time point and the first detection time point, the first delay is calculated, and the formula for calculating the first delay is as follows:

[0112] τ = t2 - t1;

[0113] The second detection time is calculated based on the first delay, the first inter-frame exposure interval, the first detection interval, and the first time; specifically, it includes:

[0114] Based on the first delay, the first inter-frame exposure interval, and the first detection interval, calculate the detection time difference between the radar sensor and the camera; the calculation formula is as follows:

[0115]

[0116] The second detection time is calculated based on the time difference between the radar sensor and the camera and the first detection time; the formula for calculating the second detection time is as follows;

[0117] t3=t2+τ enable ;

[0118] In summary, performing detection and sensing at the second detection moment ensures that the camera is exposed at the midpoint of the radar sensor's detection and sensing time window, thus achieving synchronization between camera exposure imaging and radar sensor detection. Figure 3 As shown.

[0119] The following example illustrates the calculation process for the second detection time in this embodiment: Assuming the frame rate of the camera exposure imaging is 25 frames per second, the inter-frame exposure interval is T. C =40ms, the first detection time window T of the radar sensor R =5ms, and the period of each linear frequency modulation is 1ms. Assume the camera exposes and images a clock containing timestamp information at time t1. After a delay τ = 3ms (temporarily unknown), the timing unit receives the data acquired by the camera at time t1 at time t2. Since the acquired data contains the timestamp information of time t1, the calculation unit can estimate the delay τ = t2 - t1 = 3ms. Starting from time t2, the calculation unit calculates the delay τ based on the T data pre-stored in the storage unit. R Calculate the process The delay, the timing unit triggers the control unit to control the radar sensor at time t3=t2+τ enable Detection and sensing begin at t2 + 34.5 ms. Delay τ enable The calculation formula needs to be adjusted according to the alignment between the camera's exposure time and the radar sensor's sensing time window. When the camera's exposure time is located at the midpoint of the radar sensor's detection time window, When the camera's exposure time falls at the beginning of the radar sensor's detection window, τ enable =T C -τ; When the camera's exposure time is at the end of the radar sensor's detection window, τ enable =T C -τ-T R .

[0120] In one embodiment, when the first sensor is a radar sensor and the second sensor is a camera, then: obtaining the second detection time specifically includes:

[0121] Acquire second detection information; the second detection information characterizes the detection information collected by the radar sensor; and the second detection information includes the distance and radial velocity of the rotating reflector relative to the radar sensor at the midpoint of the detection time window;

[0122] The second moment is obtained; the second moment represents the time when the second detection information was obtained.

[0123] The second delay is calculated based on the second detection information, the first relationship table, and the second time. The first relationship table represents the correspondence between the distance, radial velocity, and nominal time characteristics of the rotating reflector relative to the radar sensor, which was obtained in advance. The second delay represents the difference between the midpoint of the detection time window of the radar sensor and the second time.

[0124] The second detection time is calculated based on the second delay, the first inter-frame detection interval, and the second time.

[0125] The challenge and objective of this mode is to estimate the second moment of the radar sensor's detection and sensing data acquisition. This estimation can be achieved using a certain method before system deployment, or the detection and sensing time can be adjusted at any moment during detection and sensing. The idea is to embed obtainable timestamp information into the data. The following example illustrates the calculation process of the second detection moment in a mode where the timing of data acquisition for camera exposure imaging is controllable, but the timing of data acquisition for radar sensor detection and sensing is uncontrollable. The specific calculation method for obtaining the second detection moment includes: (wherein, the time difference between the second moment and the camera's detection and sensing is denoted by τ). enable The second time interval is denoted by t2; the second detection time is denoted by t3; the first inter-frame detection interval is denoted by T. C (The second delay is represented by τ.)

[0126] Obtain second detection information;

[0127] Obtain the second moment;

[0128] Based on the second detection information, the first relationship table, and the second time point, the second delay is calculated; the calculation process of the second delay is described as follows:

[0129] The controller's control unit controls a reflector with a controllable rotation speed, a known rotation radius, and a fixed center position. The calculation unit generates a table showing the correspondence between the distance and radial velocity (including direction and magnitude) of the rotating reflector relative to the radar sensor and nominal time characteristics, and stores this table in the storage unit. For example, [the following is implemented]. Figure 7 As shown. Figure 4 As shown, assume the radar sensor completes detection within the current time window at time t1 and acquires detection data at position t0 within the current time window. This data includes the detection results of the distance and radial velocity of the rotating reflector relative to the radar sensor at time t0. Starting from time t1, after a delay τ1 (temporarily unknown), the timing unit receives the timestamp feature information of time t0 at time t2 (provided by the timing unit measurement, so t2 is a known quantity), i.e., the detection data of the distance and radial velocity of the rotating reflector relative to the radar sensor at time t0. The calculation unit looks up the table data in the storage unit to obtain the letter label and numerical sequence number representing the nominal time feature quantity corresponding to the detection results of the distance and radial velocity of the rotating reflector relative to the radar sensor at time t0. Since the rotation speed of the rotating reflector is controllable and the rotation radius is known, the letter label and numerical sequence number representing the nominal time feature quantity corresponding to the distance and radial velocity information of the rotating reflector relative to the radar sensor at time t2 can also be obtained by looking up the table. Thus, the time difference τ from time t0 to time t2 can be estimated, such as... Figure 6 and Figure 7 As shown.

[0130] The second detection time is calculated based on the second delay, the first inter-frame detection interval, and the second time; specifically, it includes:

[0131] Based on the first inter-frame detection interval and the second delay, the time difference between the second moment and the camera's detection and sensing is calculated; the formula for calculating the time difference between the second moment and the camera's detection and sensing is as follows:

[0132] τ enable =T C -τ;

[0133] The second detection time is calculated based on the time difference between the second moment and the camera's detection and sensing, and the second moment itself; the formula for calculating the second detection time is as follows:

[0134] t3=t2+τ enable ;

[0135] In summary, performing detection and sensing at the second detection moment ensures that the camera is exposed at the midpoint of the radar sensor's detection and sensing time window, thus achieving synchronization between camera exposure imaging and radar sensor detection. Figure 4 As shown.

[0136] The following example illustrates the calculation process for the second detection time in this embodiment: Assuming the frame rate of the camera exposure imaging is 25 frames per second, the inter-frame exposure interval is T. C =40ms, the detection and sensing time window T of the radar sensor R = 5ms, and each linear frequency modulation period is 1ms. For example... Figure 6 As shown, assuming the radar sensor is located at coordinates (0,0), the center of the rotating reflector is located at coordinates (0,5), the unit is meters, the rotation radius is 1 meter, counterclockwise rotation is defined as the positive direction, and the rotation frequency is f = 20 revolutions per second, then the uniform tangential scalar velocity of the rotating reflector is v = 2πRf = 2 × 3.14 × 1 × 20 = 125.6 m / s. Figure 9 Using midpoint A as the nominal starting time, and selecting N=8 sampling points on the circumference, the distance and radial velocity of the rotating reflector relative to the radar sensor at different positions are as follows: Figure 7 As shown, the radial velocity of the rotating reflector moving away from the radar sensor is set to positive. The time difference between two adjacent points is... ms. Assume the radar sensor completes detection within the current time window at time t1 and acquires detection data at position t0 within that window. This data includes the distance between the rotating reflector and the radar sensor at time t0 being 5.1 meters, a radial velocity of 123 m / s, and the direction of the reflector towards the radar sensor. After a certain time delay τ1 (currently unknown), the controller's control unit receives this data and obtains the received data time t2 from the timing unit. The calculation unit looks up the data in the storage unit's table and finds that the detection result at time t0 corresponds to the letter E (associated with the numerical label 4; letters are used to avoid confusion with coordinate numerical labels). Since the rotation of the rotating reflector is manually set, time t2 corresponds to the letter H (associated with the numerical label 7) in the table. By comparing the results in the table, the time difference between time t0 and time t2 can be estimated.

[0137] τ=Δt×(7+N-4)modN=6.25×[(7+8-4)mod8]=18.75ms. Therefore, starting from time t2, after a delay τ enable =T C -τ=40-18.5=21.5ms, the timing unit triggers the control unit to control the camera at time t3=t2+τ enable Exposure imaging is performed.

[0138] When the measured values ​​of distance and radial velocity of the rotating reflector relative to the radar sensor are not completely consistent with the data in the table, the closest value is selected to complete the table lookup.

[0139] Secondly, according to an embodiment of the present invention, a system for multi-sensor perception synchronization is also provided, comprising:

[0140] The system comprises a first sensor, a second sensor, and a control module; the control module is connected to the first sensor and the second sensor.

[0141] The control module is used to obtain a first detection time and control the first sensor to perform detection and sensing at the first detection time; and to obtain a second detection time and control the second sensor to perform detection and sensing at the second detection time; specifically, the control module is a controller.

[0142] In one embodiment, the control module includes: a control unit, a storage unit, a timing unit, and a computing unit;

[0143] The control unit is used to control the first sensor to perform detection and sensing at the first detection time, or / and to control the second sensor to perform detection and sensing at the second detection time.

[0144] The storage unit is used to store the first detection time window and the first inter-frame exposure interval.

[0145] The timing unit is used to acquire the first detection time and the second detection time, and when the first detection time and the second detection time are acquired, to trigger the control unit to control the first sensor to perform detection and sensing at the first detection time, or to trigger the control unit to control the second sensor to perform detection and sensing at the second detection time.

[0146] The calculation unit is used to calculate the second detection time based on the first detection time, the first detection time window, and the first inter-frame exposure interval, and send the second detection time to the timing unit.

[0147] When the first sensor does not have a data acquisition timing function, but the second sensor does have a data acquisition timing function, in one embodiment, the multi-sensor sensing synchronization system further includes:

[0148] A first control line; the control module is connected to the second sensor through the first control line; so that the control module performs data acquisition timing on the second sensor through the first control line; the data acquisition timing on the second sensor indicates that at the second detection time, the control module transmits a first trigger signal to the second sensor through the first control line, triggering the second sensor to perform detection and sensing at the second detection time.

[0149] When the first sensor is a camera, a multi-sensor perception synchronization system is as follows: Figure 9 As shown;

[0150] When the first sensor is a radar sensor, the multi-sensor sensing synchronization system is as follows: Figure 10 As shown;

[0151] When both the first and second sensors have data acquisition timing functions, in one embodiment, the multi-sensor sensing synchronization system, in addition to having a first control line for timing data acquisition from the second sensor, also includes:

[0152] The second control line; the control module is connected to the first sensor via the second control line, so that the control module performs data acquisition timing on the first sensor via the second control line; the data acquisition timing on the first sensor indicates that at the first detection moment, the control module transmits a second trigger signal to the first sensor via the second control line, triggering the first sensor to perform detection and sensing at the first detection moment, such as... Figure 8 As shown.

[0153] When neither the first sensor nor the second sensor has a data acquisition timing function, that is, the multi-sensor sensing synchronization system does not have a first control line and a second control line; in one embodiment, the multi-sensor sensing synchronization system further includes:

[0154] First reset / programming line and second reset / programming line;

[0155] The first reset / programming line is connected to the first sensor; the first reset / programming line is used to transmit the first detection time to the first sensor, so that the controller controls the first sensor to perform detection and sensing at the first detection time.

[0156] The control module is connected to the second sensor via the second reset / programming line; the second reset / programming line is used to transmit the second detection time to the second sensor, so that the controller controls the second sensor to perform detection and sensing at the second detection time. Figure 11 As shown.

[0157] The timing unit triggers the control unit by using pulse triggering to control the exposure and imaging time of the camera and the detection and sensing time of the radar sensor, respectively.

[0158] The reset / programming function is generally used to restart or reinitialize the electronic system. However, the reset / programming functions of existing radar sensors and cameras are not designed to control the sensing (i.e., data sampling) timing. Specifically, the technical solution provided in this embodiment controls the camera or radar sensor to restart immediately (or restart after a program update). After startup, the sensor completes system initialization under its own program control and begins data acquisition according to the program settings. The entire startup and initialization process takes a period of time, referred to as the reset or startup delay. Under relatively stable reset delay conditions, by measuring and estimating this delay offline, the startup delay can be incorporated as a known quantity into the synchronization system for calculation, thereby achieving indirect control of the sensor data sampling time window. This achieves the following: When neither the first nor the second sensor has a data acquisition timing function, the initial time of the camera's exposure imaging and the initial time of the radar sensor's detection sensing are controlled by the reset / programming function, respectively. This ensures that the camera is exposed at the point in the radar sensor's detection sensing time window (or other positions mentioned above), thereby achieving synchronization between the camera's exposure imaging and the radar sensor's detection sensing. The difficulty and goal of this mode is to achieve an approximately equivalent effect where the timing of both the camera and the radar sensor is controllable, even when the timing of both the camera and the radar sensor is uncontrollable.

[0159] The multi-sensor sensing synchronization system also includes:

[0160] Data cables connecting the radar sensor and the controller, and data cables connecting the camera and the controller;

[0161] Each embodiment of the multi-sensor sensing synchronization system includes a first reset / programming line and a second reset / programming line. The functions of the corresponding structures and other specific structures in the multi-sensor sensing synchronization system are well-known to those skilled in the art and will not be elaborated upon here.

[0162] In addition, according to one embodiment of the present invention, an electronic device is also provided, such as... Figure 12 As shown, it includes a processor and a memory, wherein the memory is used to store code;

[0163] The processor is used to execute the code in the memory to implement the multi-sensor perception synchronization method described in the foregoing embodiments of the present invention.

[0164] Additionally, according to one embodiment of the present invention, an electronic device 10 is also provided, which is as follows: Figure 12 As shown, it includes a processor 11 and a memory 12, the memory being used to store code; the processor 11 and the memory 12 communicate via a bus 13.

[0165] The processor is used to execute the code in the memory to implement the multi-sensor perception synchronization method mentioned above.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synchronizing perception across multiple sensors, characterized in that, include: Control the first sensor to perform detection and sensing at the first detection moment; The first detection time represents the arbitrarily set detection and sensing time of the first sensor; Obtain the second detection time; Based on the second detection time, control the second sensor to perform detection and sensing at the second detection time; The first detection time and the second detection time are configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing; wherein the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable; wherein, when the first sensor is a camera and the second sensor is a radar sensor; then: obtaining the second detection time specifically includes: obtaining first detection information; the first detection information represents the detection information collected by the camera; and the first detection information includes the first detection time; obtaining a first time; the first time represents the time when the first detection information is obtained; calculating a first delay based on the first time and the first detection time; the first delay represents the difference between the first detection time and the first time; calculating the second detection time based on the first delay, the first inter-frame exposure interval, the first detection interval, and the first time; the first inter-frame exposure interval represents the inter-frame exposure interval of the camera; the first detection interval represents the time difference between the first sensor and the second sensor performing detection and sensing.

2. A method for synchronizing perception across multiple sensors, characterized in that, include: Control the first sensor to perform detection and sensing at the first detection moment; The first detection time represents the arbitrarily set detection and sensing time of the first sensor; Obtain the second detection time; Based on the second detection time, control the second sensor to perform detection and sensing at the second detection time; Wherein, the first detection time and the second detection time are configured such that: when the second sensor and the first sensor respectively perform detection and sensing at the second detection time and the first detection time, the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing; wherein, the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable; wherein, when the first sensor is a radar sensor and the second sensor is a camera; then: obtaining the second detection time specifically includes: obtaining second detection information; the second detection information represents the detection information collected by the radar sensor; and the second detection information includes the radar sensor's detection... The distance and radial velocity of the rotating reflector relative to the radar sensor at the midpoint of the time window; obtaining the second moment; the second moment represents the moment when the second detection information is acquired; calculating the second delay based on the second detection information, the first relation table, and the second moment; the first relation table represents the pre-obtained correspondence table of the distance, radial velocity, and nominal time characteristics of the rotating reflector relative to the radar sensor; the second delay represents the difference between the midpoint of the detection time window of the radar sensor and the second moment; calculating the second detection moment based on the second delay, the first inter-frame exposure interval, and the second moment, where the first inter-frame exposure interval represents the inter-frame exposure interval of the camera.

3. The method for multi-sensor sensing synchronization according to claim 1 or 2, characterized in that, The second detection time is configured as follows: The time difference between the second detection time and the first detection time is half of the first detection time window; the first detection time window represents the maximum detection and sensing time window of the first sensor and the second sensor.

4. The method for multi-sensor sensing synchronization according to claim 1 or 2, characterized in that, The second detection time is configured as follows: The second detection time is the same as the first detection time, so that the time window for the second sensor to detect and perceive is aligned with the time window for the first sensor to detect and perceive.

5. The method for multi-sensor sensing synchronization according to claim 1 or 2, characterized in that, The second detection time is configured as follows: The time difference between the second detection time and the first detection time is equal to the first detection time window.

6. The method for multi-sensor sensing synchronization according to claim 2, characterized in that, When the first sensor is a radar sensor and the second sensor is a camera, the method further includes acquiring the first detection time before controlling the first sensor to perform detection and perception at the first detection time.

7. The method for multi-sensor sensing synchronization according to claim 6, characterized in that, The first detection time window is the pre-obtained detection time window of the radar sensor, and obtaining the second detection time specifically includes: The first detection interval is calculated based on the first detection time window; the first detection interval represents the time difference between the detection and sensing by the first sensor and the second sensor; The second detection time is calculated based on the first detection interval and the first detection time.

8. A multi-sensor sensing synchronization system, characterized in that, include: The first sensor, the second sensor, and the control module; The control module connects the first sensor and the second sensor; The control module is used to obtain a first detection time and, based on the first detection time, control the first sensor to perform detection and sensing at the first detection time. And to acquire the second detection time, and control the second sensor to perform detection and sensing at the second detection time based on the second detection time; The first detection time and the second detection time are configured such that when the second sensor and the first sensor perform detection and sensing at the second detection time and the first detection time respectively, the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing; wherein the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable; wherein, when the first sensor is a camera and the second sensor is a radar sensor; then: obtaining the second detection time specifically includes: obtaining first detection information; the first detection information represents the detection information collected by the camera; and the first detection information includes the first detection time; obtaining a first time; the first time represents the time when the first detection information is obtained; calculating a first delay based on the first time and the first detection time; the first delay represents the difference between the first detection time and the first time; calculating the second detection time based on the first delay, the first inter-frame exposure interval, the first detection interval, and the first time; the first inter-frame exposure interval represents the inter-frame exposure interval of the camera; the first detection interval represents the time difference between the first sensor and the second sensor performing detection and sensing.

9. A multi-sensor sensing synchronization system according to claim 8, characterized in that, The control module includes: a control unit, a storage unit, a timing unit, and a computing unit; The control unit is used to control the first sensor to perform detection and sensing at the first detection time, or / and to control the second sensor to perform detection and sensing at the second detection time. The storage unit is used to store the first detection time window and the first inter-frame exposure interval. The timing unit is used to acquire the first detection time and the second detection time, and when the first detection time and the second detection time are acquired, to trigger the control unit to control the first sensor to perform detection and sensing at the first detection time, or to trigger the control unit to control the second sensor to perform detection and sensing at the second detection time. The calculation unit is used to calculate the second detection time based on the first detection time, the first detection time window, and the first inter-frame exposure interval, and send the second detection time to the timing unit.

10. A multi-sensor sensing synchronization system, characterized in that, include: The first sensor, the second sensor, and the control module; The control module connects the first sensor and the second sensor; The control module is used to obtain a first detection time and, based on the first detection time, control the first sensor to perform detection and sensing at the first detection time. And to acquire the second detection time, and control the second sensor to perform detection and sensing at the second detection time based on the second detection time; Wherein, the first detection time and the second detection time are configured such that: when the second sensor and the first sensor respectively perform detection and sensing at the second detection time and the first detection time, the time window of the second sensor's detection and sensing is aligned with the time window of the first sensor's detection and sensing; wherein, the data acquisition timing of the first sensor is uncontrollable, while the data acquisition timing of the second sensor is controllable; wherein, when the first sensor is a radar sensor and the second sensor is a camera; then: obtaining the second detection time specifically includes: obtaining second detection information; the second detection information represents the detection information collected by the radar sensor; and the second detection information includes the radar sensor's detection... The distance and radial velocity of the rotating reflector relative to the radar sensor at the midpoint of the time window; obtaining the second moment; the second moment represents the moment when the second detection information is acquired; calculating the second delay based on the second detection information, the first relation table, and the second moment; the first relation table represents the pre-obtained correspondence table of the distance, radial velocity, and nominal time characteristics of the rotating reflector relative to the radar sensor; the second delay represents the difference between the midpoint of the detection time window of the radar sensor and the second moment; calculating the second detection moment based on the second delay, the first inter-frame exposure interval, and the second moment, where the first inter-frame exposure interval represents the inter-frame exposure interval of the camera.

11. A multi-sensor sensing synchronization system according to claim 8 or 10, characterized in that, Multi-sensor sensing synchronization systems also include: A first control line; the control module is connected to the second sensor through the first control line; so that the control module performs data acquisition timing on the second sensor through the first control line; the data acquisition timing on the second sensor indicates that at the second detection time, the control module transmits a first trigger signal to the second sensor through the first control line, triggering the second sensor to perform detection and sensing at the second detection time.

12. A multi-sensor sensing synchronization system according to claim 8 or 10, characterized in that, Multi-sensor sensing synchronization systems also include: First control line; used for timing data acquisition from the second sensor; The second control line; the control module is connected to the first sensor through the second control line so that the control module performs data acquisition timing on the first sensor through the second control line; the data acquisition timing on the first sensor indicates that at the first detection time, the control module transmits a second trigger signal to the first sensor through the second control line, triggering the first sensor to perform detection and sensing at the first detection time.

13. An electronic device, characterized in that, Includes a processor and a memory, wherein the memory is used to store code; The processor is configured to execute code in the memory to implement the multi-sensor perception synchronization method according to any one of claims 1 to 7.

14. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for multi-sensor sensing synchronization as described in any one of claims 1 to 7.

Images