Synchronization of multi-sensor systems

Abstract

The present disclosure relates to signal synchronization of a multi-sensor system. In one aspect, the present disclosure provides a method comprising identifying, by a processor on a circuit, a trigger signal, wherein the circuit is coupled to a plurality of groups of sensors. Each group of sensors has one or more sensors, each sensor having a respective sensor type. For each sensor in each of the plurality of groups of sensors, a synchronization trigger signal is generated based at least on the respective sensor type using the identified trigger signal. The synchronization trigger signal, once received by the respective sensor, causes the respective sensor to synchronize acquisition of sensor data for a particular scene. By sending the synchronization trigger signal to the respective sensors in the plurality of groups of sensors, execution of data acquisition is controlled.

Description

Technical Field

[0001] This disclosure relates to a synchronous trigger signal for controlling the execution of a multi-sensor system. Background Technology

[0002] Some multi-sensor systems may consist of one or more sensors used to collect sensor data for a specific scene. The collected sensor data can be analyzed for various applications, such as determining environmental changes, detecting one or more objects in the scene, and determining the distance of objects from the sensors. An example application of multi-sensor systems is autonomous driving. In autonomous driving, a circuit or processor receives scene sensor data from multiple sensors and processes that data, for example, to determine the driving trajectory of the autonomous vehicle. Ideally, the collected data should represent the scene at the same time or time step. However, one or more sensors may be asynchronous due to control or transmission delays, resulting in the collected sensor data representing the scene at two or more different times or time steps. Summary of the Invention

[0003] This disclosure provides a technique for synchronizing multiple sensors in a multi-sensor system. More specifically, the technique generates synchronization trigger signals to control the execution of multiple sensors, thereby triggering each sensor to collect sensor data at the same timestamp with the same period, the same frequency, or the same period and frequency. Furthermore, the technique can further ensure that each collected sensor data is associated with a precise timestamp using a Global Navigation Satellite System (GNSS) and optionally after a calibration process. To generate the synchronization trigger signals, the technique can customize synchronization triggers for each sensor in one or more sensor groups based on the characteristics of the sensor or sensor group to compensate for transmission or control delays caused by sending the trigger signals and to coordinate the synchronization trigger signals with the specific sensing tasks of the system. Each synchronization trigger signal generated for a given sensor can specify one or more of the following: the frame rate at which the sensor collects sensor data, the phase used to trigger collection (e.g., a pulse signal), the pulse width satisfying its respective duty cycle, or the polarity based on the sensor's specific application.

[0004] One aspect of the subject matter described in this disclosure can be embodied in a method for generating a synchronization trigger signal for controlling the execution of multiple sensors in a multi-sensor system. The method further includes: identifying the trigger signal via a processor on circuitry, wherein the circuitry is coupled to multiple groups of sensors. Each group of sensors includes one or more sensors, each having a respective sensor type. The method further includes generating a synchronization trigger signal for each sensor in each group of sensors using the identified trigger signal, at least based on the respective sensor type. Once the synchronization trigger signal is received by the corresponding sensor, the corresponding sensor synchronously acquires sensor data for a specific scene. The synchronization trigger signal is sent to the corresponding sensor in the multiple groups of sensors to control the execution of data acquisition. Other embodiments of this aspect include corresponding apparatus, systems, and computer programs encoded on a computer storage device, configured to perform various aspects of the method.

[0005] These and other embodiments may optionally include one or more of the following features. In some aspects, one group of the plurality of sensors may include: a first sensor having a first sensor type and a second sensor having a second sensor type. The synchronization trigger signal of the first sensor and the synchronization trigger signal of the second sensor may specify their respective frame rates, respective phases, respective pulse widths, or respective polarities to trigger sensor operation.

[0006] To identify the trigger signal via a processor, a system implementing the technology can select a trigger signal received by the processor from the master sensor in one of a set of multiple sensors as the identification trigger signal. Alternatively / further, the system can generate a trigger signal based on time data received from GNSS, which includes pulses per second (PPS) signals. Alternatively, the system can have the processor generate the trigger signal based on a local timer or counter in the circuitry.

[0007] Before generating the synchronization trigger signal, the method may further include calibrating the identified trigger signal using a local timer or counter in the circuit.

[0008] One of the synchronization trigger signals may have a different frame rate, phase, polarity, or pulse width than the identified trigger signal. The respective sensor types may include image sensors, audio sensors, or laser sensors. Furthermore, one group of sensors may include four cameras and one LiDAR.

[0009] In some cases, the first synchronization trigger signal for a first sensor in a set of multiple sensors may have a different phase than the second synchronization trigger signal for a second sensor in a set of multiple sensors. The phase difference can be determined by the processor based on the time intervals during which the first and second synchronization trigger signals are respectively sent to the first and second sensors.

[0010] In some embodiments, the system may use a local timer or counter in the circuit or a timestamp specified by GNSS to calibrate the identification trigger signal of the multiple sets of sensors.

[0011] If the identified trigger signal is a signal received from the master sensor in one of a group of sensors, the system can transmit the synchronization trigger signal to one or more working sensors in the group of sensors.

[0012] In some cases, the circuit includes a multiplexer unit for sending one of the synchronization trigger signals to a corresponding active sensor. The multiplexer unit is coupled to a first Inter-Integrated Circuit (I2C) primary-worker bus for the first synchronization trigger signal and a second I2C primary-worker bus for the second synchronization trigger signal. The system may receive a selection signal from the broadcast data bus at the multiplexer unit and select one of the first and second synchronization trigger signals to send to the corresponding active sensor based on the selection signal.

[0013] To determine the duration of signal transmission, the system can determine the duration by calculating the offset between a first time when the time data including the PPS signal is received from the GNSS and a second time when the PPS signal is transmitted to the corresponding component in the circuit using a sequence of telemetry packets.

[0014] The contents described in this disclosure can be implemented in certain embodiments to achieve one or more of the following advantages.

[0015] The technology improves the fusion accuracy of sensor data collected from different sensors in a multi-sensor system, which further enhances the accuracy of downstream applications of the fused sensor data (e.g., image generation and processing, component control in robots or autonomous vehicles, or sensor system optimization). Sensor data from different sensors may be mismatched due to differences in timestamps, triggering, transmission delays, sampling frequencies or phases, or pulse widths. Mismatched sensor data introduces undesirable noise or errors into the fusion algorithm, further leading to inaccurate scene perception.

[0016] Therefore, to generate an accurate overall understanding of the scene, the technique ensures that each sensor in a multi-sensor system simultaneously or coordinately acquires scene-related sensor data within one or more time steps (e.g., collecting sensor data at the same time step in all or at least some of the time steps), reducing unwanted measurement noise that leads to fusion errors. To achieve this, the technique sets a reference time for generating trigger signals. This reference time can be a local timer or counter in the main circuitry, or time data from GNSS (or Global Positioning System, GPS). Furthermore, the system compensates for different experienced time periods (also known as delays) when sending trigger signals from the main circuitry (or the processor in the main circuitry) to the corresponding sensors, ensuring that the sensors are triggered substantially simultaneously to acquire sensor data in a coordinated manner.

[0017] Furthermore, the synchronous trigger signal generated by this technology is compatible with various multi-sensor systems, which may have various sensor types and numbers in one or more sensor groups. The synchronous trigger signal can take into account the different characteristics of the multi-sensor system. For example, these system characteristics may include different sensors, different routing or physical connections between sensors and the main circuitry, cross-sensor or cross-sensor group conditions, different driving software in the sensors for processing the received trigger signal, or other system characteristics. Different sensors may also have different characteristics. For example, sensor characteristics may include different operating or sampling frame rates, allowed pulse widths (such as duty cycles), preferred signal polarities for cross-sensor compatibility, or other sensor characteristics. By taking these characteristics into account, the synchronous trigger sensor is compatible with different systems and various task requirements.

[0018] Furthermore, the technique improves the efficiency of generating synchronization trigger signals and broadcasting these signals to the corresponding sensors. For example, the technique does not require significant computational resources to generate synchronization trigger signals. Instead, the generation process involves minimal computation and time. For instance, a processor can generate synchronization trigger signals for a working sensor based on the operating characteristics of the master sensor. As another example, the synchronization trigger signals can be generated based on a local timer or time data from GNSS. The technique can use a high-speed data bus, such as one or more serial communication data buses or an I2C master working data bus, to broadcast the generated trigger signals to the corresponding sensors. One or more multiplexer (MUX) units and broadcast buses can be used to further assist the broadcasting by switching trigger signals from different master buses.

[0019] Details of one or more embodiments described in this disclosure will be set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of this disclosure will become apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1 This is a block diagram of a multi-sensor system in one embodiment;

[0021] Figure 2 A schematic diagram of a sensor group with its own sensors in a multi-sensor system is shown.

[0022] Figure 3 This is a block diagram of the signal transmission process between the sensor group and the synchronization subsystem in one embodiment;

[0023] Figure 4 This is a block diagram of the signal transmission process between the sensor group and the synchronization subsystem in another embodiment;

[0024] Figure 5 This is a partial block diagram of an example implementation of a multi-sensor system;

[0025] Figure 6 This is a schematic diagram of the trigger signal and the synchronization trigger signal in one embodiment;

[0026] Figure 7 This is a schematic diagram of the trigger signal and the synchronization trigger signal in another embodiment;

[0027] Figure 8 This is a flowchart of an example process for controlling the execution of a synchronous trigger signal of a sensor in a multi-sensor system, as described in one embodiment.

[0028] Similar reference numerals and names in different figures indicate similar elements. Detailed Implementation

[0029] In general, this disclosure describes systems and techniques for generating synchronization trigger signals that, once received and executed by processors in the respective sensors, cause the respective sensors to perform sensor operations in a coordinated manner. In this disclosure, the term "coordinatedly" means that sensors are triggered to acquire sensor data at substantially the same time, and the acquisition process allows data from a first sensor to be matched with other sensors in one or more sensor groups (not necessarily for every acquisition cycle). Furthermore, each acquired sensor data (e.g., image data, reflected light signals) is tagged with an accurate timestamp for fusion. The term "fusion" generally refers to aggregating sensor data from different sensors over one or more time steps for analyzing a scene (or one or more components of a scene). To generate the synchronization trigger signal, the technique takes into account the characteristics of the system and sensors and uses a reference timer (e.g., a local timer in the main circuitry or external time data from GNSS). In some embodiments, the system calibrates the local timer with external time data and / or coordinates received input triggers with the local timer or GNSS time data. Furthermore, the technique can use a high-speed data bus, such as an I2C master-slave bus, to broadcast the synchronization trigger signal. In some cases, a system implementing the technology may include one or more multiplexer units (MUXs) and a broadcast data bus for broadcasting synchronization trigger signals.

[0030] Figure 1 This is a block diagram of a multi-sensor system 100 according to one embodiment of this disclosure. For example, system 100 may include multiple sensor groups 110 and a synchronization subsystem 130. System 100 may be a system implemented on one or more computers at one or more locations, wherein the systems, components, and techniques described below can be implemented. Some components of system 100 may be computer programs configured to run on one or more computers. The synchronization subsystem 130 may include any suitable engine or algorithm configured to generate synchronization trigger signals for controlling the sensors in the multi-sensor system 100 according to various tasks and requirements.

[0031] The synchronization subsystem 130 is configured to generate a synchronization trigger signal 155 for controlling the sensor operation of multiple sensors. Each sensor group 110 may include multiple sensors 112. Although the system 100 is referred to as a multi-sensor system, the system 100 may be external to the multiple sensors 112. The synchronization subsystem 130 on the main circuit can function independently as a system for generating the synchronization trigger signal, and the sensors in the multiple sensor groups can be coupled to the main circuit through various I / O ports.

[0032] The sensors in the multiple sensors 112 can be grouped into specific sensor groups in sensor group 110 according to different sensing tasks and requirements. The type and number of sensors in a sensor group can be specified according to various tasks and requirements. For example, a first group of sensors may include four video sensors and one position sensor for collecting data on components near the robot. For example, the robot may be an autonomous vehicle. The video sensors may include cameras, and the position sensor may include light detection and ranging sensors (LiDAR). As another implementation, a second group of sensors may include a single camera for determining a longer field of view for the autonomous vehicle. As another implementation, a third group of sensors may include dynamic sensors configured to determine vehicle dynamics, such as speed, acceleration, and position. While the embodiments of sensor groups listed above include cameras, LiDAR, and dynamic sensors, different numbers of other types of sensors may be included in the sensor groups depending on the different tasks. For example, another group of sensors may include one or two LiDARs and multiple forward-looking cameras.

[0033] Furthermore, the number of sensor groups coupled to the circuit can be specified according to different tasks. In some embodiments, the autonomous driving system includes multiple autonomous vehicles, each with onboard circuitry for controlling the operation of vehicle sensors. In this case, each circuit can be coupled to a different number of sensor groups, and different types and numbers of sensors within each group. In some embodiments, the aforementioned different circuits can control the corresponding sensors through communication coupling. For example, multiple circuits can be integrated into a cluster or subsystem to control the operation of multiple sensors, which can simplify system design and reduce overall cost.

[0034] The synchronization subsystem 130 includes a trigger generator 140. The trigger generator 140 is configured to determine the respective sampling frequency, pulse width, and signal phase of the trigger signal for each sensor, so that these trigger signals synchronously trigger sensor operation in a coordinated manner. To generate a synchronization trigger signal 155, the trigger generator 140 can process one or more input trigger signals 115 received from the corresponding sensor 112. In some embodiments, the trigger generator 140 can use a local timer or counter 145 on a host device (e.g., main circuitry) to generate the synchronization trigger signal. The local timer or counter 145 can also be used to determine a timestamp associated with the trigger signal. Generally, a local timer of the main circuitry emits a local clock signal to timestamp a specific event or process, and a local counter of the main circuitry is configured to count and store the number of times a specific event or process has occurred. Although the counter is primarily used to count the number of times a specific event has occurred in the main circuitry, the count can be associated with a clock signal, and therefore the counter can be used as a timer in this technique. In this document, the local timer or counter is also referred to as an internal timer or counter.

[0035] In some cases, the trigger generator 140 can use time data 135 from GNSS 105 to generate a synchronization trigger signal. The time data 135 may include, for example, a PPS signal, which is precisely used to generate the frame-level synchronization trigger signal. Furthermore, the PPS signal has little or no accumulated error, thus maintaining time and frequency accuracy over a considerable period. The frequency at which the time data 135 is received at the synchronization subsystem 130 can serve as the base frequency for generating the synchronization trigger signal. The time data 135 can also be used as a reference time for timestamps.

[0036] like Figure 1As shown, the synchronization subsystem 130 may include a calibration engine 150 configured to calibrate data. In some cases, the calibration process may be performed before the trigger generator 140 generates a synchronization trigger signal. The calibrator compares the time difference between time data 135 received from GNSS 105, a clock signal emitted from a local counter or timer 145, and an input trigger signal 115 received from sensor 112. For example, the calibrator 150 may reduce the cumulative error or difference between the received time data 135 and the time value indicated by the local timer or counter 145. Assuming that the time data 135 emitted by GNSS 105 is long-term accurate, the subsystem 130 will adjust the local timer or counter 145 to match the time data 135. For example, assuming a GNSS signal period is received every 3.3 milliseconds, the subsystem 130 will check whether the local timer 145 measures a period of 3.3 milliseconds. If not, the subsystem 130 will update the local timer 145 to match the GNSS signal. In some implementations, the subsystem 130 may use the difference as a factor in phase change to generate a synchronization trigger signal.

[0037] In another implementation, the calibrator 150 can determine the timestamp accuracy of the received input trigger signal 115. The synchronization subsystem 130 can compare the timestamp specified by time data 135 published from GNSS 105 or by a local counter or timer 145 with the timestamp associated with the input trigger signal 115. If they do not match, the subsystem 130 can update the timestamp associated with the input trigger signal 115. In some cases, when using the received input trigger signal 115 to generate a synchronization trigger signal, the subsystem 130 can use the timestamp difference as a factor in phase change.

[0038] In some implementations, the calibration process may also be performed after the trigger generator 140 generates the synchronization trigger signal. For example, the calibrator 150 may determine the time error or difference between the generated synchronization trigger signal and the time data 135 received from GNSS 105 or the clock signal emitted by a local counter or timer 145. Once the time error or difference is detected, the subsystem 130 may update the synchronization trigger signal with a corrected time or timestamp. In some cases, the calibrator 150 may use the time or timestamp difference as a factor in phase change to update the existing synchronization trigger signal according to the phase change.

[0039] Figure 2The diagram illustrates a structure 200 of sensor groups 205, 207, and 209 in a multi-sensor system according to an embodiment, wherein the sensor groups have sensors 210a-n, 220a-o, and 230a-p, respectively. The multi-sensor system can be integrated with… Figure 1 The multi-sensor system is similar to 100.

[0040] like Figure 2 As shown, the plurality of sensor groups 205, 207 and 209 are communicatively coupled to the trigger generator 240 (similar to the above description). Figure 1 The trigger generator 240 (mentioned above) can send a synchronization trigger signal 208 to one or more sensors in a corresponding sensor group. In some cases, one or more sensors can provide an input trigger signal to the trigger generator 240 as input to generate the synchronization trigger signal 208, which will combine Figure 3 To describe in more detail.

[0041] Each main circuit (and the trigger generator 240 included in the circuit) can be coupled to multiple sensor groups 205, 207, and 209. Although Figure 2 Three sensor groups coupled to trigger generator 240 are shown, but it will be understood that trigger generator 240 can be coupled to any suitable number of sensor groups, such as one, two, five, ten or other suitable number.

[0042] Similarly, each sensor group can include its own number of sensors and its own sensor type; each group does not need to have the same number of sensors. For example... Figure 2 As shown, sensor group 205 may include sensors 210a-n, sensor group 207 may include sensors 220a-o, and sensor group 209 may include sensors 230a-p. As a simple example, sensors 210a-n may include three sensors, all of which are motion sensors; sensors 220a-o may include five sensors, all of which are lidar sensors; and sensors 230a-p may include seven sensors, including two motion sensors, one lidar sensor, and four cameras. The number of sensors represented by n, o, and p can be different or the same. It should be understood that the above-described number and type of sensors are for illustrative purposes. Other suitable numbers and types of sensors can be configured for the sensor groups according to different sensing requirements and tasks.

[0043] In some cases, the system or user may designate one or more sensors from one or more sensor groups as master sensors, while the remaining sensors in a multi-sensor system may be designated as active sensors. The term "master sensor" typically refers to a sensor triggered by a specific trigger signal. This specific trigger signal may specify at least one or more trigger data, including sampling frame rate, pulse width, sampling phase, or preferred polarity. This specific trigger signal may be generated by the system (e.g., synchronization subsystem 130, ...). Figure 1 Trigger generator 140 or Figure 2 The trigger generator 240) processes the signal to generate a synchronous trigger signal (e.g., Figure 1 The synchronous trigger signal 155 or Figure 2 The term "working sensor" generally refers to a sensor triggered by a synchronization trigger signal, which is generated based on a specific trigger signal from a master sensor. For example, the master sensor could be a LiDAR with a sampling frequency of 10 Hz or 10 frames per second, and the corresponding working sensor could include multiple cameras with a sampling frequency of 20 Hz or 20 frames per second. Another example is a LiDAR with a sampling frequency of 10 Hz or 10 frames per second and multiple cameras with a sampling frequency of 30 Hz or 30 frames per second. Details of the master and working sensors, and their relationship to the synchronization trigger signal, will be discussed in conjunction with... Figures 3 to 5 Describe it.

[0044] Figure 3 This is a block diagram of an example process 300 for signal transmission between a sensor group 305 and a synchronization subsystem 330 in one embodiment. The sensor group 305 can be connected to... Figure 1 Sensor group 110 or Figure 2 The sensor groups 205, 207, or 209 are similar. The synchronization subsystem 330 can be connected to… Figure 1 The synchronization subsystem 130 is similar.

[0045] like Figure 3 As shown, the sensor group 305 may include multiple sensors 310a-n. Each sensor 310a-n may have a specific sensor type, including a camera, LiDAR, motion sensor, thermal sensor, or other suitable sensor. To generate a synchronous trigger signal for controlling the operation of the sensors 310a-n, the system or user may designate one of the sensors 310a-n in the sensor group 305 as the master sensor 310a and designate the remaining sensors 310b-n in the sensor group 305 as working sensors. For example, the master sensor 310a may be a LiDAR, and the working sensors may include multiple cameras.

[0046] The sensor group 305 or the main sensor 310a can provide a specific trigger signal controlling the operation of the main sensor 310a as an input trigger signal 315 to the synchronization subsystem 330. The input trigger signal 315 can be provided to the synchronization subsystem 330 wirelessly or via physical lines using any suitable data communication technology. In one embodiment, data transmission includes using general-purpose input and output (GPIO) ports integrated into the main circuitry (or synchronization subsystem 330). The synchronization subsystem 330 (or Figure 1 The trigger generator 140 can generate a synchronous trigger signal 308 based on the input trigger signal 315. For example, the trigger generator 340 can directly select the input trigger signal 315 as the synchronous trigger signal 308 for the working sensors 310b-n.

[0047] As another implementation, a trigger generator (not shown, but similar to) Figure 1 The trigger generator 140 can generate the synchronization trigger signal 308 by modifying the phase of the input trigger signal 315, thereby triggering the working sensors 310b-n to collect and / or transmit sensor data substantially simultaneously with the main sensor 310a. The phase modification can be determined based on the delay of transmitting or processing control signals. For example, when the main sensor 310a is a lidar with a sampling frequency of 10 Hz, and the working sensors include multiple cameras with a sampling frequency of 20 Hz, the synchronization trigger signal 308 can specify a 10 Hz sampling frequency for the camera working sensors. In some cases, due to various delays caused by different routing or architectures, and optionally due to the processing time required by the sensor driver, the synchronization trigger signal 308 may have slightly different phases for different working sensors. In cases where the working sensors have different duty cycle sizes (i.e., the ratio of the effective pulse width to the pulse period), the synchronization trigger signal 308 may have its own pulse width. Furthermore, even for the same type of working sensors, the synchronization trigger signal 308 may have different pulse polarities due to different task requirements. More examples of synchronization trigger signals will be discussed in conjunction with... Figure 6 and Figure 7 Provide a detailed description.

[0048] In some cases, the synchronization subsystem 130 is configured to calibrate the received input trigger signal 115 using a reference timestamp. The reference timestamp can be determined using time data from a GNSS system or by a timer or counter locally contained in the main circuitry. This allows the system to determine the accurate time to trigger the operating sensor, thereby determining phase adjustment (if necessary) to generate a synchronization trigger signal from the input trigger signal 115.

[0049] Figure 4 This is a block diagram of an example process 400 for signal transmission between sensor group 405 and synchronization subsystem 430 in another embodiment. The sensor group 405 can be connected to... Figure 1 Sensor group 110 or Figure 2 The sensor groups 205, 207, or 209 are similar. The synchronization subsystem 430 can be connected with... Figure 1 It is similar to the synchronization subsystem 130.

[0050] and Figure 3 The example process shown is different from 300. Figure 4 Sensor group 405 does not include a master sensor (or equivalently, the multi-sensor system 100 or the user does not specify a master sensor from the sensors in sensor group 405). Instead, the sensors in sensor group 405 are all operational sensors 410a-n.

[0051] To determine the synchronization trigger signal 408 for controlling the operation of the working sensors 410a-n, the synchronization subsystem 430 can receive the PPS signal 425 from the GNSS 420. The GNSS 420 can be connected to... Figure 1 Similar to GNSS 105.

[0052] The synchronization subsystem 430 can receive the PPS signal 425 via a GPIO port. In some cases, the synchronization subsystem 430 is configured to calibrate the received PPS signal 425 with a reference timestamp. The reference timestamp can be determined based on a timer or counter locally included in the main circuitry. In this way, the system can determine the accurate time to trigger the operating sensor, thereby determining phase adjustment (if necessary) to generate a synchronization trigger signal from the input trigger signal 115.

[0053] To determine the accurate timestamp of an event (e.g., receiving and transmitting a PPS signal), the subsystem 130 can further determine the elapsed time period for transmitting the PPS signal. The PPS signal can be received at the main circuitry and transmitted to the subsystem using a telemetry data packet sequence. To determine the elapsed time period, the subsystem 130 can, for example, determine a first time at the rising edge of the received PPS signal. The subsystem can further determine the period of transmitting the received PPS signal by transmitting a data packet sequence. For example, assuming the time required to transmit each data packet is approximately 2 μs, the total number of transmissions for each PPS signal is 16, and the recharge time after the last transmission is approximately 18 μs, then the total elapsed time required to transmit and recharge all 16 data packets is 50 μs, equal to (16 × 2 + 18) μs.

[0054] By determining the elapsed time period, the subsystem 130 can determine an accurate timestamp representing the time point at which the operating sensors 410a-n in the sensor group 405 are triggered. An example algorithm is described below. Assume the number of points DI (data point index) in the transmission sequence is 0-15, and the number of transmission sequences SI (sequence index) is 0-23. The total time offset T_0 = (elapsed time period × SI + transmission time × DI) μs. Assume the timestamp of the telemetry data packet is T_v (timestamp value), then the exact time point of each PPS signal is T_e = T_v + T_o.

[0055] The synchronization trigger signal 408 is generated based on the received PPS signal. Therefore, the synchronization trigger signal 408 and the received PPS signal can share a sampling frequency. However, in some cases, the synchronization trigger signal can specify different frequencies for different operating sensors 410a-n in the sensor group 405. Due to the time intervals during which the PPS signal is received and transmitted on the main circuit described above, the synchronization trigger signal 408 may have a different phase than the PPS signal.

[0056] In some cases, the synchronization subsystem 430 generates the synchronization trigger signal 408 based on a local timer instead of the PPS signal. In some cases, the synchronization trigger signal 408 may have the same sampling rate as the local timer or counter. Optionally, the synchronization trigger signal 408 may have a lower sampling rate than the local counter, for example, sampling data every four counts. Similarly, the synchronization trigger signal 408 may have its own phase, pulse width, and polarity depending on the respective sensor and sensing task.

[0057] Once the synchronization trigger signal 408 is generated, the synchronization subsystem 430 can use appropriate data communication technology to send the synchronization trigger signal 408 to the individual working sensors 410a-n in the sensor group. For example, as described above, a physical connection such as a GPIO port can be used to transmit the signal.

[0058] Figure 5 This is a partial block diagram of an example process 500 of a multi-sensor system in one embodiment. A portion of the multi-sensor system is configured to broadcast a synchronization trigger signal. Sensor group 505 can be... Figure 1 Similar to sensor group 110, and the working sensor 510a-n can be with Figure 1The sensor 112 is similar. The example process 500 described is based on a specific implementation where the synchronization trigger signal is determined based on input trigger signals from one or more master sensors. As described above, a sensor group may include zero, one or more master sensors and one or more working sensors. Each master sensor may provide an input trigger signal to a synchronization subsystem, such as synchronization subsystem 130, to generate a synchronization trigger signal. The synchronization trigger signal can be used to control the operation of working sensors in the same sensor group or other sensor groups of the master sensors, or to control both simultaneously. Thus, a working sensor may be assigned to one or more master sensors.

[0059] Generally, synchronization trigger signals generated by different master sensors can be transmitted to the corresponding working sensors using their respective data buses. For example, ... Figure 5 As shown, parts of the multi-sensor system may include different I2C master data buses to send synchronization trigger signals. More specifically, the system may use a first I2C master data bus 520a to send synchronization trigger signals 530a to the corresponding working sensors 510a-n, and use a second I2C master data bus 520b to send synchronization trigger signals 530b to the corresponding working sensors 510a-n.

[0060] Considering that a single working sensor may be "indirectly" controlled by multiple master sensors, the multi-sensor system (e.g., Figure 1 System 100 needs to determine which of a plurality of synchronization trigger signals a working sensor should receive. To determine this, the system includes a plurality of MUX units 515a-n, each assigned to a corresponding working sensor 510a-n. Each MUX unit is also coupled to a corresponding I2C master bus 520a or 520b. The multi-sensor system or user can instruct the MUX units to select one of the plurality of I2C master buses to allow the corresponding working sensor to receive the synchronization trigger signal transmitted along the selected bus. In some embodiments, the system also includes a broadcast bus 550 to facilitate a broadcast process. The broadcast bus is coupled to each of the MUX units 515a-n. The system or user can specify an I2C master bus using a selection signal 560. Once a MUX unit receives the selection signal 560 along the broadcast bus 550, the MUX unit selects the I2C master bus specified in the selection signal 560 and allows the corresponding working sensor 510a-n to receive the synchronization trigger signal transmitted along the selected bus.

[0061] Figure 6 and Figure 7 Examples 600 and 700 of the trigger signal and synchronous trigger signal are shown respectively.

[0062] refer to Figure 6 Trigger generator in a multi-sensor system (e.g., system 100) (e.g., Figure 1 The trigger generator 140 can be configured to adjust the sampling frame rate, sampling phase, or both based on the input trigger signal 610. For example, the trigger generator can determine a synchronization trigger signal 620 with a higher sampling frame rate than the input trigger signal 610. Alternatively, the trigger generator can determine a synchronization trigger signal 630 with a lower sampling frame rate than the input trigger signal 610. Both cases can be applied when the main sensor and the assigned working sensors have different sampling rates. As a simple implementation, by default, the main sensor can be a LiDAR with a sampling frame rate of 20 Hz and multiple cameras with a sampling frame rate of 10 Hz. In this case, the trigger generator can increase the sampling frame rate of the cameras from 10 Hz to 20 Hz, so that the LiDAR and cameras can operate at the same sampling frame rate. In another implementation, the main sensor can be a first dynamic sensor with a maximum sampling frame rate of 30 Hz, and the working sensors can include multiple dynamic sensors with a sampling frame rate of 50 Hz. Since the sampling frame rate of the first dynamic sensor cannot be increased, the trigger generator can reduce the sampling rate of the working sensors to 30 Hz. In this way, the primary motion sensor and the working motion sensor can operate at the same sampling frame rate.

[0063] In addition, such as Figure 6 As shown, the trigger generator can generate a synchronous trigger signal 640 with a different phase and a higher sampling rate than the input trigger signal 610. Alternatively, the trigger generator can generate a synchronous trigger signal 650 with a different phase and a lower sampling rate than the input trigger signal 610. The sampling rate difference is similar to that described above. As mentioned above, the phase difference between the synchronous trigger signals 640 and 650, and between them and the input trigger signal 610, is used to compensate for the epochal time of signal transmission from the main circuit to the corresponding operating sensor. In some cases, the phase difference can also be attributed to differences between various times determined during an optional calibration process.

[0064] refer to Figure 7 Trigger generator (e.g., Figure 1 The trigger generator 140 in the middle can further generate synchronous trigger signals with different pulse widths and polarities for different working sensors.

[0065] The trigger generator can generate a first synchronous trigger signal 710 for a first working sensor A and a second synchronous trigger signal 720 for a second working sensor B. Note that the first and second synchronous trigger signals 710 and 720 have the same sampling frame rate and phase. More specifically, signals 710 and 720 can both have the same periods 713 and 723. However, the first synchronous trigger signal 710 has a pulse width 715 that is longer or greater than the pulse width 725 of the second synchronous trigger signal 720. The corresponding invalid width 717, i.e., the pulse interval, of the first synchronous trigger signal 710 is smaller than the invalid width 727 of the second synchronous trigger signal 720. Since different sensors can have their own preferred duty cycles, the difference in pulse width is determined according to the respective sensor type. As mentioned above, the term "duty cycle" in this disclosure generally refers to the ratio of the effective time to the total signal period (or pulse period), for example, the ratio of the pulse width 715 to the period 713. The duty cycle can include 1, 0.5, 0.25, or other suitable values. If the duty cycle is 0.5, the effective time and the ineffective time each account for 50% of the pulse period. If the duty cycle is 0.25, the effective time accounts for 25% of the pulse period, and the ineffective time accounts for 75% of the pulse period.

[0066] Furthermore, trigger generator 140 can also generate a third synchronous trigger signal 730 for the third working sensor C and a fourth synchronous trigger signal 740 for the fourth working sensor D. Similarly, the third and fourth synchronous trigger signals share the same sampling frame rate and phase (e.g., periods 733 and 743 have the same magnitude, and the pulses change simultaneously). The third synchronous trigger signal 730 has a larger duty cycle than the fourth synchronous trigger signal 740. However, the polarity of the third synchronous trigger signal 730 is opposite to that of the first synchronous trigger signal 710, while the polarity of the fourth synchronous trigger signal 740 is opposite to that of the second synchronous trigger signal 720. The polarity of the working sensor is determined according to the respective task requirements. For example, the sensor can be configured to activate at the peak of the received signal (also known as positive polarity or high-level activation). As another example, the sensor can be configured to activate at the trough of the received signal (also known as negative polarity or low-level activation).

[0067] Figure 8 This is a flowchart of a process 800 in a multi-sensor system provided by an embodiment, in which a synchronous trigger signal controls the execution of sensors. The operation of the process 800 can be performed by... Figure 1 The multi-sensor system 100 or more specifically, composed of Figure 1The process 800 is executed by the synchronization subsystem 130. The operation of the process 800 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory. Executing the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of the process 800.

[0068] This system is configured to generate a synchronization trigger signal for controlling the operation of multiple sensors in a multi-sensor system. The multi-sensor system may include multiple sensor groups, each group comprising multiple sensors of different types. The multiple sensors are coupled to a main circuit, which is configured to issue the synchronization trigger signal to control the multiple sensors. Although the system is named a multi-sensor system, in some embodiments, the system may include only one or more main circuits. The multiple sensors may be external to the main circuit and coupled to it via a physical port or wireless communication.

[0069] The system identifies trigger signals via a processor on circuit (810). As described above, the circuit is coupled to multiple sensor groups, and each sensor group includes multiple sensors of a respective sensor type. Sensor types may include image sensors, audio sensors, or laser sensors. A first group of the multiple sensor groups may include a first sensor of a first sensor type and a second sensor of a second sensor type. The first sensor may be, for example, a lidar, and the second sensor may be a camera. In some embodiments, the system may include multiple circuits, each coupled to multiple sensor groups with different numbers and types of sensors.

[0070] By identifying trigger signals, the system can select one or more input trigger signals from one or more master sensors, and select one input trigger signal as the identified trigger signal. The master sensors can be specified by the system or a user. The input trigger signals are typically suitable for controlling the sensor operation of the corresponding master sensor. The input trigger signals can specify one or more of frame rate (also referred to as sampling frame rate or sampling frequency in this disclosure), phase, pulse width, or polarity to instruct or control the corresponding master sensor to acquire sensor data.

[0071] Alternatively, the system can generate a trigger signal as the identification trigger signal based on time data received from GNSS. More specifically, the GNSS can specify a reference time for timestamped events, and the time data may include PPS signals periodically received by the main circuit. The system can generate a trigger signal based on the PPS signal, and the trigger signal can be used to generate a synchronization trigger signal to control the sensors in the system.

[0072] In some cases, the trigger signal for the identification can be generated by a time value or timestamp issued by a local timer or counter in the main circuit. Details of the local timer or counter are described above and below. Figure 1 The relevant parts have been described. The identified trigger signal can also be used to generate a synchronization trigger signal to control the operation of the sensors in the system.

[0073] The system generates a synchronization trigger signal for each sensor in each of the multiple sensor groups (820). More specifically, the system generates the synchronization trigger signal using the identified trigger signal, at least based on the respective sensor type. Once the synchronization trigger signal is received by the corresponding sensor and executed by the driver in the sensor, it enables the corresponding sensor to synchronously acquire sensor data for a specific scene. Generally, the synchronization trigger signal can specify the respective frame rate, phase, pulse width, or polarity for the corresponding sensor based on system characteristics and sensor characteristics. For example, the system can determine the synchronization trigger signal after determining and compensating for the phase difference caused by delays in signal transmission. As another implementation, the system can determine the synchronization trigger signal after considering the pulse width and polarity based on the different characteristics of the sensor. Further implementations of sensor characteristics and system characteristics can be found in the description above.

[0074] Because the system takes into account the characteristics of each sensor when generating the synchronization trigger signal, one or more synchronization trigger signals may have different frame rates, phases, polarities, or pulse widths than the identified trigger signal. For example, the synchronization trigger signal may have a higher or lower sampling frame rate, or a different phase than the identified trigger signal. In some cases, the synchronization trigger signal may have a different pulse width and polarity than the identified trigger signal, or different synchronization trigger signals may have different pulse widths and polarities.

[0075] Before generating the synchronization trigger signal, the system may perform a calibration process. For example, the system may calibrate the timestamp using time values ​​or data from GNSS or a local timer or counter. As another implementation, the system may calibrate the identified trigger signal using a local (or internal) timer or counter or using a timestamp specified by GNSS. In some embodiments, the system may also use time data received from GNSS to further calibrate the timestamp of the local timer or counter. Further details of the calibration process can be found in the above description. Figure 1 Related descriptions.

[0076] The system transmits a synchronization trigger signal to the corresponding sensor (830). After the corresponding driver on the sensor receives and processes the synchronization trigger signal, it triggers multiple sets of sensors to acquire sensor data of the scene. The corresponding sensors include working sensors assigned to one or more master sensors.

[0077] To broadcast synchronization trigger signals, the system may include a multiplexer unit corresponding to each working sensor. The multiplexer unit facilitates the broadcasting process because it can be configured to select one data bus from multiple data buses along which the synchronization trigger signal is transmitted. For example, the multiplexer may be coupled to a first I2C master working data bus for the first synchronization trigger signal and a second I2C master working data bus for the second synchronization trigger signal. The multiplexer unit is also coupled to a broadcast data bus to facilitate the broadcasting process. More specifically, the multiplexer unit is configured to receive a selection signal along the broadcast data bus and, based on the selection signal, select one of the first and second synchronization trigger signals and send it to the corresponding working sensor. The selection signal may be specified by the system or a user.

[0078] In order to generate an accurate timestamp or correct phase change in the trigger signal, the system can determine whether to receive the PPS signal and send it to the synchronization subsystem (e.g., Figure 1 The system determines the exact time point of the synchronization subsystem 130. More specifically, the system can determine the elapsed time period by calculating the offset between a first time when the trigger signal, including the PPS signal, is received from the GNSS at the main circuit and a second time when the PPS signal is sent to the subsystem using a telemetry data packet sequence. Details on determining the elapsed time period can be found in the above section. Figure 4 Related descriptions.

[0079] Embodiments of the subject matter and operations described in this disclosure can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, or in a combination of one or more of these, the hardware including the structures disclosed in this disclosure and their structural equivalents. Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution or control of its operation by a data processing apparatus. Alternatively or further, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated for encoding information for transmission to a suitable receiving device for execution by the data processing apparatus. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Furthermore, while the computer storage medium is not a propagation signal, it can be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0080] The operations described in this disclosure can be performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0081] The term "data processing device" includes various devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination of the foregoing. The device may include special-purpose logic circuitry, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, the device may also include code that creates an execution environment for the relevant computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The device and execution environment can implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0082] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file containing other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to said program, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program can be deployed and executed on one or more computers located at a site or distributed across multiple sites interconnected by a communication network.

[0083] The processes and logic flows described in this disclosure can be executed by one or more programmable processors that execute one or more computer programs to perform operations by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can also be implemented as dedicated logic circuitry, such as an FPGA or ASIC.

[0084] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor that executes operations according to instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is coupled to said mass storage devices for sending and receiving data. However, a computer does not require such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by or incorporated into dedicated logic circuits.

[0085] To provide interaction with the user, the specific embodiments described in this disclosure can be implemented on a computer having a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving files to and from the device used by the user, for example, by sending a webpage to the web browser in response to a request received from the web browser on the user's client device.

[0086] The specific embodiments described in this disclosure can be implemented in a computing system that includes backend components, such as a data server; or middleware components, such as an application server; or frontend components, such as a client computer with a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described in this disclosure; or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), the Internet, and peer-to-peer networks (such as self-organizing peer-to-peer networks).

[0087] The computing system may include clients and servers. Clients and servers are typically geographically separated and generally interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, and a client-server relationship exists between them. In some implementations, the server sends data (e.g., HTML pages) to client devices (e.g., to display data to users interacting with the client devices and to receive user input from users interacting with the client devices). Data generated on the client devices (e.g., the results of user interactions) can be received from the client devices on the server.

[0088] While this disclosure contains numerous specific implementation details, these details should not be construed as limiting the scope of protection of the invention or other claimed content, but rather as descriptions of features of particular embodiments of a particular invention. Specific features described in individual embodiments of this disclosure may also be implemented in combination in a single embodiment. Similarly, various features described in the context of a single embodiment may be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the foregoing features may be described as functioning in certain combinations, and even initially claimed in this manner, in some cases one or more features may be removed from the claimed combination, and the claimed combination may be a sub-combination or a variation of a sub-combination.

[0089] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown, or requiring all of the shown operations to obtain the desired result. In some cases, multitasking and parallel processing can also achieve the desired effect. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments; rather, it should be understood that the program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0090] Therefore, this disclosure has provided specific embodiments, and other embodiments fall within the scope defined by the following claims. In some cases, the operations described in the claims can be performed in a different order and the desired result can still be obtained. Furthermore, the processes shown in the drawings do not necessarily require the specific order shown to achieve the desired result. In some implementations, multitasking and parallel processing can also achieve the desired effect.

Claims

1. A method for generating a synchronous trigger signal for controlling the execution of multiple sensors, characterized in that, include: The trigger signal is identified by a processor on the circuit, which is coupled to multiple sets of sensors, each set of sensors having one or more sensors, and each sensor having its own sensor type; For each sensor in each group of sensors, a synchronization trigger signal is generated using an identified trigger signal based at least on the corresponding sensor type, wherein once the synchronization trigger signal is received by the corresponding sensor, the corresponding sensor causes the corresponding sensor to synchronously acquire sensor data for a specific scenario; The synchronization trigger signal is sent to the corresponding sensor in the multiple sets of sensors to control the execution of data acquisition; Wherein, the first synchronization trigger signal of the first sensor in the plurality of sensors has a different phase from the second synchronization trigger signal of the second sensor in the plurality of sensors, wherein the phase difference is determined by the processor based on the time period during which the first synchronization trigger signal and the second synchronization trigger signal are sent to the first sensor and the second sensor, respectively.

2. The method according to claim 1, characterized in that, One of the multiple sets of sensors includes a first sensor of a first sensor type and a second sensor of a second sensor type, wherein the synchronization trigger signal for the first sensor and the synchronization trigger signal for the second sensor specify their respective frame rates, phases, pulse widths, or polarities to trigger sensor operation.

3. The method according to claim 1, characterized in that, The processor identifies the trigger signal by selecting a signal received by the processor from the main sensor in one of a set of multiple sensors as the trigger signal for identification.

4. The method according to claim 1, characterized in that, The processor identifies the trigger signal, including: The signal generated based on the time data received from the Global Navigation Satellite System (GNSS) is used as the trigger signal for identification, where the received time data includes a pulses per second (PPS) signal.

5. The method according to claim 1, characterized in that, The processor identifies the trigger signal, including: The processor generates the trigger signal based on a local timer or counter in the circuit.

6. The method according to claim 1, characterized in that, Before generating the synchronization trigger signal, the process also includes: The identified trigger signal is calibrated using a local timer or counter in the circuit.

7. The method according to claim 1, characterized in that, One of the synchronous trigger signals has a different frame rate, phase, polarity, or pulse width than the identified trigger signal.

8. The method according to claim 1, characterized in that, The respective sensor types include image sensors, audio sensors, or laser sensors.

9. The method according to claim 1, characterized in that, One of the multiple sets of sensors includes at least one of a camera or a lidar.

10. The method according to claim 1, characterized in that, Also includes: The identified trigger signal is calibrated on the plurality of sensors using a local timer or counter in the circuit or a timestamp specified by GNSS.

11. The method according to claim 1, characterized in that, The trigger signal for identification is a signal received from the master sensor in one of the multiple sets of sensors, wherein sending the synchronization trigger signal to the corresponding sensor includes: The synchronization trigger signal is sent to one or more of the working sensors in the multiple sets of sensors.

12. The method according to claim 1, characterized in that, The circuit includes a multiplexer unit for sending one of the synchronization trigger signals to a corresponding working sensor. The multiplexer unit is coupled to a first I2C main bus for the first synchronization trigger signal and a second I2C main bus for the second synchronization trigger signal. Sending the synchronization trigger signal to the corresponding sensor includes: The multiplexer unit receives a selection signal from the broadcast data bus. as well as Based on the selection signal, one of the first synchronization trigger signal and the second synchronization trigger signal is selected and sent to the corresponding working sensor.

13. The method according to claim 1, characterized in that, Also includes: Determining the duration of signal transmission, wherein determining the duration of signal transmission includes: The elapsed time period is determined by calculating the offset between the first time when the trigger signal, including the PPS signal, is received from the GNSS and the second time when the PPS signal is sent to the corresponding component in the circuit using a telemetry data packet sequence.

14. A multi-sensor system, characterized in that, The system includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, perform the steps of the method according to any one of claims 1 to 13.

15. A non-transitory computer-readable storage medium having instructions stored thereon, characterized in that, When one or more computers execute the instructions, the steps of the method according to any one of claims 1 to 13 are performed.

Images