Multi-sensor data synchronization method, system and device and automobile

Abstract

The invention relates to a multi-sensor data synchronization method, system and device and an automobile, and the method comprises the steps: starting a local timing module of the automobile when the automobile is powered on, and obtaining the local timing time; the local timing time is transmitted to a first sensor, supporting time synchronization, of the automobile, so that the first sensor marks a first timestamp on the collected first data according to the local timing time; when second data collected by a second sensor which does not support time synchronization is obtained, marking a second timestamp on the second data by using local timing time; after the first data is received, the first data and the second data are synchronized by using the first timestamp of the first data and the second timestamp of the second data. According to the method, the timestamps of the data of the two sensors are marked by using the same local timing time, and data alignment is realized, so that the problem that the data of the sensors which do not support time synchronization cannot be synchronized is solved.

Description

Technical Field

[0001] The present application relates to the automotive field, and in particular to a multi-sensor data synchronization method, system, device and an automobile. Background Art

[0002] The intelligent driving system needs to face a complex driving environment, which requires the system to be able to fully and accurately perceive the surrounding situation. Different types of sensors (such as cameras, radars, and lidars) each have unique advantages and can provide information from different angles. For example, cameras are good at identifying traffic signs and pedestrians, radars can accurately measure the speed and distance of objects, and lidars provide high-precision three-dimensional environmental models. Through the collaborative work of multiple sensors, the system can better cope with various environmental changes and improve safety and reliability. In this process, time synchronization is particularly important. The data collection of each sensor may be carried out at different time points. Without precise time synchronization, the system may misjudge the position and state of the object, thereby affecting decision-making and reaction speed.

[0003] In the prior art, the same time can be used to synchronize different sensors, so that the data of different sensors can be synchronized. However, in the prior art, for sensors that cannot be synchronized in time, data synchronization cannot be performed. Summary of the invention

[0004] The present application provides a multi-sensor data synchronization method, system, device and a car to solve the technical problem in the prior art that data of sensors that do not support time synchronization cannot be synchronized.

[0005] In a first aspect, the present application provides a multi-sensor data synchronization method, including: when a car is powered on, starting a local timing module of the car to obtain a local timing time; transmitting the local timing time to a first sensor of the car that supports time synchronization, so that the first sensor adds a first timestamp to the collected first data according to the local timing time; when obtaining second data collected by a second sensor that does not support time synchronization, using the local timing time to add a second timestamp to the second data; after receiving the first data, using the first timestamp of the first data and the second timestamp of the second data to synchronize the first data with the second data.

[0006] In the second aspect, the present application provides a multi-sensor data synchronization system, including: a local timing module, which is started when the car is powered on to obtain a local timing time; a first sensor that supports time synchronization, which is used to receive the above-mentioned local timing time and stamp a first timestamp on the collected first data according to the above-mentioned local timing time; a second sensor that does not support time synchronization, which is used to collect second data; a processor, which is used to stamp a second timestamp on the above-mentioned second data according to the above-mentioned local timing time when receiving the above-mentioned second data; and after receiving the above-mentioned first data, use the above-mentioned first timestamp of the above-mentioned first data and the above-mentioned second timestamp of the above-mentioned second data to synchronize the above-mentioned first data with the above-mentioned second data.

[0007] In a third aspect, the present application provides a car, comprising the multi-sensor data synchronization system described above.

[0008] In a fourth aspect, the present application provides a multi-sensor data synchronization device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute a multi-sensor data synchronization method when running computer executable instructions in the memory.

[0009] In a fifth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute any of the multi-sensor data synchronization methods described above in the present application.

[0010] Beneficial effects of this application:

[0011] For multiple sensors on a car, the present application synchronizes the local timing of the car to the sensors that can be time synchronized. For sensors that cannot be time synchronized, the local timing is used to timestamp the sensor data when it is received, so that the timestamps of the data of the two sensors are marked with the same local timing, thereby achieving data alignment, and thus solving the problem that the data of sensors that do not support time synchronization cannot be synchronized. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of the multi-sensor data synchronization method of the present application;

[0013] Figure 2 This is a flowchart for obtaining local timing time for this application;

[0014] Figure 3 A flowchart of applying a first timestamp to a first sensor according to the present application;

[0015] Figure 4 A flowchart of transmitting a local timing time to a first sensor according to the present application;

[0016] Figure 5 A flowchart of applying a second time stamp to second data according to the present application;

[0017] Figure 6 This is the system topology diagram of this application;

[0018] Figure 7 Flowchart for synchronizing the local time of this application to the physical network card;

[0019] Figure 8 This is the synchronization flow chart of the whole vehicle time for this application;

[0020] Fig. 9 A flow chart of the time recording of GPS location information for this application;

[0021] Fig.10 This is a time synchronization flow chart of the laser radar of this application;

[0022] Fig.11 This is a time synchronization flow chart of the millimeter wave radar of this application;

[0023] Fig.12 A flowchart for maintaining the local time and vehicle time deviation for the time synchronization system of this application, and deriving the sensor vehicle time based on the deviation and the sensor local time;

[0024] Fig.13 In the recharge scenario of this application, the domain controller uses the two timestamps of the recharge data to restore the overall topology of the dual-time system state;

[0025] Fig.14 This is the time synchronization flow chart for this application;

[0026] Fig.15 This is a schematic diagram of a multi-sensor data synchronization device of the present application. DETAILED DESCRIPTION

[0027] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0029] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of an element or feature as shown in the figure relative to another element or feature, such as "inside", "outside", "inner side", "outer side", "below", "below", "above", "above", "front", "back", etc. Such spatial relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figure. For example, if the device in the figure undergoes a position flip or a change in posture or a change in motion state, then these directional indications will also change accordingly, for example: an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented as "above other elements or features" or "above other elements or features". Therefore, the example term "below..." can include both upper and lower orientations. The device can be oriented otherwise (rotated 90 degrees or in other directions) and the spatial relative descriptors used in the text are interpreted accordingly.

[0030] In order to solve the technical problem in the prior art that data of sensors that do not support time synchronization cannot be synchronized, the present application provides a multi-sensor data synchronization method, system, device and a car, which can achieve the effect of aligning data from different sensors.

[0031] Figure 1 Flow chart of a multi-sensor data synchronization method provided in an embodiment of the present application. Figure 1 As shown, including:

[0032] S101, when the car is powered on, starting the local timing module of the car to obtain the local timing time;

[0033] S102-1, transmitting the local timing time to a first sensor of the vehicle that supports time synchronization, so that the first sensor adds a first timestamp to the collected first data according to the local timing time;

[0034] S102-2, when second data collected by a second sensor that does not support time synchronization is acquired, a second timestamp is stamped on the second data using a local timing time;

[0035] S103: After receiving the first data, synchronize the first data with the second data using a first timestamp of the first data and a second timestamp of the second data.

[0036] The above steps S102 - 1 and S102 - 2 do not emphasize the order, and S102 - 2 may be performed first and then S102 - 1, or both may be performed simultaneously.

[0037] The above method in this embodiment can be applied to a car, which includes multiple sensors of different types. Some sensors can support data synchronization, while others do not. A sensor that supports data synchronization can input time, and the input time will be recorded by the sensor. After the sensor collects data, it can timestamp the collected data according to the recorded time. However, a sensor that does not support time synchronization cannot input or record time, and therefore cannot timestamp the collected data.

[0038] The data collected by multiple sensors on the car need to be time-aligned. For example, for data 1-3 collected by sensor 1 and data AC collected by sensor 2, whether the time point of collecting data 1 is consistent with the time point of collecting data A; or whether the time point of collecting data 1 is consistent with the time point of collecting data B; because sensor 2 does not support time synchronization, it cannot add a timestamp when collecting data, so the data of sensor 2 cannot be synchronized with the data of sensor 1.

[0039] In order to synchronize data, this embodiment proposes that a local timing module can be deployed on the car. The local timing module is a timing module maintained by the car itself, and is maintained by the crystal oscillator in the car chip. The local timing module is not interfered by the outside world of the car, and only uses its own crystal oscillator to count. The time increases by 1 second for every number of times the crystal oscillator vibrates. Therefore, the local timing module is used to count, and the time can be guaranteed to increase steadily (the time may be different from the world time, but it does not affect the use of the time for data synchronization in this embodiment).

[0040] The time measured by the local timing module is the local timing time, and the local timing time is transmitted to the first sensor supporting time synchronization, and the first sensor supporting time synchronization will record the time. After the first sensor collects the first data, it will timestamp the collected first data according to the time recorded by itself, so that the first data collected by the first sensor has the first timestamp. Since the second sensor that does not support data synchronization cannot store the local timing time, it can add a second timestamp to the collected second data when obtaining the second data collected by the second sensor.

[0041] After the first data and the second data are acquired, the first data and the second data may be aligned according to the first timestamp and the second timestamp, so that data synchronization may be performed for sensors that cannot perform time synchronization.

[0042] In this embodiment, the second sensor transmits the second data to the processor of the car immediately after collecting the second data. Therefore, although the second data is stamped with the second timestamp when the processor receives the data, the second timestamp is close to the collection time point of the second data. Alternatively, in order to further improve the accuracy of data synchronization, different sensors of the car can be used to collect preset data, and the preset data is data that has been aligned, and the time point when the data is generated can be determined between each data. Then, after the first sensor collects the data, it can check the difference between the time point of the collected data and the time point when the data is generated, and after the second sensor collects the data and sends it to the processor, the processor can obtain the difference between the time point when the data collected by the second sensor is received and the time point when the data is generated, so that the time interval from the second sensor collecting the data to the time when the data is sent to the processor can be determined. When the data of the first sensor and the second sensor are synchronized, the data of the first sensor and the second sensor can be completely synchronized through this time interval.

[0043] For multiple sensors on a car, the present application synchronizes the local timing of the car to the sensors that can be time synchronized. For sensors that cannot be time synchronized, the local timing is used to timestamp the sensor data when it is received, so that the timestamps of the data of the two sensors are marked with the same local timing, thereby achieving data alignment, and thus solving the problem that the data of sensors that do not support time synchronization cannot be synchronized.

[0044] In one embodiment, in the above step S101, if Figure 2 As shown, when the car is powered on, the local timing module of the car is started, and the local timing time is obtained including:

[0045] S201, when the car is powered on, start the local timing module to start timing;

[0046] S202, when the car is powered off, saving the timing time recorded by the local timing module;

[0047] S203, when the car is powered on again, the timing continues from the recorded timing time.

[0048] In this embodiment, the local timing module of the car can be a module powered by the battery of the car, and timing can also be performed when the car is not started. Alternatively, if it is a module powered by the battery after the car is ignited and started, it will not perform timing when the car battery is not powered, but will be in a dormant or closed state. For example, for the local timing module, timing can be performed according to the vibration frequency of the crystal oscillator when timing. If the car is parked and the power is cut off, it can enter a dormant state. Timing is not performed at this time. After the car is started again, timing continues. The timing obtained by the local timing module is the local time. At this time, although the timing time of the car, that is, the local time, is different from the world time, since the first sensor and the second sensor both use this time to collect data or receive data and stamp it, the timing time can be used for data alignment.

[0049] The timing time in this embodiment can also be reset to zero. For example, each time the car is parked and the power is cut off, the timing time can be reset to zero, and after the vehicle is started again, the timing starts from zero.

[0050] In one example, the above-mentioned car includes a real-time core and a performance core. The real-time core is used to process real-time tasks, and the performance core is used to process high-load or high-computation tasks. The real-time core and the performance core obtain local timing time through an interface.

[0051] The above-mentioned real-time core and performance core have different divisions of labor and are used to perform different tasks. The real-time core processes tasks with high time requirements, such as real-time tasks. For example, real-time tasks have high time requirements and need to provide feedback results or interactions within a predetermined time, which can be processed by the real-time core. For example, the data collected by the sensor is handed over to the real-time and data alignment. The performance core is responsible for processing tasks with large computational workloads, such as neural network learning. For example, the aligned data is handed over to the performance core for calculation and learning.

[0052] In one example, in the above step S102-1, if Figure 3 As shown, the first sensor supporting time synchronization for transmitting the local timekeeping time to the vehicle includes:

[0053] S301, determining the type of the first sensor;

[0054] S302, matching the transmission protocol according to the type;

[0055] S303: Use a matching transmission protocol to transmit the local timing time to the first sensor.

[0056] In this embodiment, for the first sensor, there may be sensors for collecting different data, and the types of the sensors are different. For different types of first sensors, the data transmission protocols used may also be different. Therefore, it is necessary to determine the data transmission protocols used by different types of sensors, and then use the corresponding data transmission protocols to synchronize the local timing time to the corresponding type of sensor.

[0057] In this embodiment, the local timing time may be expressed in different forms in different data transmission protocols. For example, in a certain protocol, the local timing time is encapsulated as 6 bytes of data, each byte includes 8 bits of data, while in another protocol, the local timing time can be converted into binary. Through different expressions, the transmission speed can be accelerated in the corresponding protocol.

[0058] In one example, in the above step S303, if Figure 4 As shown, using a matching transmission protocol to transmit the local timing time to the first sensor includes:

[0059] S401-1, when the first sensor is of a type that communicates with the real-time core, the real-time core transmits the local timing time to the first sensor using a matching transmission protocol;

[0060] S401 - 2 : When the first sensor is of a type that communicates with the core of the performance core, the performance core transmits the local timing time to the first sensor via the network card using a matching transmission protocol.

[0061] In this embodiment, due to the different types of first sensors, the first sensors also have different synchronization methods when performing time synchronization. For example, for the first sensor that communicates with the real-time core, the real-time core is responsible for transmitting the local timing time to the first sensor according to the corresponding protocol. Depending on the type of the first sensor, the protocol used may be different. In this embodiment, a correspondence table between the first sensor and the transmission protocol used can be constructed, and one type of first sensor can correspond to one protocol, and the corresponding protocol can be used to synchronize the local timing time of the first sensor of that type. And one protocol can correspond to multiple types of first sensors, that is, multiple types of first sensors can use the protocol to synchronize local timing time.

[0062] If the first sensor is the first sensor that communicates with the performance core, the performance core is responsible for using the network card to transmit the local timing time to the first sensor according to the corresponding protocol. That is, the performance core transmits the local timing time to the network card, and then the network card uses the local timing time as the network card time and transmits the network card time to the first sensor.

[0063] In one example, in the above step S102-2, if Figure 5As shown, when second data collected by a second sensor that does not support time synchronization is acquired, using the local timing time to stamp a second timestamp on the second data includes:

[0064] S501, when the performance core receives the second data, a local timing time is added to the second data at the lowest level of data access for receiving the second data, wherein the local timing time is a second timestamp.

[0065] In this embodiment, the performance core and the second sensor are different components, so the data between the two is transmitted through a protocol. The lowest level of data access mentioned above is that when the second sensor transmits the second data to the performance core, no matter what format the second data is encapsulated or modified into, it is stamped with the second timestamp when it is received by the performance core. The performance core subsequently performs operations such as decapsulation and restoration on the second data, which will not change the time of the second timestamp.

[0066] In this embodiment, since the second sensor is not a sensor that supports time synchronization, there is no way to synchronize the local timing time to the second sensor. The second sensor has no way to timestamp the data after collecting the data. Therefore, in this embodiment, when the second sensor collects the second data and transmits the second data to the performance core, the performance core can add a second timestamp to the second data when accessing the data. Although there is a slight delay between the second timestamp and the time when the second sensor collects the second data, the second sensor sends the second data to the performance core in real time when collecting the second data, and the performance core adds a second timestamp to the second data at the bottom layer of data access, so the delay can be ignored.

[0067] In one example, the above method also includes: when receiving a transmission signal of global positioning system (GPS) positioning data, recording the current local timing time; when receiving a data packet corresponding to the transmission signal, adding the current local timing time to the data packet as a timestamp; and using the local timing time in the data packet to synchronize the first data, the second data and the GPS positioning data.

[0068] In this embodiment, GPS positioning data can be used to determine global time. That is, the position of the car on the earth is determined based on the GPS positioning data, and each position on the earth corresponds to a different time (the same time for the same longitude, and different time for different longitudes), so the current global time can be determined based on the position of the car. Since the local timing module of the car is timed by a crystal oscillator alone, it may be the same as or different from the global time. When the GPS data is received in this embodiment, the GPS data can be timestamped with the local timing time, so that the global time of the GPS data can determine the time difference with the local timing time, and the time difference is the interval between the two times. According to the interval, the GPS data can be aligned with the first data and the second data collected by the first sensor and the second sensor of the car.

[0069] Figure 6 It is the system topology diagram of this embodiment. The synchronization topology of the entire system consists of the vehicle gateway, intelligent driving domain control and sensors related to the intelligent driving domain control. The cores related to the internal time synchronization of the intelligent driving domain control mainly include the performance core and the real-time core. The performance core generally runs real-time operating systems such as Linux or QNX. The internal counter on the chip maintains the vehicle time, that is, the time synchronized with the world time determined by the GPS positioning data. The master of the vehicle time is the vehicle gateway. The vehicle gateway synchronizes the world time with the performance core of the intelligent driving domain control through Ethernet. The performance core uses the generalized precision time protocol (GPTP) synchronization protocol to first synchronize the vehicle time to the physical clock of the network card 1, and then synchronize the physical clock of the network card 1 to the system clock. The time of the system clock is the vehicle time. Since this time comes from the outside, it is affected by the jump of the master. Therefore, this system mainly uses this time to record logs and analyze problems, and will not synchronize sensor data based on this time axis. In addition to the system clock maintained by the chip and the operating system, the performance core also has the physical clock inside the above-mentioned network card 1, and the real-time clock shared with the real-time core. The real-time clock is the local time obtained by the local timing module in this embodiment.

[0070] In this embodiment, the local timing time (local time) of the local timing modules of the performance core and the real-time core starts to accumulate after the car is turned on and powered on. It is not synchronized with the external world time (vehicle time) and is not affected by external factors. The performance core and the real-time core can access this local time axis at any time through the interface. At the same time, the performance core synchronizes the local time to the physical clock inside the Ethernet card of the performance core through the charging protocol (Programmable Power Supply, referred to as PPS) connected to the hardware. For sensors with time synchronization capabilities, such as lidar, forward millimeter wave, etc., according to the synchronization protocols they support, the lidar uses the picture transfer protocol (PTP) protocol to synchronize the local time to the lidar through the performance core of the intelligent driving domain control. For forward millimeter wave and other sensors that support the controller area network (CAN) protocol, the local time is synchronized through the real-time core using the CAN synchronization protocol. For GPS location information data, the performance core records the current local time when processing the PPS signal sent by the GPS module. When the recommended positioning information GPRMC data packet is received later, after parsing the GPRMC information, the local time of the intelligent driving domain when the PPS signal is triggered is added to the GPS data packet. Because the PPS speed is extremely fast, it can be considered that the local time when the PPS information is triggered is the time when the GPS location information is generated. The vehicle time is synchronized to the system clock of the performance core through the vehicle gateway. The performance core periodically synchronizes the vehicle time to the real-time core through the synchronization protocol. At this point, the local time axis and the vehicle time axis in the intelligent driving domain control are synchronized.

[0071] The functional businesses within the intelligent driving domain control all use the local time of the data when processing business logic such as sensor fusion and data alignment. The local time is completely monotonically increasing and is not affected by external factors. For modules that need to use GPS location information, such as the positioning module, because the GPS data has been added with a local timestamp through PPS during performance kernel parsing, the local time is also used when aligning GPS data with other data.

[0072] The vehicle time synchronized from the entire vehicle is used by the intelligent driving system to record logs. When recording logs, the vehicle time and local time can be recorded at the same time. By decoupling the two time axes, when problems such as vehicle time jumps occur, it will not affect the system function and it will be easier to analyze and troubleshoot the problem.

[0073] For laser radar, millimeter wave radar, inertial navigation system and other sensors that support time synchronization, but only maintain a single time, the time synchronization module maintains the local time and vehicle time deviation. The data access module uses this deviation and the local time of the sensor data to convert the vehicle time corresponding to the data. In this way, the data acquisition module can record the two times of the sensor data at the same time, which is conducive to data re-injection. Since the vehicle time may jump, the time synchronization module performs anti-jitter processing on the deviation through median filtering.

[0074] Some intelligent driving modules rely on vehicle time or local time when processing business logic. Therefore, during the data re-injection phase, the system status at the time of data collection will be reproduced as much as possible. To this end, the two times that the intelligent driving function relies on must also follow the time of the re-injection data record. That is, in the re-injection scenario, the two time axes of the domain controller need to be synchronized to the time of the corresponding re-injection data record. This embodiment synchronizes the local time and vehicle time stored in the collected data to the two time axes of the domain controller through the following steps:

[0075] 1. The IPC refill software sorts the data according to the local time of data collection;

[0076] 2. The re-injection software pushes data to the domain control system in the sorted order;

[0077] 3. After receiving the feedback data, the real-time core parses the two timestamps in the data and synchronizes them to its own real-time clock and system clock respectively;

[0078] 4. The local time in the two time axes of the performance core is obtained by directly reading the shared local timing module, and the vehicle time is synchronized by GPS data.

[0079] Figure 7 This is a flowchart for synchronizing the local time to the physical network card. After the local timing module is powered on, it runs freely without synchronization with the outside of the car. At the same time, the time synchronization signal is transmitted to the network card clock of the physical network card through the PPS protocol inside the chip. When the network card clock detects the PPS signal, it saves the network card clock at the hardware level. At the same time, the performance core saves the local clock at this moment (the local timing time of the local timing module). In this way, the absolute deviation between the network card clock and the real-time clock can be measured. The frequency deviation between the network card clock and the real-time clock can be calculated by measuring the absolute deviation multiple times, thus completing the synchronization between the real-time clock and the network card clock.

[0080] Figure 8This is a flow chart for synchronizing the vehicle time in this embodiment. The vehicle time is maintained by the time synchronization gateway on the vehicle. The gateway maintains its own time by integrating the network and GPS time, and then synchronizes the maintained time to the performance core of the intelligent driving domain control through the GPTP protocol. The performance core first synchronizes the time to the physical clock of the physical network card 1 through the GPTP protocol, and then synchronizes the physical clock of the physical network card 1 to the system clock. At the same time, the performance core needs to synchronize its own system clock to the real-time core. At this point, the performance core and the real-time core of the intelligent driving domain control have completed the synchronization with the vehicle time. Since this time comes from the outside, time jumps and other problems may occur. The intelligent driving domain only uses this time when recording logs. When troubleshooting problems, it is combined with the local time to understand the approximate time when the problem occurred and is not affected by external factors.

[0081] Fig. 9 This is a flow chart of the time recording of GPS position information in this embodiment. The inertial navigation module transmits the GPS-synchronized PPS to the performance core of the intelligent driving module through a PPS pin. The performance core of the intelligent driving module saves the local timestamp at that time when receiving the PPS. After a period of time, when the GPRMC message of the inertial navigation module reaches the performance core of the intelligent driving module, the GPS signal receiving module adds the saved PPS triggering local timestamp to the message header of the GPS information, and then publishes the data through the communication interface. When downstream modules such as the positioning module align the sensor timestamps of the GPS data, they uniformly use the local timestamps in the data for alignment.

[0082] Fig.10 This is the time synchronization flow chart of the laser radar. The intelligent driving performance core uses another network card as the master to synchronize the local time to the laser radar through the GPTP protocol. If there are multiple laser radars, they are connected through a switch. The synchronization principle is similar. For the time synchronization process of the millimeter wave radar, see Fig.11 The millimeter-wave radar is connected to the intelligent driving real-time core through the CAN / CANFD interface, and the real-time core synchronizes the local time to the millimeter-wave radar through the AUTOSAR CANTSYNC protocol.

[0083] For sensors that do not support time synchronization, a local timestamp is added at the bottom level of data access. At this point, the performance core and real-time core in the entire intelligent driving domain control complete dual time synchronization, and all sensors are also synchronized to the local time or the local time is added at the point closest to the output generation.

[0084] Because many sensors only support synchronization of one time, that is, the sensor synchronizes the local time, the domain controller is missing the entire vehicle time when receiving data, which is not conducive to data collection and problem analysis. Fig.12The time synchronization system maintains the deviation between local time and vehicle time, and derives the process of sensor vehicle time based on the deviation and sensor local time. Deviation = median filter (vehicle time - local time), vehicle time = local time + deviation. In this way, the data acquisition software can simultaneously record the local time and vehicle time when the sensor data is generated, which is helpful for subsequent problem analysis and reproduction.

[0085] Since many intelligent driving modules are sensitive to the time system, in order to ensure the consistency of the system during re-injection and data acquisition, it is necessary to restore the state of the dual time system during data acquisition. Fig.13 In the recharge scenario, the domain controller uses the two timestamps of the recharge data to restore the overall topology of the dual-time system state. Fig.14 This is a flowchart of dual time synchronization in this scenario. By synchronizing the two timestamps stored during data collection with the domain controller, the dual time system of the domain controller can be restored to the state at the time of data collection, which can better support problem analysis and reproduction.

[0086] This embodiment also provides a multi-sensor data synchronization system, including:

[0087] A local timing module is used to start when the car is powered on to obtain the local timing time;

[0088] A first sensor supporting time synchronization, configured to receive a local time and stamp a first time stamp on the collected first data according to the local time;

[0089] A second sensor that does not support time synchronization is used to collect second data;

[0090] The processor is used to stamp the second data with a second timestamp according to the local time when receiving the second data; and after receiving the first data, synchronize the first data with the second data using the first timestamp of the first data and the second timestamp of the second data.

[0091] The above system in this embodiment can be applied to a car, which includes multiple sensors of different types. Some sensors can support data synchronization, while others do not. A sensor that supports data synchronization can input time, and the input time will be recorded by the sensor. After the sensor collects data, it can timestamp the collected data according to the recorded time. A sensor that does not support time synchronization cannot input or record time, and therefore cannot timestamp the collected data.

[0092] The data collected by multiple sensors on the car needs to be time-aligned. For example, for data 1-3 collected by sensor 1 and data AC collected by sensor 2, are the time points of data 1 collected consistent with the time points of data A collected? Or are the time points of data 1 collected consistent with the time points of data B collected? Since sensor 2 does not support time synchronization, it cannot add a timestamp when collecting data, so the data of sensor 2 cannot be synchronized with the data of sensor 1.

[0093] In order to synchronize data, this embodiment proposes that a local timing module can be deployed on the car. The local timing module is a timing module maintained by the car itself, and is maintained by the crystal oscillator in the car chip. The local timing module is not interfered by the outside world of the car, and only uses its own crystal oscillator to count. The time increases by 1 second for every number of times the crystal oscillator vibrates. Therefore, the local timing module is used to count, and the time can be guaranteed to increase steadily (the time may be different from the world time, but it does not affect the use of the time for data synchronization in this embodiment).

[0094] The time measured by the local timing module is the local timing time, and the local timing time is transmitted to the first sensor supporting time synchronization, and the first sensor supporting time synchronization will record the time. After the first sensor collects the first data, it will timestamp the collected first data according to the time recorded by itself, so that the first data collected by the first sensor has the first timestamp. Since the second sensor that does not support data synchronization cannot store the local timing time, it can add a second timestamp to the collected second data when obtaining the second data collected by the second sensor.

[0095] After the first data and the second data are acquired, the first data and the second data may be aligned according to the first timestamp and the second timestamp, so that data synchronization may be performed for sensors that cannot perform time synchronization.

[0096] In this embodiment, the second sensor transmits the second data to the processor of the car immediately after collecting the second data. Therefore, although the second data is stamped with the second timestamp when the processor receives the data, the second timestamp is close to the collection time point of the second data. Alternatively, in order to further improve the accuracy of data synchronization, different sensors of the car can be used to collect preset data, and the preset data is data that has been aligned, and the time point when the data is generated can be determined between each data. Then, after the first sensor collects the data, it can check the difference between the time point of the collected data and the time point when the data is generated, and after the second sensor collects the data and sends it to the processor, the processor can obtain the difference between the time point when the data collected by the second sensor is received and the time point when the data is generated, so that the time interval from the second sensor collecting the data to the time when the data is sent to the processor can be determined. When the data of the first sensor and the second sensor are synchronized, the data of the first sensor and the second sensor can be completely synchronized through this time interval.

[0097] For multiple sensors on a car, the present application synchronizes the local timing of the car to the sensors that can be time synchronized. For sensors that cannot be time synchronized, the local timing is used to timestamp the sensor data when it is received, so that the timestamps of the data of the two sensors are marked with the same local timing, thereby achieving data alignment, and thus solving the problem that the data of sensors that do not support time synchronization cannot be synchronized.

[0098] This embodiment also provides a car, which may include the above system, that is, the car may include a first sensor that supports time synchronization, a second sensor that does not support time synchronization, a local timing module for timing, and a processor for synchronizing data.

[0099] A car can include multiple sensors, and there are different types of sensors. Some sensors can support data synchronization, while others do not. Sensors that support data synchronization can input time, and the input time will be recorded by the sensor. After collecting data, the sensor can timestamp the collected data according to the recorded time. Sensors that do not support time synchronization cannot input or record time, so they cannot timestamp the collected data.

[0100] The data collected by multiple sensors on the car needs to be time-aligned. For example, for data 1-3 collected by sensor 1 and data AC collected by sensor 2, are the time points of data 1 collected consistent with the time points of data A collected? Or are the time points of data 1 collected consistent with the time points of data B collected? Since sensor 2 does not support time synchronization, it cannot add a timestamp when collecting data, so the data of sensor 2 cannot be synchronized with the data of sensor 1.

[0101] In order to synchronize data, this embodiment proposes that a local timing module can be deployed on the car. The local timing module is a timing module maintained by the car itself, and is maintained by the crystal oscillator in the car chip. The local timing module is not interfered by the outside world of the car, and only uses its own crystal oscillator to count. The time increases by 1 second for every number of times the crystal oscillator vibrates. Therefore, the local timing module is used to count, and the time can be guaranteed to increase steadily (the time may be different from the world time, but it does not affect the use of the time for data synchronization in this embodiment).

[0102] The time measured by the local timing module is the local timing time, and the local timing time is transmitted to the first sensor supporting time synchronization, and the first sensor supporting time synchronization will record the time. After the first sensor collects the first data, it will timestamp the collected first data according to the time recorded by itself, so that the first data collected by the first sensor has the first timestamp. Since the second sensor that does not support data synchronization cannot store the local timing time, it can add a second timestamp to the collected second data when obtaining the second data collected by the second sensor.

[0103] After the first data and the second data are acquired, the first data and the second data may be aligned according to the first timestamp and the second timestamp, so that data synchronization may be performed for sensors that cannot perform time synchronization.

[0104] In this embodiment, the second sensor transmits the second data to the processor of the car immediately after collecting the second data. Therefore, although the second data is stamped with the second timestamp when the processor receives the data, the second timestamp is close to the collection time point of the second data. Alternatively, in order to further improve the accuracy of data synchronization, different sensors of the car can be used to collect preset data, and the preset data is data that has been aligned, and the time point when the data is generated can be determined between each data. Then, after the first sensor collects the data, it can check the difference between the time point of the collected data and the time point when the data is generated, and after the second sensor collects the data and sends it to the processor, the processor can obtain the difference between the time point when the data collected by the second sensor is received and the time point when the data is generated, so that the time interval from the second sensor collecting the data to the time when the data is sent to the processor can be determined. When the data of the first sensor and the second sensor are synchronized, the data of the first sensor and the second sensor can be completely synchronized through this time interval.

[0105] For multiple sensors on a car, the present application synchronizes the local timing of the car to the sensors that can be time synchronized. For sensors that cannot be time synchronized, the local timing is used to timestamp the sensor data when it is received, so that the timestamps of the data of the two sensors are marked with the same local timing, thereby achieving data alignment, and thus solving the problem that the data of sensors that do not support time synchronization cannot be synchronized.

[0106] like Fig.15 As shown, the embodiment of the present application also provides a multi-sensor data synchronization device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0107] Memory 113, used for storing computer programs;

[0108] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the multi-sensor data synchronization method provided in any one of the above method embodiments, including:

[0109] When the car is powered on, the local timing module of the car is started to obtain the local timing time;

[0110] Transmitting the local timing time to a first sensor of the vehicle supporting time synchronization, so that the first sensor stamps the collected first data with a first timestamp according to the local timing time;

[0111] When second data collected by a second sensor that does not support time synchronization is acquired, a second timestamp is stamped on the second data using the local timing time;

[0112] After receiving the first data, the first data and the second data are synchronized using a first timestamp of the first data and a second timestamp of the second data.

[0113] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-sensor data synchronization method provided in any of the aforementioned method embodiments are implemented.

[0114] The device embodiments described above are illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0116] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0117] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A multi-sensor data synchronization method, characterized in that: include: When the car is powered on, the local timing module of the car is started to obtain the local timing time; Transmitting the local timing time to a first sensor of the vehicle supporting time synchronization, so that the first sensor stamps the collected first data with a first timestamp according to the local timing time; When second data collected by a second sensor that does not support time synchronization is acquired, a second timestamp is stamped on the second data using the local timing time; After receiving the first data, the first data and the second data are synchronized using the first timestamp of the first data and the second timestamp of the second data.

2. The method according to claim 1, characterized in that When the car is powered on, starting the local timing module of the car to obtain the local timing time includes: When the car is powered on, the local timing module is started to start timing, and the local timing time is obtained; When the vehicle is powered off, saving the timing time recorded by the local timing module; When the car is powered on again, the timing will continue from the recorded timing time.

3. The method according to claim 1, characterized in that The automobile includes a real-time core and a performance core. The real-time core is used to process real-time tasks, and the performance core is used to process high-load or high-computation tasks. The real-time core and the performance core obtain the local timing time through an interface.

4. The method according to claim 3, characterized in that The first sensor supporting time synchronization for transmitting the local timekeeping time to the vehicle comprises: determining a type of the first sensor; matching a transport protocol according to the type; The local timekeeping time is transmitted to the first sensor using a matching transmission protocol.

5. The method according to claim 4, characterized in that The transmitting the local time to the first sensor using a matching transmission protocol comprises: When the first sensor is of a type that communicates with a real-time core, the real-time core transmits the local timing time to the first sensor using a matching transmission protocol; When the first sensor is of a type that communicates with a core of a performance core, the performance core transmits the local timing time to the first sensor via a network card using a matching transmission protocol.

6. The method according to claim 3, characterized in that When acquiring second data collected by a second sensor that does not support time synchronization, using the local timing time to stamp a second timestamp on the second data includes: When the performance core receives the second data, the local timing time is stamped on the second data at the lowest level of data access for receiving the second data, wherein the local timing time is the second timestamp.

7. The method according to claim 3, characterized in that The method further comprises: When receiving the transmission signal of GPS positioning data, record the current local time; When receiving a data packet corresponding to the transmission signal, adding the current local timing time as a timestamp to the data packet; The first data, the second data and the GPS positioning data are synchronized using the local timing time in the data packet.

8. A multi-sensor data synchronization system, characterized in that: include: A local timing module is used to start when the car is powered on to obtain the local timing time; a first sensor supporting time synchronization, configured to receive the local timing time and stamp a first timestamp on the collected first data according to the local timing time; A second sensor that does not support time synchronization is used to collect second data; a processor, configured to, upon receiving the second data, stamp the second data with a second timestamp according to the local timing time; After receiving the first data, the first data and the second data are synchronized using the first timestamp of the first data and the second timestamp of the second data.

9. A car, characterized in that: A system comprising the method of claim 8.

10. A multi-sensor data synchronization device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to perform the method of any one of claims 1-7 when running the computer executable instructions in the memory.

Images