Multi-sensor time synchronization method and system based on time axis and storage medium

Abstract

The invention is applicable to the technical field of sensor time synchronization, and provides a time axis-based multi-sensor time synchronization method and system and a storage medium, and is characterized in that the method comprises the following steps: if a current slave sensor does not receive data of a master sensor, adopting a current moment as a matching timestamp; traversing the plurality of sensors, and determining two frames of scanning data with the nearest matching timestamp of each sensor; and synchronizing each sensor according to the two frames of scanning data. When the data of the main sensor is not received, the current moment is adopted as the matching timestamp, it is guaranteed that when the main sensor cannot work normally, all the sensors can still be synchronized, the anti-risk capacity of the synchronization system can be improved, all the sensors are synchronized according to the two frames of scanning data with the latest matching timestamp, and the accuracy of the synchronization system is improved. And the accuracy of the synchronous sensor is improved.

Description

Technical Field

[0001] This application belongs to the technical field of sensor time synchronization, and particularly relates to a multi-sensor time synchronization method, system and storage medium based on a time axis. Background Technique

[0002] With the development of technology, perception systems are required in more and more occasions, and multiple sensors need to work together in a perception system. For example, as an important component in vehicle-road collaboration, the roadside perception system has inconsistent time for the data received by each sensor in the roadside unit, resulting in the inability to use sensor data, which has an important impact on the algorithm module for data processing. Therefore, time synchronization between multiple sensors has become an important issue of concern in the industry.

[0003] Time synchronization is to provide a relatively more unified time scale for a distributed system environment, that is, to unify each sensor under a time coordinate system. For example, a unified master sensor provides a reference time for each sensor, and each sensor adds timestamp information to the independently collected data according to its respective calibrated time to achieve timestamp synchronization of all sensors.

[0004] The common soft synchronization method in the prior art is the nearest timestamp matching. Taking a camera and a lidar as an example, a common synchronization method is to calculate the difference between the timestamps of their data and select the set of data with the smallest difference as the synchronization result. Directly using the current moments of the data of the two as timestamps to calculate the difference in the above method is not conducive to improving the accuracy of multi-sensor time synchronization. Another common synchronization method is to use a certain sensor as the master sensor. When a new frame of data arrives at this sensor, it goes to the data queues of other sensors to calculate the difference for matching. In the above method, only the timestamp of the master sensor is used as the matching timestamp. In this case, when the master sensor fails to work properly, the entire system cannot run further, and the anti-risk ability of the system is insufficient. Summary of the Invention

[0005] The embodiments of this application provide a multi-sensor time synchronization method, system and storage medium based on a time axis, which can ensure that each sensor can still be synchronized when the master sensor fails to work properly, is conducive to improving the anti-risk ability of the synchronization system, and is conducive to improving the accuracy of the synchronized sensors.

[0006] The first aspect of the embodiments of this application provides a multi-sensor time synchronization method based on a time axis, and the method includes:

[0007] If the current slave sensor does not receive the data of the master sensor, use the current moment as the matching timestamp;

[0008] Traverse the multiple sensors to determine the two frames of scan data with the closest matching timestamps for each of the sensors;

[0009] Synchronize each of the sensors based on the two frames of scan data.

[0010] A second aspect of the embodiments of the present application provides a multi-sensor time synchronization system based on a time axis. The system includes:

[0011] A matching timestamp determination module, configured to use the current moment as the matching timestamp if data from the master sensor is not received by the current slave sensor;

[0012] A scan data determination module, configured to traverse the multiple sensors to determine the two frames of scan data with the closest matching timestamps for each of the sensors;

[0013] A synchronization module, configured to synchronize each of the sensors based on the two frames of scan data.

[0014] A third aspect of the embodiments of the present application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for multi-sensor time synchronization based on a time axis described in the first aspect above is implemented.

[0015] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the operation method for multi-sensor time synchronization based on a time axis described in the first aspect above is implemented.

[0016] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device is enabled to execute the method for multi-sensor time synchronization based on a time axis described in the first aspect above.

[0017] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: The present application proposes a method for multi-sensor time synchronization based on a time axis. When data from the master sensor is not received, the current moment is used as the matching timestamp, ensuring that each sensor can still be synchronized when the master sensor fails to work properly, which is beneficial to improving the anti-risk ability of the synchronization system. By synchronizing each sensor according to the two frames of scan data with the closest matching timestamps, it is beneficial to improve the accuracy of the synchronized sensors. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic flowchart of a multi-sensor time synchronization method based on a timeline provided by an embodiment of the present application;

[0020] Figure 2 is provided by an embodiment of the present application Figure 1 The specific flowchart of step 300 in

[0021] Figure 3 is provided by an embodiment of the present application Figure 2 The specific flowchart of step 303 in

[0022] Figure 4 It is a schematic flowchart of a time synchronization method for a radar and a camera provided by an embodiment of the present application;

[0023] Figure 5 is a schematic structural diagram of a multi-sensor time synchronization system based on a timeline provided by an embodiment of the present application;

[0024] Figure 6 is a structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0025] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0026] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0027] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] In addition, in the description of the specification and the appended claims of this application, the terms "first", "second", "third", etc. are only used for differential description and should not be construed as indicating or implying relative importance.

[0029] The reference to "one embodiment" or "some embodiments" in the description of this application means that in one or more embodiments of this application, the specific features, structures, or characteristics described in connection with that embodiment are included. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0030] It should be understood that the magnitudes of the sequence numbers of the steps in this embodiment do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not impose any limitation on the implementation process of the embodiments of this application.

[0031] With the development of technology, in more and more scenarios, a perception system is required, and multiple sensors need to work together in the perception system. For example, as an important component in vehicle-road cooperation, the roadside perception system has sensors that receive data at different times, resulting in the inability to use the sensor data, which has an important impact on the algorithm module for data processing. Therefore, time synchronization between multiple sensors has become an important issue of concern in the industry.

[0032] Time synchronization is to provide a relatively more unified time scale for a distributed system environment, that is, to unify each sensor under a time coordinate system. For example, a unified master sensor provides a reference time for each sensor, and each sensor adds timestamp information to the data independently collected according to its own calibrated time, so as to achieve timestamp synchronization of all sensors.

[0033] The prior art mainly involves three parts: time synchronization, hardware synchronization, and software synchronization. Among them, soft synchronization mainly takes into account the camera imaging and transmission processes. Millimeter-wave radars and lidars do not support hardware triggering because millimeter waves cannot receive external signal triggers, and the acquisition times of the two sensors are inconsistent. A common soft synchronization method is the nearest timestamp matching. Taking the camera and lidar as an example, a common synchronization method is to calculate the difference between the timestamps of their data and select the set of data with the smallest difference as the synchronization result. Using the current time of the data of the two sensors directly as the timestamp to calculate the difference in the above method is not conducive to improving the accuracy of multi-sensor time synchronization. Another common synchronization method is to use a certain sensor as the master sensor. When a new frame of data arrives at this sensor, it goes to the data queues of other sensors to calculate the difference for matching. Using only the timestamp of the master sensor as the matching timestamp in the above method, when the master sensor fails to work properly, the entire system will not be able to run further, and the anti-risk ability of the system is insufficient.

[0034] Compared with the traditional solution that can only perform timestamp matching through the timestamp of the master sensor, the biggest difference of this application is that when the data of the master sensor is not received, the current time is used as the matching timestamp, ensuring that each sensor can still be synchronized when the master sensor fails to work properly, which is conducive to improving the anti-risk ability of the synchronization system. On the other hand, in the prior art, the difference is directly used to calculate the difference with the matching timestamp, and the set with the smallest difference is selected as the synchronization result, ignoring the influence of the difference. This application infers the data of the matching timestamp based on the two frames of scan data closest to the matching timestamp and synchronizes each sensor, which is conducive to improving the accuracy of the synchronized sensors.

[0035] To illustrate the technical solution of this application, specific embodiments will be used below for illustration.

[0036] Referring to Figure 1 , a schematic flowchart of a multi-sensor time synchronization method based on a time axis provided by an embodiment of this application is shown. As Figure 1 shown, the multi-sensor time synchronization method may include the following steps:

[0037] Step 100, if the current slave sensor does not receive the data of the master sensor, use the current time as the matching timestamp.

[0038] In the embodiments of the present application, the sensor can be a lidar or a camera. Time synchronization is to provide a relatively more unified time scale for the distributed system environment, that is, to unify each sensor into a time coordinate system. For example, a unified master sensor provides a reference time for each sensor, and each sensor adds timestamp information to the independently collected data according to its own calibrated time, so as to achieve timestamp synchronization of all sensors. However, in actual applications, there is a situation where the master sensor fails, so that the data of the master sensor cannot be received. In the embodiments of the present application, if the current slave sensor does not receive the data of the master sensor, the sensor uses the current moment as the matching timestamp for time synchronization.

[0039] In another application scenario, the method further includes: if the current slave sensor receives the data of the master sensor, using the timestamp of the master sensor as the matching timestamp.

[0040] In the embodiments of the present application, the sensor can be a lidar, a camera, etc., which are not specifically limited here. For example, when the lidar and the camera are synchronized, since the lidar data is relatively accurate and the working frequency is low, the lidar is generally selected as the master sensor. When the data of the lidar is received, the timestamp of the lidar is used as the matching timestamp.

[0041] Step 200, traverse the multiple sensors to determine the two frames of scan data with the closest matching timestamp for each sensor.

[0042] In the embodiments of the present application, after the matching timestamp is determined through step 100, all the sensors that need to be synchronized are traversed, and the two frames of scan data closest to the matching timestamp for each sensor need to be obtained to determine the scan data of each sensor at the matching timestamp. Optical flow is a vector field that describes the displacement of pixels in an image over time. Since the pixel movement between two frames of the optical flow field is continuous, by calculating the optical flow vector field between two frames of images, the pixel positions at intermediate time points can be inferred, and the pixel values can be interpolated based on these positions. By inferring the scan data at the intermediate time point, that is, the matching timestamp, the time of each sensor is synchronized.

[0043] Specifically, the traversing the multiple sensors to determine the two frames of scan data with the closest matching timestamp for each sensor includes:

[0044] Obtain the scan image of the frame closest to the matching timestamp before the matching timestamp for each sensor and the scan image of the frame closest to the matching timestamp after the matching timestamp.

[0045] In an embodiment of the present application, the scanned images of each sensor at the moment of the matching timestamp are inferred by obtaining the nearest scanned image before the matching timestamp and the nearest scanned image after the matching timestamp of each sensor. Optical flow interpolation is an interpolation method based on motion estimation. Assuming that the pixel movement between two frames of images is continuous, by calculating the optical flow vector field between the two frames of images, it can be inferred that the pixel positions at the intermediate time point, i.e., at the matching timestamp, and the pixel values are interpolated based on these positions to synchronize each sensor.

[0046] Step 300: Synchronize each of the sensors according to the two frames of scanned data.

[0047] In an embodiment of the present application, sensor synchronization is performed according to two frames of scanned data, that is, according to the nearest scanned image before the matching timestamp and the nearest scanned image after the matching timestamp. The scanned image includes the pixel point positions and pixel values of all pixels in the scanned image.

[0048] Specifically, referring to Figure 2 , which shows the specific process schematic diagram of step 300 provided by the embodiment of the present application. As Figure 1 shown, step 300 specifically includes the following steps: Figure 2 As shown, step 300 specifically includes the following steps:

[0049] S301: Obtain the first coordinate and the first pixel value corresponding to any pixel on the first frame of scanned image.

[0050] In an embodiment of the present application, the first frame of scanned image is the nearest scanned image before the matching timestamp. Since the pixel movement between two frames of the optical flow field is continuous, by calculating the optical flow vector field between the two frames of images, for any pixel on the scanned image, the pixel position and pixel value at the intermediate time, i.e., at the matching timestamp, can be inferred in the manner of the present application according to the pixel positions and pixel values of the two frames of scanned images. For example, an image coordinate system is established on the scanned image, and the first coordinate (x, y) of any pixel of the image coordinate system is obtained, and the first pixel value I1(x, y) corresponding to the pixel at the first coordinate is obtained.

[0051] S302: Obtain the second coordinate and the second pixel value corresponding to the any pixel on the second frame of scanned image.

[0052] In the embodiment of the present application, the second frame of scanned image is the scanned image of the nearest frame after the matching timestamp. Obtain the second coordinate and the second pixel value corresponding to the same arbitrary pixel point in the second frame of scanned image in S301. For example, obtain the second coordinate (x + Δx, y + Δy) of the pixel of the same arbitrary point for S301, and the second pixel value I2(x + Δx, y + Δy) corresponding to the pixel at the second coordinate. Since the pixel points in the scanned image change continuously in the time axis, the second coordinate is the same pixel point plus the pixel displacement based on the first coordinate for the first coordinate. The change amount of the second coordinate value is the same as that of the coordinate for the first coordinate value, and the pixel change amount also needs to be added to the first pixel value.

[0053] S303. Calculate and obtain the optical flow vector field between the first frame of scanned image and the second frame of scanned image according to the first pixel value, the second pixel value, and the optical flow equation.

[0054] In the embodiment of the present application, the camera data of the matching timestamp is fitted by using the optical flow interpolation method. By calculating the optical flow vector field between two frames of images, the pixel positions at intermediate time points can be inferred, and the pixel values are interpolated according to these positions. The optical flow equation is required to calculate the optical flow vector field.

[0055] An optical flow equation is established through the first pixel value and the second pixel value. Assume that the pixel movement between two frames of images is continuous. Then, in the definition of the optical flow field, the optical flow vector field is constant everywhere. That is to say, the first pixel value and the second pixel value in the optical flow vector field should be equal. Therefore, an optical flow equation can be established to calculate the optical flow vector field.

[0056] Specifically, Figure 3 shows the specific flow schematic diagram of Figure 2 step 303 provided by the embodiment of the present application, as Figure 3 shown. Step 303 specifically includes the following steps:

[0057] S3031. Obtain the first optical flow equation according to the first pixel value and the second pixel value. The first optical flow equation is that the first pixel value is equal to the second pixel value, and the second pixel value is the first pixel value plus the pixel displacement amount.

[0058] In the embodiment of the present application, since the optical flow vector field is constant everywhere, that is to say, the first pixel value and the second pixel value in the optical flow vector field should be equal. Therefore, the first optical flow equation is established through formula (1).

[0059] I1(x, y) = I2(x + Δx, y + Δy) (1)

[0060] Among them, I1(x, y) is the first pixel value, I2(x + Δx, y + Δy) is the second pixel value, and Δx and Δy are the displacement change amounts of the pixel on the time axis.

[0061] S3032, perform a Taylor expansion on the first optical flow equation to obtain a second optical flow equation.

[0062] In the embodiment of the present application, by performing arithmetic processing on the first optical flow equation, performing a Taylor expansion on the first optical flow equation, and retaining the first order to obtain the second optical flow equation, as shown in formula (2).

[0063]

[0064] S3032, obtain a third optical flow equation according to the second optical flow equation, the first pixel value, and the second pixel value. The third optical flow equation is that the first pixel value is equal to the second pixel optical flow vector value, and the second pixel optical flow vector value is the first pixel value plus the optical flow vector field component. The optical flow vector field component includes the horizontal component and the vertical component of the optical flow vector field.

[0065] In the embodiment of the present application, let the horizontal component of the optical flow vector field be u and the vertical component be v. According to the brightness constancy hypothesis, that is, the optical flow vector field is constant everywhere. Therefore, for any pixel, the optical flow vectors of the first pixel value and the second pixel value in the optical flow vector field are also constant. Therefore, the change amounts of the horizontal component and the vertical component in the optical flow vector field of the pixel in the second coordinate should be equal to the pixel value in the first coordinate. Substitute u and v into the formula in S3031 according to the definition of the optical flow vector field component to obtain the third optical flow equation, as shown in formula (3).

[0066] I1(x, y) = I2(x + u, y + v) (3)

[0067] S3033, perform a Taylor expansion on the third optical flow equation and calculate to obtain the optical flow vector field component.

[0068] In the embodiment of the present application, by performing a Taylor expansion on formula (3) and retaining the first-order term, as shown in formula (4), solve the optical flow equation to obtain the horizontal component u of the optical flow vector field and the vertical component v of the optical flow vector field, so as to obtain the optical flow vector field (u, v) according to the horizontal component and the vertical component.

[0069]

[0070] S304, calculate and obtain the scan data of the matching timestamp according to the optical flow vector field.

[0071] Specifically, interpolation calculations are performed based on the optical flow vector field components to obtain the coordinates and pixel values of the pixel at the matching timestamp.

[0072] In the embodiment of the present application, the optical flow vector field calculated through step S303 is used to perform difference calculations based on the horizontal and vertical components in the optical flow vector field. In a possible embodiment, an interpolation method, such as bilinear interpolation, can be applied to infer the pixel positions and pixel values at the matching timestamp points.

[0073] S305, synchronize each of the sensors according to the two frames of scan data.

[0074] The pixel positions and pixel values at the matching timestamp points, that is, the scan data, are used to synchronize each sensor. In the embodiment of the present application, by utilizing the definition that the optical flow vector field is equal everywhere, the two nearest frames of scan images before and after the matching timestamp point are selected. The pixel point positions and pixel values in the two frames of scan images are used to participate in the optical flow equation calculation to obtain the optical flow vector field between the two frames of images at the matching timestamp. Since the pixel changes in the optical flow vector field are continuous, the scan data at the matching timestamp can be inferred based on this optical flow vector field, and the timestamps of each sensor are synchronized through the scan data at the matching timestamp.

[0075] When the main sensor data is not received, the current moment is used as the matching timestamp to ensure that each sensor can still be synchronized when the main sensor fails to work properly, which is beneficial to improving the anti-risk ability of the synchronization system. By inferring the data at the matching timestamp based on the two frames of scan data closest to the matching timestamp and synchronizing each sensor, it is beneficial to improve the accuracy of the synchronized sensors.

[0076] The multi-sensor time synchronization method based on the time axis further includes:

[0077] According to the current moment, the previous matching timestamp, and a preset synchronization time interval, determine whether the current moment is used as the next matching timestamp to synchronize each of the sensors.

[0078] In the embodiment of the present application, it can also be determined whether the process of synchronizing the sensors is started at the current moment. The determination process is as follows: Calculate the difference between the current moment and the previous matching timestamp. If the difference is greater than or equal to the preset synchronization time interval, it is determined that the current moment is used as the next matching timestamp to synchronize each sensor. By the above method, the continuity of the synchronized sensors is ensured. By continuously determining whether a new round of sensor synchronization cycle needs to be started at the current timestamp, it is beneficial to automatically start the automatic synchronization of the sensors.

[0079] Refer to Figure 4, which shows a schematic flowchart of a time synchronization method for a radar and a camera according to an embodiment of the present application, as Figure 4 shown.

[0080] First, connect the lidar, camera, and millimeter-wave radar. Start the time matching of the data, and turn on the first judgment condition. Calculate the difference between the current moment and the previous matching timestamp. If the difference is greater than or equal to the preset synchronization time interval, then judge the current moment as the next matching timestamp to synchronize each sensor. Then continue with the second judgment condition to determine whether the current main sensor can work normally. If it can work normally, the matching timestamp t1 is the timestamp of the main sensor. If it cannot work normally, the matching timestamp t1 is the current moment timestamp. After determining the matching timestamp, traverse all radars to obtain the two closest frame scan data of the radar at the matching timestamp, one frame is the scan data > t1, and one frame is the scan data < t1. According to the two-frame scan data, calculate the optical flow vector field, and then obtain the data at the matching timestamp according to the linear interpolation method to synchronize each radar sensor. Similarly, traverse each camera sensor to obtain the two closest frame scan data of the camera at the matching timestamp, one frame is the scan data > t1, and one frame is the scan data < t1. According to the two-frame scan data, calculate the optical flow vector field, and then obtain the data at the matching timestamp according to the linear interpolation method to synchronize each camera sensor.

[0081] By obtaining the pixel positions and pixel values, i.e., scan data, at the matching timestamp points through the above steps to synchronize each radar sensor and camera sensor. In the embodiment of the present application, by using the fact that the optical flow vector field is everywhere equal, the two closest frame scan images before and after the matching timestamp point are determined. According to the pixel point positions and pixel values in the two-frame scan images, participate in the calculation of the optical flow equation to obtain the optical flow vector field between the two-frame images at the matching timestamp. Since the pixel changes in the optical flow vector field are continuous, the scan data at the matching timestamp can be inferred according to this optical flow vector field, and the timestamps of each sensor are synchronized through the scan data at the matching timestamp.

[0082] When the main sensor data is not received, the current moment is used as the matching timestamp, which ensures that when the main sensor cannot work normally, each sensor can still be synchronized, which is beneficial to improving the anti-risk ability of the synchronization system. By inferring the data at the matching timestamp according to the two closest frame scan data at the matching timestamp and synchronizing each sensor, it is beneficial to improve the accuracy of the synchronized sensors.

[0083] See Figure 5 , which shows a schematic structural diagram of a multi-sensor time synchronization system based on a time axis according to an embodiment of the present application. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0084] The multi-sensor time synchronization system 400 based on the time axis may specifically include the following modules:

[0085] The matching timestamp determination module 410 is configured to use the current moment as the matching timestamp if the current slave sensor does not receive data from the master sensor;

[0086] If the current slave sensor receives data from the master sensor, use the timestamp of the master sensor as the matching timestamp.

[0087] The scanned data determination module 420 is configured to traverse the multiple sensors and determine two frames of scanned data with the closest matching timestamps for each of the sensors;

[0088] The synchronization module 430 is configured to synchronize each of the sensors according to the two frames of scanned data.

[0089] In an embodiment of the present application, the scanned data determination module 420 is specifically configured to: obtain the scanned image of the closest frame before the matching timestamp and the scanned image of the closest frame after the matching timestamp for each of the sensors.

[0090] In an embodiment of the present application, the synchronization module 430 may include the following sub-modules:

[0091] The first coordinate and pixel value acquisition module is configured to obtain the first coordinate and the first pixel value corresponding to any pixel on the first frame of scanned image;

[0092] The second coordinate and pixel value acquisition module is configured to obtain the second coordinate and the second pixel value corresponding to the any pixel on the second frame of scanned image;

[0093] The optical flow vector field calculation module is configured to calculate and obtain the optical flow vector field between the first frame of scanned image and the second frame of scanned image according to the first pixel value, the second pixel value, and the optical flow equation;

[0094] The scanned data calculation module is configured to perform interpolation calculation according to the optical flow vector field components to obtain the coordinate and pixel value of the pixel at the matching timestamp;

[0095] The synchronized sensor data module is configured to synchronize each of the sensors according to the two frames of scanned data.

[0096] The optical flow vector calculation module may include the following units:

[0097] The first optical flow equation unit is configured to obtain a first optical flow equation according to the first pixel value and the second pixel value, and the first optical flow equation is that the first pixel value is equal to the second pixel value, and the second pixel value is the first pixel value plus the pixel displacement amount;

[0098] A second optical flow equation unit, configured to perform Taylor expansion on the first optical flow equation to obtain a second optical flow equation;

[0099] A third optical flow equation unit, configured to obtain a third optical flow equation according to the second optical flow equation, the first pixel value, and the second pixel value, where the third optical flow equation is that the first pixel value is equal to the second pixel optical flow vector value, and the second pixel optical flow vector value is the first pixel value plus the optical flow vector field component, and the optical flow vector field component includes a horizontal component of the optical flow vector field and a vertical component of the optical flow vector field;

[0100] An optical flow vector field component calculation unit, configured to perform Taylor expansion on the third optical flow equation and calculate to obtain the optical flow vector field component.

[0101] In the embodiment of the present application, the multi-sensor time synchronization system 400 based on the time axis may further include the following modules:

[0102] A judgment module, configured to judge whether the current moment is used as the next matching timestamp to synchronize each sensor according to the current moment, the previous matching timestamp, and a preset synchronization time interval.

[0103] The multi-sensor time synchronization system based on the time axis provided by the embodiment of the present application can be applied in the foregoing method embodiment. For details, please refer to the description of the foregoing method embodiment, which will not be elaborated here.

[0104] Figure 6 It is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As Figure 6 shown, the terminal device 700 of this embodiment includes: at least one processor 710 ( Figure 6 only one is shown in the figure), a memory 720, and a computer program 721 stored in the memory 720 and operable on the at least one processor 710. When the processor 710 executes the computer program 721, the steps in the foregoing method embodiment of the multi-sensor time synchronization method based on the time axis are implemented.

[0105] The terminal device 700 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art can understand that Figure 6 merely an example of the terminal device 700, which does not constitute a limitation on the terminal device 700, and may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may further include input / output devices, network access devices, etc.

[0106] The so-called processor 710 may be a Central Processing Unit (CPU), and this processor 710 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0107] In some embodiments, the memory 720 may be an internal storage unit of the terminal device 700, such as the hard disk or memory of the terminal device 700. In other embodiments, the memory 720 may also be an external storage device of the terminal device 700, such as a plug-in hard disk equipped on the terminal device 700, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 720 may also include both the internal storage unit of the terminal device 700 and the external storage device. The memory 720 is used to store an operating system, application programs, a Boot Loader, data, and other programs, such as the program code of the computer program, etc. The memory 720 may also be used to temporarily store data that has been output or will be output.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0109] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0110] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0111] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, the functional units in the various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0114] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0115] To implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program product. When the computer program product runs on a terminal device, the terminal device can execute the steps in the above-described method embodiments when executed.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A multi-sensor time synchronization method based on a timeline, characterized in that, The method includes: If the current slave sensor does not receive data from the master sensor, use the current moment as the matching timestamp; Traverse the multiple sensors to determine the two frames of scan data with the closest matching timestamps for each sensor; Synchronize each sensor according to the two frames of scan data.

2. The multi-sensor time synchronization method according to claim 1, characterized in that The method further includes: If the current slave sensor receives data from the master sensor, use the timestamp of the master sensor as the matching timestamp.

3. The multi-sensor time synchronization method according to claim 1, wherein The traversing the multiple sensors to determine the two frames of scan data with the closest matching timestamps for each sensor includes: Obtain the scan image of the frame closest to the matching timestamp before the matching timestamp of each sensor and the scan image of the frame closest to the matching timestamp after the matching timestamp.

4. The multi-sensor time synchronization method according to claim 1, characterized in that The scan data includes the positions and pixel values of each pixel on the scan image. Synchronizing each sensor according to the two frames of scan data includes: Obtain the first coordinate and the first pixel value corresponding to any pixel on the first frame of scan image; Obtain the second coordinate and the second pixel value corresponding to the any pixel on the second frame of scan image; Calculate the optical flow vector field between the first frame of scan image and the second frame of scan image according to the first pixel value, the second pixel value and the optical flow equation; Calculate the scan data of the matching timestamp according to the optical flow vector field; Synchronize each sensor according to the two frames of scan data.

5. The multi-sensor time synchronization method according to claim 4, characterized in that, The calculating the optical flow vector field between the first frame of scan image and the second frame of scan image according to the first pixel value, the second pixel value and the optical flow equation includes: Obtain a first optical flow equation according to the first pixel value and the second pixel value. The first optical flow equation is that the first pixel value is equal to the second pixel value, and the second pixel value is the first pixel value plus the pixel displacement amount; Perform Taylor expansion on the first optical flow equation to obtain a second optical flow equation; Obtain a third optical flow equation according to the second optical flow equation, the first pixel value and the second pixel value. The third optical flow equation is that the first pixel value is equal to the second pixel optical flow vector value, and the second pixel optical flow vector value is the first pixel value plus the optical flow vector field component. The optical flow vector field component includes the horizontal component of the optical flow vector field and the vertical component of the optical flow vector field; Perform Taylor expansion on the third optical flow equation and calculate to obtain the optical flow vector field component.

6. The multi-sensor time synchronization method according to claim 4, wherein The calculating the scan data of the matching timestamp according to the optical flow vector field includes: Perform interpolation calculation according to the optical flow vector field component to obtain the coordinates and pixel values of the pixel at the matching timestamp.

7. The multi-sensor time synchronization method according to claim 1, characterized in that, The method further includes: Judge whether the current moment is used as the next matching timestamp to synchronize each sensor according to the current moment, the previous matching timestamp and the preset synchronization time interval.

8. A multi-sensor time synchronization system based on a timeline, characterized in that, It includes: A matching timestamp determination module, configured to use the current moment as the matching timestamp if the current slave sensor does not receive data from the master sensor; A scan data determination module, configured to traverse the multiple sensors and determine two frames of scan data with the closest matching timestamps for each of the sensors; A synchronization module, configured to synchronize each of the sensors according to the two frames of scan data.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images