A multi-sensor time synchronization fusion method, device, equipment and storage medium

Abstract

The application provides a multi-sensor time synchronization fusion method, system, device and storage medium, the method performs cyclic processing on the sensor data of each number bit in the array, time aligns all the sensor data, and completes the fusion update of the multi-sensor. The method provided by the application fully considers the time delay of each link in the process from detection to fusion processing of sensor data, globally track sensor data is grouped according to from old to new for prediction update, once prediction of a long time period is avoided, and the accuracy of prediction update is improved.

Description

Technical Field

[0001] This invention relates to a perception fusion module for intelligent driving of automobiles, and in particular to a method, apparatus, device and storage medium for multi-sensor time synchronization fusion. Background Technology

[0002] On mobile devices that need to perceive their environment, such as robots and autonomous vehicles, multiple sensors, such as cameras and millimeter-wave radar, are usually required to collect image data of the surrounding environment. Based on the image data collected by multiple cameras, data fusion perception is performed to obtain the perception results of the surrounding environment. Since it is necessary to fuse image data from multiple cameras, the accuracy of multi-sensor time synchronization becomes a key factor in ensuring the accuracy of the perception results.

[0003] In existing time synchronization technologies, clock synchronization is mainly achieved by using a pulse generator to unify the clock and correct it with each trigger to align the timestamps of each sensor. When processing each frame of sensor data, a reference sensor data time Tc is determined, and the preceding and following frame data t1 and t2 of another sensor are found before and after this time. The aligned data of the other sensor at time Tc is obtained by interpolation.

[0004] However, with current intelligent driving technology, multiple sensors connect to the domain controller in different ways, and some architectures are not suitable for a unified clock source. Furthermore, when processing the latest sensor data, it's not always possible to acquire data from the preceding and following frames for interpolation calculations. Clearly, existing time synchronization technologies have significant limitations. In addition, existing technologies suffer from low prediction accuracy due to long extrapolation times during time extrapolation prediction. Summary of the Invention

[0005] In view of the problems existing in the prior art described above, the purpose of this application is to provide a multi-sensor time synchronization fusion method, apparatus, device and storage medium to solve the above-mentioned technical problems.

[0006] To achieve the above and other related objectives, the present invention provides a multi-sensor time synchronization fusion method, comprising:

[0007] Receive current frame data, the current frame data including at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module;

[0008] The sensor data are sorted based on the first measurement delay time and placed in the corresponding positions of the receiving array in descending order;

[0009] Based on the sensor data of the first position of the received array and the corresponding first measurement delay time, the initial global track and initial global track delay time of the current frame are updated to obtain the first global track and the first global track delay time.

[0010] Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time are updated to obtain the final global track and the final global track delay time of the current frame.

[0011] In an optional embodiment of the present invention, updating the initial global track and initial global track delay time of the current frame based on the sensor data of the first bit of the received array and the corresponding first measurement delay time to obtain the first global track and the first global track delay time specifically includes:

[0012] The difference between the initial global track delay time of the current frame and the first measurement delay time of the sensor at the first position of the receiving array is calculated.

[0013] The initial global track and initial global track delay time of the current frame are updated based on the difference result to obtain the first global track and the first global track delay time.

[0014] In an optional embodiment of the present invention, when the current frame data includes three sets of sensor data, updating the first global track and the first global track delay time based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time to obtain the final global track and the final global track delay time of the current frame specifically includes:

[0015] The first global track and the first global track delay time are updated based on the sensor data of the second position of the receiving array and the corresponding first measurement delay time to obtain the second global track and the second global track delay time.

[0016] Based on the sensor data in the third position of the received array and the corresponding first measurement delay time, the second global track and the second global track delay time are updated to obtain the third global track and the third global track delay time. The third global track and the third global track delay time are then used as the final global track and the final global track delay time of the current frame.

[0017] In an optional embodiment of the present invention, updating the second global track and the second global track delay time based on the sensor data of the third bit of the received array and the corresponding first measurement delay time to obtain the third global track and the third global track delay time, and using the third global track and the third global track delay time as the final global track and the final global track delay time of the current frame, specifically includes:

[0018] The difference between the second global track delay time of the current frame and the first measurement delay time of the sensor at the third position of the receiving array is calculated.

[0019] The second global track and its delay time are updated based on the difference result to obtain the third global track and its delay time.

[0020] In an optional embodiment of the present invention, the initial global trajectory delay time of the current frame is the final global trajectory delay time of the previous frame plus a preset perception fusion module execution cycle.

[0021] In an optional embodiment of the present invention, if the previous frame of the current frame is the first frame, the method further includes:

[0022] The sensor data in the first position of the received array is processed to generate an initial global track and an initial global track delay time;

[0023] The initial global track and the initial global track delay time are used as the first global track and the first global track delay time of the first frame;

[0024] Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time of the first frame are updated to obtain the final global track and the final global track delay time of the first frame.

[0025] In an optional embodiment of the present invention, the initial global track of the current frame is the final global track of the previous frame.

[0026] In an optional embodiment of the present invention, the first measurement delay time is obtained in the following manner:

[0027] The first measurement delay time is obtained based on the initial time of the sensor measurement target, the time when the target is processed in the sensor and sent via CAN, the loss time of the sensor data transmission on the CAN line, the time when the sensor data enters the domain controller, and the time when the sensor data enters the fusion module.

[0028] To achieve the above and other objectives, the present invention also provides a multi-sensor time synchronization fusion device, comprising:

[0029] The data acquisition module is used to receive the current frame data, which includes at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module;

[0030] The sorting module is used to sort the sensor data based on the first measurement delay time and place them in the corresponding positions of the receiving array in descending order;

[0031] The first-time synchronization fusion module is used to update the initial global track and initial global track delay time of the current frame based on the sensor data of the first position of the receiving array and the corresponding first measurement delay time, so as to obtain the first global track and the first global track delay time.

[0032] The second time synchronization fusion module is used to update the first global track and the first global track delay time based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, so as to obtain the final global track and the final global track delay time of the current frame.

[0033] To achieve the above and other objectives, the present invention also provides an electronic device, characterized in that it comprises:

[0034] One or more processors;

[0035] A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the method described in any of the preceding descriptions.

[0036] To achieve the above and other objectives, the present invention also provides a computer-readable storage medium, characterized in that it stores computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the method as described in any one of the above descriptions.

[0037] Beneficial effects:

[0038] The method provided by this invention performs time alignment on all sensor data by cyclically processing the sensor data at each position in the array, thus completing the fusion update of multiple sensors. This method fully considers the time delay at each stage of the sensor data fusion process from detection to processing. Global track and sensor data are grouped from oldest to newest for prediction updates, avoiding predictions over a long period and improving the accuracy of prediction updates. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0040] Figure 1 This is a block diagram illustrating a sensor module in the field of intelligent driving, as an exemplary embodiment of this application.

[0041] Figure 2 This is a flowchart illustrating a multi-sensor time synchronization fusion method as an exemplary embodiment of this application.

[0042] Figure 3 for Figure 2 A flowchart of step S230 in a specific embodiment.

[0043] Figure 4 for Figure 2 A flowchart of step S240 in a specific embodiment.

[0044] Figure 5 This is a block diagram illustrating a multi-sensor time synchronization fusion device in an exemplary embodiment of this application.

[0045] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0046] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0048] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0050] Unless otherwise stated, the term "multiple" means two or more.

[0051] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0052] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0053] Figure 1 This is a block diagram illustrating a sensor module 100 in the field of intelligent driving, as shown in an exemplary embodiment of this application. The sensor module includes a camera sensor 101, a millimeter-wave radar 102, and an ultrasonic radar 103. The camera sensor 101 acquires real-world images of the vehicle's surroundings captured by a camera, extracts scene feature information from the real-world images, adjusts the image density, and preprocesses the images. The millimeter-wave radar 102 determines the physical environment information around the vehicle body (such as the relative distance, relative speed, angle, motion, and direction of motion between the vehicle and other objects) by emitting electromagnetic wave signals (millimeter-wave band) and receiving echo signals. Then, it performs target tracking and identification classification based on the detected object information, and further combines this with vehicle dynamic information for data fusion. The ultrasonic radar 103 uses ultrasonic waves to detect obstacles ahead, and is particularly applicable to automatic parking and short-range sensing during driving.

[0054] In the field of intelligent driving, a single sensor often only detects partial information about a target. To obtain complete information about a target, it is often necessary to fuse the target information detected by multiple sensors. However, to achieve information fusion of the same target detected by multiple sensors, it is first necessary to ensure that the information of the same target detected by each sensor is at the same moment; otherwise, such combination is meaningless.

[0055] In existing time synchronization technologies, clock synchronization is mainly achieved by using a pulse generator to unify the clock and correct it with each trigger to align the timestamps of each sensor. When processing each frame of sensor data, a reference sensor data time Tc is determined, and the preceding and following frame data t1 and t2 of another sensor are found before and after this time. The aligned data of the other sensor at time Tc is obtained by interpolation.

[0056] However, with current intelligent driving technology, multiple sensors connect to the domain controller in different ways, and some architectures are not suitable for a unified clock source. Furthermore, when processing the latest sensor data, it's not always possible to acquire data from the preceding and following frames for interpolation calculations. Clearly, existing time synchronization technologies have significant limitations. In addition, existing technologies suffer from low prediction accuracy due to long extrapolation times during time extrapolation prediction.

[0057] To address these issues, embodiments of this application propose a multi-sensor time synchronization fusion method, a scene segmentation device based on vehicle data, an electronic device, and a computer-readable storage medium. These embodiments will be described in detail below.

[0058] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating a multi-sensor time synchronization fusion method as shown in an exemplary embodiment of this application. Combined with... Figure 2 As shown, an embodiment of this disclosure provides a multi-sensor time synchronization fusion method, comprising:

[0059] Step S210: Receive current frame data, the current frame data including at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module;

[0060] Step S220: Sort the sensor data based on the first measurement delay time and place them in the corresponding positions of the receiving array in descending order;

[0061] Step S230: Update the initial global track and initial global track delay time of the current frame based on the sensor data of the first position of the receiving array and the corresponding first measurement delay time, so as to obtain the first global track and the first global track delay time.

[0062] Step S240: Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, update the first global track and the first global track delay time to obtain the final global track and the final global track delay time of the current frame.

[0063] The multi-sensor time synchronization fusion method provided in this disclosure involves receiving current frame data, which includes at least two sets of sensor data and corresponding first measurement delay times for entering the perception fusion module. The sensor data is sorted based on these first measurement delay times and placed sequentially in descending order at corresponding positions in a receiving array. Then, based on the sensor data at the first position in the receiving array and the corresponding first measurement delay time, the initial global trajectory and initial global trajectory delay time of the current frame are updated to obtain the first global trajectory and the first global trajectory delay time. Finally, based on the sensor data at subsequent positions in the receiving array and the corresponding first measurement delay times, the first global trajectory and the first global trajectory delay time are updated to obtain the final global trajectory and the final global trajectory delay time of the current frame. This method, by cyclically processing the sensor data at each position in the array, performs time alignment on all sensor data, thus completing the multi-sensor fusion update. The method provided by this invention fully considers the time delay of each stage in the sensor data detection and fusion process. The global track and sensor data are grouped from old to new for prediction and update, avoiding a long prediction period and improving the accuracy of prediction and update.

[0064] The following is in conjunction with the appendix Figure 2 Attached drawings and appendices Figure 4 Let's take a detailed look at the implementation process of each step:

[0065] First, step S210 is executed to receive the current frame data, which includes at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module.

[0066] In this embodiment, the time synchronization method is divided into the following moments: the initial moment when the sensor measures the target, the moment when the sensor measures the target and sends the data via CAN, the moment when the sensor data enters the domain controller, and the moment when the sensor data enters the fusion module.

[0067] It should be noted that the sensors described in this invention include at least a camera and a millimeter-wave radar. The calculation of the first measurement delay time will now be explained using the camera and millimeter-wave radar as examples respectively:

[0068] The forward-facing camera's FC data processing unit is not located within the domain controller; its data is transmitted via the CAN bus. The forward-facing camera detects data at the following points: exposure time FC.temp1, processing completion time FC.temp2 (when data is sent out via CAN), data transmission loss on the CAN bus FC.dt (a fixed value that can be measured experimentally), and data arrival time at the domain controller FC.temp3. The total time delay from exposure detection to data transmission to the domain controller is (FC.temp2 - FC.temp1 + FC.dt), in milliseconds. The timestamp for entering the perception fusion module is FC.temp4. At this point, the camera's first measurement delay FC.MeasureLatency = FC.temp4 - FC.temp3 + FC.latency.

[0069] The millimeter-wave radar's FR data processing unit is not located within the domain controller; its data is transmitted via the CAN bus. The millimeter-wave radar detects and outputs data at several points in time: the moment the echo is received (FR.temp1), the moment the data is processed and transmitted via CAN (FR.temp2), the data transmission loss on the CAN bus (FR.dt, a fixed value that can be measured experimentally), and the moment the data arrives at the domain controller (FR.temp3). The total time delay from receiving the echo signal to the data reaching the domain controller is FR.latency = (FR.temp2 - FR.temp1 + FR.dt), measured in milliseconds (ms). Therefore, the millimeter-wave radar's FR target measurement delay (FR.MeasureLatency) is calculated as FR.temp4 - FR.temp3 + FR.latency.

[0070] Next, step S220 is executed, in which the sensor data are sorted based on the first measurement delay time and placed in the corresponding positions of the receiving array in descending order;

[0071] The perception fusion module uses a preset data comparison algorithm to compare the first measurement delay time of each sensor entering the perception fusion module, and places them in the corresponding positions of the receiving array in descending order. That is, the sensor with the largest first measurement delay time is placed in the first position of the array (generally the 0th position of the array), and the sensor with the smallest first measurement delay time is placed in the last position of the array.

[0072] Next, step S230 is executed, which updates the initial global track and initial global track delay time of the current frame based on the sensor data of the first position of the receiving array and the corresponding first measurement delay time, so as to obtain the first global track and the first global track delay time.

[0073] First, it should be noted that in this embodiment, the first global track delay time of the current frame is equal to the final global track delay time of the previous frame plus the preset execution cycle of the perception fusion module. In a specific embodiment, the execution cycle of the perception fusion module is 25ms.

[0074] Please see Figure 3 As shown, in a specific embodiment, updating the initial global track and initial global track delay time of the current frame based on the sensor data of the first position of the received array and the corresponding first measurement delay time, to obtain the first global track and the first global track delay time, includes:

[0075] Step S310: Difference the initial global track delay time of the current frame with the first measurement delay time of the sensor at the first position of the receiving array;

[0076] For ease of description, this paper defines the initial global track delay time of the current frame as Track0.MeasureLatency, and the first measurement delay time of the sensor data in the first position of the receiving array as InputData[0].MeasureLatency.

[0077] The time difference Δt1 is obtained by subtracting the initial global track delay time of the current frame from the first measurement delay time of the sensor data in the first position of the received array, i.e.:

[0078] Δt1= Track0.MeasureLatency - InputData[0].MeasureLatency.

[0079] Step S320: Update the initial global track and initial global track delay time of the current frame according to the difference result to obtain the first global track and the first global track delay time.

[0080] For ease of description, the first global track delay time is defined as Track1.MeasureLatency.

[0081] If Δt1 > 0, then the initial global trajectory of the current frame is extrapolated by Δt1 time for prediction. The initial global trajectory of the current frame after extrapolation by Δt1 time is correlated and matched with the sensor data of the first position in the receiver array, and the initial global trajectory of the current frame is updated to obtain the first global trajectory. At the same time, the initial global trajectory delay time of the current frame is updated to the first measurement delay time of the sensor in the first position in the receiver array, and this is used as the first global trajectory delay time, that is:

[0082] Track1.MeasureLatency =InputData[0].MeasureLatency;

[0083] If Δt1 <= 0, extrapolate the sensor data of the first position in the receiver array by Δt1 time for prediction. Then, associate and match the sensor data of the first position in the receiver array after extrapolation by Δt1 time with the initial global track of the current frame, and update the initial global track of the current frame to form the first global track. At the same time, keep the initial global track delay time of the current frame unchanged and use it as the first global track delay time, that is, Track1.MeasureLatency = Track0.MeasureLatency.

[0084] Finally, step S240 is executed, which updates the first global track and the first global track delay time based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, so as to obtain the final global track and the final global track delay time of the current frame.

[0085] Please see Figure 4 As shown, when the current frame data includes three sets of sensor data, based on the sensor data in the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time are updated to obtain the final global track and the final global track delay time of the current frame, specifically including:

[0086] Step S410: Update the first global track and the first global track delay time based on the sensor data of the second position of the receiving array and the corresponding first measurement delay time to obtain the second global track and the second global track delay time.

[0087] First, for ease of description, the second global track delay time is defined as Track2.MeasureLatency, and the first measurement delay time of the sensor at the second position of the receiving array is defined as InputData[1].MeasureLatency.

[0088] The difference between the first global track delay time and the first measurement delay of the sensor at the second position of the receiving array is Δt2, that is, Δt2 = Track1.MeasureLatency - InputData[1].MeasureLatency. Δt2 is then compared with 0:

[0089] If Δt2 > 0, then the first global track is extrapolated by Δt2 time for prediction. The first global track after extrapolation by Δt2 time is then correlated and matched with the sensor data at position 2 of the receiver array, and the first global track is updated to form the second global track. At the same time, the delay time of the first global track is updated to the first measurement delay time of the sensor at position 2 of the receiver array, and this delay time is used as the second global delay time. That is:

[0090] Track2.MeasureLatency =InputData[1].MeasureLatency;

[0091] If Δt2<=0, extrapolate the sensor data at position 2 of the received array by Δt2 time for prediction. Then, associate and match the sensor data at position 2 of the received array after extrapolation by Δt2 time with the first global track and update the first global track to form the second global track. At the same time, keep the delay of the first global track unchanged and use it as the delay time of the second global track, that is, Track2.MeasureLatency = Track1.MeasureLatency.

[0092] Step S420: Update the second global track and the second global track delay time based on the sensor data of the third position of the received array and the corresponding first measurement delay time to obtain the third global track and the third global track delay time, and use the third global track and the third global track delay time as the final global track and the final global track delay time of the current frame.

[0093] First, for ease of description, the third global track delay time is defined as Track3.MeasureLatency, and the first measurement delay time of the sensor at the third position of the receiving array is defined as InputData[2].MeasureLatency.

[0094] The difference between the second global track delay time and the first measurement delay of the sensor at the third position of the receiving array is Δt3, that is, Δt3 = Track2.MeasureLatency-InputData[2].MeasureLatency. Δt3 is then compared with 0:

[0095] If Δt3 > 0, the second global track is extrapolated by Δt3 time for prediction. The extrapolated second global track is then matched with the sensor data at position 3 of the receiver array, and the second global track is updated to form the third global track. Simultaneously, the delay time of the second global track is updated to the first measurement delay time of the sensor at position 3 of the receiver array to obtain the third global delay time, which is used as the final global track delay time for the current frame. That is:

[0096] Track3.MeasureLatency =InputData[2].MeasureLatency;

[0097] If Δt3 <= 0, extrapolate the sensor data at position 3 of the received array by Δt3 time for prediction. Then, associate and match the sensor data at position 3 of the received array after extrapolation by Δt3 time with the current first global track, and update the current second global track to form the third global track. At the same time, keep the delay time of the second global track unchanged and use it as the delay time of the third global track, that is, Track3.MeasureLatency = Track2.MeasureLatency.

[0098] It is understood that in other embodiments, when the current frame data includes more than three sets of sensor data, the subsequent steps are to repeat step S420 until all sensor data have been synchronously fused. This will not be repeated here.

[0099] It should be noted that existing technologies, when calculating extrapolation times for prediction, simply compare the global trajectory time with the data from each sensor in the array that has the smallest delay, and then directly extrapolate the delay times of other sensors to the time with the smallest delay. However, a single extrapolation takes a relatively long time, during which the target's motion state may change, such as sudden acceleration, deceleration, or turning. Clearly, using existing methods introduces significant errors. In contrast, this invention calculates the extrapolation time by comparing the global trajectory time with the data from each sensor in the array one by one, and then derives the extrapolation time for each iteration. This method avoids making predictions over a long period at once, thus improving prediction accuracy.

[0100] Finally, it should be noted that in this embodiment, if the frame preceding the current frame is the first frame, the method further includes:

[0101] The sensor data in the first position of the received array is processed to generate an initial global track and an initial global track delay time;

[0102] The initial global track and the initial global track delay time are used as the first global track and the first global track delay time of the first frame;

[0103] Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time of the first frame are updated to obtain the final global track and the final global track delay time of the first frame.

[0104] Figure 5A block diagram of a multi-sensor time synchronization fusion device 500, illustrating an exemplary embodiment of this application, is shown. The multi-sensor time synchronization fusion device 500 includes a data acquisition module 501, a sorting module 502, a first time synchronization fusion module 503, and a second time synchronization fusion module 504. The data acquisition module 501 is used to receive current frame data, which includes at least two sets of sensor data and corresponding first measurement delay times for entering the perception fusion module; the sorting module 502 is used to sort the sensor data based on the first measurement delay times and place them in the corresponding positions of the receiving array in descending order; the first time synchronization fusion module 503 is used to update the initial global trajectory and the initial global trajectory delay time of the current frame based on the sensor data of the first position of the receiving array and the corresponding first measurement delay time, so as to obtain the updated first global trajectory and the first global trajectory delay time; the second time synchronization fusion module 504 is used to update the updated first global trajectory and the first global trajectory delay time based on the sensor data of the subsequent positions of the receiving array and the corresponding first measurement delay times, so as to obtain the final global trajectory and the final global trajectory delay time of the current frame.

[0105] It should be noted that the multi-sensor time synchronization fusion device 500 provided in the above embodiments and the multi-sensor time synchronization method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the multi-sensor time synchronization device 500 provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0106] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the multi-sensor time synchronization fusion method provided in the above embodiments.

[0107] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0108] like Figure 6As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from Storage Unit 608 into Random Access Memory (RAM) 603, such as performing the methods described in the above embodiments. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0109] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0110] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0111] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0114] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-sensor time synchronization fusion method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

Claims

1. A multi-sensor time synchronization fusion method, characterized in that, Applied to the perception fusion module, including: Receive current frame data, the current frame data including at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module; The sensor data are sorted based on the first measurement delay time and placed in the corresponding positions of the receiving array in descending order; Based on the sensor data of the first position of the received array and the corresponding first measurement delay time, the initial global track and initial global track delay time of the current frame are updated to obtain the first global track and the first global track delay time. Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time are updated to obtain the final global track and the final global track delay time of the current frame. Specifically, updating the initial global trajectory and initial global trajectory delay time of the current frame based on the sensor data of the first position of the received array and the corresponding first measurement delay time to obtain the first global trajectory and first global trajectory delay time includes: The difference between the initial global track delay time of the current frame and the first measurement delay time of the sensor at the first position of the receiving array is used to obtain the difference result Δt1: When the difference result Δt1 is greater than 0, the initial global trajectory of the current frame is extrapolated by Δt1 time for prediction. The initial global trajectory of the current frame after extrapolation by Δt1 time is associated and matched with the sensor data of the first position of the receiving array, and the initial global trajectory of the current frame is updated to obtain the first global trajectory. At the same time, the initial global trajectory delay time of the current frame is updated to the first measurement delay time of the sensor of the first position of the receiving array, which is used as the first global trajectory delay time. When the difference result Δt1 is less than or equal to 0, the sensor data of the first position of the receiving array is extrapolated by Δt1 time for prediction. The sensor data of the first position of the receiving array after extrapolation by Δt1 time is associated and matched with the initial global track of the current frame, and the initial global track of the current frame is updated to form the first global track. At the same time, the initial global track delay time of the current frame is kept unchanged and used as the first global track delay time.

2. The multi-sensor time synchronization fusion method according to claim 1, characterized in that, When the current frame data includes three sets of sensor data, updating the first global track and the first global track delay time based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time to obtain the final global track and the final global track delay time of the current frame specifically includes: The first global track and the first global track delay time are updated based on the sensor data in the second position of the receiving array and the corresponding first measurement delay time to obtain the second global track and the second global track delay time. Based on the sensor data in the third position of the received array and the corresponding first measurement delay time, the second global track and the second global track delay time are updated to obtain the third global track and the third global track delay time. The third global track and the third global track delay time are then used as the final global track and the final global track delay time of the current frame.

3. The multi-sensor time synchronization fusion method according to claim 2, characterized in that, The process of updating the second global track and the second global track delay time based on the sensor data of the third bit of the received array and the corresponding first measurement delay time to obtain the third global track and the third global track delay time, and using the third global track and the third global track delay time as the final global track and the final global track delay time of the current frame, specifically includes: The difference between the second global track delay time of the current frame and the first measurement delay time of the sensor at the third position of the receiving array is calculated. The second global track and its delay time are updated based on the difference result to obtain the third global track and its delay time.

4. The multi-sensor time synchronization fusion method according to claim 1, characterized in that, The initial global trajectory delay time of the current frame is the final global trajectory delay time of the previous frame plus the preset execution cycle of the perception fusion module.

5. The multi-sensor time synchronization fusion method according to claim 4, characterized in that, If the previous frame of the current frame is the first frame, the method further includes: The sensor data in the first position of the received array is processed to generate an initial global track and an initial global track delay time; The initial global track and the initial global track delay time are used as the first global track and the first global track delay time of the first frame; Based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, the first global track and the first global track delay time of the first frame are updated to obtain the final global track and the final global track delay time of the first frame.

6. The multi-sensor time synchronization fusion method according to claim 1, characterized in that, The initial global track of the current frame is the final global track of the previous frame.

7. The multi-sensor time synchronization fusion method according to claim 1, characterized in that, The first measurement delay time is obtained in the following manner: The first measurement delay time is obtained based on the initial time of the sensor measurement target, the time when the target is processed in the sensor and sent via CAN, the loss time of the sensor data transmission on the CAN line, the time when the sensor data enters the domain controller, and the time when the sensor data enters the fusion module.

8. A multi-sensor time synchronization fusion device, characterized in that, include: The data acquisition module is used to receive the current frame data, which includes at least two sets of sensor data and the corresponding first measurement delay time for entering the perception fusion module; The sorting module is used to sort the sensor data based on the first measurement delay time and place them in the corresponding positions of the receiving array in descending order; The first time synchronization fusion module is used to update the initial global track of the current frame and the initial global track delay time of the current frame based on the sensor data of the first position of the receiving array and the corresponding first measurement delay time, so as to obtain the updated first global track and the first global track delay time. The second time synchronization fusion module is used to update the updated first global track and first global track delay time based on the sensor data of the subsequent positions of the received array and the corresponding first measurement delay time, so as to obtain the final global track and final global track delay time of the current frame. Specifically, updating the initial global trajectory and initial global trajectory delay time of the current frame based on the sensor data of the first position of the received array and the corresponding first measurement delay time to obtain the first global trajectory and first global trajectory delay time includes: The difference between the initial global track delay time of the current frame and the first measurement delay time of the sensor at the first position of the receiving array is used to obtain the difference result Δt1: When the difference result Δt1 is greater than 0, the initial global trajectory of the current frame is extrapolated by Δt1 time for prediction. The initial global trajectory of the current frame after extrapolation by Δt1 time is associated and matched with the sensor data of the first position of the receiving array, and the initial global trajectory of the current frame is updated to obtain the first global trajectory. At the same time, the initial global trajectory delay time of the current frame is updated to the first measurement delay time of the sensor of the first position of the receiving array, which is used as the first global trajectory delay time. When the difference result Δt1 is less than or equal to 0, the sensor data of the first position of the receiving array is extrapolated by Δt1 time for prediction. The sensor data of the first position of the receiving array after extrapolation by Δt1 time is associated and matched with the initial global track of the current frame, and the initial global track of the current frame is updated to form the first global track. At the same time, the initial global track delay time of the current frame is kept unchanged and used as the first global track delay time.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of any one of claims 1 to 7.

Images