Time alignment method and device based on multi-modal sensor, equipment and medium

Abstract

The application discloses a time alignment method and device based on a multi-modal sensor, equipment and a medium. The method comprises the following steps: generating a target timestamp sequence based on a collection timestamp of a target sensor during a data collection period; generating an original sensor collection data sequence according to sensor collection data collected by the candidate sensor during the data collection period; performing a vectorized timestamp neighbor alignment processing on the original sensor collection data sequence based on the target timestamp sequence to generate a candidate sensor collection data sequence; and performing an interpolation processing and a dynamic frame rate adjustment on the candidate sensor collection data sequence to generate a target sensor collection data sequence after timestamp alignment. Through the above technical solution, the timestamp alignment accuracy of the target sensor collection data sequence corresponding to different sensors is improved.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, specifically to a time alignment method, apparatus, device, and medium based on a multimodal sensor. Background Technology

[0002] In the ROS (Robot Operating System) ecosystem, sensor-collected data is published as messages via topics, with each message accompanied by a ROS timestamp.

[0003] Currently, in the ROS ecosystem, multi-sensor time synchronization mainly uses the time synchronizer provided by ROS to pair messages from different topics by timestamp by setting a time tolerance window, or publishes a simulated clock to align the time of each sensor node according to the topic, relying on hardware clock synchronization protocols to maintain clock consistency among distributed nodes. However, existing multi-sensor time synchronization methods cannot guarantee the time alignment accuracy between different types of sensors. Summary of the Invention

[0004] This application provides a time alignment method, apparatus, device, and storage medium based on multimodal sensors to improve the timestamp alignment accuracy of data acquired by sensors of different sensor types.

[0005] According to one aspect of this application, a time alignment method based on a multimodal sensor is provided, the method comprising: Based on the data acquisition timestamps of the target sensor during the data acquisition period, a target timestamp sequence is generated; where the target sensor refers to any one of the candidate sensors; and the candidate sensors refer to the sensors to be time-aligned. Based on the sensor acquisition data collected by the candidate sensors during the data acquisition period, an original sensor acquisition data sequence is generated; wherein, the original sensor acquisition data sequence includes the sensor acquisition data and the acquisition timestamps corresponding to the sensor acquisition data; Based on the target timestamp sequence, the original sensor acquisition data sequence is vectorized and aligned to generate a candidate sensor acquisition data sequence; wherein, the candidate sensor acquisition data sequence has the same sequence size as the target timestamp sequence; The candidate sensor acquisition data sequence is subjected to frame interpolation and dynamic frame rate adjustment to generate a target sensor acquisition data sequence with timestamp alignment.

[0006] According to another aspect of this application, a time alignment device based on a multimodal sensor is provided. The device includes a target timestamp sequence module for generating a target timestamp sequence based on the acquisition timestamps of data acquisition performed by the target sensor within the data acquisition period. The target sensor refers to any one of the candidate sensors, and the candidate sensor refers to the sensor to be time aligned. The raw data sequence module is used to generate a raw sensor acquisition data sequence based on the sensor acquisition data collected by the candidate sensor during the data acquisition period; wherein, the raw sensor acquisition data sequence includes the sensor acquisition data and the acquisition timestamps corresponding to the sensor acquisition data; The candidate data sequence module is used to perform vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequence based on the target timestamp sequence to generate a candidate sensor acquisition data sequence; wherein, the candidate sensor acquisition data sequence has the same sequence size as the target timestamp sequence; The timestamp alignment module is used to perform frame interpolation and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the timestamp-aligned target sensor acquisition data sequence, including: .

[0007] According to another aspect of this application, an electronic device is provided, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the time alignment methods based on multimodal sensors provided in the embodiments of this application.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the time alignment methods based on multimodal sensors provided in the embodiments of this application.

[0009] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the time alignment methods based on multimodal sensors provided in the embodiments of this application.

[0010] This application improves the timestamp alignment accuracy of target sensor acquisition data sequences corresponding to different sensors by determining the target timestamp sequence as the global timestamp, performing vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequences corresponding to candidate sensors to generate candidate sensor acquisition data sequences, and performing frame interpolation processing and dynamic frame rate adjustment on the candidate sensor acquisition data sequences to generate target sensor acquisition data sequences corresponding to candidate sensors. Attached Figure Description

[0011] Figure 1 This is a flowchart of a time alignment method based on a multimodal sensor provided in Embodiment 1 of this application; Figure 2 This is a flowchart of a time alignment method based on a multimodal sensor provided in Embodiment 2 of this application; Figure 3 This is a schematic diagram of a time alignment device based on a multimodal sensor provided in Embodiment 3 of this application; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the time alignment method based on a multimodal sensor according to Embodiment 4 of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Example 1 Figure 1This is a flowchart of a time alignment method based on a multimodal sensor according to Embodiment 1 of this application. This embodiment is applicable to the situation where, during data acquisition by multimodal sensors, timestamps of sensor data acquired by different sensors are aligned. This alignment can be performed by a time alignment device based on a multimodal sensor, which can be implemented in hardware and / or software and can be configured in a computer device, such as a server. Figure 1 As shown, the method includes: S110. Generate a target timestamp sequence based on the data acquisition timestamps of the target sensor during the data acquisition period.

[0015] The target sensor can refer to any one of the candidate sensors; the candidate sensor can refer to the sensor to be time-aligned. It should be noted that in this embodiment of the invention, there can be multiple candidate sensors, such as multimodal sensors in a robot (head camera sensor, wrist camera sensor, depth sensor, and motion sensor, etc.). Optionally, the data acquisition period can be adaptively set according to those skilled in the art.

[0016] Optionally, a target timestamp sequence is generated based on the acquisition timestamps of the target sensor during the data acquisition period, including: generating an original timestamp sequence based on the acquisition timestamps of the target sensor during the data acquisition period; performing timestamp interpolation and padding on the original timestamp sequence based on the sensor type of the target sensor to generate a candidate timestamp sequence; and downsampling the candidate timestamp sequence based on the original sampling frame rate and the target sampling frame rate of the target sensor to generate the target timestamp sequence.

[0017] The original sampling frame rate is greater than the target sampling frame rate. It should be noted that the original sampling frame rate refers to the sampling frequency during data acquisition by the sensor. The original sampling frequency can be set according to the sensor type and can be the same or different. The target sampling frequency can be used to process the acquired sensor data to reduce data redundancy. Optionally, the target sampling frequency can be adaptively set according to those skilled in the art.

[0018] Optionally, based on the sensor type of the target sensor, timestamp interpolation padding is performed on the original timestamp sequence, including: determining the maximum sampling time interval corresponding to the target sensor according to the sensor type of the target sensor; traversing the original timestamp sequence to determine the time interval corresponding to each pair of adjacent timestamps, and if the time interval is greater than the maximum sampling time interval, then the adjacent timestamp is taken as the adjacent timestamp pair to be filled; determining the number of padding timestamps to be inserted in the adjacent timestamp pair to be filled according to the time interval corresponding to the adjacent timestamp pair to be filled and the maximum sampling time interval, and uniformly inserting padding timestamps in the adjacent timestamp pair to be filled using a linear interpolation method.

[0019] The maximum sampling time interval can refer to the maximum allowed sampling time interval between two adjacent sensor data acquisitions.

[0020] In one alternative implementation, the acquisition timestamps of the target sensor during the data acquisition period can be obtained from a storage tool (e.g., a ROS bag data logging tool) that stores the sensor acquisition data, and the original timestamp sequence can be generated. Different sensor types have different maximum sampling time intervals. For example, the maximum sampling time interval for a head camera sensor is 39.8 ms, the maximum sampling time interval for a wrist camera sensor is 8 ms, the maximum sampling time interval for a depth sensor is 8 ms, and the maximum sampling time interval for a motion sensor is 4 ms.

[0021] It should be noted that if the time interval corresponding to the adjacent timestamp pair to be filled is greater than the maximum sampling time interval, the time interval corresponding to the adjacent timestamp pair to be filled can be divided by the maximum sampling time interval and rounded down to determine the number of fill timestamps to be inserted into the adjacent timestamp pair to be filled. Then, a linear interpolation method is used to uniformly insert the fill timestamps corresponding to the number of fill timestamps into the adjacent timestamp pair to be filled. Optionally, uniformly inserting fill timestamps can mean uniformly filling the time interval corresponding to the adjacent timestamp pair to be filled according to the number of fill timestamps.

[0022] It should be noted that the downsampling step size can be determined by rounding down the result of dividing the original sampling frame rate of the target sensor by the target sampling frame rate. For example, if the downsampling step size is 1, then the target timestamp sequence of the target sensor is consistent with the candidate timestamp sequence of the target sensor.

[0023] By padding the original timestamp sequence of the target sensor according to its sensor type and downsampling the corresponding timestamp sequence, the accuracy of the target timestamp sequence as a global timestamp standard is improved.

[0024] Optionally, in this embodiment of the invention, after generating the original timestamp sequence, the method further includes: performing unit precision conversion on the original timestamp sequence, and performing deduplication processing on the original timestamp sequence after unit precision conversion.

[0025] For example, each timestamp in the original timestamp sequence is uniformly converted into a nanosecond integer, and the original timestamp sequence is traversed. The first hash set is used to record the timestamps that have appeared, and duplicate timestamps are deduplicated to obtain the deduplicated original timestamp sequence. It should be noted that the time format of the timestamps in the original timestamp sequence before unit precision conversion is a second-level floating-point number. However, second-level floating-point numbers will produce rounding errors for nanosecond-level time differences, causing two originally different timestamps to become the same value in floating-point representation.

[0026] By performing unit precision conversion and deduplication on the original timestamp sequence, the accuracy and data quality of the original timestamp sequence were improved.

[0027] S120. Generate the original sensor acquisition data sequence based on the sensor acquisition data collected by the candidate sensor during the data acquisition period.

[0028] The original sensor data sequence may include the sensor data and the timestamps corresponding to the sensor data.

[0029] Optionally, in this embodiment of the invention, after generating the original sensor acquisition data sequence, the method further includes: performing deduplication processing on the original sensor acquisition data sequence based on the acquisition timestamp, and performing stutter detection and interpolation processing on the deduplicated original sensor acquisition data sequence.

[0030] For example, deduplication of the original sensor acquisition data sequence may include: for each acquisition timestamp corresponding to each sensor acquisition data in the original sensor acquisition data sequence, converting the acquisition timestamp into a nanosecond integer, traversing each acquisition timestamp, using a second hash set to record the timestamps that have appeared, and deleting the acquisition timestamp and the corresponding sensor acquisition data from the original sensor acquisition data sequence for duplicate acquisition timestamps.

[0031] Optionally, stuttering detection and interpolation based on the deduplicated original sensor data sequence may include: determining the maximum sampling time interval corresponding to the candidate sensors based on their sensor types; iterating through the sampling timestamps in the original sensor data sequence to determine the time interval corresponding to each pair of adjacent sampling timestamps; if this time interval is greater than the maximum sampling time interval, then using the adjacent sampling timestamps as the pair to be filled; determining the number of fill timestamps to be inserted into the pair to be filled based on the time intervals corresponding to the pairs to be filled and the maximum sampling time interval; and uniformly inserting the fill timestamps corresponding to this number of fill timestamps into the pairs to be filled using a linear interpolation method. It should be noted that, in the original sensor data sequence, for the fill timestamp, the sensor data corresponding to the previous frame timestamp can be copied as the sensor data corresponding to that fill timestamp.

[0032] S130. Based on the target timestamp sequence, the original sensor data sequence is vectorized and aligned with adjacent timestamps to generate candidate sensor data sequences.

[0033] The candidate sensor data sequence can have the same sequence size as the target timestamp sequence.

[0034] S140. Perform frame interpolation and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence after timestamp alignment.

[0035] Optionally, in this embodiment of the invention, after generating the target timestamp sequence, the method further includes: performing streaming continuity verification based on the current target timestamp sequence corresponding to the current data acquisition period and the historical target timestamp sequence corresponding to the previous data acquisition period; if the first acquisition timestamp in the current target timestamp sequence is less than or equal to the last acquisition timestamp in the historical target timestamp sequence, then the current target timestamp sequence is shifted backward as a whole, so that the first acquisition timestamp in the current target timestamp sequence is greater than the last acquisition timestamp in the historical target timestamp sequence, and the time difference between the two is within a preset time difference range; if the first acquisition timestamp in the current target timestamp sequence is greater than the last acquisition timestamp in the historical target timestamp sequence, then the method further includes: performing streaming continuity verification based on the current target timestamp sequence corresponding to the current data acquisition period and the historical target timestamp sequence corresponding to the previous data acquisition period .... If the time difference between the two timestamps is less than a first preset target time difference, then the current target timestamp sequence is shifted backward as a whole, so that the first collection timestamp in the current target timestamp sequence is greater than the last collection timestamp in the historical target timestamp sequence, and the time difference between them is within a preset time difference range. If the first collection timestamp in the current target timestamp sequence is greater than the last collection timestamp in the historical target timestamp sequence, and the time difference between them is greater than a second preset target time difference, then the current target timestamp sequence is shifted forward as a whole, so that the first collection timestamp in the current target timestamp sequence is greater than the last collection timestamp in the historical target timestamp sequence, and the time difference between them is within a preset time difference range. Optionally, the second preset target time difference can be three times the first preset target time difference. The first preset target time difference, the second preset target time difference, and the preset time difference range can be adaptively set according to those skilled in the art.

[0036] Optionally, the accuracy of the target timestamp sequence within the current data acquisition period is determined by performing streaming continuity verification on the target timestamp sequence.

[0037] This application embodiment determines the target timestamp sequence as the global timestamp, performs vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequence corresponding to the candidate sensor to generate the candidate sensor acquisition data sequence, and performs frame interpolation processing and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence corresponding to the candidate sensor. This improves the timestamp alignment accuracy of the target sensor acquisition data sequence corresponding to different sensors, thereby improving the data quality of the sensor data acquired by the multimodal sensor.

[0038] Example 2 Figure 2This is a flowchart of a time alignment method based on a multimodal sensor according to Embodiment 2 of this application. Based on the technical solutions of the above embodiments, this embodiment further refines the process of "performing frame interpolation and dynamic frame rate adjustment on the candidate sensor acquired data sequence to generate a timestamp-aligned target sensor acquired data sequence." It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments. Figure 2 As shown, the method includes: S210. Generate a target timestamp sequence based on the data acquisition timestamps of the target sensor during the data acquisition period.

[0039] S220. Generate the original sensor acquisition data sequence based on the sensor acquisition data collected by the candidate sensor during the data acquisition period.

[0040] S230. Based on the target timestamp sequence, the original sensor data sequence is vectorized and aligned with adjacent timestamps to generate candidate sensor data sequences.

[0041] Optionally, the original sensor acquisition data sequence is vectorized into a timestamp neighbor alignment process based on the target timestamp sequence to generate a candidate sensor acquisition data sequence. This includes: traversing each target timestamp in the target timestamp sequence and using a binary search algorithm to determine the target insertion time point corresponding to the target timestamp in the original sensor acquisition data sequence; determining two adjacent data points in the original sensor acquisition data sequence based on the target insertion time point, and vectorizing the adjacent data to generate a left adjacent data vector and a right adjacent data vector; for each target timestamp, calculating the timestamp difference with the corresponding left adjacent data in the left adjacent data vector and the corresponding right adjacent data in the right adjacent data vector, and selecting the adjacent data with the smallest timestamp difference as the candidate sensor acquisition data to generate a candidate sensor acquisition data sequence.

[0042] It should be noted that the left adjacent data vector records the index of the left adjacent data corresponding to each target insertion time point, and the right adjacent data vector records the index of the right adjacent data corresponding to each target insertion time point. It should also be noted that the target insertion time point can refer to the acquisition timestamp corresponding to the sensor acquisition data in the original sensor acquisition data sequence.

[0043] For example, a binary search can be used to determine the target insertion time point sequence corresponding to each target timestamp in the original sensor data acquisition sequence. By traversing the target insertion time points in the target insertion time point sequence, the left and right adjacent time points corresponding to each target insertion time point are determined. The sequence of left adjacent time points corresponding to each target insertion time point is taken as the left adjacent data vector, and the sequence of right adjacent time points corresponding to each target insertion time point is taken as the right adjacent data vector. It should be noted that the acquisition timestamps in the original sensor data acquisition sequence can be regarded as index labels of the sensor acquisition data.

[0044] By vectorizing adjacent data, the efficiency of determining candidate sensor data is improved.

[0045] S240. Traverse the candidate sensor data sequence and the target timestamp sequence respectively to determine the sensor data to be corrected.

[0046] The candidate sensor data to be corrected can be used to indicate that the time difference between the timestamp carried by the sensor data and its corresponding target timestamp is greater than a preset time difference. Optionally, the preset time difference can be adaptively set according to those skilled in the art.

[0047] Specifically, the candidate sensor data sequences and the target timestamp sequence can be traversed sequentially. The difference between the acquisition timestamp and the target timestamp of the currently traversed candidate sensor data can be calculated to identify candidate sensor data sequences that require correction. For example, if the difference between the acquisition timestamp and the target timestamp of a candidate sensor data sequence is greater than 10ms, then that candidate sensor data can be identified as candidate sensor data requiring correction.

[0048] S250. Based on the original sensor data sequence corresponding to the candidate sensor data sequence, perform data frame replacement on the sensor data to be corrected.

[0049] Specifically, for the sensor data to be corrected in the candidate sensor data acquisition sequence, the sensor data corresponding to the acquisition timestamp closest to the acquisition timestamp of the sensor data to be corrected is found in the original sensor data acquisition sequence corresponding to the candidate sensor data acquisition sequence, and the sensor data to be corrected is replaced until the time difference between the acquisition timestamp of the replaced sensor data and its corresponding target timestamp is less than the preset time difference.

[0050] S260. Based on the target sampling frame rate of the target timestamp sequence, dynamically adjust the frame rate of the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence.

[0051] Optionally, based on the target sampling frame rate of the target timestamp sequence, the candidate sensor acquisition data sequence is dynamically adjusted to generate the target sensor acquisition data sequence, including: if the actual sampling frame rate corresponding to the candidate sensor acquisition data sequence is less than the target sampling frame rate, then the candidate sensor acquisition data sequence is subjected to frame interpolation; if the actual sampling frame rate corresponding to the candidate sensor acquisition data sequence is greater than the target sampling frame rate, then the candidate sensor acquisition data sequence is subjected to frame extraction.

[0052] Frame interpolation refers to inserting new frame data between data points with the longest time interval in a candidate sensor data sequence. The acquisition timestamp corresponding to the interpolated frame can be the median of the longest time interval. The data value of the interpolated frame can be the same as the data value of its left-adjacent data. It should be noted that the interpolated frame data can refer to sensor data carrying acquisition timestamps.

[0053] Frame extraction can refer to deleting the sensor data with the smallest sum of time intervals between the acquisition timestamps of the candidate sensor data sequence and the data acquired by the two adjacent sensors, and then sorting the candidate sensor data sequence after the deletion operation according to the actual sampling timestamps (the data acquired by the first and last sensors remain unchanged).

[0054] It should be noted that the actual sampling frame rate corresponding to the candidate sensor data sequence can be determined based on the total number of data frames acquired in the candidate sensor data sequence and the time difference between the acquisition timestamps of the first and last data frames in the candidate sensor data sequence. Optionally, the termination condition for frame interpolation can be that the maximum time interval in the candidate sensor data sequence is less than the target time interval, and the target time interval can be determined based on the target sampling frame rate.

[0055] By adaptively adjusting the frame rate of the candidate sensor data sequences, the consistency between the target sensor data sequences and the target timestamp sequences in the time dimension is ensured.

[0056] This application embodiment replaces the candidate sensor acquisition data sequence with data frames by using the original sensor acquisition data sequence, and dynamically adjusts the frame rate of the candidate sensor acquisition data sequence based on the target sampling frame rate, thereby further improving the time alignment accuracy of multimodal sensor data.

[0057] Example 3 Figure 3This is a schematic diagram of a time alignment device based on a multimodal sensor according to Embodiment 3 of this application. It is applicable to situations where, during data acquisition by multimodal sensors, timestamps of sensor data collected by different sensors are aligned. This time alignment device based on a multimodal sensor can be implemented in hardware and / or software and can be configured in a computer device, such as a server. Figure 3 As shown, the device includes: The target timestamp sequence module 310 is used to generate a target timestamp sequence based on the acquisition timestamps of data acquired by the target sensor during the data acquisition period; wherein, the target sensor refers to any one of the candidate sensors; and the candidate sensor refers to the sensor to be time aligned. The raw data sequence module 320 is used to generate a raw sensor acquisition data sequence based on the sensor acquisition data collected by the candidate sensor during the data acquisition period; wherein, the raw sensor acquisition data sequence includes the sensor acquisition data and the acquisition timestamps corresponding to the sensor acquisition data; The candidate data sequence module 330 is used to perform vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequence based on the target timestamp sequence to generate a candidate sensor acquisition data sequence; wherein, the candidate sensor acquisition data sequence has the same sequence size as the target timestamp sequence; The timestamp alignment module 340 is used to perform frame interpolation and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence after timestamp alignment.

[0058] This application embodiment determines the target timestamp sequence as the global timestamp, performs vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequence corresponding to the candidate sensor to generate the candidate sensor acquisition data sequence, and performs frame interpolation processing and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence corresponding to the candidate sensor. This improves the timestamp alignment accuracy of the target sensor acquisition data sequence corresponding to different sensors, thereby improving the data quality of the sensor data acquired by the multimodal sensor.

[0059] Optional, the timestamp alignment module 340 includes: The data unit to be corrected is used to traverse the candidate sensor acquisition data sequence and the target timestamp sequence respectively to determine the sensor acquisition data to be corrected; wherein, the candidate sensor acquisition data to be corrected is used to indicate that the time difference between the timestamp carried by the sensor acquisition data and its corresponding target timestamp is greater than a preset time difference. The data replacement unit is used to replace the data frames of the sensor data to be corrected with the original sensor data sequence corresponding to the candidate sensor data sequence. The frame rate adjustment unit is used to dynamically adjust the frame rate of the candidate sensor acquisition data sequence based on the target sampling frame rate of the target timestamp sequence, and generate the target sensor acquisition data sequence.

[0060] Optionally, the frame rate adjustment unit includes: The frame interpolation subunit is used to perform frame interpolation processing on the candidate sensor data sequence if the actual sampling frame rate corresponding to the candidate sensor data sequence is less than the target sampling frame rate; wherein, the frame interpolation processing refers to inserting new interpolated frame data between the data with the largest time interval in the candidate sensor data sequence; the data value of the interpolated frame data is the same as the data value of its left adjacent data; The frame extraction subunit is used to perform frame extraction processing on the candidate sensor data sequence if the actual sampling frame rate corresponding to the candidate sensor data sequence is greater than the target sampling frame rate.

[0061] Optionally, the candidate data sequence module 330 includes: The target insertion time determination unit is used to traverse each target timestamp in the target timestamp sequence and use a binary search algorithm to determine the target insertion time point corresponding to the target timestamp in the original sensor data acquisition sequence. The vectorization unit is used to determine two adjacent data points in the original sensor data sequence based on the target insertion time point, and to vectorize the adjacent data to generate a left adjacent data vector and a right adjacent data vector. The candidate data determination unit is used to calculate the time difference between each target timestamp and the corresponding left adjacent data in the left adjacent data vector and the corresponding right adjacent data in the right adjacent data vector, respectively. The adjacent data with the smallest timestamp difference is selected as candidate sensor acquisition data to generate a candidate sensor acquisition data sequence.

[0062] Optionally, the target timestamp sequence module 310 includes: The raw timestamp sequence unit is used to generate a raw timestamp sequence based on the acquisition timestamps of the target sensor during the data acquisition period. The candidate timestamp sequence unit is used to perform timestamp interpolation and padding on the original timestamp sequence based on the sensor type of the target sensor to generate a candidate timestamp sequence. The target timestamp sequence unit is used to downsample the candidate timestamp sequence based on the original sampling frame rate and the target sampling frame rate of the target sensor to generate a target timestamp sequence; wherein the original sampling frame rate is greater than the target sampling frame rate.

[0063] Optionally, the candidate timestamp sequence unit can be specifically used for: Based on the sensor type of the target sensor, determine the maximum sampling time interval corresponding to the target sensor; The original timestamp sequence is traversed to determine the time interval corresponding to each pair of adjacent timestamps. If the time interval is greater than the maximum sampling time interval, the adjacent timestamp is taken as the pair of adjacent timestamps to be filled. Based on the time interval corresponding to the adjacent timestamp pairs to be filled and the maximum sampling time interval, the number of filling timestamps to be inserted into the adjacent timestamp pairs to be filled is determined, and the filling timestamps are uniformly inserted into the adjacent timestamp pairs to be filled using a linear interpolation method.

[0064] The time alignment device based on multimodal sensors provided in this application can execute the time alignment method based on multimodal sensors provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each time alignment method based on multimodal sensors.

[0065] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0066] Example 4 Figure 4 This is a schematic diagram of the structure of an electronic device 410 implementing the time alignment method based on a multimodal sensor according to embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0067] like Figure 4As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory 412 or a random access memory 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 412 or loaded from storage unit 418 into the random access memory 413. The random access memory 413 can also store various programs and data required for the operation of the electronic device 410. The processor 411, read-only memory 412, and random access memory 413 are interconnected via a bus 414. An input / output interface 415 is also connected to the bus 414.

[0068] Multiple components in electronic device 410 are connected to input / output interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of monitors, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0069] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as time alignment methods based on multimodal sensors.

[0070] In some embodiments, the multimodal sensor-based time alignment method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 410 via read-only memory 412 and / or communication unit 419. When the computer program is loaded into random access memory 413 and executed by processor 411, one or more steps of the multimodal sensor-based time alignment method described above can be performed. Alternatively, in other embodiments, processor 411 can be configured for the multimodal sensor-based time alignment method by any other suitable means (e.g., by means of firmware).

[0071] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable time alignment device based on a multimodal sensor, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0073] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0074] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0075] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0076] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0077] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0078] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A time alignment method based on a multimodal sensor, characterized in that, include: Based on the data acquisition timestamps of the target sensor during the data acquisition period, a target timestamp sequence is generated; where the target sensor refers to any one of the candidate sensors; and the candidate sensors refer to the sensors to be time-aligned. Based on the sensor acquisition data collected by the candidate sensors during the data acquisition period, an original sensor acquisition data sequence is generated; wherein, the original sensor acquisition data sequence includes the sensor acquisition data and the acquisition timestamps corresponding to the sensor acquisition data; Based on the target timestamp sequence, the original sensor acquisition data sequence is vectorized and aligned to generate a candidate sensor acquisition data sequence; wherein, the candidate sensor acquisition data sequence has the same sequence size as the target timestamp sequence; The candidate sensor acquisition data sequence is subjected to frame interpolation and dynamic frame rate adjustment to generate a target sensor acquisition data sequence with timestamp alignment.

2. The method according to claim 1, characterized in that, The step of performing frame interpolation and dynamic frame rate adjustment on the candidate sensor data sequence to generate a timestamp-aligned target sensor data sequence includes: The candidate sensor data sequence and the target timestamp sequence are traversed to determine the sensor data to be corrected; wherein, the candidate sensor data to be corrected is used to indicate that the time difference between the timestamp carried by the sensor data and its corresponding target timestamp is greater than a preset time difference. Based on the original sensor acquisition data sequence corresponding to the candidate sensor acquisition data sequence, the data frame of the sensor acquisition data to be corrected is replaced. Based on the target sampling frame rate of the target timestamp sequence, the candidate sensor acquisition data sequence is dynamically adjusted to generate the target sensor acquisition data sequence.

3. The method according to claim 2, characterized in that, The step of dynamically adjusting the frame rate of the candidate sensor acquisition data sequence based on the target sampling frame rate of the target timestamp sequence to generate the target sensor acquisition data sequence includes: If the actual sampling frame rate corresponding to the candidate sensor data sequence is less than the target sampling frame rate, then the candidate sensor data sequence is subjected to frame interpolation processing; wherein, the frame interpolation processing refers to inserting new interpolated frame data between the data with the largest time interval in the candidate sensor data sequence; the data value of the interpolated frame data is the same as the data value of its left adjacent data; If the actual sampling frame rate corresponding to the candidate sensor data sequence is greater than the target sampling frame rate, then the candidate sensor data sequence is subjected to frame extraction processing.

4. The method according to claim 1, characterized in that, The step of performing vectorized timestamp neighbor alignment processing on the original sensor-acquired data sequence based on the target timestamp sequence to generate candidate sensor-acquired data sequences includes: For each target timestamp in the target timestamp sequence, the binary search algorithm is used to determine the target insertion time point corresponding to the target timestamp in the original sensor data acquisition sequence. Based on the target insertion time point, determine two adjacent data points in the original sensor data acquisition sequence at the target insertion time point, and vectorize the adjacent data to generate a left adjacent data vector and a right adjacent data vector; For each target timestamp, the timestamp difference is calculated between the corresponding left adjacent data in the left adjacent data vector and the corresponding right adjacent data in the right adjacent data vector. The adjacent data with the smallest timestamp difference is selected as candidate sensor data to generate a candidate sensor data sequence.

5. The method according to claim 1, characterized in that, The generation of a target timestamp sequence based on the data acquisition timestamps of the target sensor during the data acquisition period includes: Generate the original timestamp sequence based on the timestamps of the data acquisition performed by the target sensor during the data acquisition period; Based on the sensor type of the target sensor, the original timestamp sequence is timestamp interpolated and padded to generate a candidate timestamp sequence; Based on the original sampling frame rate and the target sampling frame rate of the target sensor, the candidate timestamp sequence is downsampled to generate the target timestamp sequence; wherein the original sampling frame rate is greater than the target sampling frame rate.

6. The method according to claim 5, characterized in that, The process of performing timestamp interpolation padding on the original timestamp sequence based on the sensor type of the target sensor includes: Based on the sensor type of the target sensor, determine the maximum sampling time interval corresponding to the target sensor; The original timestamp sequence is traversed to determine the time interval corresponding to each pair of adjacent timestamps. If the time interval is greater than the maximum sampling time interval, the adjacent timestamp is taken as the pair of adjacent timestamps to be filled. Based on the time interval corresponding to the adjacent timestamp pairs to be filled and the maximum sampling time interval, the number of filling timestamps to be inserted into the adjacent timestamp pairs to be filled is determined, and the filling timestamps are uniformly inserted into the adjacent timestamp pairs to be filled using a linear interpolation method.

7. A time alignment device based on a multimodal sensor, characterized in that, include: The target timestamp sequence module is used to generate a target timestamp sequence based on the acquisition timestamps of data acquired by the target sensor within the data acquisition period; where the target sensor refers to any one of the candidate sensors; and the candidate sensors refer to the sensors to be time-aligned. The raw data sequence module is used to generate a raw sensor acquisition data sequence based on the sensor acquisition data collected by the candidate sensor during the data acquisition period; wherein, the raw sensor acquisition data sequence includes the sensor acquisition data and the acquisition timestamps corresponding to the sensor acquisition data; The candidate data sequence module is used to perform vectorized timestamp neighbor alignment processing on the original sensor acquisition data sequence based on the target timestamp sequence to generate a candidate sensor acquisition data sequence; wherein, the candidate sensor acquisition data sequence has the same sequence size as the target timestamp sequence; The timestamp alignment module is used to perform frame interpolation and dynamic frame rate adjustment on the candidate sensor acquisition data sequence to generate the target sensor acquisition data sequence after timestamp alignment.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the time alignment method based on a multimodal sensor as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the time alignment method based on a multimodal sensor as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the time alignment method based on a multimodal sensor according to any one of claims 1-6.

Images