Multimodal data synchronization method, device, collection system and electronic equipment

Abstract

The present application discloses a multimodal data synchronization method, device, collection system and electronic device. The method includes: determining the initial timestamp corresponding to the to-be-processed data in the to-be-processed data set; determining the confidence of the initial timestamp of the to-be-processed data, and determining the reference timeline based on the confidence, and determining the reference data from the to-be-processed data set; determining the reference event in the reference data, and the associated event corresponding to the reference event in the target to-be-processed data, the target to-be-processed data being any other to-be-processed data except the reference data; determining the reference timestamp of the reference event based on the reference timeline, and correcting the time flow of the target to-be-processed data based on the reference timestamp, the reference timeline and the associated event, so that the other to-be-processed data are aligned with the reference data. The present application solves the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in the related art.

Description

Technical Field

[0001] The present application relates to the field of electrical digital data processing, and in particular, to a multi-modal data synchronization method, apparatus, collection system, and electronic device. Background Art

[0002] In the related art, when collecting multi-modal data, since different modal data are usually collected separately, and there may be problems such as time deviation of data sources during the collection process, the multi-modal data that was originally synchronized in time becomes unsynchronized, resulting in a decrease in the effectiveness of the data.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present application provide a multi-modal data synchronization method, apparatus, collection system, and electronic device, so as to at least solve the technical problem of low data effectiveness caused by the inability to achieve time synchronization between multi-modal data in the related art.

[0005] According to one aspect of the embodiments of the present application, a multi-modal data synchronization method is provided, including: determining an initial timestamp corresponding to the data to be processed in the set of data to be processed, where different data to be processed have different data modalities; determining the confidence level of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence level, and determining reference data from the set of data to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed other than the reference data; determining the reference timestamp of the reference event based on the reference time axis, and correcting the time stream of the target data to be processed based on the reference timestamp, the reference time axis, and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference time axis and the reference timestamp is within a preset value range.

[0006] Optionally, correcting the time stream of the target data to be processed based on the reference timestamp, the reference time axis, and the associated event includes: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed on the reference time axis according to the preset compression ratio and the duration of the correction process until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, where each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

[0007] Optionally, correcting the data reporting times corresponding to each group of data in the target data to be processed on the reference time axis according to a preset compression ratio and the duration of the correction process includes: correcting the data reporting times corresponding to each group of data on the reference time axis by using the following formula:

[0008]

[0009] where t i represents the corrected data reporting time, t i-1 represents the previous data reporting time adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

[0010] Optionally, the correction stop condition expression is as follows:

[0011]

[0012] where α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold.

[0013] Optionally, the method further includes: when the preset compression ratio and the duration of the correction process meet the correction stop condition, putting the target data to be processed into the compensation mechanism queue until there is a correction process in a preset number of correction processes such that the difference between the time stamp of the associated event in the target data to be processed and the reference time stamp with respect to the reference time axis is within a preset value range, where different correction processes correspond to different reference events, and different associated events correspond to different reference events; in the case where there is no correction process such that the difference between the time stamp of the associated event in the target data to be processed and the reference time stamp with respect to the reference time axis is within a preset value range, performing a rollback operation on the target data to be processed in the compensation mechanism queue so that the data reporting times corresponding to each group of data in the target data to be processed are rolled back to the data reporting times before the correction process is executed; and reducing the confidence level of the reference data, and then determining the reference data from various types of data to be processed again according to the confidence levels of various types of data to be processed.

[0014] Optionally, determining the initial time stamp corresponding to the data to be processed in the data set to be processed includes: when the data modality of the data to be processed is video data of the long-time stream data type, determining the average time elapsed speed of the first segment of video data and the second segment of video data in the data to be processed; determining the preset critical multiple and the number of windows in the data to be processed, where each window corresponds to a group of data in the data to be processed; and determining the initial time stamp of the data corresponding to each window according to the number of windows, the preset critical multiple, and the average time elapsed speed.

[0015] Optionally, the initial time stamp is determined by the following formula:

[0016]

[0017] where t k represents the data reporting time corresponding to the first window, and t k+i represents the data reporting time corresponding to the i-th window. The data reporting times corresponding to each window are used to determine the initial timestamp, and μ represents a preset critical multiple, representing the average speed of time passage.

[0018] Optionally, determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: in the case where the data modality of the data to be processed is a fixed change time event, determining the initial data reporting times corresponding to each group of data in the data to be processed, the time interval between adjacent groups of data, and a preset data reporting time compensation amount; determining the data reporting times corresponding to each group of data based on the initial data reporting times, the time interval between adjacent groups of data, and the preset data reporting time compensation amount, where the data reporting times corresponding to each group of data are used to determine the initial timestamp corresponding to the data to be processed.

[0019] According to another aspect of the embodiments of the present application, there is also provided a multi-modal data synchronization collection system, including a proxy server and a real-time acquisition preprocessing framework. The proxy server includes a preprocessing module and a time synchronization module. The preprocessing module is used to preprocess the data to be processed in the data set to be processed collected from the multi-modal data source; the time synchronization module is used to determine the initial timestamp corresponding to the data to be processed in the data set to be processed, where the data modalities of different data to be processed are different; determining the confidence level of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence level, and determining reference data from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed except the reference data; determining the reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis, and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference time axis and the reference timestamp is within a preset value range; the real-time acquisition preprocessing framework includes a flow acquisition engine and a multi-modal database, where the flow acquisition engine is used to store the aligned data into the corresponding multi-modal database.

[0020] According to another aspect of the embodiments of the present application, there is also provided a multimodal data synchronization device, including: a first processing module, configured to determine an initial timestamp corresponding to the data to be processed in the set of data to be processed, where different data to be processed have different data modalities; a second processing module, configured to determine the confidence level of the initial timestamp of the data to be processed, and determine a reference timeline based on the confidence level, and determine reference data from the set of data to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the set of data to be processed; a third processing module, configured to determine a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed other than the reference data; a fourth processing module, configured to determine the reference timestamp of the reference event based on the reference timeline, and correct the time stream of the target data to be processed based on the reference timestamp, the reference timeline, and the associated event, so that other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference timeline and the reference timestamp is within a preset value range.

[0021] According to another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, in which a program is stored, and when the program runs, it controls the device where the non-volatile storage medium is located to execute the multimodal data synchronization method.

[0022] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory and a processor, where the processor is configured to run the program stored in the memory, and when the program runs, it executes the multimodal data synchronization method.

[0023] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal data synchronization method.

[0024] In an embodiment of the present application, an initial timestamp corresponding to the data to be processed in the data set to be processed is determined, where different data to be processed have different data modalities; the confidence level of the initial timestamp of the data to be processed is determined, and a reference time axis is determined based on the confidence level, and reference data is determined from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed; a reference event is determined in the reference data, and an associated event corresponding to the reference event in the target data to be processed is determined, where the target data to be processed is any other data to be processed except the reference data; the reference timestamp of the reference event is determined based on the reference time axis, and the time stream of the target data to be processed is corrected based on the reference timestamp, the reference time axis, and the associated event, so that other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference time axis and the reference timestamp is within a preset value range. By determining the reference data and the reference time axis according to the timestamp confidence level of the data to be processed, determining the reference event in the reference data and the associated event in other data to be processed, and finally correcting the time stream of other data to be processed according to the reference timestamp of the reference event, the associated event, and the reference time axis, the purpose of time synchronization of multimodal data is achieved, thereby realizing the technical effect of improving the correlation between different modal data for subsequent further data mining and analysis, and further solving the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0026] Figure 1 is a schematic structural diagram of a multimodal data synchronous acquisition system provided according to an embodiment of the present application;

[0027] Figure 2 is a schematic structural diagram of a preprocessing module provided according to an embodiment of the present application;

[0028] Figure 3 is a schematic flowchart of a multimodal data synchronization method provided according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of the reporting time of multimodal data in an ideal state provided according to an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of the reporting time of another multimodal data provided according to an embodiment of the present application;

[0031] Figure 6 is a schematic structural diagram of a multimodal data synchronization device provided according to an embodiment of the present application;

[0032] Figure 7 is a schematic structural diagram of a computer terminal (portable terminal) provided according to an embodiment of the present application; Detailed implementation manners

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0035] To better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0036] Multimodal Data: Data containing multiple forms, such as text, images, audio, and video, etc.

[0037] Cluster Architecture: A system composed of multiple servers or computers that work together to achieve high performance, high availability, and scalability.

[0038] Distributed System: System components are distributed on different network nodes and jointly complete tasks through communication and coordination.

[0039] Data Processing: Operations such as collecting, organizing, transforming, and analyzing data to extract useful information and support decision-making.

[0040] In the multi-modal data processing solutions in the related art, fixed data types are usually used to correspond to fixed solutions, which have the following problems:

[0041] Data isolation: For example, image data is processed and saved using computer vision (CV) technology; audio data is transcribed and classified using automatic speech recognition (ASR) technology, etc. However, the diversity of the same related data is difficult to analyze and recognize in a single dimension, and it is easy to lose the correlation of the data, resulting in information fragmentation.

[0042] Time asynchronization: In multi-modal data processing, the data generated by different data sources has a time effect. However, due to often separate collection during acquisition, or time deviation in the generation time of the data sources, time asynchronization occurs, which reduces the effectiveness of data processing. For example, in intelligent monitoring, obtaining video and audio data simultaneously can better predict the occurrence of special events and improve the accuracy of event recognition.

[0043] Centralized operation: Multi-modal data often needs to be synchronized batch by batch offline and requires centralized unified processing. It is impossible to perform preliminary processing and synchronization at the data source; endpoint computing is required to reduce transmission latency and improve transmission speed.

[0044] Due to the above problems in the related technology, it is necessary to perform unified and real-time acquisition and processing of multi-modal data, so as to achieve time synchronization between the data, and then realize the same integration of features and joint modeling to improve the analysis accuracy.

[0045] To solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0046] According to the embodiments of the present application, a method embodiment of a multi-modal data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0047] The method provided by the embodiments of the present application can run in a multi-modal data synchronization and collection system as Figure 1 shown. As Figure 1 shown, the system includes: a proxy server 10, a real-time acquisition and preprocessing framework 12, where

[0048] The proxy server 10 includes a preprocessing module 102 and a time synchronization module 104, where the preprocessing module is used to preprocess the data to be processed in the data set to be processed collected from the multi-modal data source;

[0049] The time synchronization module 104 is used to determine the initial timestamp corresponding to the data to be processed in the data set to be processed, where the data modalities of different data to be processed are different; determine the confidence level of the initial timestamp of the data to be processed, and determine the reference time axis based on the confidence level, and determine the reference data from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed; determine the reference event in the reference data, and the associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed except the reference data; determine the reference timestamp of the reference event based on the reference time axis, and correct the time stream of the target data to be processed according to the reference timestamp, the reference time axis and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference time axis and the reference timestamp is within a preset value range;

[0050] The real-time acquisition preprocessing framework 12 includes a stream acquisition engine 122 and a multimodal database, where the stream acquisition engine 122 is used to store the aligned data into the corresponding multimodal database.

[0051] Specifically, as Figure 1 shown, it is assumed that data of different modalities are acquired in real time from the multimodal data source A, the multimodal data source B, and the multimodal data source C, and the multimodal data source A, the multimodal data source B, and the multimodal data source C are data in the same environment and logically have the same time and space attributes. Additionally, it can be understood that the three multimodal data sources in this application are only for illustration and do not mean that the number of multimodal data sources can only be three. In the actual process of using this solution, the number of multimodal data sources can be more or less.

[0052] After the multimodal data is acquired from each multimodal data source, the preprocessing module 102 can be used to preprocess the multimodal data first, and then the time synchronization module 104 can be used to perform time synchronization processing on the preprocessed multimodal data and format the data synchronization information. Among them, the methods adopted by the preprocessing module 102 when preprocessing the multimodal data include screening available data information, standard formatting data information, etc.

[0053] In some embodiments of the present application, the preprocessing module 102 is as Figure 2As shown in the figure, various defined custom preprocessing plugins such as Filter, Standardization, Mapper, and AddAttribute can be arranged. Users can select corresponding plugins to perform corresponding operations on the corresponding data to collect data in a personalized manner. Among them, Filter is used to customize data filtering. The multi-modal data filtering conditions are not limited to value filtering and may be other formats of data filtering. Standardization mainly standardizes the collected data, and there may be different standardization methods for different data, such as video frame rate, picture color ratio, audio noise filtering, etc. Mapper mapping mainly performs the simplest tagging operation or value class mapping, such as converting basic attributes into codes; associating with relevant small dimension tables, etc. AddAttribute performs additional description of the data and adds additional attributes based on the data generation time, scenario, location, etc.

[0054] Under the above operating environment, an embodiment of the present application provides a multi-modal data synchronization method, as Figure 3 shown, the method includes the following steps:

[0055] Step S302, determine the initial timestamp corresponding to the data to be processed in the data set to be processed, where the data modalities of different data to be processed are different;

[0056] It should be noted that when collecting multi-modal data, due to different data sizes, generation delays, collection delays, or other reasons, there may be delays or differences in the time described by each dimension of the data, and even the time dimension may be lost or unable to be aligned. Data time synchronization standardizes and adds time dimension attributes during unified collection.

[0057] Suppose in an intelligent transportation scenario, there are three types of data, namely:

[0058] Traffic lights: a non-strict timed polling data type, and the defined data set is Da.

[0059] Surveillance photography: an event-triggered data type, and the defined data set is Db.

[0060] Roadside surveillance: a long-term streaming data type, that is, a continuous real-time video stream, and the defined data set is Dc.

[0061] The reporting times of each group of these three types of data in the ideal state are as Figure 4As shown, it should be reported synchronously. Within a period of time, the reporting times of the traffic light Internet of Things are {t1, t5, t9}; the photographing times of the monitoring and photographing are {Db1}; and the roadside monitoring is {Dc}, uploading data at all times. It can be seen that {Da1, Db1, Dc1}, {Da2, Dc2}, and {Da3, Db2, Dc3} are generated at the same time.

[0062] However, in the actual scenario, due to various interference factors, the actual reporting times of various types of data may be as Figure 5 shown. The actual reporting time of Da2 may change from t5 to t6. To solve this problem and achieve synchronization between different modalities of data, the embodiments of the present application provide a multi-modal data synchronization method as Figure 3 shown.

[0063] In some embodiments of the present application, the watermark concept is also introduced in the method provided by the embodiments of the present application. Watermark is a mechanism for processing data event time, indicating the progress of the event time stream. In the progress of the multi-modal data stream, since the event time may be uneven due to differences in reporting processes, device times, etc., the framework requires a mechanism to measure the progress of the event time and uniformly identify the occurrence of events in order to unify multi-modal real-time data.

[0064] Different from the watermark of the related technology, the watermark of the related technology is to solve the problem of disorder when data arrives, and most of them are judged by the time carried by the data. However, the watermark introduced by the framework provided by the embodiments of the present application is more inclined to unify the data time between multi-modal and multi-data, and solve the problem of different physical times collected by different collectors and Internet of Things devices.

[0065] The watermark introduced by the framework provided by the embodiments of the present application includes core components such as a watermark generator (WatermarkGenerator), a watermark confidence calculator (Watermark Confidence), a watermark synchronizer (Watermark SyncStrategy), and an event mark barrier (Eventmark Barrier).

[0066] Among them, the watermark generator is used to label the acquired streaming data of different modalities one by one to generate basic time data information TN, that is, the initial timestamp. Generally, when generating a timestamp, the watermark can be directly generated according to the time carried by the data itself. For unordered data sources, the framework provided in the embodiments of the present application also supports custom heuristic plugins to generate corresponding watermarks. In addition, it should be noted that the timestamps mentioned in the embodiments of the present application are timestamps corresponding to each group of data in various modality data, and the timestamp is used to indicate the reporting time of the group of data.

[0067] In the technical solution provided in step S302, when the data modality of the data to be processed is a fixed change time event, the method for determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: determining the initial data reporting time corresponding to each group of data in the data to be processed, the time interval between adjacent groups of data, and the preset data reporting time compensation amount; determining the data reporting time corresponding to each group of data according to the initial data reporting time, the time interval between adjacent groups of data, and the preset data reporting time compensation amount, where the data reporting time corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

[0068] For example, for a fixed change time event such as a traffic light, when there is data without time information in each group of uploaded data, a fixed time change watermark generator can be used to generate the initial timestamp. The specific calculation formula is as follows:

[0069] T n =T 0 +n*t interval ±t compensation

[0070] In the above formula, T 0 represents the reporting time of the earliest group of data arranged in time in the current processing cycle, T n is the reporting time of the (n + 1)-th group of data, t interval represents the data interval between each group of data, and t compensation is the preset confidence compensation amount.

[0071] In the technical solution provided in step S302, the steps of determining the initial timestamp corresponding to the data to be processed in the data set to be processed include: in the case where the data modality of the data to be processed is video data of the long-time stream data type, determining the average time elapsed speed of the first segment of video data and the second segment of video data in the data to be processed; determining a preset critical multiple and the number of windows in the data to be processed, where each window corresponds to a group of data in the data to be processed; and determining the initial timestamp of the data corresponding to each window according to the number of windows, the preset critical multiple, and the average time elapsed speed. The above-mentioned video data of the long-time stream data type includes roadside monitoring data, etc., and generally continuously outputs time signals. However, in some cases, when the data carries little time information or the confidence in the time information is low, the above-mentioned method for generating the initial timestamp can be adopted to generate the time series data interpolation of the video stream through a long monitoring stream.

[0072] As an alternative implementation, the initial timestamp of the video data of the long-time stream data type is determined by the following formula:

[0073]

[0074] where t k represents the data reporting time corresponding to the first window, t k+i represents the data reporting time corresponding to the i-th window, and the data reporting times corresponding to each window are used to determine the initial timestamp, μ represents the preset critical multiple, represents the average time elapsed speed, and the calculation formula is as follows:

[0075]

[0076] In the above formula, represents the average elapsed speed of the first K groups of data in the data, represents the average elapsed speed of the last K groups of data.

[0077] The value of the above critical multiple μ is 1, k is the number of windows, and the larger k is, the smoother the data tends to be, k ∈ N + . When the calculated average time elapsed speed exceeds the critical limit, it will be normalized to the preset critical limit.

[0078] Step S304, determining the confidence level of the initial timestamp of the data to be processed, and determining the reference time axis according to the confidence level, and determining the reference data from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed;

[0079] In the technical solution provided in step S304, after the watermark generator generates the watermark of the initial timestamp, the watermark confidence calculator will judge the confidence of the initial timestamp of each modality data. Generally speaking, the smoother the time data is, the higher the confidence; for the missing time data, the lower the confidence. The above judgment rule can be implemented by plug-in configuration to achieve the judgment process, and different algorithms can also be used to judge and construct specific data. For example, for videos, the offset can be calculated through video frame data, and the calculation formula is as follows:

[0080]

[0081] In the above formula, FR k is the frame rate at k seconds, is the average frame rate of the n seconds before k. Generally, the sampling n≥60, and C n represents the stability, that is, the confidence in the n time period. The closer the confidence is to 0, the more stable it is, and the more worthy of subsequent calculation trust.

[0082] When determining the reference time axis, the time axis of the data to be processed with the highest confidence can be directly used as the reference time axis, or the time axes of various types of data to be processed can be fitted according to the confidence levels of various types of data to obtain the reference time axis.

[0083] Step S306, determine the reference event in the reference data, and the associated event corresponding to the reference event in the target data to be processed. The target data to be processed is any other data to be processed except the reference data;

[0084] In the technical solution provided in step S306, the associated event refers to the event that occurs synchronously or with a high probability of synchronism when the reference event occurs. For example, since the roadside monitoring is a long-term data stream and has good confidence and basic time data. Therefore, the roadside monitoring can be used as the reference data. By analyzing the roadside monitoring data, a large number of vehicle starts and stops or other events can be used as the reference events to establish barriers. The associated events can be traffic light switching, running a yellow light, etc. Then, it can be checked whether there is a corresponding event barrier for the traffic light switching event in other data to be processed. If it exists, alignment is performed. In this way, in the scenario shown in Figure 5 as shown, at time D a2 , due to the existence of the barrier, it will be compressed from time t 6 and biased towards time t 5 .

[0085] Step S308: Determine the reference timestamp of the reference event based on the reference timeline, and correct the time stream of the target data to be processed based on the reference timestamp, the reference timeline, and the associated event, so that other data to be processed is aligned with the reference data. Among them, the difference between the timestamp of the associated event in the aligned data to be processed in the reference timeline and the reference timestamp is within a preset value range.

[0086] In some embodiments of the present application, the step of correcting the time stream of the target data to be processed based on the reference timestamp, the reference timeline, and the associated event includes: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference timeline according to the preset compression ratio and the duration of the correction process until the difference between the timestamp of the associated event in the target data to be processed relative to the reference timeline and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition. Among them, each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

[0087] As an alternative implementation, the step of correcting the data reporting time corresponding to each group of data in the target data to be processed according to the preset compression ratio and the duration of the correction process includes: correcting the data reporting time corresponding to each group of data in the reference timeline by using the following formula:

[0088]

[0089] where, t i represents the corrected data reporting time, t i-1 represents the previous data reporting time adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio. Through the above formula, the event stream of the target data to be processed can be compressed. Specifically, the data reporting times of each group of data in the target data to be processed can be corrected so that the time difference between the data reporting times of adjacent two groups of data becomes smaller and smaller. In addition, it can be seen from the above formula that as T continues to increase, the growth rate of t i continues to decline until the non-standard event is aligned or the water level collapses, where the water level collapse means that the correction stop condition is met.

[0090] In some embodiments of the present application, the correction stop condition expression is as follows:

[0091]

[0092] where, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold, generally less than 0.0001.

[0093] In some embodiments of the present application, when the preset compression ratio and the duration of the calibration process satisfy the calibration stop condition, the target data to be processed is placed in the compensation mechanism queue until there is a calibration process during a preset number of calibration processes such that the difference between the time stamp of the associated event in the target data to be processed and the reference time stamp with respect to the reference time axis is within a preset value range. Among them, different calibration processes correspond to different reference events, and different associated events correspond to different reference events; in the case where there is no calibration process such that the difference between the time stamp of the associated event in the target data to be processed and the reference time stamp with respect to the reference time axis is within a preset value range, a rollback operation is performed on the target data to be processed in the compensation mechanism queue, so that the data reporting time of each group of data in the target data to be processed is rolled back to the data reporting time before the calibration process is executed; and, the confidence level of the reference data is reduced, and the reference data is determined again from various types of data to be processed according to the confidence levels of various types of data to be processed.

[0094] Specifically, since compression takes effect when the water level collapses, it is necessary to put the compressed time stream data into the compensation mechanism queue until there is a successfully calibrated calibration process during the calibration process within the limited number ε. Among them, different calibration processes correspond to different reference events. If each calibration process satisfies the calibration stop condition, it will cause {t i ...t i …t i+ε} rollback.

[0095] In some embodiments of the present application, the process of time synchronization processing for multimodal data is as follows:

[0096] First step, when collecting multiple data sources, calculate the confidence levels of the data collected in each data source respectively.

[0097] Second step, preferentially select the data source with a higher confidence level as the reference time data.

[0098] Third step, perform event marking on the reference data. For different types of data, corresponding algorithms can be selected for marking according to different data types.

[0099] Fourth step, establish a barrier for non-reference data and wait for the data stream to continue flowing when it reaches the barrier.

[0100] Fifth step, when the time water level flows, the time flows normally; when the barrier is established, the time water level starts to flow slowly until the barrier event is established or the water level collapses.

[0101] In the sixth step, when the water level collapses, non-reference data may not generate or detect barrier corresponding events, and the time confidence level thereof drops to a preset value, such as -100.

[0102] By adopting the method of determining the initial timestamps corresponding to the data to be processed in the set of data to be processed, wherein different data to be processed have different data modalities; determining the confidence levels of the initial timestamps of the data to be processed, and determining a reference time axis based on the confidence levels, and determining reference data from the set of data to be processed, wherein the reference data is the data with the highest confidence level of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and the associated event corresponding to the reference event in the target data to be processed, the target data to be processed being any other data to be processed except the reference data; determining the reference timestamp of the reference event based on the reference time axis, and correcting the time stream of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that the other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range. By determining the reference data and the reference time axis according to the timestamp confidence level of the data to be processed, determining the reference event in the reference data and the associated event in the other data to be processed, and finally correcting the time stream of the other data to be processed according to the reference timestamp of the reference event, the associated event and the reference time axis, the purpose of time synchronization of multimodal data is achieved, thereby realizing the technical effect of improving the correlation between different modal data for subsequent further data mining and analysis, and further solving the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in the related art.

[0103] In addition, a time synchronization mechanism framework is also provided in the multimodal data synchronization method provided by the embodiments of the present application. A time synchronization algorithm abstraction module is introduced, including timestamp alignment, time synchronization algorithm, interpolation time method, etc., to ensure the time accuracy of multimodal data during joint analysis. The problem of out-of-sync time of multimodal data in the prior art is solved, and the accuracy and effectiveness of data analysis are enhanced. Moreover, through the watermark concept, the problem of time dimension differences caused by delays or other reasons in different data sources is solved, ensuring the time consistency of multimodal data during the acquisition and preprocessing process, and improving the data accuracy in application scenarios such as data analysis and AI model training.

[0104] The high-performance data processing engine provided in the multi-modal data synchronous acquisition system according to the embodiments of the present application supports selective preprocessing and analysis, and expands functions through a plug-in mechanism. Users can perform specific preprocessing modules according to requirements. This improves the real-time performance and flexibility of the system, meets the real-time requirements of multi-modal data processing, and overcomes the performance bottleneck of the existing system. In addition, a preprocessing module connected to the multi-modal data source is provided in the system, which supports custom filtering, standardization, mapping, and attribute addition plug-ins. This solves the problem of data processing isolation, speeds up data acquisition, and improves the scalability and flexibility of the system.

[0105] The embodiments of the present application provide a multi-modal data synchronization device. Figure 6 is a schematic structural diagram of the device. As can be seen from Figure 6 it, the device includes: a first processing module 60, configured to determine an initial timestamp corresponding to the data to be processed in the data set to be processed, where different data to be processed have different data modalities; a second processing module 62, configured to determine the confidence level of the initial timestamp of the data to be processed, and determine a reference time axis based on the confidence level, and determine reference data from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed; a third processing module 64, configured to determine a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed except the reference data; a fourth processing module 66, configured to determine a reference timestamp of the reference event based on the reference time axis, and correct the time stream of the target data to be processed according to the reference timestamp, the reference time axis, and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed on the reference time axis and the reference timestamp is within a preset value range.

[0106] In some embodiments of the present application, the step of the first processing module 60 determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: in the case where the data modality of the data to be processed is video data of the long-time flow data type, determining the average time passing speed of the first segment of video data and the second segment of video data in the data to be processed; determining a preset critical multiple and the number of windows in the data to be processed, where each window corresponds to a group of data in the data to be processed; and determining the initial timestamp of the data corresponding to each window according to the number of windows, the preset critical multiple, and the average time passing speed.

[0107] In some embodiments of the present application, the initial timestamp is determined by the following formula:

[0108]

[0109] Among them, t k represents the data reporting time corresponding to the first window, and t k+i represents the data reporting time corresponding to the i-th window. The data reporting times corresponding to each window are used to determine the initial timestamp, and μ represents a preset critical multiple. represents the average speed of time elapse.

[0110] In some embodiments of the present application, the steps for the first processing module 60 to determine the initial timestamp corresponding to the data to be processed in the set of data to be processed include: when the data modality of the data to be processed is a fixed change time event, determining the initial data reporting times corresponding to each group of data in the data to be processed, the time interval between adjacent groups of data, and a preset data reporting time compensation amount; determining the data reporting times corresponding to each group of data based on the initial data reporting times, the time interval between adjacent groups of data, and the preset data reporting time compensation amount, where the data reporting times corresponding to each group of data are used to determine the initial timestamp corresponding to the data to be processed.

[0111] In some embodiments of the present application, the steps for the fourth processing module 66 to correct the time stream of the target data to be processed based on the reference timestamp, the reference time axis, and the associated event include: determining a preset compression ratio; correcting the data reporting times corresponding to each group of data in the target data to be processed on the reference time axis according to the preset compression ratio and the duration of the correction process until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, where each group of data in the target data to be processed corresponds one-to-one with each data reporting time.

[0112] In some embodiments of the present application, the steps for the fourth processing module 66 to correct the data reporting times corresponding to each group of data in the target data to be processed on the reference time axis according to the preset compression ratio and the duration of the correction process include:

[0113] Correcting the data reporting times corresponding to each group of data on the reference time axis by using the following formula:

[0114]

[0115] Among them, t i represents the corrected data reporting time, t i-1 represents the previous data reporting time adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

[0116] In some embodiments of the present application, the calibration stop condition expression is as follows:

[0117]

[0118] Wherein, α represents a preset compression ratio, T represents a duration, and δ represents a preset threshold.

[0119] In some embodiments of the present application, the multimodal data synchronization device is further configured to: when the preset compression ratio and the duration of the calibration process meet the calibration stop condition, put the target data to be processed into the compensation mechanism queue until there is a calibration process in a preset number of calibration processes such that the difference between the timestamp of the associated event in the target data to be processed and the reference timestamp with respect to the reference time axis is within a preset value range, wherein different calibration processes correspond to different reference events, and different associated events correspond to different reference events; in the case where there is no calibration process such that the difference between the timestamp of the associated event in the target data to be processed and the reference timestamp with respect to the reference time axis is within a preset value range, perform a rollback operation on the target data to be processed in the compensation mechanism queue, so that the data reporting time of each group of data in the target data to be processed is rolled back to the data reporting time before the calibration process is executed; and, reduce the confidence level of the reference data, and re-determine the reference data from various data to be processed according to the confidence levels of various data to be processed.

[0120] It should be noted that each module in the above multimodal data synchronization device may be a program module (for example, a set of program instructions for implementing a specific function), or a hardware module. For the latter, it may be presented in the following forms, but not limited to: the forms of the above modules are all a processor, or the functions of the above modules are implemented by a processor.

[0121] The method embodiments provided by the embodiments of the present application may be executed on a mobile terminal, a computer terminal or a similar computing device. Figure 7 The hardware structure block diagram of a computer terminal (or mobile device) for implementing the multimodal data synchronization method is shown. As Figure 7As shown, the computer terminal 70 (or mobile device 70) may include one or more processors 702 (shown as 702a, 702b, ……, 702n in the figure) (the processor 702 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission module 706 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 7 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 70 may further include more or fewer components than Figure 7 shown in, or have a different configuration from Figure 7 that shown.

[0122] It should be noted that the above one or more processors 702 and / or other data processing circuits are generally referred to as "data processing circuits" in this article. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 70 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0123] The memory 704 may be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multi-modal data synchronization method in the embodiments of the present application. The processor 702 executes various functional applications and data processing by running the software programs and modules stored in the memory 704, that is, implements the above-mentioned multi-modal data synchronization method. The memory 704 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some instances, the memory 704 may further include a memory remotely set relative to the processor 702, and these remote memories may be connected to the computer terminal 70 through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The transmission device 706 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal 70. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 706 may be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0125] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computer terminal 70 (or mobile device).

[0126] According to an embodiment of the present application, a non-volatile storage medium is provided. A program is stored in the non-volatile storage medium. When the program runs, it controls the device where the non-volatile storage medium is located to execute the following multi-modal data synchronization method: determining an initial timestamp corresponding to the data to be processed in the data set to be processed, where different data to be processed have different data modalities; determining the confidence level of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence level, and determining reference data from the data set to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the data set to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed except the reference data; determining the reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed according to the reference timestamp, the reference time axis and the associated event, so that other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed and the reference timestamp in the reference time axis is within a preset value range.

[0127] According to an embodiment of the present application, an electronic device is provided, including: a memory and a processor, where the processor is configured to run a program stored in the memory. When the program runs, the following multimodal data synchronization method is executed: determining an initial timestamp corresponding to the data to be processed in the set of data to be processed, where different data to be processed have different data modalities; determining the confidence level of the initial timestamp of the data to be processed, and determining a reference timeline based on the confidence level, and determining reference data from the set of data to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed other than the reference data; determining the reference timestamp of the reference event based on the reference timeline, and correcting the time stream of the target data to be processed based on the reference timestamp, the reference timeline, and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed in the reference timeline and the reference timestamp is within a preset value range.

[0128] According to an embodiment of the present application, a computer program product is further provided, including a computer program that, when executed by a processor, implements the steps of the following multimodal data synchronization method: determining an initial timestamp corresponding to the data to be processed in the set of data to be processed, where different data to be processed have different data modalities; determining the confidence level of the initial timestamp of the data to be processed, and determining a reference timeline based on the confidence level, and determining reference data from the set of data to be processed, where the reference data is the data with the highest confidence level of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed other than the reference data; determining the reference timestamp of the reference event based on the reference timeline, and correcting the time stream of the target data to be processed based on the reference timestamp, the reference timeline, and the associated event, so that the other data to be processed is aligned with the reference data, where the difference between the timestamp of the associated event in the aligned data to be processed in the reference timeline and the reference timestamp is within a preset value range.

[0129] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0130] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0132] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the related technology, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0133] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A multimodal data synchronization method, characterized in that: include: Determine an initial timestamp corresponding to the data to be processed in the data set to be processed, wherein different data to be processed have different data modalities, and the initial timestamp is determined according to the data reporting time corresponding to the data to be processed; Determine the confidence of the initial timestamp of the data to be processed, determine a reference time axis according to the confidence, and determine reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; Determining a reference event in the reference data, and an associated event corresponding to the reference event in target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; Determine a reference timestamp of the reference event based on the reference time axis, and correct the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated events so that the other data to be processed are aligned with the reference data, including: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds to each data reporting time one by one, and the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

2. The multimodal data synchronization method according to claim 1, characterized in that: Correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process includes: The following formula is used to correct the data reporting time corresponding to each group of data in the reference time axis: Among them, t i Indicates the reporting time of the corrected data, t i-1 represents the last data reporting time immediately adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

3. The multimodal data synchronization method according to claim 1, characterized in that: The correction stop bar expression is as follows: Wherein, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold.

4. The multimodal data synchronization method according to claim 1, characterized in that: The method further comprises: In the case where the preset compression ratio and the duration of the correction process satisfy the correction stop condition, the target data to be processed is placed in the compensation mechanism queue until there is a correction process in a preset number of correction processes that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, wherein different correction processes correspond to different reference events, and different reference events correspond to different associated events; In the absence of a correction process that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, a rollback operation is performed on the target data to be processed in the compensation mechanism queue, so that the data reporting time corresponding to each group of data in the target data to be processed is rolled back to the data reporting time before the correction process is performed; and The confidence of the reference data is reduced, and the reference data is determined again from the various types of data to be processed according to the confidence of the various types of data to be processed.

5. The multimodal data synchronization method according to claim 1, characterized in that: Determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: In a case where the data modality of the data to be processed is video data of a long-time stream data type, determining an average speed of time lapse of a first segment of video data and a second segment of video data in the data to be processed; Determining a preset critical multiple and the number of windows in the data to be processed, wherein each window corresponds to a group of data in the data to be processed; The initial timestamp of the data corresponding to each window is determined according to the number of windows, the preset critical multiple and the average speed of time passage.

6. The multimodal data synchronization method according to claim 5, characterized in that: The initial timestamp is determined by the following formula: Among them, t k Indicates the data reporting time corresponding to the first window, t k+i represents the data reporting time corresponding to the i-th window, the data reporting time corresponding to each window is used to determine the initial timestamp, μ represents the preset critical multiple, Represents the average speed at which time passes.

7. The multimodal data synchronization method according to claim 1, characterized in that: Determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: In the case where the data modality of the data to be processed is a fixed change time event, determining the initial data reporting time corresponding to each group of data in the data to be processed, the time interval between each adjacent group of data, and the preset data reporting time compensation amount; The data reporting time corresponding to each group of data is determined based on the initial data reporting time, the time interval between each adjacent group of data, and the preset data reporting time compensation amount, wherein the data reporting time corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

8. A multimodal data synchronous collection system, characterized in that: It includes proxy server and real-time acquisition preprocessing framework, among which, The proxy server comprises a preprocessing module and a time synchronization module, wherein the preprocessing module is used to preprocess the data to be processed in the data set to be processed collected from the multimodal data source; The time synchronization module is used to determine the initial timestamp corresponding to the unprocessed data in the unprocessed data set, wherein different unprocessed data have different data modes, and the initial timestamp is determined according to the data reporting time corresponding to the unprocessed data; determine the confidence of the initial timestamp of the unprocessed data, and determine the reference time axis according to the confidence, and determine the reference data from the unprocessed data set, wherein the reference data is the data with the highest confidence of the initial timestamp in the unprocessed data set; determine the reference event in the reference data, and the associated event corresponding to the reference event in the target unprocessed data, wherein the target unprocessed data is any other unprocessed data except the reference data; determine the reference timestamp of the reference event according to the reference time axis, and determine the reference timestamp of the reference event according to the reference timestamp, and determine the reference time axis according to the reference timestamp, ... The time axis and the associated events correct the time flow of the target data to be processed so that the other data to be processed are aligned with the benchmark data, including: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the benchmark time axis according to the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated events in the target data to be processed relative to the benchmark time axis and the benchmark timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds to each data reporting time one by one, and the difference between the timestamp of the associated events in the aligned data to be processed in the benchmark time axis and the benchmark timestamp is within a preset value range; The real-time acquisition preprocessing framework includes a stream acquisition engine and a multimodal database, wherein the stream acquisition engine is used to store the aligned data into the corresponding multimodal database.

9. A multimodal data synchronization device, characterized in that: include: A first processing module is used to determine an initial timestamp corresponding to the to-be-processed data in the to-be-processed data set, wherein different to-be-processed data have different data modalities, and the initial timestamp is determined according to a data reporting time corresponding to the to-be-processed data; A second processing module is used to determine the confidence of the initial timestamp of the data to be processed, determine a reference time axis according to the confidence, and determine reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; A third processing module is used to determine a reference event in the reference data, and an associated event corresponding to the reference event in target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; The fourth processing module is used to determine the reference timestamp of the reference event according to the reference time axis, and to correct the time flow of the target data to be processed according to the reference timestamp, the reference time axis and the associated event, so that the other data to be processed are aligned with the reference data, including: determining a preset compression ratio; The data reporting time corresponding to each group of data in the target data to be processed in the reference time axis is corrected according to the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds one-to-one to each data reporting time, and the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the multimodal data synchronization method according to any one of claims 1 to 7.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the multimodal data synchronization method described in any one of claims 1 to 7 when running.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal data synchronization method described in any one of claims 1 to 7 are implemented.

Images