Multi-modal data synchronization method and device, collection system and electronic equipment

Abstract

The invention discloses a multi-modal data synchronization method and device, a collection system and electronic equipment. The method comprises the following steps: determining an initial timestamp corresponding to data to be processed in a data set to be processed; determining the confidence coefficient of the initial timestamp of the to-be-processed data, determining a reference time axis according to the confidence coefficient, and determining reference data from the to-be-processed data set; determining a reference event in the reference data and an associated event corresponding to the reference event in target to-be-processed data, wherein the target to-be-processed data is any other to-be-processed data except the reference data; and determining a reference timestamp of the reference event according to the reference time axis, and correcting the time stream of the target to-be-processed data according to the reference timestamp, the reference time axis and the associated event, so that other to-be-processed data are aligned with the reference data. According to the method and the device, the technical problem of low data validity caused by incapability of realizing time synchronization among multi-modal data in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular, to a multimodal data synchronization method, device, collection system and electronic equipment. Background Art

[0002] In the related art, when collecting multimodal data, data of different modes are usually collected separately, and there may be problems such as time deviation in the data source during the collection process, which causes the multimodal data that were originally synchronized in time to become out of sync, resulting in reduced data validity.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a multimodal data synchronization method, device, collection system and electronic device to at least solve the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in related technologies.

[0005] According to one aspect of an embodiment of the present application, a multimodal data synchronization method is provided, including: determining an initial timestamp corresponding to the data to be processed in a set of data to be processed, wherein different data to be processed have different data modalities; determining the confidence of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence, and determining reference data from the set of data to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determining a reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

[0006] Optionally, correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated events includes: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis based on the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

[0007] Optionally, correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process includes: correcting the data reporting time corresponding to each group of data in the reference time axis using the following formula:

[0008]

[0009] Among them, t i Indicates the reporting time of the corrected data, t i-1 represents the last data reporting time immediately adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

[0010] Optionally, the correction stop bar expression is as follows:

[0011]

[0012] Among them, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold.

[0013] Optionally, the method also includes: when a preset compression ratio and the duration of the correction process meet the correction stop condition, the target data to be processed is placed in a compensation mechanism queue until there is a correction process in a preset number of correction processes that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, wherein different correction processes correspond to different reference events, and different reference events correspond to different associated events; when there is no correction process that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, a rollback operation is performed on the target data to be processed in the compensation mechanism queue, so that the data reporting time corresponding to each group of data in the target data to be processed is rolled back to the data reporting time before the correction process is performed; and, the confidence of the reference data is reduced, and the reference data is determined again from each type of data to be processed based on the confidence of each type of data to be processed.

[0014] Optionally, determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: when the data modality of the data to be processed is video data of a long-time stream data type, determining the average speed of time passage of the first video data segment and the second video data segment in the data to be processed; determining a preset critical multiple and the number of windows in the data to be processed, wherein each window corresponds to a group of data in the data to be processed; and determining the initial timestamp of the data corresponding to each window based on the number of windows, the preset critical multiple and the average speed of time passage.

[0015] Optionally, the initial timestamp is determined by the following formula:

[0016]

[0017] Among them, t k Indicates the data reporting time corresponding to the first window, t k+i Indicates the data reporting time corresponding to the i-th window. The data reporting time corresponding to each window is used to determine the initial timestamp. μ indicates the preset critical multiple. Represents the average speed at which time passes.

[0018] Optionally, determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: when the data mode of the data to be processed is a fixed-changing time event, determining the initial data reporting moment corresponding to each group of data in the data to be processed, the time interval between each adjacent group of data, and a preset data reporting moment compensation amount; determining the data reporting moment corresponding to each group of data based on the initial data reporting moment, the time interval between each adjacent group of data, and the preset data reporting moment compensation amount, wherein the data reporting moment corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

[0019] According to another aspect of an embodiment of the present application, a multimodal data synchronization collection system is also provided, including a proxy server and a real-time collection preprocessing framework, wherein the proxy server includes a preprocessing module and a time synchronization module, wherein the preprocessing module is used to preprocess the unprocessed data in the unprocessed data set collected from the multimodal data source; the time synchronization module is used to determine the initial timestamp corresponding to the unprocessed data in the unprocessed data set, wherein different unprocessed data have different data modalities; determine the confidence of the initial timestamp of the unprocessed data, and determine the reference time axis based on the confidence, and determine the reference data from the unprocessed data set, wherein the reference data is the initial timestamp with the highest confidence in the unprocessed data set; high data; determining a benchmark event in the benchmark data, and an associated event corresponding to the benchmark event in the target data to be processed, the target data to be processed being any other data to be processed except the benchmark data; determining a benchmark timestamp of the benchmark event according to a benchmark time axis, and correcting the time flow of the target data to be processed according to the benchmark timestamp, the benchmark time axis and the associated event, so that other data to be processed are aligned with the benchmark data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the benchmark time axis and the benchmark timestamp is within a preset value range; the real-time acquisition preprocessing framework includes a stream acquisition engine and a multimodal database, wherein the stream acquisition engine is used to store the aligned data in the corresponding multimodal database.

[0020] According to another aspect of an embodiment of the present application, a multimodal data synchronization device is also provided, including: a first processing module, used to determine the initial timestamp corresponding to the data to be processed in the data set to be processed, wherein different data to be processed have different data modalities; a second processing module, used to determine the confidence of the initial timestamp of the data to be processed, and determine the reference time axis based on the confidence, and determine the reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; a third processing module, used to determine the reference event in the reference data, and the associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; a fourth processing module, used to determine the reference timestamp of the reference event based on the reference time axis, and correct the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

[0021] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a multimodal data synchronization method.

[0022] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the multimodal data synchronization method is executed when the program is run.

[0023] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the steps of the multimodal data synchronization method when executed by a processor.

[0024] In an embodiment of the present application, an initial timestamp corresponding to the data to be processed in a data set to be processed is determined, wherein different data to be processed have different data modes; the confidence of the initial timestamp of the data to be processed is determined, and a reference time axis is determined based on the confidence; and reference data is determined from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; a reference event is determined in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; a reference timestamp of the reference event is determined based on the reference time axis, and the time flow of the target data to be processed is corrected based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are Aligned with the benchmark data, wherein the difference between the timestamps of the associated events in the aligned data to be processed in the benchmark timeline and the benchmark timestamp is within a preset value range, by determining the benchmark data and the benchmark timeline according to the timestamp confidence of the data to be processed, and determining the benchmark events and associated events in other data to be processed from the benchmark data, and finally correcting the time flow of other data to be processed according to the benchmark timestamps of the benchmark events, the associated events and the benchmark timeline, thereby achieving the purpose of time synchronization of multimodal data, thereby achieving the technical effect of improving the correlation between different modal data for subsequent further data mining and analysis, and thus solving the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 It is a structural schematic diagram of a multimodal data synchronous acquisition system provided according to an embodiment of the present application;

[0027] Figure 2 is a structural schematic diagram of a preprocessing module provided according to an embodiment of the present application;

[0028] Figure 3 It is a flowchart of a multimodal data synchronization method provided according to an embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of reporting timing of multimodal data under an ideal state provided according to an embodiment of the present application;

[0030] Figure 5 is another schematic diagram of reporting timing of multimodal data provided according to an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of a multimodal data synchronization device provided according to an embodiment of the present application;

[0032] Figure 7 is a schematic diagram of the structure of a computer terminal (mobile terminal) provided according to an embodiment of the present application; DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0036] Multimodal Data: contains data in multiple forms, such as text, images, audio, and video.

[0037] Cluster Architecture: A system consisting of multiple servers or computers that work together to achieve high performance, high availability, and scalability.

[0038] Distributed System: System components are distributed on different network nodes and complete tasks together through communication and coordination.

[0039] Data Processing: Collect, organize, convert, and analyze data to extract useful information and support decision-making.

[0040] The multimodal data processing solutions in related technologies usually use fixed data types corresponding to fixed solutions, which have the following problems:

[0041] Data isolation: For example, image data is processed and stored using computer vision CV technology; audio data is transcribed and classified using speech recognition ASR technology, etc. However, the diversity of the same related data is difficult to analyze and identify in a single dimension, which can easily lead to loss of data relevance and information fragmentation.

[0042] Time asynchrony: In multimodal data processing, data generated by different data sources have time effects, but because they are often collected separately or the data sources have time deviations that lead to time asynchrony, this situation will reduce the effectiveness of data processing. For example, in intelligent monitoring, acquiring video and audio data at the same time can better predict the occurrence of special events and improve the accuracy of event recognition.

[0043] Centralized computing: Multimodal data often needs to be synchronized offline in batches, so centralized and unified processing is required. It is impossible to perform preliminary processing and synchronization at the source of the data; endpoint computing is required to reduce transmission delays and increase transmission speeds.

[0044] Due to the above-mentioned problems in related technologies, it is necessary to uniformly collect and process multimodal data in real time to achieve time synchronization between data, and then realize feature integration and joint modeling to improve analysis accuracy.

[0045] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0046] According to an embodiment of the present application, a method embodiment of a multimodal data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] The method provided in the embodiments of the present application can be used in Figure 1 The multi-modal data synchronization collection system shown in FIG. Figure 1 As shown, the system includes: a proxy server 10, a real-time acquisition preprocessing framework 12, wherein:

[0048] The proxy server 10 includes a preprocessing module 102 and a time synchronization module 104, wherein the preprocessing module is used to preprocess the data to be processed in the data set to be processed collected from the multimodal data source;

[0049] The time synchronization module 104 is used to determine the initial timestamp corresponding to the data to be processed in the data set to be processed, wherein different data to be processed have different data modes; determine the confidence of the initial timestamp of the data to be processed, and determine the reference time axis based on the confidence, and determine the reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; determine the reference event in the reference data, and the associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determine the reference timestamp of the reference event based on the reference time axis, and correct the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range;

[0050] The real-time acquisition preprocessing framework 12 includes a stream acquisition engine 122 and a multimodal database, wherein the stream acquisition engine 122 is used to store the aligned data into the corresponding multimodal database.

[0051] Specifically, Figure 1 As shown, it is assumed that data of different modes are collected in real time from multimodal data source A, multimodal data source B, and multimodal data source C, and multimodal data source A, multimodal data source B, and multimodal data source C are data in the same environment and logically have the same time and space attributes. In addition, it can be understood that the three multimodal data sources in this application are only for illustration and do not mean that the number of multimodal data sources can only be three. In the actual use of this solution, the number of multimodal data sources can be more or less.

[0052] After collecting multimodal data from various multimodal data sources, the preprocessing module 102 can be used to preprocess the multimodal data, and then the time synchronization module 104 can be used to perform time synchronization processing on the preprocessed multimodal data and format data synchronization information. Among them, the preprocessing module 102 uses methods such as screening available data information, standard formatting data information, etc. when preprocessing the multimodal data.

[0053] In some embodiments of the present application, the preprocessing module 102 is as follows: Figure 2As shown in the figure, various defined custom pre-processing plug-ins such as Filter, Standardization, Mapper, AddAttribute, etc. can be arranged. Users can select the corresponding data to execute the corresponding plug-in to personalize the collected data. Among them, Filter is used to customize the filtering data. The filtering conditions of multimodal data are not limited to value filtering, but may be filtering of data in other formats. Standardization is mainly to standardize the collected data. There may be different standardization methods for different data, such as video frame rate, picture color ratio, audio noise filtering, etc. Mapper mapping mainly performs the simplest marking operation or value class mapping, such as converting basic attributes into encoding; related small dimension table association, etc. AddAttribute is used to provide additional description of the data, and additional attributes are added based on the time, scene, location, etc. of the data.

[0054] In the above operating environment, the present application embodiment provides a multimodal data synchronization method, such as Figure 3 As shown, the method comprises the following steps:

[0055] Step S302, determining the initial timestamp corresponding to the data to be processed in the data set to be processed, wherein different data to be processed have different data modalities;

[0056] It should be noted that when collecting multimodal data, due to different data sizes, generation delays, collection delays or other reasons, there may be delays or differences in the data time described in each dimension, and even the time dimension may be lost or misaligned. Data time synchronization standardizes the time dimension attributes during unified collection.

[0057] Assume that in an intelligent traffic scenario, there are three types of data:

[0058] Traffic light: non-strictly timed round-robin data type, the data set is defined as Da.

[0059] Monitoring and taking photos: Event trigger data type, define the data set as Db.

[0060] Roadside monitoring: Long-term streaming data type, that is, continuous real-time video stream, the data set is defined as Dc.

[0061] Under ideal conditions, the reporting time of each group of data for these three types of data is as follows: Figure 4As shown, they should be reported synchronously. In a period of time, the traffic light IoT reports data at {t1, t5, t9}; the monitoring camera takes photos at {Db1}; and the roadside monitoring is {Dc}. Data is uploaded every moment, and it can be seen that {Da1, Db1, Dc1}, {Da2, Dc2}, and {Da3, Db2, Dc3} are all generated at the same time.

[0062] In actual scenarios, due to various interference factors, the actual reporting time of various data may be Figure 5 As shown, the actual reporting time of Da2 may change from t5 to t6. In order to solve this problem and achieve synchronization between data of different modes, the present application embodiment provides the following Figure 3 The multimodal data synchronization method shown.

[0063] In some embodiments of the present application, the concept of watermark is also introduced in the method provided in the embodiments of the present application. Watermark is a mechanism for processing data event time, which indicates the progress of event time stream. In the progress of multimodal data stream, since event time may be uneven due to differences in reporting processes, device time differences, etc., the framework needs a mechanism to measure the progress of event time and uniformly identify event occurrences in order to unify multimodal real-time data.

[0064] Unlike the watermark in related technologies, the watermark in related technologies is to solve the disorder problem when data arrives, and most of them are judged by the time the data is carried. The watermark introduced by the framework provided in the embodiment of the present application is more inclined to unify the data time problem between multi-modal and multi-data, and solve the problem of different physical times collected by different collectors and IoT devices.

[0065] The watermark introduced in the framework provided in the embodiment of the present application includes core components such as a watermark generator (WatermarkGenerator), a watermark confidence calculator (Watermark Confidence), a watermark synchronizer (Watermark SyncStrategy) and an event marker barrier (Eventmark Barrier).

[0066] Among them, the watermark generator is used to mark the acquired stream data of different modes one by one, and generate basic time data information TN, that is, the initial timestamp. Generally speaking, when generating a timestamp, a watermark can be directly generated according to the time carried by the data itself. For unordered data sources, the framework provided in the embodiment of the present application also supports custom heuristic plug-ins to generate corresponding watermarks. It should also be noted that the timestamps mentioned in the embodiments of the present application are timestamps corresponding to each group of data in various modal data, and the timestamp is used to indicate the reporting time of the group of data.

[0067] In the technical solution provided in step S302, when the data mode of the data to be processed is a fixed-changing time event, the method of determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: determining the initial data reporting time corresponding to each group of data in the data to be processed, the time interval between each adjacent group of data, and the preset data reporting time compensation amount; determining the data reporting time corresponding to each group of data based on the initial data reporting time, the time interval between each adjacent group of data, and the preset data reporting time compensation amount, wherein the data reporting time corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

[0068] For example, for a fixed-time event such as a traffic light, when there is data without time information in each set of uploaded data, a fixed-time-varying watermark generator can be used to generate an initial timestamp. The specific calculation formula is as follows:

[0069] T n =T0+n*t interval ±t compensation

[0070] In the above formula, T0 represents the reporting time of the earliest set of data in the current processing cycle, and T n is the reporting time of the n+1th group of data, t interval Represents the data interval between each set of data, t compensation is the preset confidence compensation amount.

[0071] In the technical solution provided in step S302, the step of determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: in the case where the data modality of the data to be processed is video data of a long-time stream data type, determining the average speed of time passage of the first video data and the second video data in the data to be processed; determining a preset critical multiple and the number of windows in the data to be processed, wherein each window corresponds to a group of data in the data to be processed; determining the initial timestamp of the data corresponding to each window based on the number of windows, the preset critical multiple and the average speed of time passage. The above-mentioned video data of the long-time stream data type includes roadside monitoring data, etc., and generally outputs time signals continuously. However, in some cases, when the data carries a small amount of time information or the confidence in the time information is low, the above-mentioned initial timestamp generation method can be used to generate a time series data interpolation of the video stream through a long monitoring stream.

[0072] As an optional implementation, the initial timestamp of the video data of the long-duration stream data type is determined by the following formula:

[0073]

[0074] Among them, t k Indicates the data reporting time corresponding to the first window, t k+i Indicates the data reporting time corresponding to the i-th window. The data reporting time corresponding to each window is used to determine the initial timestamp. μ indicates the preset critical multiple. It represents the average speed of time passing, and the calculation formula is as follows:

[0075]

[0076] In the above formula, Indicates the average flow rate of the first K groups of data in the data, Represents the average flow rate of the last K groups of data.

[0077] The critical multiple μ is 1, k is the number of windows, and the larger k is, the smoother the data is. k∈N + When the calculated average speed of time passage exceeds the critical limit, it will be normalized to the preset critical limit.

[0078] Step S304, determining the confidence of the initial timestamp of the data to be processed, and determining the reference time axis according to the confidence, and determining the reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed;

[0079] In the technical solution provided in step S304, after the watermark generator generates the watermark of the initial timestamp, the watermark confidence calculator will judge the confidence of the initial timestamp of each modal data. Generally speaking, the more stable and smooth the time data is, the higher the confidence; for missing time data, the confidence is lower. The above judgment rules can be configured through plug-ins to implement the judgment process, and specific data can also be constructed by different algorithms. For example, for video, the offset can be calculated by video frame data, and the calculation formula is as follows:

[0080]

[0081] In the above formula, FR k is the frame rate in k seconds, is the average frame rate of n seconds before k, generally sampling n≥60, C n It represents stability, that is, the confidence level in time period n. The closer the confidence level is to 0, the more stable it is and the more trustworthy it is for subsequent calculations.

[0082] When determining the reference time axis, the time axis of the data to be processed with the greatest confidence level may be directly used as the reference time axis, or the reference time axis may be obtained by fitting the time axes of various types of data to be processed according to the confidence levels of the various types of data to be processed.

[0083] Step S306, determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, where the target data to be processed is any other data to be processed except the reference data;

[0084] In the technical solution provided in step S306, an associated event refers to an event that occurs synchronously or with a high probability when a benchmark event occurs. For example, since roadside monitoring is a long-term data stream with good confidence and basic time data, roadside monitoring can be used as benchmark data. By analyzing roadside monitoring data, a large number of vehicle starts and stops or other events can be used as benchmark events to establish barriers. Associated events can be traffic light switching, running a yellow light, etc. Afterwards, you can check whether there is a corresponding event barrier for the traffic light switching event in other data to be processed. If so, align them. In this way, Figure 5 In the scenario shown, D a2 Due to the existence of the barrier, the time will be compressed from time t6 to time t5.

[0085] Step S308, determine the baseline timestamp of the baseline event based on the baseline time axis, and correct the time flow of the target data to be processed based on the baseline timestamp, the baseline time axis and the associated events, so that other data to be processed are aligned with the baseline data, wherein the difference between the timestamp of the associated events in the aligned data to be processed in the baseline time axis and the baseline timestamp is within a preset value range.

[0086] In some embodiments of the present application, the step of correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated events includes: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis based on the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

[0087] As an optional implementation, the step of correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process includes: correcting the data reporting time corresponding to each group of data in the reference time axis using the following formula:

[0088]

[0089] Among them, t i Indicates the reporting time of the corrected data, t i-1 represents the previous data reporting time immediately adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio. The above formula can be used to compress the event stream of the target data to be processed. Specifically, the reporting time of each group of data in the target data to be processed can be corrected so that the time difference between the reporting times of two adjacent groups of data becomes smaller and smaller. In addition, it can be seen from the above formula that as T increases, t i The growth rate continues to decrease until the non-standard events align or the water level collapses, where the water level collapse refers to meeting the correction stop condition.

[0090] In some embodiments of the present application, the correction stop bar expression is as follows:

[0091]

[0092] Among them, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold, which is generally less than 0.0001.

[0093] In some embodiments of the present application, when a preset compression ratio and the duration of the correction process satisfy a correction stop condition, the target data to be processed is placed in a compensation mechanism queue until there is a correction process in a preset number of correction processes in which the difference between the timestamp of an associated event in the target data to be processed relative to a reference time axis and a reference timestamp is within a preset value range, wherein different correction processes correspond to different reference events, and different reference events correspond to different associated events; in the absence of a correction process in which the difference between the timestamp of an associated event in the target data to be processed relative to a reference time axis and a reference timestamp is within a preset value range, a rollback operation is performed on the target data to be processed in the compensation mechanism queue, so that the data reporting time corresponding to each group of data in the target data to be processed is rolled back to the data reporting time before the correction process is executed; and, the confidence of the reference data is reduced, and the reference data is again determined from each type of data to be processed based on the confidence of each type of data to be processed.

[0094] Specifically, due to the collapse of the water level t i Compression will take effect, so the compressed time stream data needs to be put into the queue of the compensation mechanism until there is a successful correction process within the limited number of correction processes ε. Each correction process corresponds to a different benchmark event. If each correction process meets the correction stop condition, it will cause {t i …t i+ε}rollback.

[0095] In some embodiments of the present application, the process of performing time synchronization processing on multimodal data is as follows:

[0096] The first step is to calculate the confidence of the data collected from each data source when collecting multiple data sources.

[0097] The second step is to give priority to data sources with higher confidence as benchmark time data.

[0098] The third step is to mark the events of the benchmark data. For different types of data, corresponding algorithms can be selected for marking according to different data types.

[0099] The fourth step is to establish a barrier for non-benchmark data and wait for the data flow to reach the barrier before continuing the time data flow.

[0100] Step 5: When the time water level flows, time flows normally; when the barrier is established, the time water level begins to flow slowly until the barrier event is established or the water level collapses.

[0101] In the sixth step, when the water level collapses, the non-benchmark data may not generate or find the barrier corresponding event, and the corresponding time trust drops to the preset value, such as -100.

[0102] The method comprises the following steps: determining an initial timestamp corresponding to the data to be processed in a data set to be processed, wherein different data to be processed have different data modes; determining the confidence of the initial timestamp of the data to be processed, and determining a reference time axis according to the confidence; and determining reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determining a reference timestamp of the reference event according to the reference time axis, and correcting the time flow of the target data to be processed according to the reference timestamp, the reference time axis and the associated event, so that other data to be processed are consistent with the reference Data alignment, wherein the difference between the timestamps of associated events in the aligned data to be processed in the benchmark timeline and the benchmark timestamp is within a preset value range, by determining the benchmark data and the benchmark timeline according to the timestamp confidence of the data to be processed, and determining the benchmark events and associated events in other data to be processed from the benchmark data, and finally correcting the time flow of other data to be processed according to the benchmark timestamps of the benchmark events, the associated events and the benchmark timeline, thereby achieving the purpose of time synchronization of multimodal data, thereby achieving the technical effect of improving the correlation between different modal data for subsequent further data mining and analysis, and thus solving the technical problem of low data validity caused by the inability to achieve time synchronization between multimodal data in related technologies.

[0103] In addition, the multimodal data synchronization method provided in the embodiment of the present application also provides a time synchronization mechanism framework, which introduces a time synchronization algorithm abstract module, including timestamp alignment, time synchronization algorithm, interpolation time method, etc., to ensure the time accuracy of multimodal data during joint analysis. It solves the problem of time asynchrony of multimodal data in the prior art and enhances the accuracy and effectiveness of data analysis. Moreover, through the concept of watermark, it solves the problem of time dimension differences caused by delays or other reasons in different data sources, ensures the time consistency of multimodal data during the collection and preprocessing process, and improves the data accuracy in application scenarios such as data analysis and AI model training.

[0104] The multimodal data synchronous acquisition system provided in the embodiment of the present application also provides a high-performance data processing engine, supports selective preprocessing and analysis, and expands functions through a plug-in mechanism, so that users can perform specific preprocessing modules according to their needs. The real-time and flexibility of the system are improved, the real-time requirements of multimodal data processing are met, and the performance bottleneck of the existing system is overcome. In addition, a preprocessing module connected to a multimodal data source is also provided in the system, which supports custom filtering, standardization, mapping, and attribute addition plug-ins. The problem of isolation of data processing is solved, the speed of data acquisition is accelerated, and the scalability and flexibility of the system are improved.

[0105] The present application embodiment provides a multi-modal data synchronization device, Figure 6 is a schematic diagram of the structure of the device. Figure 6 As can be seen from the figure, the device includes: a first processing module 60, which is used to determine the initial timestamp corresponding to the to-be-processed data in the to-be-processed data set, wherein different to-be-processed data have different data modes; a second processing module 62, which is used to determine the confidence of the initial timestamp of the to-be-processed data, and determine the reference time axis according to the confidence, and determine the reference data from the to-be-processed data set, wherein the reference data is the data with the highest confidence of the initial timestamp in the to-be-processed data set; a third processing module 64, which is used to determine the reference event in the reference data, and the associated event corresponding to the reference event in the target to-be-processed data, wherein the target to-be-processed data is any other to-be-processed data except the reference data; a fourth processing module 66, which is used to determine the reference timestamp of the reference event according to the reference time axis, and correct the time flow of the target to-be-processed data according to the reference timestamp, the reference time axis and the associated event, so that the other to-be-processed data are aligned with the reference data, wherein the difference between the timestamp of the associated event in the to-be-processed data after alignment and the reference timestamp is within a preset value range.

[0106] In some embodiments of the present application, the step in which the first processing module 60 determines the initial timestamp corresponding to the data to be processed in the data set to be processed includes: when the data modality of the data to be processed is video data of a long-time stream data type, determining the average speed of time passage of the first video data segment and the second video data segment in the data to be processed; determining a preset critical multiple and the number of windows in the data to be processed, wherein each window corresponds to a group of data in the data to be processed; and determining the initial timestamp of the data corresponding to each window based on the number of windows, the preset critical multiple and the average speed of time passage.

[0107] In some embodiments of the present application, the initial timestamp is determined by the following formula:

[0108]

[0109] Among them, t k Indicates the data reporting time corresponding to the first window, t k+i Indicates the data reporting time corresponding to the i-th window. The data reporting time corresponding to each window is used to determine the initial timestamp. μ indicates the preset critical multiple. Represents the average speed at which time passes.

[0110] In some embodiments of the present application, the step of the first processing module 60 determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: when the data mode of the data to be processed is a fixed changing time event, determining the initial data reporting time corresponding to each group of data in the data to be processed, the time interval between each adjacent group of data, and the preset data reporting time compensation amount; determining the data reporting time corresponding to each group of data based on the initial data reporting time, the time interval between each adjacent group of data, and the preset data reporting time compensation amount, wherein the data reporting time corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

[0111] In some embodiments of the present application, the step of the fourth processing module 66 correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated events includes: determining a preset compression ratio; correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

[0112] In some embodiments of the present application, the fourth processing module 66 corrects the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process, including:

[0113] The following formula is used to correct the data reporting time corresponding to each group of data in the reference time axis:

[0114]

[0115] Among them, t i Indicates the reporting time of the corrected data, t i-1 represents the last data reporting time immediately adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

[0116] In some embodiments of the present application, the correction stop bar expression is as follows:

[0117]

[0118] Among them, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold.

[0119] In some embodiments of the present application, the multimodal data synchronization device is also used to: when a preset compression ratio and the duration of the correction process meet the correction stop condition, place the target data to be processed into a compensation mechanism queue until there is a correction process in a preset number of correction processes that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, wherein different correction processes correspond to different reference events, and different reference events correspond to different associated events; in the absence of a correction process that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, perform a rollback operation on the target data to be processed in the compensation mechanism queue, so that the data reporting time corresponding to each group of data in the target data to be processed is rolled back to the data reporting time before the correction process is executed; and, reduce the confidence of the reference data, and again determine the reference data from each type of data to be processed based on the confidence of each type of data to be processed.

[0120] It should be noted that the various modules in the above-mentioned multimodal data synchronization device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0121] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 7 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a multimodal data synchronization method. Figure 7As shown, the computer terminal 70 (or mobile device 70) may include one or more (shown as 702a, 702b, ..., 702n in the figure) processors 702 (the processor 702 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission module 706 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations shown.

[0122] It should be noted that the one or more processors 702 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 70 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0123] The memory 704 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimodal data synchronization method in the embodiment of the present application. The processor 702 executes various functional applications and data processing by running the software programs and modules stored in the memory 704, that is, the multimodal data synchronization method described above is realized. The memory 704 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 704 may further include a memory remotely arranged relative to the processor 702, and these remote memories may be connected to the computer terminal 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The transmission device 706 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 70. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 706 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0125] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 70 (or mobile device).

[0126] According to an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following multimodal data synchronization method: determining an initial timestamp corresponding to the data to be processed in a set of data to be processed, wherein different data to be processed have different data modalities; determining the confidence of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence, and determining reference data from the set of data to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determining a reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

[0127] According to an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor, the processor being used to run a program stored in the memory, wherein the following multimodal data synchronization method is executed when the program is run: determining an initial timestamp corresponding to the data to be processed in a set of data to be processed, wherein different data to be processed have different data modalities; determining the confidence of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence, and determining reference data from the set of data to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determining a reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

[0128] According to an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the following steps of a multimodal data synchronization method when executed by a processor: determining an initial timestamp corresponding to the data to be processed in a set of data to be processed, wherein different data to be processed have different data modalities; determining the confidence of the initial timestamp of the data to be processed, and determining a reference time axis based on the confidence, and determining reference data from the set of data to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the set of data to be processed; determining a reference event in the reference data, and an associated event corresponding to the reference event in the target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; determining a reference timestamp of the reference event based on the reference time axis, and correcting the time flow of the target data to be processed based on the reference timestamp, the reference time axis and the associated event, so that other data to be processed are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the reference time axis and the reference timestamp is within a preset value range.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0133] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal data synchronization method, characterized in that: include: Determine an initial timestamp corresponding to the data to be processed in the data set to be processed, wherein different data to be processed have different data modalities; Determine the confidence of the initial timestamp of the data to be processed, determine a reference time axis according to the confidence, and determine reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; Determining a reference event in the reference data, and an associated event corresponding to the reference event in target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; Determine a baseline timestamp of the baseline event based on the baseline time axis, and correct the time flow of the target data to be processed based on the baseline timestamp, the baseline time axis and the associated event, so that other data to be processed are aligned with the baseline data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the baseline time axis and the baseline timestamp is within a preset value range.

2. The multimodal data synchronization method according to claim 1, characterized in that: Correcting the time flow of the target data to be processed according to the reference timestamp, the reference time axis and the associated event includes: Determine the preset compression ratio; The data reporting time corresponding to each group of data in the target data to be processed in the reference time axis is corrected according to the preset compression ratio and the duration of the correction process, until the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp is within a preset value range, or the preset compression ratio and the duration of the correction process meet the correction stop condition, wherein each group of data in the target data to be processed corresponds one-to-one to each data reporting time.

3. The multimodal data synchronization method according to claim 2, characterized in that: Correcting the data reporting time corresponding to each group of data in the target data to be processed in the reference time axis according to the preset compression ratio and the duration of the correction process includes: The following formula is used to correct the data reporting time corresponding to each group of data in the reference time axis: Among them, t i Indicates the reporting time of the corrected data, t i-1 represents the last data reporting time immediately adjacent to the corrected data reporting time, T represents the duration, and α represents the preset compression ratio.

4. The multimodal data synchronization method according to claim 2, characterized in that: The correction stop bar expression is as follows: Wherein, α represents the preset compression ratio, T represents the duration, and δ represents the preset threshold.

5. The multimodal data synchronization method according to claim 2, characterized in that: The method further comprises: In the case where the preset compression ratio and the duration of the correction process satisfy the correction stop condition, the target data to be processed is placed in the compensation mechanism queue until there is a correction process in a preset number of correction processes that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, wherein different correction processes correspond to different reference events, and different reference events correspond to different associated events; In the absence of a correction process that makes the difference between the timestamp of the associated event in the target data to be processed relative to the reference time axis and the reference timestamp within a preset value range, a rollback operation is performed on the target data to be processed in the compensation mechanism queue, so that the data reporting time corresponding to each group of data in the target data to be processed is rolled back to the data reporting time before the correction process is performed; and The confidence of the reference data is reduced, and the reference data is determined again from the various types of data to be processed according to the confidence of the various types of data to be processed.

6. The multimodal data synchronization method according to claim 1, characterized in that: Determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: In a case where the data modality of the data to be processed is video data of a long-time stream data type, determining an average speed of time lapse of a first segment of video data and a second segment of video data in the data to be processed; Determining a preset critical multiple and the number of windows in the data to be processed, wherein each window corresponds to a group of data in the data to be processed; The initial timestamp of the data corresponding to each window is determined according to the number of windows, the preset critical multiple and the average speed of time passage.

7. The multimodal data synchronization method according to claim 6, characterized in that: The initial timestamp is determined by the following formula: Among them, t k Indicates the data reporting time corresponding to the first window, t k+i represents the data reporting time corresponding to the i-th window, the data reporting time corresponding to each window is used to determine the initial timestamp, μ represents the preset critical multiple, Represents the average speed at which time passes.

8. The multimodal data synchronization method according to claim 1, characterized in that: Determining the initial timestamp corresponding to the data to be processed in the data set to be processed includes: In the case where the data modality of the data to be processed is a fixed change time event, determining the initial data reporting time corresponding to each group of data in the data to be processed, the time interval between each adjacent group of data, and the preset data reporting time compensation amount; The data reporting time corresponding to each group of data is determined based on the initial data reporting time, the time interval between each adjacent group of data, and the preset data reporting time compensation amount, wherein the data reporting time corresponding to each group of data is used to determine the initial timestamp corresponding to the data to be processed.

9. A multimodal data synchronous collection system, characterized in that: It includes proxy server and real-time acquisition preprocessing framework, among which, The proxy server comprises a preprocessing module and a time synchronization module, wherein the preprocessing module is used to preprocess the data to be processed in the data set to be processed collected from the multimodal data source; The time synchronization module is used to determine the initial timestamp corresponding to the to-be-processed data in the to-be-processed data set, wherein different to-be-processed data have different data modes; determine the confidence of the initial timestamp of the to-be-processed data, and determine the reference time axis based on the confidence, and determine the reference data from the to-be-processed data set, wherein the reference data is the data with the highest confidence of the initial timestamp in the to-be-processed data set; determine the reference event in the reference data, and the associated event corresponding to the reference event in the target to-be-processed data, wherein the target to-be-processed data is any other to-be-processed data except the reference data; determine the reference timestamp of the reference event based on the reference time axis, and correct the time flow of the target to-be-processed data based on the reference timestamp, the reference time axis and the associated event, so that the other to-be-processed data are aligned with the reference data, wherein the difference between the timestamp of the associated event in the aligned to-be-processed data in the reference time axis and the reference timestamp is within a preset value range; The real-time acquisition preprocessing framework includes a stream acquisition engine and a multimodal database, wherein the stream acquisition engine is used to store the aligned data into the corresponding multimodal database.

10. A multimodal data synchronization device, characterized in that: include: A first processing module is used to determine an initial timestamp corresponding to the to-be-processed data in the to-be-processed data set, wherein different to-be-processed data have different data modalities; A second processing module is used to determine the confidence of the initial timestamp of the data to be processed, determine a reference time axis according to the confidence, and determine reference data from the data set to be processed, wherein the reference data is the data with the highest confidence of the initial timestamp in the data set to be processed; A third processing module is used to determine a reference event in the reference data, and an associated event corresponding to the reference event in target data to be processed, wherein the target data to be processed is any other data to be processed except the reference data; A fourth processing module is used to determine a baseline timestamp of the baseline event based on the baseline time axis, and to correct the time flow of the target data to be processed based on the baseline timestamp, the baseline time axis and the associated event, so that other data to be processed are aligned with the baseline data, wherein the difference between the timestamp of the associated event in the aligned data to be processed in the baseline time axis and the baseline timestamp is within a preset value range.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the multimodal data synchronization method according to any one of claims 1 to 8.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program, when running, executes the multimodal data synchronization method described in any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal data synchronization method described in any one of claims 1 to 8 are implemented.

Images