A multimodal data time synchronization and labeling and edge inference method and system

Abstract

The application provides a multimodal data time synchronization, labeling and edge inference method and system, and relates to the technical field of cognitive evaluation. The multimodal data time synchronization, labeling and edge inference method comprises the following steps: collecting multimodal data; using a global logical clock + network time protocol + sliding window drift correction method to align the time of the multimodal data; identifying three dimensions of a task phase, a behavior event and an emotional state, and outputting three types of annotation labels; deploying an edge inference model, and automatically triggering cloud review when the confidence is lower than a threshold; generating a unique timestamp and a task identification for each collected frame, and establishing a unified log database in the cloud. The application combines a time sequence correction algorithm and AI model annotation logic in depth, realizes millisecond-level time alignment and intelligent labeling of multimodal data, and solves the technical problems of asynchronous data, high delay and large amount of manual labeling in the cognitive evaluation and rehabilitation training scene.

Description

Technical Field

[0001] This invention relates to the field of cognitive assessment technology, and in particular to a method and system for multimodal data time synchronization and annotation and edge reasoning. Background Technology

[0002] During cognitive assessment and rehabilitation training, subjects typically need to perform multiple interactive tasks simultaneously, such as verbal responses, hand movements, drawing tasks, touch operations, and facial expression responses. The system needs to simultaneously collect multi-source modalities such as voice, video, motion trajectories, and touch data, and complete time alignment within milliseconds to ensure accurate identification of behavioral events.

[0003] In traditional scenarios, multimodal data often relies on post-event manual annotation or offline analysis, resulting in significant time skew and a lack of a unified time series benchmark. This leads to inaccurate event localization, low annotation efficiency, and difficulty in providing real-time feedback. This results in a series of problems:

[0004] (1) Time asynchrony: The sampling frequency and time reference of each sensor are inconsistent, resulting in a shift of more than milliseconds in the collected data, which affects the alignment of task events and the accuracy of AI feature extraction.

[0005] (2) Low efficiency of manual annotation: The amount of multimodal data is large, and the cost of manually marking task stages, emotional states or erroneous behaviors frame by frame is high and highly subjective.

[0006] (3) High latency of cloud inference: Existing systems usually need to upload raw data to the cloud for unified analysis, which leads to feedback delay and is not suitable for real-time training and instant evaluation.

[0007] (4) Lack of a unified time-series audit and traceability mechanism: In multi-device collaborative data collection scenarios, it is impossible to quickly track the time-series accuracy and source integrity of data.

[0008] Therefore, there is an urgent need for a unified system architecture that can realize multimodal data time synchronization, intelligent annotation, edge reasoning and tracing to improve the real-time performance and reliability of data processing. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a multimodal data time synchronization and annotation method and system for edge inference. It is applicable to various intelligent medical and behavioral assessment scenarios with strong multimodal perception requirements and high sensitivity to time accuracy and behavioral event localization, including applications such as cognitive screening, psychological assessment, neurological function monitoring, rehabilitation training assessment, and intelligent companionship devices.

[0010] This invention achieves an overall framework of "synchronization upon data acquisition, annotation upon recognition, and real-time decision-making at the edge" through hardware-triggered synchronization, logical clock fusion, AI automatic annotation, and edge inference mechanisms. It is applicable to smart hospital terminals, community digital health stations, elderly care assessment equipment, mobile rehabilitation equipment, and home-based intelligent companionship systems. This system ensures that multimodal data has a highly consistent time reference during the acquisition phase and possesses a traceable, low-latency, and scalable structure, making it a crucial foundational capability for intelligent medical devices.

[0011] In a first aspect, the present invention provides a method for multimodal data time synchronization, annotation, and edge inference, comprising the following steps:

[0012] S1. Multimodal data acquisition is carried out during the cognitive assessment process; wherein, the data collected by the same sensor within one hardware sampling period is set as a collection frame, and the data modalities include: audio stream, video stream, touch and handwriting trajectory, motion / skeletal trajectory, and task event stream;

[0013] S2. Time Synchronization and Drift Correction: Based on the main control MCU reference sampling signal to synchronize the hardware sampling period, the method of global logic clock + network time protocol + sliding window drift correction is adopted to align the time of multimodal data and control the error within 2ms.

[0014] S3, AI Intelligent Annotation: Unifies the encoding of multimodal data into feature vectors; identifies three dimensions: task stage, behavioral event, and emotional state, and outputs three types of annotation labels; performs smoothing and conflict detection on the annotation results, and filters out annotation conflicts or abnormal annotations.

[0015] S4, Edge Inference and Cloud-Edge Collaboration: Lightweight deployment of AI models on the device to achieve real-time judgment and label generation, and automatic cloud review when the confidence level is lower than the threshold;

[0016] S5. Log Auditing and Traceability: Generate a unique timestamp and task identifier for each collection frame, establish a unified log database in the cloud, and record the collection, inference, storage and AI annotation information of multimodal data to achieve full-process data traceability, playback and quality assessment.

[0017] As a further improvement of the present invention, the data parameters of step S1 include:

[0018] Video sampling frame rate ≥ 25fps, resolution ≥ 1080p;

[0019] Audio sampling rate ≥ 16kHz / 24bit, signal-to-noise ratio ≥ 60dB;

[0020] Touch sampling frequency ≥ 60Hz, trajectory accuracy ≤ 2mm;

[0021] The motion detection module uses a high-precision skeleton recognition model with inter-frame drift ≤ 5px.

[0022] As a further improvement of the present invention, the specific method of step S2 includes:

[0023] Correct the local time:

[0024] ;

[0025] in:

[0026] T global To standardize time, T local K(t) is the local time obtained by the sensor, K(t) is the dynamic compensation parameter calculated based on the sliding window drift, and ΔNTP is the network time deviation.

[0027] The calculation method for dynamic compensation parameters is as follows:

[0028] ;

[0029] Among them, E avg K(t+1) is the average drift error per unit time within the sliding window, K(t+1) is the corrected dynamic compensation parameter, K(t) is the current dynamic compensation parameter, and α is the dynamic learning rate.

[0030] As a further improvement to the present invention, E avg The calculation methods include simplification methods and robust methods:

[0031] Simplified method: ;

[0032] in, For the current alignment residual, It is the local time increment of the sampling point corresponding to the i-th residual sample within the sliding window from the anchor point, where the anchor point is the drift correction reference time formed based on the time synchronization event;

[0033] Robust method for calculating the slope of linear regression: ;

[0034] Where x is x i The set, e represents the residual. A set of.

[0035] As a further improvement of the present invention, when the synchronization error E > 2ms, automatic drift retraining is triggered to perform drift correction; wherein...

[0036] ;T sensor For sensor time.

[0037] As a further improvement of the present invention, the specific steps of step S3 include:

[0038] S31. Multimodal feature fusion: The Transformer / Time-series Fusion vector encoding model is used to encode visual keyframe features, speech spectrum features, motion trajectory and eye movement sequence features, and touch trajectory features through cross-modal attention and temporal position encoding, and output a fixed-dimensional feature vector.

[0039] S32. Annotation Task Output: Through classifier identification, output three types of annotation labels: task stage labels, including preparation, execution, completion, and hesitation; behavioral event labels, including error, repetition, slowness, and incompleteness; and emotional state labels, including tension, joy, focus, and confusion. The annotation labels are bound to the original data in a JSON structure.

[0040] S33. Confidence Smoothing and Conflict Detection: Voting smoothing is performed using a time window ΔT to smooth the annotation results and filter out jittery labels.

[0041] ;

[0042] Where confidence' is the confidence level after smoothing. i Here, N represents the confidence level for each category, and N is the number of categories.

[0043] As a further improvement of the present invention, step S33 includes:

[0044] Events of the same type are merged according to annotation ID. Fragments and smooth confidence scores for each type of event are calculated based on a sliding window step of 50ms. Then, conflict detection is performed. The specific method is as follows:

[0045] For conflict event labels, retain labels with higher confidence, or set retention priority according to preset event types; when similar labels from different source modalities conflict within 100ms, retain the merged / majority side; for long-term conflict situations, output a "conflict" flag and reduce its weight.

[0046] As a further improvement of the present invention, the specific steps of step S4 include:

[0047] S41, Lightweight Edge Model Deployment: Using 8-bit quantization and distillation technology, the original AI model is compressed by 70% and deployed to NPU, DSP, and CPU devices for speech keyword recognition, facial emotion recognition, and behavior detection;

[0048] S42, Edge Decision Reasoning: When the confidence level of the device-side recognition result is not less than 0.85, output a valid label; otherwise, mark the data segment as pending cloud verification.

[0049] S43, Cloud-Edge Collaboration: Upload the device-side inference results and timestamps to the cloud, where the main model verifies their consistency.

[0050] As a further improvement of the present invention, in step S43, the main model verifies consistency using a consistency index C:

[0051] ;

[0052] Among them, Score edge Score is the post-calibration confidence of edge nodes for the same event within the alignment time window. cloud The post-calibration confidence of the cloud-based master model for the same event within the same aligned time window;

[0053] If C < 0.6, the edge model is automatically marked as a deviation state.

[0054] As a further improvement of the present invention, step S5 includes: backtracking and auditing are based on a unified log database in the cloud, with real-time upload of summaries at the edge, and timeline indexes established after aggregation in the cloud to achieve multimodal retrieval, alignment, playback and auditing; local caching is performed in offline scenarios, and re-uploaded and merged with cloud logs after networking.

[0055] As a further improvement to the present invention, a data structure is constructed, including:

[0056] In step S1, metadata is added to each acquisition frame:

[0057] {timestamp_local, sensorID, modalityType, frameIndex, deviceID};

[0058] Where timestamp_local is the local timestamp, sensorID is the sensor identifier, modalityType is the modality type, frameIndex is the frame number, and deviceID is the device ID;

[0059] In step S3, the data for outputting the annotation labels is as follows:

[0060] {timestamp_global, eventType, eventConfidence, modalitySource,sensorID, annotationID};

[0061] Among them, timestamp_global is the unified timestamp, eventType is the event type, eventConfidence is the recognition confidence level, modalitySource is the source modality, sensorID is the sensor identifier, and annotationID is the annotation ID;

[0062] In step S5, the data recorded in each frame is as follows:

[0063] {taskID, sessionToken, timestamp_global, sensorID, deviceID, syncError, modelVersion, edgeDecisionFlag, , confidence};

[0064] Among them, taskID is the task ID, sessionToken is the session identifier, timestamp_global is the unified timestamp, sensorID is the sensor identifier, deviceID is the device ID, syncError is the time error, modelVersion is the model version, edgeDecisionFlag is the edge decision flag, edgeTag is the decision path label, and confidence is the decision confidence level.

[0065] Secondly, this invention provides a multimodal data time synchronization and annotation and edge inference system, the system comprising:

[0066] The multimodal acquisition terminal specifically includes: a voice acquisition module, an eye-tracking acquisition module, a video acquisition module, and a motion / touch acquisition module;

[0067] The device-side / edge computing node specifically includes: a time synchronization and drift correction controller, a multimodal buffer and alignment module, an AI intelligent annotation module, and a log and data packaging module;

[0068] The cloud-based central system specifically includes: a cloud-based master model inference and verification module, a time base synchronization module, a log and traceability database, and a data management and reporting system;

[0069] Among them, the cloud-based main model inference and verification module uses high-precision AI model analysis, and its results are used to update the task identifier and traceability information generated by the log and data packaging module; the time base synchronization module uses an NTP / PTP global clock to perform time synchronization correction on the time synchronization and drift correction controller.

[0070] Thirdly, the present invention provides an embodiment of a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0071] Fourthly, the present invention provides a computer program product embodiment, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0072] The core innovation of this invention lies in proposing a unified time synchronization and annotation mechanism for multimodal sensors. This mechanism, through a four-layer architecture of "logic clock + hardware-software collaborative drift correction + intelligent annotation algorithm + edge inference control," achieves high-precision time alignment, automated annotation, and integrated cloud-edge collaborative inference processing of multimodal data. Traditional systems can only perform single-channel time correction or post-processing data alignment, while this invention deeply integrates the time correction algorithm with AI model annotation logic, enabling millisecond-level time alignment and intelligent annotation of multimodal data such as voice, video, motion, touch, and eye tracking during the acquisition stage. Low-latency recognition and feedback are completed through edge computing nodes, fundamentally solving the technical challenges of "asynchronous data," "high latency," and "large amount of manual annotation" in cognitive assessment and rehabilitation training scenarios.

[0073] This invention is particularly suitable for applications sensitive to temporal precision and reaction time, such as cognitive assessment, brain function evaluation, and psychological rehabilitation. It achieves an organic integration of time synchronization, intelligent annotation, and edge reasoning; specific technical effects include:

[0074] A unified time reference and high-precision alignment for data collected by various sensors;

[0075] Utilize AI models to automate event detection and tagging;

[0076] Low-latency initial inference is performed on the device via an edge computing module;

[0077] Build a cloud-edge collaborative architecture to achieve efficient data transmission, unified management, and traceability recording. Attached Figure Description

[0078] Figure 1 This is a flowchart of a method for multimodal data time synchronization, annotation, and edge inference disclosed in this invention.

[0079] Figure 2 This is a flowchart of the time synchronization mechanism algorithm.

[0080] Figure 3 This is a structural diagram of the AI ​​intelligent annotation module.

[0081] Figure 4 This is a schematic diagram of cloud-edge collaborative data flow. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Steps S1, S2… in the described embodiments of the present invention do not limit the scope of execution of the present invention; the various models, simulation environments, and software described in the present invention are not considered as the only limiting methods of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0083] In this invention, computer device / equipment / system refers to a related entity applied to a computer, such as hardware, a combination of hardware and software, software, or software in execution. More specifically, for example, software includes, but is not limited to, a process running on a processor, a processor, an object, executable software, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be software. One or more software programs may be in an execution process and / or thread, and the software may be localized on one computer and / or distributed across two or more computers, and may be run on various computer-readable media.

[0084] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0085] In a first aspect, the present invention provides an embodiment of a multimodal data time synchronization and annotation and edge inference method, such as... Figure 1 As shown, the specific process can be as follows:

[0086] S1, Multimodal data acquisition;

[0087] The modalities collected by the system include, but are not limited to:

[0088] Audio stream (voice response), video stream (facial expressions, eye movements, motion), touch and handwriting trajectory, motion / skeletal trajectory, task event stream (start / end / number of prompts).

[0089] Technical requirements and parameters:

[0090] Video sampling frame rate ≥ 25fps, resolution ≥ 1080p;

[0091] Audio sampling rate ≥ 16kHz / 24bit, signal-to-noise ratio ≥ 60dB;

[0092] Touch sampling frequency ≥ 60Hz, trajectory accuracy ≤ 2mm;

[0093] The motion detection module uses a high-precision skeleton recognition model with inter-frame drift ≤ 5px.

[0094] Data Structures:

[0095] Add metadata to each acquisition frame:

[0096] {timestamp_local, sensorID, modalityType, frameIndex, deviceID};

[0097] The data types in the metadata include: local timestamp, sensor identifier, modality type, frame sequence number, and device ID.

[0098] The acquisition frame refers to a recording unit generated by the same sensor within a fixed sampling period under the drive of a unified acquisition clock, and written as the smallest atomic unit of the data stream; it contains the original / encoded data payload of that period and the necessary metadata {timestamp_local, sensorID, modalityType, frameIndex, deviceID}. For the same modality, the timestamp is monotonically undecreasing and the frame sequence number is globally unique.

[0099] In one embodiment of the present invention, the metadata collection content and description are shown in Table 1.

[0100] Table 1:

[0101]

[0102] S2, Time Synchronization and Drift Correction;

[0103] like Figure 2 As shown, multimodal data needs to achieve millisecond-level synchronization accuracy (≤2ms) to accurately identify the relationship between action and speech sequence.

[0104] S21. Logical clock synchronization (Hybrid Timestamp Protocol).

[0105] The global logical clock is maintained using edge nodes, and a unified timestamp is generated by combining the Network Time Protocol (NTP) with the local clock drift model.

[0106] Specifically, a local hardware timer (clock_local), a cloud-based reference time (T_cloud), and a drift correction model (offset(t)) are used. Each sensor's data frame is accompanied by a local timestamp and an offset correction value Δt, i.e., K(t).

[0107] Time correction formula:

[0108] ;

[0109] in:

[0110] T global To standardize time, T local Where is the local time, K(t) is the dynamic compensation parameter calculated based on the sliding window drift, and ΔNTP is the network time deviation, which is automatically estimated and corrected by the system.

[0111] Synchronization error:

[0112] ;

[0113] T sensor For sensor time, when E > 2ms, automatic drift retraining is triggered to perform drift correction.

[0114] S22, Hardware-triggered synchronization;

[0115] The camera module, voice acquisition module, and touch control module share the hardware synchronization trigger pulse reference signal issued by the main controller; specifically:

[0116] The main control MCU provides the reference sampling signal, and all modules start sampling and enter the sampling cycle simultaneously.

[0117] Hardware synchronization error: ≤1ms.

[0118] S23, Drift correction;

[0119] The sliding window method is adopted: the offset of the past N frames is recorded using the sliding window method, and the synchronization parameter K(t) is dynamically updated.

[0120] ;

[0121] Among them, E avg K(t+1) is the average drift error per unit time within the sliding window, K(t+1) is the corrected dynamic compensation parameter, K(t) is the current dynamic compensation parameter, and α is the dynamic learning rate.

[0122] Preferably, the present invention provides a calculation of E avg Two methods:

[0123] (1) Simplification method: , ( =Current alignment residual, =The local time increment of the sampling point corresponding to the i-th residual sample within the sliding window from the anchor point, where the anchor point is the reference time used for the current drift correction, consisting of a pair of timestamps formed by the same time synchronization event and an optional sequence number);

[0124] (2) Robust approach: for With residual Calculate the slope of the linear regression: .

[0125] Based on the synchronization parameter K(t), the local time is corrected to achieve millisecond-level time alignment. The correction formula is as follows:

[0126] T' = T local + K(t);

[0127] Compared to the traditional "single synchronization" mode, the synchronization parameter correction mechanism can maintain dynamic correction accuracy in long-term tasks, with the error controlled within ±2ms.

[0128] S3, AI intelligent annotation, such as Figure 3 As shown;

[0129] S31, Multimodal feature fusion;

[0130] Using a Transformer / Time-series Fusion vector encoding model, cross-modal attention and temporal position encoding are performed on aligned visual keyframe features (CNN + Pose Estimation), speech spectral features (MFCC, Mel spectrum), motion trajectory and eye movement sequence features, and touch trajectory features, outputting a fixed-dimensional feature vector Z:

[0131] ;

[0132] Among them, F fusion This is a vector encoding model, where video is a sequence of visual keyframe features. Features such as face and hand are extracted using CNN / ViT+ROI, and audio is a sequence of speech spectral features. pose / skeletal keypoint sequence tace is a sequence of touch / handwriting trajectories. This includes coordinates, velocity, acceleration, pressure sensitivity, etc.

[0133] S32, Comment out the task output;

[0134] The classifier identifies and outputs three types of labels:

[0135] Task stage tags (Preparation, Execution, Completion, Hesitation).

[0136] Behavioral event labels (error, repetition, slow, incomplete).

[0137] Emotional state labels (tension, pleasure, focus, confusion);

[0138] Output data structure:

[0139] {timestamp_global, eventType, eventConfidence, modalitySource,sensorID, annotationID};

[0140] The output data types include: uniform timestamp, event type, identification confidence, source modality, sensor identifier, and annotation ID;

[0141] In one embodiment of the present invention, examples and descriptions of the output data are shown in Table 2.

[0142] Table 2:

[0143]

[0144] The annotation results are bound to the original data in a JSON structure, which supports subsequent manual review or model retraining.

[0145] S33, Confidence Smoothing and Collision Detection;

[0146] The AI ​​output is smoothed by using a time window ΔT (200ms) to filter out jittery labels.

[0147] ;

[0148] Where confidence' is the confidence level after smoothing. i Here, N represents the confidence level for each category, and N is the number of categories.

[0149] The system performs consistency checks on the annotation results. In cases of annotation conflicts or abnormal annotations (such as identifying "preparation action" in the "execution phase"), the system prompts for manual review or automatically backtracks to the previous level model.

[0150] This step performs time smoothing on the confidence scores; the labels are obtained by taking the maximum of the smoothed confidence scores for each category (plus the threshold / hysteresis), which is equivalent to smoothing the annotation results through voting. Specific steps include:

[0151] S331. Aggregation and Deduplication: Merge the same events by annotationID and unify them into timestamp_global.

[0152] S332. Smooth the code by namespace:

[0153] Task stages (stage.*) are grouped together;

[0154] Multiple action events (action.*) can coexist, but those of the same type are mutually exclusive;

[0155] Emotional states (affect.*) can be categorized using either multi-label or single-label strategies.

[0156] S333, Sliding window calculation: Calculate for each type of event in 50ms increments.

[0157] S334, Label Output: Form segments (tStart / tEnd) with smooth confidence using hysteresis threshold / minimum duration.

[0158] S335, Collision Detection:

[0159] Decision set with mutual exclusion: allows unique activity labels; when overlaps, higher confidence is preserved or decisions are made according to priority (done > exec > prep > hesitate);

[0160] Cross-source consistency: When similar tags from different modalitySources conflict within ≤100ms, retain the merged / majority side; for long-term conflicts, output a "conflict" flag and reduce the weight.

[0161] S336. Output fields: retain eventType, confidence_smooth, tStart / tEnd, modalitySource, sensorID, annotationID, producer, and modelVersion.

[0162] S4, Marginal Reasoning;

[0163] S41, Lightweight Edge Model Deployment;

[0164] Using 8-bit quantization and distillation techniques, the original AI model is compressed by 70% and deployed to devices such as NPU, DSP, and CPU. This ensures that the deployed model can perform its main tasks: speech keyword recognition, facial emotion recognition, and behavior detection.

[0165] S42, Edge Decision Reasoning;

[0166] If the confidence level of the device-side recognition result is ≥ 0.85, generate the event result and cache it; the edge device directly outputs the valid label.

[0167] If the confidence level of the device-side identification result is < 0.85, mark the data segment as "awaiting cloud verification".

[0168] S43, Cloud-edge collaboration process, such as Figure 4 As shown:

[0169] Edge inference → Upload summary → Cloud review → Return results → Update log;

[0170] Specifically, edge nodes perform lightweight model inference; upload the results along with timestamps to the cloud; the cloud-based main model verifies consistency, generates the final results, and sends them back to the edge nodes; and the cloud synchronously updates the global log index.

[0171] Cloud output consistency index C:

[0172] ;

[0173] Among them, Score edge For edge nodes, the post-calibration confidence score within the alignment time window (e.g., ±100ms) for the same event type is recommended; the smoothed confidence score is preferred. Score cloud This represents the calibrated confidence level (probability value, 0–1) of the cloud-based main model for the same event type within the same aligned time window. The smoothed confidence level is also used. .

[0174] If C < 0.6, the edge model is automatically marked as a deviation state.

[0175] S5. Log auditing and traceability;

[0176] The data recorded in each frame is:

[0177] {taskID, sessionToken, timestamp_global, sensorID, deviceID,syncError, modelVersion, edgeDecisionFlag, edgeTag, confidence};

[0178] The data types include: task ID, session identifier, unified timestamp, sensor identifier, device ID, time error, model version, edge decision marker, decision path label, and decision confidence.

[0179] In one embodiment of the present invention, examples and descriptions of data recording for each frame are shown in Table 3.

[0180] Table 3:

[0181]

[0182] Each data collection task generates a unique task ID and session token; a unified log database (TimeLineLog) is established in the cloud to enable multimodal data retrieval, alignment, playback, and auditing. The system can quickly trace back the collected data, inference, storage, and AI annotation records for a specific time period based on the task ID.

[0183] Retrospection and auditing are based on a unified log database in the cloud: summaries are uploaded in real time from the edge, aggregated in the cloud, and a timeline index is built to enable multimodal retrieval, alignment, playback, and auditing. In offline scenarios, local caching is performed, and logs are re-uploaded and merged with cloud logs after connecting to the network.

[0184] The system can achieve: full modal playback, synchronous quality assessment, model version tracking, and behavior anomaly tracing.

[0185] Table 4 shows some system data interface definitions and field examples.

[0186] Table 4:

[0187]

[0188] Secondly, the present invention provides an embodiment of a multimodal data time synchronization and annotation and edge inference system, the system comprising:

[0189] The multimodal acquisition terminal specifically includes: a voice acquisition module, an eye-tracking acquisition module, a video acquisition module, and a motion / touch acquisition module;

[0190] The device-side / edge computing node specifically includes: a time synchronization and drift correction controller, a multimodal buffer and alignment module, an AI intelligent annotation module, and a log and data packaging module;

[0191] The cloud-based central system specifically includes: a cloud-based master model inference and verification module, a time base synchronization module, a log and traceability database, and a data management and reporting system;

[0192] Among them, the cloud-based main model inference and verification module uses high-precision AI model analysis, and its results are used to update the task identifier and traceability information generated by the log and data packaging module; the time base synchronization module uses an NTP / PTP global clock to perform time synchronization correction on the time synchronization and drift correction controller.

[0193] Example 1: Cognitive Assessment Task Scenario:

[0194] In the brain intelligence testing machine, the voice module collects the patient's audio responses, the camera captures facial expressions, and the touch module records the reaction time.

[0195] The system uses a time synchronization module to align all data frames in time; the AI ​​annotation module identifies abnormalities in the patient's speech fluency, facial tension, and reaction time, and performs preliminary analysis on the device.

[0196] If the device determines the confidence level is high, it will directly output the "Task correct / incorrect" label; otherwise, it will upload the model to the cloud for verification.

[0197] Results are transmitted back to the cloud and logs are synchronized, enabling rapid diagnosis and real-time feedback.

[0198] Example 2: Rehabilitation training task scenario:

[0199] During rehabilitation training games, the system captures the patient's hand movements, voice commands, and facial expressions in real time.

[0200] Edge nodes execute lightweight AI models locally to detect patient action completion and generate training scores;

[0201] The system receives synchronized time data from the cloud, merges voice and visual information, and generates AI analysis reports and trend curves.

[0202] Thirdly, the present invention provides an embodiment of a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0203] Fourthly, the present invention provides a computer program product embodiment, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0204] Existing multimodal data acquisition systems generally suffer from problems such as inconsistent time references among sensors, difficulties in data fusion, high costs of manual annotation, severe latency in cloud-based inference, and lack of source tracing records, which seriously limit their real-time application in scenarios such as cognitive assessment and rehabilitation training. This invention proposes an integrated architecture of "synchronization-annotation-inference-source tracing," which establishes a unified time reference from the acquisition stage. Through the collaboration of logical clocks and hardware triggers, it ensures that multimodal data is aligned with millisecond-level precision, significantly improving the accuracy of task event recognition.

[0205] Meanwhile, this invention achieves automated annotation through a multimodal fusion AI model, integrating and recognizing information such as voice, actions, expressions, and trajectories. This not only reduces the cost of manual annotation but also significantly improves annotation consistency. Furthermore, this invention achieves low-latency event judgment through an edge inference module, forming a closed loop with cloud-based verification, improving system stability under low network quality or high concurrency conditions. A unified logging and traceability mechanism ensures data traceability, auditability, and medical compliance. Compared to existing technologies, this invention achieves highly accurate real-time processing at the system level.

Claims

1. A method for multimodal data time synchronization, annotation, and edge inference, characterized in that, Includes the following steps: S1. Multimodal data acquisition is carried out during the cognitive assessment process; wherein, the data collected by the same sensor within one hardware sampling period is set as a collection frame, and the data modalities include: audio stream, video stream, touch and handwriting trajectory, motion / skeletal trajectory, and task event stream; S2. Time Synchronization and Drift Correction: Based on the main control MCU reference sampling signal, the hardware sampling period is synchronized. The method of global logic clock + network time protocol + sliding window drift correction is used to align the time of multimodal data. Drift correction is performed when the error is greater than 2ms. The sliding window drift correction method includes: dynamically updating compensation parameters based on the offset of the past N frames recorded by the sliding window; S3, AI Intelligent Annotation: Unifies the encoding of multimodal data into feature vectors; identifies three dimensions: task stage, behavioral event, and emotional state, and outputs three types of annotation labels; performs smoothing and conflict detection on the annotation results, and filters out annotation conflicts or abnormal annotations. S4, Edge Inference and Cloud-Edge Collaboration: Lightweight deployment of AI models on the device to achieve real-time judgment and output of valid labels, and automatic cloud review when the confidence level is lower than the threshold; S5. Log Auditing and Traceability: Generate a unique timestamp and task identifier for each collection frame, establish a unified log database in the cloud, and record the collection, inference, storage and AI annotation information of multimodal data to achieve full-process data traceability, playback and quality assessment.

2. The method according to claim 1, characterized in that, The data parameters for step S1 include: Video sampling frame rate ≥ 25fps, resolution ≥ 1080p; Audio sampling rate ≥ 16kHz / 24bit, signal-to-noise ratio ≥ 60dB; Touch sampling frequency ≥ 60Hz, trajectory accuracy ≤ 2mm; The motion detection module uses a high-precision skeleton recognition model with inter-frame drift ≤ 5px.

3. The method according to claim 1, characterized in that, The specific method of step S2 includes: Correct the local time: ； in: T global To standardize time, T local K(t) is the local time obtained by the sensor, K(t) is the dynamic compensation parameter calculated based on the sliding window drift, and ΔNTP is the network time deviation. The calculation method for dynamic compensation parameters is as follows: ； Among them, E avg K(t+1) is the average drift error per unit time within the sliding window, K(t+1) is the corrected dynamic compensation parameter, K(t) is the current dynamic compensation parameter, and α is the dynamic learning rate.

4. The method according to claim 3, characterized in that, E avg The calculation methods include simplification methods and robust methods: Simplified method: ; in, For the current alignment residual, It is the local time increment of the sampling point corresponding to the i-th residual sample within the sliding window from the anchor point, where the anchor point is the drift correction reference time formed based on the time synchronization event; Robust method for calculating the slope of linear regression: ; Where x is x i The set, e represents the residual. A set of.

5. The method according to claim 3, characterized in that, When the synchronization error E > 2ms, automatic drift retraining is triggered to perform drift correction; whereby... ;T sensor For sensor time.

6. The method according to claim 1, characterized in that, The specific steps of step S3 include: S31. Multimodal feature fusion: The Transformer / Time-series Fusion vector encoding model is used to encode visual keyframe features, speech spectrum features, motion trajectory and eye movement sequence features, and touch trajectory features through cross-modal attention and temporal position encoding, and output a fixed-dimensional feature vector. S32. Annotation Task Output: Through classifier identification, output three types of annotation labels: task stage labels, including preparation, execution, completion, and hesitation; behavioral event labels, including error, repetition, slowness, and incompleteness; and emotional state labels, including tension, joy, focus, and confusion. The annotation labels are bound to the original data in a JSON structure. S33. Confidence Smoothing and Conflict Detection: Voting smoothing is performed using a time window ΔT to smooth the annotation results and filter out jittery labels. ； Where confidence' is the confidence level after smoothing. i Here, N represents the confidence level for each category, and N is the number of categories.

7. The method according to claim 6, characterized in that, Step S33 includes: Events of the same type are merged according to annotation ID. Fragments and smooth confidence scores for each type of event are calculated based on a sliding window step of 50ms. Then, conflict detection is performed. The specific method is as follows: For conflict event labels, retain labels with higher confidence, or set retention priority according to preset event types; when similar labels from different source modalities conflict within 100ms, merge or retain the majority one; for long-term conflict situations, output a "conflict" flag and reduce its weight.

8. The method according to claim 1, characterized in that, The specific steps of step S4 include: S41, Lightweight Edge Model Deployment: Using 8-bit quantization and distillation technology, the original AI model is compressed by 70% and deployed to NPU, DSP, and CPU devices for speech keyword recognition, facial emotion recognition, and behavior detection; S42, Edge Decision Reasoning: When the confidence level of the device-side recognition result is not less than 0.85, output a valid label; otherwise, mark the data segment as pending cloud verification. S43, Cloud-Edge Collaboration: Upload the device-side inference results and timestamps to the cloud, where the main model verifies their consistency.

9. The method according to claim 8, characterized in that, In step S43, the main model verifies consistency using the consistency index C: ； Among them, Score edge Score is the post-calibration confidence of edge nodes for the same event within the alignment time window. cloud The post-calibration confidence of the cloud-based master model for the same event within the same aligned time window; If C < 0.6, the edge model is automatically marked as a deviation state.

10. The method according to claim 1, characterized in that, Step S5 includes: backtracking and auditing are based on a unified log database in the cloud, with real-time upload of summaries at the edge, and timeline indexing after aggregation in the cloud to achieve multimodal retrieval, alignment, playback and auditing; local caching is performed in offline scenarios, and re-uploaded and merged with cloud logs after connecting to the network.

11. The method according to claim 1, characterized in that, Constructing data structures, including: In step S1, metadata is added to each acquisition frame: {timestamp_local, sensorID, modalityType, frameIndex, deviceID}; Where timestamp_local is the local timestamp, sensorID is the sensor identifier, modalityType is the modality type, frameIndex is the frame number, and deviceID is the device ID; In step S3, the data for outputting the annotation labels is as follows: {timestamp_global, eventType, eventConfidence, modalitySource, sensorID,annotationID}; Among them, timestamp_global is the unified timestamp, eventType is the event type, eventConfidence is the recognition confidence level, modalitySource is the source modality, sensorID is the sensor identifier, and annotationID is the annotation ID; In step S5, the data recorded in each frame is as follows: {taskID, sessionToken, timestamp_global, sensorID, deviceID, syncError, modelVersion, edgeDecisionFlag, edgeTag, confidence}; Among them, taskID is the task ID, sessionToken is the session identifier, timestamp_global is the unified timestamp, sensorID is the sensor identifier, deviceID is the device ID, syncError is the time error, modelVersion is the model version, edgeDecisionFlag is the edge decision flag, edgeTag is the decision path label, and confidence is the decision confidence level.

12. A multimodal data time synchronization and annotation and edge inference system, wherein the various parts of the system operate collaboratively to implement the steps of the method according to any one of claims 1-11, the system comprising: The multimodal acquisition terminal specifically includes: a voice acquisition module, an eye-tracking acquisition module, a video acquisition module, and a motion / touch acquisition module; The device-side / edge computing node specifically includes: a time synchronization and drift correction controller, a multimodal buffer and alignment module, an AI intelligent annotation module, and a log and data packaging module; The cloud-based central system specifically includes: a cloud-based master model inference and verification module, a time base synchronization module, a log and traceability database, and a data management and reporting system; Among them, the cloud-based main model inference and verification module uses high-precision AI model analysis, and its results are used to update the task identifier and traceability information generated by the log and data packaging module; the time base synchronization module uses an NTP / PTP global clock to perform time synchronization correction on the time synchronization and drift correction controller.

13. A computer apparatus comprising a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method according to any one of claims 1-11.

14. A computer program product, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-11.

Images