Course Management and Analysis System for Multimodal Data

Abstract

This invention belongs to the field of multimodal data management technology. It provides a course management and analysis system for multimodal data, comprising: synchronously transmitting physical feature signals carrying pseudo-random codes through a physical reference source; acquiring visual image streams and non-visual physical perception streams, extracting coded phases and time-domain feature points; performing time-domain mapping and calculating nonlinear deviations to construct a dynamic second-order clock drift compensation function; using this function to correct timestamps and normalize the time axis to achieve multimodal data alignment; and finally generating a training course continuity analysis and evaluation report based on the aligned data. This invention can suppress clock drift and transmission jitter under strong electromagnetic interference, achieving high-precision multimodal time alignment and ensuring the timing consistency and evaluation accuracy of training data.

Description

Technical Field

[0001] This invention belongs to the field of multimodal data management technology, specifically a course management and analysis system for multimodal data. Background Technology

[0002] Management systems for courses such as mechanical maintenance, welding practice, and high-voltage electrical engineering typically need to integrate visual modalities (such as surveillance cameras) and physical perception modalities (such as inertial sensors worn on the operator's wrist and pressure sensors in smart tools) to comprehensively evaluate operational accuracy.

[0003] However, in actual teaching scenarios, especially in training rooms involving arc welding machines, high-voltage discharge devices, or large rotating electrical machines, there are extremely strong transient electromagnetic pulses (EMPs) and continuous electromagnetic interference (EMI). Traditional course management systems typically rely on the Network Time Protocol (NTP) or the built-in quartz crystal clocks of each device when handling multimodal alignment. But in environments with strong interference, these solutions have the following drawbacks: Nonlinear clock drift: Strong magnetic field environments can interfere with the oscillation frequency of the crystal oscillator inside the sensor, causing a small but nonlinear shift in its physical sampling frequency, resulting in cumulative errors in the timestamps of the sensor data.

[0004] Transmission layer jitter: Electromagnetic noise can cause severe packet retransmission and latency jitter in wireless transmissions (such as Bluetooth and Zigbee), rendering alignment methods based on receiver stamping completely ineffective.

[0005] Software synchronization failure: Most existing software alignment algorithms are based on the constant drift assumption, which cannot cope with the sudden, nonlinear time axis stretching caused by equipment start-up and shutdown (such as the moment the welding machine starts arc) during actual operation.

[0006] The aforementioned issues cause a mismatch between the visually perceived actions and the force / acceleration measured by the sensors on the timeline, directly leading to misjudgments by the teaching analysis system in determining the practical steps, and seriously affecting the fairness and scientific nature of course management.

[0007] To address this, the present invention provides a course management and analysis system for multimodal data. Summary of the Invention

[0008] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0009] The technical solution adopted by this invention to solve its technical problem is: In one aspect, the present invention provides a course management and analysis system for multimodal data, comprising: The feature signal synchronous transmission module synchronously transmits physical feature signals with specific pseudo-random coding sequences through a physical reference source within the training space; The data acquisition and feature recognition module acquires the image stream of the visual acquisition mode and the physical perception stream of the non-visual acquisition mode in real time. It uses the sliding window algorithm to identify the encoded phase of the physical feature signal from the image stream and extracts the time domain feature points of the physical feature signal from the physical perception stream. The clock drift compensation model construction module maps the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal in the time domain, calculates the nonlinear deviation value, and constructs a dynamic second-order clock drift compensation function based on the nonlinear deviation value. The multimodal data time alignment module uses a dynamic second-order clock drift compensation function to resample and nonlinearly interpolate the timestamps of the raw data generated by non-visual sensors, normalizing the time axis of the non-visual acquisition mode to the reference clock domain of the visual acquisition mode, thereby achieving multimodal data alignment. The training data evaluation and analysis module generates a continuity analysis and evaluation report for practical courses based on the aligned multimodal dataset.

[0010] As a further improvement of the present invention, the specific process of synchronously transmitting physical characteristic signals with specific pseudo-random coding sequences through a physical reference source in the training space is as follows: At least one electromagnetic interference-resistant physical reference source device is set up in the training space to acquire physical characteristic signals.

[0011] As a further improvement of the present invention, the physical reference source device is specifically: The physical reference source device includes a high-frequency controllable LED light-emitting module, an ultrasonic emission module, and a synchronous coding control unit; A preset pseudo-random encoding sequence is generated by the synchronous encoding control unit. The synchronous encoding control unit is driven by the same source clock and controls the high-frequency controllable LED light-emitting module and the ultrasonic transmitting module, so that the high-frequency flashing light signal emitted by the LED light-emitting module and the ultrasonic pulse signal emitted by the ultrasonic transmitting module are strictly synchronized. The on / off timing of the high-frequency flicker light signal corresponds one-to-one with the bits of the pseudo-random coding sequence. The transmission time of the ultrasonic pulse signal is aligned with the synchronization header and coding transition edge of the pseudo-random coding sequence. The frequency of the high-frequency flicker light signal is set within the frame rate range that the visual acquisition device can clearly acquire. The frequency, amplitude, and pulse width of the ultrasonic pulse signal are set within the effective detection bandwidth of the non-visual sensor.

[0012] As a further improvement of the present invention, the specific process of acquiring the image stream of the visual acquisition modality and the physical perception stream of the non-visual acquisition modality in real time is as follows: The system acquires a continuous image stream output by an image acquisition device in a visual acquisition mode, and a physical perception stream output by an inertial sensor, a pressure sensor, and a non-visual sensor in a non-visual acquisition mode. The image acquisition device and the non-visual sensor are both deployed in the same training space and are within the signal coverage range of the physical reference source.

[0013] As a further improvement of the present invention, the specific process of identifying the encoded phase of the physical feature signal from the image stream is as follows: Set the sliding window length to For each frame in the image stream within the sliding window, local region grayscale statistics are performed to extract the average brightness value of the region where the physical reference source is located. ; The brightness sequence is converted into a binary sequence by using a preset dynamic threshold. ,in The binarized sequence is cyclically cross-correlated with a preset standard pseudo-random coding sequence. When the correlation coefficient reaches its maximum value, the corresponding offset is determined as the coding phase at the current time. .

[0014] As a further improvement of the present invention, the specific process of extracting the time-domain feature points of the physical feature signal from the physical sensing stream is as follows: A sliding window of the same length as the encoding phase calculation process is used as the time sliding window. The raw signal output from the non-visual acquisition mode is extracted, and the data points within the time sliding window are differentially processed using the first-order gradient operator to identify the abrupt change points in the signal energy. Let the signal sequence be Calculate its gradient function ;when When the pulse trigger threshold is exceeded and a specific pulse width characteristic is met, the corresponding moment is marked as a time-domain feature point. .

[0015] As a further improvement of the present invention, the specific process for calculating the nonlinear deviation value is as follows: Obtain the i-th encoded phase under visual sampling Corresponding visual local timestamp And the corresponding temporal feature point timestamps under non-visual sampling. Based on a preset standard pseudo-random coding sequence period With bit width Calculate the standard reference time under ideal conditions. Construct a three-dimensional mapping sample point set ; Using the reference clock of the visual acquisition modality as a benchmark, calculate the original deviation value of the non-visual acquisition modality relative to the visual modality. : , and These represent the times of the initial alignment point.

[0016] As a further improvement of the present invention, the specific process of constructing the dynamic second-order clock drift compensation function based on the nonlinear deviation value is as follows: Real-time monitoring of electromagnetic intensity evolution parameters in the training environment At the moment the welding machine starts igniting the arc, The value undergoes a drastic nonlinear jump, and the raw time deviation values ​​of multiple non-visual sensors are obtained. ; Using the least squares method, the obtained n sets of sample points Substitute the values ​​into a second-order polynomial model for fitting and construct the compensation function: , The coefficients of the second-order terms, The coefficients of the first-order terms, This is a constant term.

[0017] As a further improvement of the present invention, the specific process of resampling and nonlinear interpolating the original data timestamps generated by non-visual sensors is as follows: The dynamic second-order clock drift compensation function is obtained, and the original timestamp sequence corresponding to the physical perception flow under the non-visual acquisition mode is obtained. Based on the sliding window traversal mechanism, each original timestamp is substituted into the dynamic second-order clock drift compensation function to calculate the corresponding nonlinear correction offset.

[0018] As a further improvement of the present invention, the specific process of resampling and nonlinear interpolation correction of the original data timestamps generated by non-visual sensors further includes: A resampling mechanism is used to homogenize the corrected timestamp sequence, and a nonlinear interpolation algorithm is used to smooth and fill discrete timestamp points and perform time-series calibration.

[0019] On the other hand, this invention provides a course management and analysis method for multimodal data, including: S1: Within the training space, a physical characteristic signal with a specific pseudo-random coding sequence is synchronously transmitted through a physical reference source; S2: Real-time acquisition of image streams from visual acquisition mode and physical perception streams from non-visual acquisition mode; using a sliding window algorithm to identify the encoded phases of physical feature signals from the image streams and extracting time-domain feature points of physical feature signals from the physical perception streams. S3: Map the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal in the time domain, calculate the nonlinear deviation value, and construct a dynamic second-order clock drift compensation function based on the nonlinear deviation value. S4: Using a dynamic second-order clock drift compensation function, the original data timestamps generated by non-visual sensors are resampled and nonlinearly interpolated to normalize the time axis of the non-visual acquisition mode to the reference clock domain of the visual acquisition mode, thereby achieving multimodal data alignment. S5: Based on the aligned multimodal dataset, generate a continuity analysis and evaluation report for the practical course.

[0020] The beneficial effects of this invention are as follows: 1. This invention synchronously transmits physical feature signals carrying pseudo-random codes through a physical reference source, and combines time-domain mapping and nonlinear deviation calculation methods to effectively suppress clock drift and transmission jitter in strong electromagnetic interference environments. This achieves high-precision time synchronization between visual acquisition devices and non-visual sensor data, and solves the technical defects of traditional methods such as inconsistent timestamps and large timing matching errors.

[0021] 2. This invention provides a time-consistent and accurately matched data foundation for the continuity analysis and evaluation of practical training courses. The analysis and evaluation report generated based on aligned data eliminates evaluation errors caused by time-series deviations, significantly improving the credibility and scientific rigor of the evaluation results regarding the continuity, operational standardization, and effectiveness of key nodes in practical training courses.

[0022] 3. The dynamic second-order clock drift compensation function used in this invention can respond to changes in environmental parameters in real time and has good adaptability to complex operating conditions such as strong electromagnetic interference and signal attenuation. Compared with traditional static compensation methods, this invention significantly improves the system's operational stability and data acquisition reliability in harsh environments, and broadens the application scenarios of the technical solution. Attached Figure Description

[0023] The invention will now be further described with reference to the accompanying drawings.

[0024] Figure 1 This is a system module diagram of the course management and analysis system for multimodal data according to the present invention; Figure 2 This is a flowchart illustrating the steps of the course management and analysis method for multimodal data according to the present invention. Detailed Implementation

[0025] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0026] Example 1 like Figure 1 As shown in the embodiment of the present invention, the course management and analysis system for multimodal data includes: The feature signal synchronous transmission module synchronously transmits physical feature signals with specific pseudo-random coding sequences through a physical reference source within the training space; In the feature signal synchronous transmission module, the specific process of synchronously transmitting physical feature signals with specific pseudo-random coding sequences through a physical reference source within the training space is as follows: At least one electromagnetic interference-resistant physical reference source device is arranged in the training space. The physical reference source device includes a high-frequency controllable LED light-emitting module, an ultrasonic emission module, and a synchronous coding control unit. A preset pseudo-random coding sequence is generated by the synchronization coding control unit. This pseudo-random coding sequence has uniqueness and time-domain locationability, and can be used for subsequent time-domain positioning and phase matching. The synchronous encoding control unit uses the same source clock for driving and synchronously controls the high-frequency controllable LED light-emitting module and the ultrasonic transmitting module, so that the high-frequency flashing light signal emitted by the LED light-emitting module and the ultrasonic pulse signal emitted by the ultrasonic transmitting module are strictly synchronously emitted from the same source. Among them, the on / off timing of the high-frequency flickering optical signal corresponds one-to-one with the bit of the pseudo-random coding sequence, and the transmission time of the ultrasonic pulse signal is strictly aligned with the synchronization header and coding transition edge of the pseudo-random coding sequence to ensure that there is no inherent time difference between the optical signal and the ultrasonic signal. Meanwhile, the parameters of the physical feature signal are configured, the frequency of the high-frequency flickering light signal is set within the frame rate range that the visual acquisition device can clearly acquire, and the frequency, amplitude and pulse width of the ultrasonic pulse signal are set within the effective detection bandwidth of the non-visual sensor, so that the physical feature signal can simultaneously cover the physical perception bandwidth of both the visual acquisition mode and the non-visual acquisition mode. Specifically, the physical characteristic signals include: High-frequency flickering light signal: emitted by a high-frequency controllable LED light-emitting module, whose on-off timing corresponds one-to-one with the bits of the pseudo-random coding sequence, and the frequency is set within the frame rate range that the visual acquisition device can acquire; Ultrasonic pulse signal: emitted by ultrasonic transmitting module, its emission time is strictly aligned with the synchronization head and encoding transition edge of pseudo-random encoded sequence, and its frequency, amplitude and pulse width are set within the effective detection bandwidth of non-visual sensor; By driving with the same source clock, it is ensured that there is no inherent time difference between the above optical signal and ultrasonic signal, so as to achieve full coverage of the visual and non-visual dual-modal perception bandwidth of physical feature signal.

[0027] The data acquisition and feature recognition module acquires the image stream of the visual acquisition mode and the physical perception stream of the non-visual acquisition mode in real time. It uses the sliding window algorithm to identify the encoded phase of the physical feature signal from the image stream and extracts the time domain feature points of the physical feature signal from the physical perception stream. In the data acquisition and feature recognition module, the specific process of acquiring the image stream from the visual acquisition modality and the physical perception stream from the non-visual acquisition modality in real time is as follows: The system acquires and obtains continuous image streams output by image acquisition devices in visual acquisition mode and physical perception streams output by inertial sensors, pressure sensors and non-visual sensors in non-visual acquisition mode in real time. The image acquisition devices and non-visual sensors are deployed in the same training space and are within the signal coverage range of the physical reference source. For example, take an electric welding training course: In the visual acquisition mode, the image acquisition device can be an industrial camera placed directly above the training station to capture and output a continuous stream of images in real time, including the trainee's operation gestures and the weld formation process. Non-visual sensors in the non-visual acquisition mode may include a six-axis inertial sensor (IMU) integrated in the welding torch handle and a pressure sensor installed at the bottom of the welding workbench, which respectively output an inertial data stream reflecting the spatial attitude change of the welding torch and a pressure sensing stream reflecting the welding pressure intensity. At this time, the physical reference source device is installed on a fixed bracket on the side of the training station to ensure that the high-frequency scintillation light signal emitted by it is within the effective field of view of the industrial camera, and the ultrasonic pulse signal emitted by it is within the effective acoustic sensing radius of the welding torch and the workbench sensor, thereby realizing the synchronous acquisition of visual image stream and multi-dimensional physical sensing stream under the same physical reference. In the data acquisition and feature recognition module, the specific process of using the sliding window algorithm to identify the encoded phase of physical feature signals from the image stream and extracting the temporal feature points of physical feature signals from the physical sensing stream is as follows: Based on a sliding window algorithm of preset length, the real-time acquired image stream is traversed frame by frame and features are detected. The encoded phase corresponding to the physical feature signal emitted by the physical reference source is located and identified from the image stream. The on / off timing features that match the pseudo-random encoded sequence are extracted to obtain the encoded phase information in the visual dimension. Synchronously, the sliding window length of the non-visual acquisition modality is proportionally mapped according to the sampling frequency of the non-visual sensor to ensure that the visual and non-visual windows are consistent in physical duration. The same sliding window traversal logic is used to perform temporal feature analysis on the physical perception stream of the non-visual acquisition modality, and to extract the temporal feature points corresponding to the physical feature signals from the physical perception data stream. The temporal feature points include the signal start time, the transition edge time, and the encoding synchronization position to ensure that the encoding phase and the detection process of the temporal feature points maintain temporal consistency. The identified coded phase and the extracted temporal feature points are cached in real time to provide stable and reliable basic data for subsequent temporal mapping and bias calculation. The preset length of the sliding window is positively correlated with the number of frames occupied by a complete cycle of the pseudo-random coding sequence in the image stream, so as to ensure that the data within the window covers the complete coding phase features; preferably, the preset length is not less than the total number of frames of a complete cycle of the pseudo-random coding sequence, thereby realizing the unique identification of the coding phase. The present invention does not impose a specific numerical limit on this. The process for calculating the encoded phase and temporal feature points is as follows: Set the sliding window length to frame( Depending on the number of frames corresponding to a complete cycle of the pseudo-random coding sequence, local region grayscale statistics are performed on each frame in the image stream within the sliding window to extract the average brightness value of the region where the physical reference source (LED) is located. ; The brightness sequence is converted into a binary sequence using a preset dynamic threshold (e.g., a weighted average of the maximum and minimum brightness values ​​of a local area within the current sliding window, or an adaptive grayscale segmentation threshold determined based on the real-time statistical distribution of ambient light intensity). ,in Perform a cyclic cross-correlation operation between the binarized sequence and a preset standard pseudo-random encoded sequence: Binarize the sequence With the preset standard pseudo-random coding sequence Perform a cyclic cross-correlation operation; the specific calculation process is as follows: Sequence preprocessing for binarized sequences With standard pseudo-random coding sequence Perform periodic extension to satisfy the operational conditions of cyclic cross-correlation, ensuring that the sequence length is N points, thus forming a discrete sequence with a period of N; Definition of cyclic cross-correlation, cyclic cross-correlation sequence The formula for calculation is: in: m is the time-shift index, representing the relative offset between sequences; This is a cyclic shift operation, ensuring that the sequence matches cyclically within the period; , These represent the values ​​(0 or 1) of the two sequences at corresponding positions. Calculate the time-shift traversal by iterating through the time-shift indices m from 0 to N-1, and perform the following steps for each m: Perform a cyclic shift on sequence P by a shift amount of m; Calculate the shifted sequence and By multiplying and summing the products position by position, we obtain the cross-correlation value at that time shift. After determining the maximum correlation coefficient and completing the traversal, a complete cyclic cross-correlation sequence is obtained. Find the maximum value in the sequence, and its corresponding time shift index mopt is the optimal matching offset; When the correlation coefficient reaches its maximum value, the corresponding offset is determined as the encoding phase at the current moment. ; A sliding window of the same length as the encoding phase calculation process is used as the time sliding window. The raw signal output from non-visual acquisition modes (such as ultrasonic receivers) is captured, and the data points within the time sliding window are differentially processed using the first-order gradient operator to identify abrupt changes in signal energy. Let the signal sequence be Calculate its gradient function ,when When the pulse trigger threshold is exceeded and a specific pulse width characteristic is met, that moment is marked as a time-domain feature point. (e.g., the timing of the encoding transition edge); Real-time output of a set of related pairs That is, in the visual phase At that instant, the timestamp of the physical feature recorded by the non-visual sensor was... This provides accurate discrete mapping sample points for the calculation of nonlinear deviation in the subsequent clock drift compensation model construction module; The clock drift compensation model construction module maps the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal in the time domain, calculates the nonlinear deviation value, and constructs a dynamic second-order clock drift compensation function based on the nonlinear deviation value. In the clock drift compensation model construction module, the specific process of mapping the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal and calculating the nonlinear deviation value is as follows: The encoding phase of the physical feature signal extracted from the visual acquisition modal image stream and the temporal feature points of the physical feature signal extracted from the non-visual acquisition modal physical perception stream are obtained in the data acquisition and feature recognition module. Based on the standard reference timing corresponding to the specific pseudo-random encoding sequence emitted by the physical reference source and preset in the feature signal synchronous transmission module, a temporal mapping relationship between the visual dimension and the physical dimension is established. Specifically, with reference to the benchmark timing, the time domain time corresponding to the encoded phase is matched one by one with the time domain time corresponding to the time domain feature point, and the nonlinear deviation value between the two is calculated. This nonlinear deviation value includes not only the static cumulative deviation caused by the shift in crystal oscillation frequency due to strong electromagnetic interference, but also the sudden, nonlinear time axis stretching or compression deviation caused by practical actions such as arc initiation of welding machines and start-up and shutdown of high-voltage equipment. The specific process for calculating the nonlinear deviation value is as follows: Obtain the i-th encoded phase under visual sampling Corresponding visual local timestamp And the corresponding temporal feature point timestamps under non-visual sampling. Based on a preset standard pseudo-random coding sequence period With bit width Calculate the standard reference time of this phase under ideal conditions. Thus, a three-dimensional mapping sample point set is constructed. ; Using the reference clock of the visual acquisition modality as a benchmark, calculate the original deviation value of the non-visual acquisition modality relative to the visual modality. ; ; in, and The times at which the initial alignment points are located are respectively. This reflects the cumulative time difference between the visual acquisition modal system and the non-visual acquisition modal system at the i-th observation point; In the clock drift compensation model construction module, the specific process of constructing a dynamic second-order clock drift compensation function based on the nonlinear deviation value is as follows: Multiple sets of nonlinear deviation values ​​are obtained, and a dynamic second-order clock drift compensation function is constructed using a polynomial fitting algorithm. This compensation function takes the electromagnetic intensity evolution parameters in the actual operating environment as input variables and the correction amount of the nonlinear deviation value as output variables. It can reflect and fit the clock drift law that changes dynamically with the electromagnetic environment in real time, breaking through the limitation of traditional linear compensation functions that can only deal with constant drift, and providing accurate and dynamic mathematical basis for time axis correction of non-visual acquisition modes. Real-time monitoring of electromagnetic intensity evolution parameters in the training environment (For example: the peak value of the induced electromotive force or the decrease in the signal-to-noise ratio in decibels acquired by the sensor); at the moment the welding machine ignites the arc, The value undergoes a drastic nonlinear jump; simultaneously, the raw time deviation values ​​of multiple non-visual sensors within this time period are obtained through the data acquisition and feature recognition module. ; Using the least squares method, the obtained n sets of sample points Substitute the values ​​into a second-order polynomial model for fitting and construct the compensation function: ; in: Second-order term coefficients It captures the changes in time-axis acceleration caused by the violent evolution of electromagnetic fields (such as the moment of arc initiation), reflecting the intensity of nonlinear distortion; First-order term coefficients Corrects the drift component that increases linearly with the electromagnetic background intensity; constant term : Compensate for the inherent static clock skew of the system; The multimodal data time alignment module uses a dynamic second-order clock drift compensation function to resample and nonlinearly interpolate the timestamps of the raw data generated by non-visual sensors, normalizing the time axis of the non-visual acquisition mode to the reference clock domain of the visual acquisition mode, thereby achieving multimodal data alignment. In the multimodal data time alignment module, the specific process of resampling and nonlinear interpolation correction of the original data timestamps generated by non-visual sensors using a dynamic second-order clock drift compensation function is as follows: The dynamic second-order clock drift compensation function, constructed in the clock drift compensation model construction module, is obtained. This compensation function is used as the core mathematical model for time axis correction of non-visual acquisition modality. Global time series correction processing is performed on the original data timestamps generated by non-visual sensor acquisition. The specific processing procedure is as follows: First, the original timestamp sequence corresponding to the physical perception flow under the non-visual acquisition mode is obtained. Based on the sliding window traversal mechanism, each original timestamp is substituted into the dynamic second-order clock drift compensation function to calculate the nonlinear correction offset corresponding to the timestamp, thus completing the initial deviation elimination of the timestamp. Based on this, a resampling mechanism is used to homogenize the corrected timestamp sequence, and a nonlinear interpolation algorithm is used to smoothly fill and calibrate the discrete timestamp points, eliminating time axis jumps, missing or redundant problems caused by sudden electromagnetic interference, and ensuring the continuity and smoothness of non-visual acquisition modal data in time.

[0028] After timing calibration, the clock signal of the visual acquisition modality is used as a unified reference clock domain. The time axis of the non-visual acquisition modality, which has been corrected by resampling and nonlinear interpolation, is normalized and mapped in the whole domain. This ensures that the physical perception data output by the non-visual sensor and the image data output by the visual acquisition device are strictly aligned in the time domain and matched one-to-one in time. Finally, the multimodal data alignment of the visual acquisition modality and the non-visual acquisition modality in the training scenario is completed, providing a time-unified, accurate and reliable multimodal dataset for subsequent practical course analysis and evaluation. For example, consider the data correction of the welding torch attitude sensor (IMU) during practical training: Assuming that at the instant of arc ignition during electric welding, due to strong electromagnetic pulse interference, the crystal oscillator inside the IMU sensor experiences momentary jitter, acquire a set of raw timestamp sequences at that moment. Substitute these timestamps into the compensation function constructed by the clock drift compensation model construction module. In the middle, calculate the corresponding correction amount (such as The corrected initial timestamp sequence becomes: This eliminates the cumulative drift caused by interference; In the above In the sequence, a large time gap (approximately 14.8 ms) was found between 108.2 ms and 123.0 ms. This was due to data packet loss or sampling delay caused by electromagnetic interference during this period. The Cubic Spline Interpolation algorithm was used to perform nonlinear simulation calculations at the gap based on the known attitude data points before and after, generating smooth virtual data points. This filled in the time axis gap caused by interference, ensuring the continuity of the attitude data stream in time and avoiding action jumps in subsequent analysis. The system uses a fixed frame rate (e.g., 60fps, i.e., one frame every 16.67ms) of the visual acquisition mode as a unified reference clock domain; using a resampling mechanism, the non-visual perception stream after interpolation correction is remapped to standard time nodes {16.67ms, 33.34ms, 50.01ms, ...}; the final alignment result is: when the visual image stream shows the 10th frame of the welding torch touching the workpiece (corresponding to the timestamp 166.7ms), the resampled pressure sensor data and IMU attitude data are also precisely locked at the moment of 166.7ms; Through the above process, the system successfully straightened and aligned the sensor time axis, which was originally distorted and broken due to electromagnetic interference, onto the visual time axis. Even in the environment of strong interference from electric welding, it can achieve a strict one-to-one correspondence between the welding action in the visual image and the current / pressure characteristics recorded by the sensor within the microsecond level of error, thus solving the underlying technical problem of mismatch between multimodal data. The training data evaluation and analysis module generates a continuity analysis and evaluation report for practical courses based on the aligned multimodal dataset.

[0029] In the practical training data evaluation and analysis module, the specific process of generating a continuity analysis and evaluation report for the practical course based on the aligned multimodal dataset is as follows: The multimodal dataset after time-series normalization and alignment is obtained from the multimodal data time alignment module. This dataset contains time-accurate image stream data in the visual acquisition mode and physical perception stream data in the non-visual acquisition mode after dynamic second-order clock drift compensation, resampling and nonlinear interpolation correction. All data have been unified to the reference clock domain of the visual acquisition mode, and have temporal consistency and feature correlation. The aligned multimodal dataset is subjected to full-domain data parsing and feature fusion processing. Spatiotemporal features of operational behavior in the image stream, tool state change features and environmental interaction features in the physical perception stream are extracted respectively. Based on the preset training course evaluation rules and time-series continuity analysis model, the multimodal fusion features are quantitatively analyzed on a time-by-time and action-by-action basis.

[0030] By comprehensively judging and statistically calculating the temporal integrity, action continuity, and data consistency of the entire practical process, a practical course continuity analysis and evaluation report is generated, which includes temporal alignment accuracy, operational process standardization, multimodal data matching degree, and the integrity of key practical nodes.

[0031] The report is output in the form of visualized data, time-series comparison curves and comprehensive scores, which can be directly used for the effect evaluation, process traceability and quality judgment of practical training, realizing the time alignment and application of multimodal data of practical training courses under strong electromagnetic interference environment; Example 2 like Figure 2 As shown, based on the specific implementation of Embodiment 1, the present invention provides a course management and analysis method for multimodal data, including: S1: Within the training space, a physical characteristic signal with a specific pseudo-random coding sequence is synchronously transmitted through a physical reference source; S2: Real-time acquisition of image streams from visual acquisition mode and physical perception streams from non-visual acquisition mode; using a sliding window algorithm to identify the encoded phases of physical feature signals from the image streams and extracting time-domain feature points of physical feature signals from the physical perception streams. S3: Map the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal in the time domain, calculate the nonlinear deviation value, and construct a dynamic second-order clock drift compensation function based on the nonlinear deviation value. S4: Using a dynamic second-order clock drift compensation function, the original data timestamps generated by non-visual sensors are resampled and nonlinearly interpolated to normalize the time axis of the non-visual acquisition mode to the reference clock domain of the visual acquisition mode, thereby achieving multimodal data alignment. S5: Based on the aligned multimodal dataset, generate a continuity analysis and evaluation report for the practical course.

[0032] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A course management analytics system oriented to multi-modal data, characterized by: include: The feature signal synchronous transmission module synchronously transmits physical feature signals with specific pseudo-random coding sequences through a physical reference source within the training space; The data acquisition and feature recognition module acquires the image stream of the visual acquisition mode and the physical perception stream of the non-visual acquisition mode in real time. It uses the sliding window algorithm to identify the encoded phase of the physical feature signal from the image stream and extracts the time domain feature points of the physical feature signal from the physical perception stream. The clock drift compensation model construction module maps the encoded phase of the physical feature signal to the time-domain feature points of the physical feature signal in the time domain, calculates the nonlinear deviation value, and constructs a dynamic second-order clock drift compensation function based on the nonlinear deviation value. The multimodal data time alignment module uses a dynamic second-order clock drift compensation function to resample and nonlinearly interpolate the timestamps of the raw data generated by non-visual sensors, normalizing the time axis of the non-visual acquisition mode to the reference clock domain of the visual acquisition mode, thereby achieving multimodal data alignment. The training data evaluation and analysis module generates a continuity analysis and evaluation report for practical courses based on the aligned multimodal dataset.

2. The multi-modal data oriented course management analytics system of claim 1, wherein, The specific process of synchronously transmitting physical characteristic signals with specific pseudo-random coding sequences through a physical reference source within the training space is as follows: At least one electromagnetic interference-resistant physical reference source device is set up in the training space to acquire physical characteristic signals.

3. The course management and analysis system for multimodal data according to claim 2, characterized in that, The physical reference source device is specifically: The physical reference source device includes a high-frequency controllable LED light-emitting module, an ultrasonic emission module, and a synchronous coding control unit; A preset pseudo-random encoding sequence is generated by the synchronous encoding control unit. The synchronous encoding control unit is driven by the same source clock and controls the high-frequency controllable LED light-emitting module and the ultrasonic transmitting module, so that the high-frequency flashing light signal emitted by the LED light-emitting module and the ultrasonic pulse signal emitted by the ultrasonic transmitting module are strictly synchronized. The on / off timing of the high-frequency flicker light signal corresponds one-to-one with the bits of the pseudo-random coding sequence. The transmission time of the ultrasonic pulse signal is aligned with the synchronization header and coding transition edge of the pseudo-random coding sequence. The frequency of the high-frequency flicker light signal is set within the frame rate range that the visual acquisition device can clearly acquire. The frequency, amplitude, and pulse width of the ultrasonic pulse signal are set within the effective detection bandwidth of the non-visual sensor.

4. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process of acquiring the image stream of the visual acquisition modality and the physical perception stream of the non-visual acquisition modality in real time is as follows: The system acquires a continuous image stream output by an image acquisition device in a visual acquisition mode, and a physical perception stream output by an inertial sensor, a pressure sensor, and a non-visual sensor in a non-visual acquisition mode. The image acquisition device and the non-visual sensor are both deployed in the same training space and are within the signal coverage range of the physical reference source.

5. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process for identifying the encoded phase of physical feature signals from the image stream is as follows: Set the sliding window length to For each frame in the image stream within the sliding window, local region grayscale statistics are performed to extract the average brightness value of the region where the physical reference source is located. ; The brightness sequence is converted into a binary sequence by using a preset dynamic threshold. ,in The binarized sequence is cyclically cross-correlated with a preset standard pseudo-random coding sequence. When the correlation coefficient reaches its maximum value, the corresponding offset is determined as the coding phase at the current time. .

6. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process for extracting time-domain feature points of physical feature signals from the physical sensing stream is as follows: A sliding window of the same length as the encoding phase calculation process is used as the time sliding window. The raw signal output from the non-visual acquisition mode is extracted, and the data points within the time sliding window are differentially processed using the first-order gradient operator to identify the abrupt change points in the signal energy. Let the signal sequence be Calculate its gradient function ;when When the pulse trigger threshold is exceeded and a specific pulse width characteristic is met, the corresponding moment is marked as a time-domain feature point. .

7. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process for calculating the nonlinear deviation value is as follows: Obtain the i-th encoded phase under visual sampling Corresponding visual local timestamp And the corresponding temporal feature point timestamps under non-visual sampling. Based on a preset standard pseudo-random coding sequence period With bit width Calculate the standard reference time under ideal conditions. Construct a three-dimensional mapping sample point set ; Using the reference clock of the visual acquisition modality as a benchmark, calculate the original deviation value of the non-visual acquisition modality relative to the visual modality. : , and These represent the times of the initial alignment point.

8. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process of constructing the dynamic second-order clock drift compensation function based on the nonlinear deviation value is as follows: Real-time monitoring of electromagnetic intensity evolution parameters in the training environment At the moment the welding machine starts igniting the arc, The value undergoes a drastic nonlinear jump, and the raw time deviation values ​​of multiple non-visual sensors are obtained. ; Using the least squares method, the obtained n sets of sample points Substitute the values ​​into a second-order polynomial model for fitting and construct the compensation function: , The coefficients of the second-order terms, The coefficients of the first-order terms, This is a constant term.

9. The course management and analysis system for multimodal data according to claim 1, characterized in that, The specific process of resampling and nonlinear interpolating the timestamps of the raw data generated by non-visual sensors is as follows: The dynamic second-order clock drift compensation function is obtained, and the original timestamp sequence corresponding to the physical perception flow under the non-visual acquisition mode is obtained. Based on the sliding window traversal mechanism, each original timestamp is substituted into the dynamic second-order clock drift compensation function to calculate the corresponding nonlinear correction offset.

10. The course management and analysis system for multimodal data according to claim 9, characterized in that, The specific process of resampling and nonlinearly interpolating the timestamps of the raw data generated by non-visual sensors also includes: A resampling mechanism is used to homogenize the corrected timestamp sequence, and a nonlinear interpolation algorithm is used to smooth and fill discrete timestamp points and perform time-series calibration.

Images