Multi-modal data alignment method and system

Abstract

The invention relates to the technical field of data processing, in particular to a multi-modal data alignment method and system, and the method comprises the steps: collecting multi-modal data, and carrying out the preprocessing of the multi-modal data; the preprocessed multi-modal data are sent to an event anchor point detection module, and multi-modal emergency detection, anchor point marking and preliminary global alignment are carried out; performing time resampling and numerical value standardization on the multi-modal data after coarse alignment; mapping the resampled and standardized multi-modal data to a unified time axis; performing local fine alignment and global adjustment on the multi-modal feature sequence after unified time axis mapping; and performing confidence evaluation and anomaly correction on the fine alignment result, and outputting a final alignment sequence. According to the invention, a high-precision and multi-modal data real-time alignment method which is light in weight and has adaptive correction capability can be realized on end side equipment with limited resources.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for aligning multimodal data. Background Art

[0002] In recent years, with the rapid adoption of intelligent driving, in-vehicle systems, digital human interaction, and various smart in-vehicle devices, multimodal perception has become a core technology for improving environmental understanding and decision-making accuracy. Onboard sensors such as cameras and microphones can simultaneously capture visual and acoustic information, providing the data foundation for functions such as event detection and sentiment analysis. However, different modalities exhibit significant differences in sampling frame rate, data format, timestamp accuracy, sensor link latency, and network jitter. This leads to challenges in cross-modal fusion, such as frame rate mismatch, timing drift, and environmental noise interference. Traditional methods that rely on unified timestamps, linear interpolation, and fixed-window cross-correlation struggle to accurately map short-duration, sudden events such as horns and sudden braking, resulting in frequent false detections and alignment shifts. Furthermore, due to computing power, storage, and power consumption constraints on device-side devices, large offline models, while capable of improving alignment quality, are difficult to deploy in real time, and adaptive correction and robustness remain prominent issues. Therefore, developing a lightweight, robust, and real-time multimodal data alignment technology that can run on-device has become a key technical requirement for achieving high-precision multimodal perception.

[0003] Currently, the industry's technical solutions for aligning multimodal data in edge devices mainly include the following categories: The first is interpolation synchronization based on timestamp order, in which each modal data is matched one-to-one according to the acquisition timestamp, and linear interpolation is applied to the audio signal with a higher frame rate to align the corresponding video frames. The second is unified frame rate resampling combined with global dynamic time warping. First, the video and audio data are resampled to the same frame rate, and then the DTW algorithm is used to search for the optimal time mapping path on the entire sequence to achieve alignment under nonlinear time scales. The third is event detection-driven alignment, which identifies key events in the video or audio stream (such as vehicle horns, sudden braking, sudden acoustic signals, etc.) and uses these event moments as anchor points. The corresponding positions in each modal sequence are located and then locally aligned. The fourth is deep cross-modal embedding alignment: With the help of deep models such as lightweight Transformer CNN and LSTM, the features of different modalities are mapped to a unified embedding space, and then aligned through feature similarity matching.

[0004] While the aforementioned technical solutions can address some scenarios based on their respective focus, they generally face the following bottlenecks in real-time edge applications: unified interpolation or fixed time window methods struggle to accurately map dynamic events; global DTW is computationally intensive, making millisecond-level responses difficult to achieve; event-driven solutions lack global consistency across the entire sequence; and deep models rely on offline training and lack online adaptive correction mechanisms. This is particularly true in environments where computing power, storage, and power consumption are highly constrained on in-vehicle and other edge devices. The need to ensure high-precision synchronization of multimodal data while also maintaining low latency and high robustness presents a challenge for which existing technologies offer a straightforward solution. Summary of the Invention

[0005] This application provides a multimodal data alignment method and system, which can achieve a high-precision, real-time multimodal data alignment method that is both lightweight and has adaptive correction capabilities on resource-constrained end-side devices. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for aligning multimodal data, the method comprising:

[0007] Collecting multimodal data and preprocessing the multimodal data;

[0008] The pre-processed multimodal data is fed into the event anchor detection module for multimodal sudden event detection, anchor tagging, and preliminary global alignment.

[0009] The coarsely aligned multimodal data are temporally resampled and numerically normalized;

[0010] Map the resampled and normalized multimodal data to a unified time axis;

[0011] Perform local fine alignment and global adjustment on the multimodal feature sequence after unified time axis mapping;

[0012] Perform confidence assessment and anomaly correction on the fine alignment results and output the final aligned sequence.

[0013] In a specific embodiment, the collecting multimodal data and preprocessing the multimodal data include:

[0014] Synchronously collect multimodal data streams from the vehicle environment;

[0015] For audio signals, noise is removed through a noise suppressor and filter, and the signal amplitude is smoothed using dynamic range compression technology, adjusting overly strong or weak audio segments to a preset dynamic range. The processed audio is divided into short time frames, and the Mel-frequency cepstral coefficients or Mel-spectrogram embedding are extracted to map the continuous time domain waveform to a frequency domain feature space that conforms to human auditory perception.

[0016] For video signals, spatial noise reduction, exposure correction and motion compensation are applied to each video frame to eliminate blur; a lightweight convolutional neural network is used to perform forward inference on the cleaned image to extract deep feature vectors containing edges, textures and scene semantics.

[0017] In a specific implementation scheme, the pre-processed multimodal data is fed into the event anchor detection module to perform multimodal sudden event detection, anchor tagging, and preliminary global alignment, including:

[0018] The event anchor detection module scans the mel-spectrogram sequence of the audio end and the edge map change sequence of the video end, and calculates the short-term change amount respectively: the audio end constructs a spectrum change map by the amplitude difference of the mel-spectrogram at adjacent moments to identify the spectrum jump of sudden sound events; the video end differentiates the edge maps of adjacent frames, and when a sudden change in edge density or structure is detected, it determines that there may be occlusion or object emergence; at any time, when the difference between the audio change map and the visual edge exceeds a preset threshold, and the time interval between the triggering of the two is less than tens of milliseconds, the event anchor detection module determines that the time point is a multimodal event anchor point and puts this anchor point into the processing window of the alignment operation;

[0019] For each identified event anchor window, short-term signal segments are extracted from the corresponding audio and visual data streams, and the cross-power spectrum of the two is calculated in the frequency domain. The cross-power spectrum is then phase-weighted and inversely transformed back to the time domain to estimate the global time offset.

[0020] The target signal is resampled for a series of candidate scaling ratios, and the generalized cross-correlation phase transform algorithm is applied. The correlation peak amplitude at each scaling ratio is recorded, and the scaling ratio that makes the cross-correlation peak most significant is selected as the optimal time axis correction factor. The preliminary calibration of the global delay and time axis stretching is completed in a single iteration.

[0021] In a specific implementation scheme, the temporal resampling and numerical normalization of the roughly aligned multimodal data includes:

[0022] A unified time grid is determined based on the alignment results, and an interpolation algorithm is used to resample the feature sequences of each modality. For the audio feature stream, the Mel-spectrogram amplitude is interpolated and calculated at each aligned time point. For the visual feature vector, the deep features output by the convolutional network are smoothly interpolated at the resampling points.

[0023] Perform numerical normalization on the multimodal features after time series resampling. First, calculate the historical mean and standard deviation of each modal feature dimension, and then apply zero-mean unit variance normalization based on this to convert features of different modes and dimensions into the same numerical range.

[0024] After standardization, the time series features of all modes are stored in a unified format to form a multi-channel, equal-time series, and numerically unified feature matrix.

[0025] In a specific embodiment, mapping the resampled and normalized multimodal data to a unified time axis includes:

[0026] Determine the time baseline, using a fixed time step as the unit, and divide several equally spaced time points between the start and end of the task as reference coordinates for multimodal alignment;

[0027] Read the timestamp information of each modal sequence and map it to the closest unified time point;

[0028] After the mapping is completed, a multi-channel time series feature matrix is generated: each column corresponds to a unified time point, and each row corresponds to the normalized feature vector of a sensing modality.

[0029] In a specific implementation scheme, performing local fine alignment and global adjustment on the multimodal feature sequence mapped to the unified time axis includes:

[0030] The mapped audio and visual feature sequences are subjected to short-time Fourier transforms to calculate their cross-power spectra. Phase transform weighting is applied in the frequency domain, and the cross-correlation function is obtained through inverse Fourier transform. The overall time offset is accurately estimated using the cross-correlation peak.

[0031] The frequency domain cross-comparison process is repeatedly performed on the same feature sequence within a set of predefined candidate scaling ratios, the cross-correlation peak amplitude at each ratio is recorded, and the one with the strongest peak is selected as the optimal scaling factor;

[0032] After completing the global adjustment, the entire alignment sequence is divided into several short-time windows, and local fine alignment is started in each window. First, based on metrics such as Euclidean distance or cosine similarity, the Hungarian optimal matching algorithm is used to establish a preliminary frame pair mapping between the audio and visual features in the same time window to determine the most likely corresponding feature pairs; then, the dynamic time warping algorithm is applied to the mapping results to carefully optimize the time path, and the matching path is dynamically adjusted to minimize the cumulative deviation.

[0033] In a specific embodiment, the performing confidence assessment and anomaly correction on the fine alignment results and outputting the final aligned sequence includes:

[0034] For each corresponding frame mapping pair, three normalized indicators, namely, alignment error residual, cross-modal feature similarity, and ambient noise level, are calculated and synthesized into a single confidence score according to a preset linear fusion strategy.

[0035] The confidence score obtained after fusion is compared with the multi-level threshold to determine the processing method of the alignment result: sliding window statistics and exponentially weighted moving average methods are used simultaneously, and the weights and thresholds are continuously updated online;

[0036] After completing confidence assessment and multi-level anomaly correction, all correction results are integrated to generate a multimodal alignment sequence.

[0037] In a second aspect, the present application provides a multimodal data alignment system, which adopts the following technical solutions:

[0038] A multimodal data alignment system, comprising:

[0039] A data preprocessing module, configured to collect multimodal data and preprocess the multimodal data;

[0040] The coarse alignment module is used to feed the pre-processed multimodal data into the event anchor detection module to perform multimodal sudden event detection, anchor tagging and preliminary global alignment;

[0041] The data normalization module is used to perform temporal resampling and numerical normalization on the coarsely aligned multimodal data;

[0042] The time unification module is used to map the resampled and normalized multimodal data to a unified time axis;

[0043] The fine alignment module is used to perform local fine alignment and global adjustment on the multimodal feature sequence after unified time axis mapping;

[0044] The alignment sequence output module is used to perform confidence assessment and anomaly correction on the fine alignment results and output the final alignment sequence.

[0045] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement a multimodal data alignment method as described in the first aspect.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and when the program is executed by a processor, it is used to implement a multimodal data alignment method as described in the first aspect.

[0047] In summary, the beneficial effects of this application include at least:

[0048] (1) The unified processing in the multimodal data preprocessing stage greatly improves the subsequent alignment performance. By performing integrated noise suppression, frame rate resampling and feature normalization on the raw data from different sensors before core alignment, the system can effectively eliminate the timing offset and amplitude differences caused by different sampling frequencies, resolutions or encoding accuracies of each modality, so that multi-source data such as audio and video can be directly compared under the same time reference and the same numerical scale. This preprocessing not only significantly reduces the interference of irrelevant noise and outliers on the alignment results, but also simplifies the complexity of the subsequent alignment algorithm when processing diverse inputs, thereby ensuring high-precision alignment and reducing the overall computational overhead in the vehicle-side environment with limited computing power and storage.

[0049] (2) The robustness of the dynamic confidence alignment mechanism is adaptive to sudden interference and data loss. After completing the fine alignment, the system does not simply solidify the alignment results. Instead, it dynamically calculates the confidence score of each mapping based on multi-dimensional indicators such as alignment error residual, cross-modal feature similarity, and environmental noise level, and adopts multi-level correction strategies such as smoothing filtering, neighborhood compensation interpolation, or back-off resampling according to different confidence levels of high, medium, and low. At the same time, the weights and thresholds are updated online through sliding windows or exponential weighted averages to achieve real-time adaptive adjustment for sudden changes in environmental noise, frame loss, or transmission interference. This mechanism can ensure the continuity and stability of the alignment results in extreme vehicle-mounted scenarios, effectively overcome the damage to multi-modal synchronization caused by sudden interference, and significantly improve the robustness of the system in complex dynamic environments.

[0050] First, the synchronously collected audio and video streams are denoised, compensated, and feature extracted. Then, with the help of an event anchor detection module, sudden sound and light events are automatically captured and time windows are marked. Within this window, global coarse alignment is performed using generalized cross-correlation phase transformation. The coarsely aligned feature sequence is then uniformly time-resampled and numerically normalized to map all modal data to the same time axis. Hungarian matching and dynamic time warping are then applied within short-term sub-windows for local fine alignment. Finally, confidence is dynamically calculated based on metrics such as alignment error, cross-modal similarity, and ambient noise, and multi-level anomaly correction is performed to output a continuous and stable high-precision alignment sequence. Through event-driven focused computation, a hierarchical alignment process, and online confidence feedback, this solution achieves millisecond-level high-precision mapping of dynamic sudden events while balancing global consistency and real-time robustness in edge-side environments with limited computing power, storage, and power consumption. This effectively overcomes the accuracy, latency, and adaptability bottlenecks of unified interpolation, static DTW, and offline models.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and to implement it in accordance with the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 3 is a flow chart of a method for aligning multimodal data in an embodiment of the present application.

[0053] Figure 2 It is a schematic diagram of the overall process of the multimodal data alignment method in the embodiment of the present application.

[0054] Figure 3 It is a structural block diagram of the multimodal data alignment system in an embodiment of the present application.

[0055] Figure 4 4 is a block diagram of an electronic device for aligning multimodal data in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0057] Optionally, the present application uses the multimodal data alignment method provided in each embodiment as an example for explanation in an electronic device, where the electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0058] Reference Figure 1 , is a flow chart of a multimodal data alignment method provided by an embodiment of the present application, which includes at least the following steps:

[0059] Step S101: Collect multimodal data and preprocess the multimodal data.

[0060] In step S101, the multimodal data stream from the vehicle environment is first synchronously collected, and a unified time base is established at the time series level to ensure that the subsequent modal features can be aligned and compared at the same time point. Specifically, the audio sensors, cameras and other environmental perception devices carried by the vehicle divide the sampled data into several time segments through the internal clock or GPS timestamp, and uniformly adjust them to millisecond-level accuracy. After completing the time synchronization, the collected raw data is lightweight preprocessed to remove noise, correct distortion and extract preliminary features, so that each modality can reduce the computing pressure and maintain sufficient discrimination ability in the subsequent fusion and reasoning process.

[0061] During implementation, for audio signals, the environmental wind noise and mechanical vibration noise are removed in sequence through an adaptive noise suppressor and a multi-stage bandpass filter, and then the signal amplitude is smoothed using dynamic range compression technology, and audio clips that are too strong or too weak are adjusted to the preset dynamic range; then, the processed audio is divided into short time frames, and the Mel-frequency cepstral coefficients or Mel-spectrogram embedding are extracted to map the continuous time domain waveform to a frequency domain feature space that conforms to the human auditory perception. In terms of visual data, spatial noise reduction, exposure correction and motion compensation are applied to each video frame to eliminate blurring caused by sudden changes in lighting and high-speed driving; then, a lightweight convolutional neural network is used to perform forward reasoning on the cleaned image to extract deep feature vectors containing edges, textures and scene semantics. All modal features are normalized to the same dimension at this stage and aggregated into a unified buffer through time tags to provide high-quality input data for subsequent fusion, matching and reinforcement learning strategies.

[0062] Step S102: The pre-processed multimodal data is sent to the event anchor detection module to perform multimodal emergency event detection, anchor point marking and preliminary global alignment.

[0063] In step S102, after preprocessing the multimodal data, the preprocessed audio and visual features are first fed into the event anchor detection module. The core purpose of this module is to capture synchronous responses caused by sudden physical events in the multimodal data stream and use them as triggers for subsequent fine-grained alignment.

[0064] Specifically, the event anchor detection module scans the mel-spectrogram sequence on the audio side and the edge map change sequence on the video side, calculating the short-term changes in each. The audio side constructs a spectrum change map by taking the difference in the mel-spectrogram amplitudes at adjacent moments to identify spectral jumps of sudden sound events. The video side then differentiates the edge maps of adjacent frames and, when a sudden change in edge density or structure is detected, determines that there may be occlusion or an object appearing. At any moment, when the difference between the audio change map and the visual edge exceeds a preset threshold, and the time interval between their triggering is less than tens of milliseconds, the event anchor detection module identifies that time point as a multimodal event anchor and places this anchor point in the pending window for subsequent alignment operations.

[0065] For each identified event anchor window, a coarse alignment submodule is activated to rapidly correct for global time offset and axial scaling errors between the multimodal data. This submodule uses the generalized cross-correlation phase transform algorithm as its core matching algorithm. First, short-duration signal segments are extracted from the audio and visual data streams, and their cross-power spectra are calculated in the frequency domain. The cross-power spectra are then phase-weighted and inversely transformed back into the time domain to obtain the peak position of the cross-correlation function, which is then used to estimate the global time offset. To simultaneously correct for possible sampling rate bias or time compression / stretching, the target signal is resampled at a series of candidate scaling ratios. The generalized cross-correlation phase transform algorithm is then applied again, and the correlation peak amplitude at each scaling ratio is recorded. Finally, the scaling ratio that results in the most pronounced cross-correlation peak is selected as the optimal time correction factor, completing the initial calibration of global delay and time stretching in a single iteration.

[0066] In summary, all multimodal frames occurring in the same physical event are uniformly mapped to the same time base through the above-mentioned coarse alignment method, forming precisely aligned event anchor areas. Subsequently, high-precision alignment operations are initiated only for these areas, concentrating computing resources on emergency event segments that truly require fine synchronization, rather than fully aligning the entire data stream. Through this event-driven + coarse alignment strategy, it is possible to ensure high-quality synchronization of audio and video information in key emergency scenarios, significantly reduce overall computing overhead, and improve the system's response speed and alignment accuracy to emergencies in the vehicle environment.

[0067] Step S103: performing time resampling and value normalization on the roughly aligned multimodal data.

[0068] In step S103 , time resampling is first performed on the multimodal feature sequence that has been coarsely aligned with event anchor points to eliminate the time step inconsistency problem caused by the frame rate difference between the original acquisition devices.

[0069] Specifically, a unified time grid is determined based on the alignment results, and an appropriate interpolation algorithm, such as linear interpolation or high-order spline interpolation, is used to resample the feature sequences of each modality. For the audio feature stream, the mel-spectrogram amplitude is interpolated and calculated at each aligned time point; for the visual feature vector, the deep features output by the convolutional network are smoothly interpolated at the resampling points to ensure that each time step has corresponding feature data. In this way, data from all modalities are mapped to the same time coordinate system, forming an equally spaced, continuous multimodal sequence, laying a consistent time foundation for subsequent fusion and model input.

[0070] In implementation, numerical normalization is then performed on the multimodal features after time series resampling to unify the differences in numerical range and distribution of each mode. To this end, the historical mean and standard deviation of each modal feature dimension are first calculated, and based on this, zero-mean unit variance normalization is applied to convert features of different modalities and dimensions into the same numerical range. This processing not only eliminates the bias caused by the difference in feature dimensions to the subsequent fusion network or scheduling model, but also improves the comparability of features in the multimodal fusion layer or decision module. After standardization is completed, the time series features of all modalities are stored in a unified format to form a multi-channel, equal-time series, numerically unified feature matrix, which is used as the input of subsequent feature fusion or reinforcement learning scheduling models to ensure the integrity and consistency of the data preprocessing stage.

[0071] Step S104: Map the resampled and normalized multimodal data to a unified time axis.

[0072] In step S104, after completing time resampling and numerical standardization, a global unified time baseline is first determined based on the overall task cycle or a pre-set processing window. The time baseline uses a fixed time step as a unit, and divides a number of equally spaced time points between the start and end times of the task as reference coordinates for multimodal alignment. Read the respective timestamp information in each modal sequence and correspond it to the closest unified time point; for the case where a certain modal data is missing at certain time points, adopt strategies such as keeping the nearest value or interpolation filling to smoothly map adjacent known features to ensure that each modality has a valid feature vector representation at each reference time point. This global mapping method not only solves the synchronization problem of each modality due to sampling frequency and initial offset, but also enables subsequent feature fusion, model inference or scheduling decisions to be processed in parallel at the same time granularity, greatly reducing alignment error and computational overhead.

[0073] After the mapping is completed, a multi-channel time series feature matrix is generated: each column corresponds to a unified time point, and each row corresponds to the standardized feature vector of a sensing modality. The entire matrix is strictly normalized in the numerical interval to ensure that the features of each modality are comparable on the same scale. The matrix is loaded into the subsequent multimodal fusion module or reinforcement learning scheduling model to perform event detection, pattern recognition or unloading decisions on the same time scale. Through this "unified time-unified value-unified format" mapping process, not only is the high consistency of multimodal data in timing and amplitude guaranteed, but it also provides a solid data foundation for the system to collaboratively process asynchronous and multi-source information in complex scenarios.

[0074] Step S105: performing local fine alignment and global adjustment on the multimodal feature sequence mapped to the unified time axis.

[0075] In step S105, based on the generated multimodal feature matrix after unified time axis mapping, global time offset and rate correction are first carried out. Specifically, all pre-processed and mapped audio and visual feature sequences are subjected to short-time Fourier transform respectively, and the cross-power spectrum of the two is calculated; then, phase transformation weighting is applied in the frequency domain, and the cross-correlation function is obtained by inverse Fourier transform, and the overall time offset is accurately estimated with the help of the cross-correlation peak. In order to eliminate the sampling rate or timing scaling error, the above-mentioned frequency domain cross-comparison process is repeatedly performed on the same feature sequence within a set of predefined candidate scaling ratios, and the cross-correlation peak amplitude at each ratio is recorded, and the one with the strongest peak is selected as the optimal scaling factor. Through this process, the global time alignment and axial scaling correction of the entire multimodal sequence can be automatically completed to ensure that different modalities have a consistent timing benchmark at the macro level.

[0076] In implementation, after completing the global adjustment, the entire alignment sequence is divided into several short-time windows, and a local fine alignment process is started in each window. First, based on metrics such as Euclidean distance or cosine similarity, the Hungarian optimal matching algorithm is used to establish a preliminary frame pair mapping between the audio and visual features in the same time window to quickly determine the most likely corresponding feature pairs; then, the dynamic time warping (DTW) algorithm is applied to the mapping result to carefully optimize the time path and dynamically adjust the matching path to minimize the cumulative deviation, thereby compensating for short-term nonlinear time series distortion. Through the two-stage alignment strategy of coarse first and fine later, not only the calibration consistency at the global level is guaranteed, but also high-precision multimodal synchronization is achieved in the local range, providing coordinated and accurate time series input for subsequent feature fusion and reinforcement learning scheduling modules.

[0077] Step S106: perform confidence assessment and anomaly correction on the fine alignment result, and output the final alignment sequence.

[0078] In step S106, the resulting multimodal feature sequence after local fine alignment is used as input. Three normalized metrics, namely, alignment error residual, cross-modal feature similarity, and ambient noise level, are first calculated for each corresponding frame mapping pair. Alignment error residual reflects the temporal and amplitude offset between two signals; cross-modal feature similarity measures the semantic relevance of features from different modalities, such as audio and visual, in the current frame; and ambient noise level captures the potential impact of ambient noise interference on alignment quality. Based on the historical performance and real-time uncertainty of each modality, the weights of the three metrics are dynamically adjusted and synthesized into a single confidence score using a pre-set linear fusion strategy. This score not only quantifies the reliability of the current frame alignment but also provides a basis for subsequent corrective measures.

[0079] The fused confidence score is then compared with multiple thresholds to determine how the alignment results should be handled. For example, a score above 0.7 is considered a high-confidence mapping, and the alignment curve is directly smoothed to eliminate slight jitter caused by occasional noise. A score between 0.4 and 0.7 is considered a medium-confidence mapping, and local compensation interpolation is performed within the current frame's neighborhood time window, correcting small deviations using information from neighboring high-confidence frames. A score below 0.4 is considered a low-confidence mapping, and the algorithm automatically falls back to the interpolation compensation mechanism or, if necessary, re-performs the temporal resampling and numerical normalization process to increase alignment tolerance. Furthermore, sliding window statistics and exponentially weighted moving average methods are used to continuously update weights and thresholds online to adapt to dynamic changes in vehicle motion, environmental noise, and sensor status.

[0080] At this point, after completing confidence assessment and multi-level anomaly correction, all correction results are integrated to generate a final continuous, stable, and high-precision multimodal alignment sequence, which is then output to subsequent feature fusion or scheduling decision modules to provide reliable timing input for the entire on-board edge computing system.

[0081] In summary, combined with Figure 2 First, the synchronously collected audio and video streams are denoised, compensated, and feature extracted. Then, the event anchor detection module automatically captures sudden sound and light events and marks a time window. Within this window, global coarse alignment is performed using generalized cross-correlation phase transformation. The coarsely aligned feature sequence is then uniformly time-resampled and numerically normalized to map all modal data to the same time axis. Hungarian matching and dynamic time warping are then applied within short-term sub-windows for local fine alignment. Finally, confidence is dynamically calculated based on metrics such as alignment error, cross-modal similarity, and ambient noise, and multi-level anomaly correction is performed to output a continuous and stable high-precision alignment sequence. Through event-driven focused computation, a hierarchical alignment process, and online confidence feedback, this solution achieves millisecond-level high-precision mapping of dynamic sudden events while maintaining global consistency and real-time robustness in edge-side environments with limited computing power, storage, and power consumption. This effectively overcomes the accuracy, latency, and adaptability bottlenecks of unified interpolation, static DTW, and offline models.

[0082] Figure 3 This is a block diagram of a multimodal data alignment system provided by one embodiment of the present application. The system includes at least the following modules:

[0083] A data preprocessing module, used to collect and preprocess multimodal data;

[0084] The coarse alignment module is used to feed the pre-processed multimodal data into the event anchor detection module to perform multimodal sudden event detection, anchor tagging and preliminary global alignment;

[0085] The data normalization module is used to perform temporal resampling and numerical normalization on the coarsely aligned multimodal data;

[0086] The time unification module is used to map the resampled and normalized multimodal data to a unified time axis;

[0087] The fine alignment module is used to perform local fine alignment and global adjustment on the multimodal feature sequence after unified time axis mapping;

[0088] The alignment sequence output module is used to perform confidence assessment and anomaly correction on the fine alignment results and output the final alignment sequence.

[0089] For relevant details, please refer to the above method embodiment.

[0090] Figure 4 4 is a block diagram of an electronic device provided in one embodiment of the present application. The device includes at least a processor 401 and a memory 402.

[0091] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0092] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the multimodal data alignment method provided in the method embodiment of the present application.

[0093] In some embodiments, the electronic device may optionally include a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or circuit boards. Illustratively, the peripheral devices include, but are not limited to, a radio frequency circuit, a touchscreen display, an audio circuit, and a power supply.

[0094] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0095] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored. The program is loaded and executed by a processor to implement the multimodal data alignment method of the above method embodiment.

[0096] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored. The program is loaded and executed by a processor to implement the multimodal data alignment method of the above method embodiment.

[0097] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for aligning multimodal data, characterized in that: The method comprises: Collecting multimodal data and preprocessing the multimodal data; The pre-processed multimodal data is fed into the event anchor detection module for multimodal sudden event detection, anchor tagging, and preliminary global alignment. The coarsely aligned multimodal data are temporally resampled and numerically normalized; Map the resampled and normalized multimodal data to a unified time axis; Perform local fine alignment and global adjustment on the multimodal feature sequence after unified time axis mapping; Perform confidence assessment and anomaly correction on the fine alignment results and output the final aligned sequence.

2. The multimodal data alignment method according to claim 1, characterized in that: The collecting of multimodal data and preprocessing of the multimodal data include: Synchronously collect multimodal data streams from the vehicle environment; For audio signals, noise is removed through a noise suppressor and filter, and the signal amplitude is smoothed using dynamic range compression technology, adjusting overly strong or weak audio segments to a preset dynamic range. The processed audio is divided into short time frames, and the Mel-frequency cepstral coefficients or Mel-spectrogram embedding are extracted to map the continuous time domain waveform to a frequency domain feature space that conforms to human auditory perception. For video signals, spatial noise reduction, exposure correction and motion compensation are applied to each video frame to eliminate blur; a lightweight convolutional neural network is used to perform forward inference on the cleaned image to extract deep feature vectors containing edges, textures and scene semantics.

3. The multimodal data alignment method according to claim 2, characterized in that: The pre-processed multimodal data is fed into the event anchor detection module to perform multimodal emergency event detection, anchor point marking, and preliminary global alignment, including: The event anchor detection module scans the mel-spectrogram sequence of the audio end and the edge map change sequence of the video end, and calculates the short-term change amount respectively: the audio end constructs a spectrum change map by the amplitude difference of the mel-spectrogram at adjacent moments to identify the spectrum jump of sudden sound events; the video end differentiates the edge maps of adjacent frames, and when a sudden change in edge density or structure is detected, it determines that there may be occlusion or object emergence; at any time, when the difference between the audio change map and the visual edge exceeds a preset threshold, and the time interval between the triggering of the two is less than tens of milliseconds, the event anchor detection module determines that the time point is a multimodal event anchor point and puts this anchor point into the processing window of the alignment operation; For each identified event anchor window, short-term signal segments are extracted from the corresponding audio and visual data streams, and the cross-power spectrum of the two is calculated in the frequency domain. The cross-power spectrum is then phase-weighted and inversely transformed back to the time domain to estimate the global time offset. The target signal is resampled for a series of candidate scaling ratios, and the generalized cross-correlation phase transform algorithm is applied. The correlation peak amplitude at each scaling ratio is recorded, and the scaling ratio that makes the cross-correlation peak most significant is selected as the optimal time axis correction factor. The preliminary calibration of the global delay and time axis stretching is completed in a single iteration.

4. The multimodal data alignment method according to claim 1, characterized in that: The time resampling and value normalization of the roughly aligned multimodal data includes: A unified time grid is determined based on the alignment results, and an interpolation algorithm is used to resample the feature sequences of each modality. For the audio feature stream, the Mel-spectrogram amplitude is interpolated and calculated at each aligned time point. For the visual feature vector, the deep features output by the convolutional network are smoothly interpolated at the resampling points. Perform numerical normalization on the multimodal features after time series resampling. First, calculate the historical mean and standard deviation of each modal feature dimension, and then apply zero-mean unit variance normalization based on this to convert features of different modes and dimensions into the same numerical range. After standardization, the time series features of all modes are stored in a unified format to form a multi-channel, equal-time series, and numerically unified feature matrix.

5. The multimodal data alignment method according to claim 4, characterized in that: Mapping the resampled and standardized multimodal data to a unified time axis includes: Determine the time baseline, using a fixed time step as the unit, and divide several equally spaced time points between the start and end of the task as reference coordinates for multimodal alignment; Read the timestamp information of each modal sequence and map it to the closest unified time point; After the mapping is completed, a multi-channel time series feature matrix is generated: each column corresponds to a unified time point, and each row corresponds to the normalized feature vector of a sensing modality.

6. The multimodal data alignment method according to claim 1, characterized in that: The performing of local fine alignment and global adjustment on the multimodal feature sequence mapped to the unified time axis includes: The mapped audio and visual feature sequences are subjected to short-time Fourier transforms to calculate their cross-power spectra. Phase transform weighting is applied in the frequency domain, and the cross-correlation function is obtained through inverse Fourier transform. The overall time offset is accurately estimated using the cross-correlation peak. The frequency domain cross-comparison process is repeatedly performed on the same feature sequence within a set of predefined candidate scaling ratios, the cross-correlation peak amplitude at each ratio is recorded, and the one with the strongest peak is selected as the optimal scaling factor; After completing the global adjustment, the entire alignment sequence is divided into several short-time windows, and local fine alignment is started in each window. First, based on metrics such as Euclidean distance or cosine similarity, the Hungarian optimal matching algorithm is used to establish a preliminary frame pair mapping between the audio and visual features in the same time window to determine the most likely corresponding feature pairs; then, the dynamic time warping algorithm is applied to the mapping results to carefully optimize the time path, and the matching path is dynamically adjusted to minimize the cumulative deviation.

7. The multimodal data alignment method according to claim 1, characterized in that: The confidence assessment and anomaly correction of the fine alignment results and output of the final alignment sequence include: For each corresponding frame mapping pair, three normalized indicators, namely, alignment error residual, cross-modal feature similarity, and ambient noise level, are calculated and synthesized into a single confidence score according to a preset linear fusion strategy. The confidence score obtained after fusion is compared with the multi-level threshold to determine the processing method of the alignment result: sliding window statistics and exponentially weighted moving average methods are used simultaneously, and the weights and thresholds are continuously updated online; After completing confidence assessment and multi-level anomaly correction, all correction results are integrated to generate a multimodal alignment sequence.

8. A multimodal data alignment system, characterized in that: include: A data preprocessing module, configured to collect multimodal data and preprocess the multimodal data; The coarse alignment module is used to feed the pre-processed multimodal data into the event anchor detection module to perform multimodal sudden event detection, anchor tagging and preliminary global alignment; The data normalization module is used to perform temporal resampling and numerical normalization on the coarsely aligned multimodal data; The time unification module is used to map the resampled and normalized multimodal data to a unified time axis; The fine alignment module is used to perform local fine alignment and global adjustment on the multimodal feature sequence after unified time axis mapping; The alignment sequence output module is used to perform confidence assessment and anomaly correction on the fine alignment results and output the final alignment sequence.

9. An electronic device, characterized in that: The device includes a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a multimodal data alignment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, which, when executed by a processor, is used to implement a multimodal data alignment method according to any one of claims 1 to 7.

Images