A method for collecting information from micro virtual instruments

Abstract

The present invention discloses a micro-virtual instrument information acquisition method, which relates to the field of information acquisition technology. For a resource-constrained micro-virtual instrument environment, a multimodal data acquisition and time alignment scheme is provided, comprising: step one: normalizing image sequences, audio sequences and sensor value sequences and extracting benchmark events; step two: generating a global time benchmark and multimodal alignment parameters based on a distributed consistency algorithm and an initial correction amount; step three: interpolating and correcting abnormal data segments according to the alignment parameters, thereby significantly improving timing consistency; step four: dynamically monitoring and cyclically self-correcting through a drift metric function to ensure accurate synchronization for long-term operation, which can take into account both high precision and low resource occupancy, has significant robustness and adaptability, and can be widely used in industrial monitoring, smart terminals and the Internet of Things, etc., to achieve deep fusion and reliable analysis of cross-modal data, and support maintaining long-term time consistency in multiple iterations.

Description

Technical Field

[0001] The present invention relates to the technical field of information collection, in particular to a micro virtual instrument information collection method. Background Art

[0002] In diverse scenarios such as IoT monitoring, industrial process control, intelligent driving, and AR / VR, it is often necessary to simultaneously capture image frames, audio sequences, and multiple sensor values ​​to comprehensively perceive the external environment and system status. However, as the scale of application devices continues to expand and their functionality continues to increase, the need for real-time acquisition and analysis of multimodal data sources is becoming increasingly prominent. This is especially true in micro-virtual instrument environments, where processor performance and storage resources are generally limited, yet the system still needs to handle the high-frequency input of massive amounts of heterogeneous data.

[0003] In the above scenario, the system must ensure time synchronization and reliable acquisition between different modal data, while also taking into account requirements such as low power consumption, low bandwidth and convenient deployment, which brings many challenges to data management and alignment.

[0004] In a patent application with publication number CN112069484A, a multimodal interactive information collection method and system are disclosed, which include: identity authentication based on a facial image and voice clip of a person to be verified; pre-identifying attribute information related to the person to be verified through the facial image and voice clip of the person to be verified; obtaining demand information of the person to be verified; determining the content of a questionnaire based on the demand information and the attribute information; presenting the questionnaire content to the person to be verified in a multimodal interactive form to obtain the multimodal questionnaire answers of the person to be verified; fusing the multimodal questionnaire answers; and verifying the validity of the information based on the fusion results.

[0005] However, combined with actual application scenarios and existing technologies:

[0006] At present, the core technical problems in multimodal data acquisition mainly focus on high-load conflicts and clock drift accumulation when performing precise time correction and dynamic synchronization of cross-modal data.

[0007] Specifically, when the image and audio sampling periods are significantly different, or there is a jitter error between the sensor update frequency and the master clock, the multimodal data will present a loose form that cannot be directly aligned; micro-virtual instruments are also prone to clock offset during continuous operation. Without a dedicated hierarchical synchronization processing and interpolation correction mechanism, the collected multimodal data will inevitably have timestamp drift or missing phenomena, which will lead to serious consequences such as inaccurate subsequent analysis results of the system, missed key events or poor interactive experience. Therefore, how to achieve accurate collection and dynamic time alignment of multimodal data under resource-constrained conditions has become a technical bottleneck that needs to be overcome urgently.

[0008] To this end, the present invention provides a micro virtual instrument information acquisition method. Summary of the Invention

[0009] (1) Technical problems solved

[0010] In response to the shortcomings of the existing technology, the present invention provides a micro-virtual instrument information acquisition method, which normalizes image sequences, audio sequences and sensor numerical sequences and extracts benchmark events for resource-constrained micro-virtual instrument environments; generates a global time benchmark and multimodal alignment parameters based on a distributed consistency algorithm and an initial correction amount; interpolates and corrects abnormal data segments according to the alignment parameters, significantly improving the timing consistency; dynamically monitors and cyclically self-corrects through a drift measurement function to ensure accurate synchronization during long-term operation, taking into account high precision and low resource occupancy, has significant robustness and adaptability, and can be widely used in industrial monitoring, smart terminals and the Internet of Things, etc., to achieve deep fusion and reliable analysis of cross-modal data, and solves the technical problems recorded in the background technology.

[0011] (2) Technical solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0013] A micro virtual instrument information acquisition method, comprising:

[0014] When the micro-virtual instrument detects multimodal data input, it reads the image sequence , audio sequence and sensor value sequence The core objects such as image brightness mutation, audio pulse and sensor burst value are detected by calling the feature analysis module and normalized preprocessing is performed to remove noise and calibrate identifiable benchmark events. Finally, a feature information set with a unified time base mark is generated in the preliminary data buffer area.

[0015] When a time offset between a reference event and an external or local timing source is detected and needs to be calibrated, the optimal global offset is used. Calling a distributed consensus algorithm to execute a global time benchmark and local compensation Compensation and generation of image sequences , audio sequence and sensor value sequence The multimodal alignment parameters are finally written into the time management unit;

[0016] When based on the global time base Multimodal alignment parameters detected for image sequences , audio sequence and sensor value sequence There is abnormal delay or missing frame and it is recorded as abnormal data segment collection When marked, the interpolation correction module prioritizes the abnormal segment according to its importance, and then calls the weighted coupling function to interpolate and update the original data corresponding to the sampling, ultimately forming a valid sequence with consistent time sequence in the multimodal data storage area;

[0017] When the benchmark event is running for a long time and a significant drift is detected between the newly collected data and the corrected data, causing the drift to exceed the threshold, the clock synchronization module is first called back to generate a corrected global time benchmark according to the hierarchical compensation strategy. and local compensation , then call the interpolation correction logic to review the suspicious segment, and finally record the self-correction result to the time management unit and overwrite the original benchmark;

[0018] Preferably, the input interface of the micro virtual instrument obtains image sequence, audio sequence and sensor value sequence, and preprocesses them to obtain multimodal data, including preprocessed image frame sequence , audio sequence and sensor value sequence ;

[0019] Preferably, feature analysis is performed on the pre-processed multimodal data to extract potential event candidate points, where:

[0020] A feature detection model based on wavelet transform is used to better identify instantaneous pulses. To ensure the detection effect and computational efficiency, the following wavelet analysis expression is defined:

[0021] Where: For the modal Wavelet transform output on ; Indicates in Continuous signal form under different modes; is the complex conjugate form of the selected wavelet basis function; is the scale parameter 0; is the translation parameter;

[0022] By wavelet transforming the output Perform peak retrieval at multiple scales to obtain a set of candidate events ; ,in Indicates the The first detected event candidates;

[0023] Preferably, cross-modal correlation analysis is performed on events of different modalities to screen key events. , where the time domain overlap judgment function is defined , which is used to determine whether events between different modalities can be considered as the same benchmark event:

[0024]

[0025] Where: Refers to different modes respectively. For the modal Medium Event The time point of occurrence; is the threshold for cross-modal event fusion;

[0026] If there are multiple modal candidate events that satisfy , then merge the set into the same benchmark event , the aggregated benchmark event set is recorded as After completing the aggregation, the benchmark event collection and its corresponding timestamp is , stored in the preliminary data buffer area;

[0027] Preferably, if the deployment environment allows receiving high-precision signals , then the local time of each benchmark event The corresponding external reference time Compare and form offset sequences ,in:

[0028]

[0029] If no external reference is available, a selected mode is used as an approximate reference locally to obtain the corresponding ; After obtaining the offset sequence Post-evaluation and solving for the optimal global offset ,in:

[0030]

[0031] in, is a positive real number, representing the sensitivity coefficient to the offset error; is the total number of benchmark events; by iterative search or analytical approximation method, the penalty function is obtained Minimize the optimal global offset :

[0032]

[0033] Get the optimal global offset Finally, it can be regarded as the uniform initial clock correction of the micro virtual instrument relative to the external reference; the final optimal global offset Together with the penalty function Solution parameters , and stored in the time management unit of the micro virtual instrument;

[0034] Preferably, when multiple micro virtual instruments each obtain the local optimal global offset After that, a simplified consensus protocol is executed between devices, periodically exchanging and comparing their respective optimal global offsets. and some benchmark event times Observation data; after the protocol is completed, the micro virtual instrument platform obtains the global consistency correction value ,in Represents the global domain correction, each is the local compensation amount;

[0035] Based on the above global time base and local compensation The consistency results are further subdivided into modalities such as image, audio, and sensor. Specifically, based on the corrected global time reference, the frame rate and frame arrival time are recalibrated to obtain the image modality alignment parameters. and ; and the obtained and multimodal alignment parameters Also write into the multimodal alignment parameter area of ​​the time management unit;

[0036] Preferably, the multimodal alignment parameters , first calculate the ideal sampling index of each mode under the unified time axis; compare the deviation between the actual sampling time and the ideal sampling time, if the deviation of a certain section is greater than the established threshold, it is considered as an abnormal delay section;

[0037] If a continuous missing frame or missing sampling point is found in the data record of any mode, it is recorded as a missing segment, and the judgment result is recorded as an abnormal data segment set. , which contains the start and end time and modal information of each abnormal segment;

[0038] The importance of index intervals identified as abnormally delayed or missing segments for subsequent analysis. If anomalies occur in high-importance segments, interpolation correction is performed. If the segments are of low importance or marked as having no critical content, interpolation can be skipped to save resources.

[0039] Preferably, for abnormal data segment collection , based on the multimodal alignment parameters and global time base , for different modes and anomaly types, corresponding interpolation algorithms are selected:

[0040] When the abnormal segment involves multiple modes that are misplaced or missing at the same time, let and Respectively indicate time The combination of different modal interpolation results or correction sequences, For the The similarity or difference measurement function of the modalities in the time interval On the other hand, the following weighted coupling function is introduced Constrain the correction results, where:

[0041]

[0042] Where: Indicates time Interpolation or correction results of multimodal data; As an evaluation An evaluation function for the quality or similarity of interpolation between modalities; For modal Used to measure the interpolation results in this mode. With reference data degree of difference or similarity;

[0043] express About time The first-order partial derivative of The rate of change of similarity over time under each modality; is the weighting coefficient, is the mode weight in the overall coupling;

[0044] is a positive real sensitivity factor, To process the window;

[0045] By minimizing , seeking to make the multimodal interpolation synchronize with the actual observation data, and finally to achieve the correction result on the global time base. Make the data segments of each modality maintain the same timing; after completing the cross-modal interpolation correction, the benchmark event The corresponding correction time is recorded as , saved in the time management unit;

[0046] The new frames, new sampling points or new data segments generated by interpolation correction are named in a manner consistent with the original data objects and are combined with the multimodal data storage area of ​​the micro virtual instrument to overwrite or mark the original abnormal segment;

[0047] Preferably, read the benchmark event set and its timestamp , and combined with the global time base , abnormal data segment collection , locate the occurrence time of these benchmark events in the newly collected data and obtain the actual observation time ;

[0048] The actual observation time Aligned with or corrected to a previously recorded reference time To calculate the difference, in order to avoid using simple mean difference or standard deviation, a drift metric function is defined. , as follows:

[0049] Assume that there is a micro virtual instrument platform Each benchmark event needs to be uniformly measured and defined in a time interval The drift vector on :

[0050]

[0051] in, Indicates the The drift of the benchmark event over time; A benchmark event can be Composition drift vector ;

[0052] For measurement In the interval The overall drift accumulation degree on the , and give a strong penalty for large changes, introduce the drift metric function :

[0053]

[0054] Where: The dimension is The drift vector includes all the benchmark events or multi-modal real-time drift values ​​that the micro virtual instrument platform is concerned about; different represents different detection benchmark event offsets; Represents the drift vector In time The first derivative / gradient of ; For size The weight matrix of For selection The norm is used to measure W the overall magnitude of is a positive real number sensitivity factor / amplification factor, is the time integration interval;

[0055] The drift metric function Compare it with the preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has produced significant drift and needs to trigger self-correction. Otherwise, the current value can be recorded into the time management unit to update the recent drift level, and no large-scale correction operation will be performed for the time being.

[0056] Preferably, when the micro virtual instrument platform finds a significant drift, it will first call this module to estimate and correct the current clock offset again to generate the latest correction value. and local compensation ;

[0057] If there is obvious multimodal data missing or abnormality in the self-correction trigger period, the data of this period will be re-interpolated or interpolated again through the interpolation correction algorithm; or the interpolation result suitable for the current drift situation will be regenerated;

[0058] When the drift metric function When the drift threshold exceeds the first level (mild), only local compensation is used, and interpolation adjustments are made to a very small number of abnormal segments. ;

[0059] When the drift metric function The second level drift threshold (medium) is reached, and the corrected global time base is activated The refresh is combined with the interpolation module to correct all suspicious segments;

[0060] When the drift metric function If the drift exceeds the third level threshold (severe), the system enters full callback mode, which includes dynamically adjusting the core parameters of the distributed consistency module and the cross-modal interpolation algorithm (such as kernel function and weight coefficient) to maximize the correction accuracy.

[0061] Preferably, the corrected data and global time base are verified again, and the results are written back to the key storage and monitoring structure within the micro virtual instrument platform, wherein:

[0062] Read the latest correction parameters 、 , multimodal data generated by secondary interpolation of drift, and re-calibration of benchmark events Compare the time in the data; if the newly generated drift metric function If the value has fallen back to within the threshold, the self-calibration is confirmed to be complete: if it is still out of limit, the compensation strategy can be repeatedly upgraded within the scope allowed by the resources of the micro virtual instrument platform;

[0063] The final confirmed correction value of the global time base and corrected local compensation Overwrite the original clock reference configuration and write the timestamp, trigger reason, execution details and other information of the self-correction operation into the time management unit to facilitate subsequent audit or troubleshooting.

[0064] (3) Beneficial effects

[0065] The present invention provides a micro virtual instrument information acquisition method, which has the following beneficial effects:

[0066] The feature analysis module is used to normalize and preprocess image sequences, audio sequences, and sensor value sequences, and extract benchmark events. This module can remove noise and calibrate recognizable time anchor points even under resource-constrained conditions.

[0067] Combined with an external high-precision reference or a local stable timing source, a distributed consensus algorithm and the optimal global offset are used to calculate the global time base and local compensation. Differentiated alignment parameters are generated based on the characteristics of image, audio, and sensor modalities, significantly reducing the synchronization overhead during subsequent interpolation corrections while ensuring timing accuracy.

[0068] Cross-modal interpolation correction is performed around a collection of abnormal data segments, which not only eliminates missing or delayed data in a single modality but also allows multi-modal data to coincide with each other on the same timeline, reducing uncertainty in subsequent fusion analysis.

[0069] Monitor the deviation between benchmark events and collected data over long periods of time. If drift exceeds the limit, the system automatically calls back to update the global time base, local compensation, and interpolation results for abnormal sections. Using a graded compensation strategy, the micro-VI dynamically decides whether to perform local repairs or global refreshes based on different drift levels, maintaining consistent timing over multiple cycles.

[0070] Despite the limited computing power and storage resources of the micro virtual instrument, it is still possible to overcome the problems of error accumulation and environmental fluctuations that are difficult to meet with hardware synchronization or single calibration in traditional solutions through the coordinated cooperation of multimodal normalized feature extraction, hierarchical synchronous processing, cross-modal index correction and long-term self-correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 The figure is a flow chart of a micro virtual instrument information acquisition method of the present invention. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] See also Figure 1 The present invention provides a micro virtual instrument information acquisition method, comprising:

[0074] Step 1: When the micro-virtual instrument detects multimodal data input, it reads the image sequence , audio sequence and sensor value sequence The core objects such as image brightness mutation, audio pulse and sensor burst value are detected by calling the feature analysis module and normalized preprocessing is performed to remove noise and calibrate identifiable benchmark events. Finally, a feature information set with a unified time base mark is generated in the preliminary data buffer area.

[0075] Step 101: Multimodal data preprocessing and input sequence normalization

[0076] The input interface of the micro virtual instrument obtains image sequences, audio sequences, and sensor value sequences, and preprocesses them to obtain multimodal data, including:

[0077] Unify the size and chroma of image data of different resolutions or color spaces to generate a pre-processed image frame sequence ; At a uniform sampling rate Under this condition, the amplitude of the audio signal is normalized to obtain the normalized audio sequence ;

[0078] Perform unified data format conversion on different types of sensor outputs (such as temperature, pressure, acceleration, etc.) to form a sensor value sequence ;

[0079] Normalization processing ensures that when subsequent feature extraction is performed on image, audio, and sensor data, features can be compared and screened under unified coordinates or dimensions.

[0080] Step 102: Multimodal feature analysis and event candidate extraction

[0081] Perform feature analysis on the preprocessed multimodal data to extract potential event candidate points, where:

[0082] A feature detection model based on wavelet transform is used to better identify instantaneous pulses (common in audio and sensor sequences) or high-frequency mutations (possible in images and sensor sequences). To ensure detection effectiveness and computational efficiency, the following wavelet analysis expression is defined:

[0083] Where: For the modal Wavelet transform output on ; Indicates in Continuous signal form under each mode (for images, frame signals can be extracted in the time dimension, for audio, it is the time domain waveform, and for sensors, discrete sampling can be approximated as a continuous function after interpolation);

[0084] is the complex conjugate form of the selected wavelet basis function; is a scale parameter, the value range is dynamically adjusted according to the actual signal frequency band, and the value is greater than 0; is a translation parameter between 0 and T, used to scan the characteristic distribution of the signal on the time axis;

[0085] By wavelet transforming the output Perform peak retrieval at multiple scales to obtain a set of candidate events ; ,in Indicates the The first detected event candidates;

[0086] Depending on the modality type, the threshold setting and peak judgment conditions can be configured differently, but the core is to determine possible transient mutations (such as audio pulses, acceleration spikes, image brightness flicker, etc.) in each modality.

[0087] Using wavelet transform instead of the common time window sliding or Fourier transform method can capture instantaneous features in a wider frequency band and has better noise suppression capabilities.

[0088] Step 103: Cross-modal benchmark event screening and time calibration

[0089] In the candidate event set Based on this, we conduct cross-modal correlation analysis on events in different modalities to screen out key events that can appear simultaneously or approximately simultaneously in multiple modalities. , define the time domain overlap judgment function , which is used to determine whether events between different modalities can be considered as the same benchmark event:

[0090]

[0091] Where: Refers to different modalities (such as image modality, audio modality or a sensor modality), For the modal Medium Event The time point of occurrence; The threshold for cross-modal event fusion determination is set according to the clock accuracy of the micro virtual instrument or the real-time requirements of the application scenario;

[0092] By calculating the time domain overlap judgment results between each mode in turn, if there are multiple mode candidate events that meet , then merge the set into the same benchmark event The final calibration time can be taken as weighted average or uniformly selected from the timestamp value of a certain priority mode. The aggregated benchmark event set is recorded as After completing the aggregation, the benchmark event collection and its corresponding timestamp is , stored in the preliminary data buffer area;

[0093] Through cross-modal correlation analysis and time domain overlap judgment function, key events are screened out It has significant features that can be detected in multiple modalities, and can better serve as the basis for the alignment of the global time benchmark, fusing the key events of multiple modalities into a unified benchmark event, which helps to achieve more accurate alignment parameter calculation and improve the reliability of multimodal collaboration.

[0094] Step 2: When the time offset between the reference event and the external or local timing source needs to be calibrated, the optimal global offset is used. Calling a distributed consensus algorithm to execute a global time benchmark and local compensation Compensation and generation of image sequences , audio sequence and sensor value sequence The multimodal alignment parameters are finally written into the time management unit;

[0095] Step 201: Initial clock offset estimation and external reference correction

[0096] Based on the benchmark event set and the corresponding timestamp , estimate the initial clock offset of the micro virtual instrument platform and perform benchmark correction in the presence of external high-precision reference signals (such as GPS, PPS, laser pulses, etc.). The specific processing includes:

[0097] If the deployment environment allows receiving high-precision signals , then the local time of each benchmark event The corresponding external reference time Compare and form offset sequences ,in:

[0098] ;

[0099] If no external reference is available, a selected mode (or a stable reference timing source) is used locally as an approximate reference to obtain the corresponding , the offset sequence records the difference between the local benchmark event of the micro virtual instrument and the external reference on the time axis; Then, the following exponential penalty function is used to evaluate and solve the optimal global offset , so that all benchmark events are corrected uniformly:

[0100]

[0101] in, is a positive real number, representing the sensitivity coefficient to the offset error; is the total number of baseline events;

[0102] By iterative search or analytical approximation method, the penalty function Ω is obtained. Minimize the optimal global offset :

[0103]

[0104] Get the optimal global offset Finally, it can be regarded as the uniform initial clock correction of the micro virtual instrument relative to the external reference (or local reference), which is used to eliminate the time offset in a large range; the final optimal global offset Together with the penalty function Ω Solution parameters , and stored in the time management unit of the micro virtual instrument to perform detailed calculation of distributed consistency or multimodal alignment parameters.

[0105] When used, through the exponential penalty function Ω , when an external reference signal is available, the overall time base can be actively aligned to a high-precision clock; if an external reference is not available, preliminary offset correction can also be completed based on a local stable source;

[0106] Step 202: Distributed consistency collaboration and multimodal alignment parameter generation

[0107] After the initial offset correction, the local clock undergoes further distributed or multi-channel consistency processing to generate alignment parameters suitable for subsequent interpolation correction. This is mainly aimed at multi-device collaboration or multi-threaded concurrent scenarios, specifically including:

[0108] When multiple micro-virtual instruments each obtain the local optimal global offset Finally, to ensure the consistency of the overall time base, a simplified consensus protocol can be executed between devices to periodically exchange and compare their respective optimal global offsets. and some benchmark event times Observational data of

[0109] After the protocol is completed, the micro virtual instrument platform obtains the global consistency correction value , , },in Represents the global time base, each is the local compensation amount;

[0110] Based on the above global time base and local compensation The consistency results are further subdivided into image, audio, and sensor modalities, as follows:

[0111] Based on the corrected global time reference, the frame rate and frame arrival time are recalibrated to obtain the image modality alignment parameters. : Combined with sampling frequency And preliminary corrections to ensure that the audio data is synchronized to the global time with high precision and obtain the audio mode alignment parameters Based on the characteristics of each sampling period of the sensor, a timing correction scheme suitable for resource-constrained environments is derived, and Linkage control, obtaining sensor modal alignment parameters ; calculated and multimodal alignment parameters Also write into the multimodal alignment parameter area of ​​the time management unit;

[0112] Through the distributed consistency protocol, the global time base of multiple micro virtual instruments or multi-threaded systems is unified, reducing the accumulation of clock drift between nodes and refining the multi-modal alignment parameters. Different synchronization frequencies and computational overheads can be allocated according to modal characteristics to suit resource-constrained scenarios. Perform rebalancing or distributed consensus processing to ensure high-precision and robust time synchronization at both macro and micro levels.

[0113] Step 3: Based on the global time base Multimodal alignment parameters detected for image sequences , audio sequence and sensor value sequence There is abnormal delay or missing frame and it is recorded as abnormal data segment collection When marked, the interpolation correction module prioritizes the abnormal segment according to its importance, and then calls the weighted coupling function to interpolate and update the original data corresponding to the sampling, ultimately forming a valid sequence with consistent time sequence in the multimodal data storage area;

[0114] Step 301: Identify abnormal data segments and determine interpolation requirements

[0115] Based on the global time base and multimodal alignment parameters ,Identify potential abnormal delays or missing data segments from the multimodal data storage area of ​​the micro virtual instrument and prepare for subsequent interpolation correction.,The main processing flow is as follows:

[0116] For multimodal alignment parameters (e.g. Corresponding image frame, Corresponding audio frame, Corresponding sensor sampling), first calculate the ideal sampling index of each modality under the unified time axis, for the image sequence , we can make:

[0117]

[0118] For audio sequences and sensor sequence Similar ideal sampling times can also be defined separately and , by adding a global time base and local compensation After that, the expected sampling position under the global time reference is obtained;

[0119] Compare the deviation between the actual sampling time (stored in the data storage area) and the ideal sampling time. If the deviation of a certain segment is greater than the established threshold (set by the micro virtual instrument platform requirements or application scenarios), it is considered an abnormal delay segment;

[0120] If a continuous missing frame or missing sampling point is found in the data record of any mode, it is recorded as a missing segment, and the judgment result is recorded as an abnormal data segment set. , which contains the start and end time and modal information of each abnormal segment;

[0121] For index intervals that have been determined to be abnormal delay segments or missing segments, the user or the trained neural network can determine their importance for subsequent analysis; if an abnormality occurs in a high-importance segment, focused interpolation correction is required; if it is a low-importance segment or has been marked as a segment without key content, interpolation can be skipped to save resources; through the coordinated use of alignment parameters and the global time base, rapid discovery of abnormal delays or data missing is achieved, and the ability to perceive the quality of multimodal data is enhanced. Abnormal segments and missing segments are classified and processed separately, so that subsequent interpolation correction can allocate computing resources in a targeted manner, ensuring that the micro virtual instrument can still maintain overall timing consistency in a resource-constrained environment.

[0122] Step 302: Multimodal interpolation correction and data level alignment execution

[0123] For abnormal data segment collection , based on the multimodal alignment parameters and global time base , perform targeted cross-modal interpolation correction to improve the matching degree between actual data records and theoretical time series. The main processing process is as follows: for different modes and anomaly types, select the corresponding interpolation algorithm:

[0124] Image Modality :Using the time interpolation method based on multidimensional interpolation kernel, the kernel function can be defined To measure the similarity between adjacent frames; audio modality : When missing segments are detected, weighted phase compensation interpolation can be used to maintain spectrum continuity; sensor mode :Multi-point interpolation or piecewise polynomial fitting is commonly used, especially suitable for smoothing sudden changes or extreme peak position drift;

[0125] When the abnormal segment involves multiple modes that are misplaced or missing at the same time, let and Respectively indicate time The combination of different modal interpolation results or correction sequences (which can be regarded as combining all modal data into a multidimensional vector evolving over time), For the The similarity or difference measurement function of the modalities (defined in the previous article, its specific form can be based on the local texture of the image, audio phase or sensor curve, etc.) is calculated in the time interval On the other hand, the following weighted coupling function is introduced Constrain the correction results, where:

[0126]

[0127] Where: Indicates time The interpolation or correction results of the multimodal data can be regarded as the distribution of high-dimensional vectors over time; As an evaluation An evaluation function for the interpolation quality or similarity of each modality, which evaluates the similarity of image frames, the degree of coincidence of audio spectra, the matching degree of sensor trend curves, etc. For modal Used to measure the interpolation results in this mode. With reference (or real) data degree of difference or similarity;

[0128] express About time The first-order partial derivative of The rate of change of similarity (or difference) over time under each modality; Corresponding to the original weighting coefficient, it is the modal The weight in the overall coupling reflects the degree of influence of the mode on the target of the micro virtual instrument platform, and its value range is a positive real number;

[0129] is a positive real number sensitivity factor used to amplify (or suppress) modal differences with a higher rate of change over time. To process the window;

[0130] By minimizing , seeking to make the multimodal interpolation synchronize with the actual observation data, and finally to achieve the correction result on the global time base. Make the data segments of each modality maintain the same timing; after completing the cross-modal interpolation correction, the benchmark event The corresponding correction time is recorded as , saved in the time management unit;

[0131] The new frames, new sampling points or new data segments generated by the interpolation correction are stored in the multimodal data storage area of ​​the micro virtual instrument in a manner consistent with the naming of the original data objects, overwriting or marking the original abnormal segments: at the same time, the kernel function, weight coefficient and interpolation interval used in the interpolation operation are registered in the time management unit;

[0132] When using, differentiated interpolation strategies are selected for different modalities such as images, audio, and sensors to improve the accuracy and stability of the interpolation results. The multimodal interpolation process is coupled to ensure that multiple modes maintain consistency on the same time basis, rather than just the separate storage of independent correction data records and interpolation parameters of a single mode, which makes it easy to quickly locate the corrected segment and perform re-correction or backtracking operations.

[0133] Step 4: When the benchmark event is running for a long time and a significant drift is detected between the newly collected data and the corrected data, causing the drift to exceed the threshold, the clock synchronization module is first called back to generate a corrected global time benchmark according to the hierarchical compensation strategy. and local compensation , then call the interpolation correction logic to review the suspicious segment, and finally record the self-correction result to the time management unit and overwrite the original benchmark;

[0134] Step 401, cross-modal drift detection and abnormal threshold determination are based on the latest data (denoted as , representing the correction sequence of image, audio, and sensor respectively), for each modality’s newly acquired data (denoted as ) for comparison;

[0135] Identifying potential drift that accumulates over time. The main process is as follows:

[0136] Reading a benchmark event collection and its timestamp , and combined with the global time base , abnormal data segment collection , locate the occurrence time of these benchmark events in the newly collected data and obtain the actual observation time ;

[0137] The actual observation time Aligned with or corrected to a previously recorded reference time To calculate the difference, in order to avoid using simple mean difference or standard deviation, a drift metric function is defined. , as follows:

[0138] Assume that there is a micro virtual instrument platform Each benchmark event (or multiple modal offsets) needs to be measured uniformly and defined in the time interval The drift vector on :

[0139] in, Indicates the The drift (or offset function) of each benchmark event over time can be obtained by interpolating or smoothing the difference sequence between the new observation time and the reference time. A benchmark event can be Composition drift vector ;

[0140] For measurement In the interval The overall drift accumulation degree on the , and give a strong penalty for large changes, introduce the drift metric function :

[0141]

[0142] Where: The dimension is The drift vector includes all the benchmark events or multi-modal real-time drift values ​​that the micro virtual instrument platform is concerned about; different represents different detection benchmark event offsets; Represents the drift vector In time The first-order differential / gradient of If it is discrete sampling, it can be approximated by difference or interpolation;

[0143] For size The weight matrix is ​​used to distinguish the importance or coupling relationship of different benchmark events in the vectorized measurement; the diagonal elements You can zoom in / out The off-diagonal elements can be used to describe the cross-constraints or mutual influences between different events (if no coupling is required, it can also be set as a diagonal matrix);

[0144] For selection The norm is used to measure W The overall amplitude of , 2 or infinite norm; is a positive real number sensitivity factor / amplification factor, The time integration interval is usually the time window of current monitoring or evaluation;

[0145] The drift metric function Compare it with the preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has produced significant drift and needs to trigger self-correction. Otherwise, the current value can be recorded into the time management unit to update the recent drift level, and no large-scale correction operation will be performed for the time being.

[0146] When in use, by comparing the moments of benchmark events in new and old data, the time offset under multimodality can be quickly detected, so that the points with large drifts can be given higher weights to trigger self-correction more effectively. The threshold judgment is combined with the time management unit to maintain continuous perception of the drift of the micro virtual instrument platform.

[0147] Step 402: Self-correction triggering and hierarchical compensation strategy

[0148] When the drift metric is determined to be out of limit, the micro-virtual instrument platform automatically calls back the previous clock synchronization processing and interpolation correction module, and formulates a hierarchical compensation strategy to control the correction scale and resource overhead. The main sequence is as follows:

[0149] In the defined clock synchronization processing module, there is a global time base and local compensation Synchronization logic: When the micro virtual instrument platform finds significant drift, it will first call this module to estimate and correct the current clock offset again to generate the latest correction value and local compensation ;

[0150] If there is obvious multimodal data missing or abnormality in the self-correction trigger period, the data of this period will be re-interpolated or interpolated again through the interpolation correction algorithm; or the interpolation result suitable for the current drift situation will be regenerated;

[0151] When the drift metric function Exceeding the first level drift threshold (mild), only using local compensation , and make interpolation adjustments for a very small number of abnormal segments;

[0152] When the drift metric function The second level drift threshold (medium) is reached, and the corrected global time base is activated The refresh is combined with the interpolation module to correct all suspicious segments;

[0153] When the drift metric function If the drift exceeds the third level threshold (severe), the system enters full callback mode, which includes dynamically adjusting the distributed consistency module (if the micro virtual instrument platform is multi-device collaborative) and the core parameters of the cross-modal interpolation algorithm (such as kernel function and weight coefficients) to maximize correction accuracy.

[0154] When in use, through a hierarchical compensation strategy, corrections can be selectively performed in a resource-constrained micro-virtual instrument environment, avoiding unnecessary global large-scale realignment to ensure the continuous stability of multimodal data under a unified benchmark and avoid the occurrence of uncontrolled cumulative offsets.

[0155] Step 403: cyclic self-calibration result confirmation and status update

[0156] The corrected data and global time base are verified again and the results are written back to the key storage and monitoring structures within the micro virtual instrument platform, where:

[0157] Read the latest correction parameters 、 , image frame sequence generated after secondary interpolation for drift , audio sequence and sensor value sequence , recalibrate the benchmark event in the same way Compare the time in the data; if the newly generated drift metric function If the value has fallen back to within the threshold, the self-calibration is confirmed to be complete: if it is still out of limit, the compensation strategy can be repeatedly upgraded within the scope allowed by the resources of the micro virtual instrument platform;

[0158] The final confirmed correction value of the global time base and corrected local compensation Overwrite the original clock reference configuration and write the timestamp, trigger reason, execution details and other information of the self-correction operation into the time management unit to facilitate subsequent audit or troubleshooting;

[0159] For multimodal data correction, new multimodal data is generated, such as image frame sequence , audio sequence and sensor value sequence ), retain the difference information or differential mapping between the old version and the new version to support further historical tracing and correction and restoration needs;

[0160] After confirming that the drift has returned to the normal range, the micro-virtual instrument platform can enter a new monitoring cycle, thus forming a complete closed loop through cross-modal drift detection and abnormal threshold determination; if abnormal drift occurs again, the same sequence can be reused;

[0161] The validity of the self-calibration results is ensured through verification, and multiple rounds of calibration can be performed for residual offsets. The self-calibration operation records are linked to long-term information. This not only provides traceability at the micro-virtual instrument platform level, but also provides a data basis for possible subsequent drifts, continuously ensuring high-precision timing consistency of multimodal data during long-term operation.

[0162] The hierarchical compensation strategy can flexibly control the correction scope and resource allocation, and adapt to the power consumption and computing power limitations in the micro virtual instrument environment. The self-correction results are not only applied immediately to the time reference and data correction, but also achieve long-term traceability through the time management unit, providing auxiliary decision-making for the next round of detection and correction; it can ensure that even in long-term deployment or complex external environments, multimodal micro virtual instruments can still maintain a stable level of time alignment and retain relevant information in full.

[0163] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a division of some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0167] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A micro virtual instrument information acquisition method, characterized in that: include, When the micro-virtual instrument detects multimodal data input, it pre-processes high-contrast frames, audio pulses and mutation values ​​to generate a multimodal feature set with benchmark event markers; If an offset between the reference event and the external timing source is detected and needs to be corrected, the distributed consistency algorithm is called and the initial correction value is referenced to generate a global time reference and local compensation, and output the multimodal data alignment parameters; When an abnormal segment of multimodal data is detected according to the global time reference and the multimodal data alignment parameters, multimodal interpolation compensation is performed on the abnormal segment based on the weighted coupling function, and the data is finally written back to the multimodal data storage area; When significant drift is found between the newly collected data and the corrected benchmark event during long-term monitoring, the clock synchronization and interpolation correction are called back to update the global time benchmark and the local compensation and correction segments. The self-correction results are then recorded in the time management unit and overwrite the original benchmark. When the abnormal segment involves multiple modes that are dislocated or missing at the same time, a weighted coupling function is introduced. Constrain the correction results, where: ； Where: Indicates time Interpolation or correction results of multimodal data; As an evaluation An evaluation function for the quality or similarity of interpolation between modalities; For modal Used to measure the interpolation results in this mode. With reference data degree of difference or similarity; express About time The first-order partial derivative of The rate of change of similarity over time under each modality; is the weighting coefficient, is the mode weight in the overall coupling; is a positive real sensitivity factor, To process the window; By minimizing the weighted coupling function, we seek to synchronize the multimodal interpolation with the actual observation data, and ultimately keep the time sequence of each modal data segment consistent under the global time reference. The new frame, new sampling point or new data segment generated by interpolation correction is named in the same way as the original data object in the multimodal data storage area of ​​the micro virtual instrument, covering or marking the original abnormal segment.

2. The micro virtual instrument information acquisition method according to claim 1, characterized in that: The input interface of the micro virtual instrument obtains image sequences, audio sequences and sensor value sequences, and preprocesses them to obtain multimodal data, including preprocessed image frame sequences, audio sequences and sensor value sequences; The pre-processed multimodal data is subjected to feature analysis, and potential event candidate points are extracted by performing peak retrieval on wavelet transform at multiple scales to obtain a set of candidate events.

3. The micro virtual instrument information acquisition method according to claim 2, characterized in that: Conduct cross-modal correlation analysis on events of different modalities to screen key events. A time-domain overlap determination function is defined to determine whether events of different modalities can be considered as the same benchmark event. If there are multiple modal candidate events that meet the preset conditions, the sets are merged into the same benchmark event, and the aggregated benchmark event set and its corresponding timestamp are obtained and stored in the preliminary data buffer.

4. The micro virtual instrument information acquisition method according to claim 3, characterized in that: Compare the local time of each benchmark event with the corresponding external reference time to form an offset sequence; After obtaining the offset sequence, the optimal global offset is evaluated and solved. After obtaining the optimal global offset, it is used as the unified initial clock correction value of the micro virtual instrument relative to the external reference.

5. The micro virtual instrument information acquisition method according to claim 4, characterized in that: A simplified consensus protocol is executed between devices, periodically exchanging and comparing their respective optimal global offsets and observation data of some benchmark event times. After the protocol is completed, the global time reference and local compensation are obtained. Based on the corrected global time reference, the frame rate and frame arrival time are recalibrated, and after obtaining the image modality alignment parameters, they are written into the multimodal alignment parameter area.

6. The micro virtual instrument information acquisition method according to claim 5, characterized in that: The ideal sampling index of each modality under the unified time axis is first calculated for the multimodal alignment parameters; the deviation between the actual sampling time and the ideal sampling time is compared. If the deviation of a certain segment is greater than the established threshold, it is considered an abnormal delay segment; If continuous missing frames or missing sampling points are found in the data records of any modality, they are recorded as missing segments, and the judgment results are recorded as an abnormal data segment set; For index intervals that have been determined to be abnormally delayed or missing segments, their importance to subsequent analysis; if an abnormality occurs in a high-importance segment, focus on interpolation correction; if it is a low-importance segment or has been marked as a segment without key content, choose to skip interpolation to save resources.

7. A micro virtual instrument information acquisition method according to claim 6, characterized in that: Locate and obtain the actual observation time in the newly collected data, calculate the difference with the previously aligned or corrected reference time, and obtain the drift measurement function; The drift measurement function is compared with the preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has produced significant drift and needs to trigger self-correction. Otherwise, the current value is recorded and entered into the time management unit to update the recent drift level. No large-scale correction operation is performed for the time being.

8. The micro virtual instrument information acquisition method according to claim 7, characterized in that: When the micro-virtual instrument platform finds significant drift, it will first call to estimate and correct the current clock offset again to generate the latest correction value; If there is obvious multimodal data missing or abnormality in the self-correction trigger period, the data of this period will be interpolated again or interpolated again through the interpolation correction algorithm, or the interpolation result suitable for the current drift situation will be regenerated.

9. The micro virtual instrument information acquisition method according to claim 8, characterized in that: Read the latest correction parameters, perform secondary interpolation for drift, and generate multimodal data. Re-compare the time of the benchmark event in this data. If the newly generated drift metric function has fallen back within the threshold, the self-correction is confirmed to be complete: If the limit is still exceeded, the compensation strategy is repeatedly upgraded within the scope allowed by the resources of the micro virtual instrument platform; the final confirmed correction value of the global time base and the corrected local compensation are used to overwrite the original clock base configuration.

Images