Miniature virtual instrument information acquisition method

Abstract

The invention discloses a miniature virtual instrument information acquisition method, relates to the technical field of information acquisition, provides a multi-modal data acquisition and time alignment scheme for a resource-limited miniature virtual instrument environment, and comprises the following steps: 1, carrying out normalization processing on an image sequence, an audio sequence and a sensor value sequence, and extracting a reference event; 2, generating a global time reference and a multi-modal alignment parameter based on a distributed consistency algorithm and the initial correction amount; 3, performing interpolation correction on the abnormal data segments according to the alignment parameters, and remarkably improving the time sequence consistency; and step 4, dynamic monitoring and cyclic self-correction are carried out through a drift metric function, accurate synchronization of long-time operation is ensured, high precision and low resource occupation can be considered, and the method has significant robustness and adaptability, can be widely applied to industrial monitoring, intelligent terminals, the Internet of Things and the like, realizes deep fusion and reliable analysis of cross-modal data, and has good application prospects. And long-term time consistency can be maintained in multiple iterations.

Description

Technical Field

[0001] The present invention relates to the technical field of information acquisition, and specifically to a method for acquiring information of a micro virtual instrument. Background Art

[0002] In diversified scenarios such as Internet of Things monitoring, industrial process control, intelligent driving, and AR / VR, it is often necessary to simultaneously collect image frames, audio sequences, and various sensor values to comprehensively perceive the external environment and system status. However, due to the continuous expansion of the scale of application devices and the continuous enrichment of functions, the demand for real-time acquisition and analysis of multi-modal data sources is becoming increasingly prominent; especially in the micro virtual instrument environment, the performance of processors and storage resources are generally limited, but still need to face the high-frequency input of massive heterogeneous data.

[0003] In the above scenarios, the system not only needs to ensure the time synchronization and reliable acquisition between different modal data, but also needs to take into account requirements such as low power consumption, low bandwidth, and convenient deployment, which brings many challenges to data management and alignment.

[0004] In the patent application with the publication number CN112069484A, a method and system for information acquisition based on multi-modal interaction are disclosed. The information acquisition method includes: performing identity authentication based on the face image and voice segment of the person to be verified; pre-identifying the attribute information related to the person to be verified through the face image and voice segment of the person to be verified; obtaining the demand information of the person to be verified; determining the questionnaire content according to the demand information and the attribute information; presenting the questionnaire content to the person to be verified in a multi-modal interaction form to obtain the multi-modal questionnaire answers of the person to be verified; performing information fusion on the multi-modal questionnaire answers; and verifying the validity of the information according to the fusion result.

[0005] However, combining the actual application scenarios and the content in the prior art: Currently, in multi-modal data acquisition, the core technical problems mainly focus on the high-load conflict and clock drift accumulation during precise time correction and dynamic synchronization of cross-modal data.

[0006] Specifically, when the image and audio sampling periods are significantly different, or there are jitter errors between the sensor update frequency and the master clock, the multi-modal data will present a loose form that cannot be directly aligned; the micro virtual instrument is also prone to clock offset during continuous operation. Without a dedicated hierarchical synchronization processing and interpolation correction mechanism, the time stamps of the collected multi-modal data will inevitably drift or be missing, resulting in serious consequences such as inaccurate subsequent analysis results of the system, missed detection of key events, or poor interaction experience. Therefore, how to achieve precise acquisition and dynamic time alignment of multi-modal data under resource-constrained conditions has become a technical bottleneck that needs to be overcome urgently.

[0007] For this reason, the present invention provides a method for collecting information of a micro virtual instrument. Summary of the Invention

[0008] (I) Technical problems to be solved Aiming at the deficiencies of the prior art, the present invention provides a method for collecting information of a micro virtual instrument. For a micro virtual instrument environment with limited resources, the image sequence, audio sequence and sensor numerical sequence are normalized and the reference events are extracted; the global time reference and multi-modal alignment parameters are generated based on the distributed consistency algorithm and the initial correction amount; the abnormal data segments are interpolated and corrected according to the alignment parameters, significantly improving the timing consistency; through the drift measurement function, dynamic monitoring and cyclic self-correction are carried out to ensure accurate synchronization during long-term operation, taking into account high precision and low resource occupancy, with significant robustness and adaptability, and can be widely applied to industrial monitoring, intelligent terminals and the Internet of Things, etc., to realize the deep fusion and reliable analysis of cross-modal data, and solve the technical problems recorded in the background art.

[0009] (II) Technical solutions To achieve the above objectives, the present invention is realized through the following technical solutions: A method for collecting information of a micro virtual instrument, including, When the micro virtual instrument detects the input of multi-modal data, read the image sequence , audio sequence and sensor numerical sequence and other core objects, call the feature analysis module to detect the sudden change of picture brightness, audio pulse and sensor burst value and perform normalization preprocessing, remove noise and calibrate the recognizable reference events, and finally generate a set of feature information with unified time base marks in the preliminary data buffer area; When it is detected that the time offset between the reference event and the external or local time source needs to be calibrated, call the distributed consistency algorithm to execute the global time reference and local compensation amount compensation, and generate multi-modal alignment parameters of the image sequence , audio sequence , and sensor numerical sequence , and finally write them into the time management unit; When abnormal time delay or frame loss is detected in the image sequence , audio sequence , and sensor numerical sequence based on the global time reference and multi-modal alignment parameters and recorded as an abnormal data segment set and When marked, the interpolation correction module filters the priority according to the importance of the abnormal segment, then calls the weighted coupling function to sample correspondingly for interpolation to update the original data, and finally forms a valid sequence with consistent time series in the multi-modal data storage area; When running for a long time and detecting that there is a significant drift between the benchmark event in the newly collected data and the corrected data, and the drift amount exceeds the threshold, according to the hierarchical compensation strategy, first call back the clock synchronization module to generate the corrected global time reference and local compensation Then call the interpolation correction logic to review the suspicious section, and finally record the self-correction result in the time management unit and overwrite the original reference; Preferably, the input interface of the micro virtual instrument acquires image sequences, audio sequences and sensor value sequences, and preprocesses them to obtain multi-modal data, including the preprocessed image frame sequences audio sequences and sensor value sequences ; Preferably, perform feature analysis on the preprocessed multi-modal data to extract potential event candidate points, where: Adopt a feature detection model based on wavelet transform to better identify instantaneous pulses. To ensure the detection effect and calculation efficiency, define the following wavelet analysis expression:

[0010] In the formula: is the wavelet transform output in the modality ; represents the continuous signal form in the th modality; is the complex conjugate form of the selected wavelet basis function; is the scale parameter 0; is the translation parameter; By performing peak retrieval on the wavelet transform output at multiple scales, the candidate event set is obtained; , where represents the th event candidate detected in the th modality; Preferably, perform cross-modal correlation analysis on events of different modalities to screen key events , where the time-domain overlap determination function is defined to determine whether events between different modalities can be regarded as the same reference event:

[0011] In the formula: respectively refer to different modalities, For the time occurrence point of an event in a modality ; For the cross-modal event fusion determination threshold; If candidate events in multiple modalities satisfy , then merge this set into the same reference event . After aggregation, the set of reference events is denoted as . After completion of aggregation, the set of reference events and its corresponding timestamps are denoted as and stored in the preliminary data buffer; Preferably, if the deployment environment allows reception of high-precision signals , then compare the local time of each reference event with the corresponding external reference time to form an offset sequence , where:

[0012] If there is no available external reference, use a selected modality as an approximate reference locally to obtain the corresponding ; After obtaining the offset sequence , evaluate and solve for the optimal global offset , where:

[0013] Among them, is a positive real number representing the sensitivity coefficient to the offset error; is the total number of reference events; Through iterative search or analytical approximation methods, find the optimal global offset that minimizes the penalty function :

[0014] After obtaining the optimal global offset , it can be regarded as the unified initial clock correction amount of the micro virtual instrument relative to the external reference; The final optimal global offset together with the solution parameters of the penalty function are stored in the time management unit of the micro virtual instrument; Preferably, when multiple micro virtual instruments each obtain their local optimal global offsets , execute a simplified consensus protocol between devices to periodically exchange and compare their respective optimal global offsets and the observation data of some reference event times ; After the protocol is completed, the micro virtual instrument platform obtains a globally consistent correction value , where represents the global domain correction amount, and each is the local compensation amount; Based on the above global time reference and local compensation amount and other consistency results, it is further subdivided into modalities such as images, audio, and sensors as follows: On the basis of the corrected global time reference, the frame rate and frame arrival time are calibrated again to obtain the image modality alignment parameters and ; and the obtained and the multi-modal alignment parameters are written into the multi-modal alignment parameter area of the time management unit together; Preferably, for the multi-modal alignment parameters , first calculate the ideal sampling index of each modality on the unified time axis; compare the deviation between the actual sampling time and the ideal sampling time. If the deviation amount in a certain section is greater than the established threshold, it is regarded as an abnormal delay section; If continuous missing frames or missing sampling points are found in the data records of any modality, it is recorded as a missing section, and the judgment result is recorded as a set of abnormal data sections , which contains the start and end times and modality information of each abnormal section; Regarding the index interval determined to be an abnormal delay section or a missing section, its importance for subsequent analysis; if an abnormality occurs in a high-importance section, key interpolation correction is performed. If it is a low-importance section or a section marked as having no key content, interpolation can be skipped to save resources; Preferably, for the set of abnormal data sections , based on the multi-modal alignment parameters and the global time reference , corresponding interpolation algorithms are selected for different modalities and abnormal types respectively: When misalignment or missing occurs simultaneously in multiple modalities in the abnormal section, let and represent the combinations of interpolation results or correction sequences of different modalities at time respectively, is the similarity or difference metric function of the th modality. In the time interval , the following weighted coupling function is introduced to constrain the correction result, where:

[0015] In the formula: represents the interpolation or correction result of multi-modal data at time ; As used to evaluate the An evaluation function for the quality or similarity of modal interpolation; For a modal Used to measure the interpolation result in this modal And the reference data Of the difference or similarity; Denote The first-order partial derivative with respect to time Substantially measures the rate of change of similarity with time under the th modal; Is the weighting coefficient, the weight of the modal In the overall coupling; Is the sensitivity factor of positive real numbers, Is the processing window; By minimizing , seeking the correction result that makes the multi-modal interpolation synchronize with the actual observation data, and finally making the time series of each modal data segment consistent under the global time reference ; after completing the cross-modal interpolation correction, record the correction moment corresponding to the reference event As , and save it in the time management unit; The new frame, new sampling point or new data segment generated by the interpolation correction, in the same way as the naming of the original data object, covers or marks the original abnormal segment in the multi-modal data storage area of the micro virtual instrument; Preferably, read the reference event set And its timestamp , and combine the global time reference , the abnormal data segment set , locate the occurrence moments of these reference events in the newly acquired data to obtain the actual observation moment ; Calculate the difference between the actual observation moment And the previously aligned or corrected reference time . To avoid using simple mean difference or standard deviation, define the drift metric function As follows: Suppose there are Reference events in the current micro virtual instrument platform that need to be uniformly measured, and define the drift vector On the time interval :

[0016] Among them, Represents the drift amount of the th reference event evolving with time; for all A reference event can form a drift vector ; is a metric in the interval to measure the overall drift accumulation degree, and impose a stronger penalty on large fluctuations. A drift metric function is introduced:

[0017] In the formula: is a drift vector with dimension , which contains all reference events or real-time drift amounts of multimodality that the micro virtual instrument platform is concerned about; different represent different offsets of detection reference events; represents the first-order differential / gradient of the drift vector at time ; is a weight matrix with size ; is to select norm to measure the overall amplitude of W ; is a sensitivity factor / amplification factor of positive real numbers, is the time integration interval; Compare the drift metric function with a preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has generated significant drift and self-correction needs to be triggered: otherwise, the current value can be recorded and entered into the time management unit to update the recent drift level, and no large-scale correction operation is performed for the time being.

[0018] Preferably, when the micro virtual instrument platform discovers significant drift, it will first call this module to estimate and correct the current clock offset again to generate the latest correction value and local compensation ; If there are obvious multimodal data missing or anomalies during the self-correction trigger period, the data in this period is interpolated or frame interpolated again through the interpolation correction algorithm; or the interpolation result suitable for the current drift situation is regenerated; When the drift metric function exceeds the first-level drift threshold (mild), only local compensation is used, and a very small number of abnormal segments are interpolated and adjusted ; When the drift metric function reaches the second-level drift threshold (moderate), the refresh of the corrected global time reference is started, and all suspicious segments are corrected in combination with the interpolation module; When the drift metric function is higher than the third-level drift threshold (severe), it enters the full callback mode: including dynamically adjusting the core parameters (such as kernel functions, weight coefficients, etc.) of the distributed consistency module and the cross-modal interpolation algorithm again to maximize the correction accuracy; Preferably, verify the corrected data and the global time reference again, and write the results back to the key storage and monitoring structure in the micro virtual instrument platform, where: Read the latest correction parameters 、 、For the multi-modal data generated after quadratic interpolation for drift, re-compare the moment of the reference event in this data; if the newly generated drift metric function has fallen back within the threshold, it is confirmed that the self-correction is completed: if it is still out of limit, the upgrade compensation strategy can be repeated within the range allowed by the resources of the micro virtual instrument platform; The corrected value of the globally confirmed global time reference and the corrected local compensation overwrite the original clock reference configuration, and write information such as the timestamp, trigger reason, execution details, etc. of the self-correction operation in the time management unit, which is convenient for subsequent auditing or troubleshooting.

[0019] (III)Beneficial effects The present invention provides a method for collecting information of a micro virtual instrument, which has the following beneficial effects: Using the feature analysis module to perform normalization preprocessing on the image sequence, audio sequence and sensor numerical sequence, and extract the reference event, it can still eliminate noise and calibrate recognizable time anchors under resource-limited conditions; Combined with an external high-precision reference or a local stable timing source, the global time reference and local compensation are calculated through the distributed consistency algorithm and the optimal global offset, and differential alignment parameters are generated according to the characteristics of the image, audio and sensor modalities respectively, so that the synchronization overhead during subsequent interpolation correction is greatly reduced and the timing accuracy is guaranteed; Carry out cross-modal interpolation correction around the abnormal data segment set, not only eliminate missing or delay in a single modality, but also make the multi-modal data coincide with each other on the same time axis, reducing the uncertainty in subsequent fusion analysis; Monitor the deviation between the reference event and the collected data during long-term operation. Once it is found that the drift exceeds the limit, it will automatically callback to realize the re-update of the global time reference, local compensation and the interpolation result of the abnormal section. Through the hierarchical compensation strategy, the micro virtual instrument can dynamically decide whether to perform local repair or global refresh for different drift levels, so as to continuously maintain timing consistency in multiple cycles.

[0020] With limited computing power and storage resources of the micro virtual instrument, through the coordinated cooperation of multi-modal normalization feature extraction, hierarchical synchronous processing, cross-modal finger value correction, and long-term self-calibration, the problems of error accumulation and environmental fluctuation that are difficult to meet by hardware synchronization or single calibration in the traditional scheme are overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flow chart of a method for collecting information of a micro virtual instrument according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0023] Please refer to Figure 1 , the present invention provides a method for collecting information of a micro virtual instrument, including Step 1. When the micro virtual instrument detects the input of multi-modal data, read the image sequence , audio sequence and sensor value sequence and other core objects, call the feature analysis module to detect the sudden change of picture brightness, audio pulse and sudden sensor value, and perform normalization preprocessing, eliminate noise and calibrate recognizable reference events, and finally generate a set of feature information with unified time base marks in the preliminary data buffer area; Step 101. Preprocessing of multi-modal data and normalization of input sequence The input interface of the micro virtual instrument acquires the image sequence, audio sequence and sensor value sequence, and preprocesses them to obtain multi-modal data, specifically including: Unify the size and chromaticity of image data with different resolutions or color spaces to generate a sequence of preprocessed image frames ; at a unified sampling rate , perform amplitude normalization on the audio signal to obtain a normalized audio sequence ; Convert the outputs of different types of sensors (such as temperature, pressure, acceleration, etc.) into a unified data format to form a sensor value sequence ; Through normalization processing, it is ensured that when extracting features from image, audio and sensor data subsequently, features can be compared and screened under a unified coordinate or dimension.

[0024] Step 102, Multimodal Feature Analysis and Event Candidate Extraction Perform feature analysis on the preprocessed multimodal data to extract potential event candidate points, where: Adopt a feature detection model based on wavelet transform to better identify instantaneous pulses (common in audio and sensor sequences) or high-frequency mutations (which may occur in image and sensor sequences). To ensure the detection effect and computational efficiency, define the following wavelet analysis expression:

[0025] In the formula: is the wavelet transform output on modality ; represents the continuous signal form in the th modality (for images, frame signals can be extracted in the time dimension, for audio it is the time-domain waveform, and for sensors, the discrete samples can be approximated as a continuous function after interpolation); is the complex conjugate form of the selected wavelet basis function; is the scale parameter, and its value range is dynamically adjusted according to the actual signal frequency band, with a value greater than 0; is the translation parameter, between 0 and T, used to scan the feature distribution of the signal on the time axis; By performing peak retrieval on the wavelet transform output at multiple scales, obtain the candidate event set ; , where represents the th event candidate detected in the th modality; According to the different modality types, the threshold setting and peak judgment conditions can be configured differently, but the core is to judge the possible instantaneous mutations in each modality (such as audio pulses, acceleration spikes, brightness flickers in images, etc.); Using wavelet transform to replace the common time-window sliding or Fourier transform methods can capture instantaneous features in a wider frequency band and has better noise suppression ability.

[0026] Step 103, Cross-modal Benchmark Event Screening and Moment Calibration Based on the candidate event set , perform cross-modal correlation analysis on events of different modalities to screen out key events that can occur simultaneously or approximately simultaneously in multiple modalities , and define a time-domain overlap judgment function to determine whether events between different modalities can be regarded as the same benchmark event:

[0027] Wherein: respectively represent different modalities (such as image modality, audio modality or a certain sensor modality), is the time occurrence point of the event in the modality ; is the cross-modal event fusion determination threshold, which is set according to the clock accuracy of the micro virtual instrument or the real-time requirement of the application scenario; By sequentially calculating the time domain overlap determination results between each modality, if the candidate events of multiple modalities satisfy , then the set is combined into the same reference event , and its final calibration time can take the weighted average or uniformly select the timestamp value of a certain preferred modality. The aggregated reference event set is denoted as . After aggregation, the reference event set and its corresponding timestamp are denoted as , and stored in the preliminary data buffer; The key events screened through cross-modal correlation analysis and time domain overlap determination function have significant features that can be detected in multiple modalities, and can better serve as the alignment basis for the global time reference. Fusing the key events of multiple modalities into a unified reference event helps to achieve more accurate alignment parameter calculation and improve the reliability of multi-modal collaboration.

[0028] Step 2: When it is necessary to calibrate the time offset between the reference event and the external or local timing source, call the distributed consistency algorithm to execute the global time reference and the local compensation amount for compensation, and generate the multi-modal alignment parameters of the image sequence , audio sequence , and sensor value sequence , and finally write them into the time management unit; Step 201: Initial clock offset estimation and external reference correction Based on the reference event set and the corresponding timestamps , estimate the initial clock offset of the micro virtual instrument platform, and perform reference correction in the presence of an external high-precision reference signal (such as GPS, PPS, laser pulse, etc.). The specific processing includes: If the deployment environment allows receiving high-precision signals , then compare the local time of each reference event with the corresponding external reference time to form an offset sequence , where: ; If there is no available external reference, a selected mode (or a stable reference timing source) is used locally as an approximate reference to obtain the corresponding , and this offset sequence records the difference between the local reference event of the micro virtual instrument and the external reference on the time axis; after obtaining the offset sequence , the following exponential penalty function is used to evaluate and solve the optimal global offset in order to uniformly correct all reference events:

[0029] where is a positive real number representing the sensitivity coefficient to the offset error; is the total number of reference events; Through iterative search or analytical approximation methods, the optimal global offset that minimizes the penalty function Ω is obtained:

[0030] After obtaining the optimal global offset , it can be regarded as the unified initial clock correction amount of the micro virtual instrument relative to the external reference (or local reference) to eliminate large-scale time offsets; the final optimal global offset , together with the solution parameters of the penalty function Ω , are stored in the time management unit of the micro virtual instrument for fine calculation of distributed consistency or multi-modal alignment parameters.

[0031] When in use, through the exponential penalty function Ω , when there is an external reference signal, the overall time reference can be actively aligned to a high-precision clock; if the external reference is not available, preliminary offset correction can also be completed based on the local stable source; Step 202, Distributed Consistency Collaboration and Multi-modal Alignment Parameter Generation Further perform distributed or multi-channel consistency processing on the local clocks that have completed the initial offset correction to generate alignment parameters applicable to subsequent interpolation correction. Here, it mainly faces multi-device collaboration or multi-thread concurrent scenarios, specifically including: When multiple micro virtual instruments each obtain their local optimal global offsets , to ensure the consistency of the overall time reference, a simplified consensus protocol can be executed between devices to periodically exchange and compare their respective optimal global offsets and the observation data of some reference event times ; After the protocol is completed, the micro virtual instrument platform obtains the global consistency correction value , , },in Represents the global time base, each is the local compensation amount; Based on the above global time base And local compensation The consistency results are further subdivided into image, audio, and sensor modalities, as follows: Based on the corrected global time reference, the frame rate and frame arrival time are recalibrated to obtain the image modality alignment parameters. : Combined with sampling frequency And preliminary corrections to ensure that the audio data is synchronized to the global time with high precision and obtain the audio mode alignment parameters ; Based on the characteristics of each sampling period of the sensor, a timing correction scheme suitable for resource-constrained environments is derived, and Linkage control, obtain sensor modal alignment parameters ; Calculated And the multimodal alignment parameters The multimodal alignment parameter area of ​​the time management unit is written together; Through the distributed consistency protocol, the global time base of multiple micro virtual instruments or multi-threads is unified, reducing the accumulation of clock drift between nodes and refining the multi-modal alignment parameters. Different synchronization frequencies and computing costs can be allocated according to modal characteristics to suit resource-constrained scenarios. Perform rebalancing or distributed consensus processing to ensure high-precision and robust time synchronization at both macro and micro levels.

[0032] Step 3: Based on the global time base Multimodal alignment parameters detected in image sequences , audio sequence and sensor value sequence There is abnormal delay or missing frame and it is recorded as abnormal data segment collection When marked, the priority is screened according to the importance of the abnormal segment in the interpolation correction module, and then the weighted coupling function is called to interpolate and update the original data corresponding to the sampling, and finally a valid sequence with consistent time sequence is formed in the multimodal data storage area; Step 301: Identification of abnormal data segments and determination of interpolation requirements Based on the global time base and multimodal alignment parameters , identify potential abnormal delays or missing data segments from the multimodal data storage area of ​​the micro virtual instrument, and prepare for subsequent interpolation correction. The main processing flow is as follows: For multimodal alignment parameters (e.g. Corresponding image frame, Corresponding audio frame, Corresponding to sensor sampling), first calculate the ideal sampling index of each modality under the same time axis, for the image sequence , we can order:

[0033] For audio sequence and sensor sequence Similar ideal sampling times can also be defined separately and , by adding a global time reference and local compensation After that, the expected sampling position under the global time reference is obtained; Compare the deviation between the actual sampling time (stored in the data storage area) and the ideal sampling time. If the deviation of a certain section is greater than a predetermined threshold (set by the micro-virtual instrument platform requirements or application scenarios), it is considered an abnormal delay section. If continuous missing frames or missing sampling points are found in the data records of any mode, they are recorded as missing segments, and the judgment results are recorded as abnormal data segment sets. , which contains the start and end time and modal information of each abnormal segment; For index intervals that have been determined to be abnormal delay segments or missing segments, the user or the trained neural network can determine their importance for subsequent analysis; if an abnormality occurs in a high-importance segment, it is necessary to perform focused interpolation correction; if it is a low-importance segment or has been marked as a segment without key content, you can choose to skip interpolation to save resources; through the coordinated use of alignment parameters and the global time base, rapid discovery of abnormal delays or data missing is achieved, and the ability to perceive the quality of multimodal data is enhanced. The abnormal segments and missing segments are classified and processed separately, so that subsequent interpolation correction can allocate computing resources in a targeted manner, ensuring that the micro virtual instrument can still maintain the consistency of the overall timing in a resource-constrained environment.

[0034] Step 302: Multimodal interpolation correction and data level alignment execution For abnormal data segment collection , based on the multimodal alignment parameters and global time base , perform targeted cross-modal interpolation correction to improve the matching degree between actual data records and theoretical time series. The main processing process is as follows: For different modes and anomaly types, select corresponding interpolation algorithms respectively: Image modality : Adopt a temporal interpolation method based on a multi-dimensional interpolation kernel, and the kernel function can be defined to measure the similarity between adjacent frames; Audio modality : When a missing segment is detected, weighted phase compensation interpolation can be selected to maintain spectral continuity; Sensor modality : Commonly use multi-point interpolation or piecewise polynomial fitting, especially suitable for smoothing mutations or extreme peak position drifts; When abnormal segments involve misalignment or missing of multiple modalities simultaneously, let and respectively represent the combination of interpolation results or correction sequences of different modalities at time (which can be regarded as the evolution of the data of all modalities combined into a multi-dimensional vector over time), is the similarity or difference metric function for the th modality (defined previously, and its specific form can be based on image local texture, audio phase, or sensor curve, etc.). In the time interval , introduce the following weighted coupling function to constrain the correction results, where:

[0035] In the formula: represents the interpolation or correction result of multi-modal data at time , which can be regarded as the distribution of high-dimensional vectors over time; As an evaluation function used to evaluate the interpolation quality or similarity of the th modality, evaluate the similarity of image frames, the coincidence degree of audio spectra, the matching degree of sensor trend curves, etc.; For modality used to measure the difference or similarity between the interpolation result and the reference (or true) data in this modality; represents the first-order partial derivative with respect to time , and essentially measures the change rate of similarity (or difference) with respect to time under the th modality; Corresponding to the original weighting coefficient, it is the weight of modality in the overall coupling, reflecting the influence degree of this modality on the target of the micro virtual instrument platform, and the value range is positive real numbers; is a sensitivity factor of positive real numbers, used to amplify (or suppress) the modality differences with a relatively high change rate over time, is the processing window; By minimizing , seeking a correction result that synchronizes multimodal interpolation to fit the actual observed data, and finally making the timing of each modal data segment consistent under the global time reference ; after completing the cross-modal interpolation correction, record the correction time corresponding to the reference event as , and save it in the time management unit; For the new frames, new sampling points, or new data segments generated by interpolation correction, overwrite or label the original abnormal segments in the multimodal data storage area of the micro virtual instrument in the same way as the original data object naming; at the same time, register the kernel function, weight coefficient, and interpolation interval used in the interpolation operation in the time management unit; When in use, select a differentiated interpolation strategy for different modalities such as images, audio, and sensors to improve the accuracy and stability of the interpolation results. By coupling the multimodal interpolation process, ensure that multiple modalities are consistent on the same time reference, rather than just the separate storage of independent correction data records and interpolation parameters for single modalities, which is convenient for quickly locating the corrected segments and performing re-calibration or backtracking operations.

[0036] Step 4: When running for a long time and detecting that there is a significant drift between the reference event in the newly acquired data and the corrected data, and the drift amount exceeds the threshold, first call back the clock synchronization module to generate a corrected global time reference and local compensation , then call the interpolation correction logic to review the suspicious segments, and finally record the self-calibration result in the time management unit and overwrite the original reference; Step 401: Cross-modal drift detection and abnormal threshold determination. According to the latest data (denoted as , representing the correction sequences of images, audio, and sensors respectively) stored in the multimodal data storage area of the micro virtual instrument after interpolation correction, compare the newly acquired data of each modality (denoted as ); Identify potential drifts accumulated over time. The main process is as follows: Read the reference event set and its time stamps , and combine with the global time reference , the abnormal data segment set , locate the occurrence times of these reference events in the newly acquired data to obtain the actual observation times ; Calculate the difference between the actual observation times and the previously aligned or corrected reference times . To avoid using simple mean differences or standard deviations, define a drift metric function , as follows: Suppose there are benchmark events (or multiple modal offsets) in the current micro virtual instrument platform that need to be uniformly measured, and the drift vector defined on the time interval :

[0037] where represents the drift amount (or offset function) of the th benchmark event evolving over time, which can be obtained by interpolating or smoothing the sequence of differences between the newly acquired observation time and the reference time; for all benchmark events, can be composed into the drift vector ; To measure the overall drift accumulation degree of in the interval and impose a stronger penalty on large fluctuations, the drift metric function is introduced:

[0038] In the formula: is the drift vector of dimension , which contains the real-time drift amounts of all benchmark events or multiple modalities that the micro virtual instrument platform is concerned about; different represent different detected benchmark event offsets; represents the first-order differential / gradient of the drift vector at time . If is discrete sampling, it can be approximated by difference or interpolation methods; is the weight matrix of size , which is used to distinguish the importance or coupling relationship of different benchmark events in the vectorized measurement; the diagonal element can amplify / diminish the influence of the th drift component, and the non-diagonal elements can be used to describe the cross-constraints or mutual influences between different events (if no coupling is required, it can also be set as a diagonal matrix); is to select norm to measure the overall amplitude of W . Common values are , 2, or the infinity norm; is the sensitivity factor / amplification factor of positive real numbers, is the time integration interval, which often selects the current monitoring or evaluation time window; The drift metric function Compare with the preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has generated significant drift and self-calibration needs to be triggered; otherwise, the current value can be recorded and entered into the time management unit to update the recent drift level, and large-scale calibration operations are not performed temporarily.

[0039] When in use, by comparing the moments of the reference event in the old and new data, the time offset in the multi-modal mode can be quickly detected, so that the large drift points obtain higher weights and thus trigger self-calibration more effectively. Combine the threshold determination with the time management unit to maintain continuous perception of the drift of the micro virtual instrument platform.

[0040] Step 402, Self-calibration Trigger and Hierarchical Compensation Strategy When it is determined that the drift metric exceeds the limit, the micro virtual instrument platform will automatically call back the previous clock synchronization processing and interpolation correction module, and at the same time formulate a hierarchical compensation strategy to control the correction scale and resource overhead. The main sequence is as follows: In the defined clock synchronization processing module, there is a synchronization logic for the global time reference and the local compensation amount : When the micro virtual instrument platform detects significant drift, it will first call this module to estimate and correct the current clock offset again to generate the latest correction value and local compensation ; If there are obvious multi-modal data missing or anomalies during the self-calibration trigger period, the data in this period will be interpolated or frame-inserted again through the interpolation correction algorithm; or regenerate the interpolation result suitable for the current drift situation; When the drift metric function exceeds the first-level drift threshold (mild), only local compensation is used, and a very small number of abnormal segments are interpolated and adjusted; When the drift metric function reaches the second-level drift threshold (moderate), the refresh of the corrected global time reference is started, and all suspicious segments are corrected in combination with the interpolation module; When the drift metric function is higher than the third-level drift threshold (severe), it enters the full callback mode: including dynamically adjusting the core parameters (such as kernel function, weight coefficient, etc.) of the distributed consistency module (if the micro virtual instrument platform is multi-device collaborative) and the cross-modal interpolation algorithm to maximize the correction accuracy; When in use, through the leveled compensation strategy, corrections can be selectively performed in the resource-constrained micro virtual instrument environment, avoiding unnecessary global large-scale realignment, ensuring the continuous stability of multi-modal data under the unified reference, and avoiding the situation of out-of-control cumulative offset.

[0041] Step 403, Loop for Self-Calibration Result Confirmation and Status Update Verify the corrected data and the global time reference again, and write the results back to the key storage and monitoring structure within the micro virtual instrument platform, where: Read the latest correction parameters , , the image frame sequence generated after quadratic interpolation for drift , audio sequence and sensor value sequence , and compare the time of the reference event in this data in the same way; if the newly generated drift metric function has fallen back within the threshold, confirm that the self-calibration is complete: if it is still over the limit, the upgrade compensation strategy can be repeated within the scope allowed by the resources of the micro virtual instrument platform; The corrected value of the finally confirmed global time reference and the corrected local compensation overwrite the original clock reference configuration, and write information such as the timestamp, trigger reason, and execution details of the self-calibration operation in the time management unit, which is convenient for subsequent auditing or troubleshooting; For the new multi-modal data generated by the correction of multi-modal data, such as the image frame sequence , audio sequence and sensor value sequence ), retain the difference information or differential mapping between the old version and the new version to support further historical traceability and correction restoration requirements; After confirming that the drift has returned to the normal range, the micro virtual instrument platform can enter a new monitoring cycle, thus forming a complete loop through cross-modal drift detection and abnormal threshold determination; if abnormal drift occurs again, the same sequence can still be reused; Ensure the effectiveness of the self-calibration result through the verification link, and perform multiple rounds of correction for the residual offset. Link the self-calibration operation record with the long-cycle information, which not only has traceability at the micro virtual instrument platform level, but also provides data basis for possible multiple drifts in the future, continuously ensuring the high-precision timing consistency of multi-modal data during long-term operation; The hierarchical compensation strategy can flexibly control the correction range and resource allocation, adapt to the power consumption and computing power limitations in the micro virtual instrument environment. The self-calibration result is not only immediately applied to the time reference and data correction, but also realizes long-cycle traceability through the time management unit, providing auxiliary decision-making for the next round of detection and correction; it can ensure that even in long-term deployment or complex external environments, the multi-modal micro virtual instrument can still maintain a stable time alignment level and retain all relevant information comprehensively.

[0042] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0043] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0044] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only for some logical function divisions. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0045] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0046] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A method for collecting information of a micro virtual instrument, characterized in that: including When the micro virtual instrument detects multi-modal data input, it preprocesses high-contrast frames, audio pulses, and mutation values to generate a multi-modal feature set with reference event markers; If it is detected that the reference event needs to be corrected for the offset from the external timing source, a distributed consensus algorithm is called and the initial correction amount is referenced to generate a global time reference and local compensation, and the multi-modal data alignment parameters are output; Based on the global time reference and multi-modal data alignment parameters, when an abnormal segment of multi-modal data is detected, multi-modal interpolation compensation is performed on the abnormal section based on a weighted coupling function, and finally written back to the multi-modal data storage area; When a significant drift is found between the newly acquired data and the corrected reference event during long-term monitoring, the clock synchronization and interpolation correction are called back to update the global time reference, local compensation, and corrected section, and then the self-correction result is recorded in the time management unit and the original reference is overwritten.

2. A method for collecting information of a micro virtual instrument according to claim 1, characterized in that: The input interface of the micro virtual instrument acquires an image sequence, an audio sequence, and a sensor value sequence, and preprocesses them to obtain multi-modal data, including the preprocessed image frame sequence, audio sequence, and sensor value sequence; Feature analysis is performed on the preprocessed multi-modal data. By performing peak retrieval on the wavelet transform at multiple scales, potential event candidate points are extracted to obtain a candidate event set.

3. A method for collecting information of a micro virtual instrument according to claim 2, characterized in that: Cross-modal correlation analysis is performed on events of different modalities to screen key events. Among them, a time-domain overlap determination function is defined to determine whether events between different modalities can be regarded as the same reference event; If the candidate events of multiple modalities meet the preset conditions, the set is combined into the same reference event, and the aggregated reference event set and its corresponding time stamp are stored in the preliminary data buffer.

4. A method for collecting information of a micro virtual instrument according to claim 3, characterized in that: The local time of each reference event is compared with the corresponding external reference moment to form an offset sequence; After obtaining the offset sequence, the optimal global offset is evaluated and solved. After obtaining the optimal global offset, it is used as the unified initial clock correction amount of the micro virtual instrument relative to the external reference.

5. A method for collecting information of a micro virtual instrument according to claim 4, characterized in that: A simplified consensus protocol is executed between devices to periodically exchange and compare their respective optimal global offsets and observation data of the moments of some reference events. After the protocol is completed, a global time reference and local compensation amount are obtained; Based on the corrected global time reference, the frame rate and frame arrival time are calibrated again. After obtaining the image modality alignment parameters, they are written into the multi-modal alignment parameter area.

6. A method for collecting information of a micro virtual instrument according to claim 5, characterized in that: For the multi-modal alignment parameters, the ideal sampling index of each modality under the unified time axis is calculated first; the deviation between the actual sampling moment and the ideal sampling moment is compared. If the deviation amount of a certain section is greater than the established threshold, it is regarded as an abnormal time delay section; If continuous missing frames or missing sampling points are found in the data records of any modality, they are recorded as missing segments, and the determination result is recorded as a set of abnormal data segments; Regarding the index intervals that have been determined to be abnormal delay segments or missing segments, their importance for subsequent analysis; if an abnormality occurs in a high-importance section, key interpolation correction is performed. If it is a low-importance section or a section that has been marked as having no key content, interpolation is skipped to save resources.

7. A method for collecting information of a micro virtual instrument according to claim 6, characterized in that: When abnormal segments involve misalignment or missing of multiple modalities simultaneously, the following weighted coupling function is introduced to constrain the correction result. By minimizing the weighted coupling function, a correction result that makes the multi-modal interpolation synchronously fit the actual observed data is sought, and finally, the data segments of each modality are kept in consistent time sequence under the global time reference; The new frames, new sampling points or new data segments generated by interpolation correction are named in the same way as the original data object and are used to overwrite or mark the original abnormal segments in the multi-modal data storage area of the micro virtual instrument.

8. A method for collecting information of a micro virtual instrument according to claim 7, characterized in that: Locate and obtain the actual observation time in the newly collected data, calculate the difference degree with the reference time recorded after previous alignment or correction, and obtain the drift metric function; Compare the drift metric function with a preset threshold. If it exceeds the drift threshold, it is determined that the micro virtual instrument platform has generated significant drift and self-calibration needs to be triggered. Otherwise, record the current value and enter the time management unit to update the recent drift level, and no large-scale calibration operation is performed temporarily.

9. A method for collecting information of a micro virtual instrument according to claim 8, characterized in that: When the micro virtual instrument platform detects significant drift, it will be called first to estimate and correct the current clock offset again to generate the latest correction value; If there are obvious multi-modal data missing or abnormalities during the self-calibration trigger period, the data in this period is interpolated or frame-inserted again through the interpolation correction algorithm, or an interpolation result suitable for the current drift situation is regenerated.

10. A method for collecting information of a micro virtual instrument according to claim 9, characterized in that: Read the latest correction parameters and the multi-modal data generated after secondary interpolation for drift, and re-compare the time of the reference event in this data; if the newly generated drift metric function has fallen back within the threshold, confirm that the self-calibration is completed: If it still exceeds the limit, repeat the upgrade compensation strategy within the range allowed by the resources of the micro virtual instrument platform; overwrite the original clock reference configuration with the correction value of the finally confirmed global time reference and the corrected local compensation.

Images