Asynchronous multi-modal data timestamp alignment and data frame construction and validation method

Abstract

The application discloses an asynchronous multi-modal data timestamp alignment, data frame construction and verification method, belongs to the field of asynchronous multi-modal data fusion, and comprises the following steps: acquiring a plurality of data streams which are asynchronous with each other; constructing a reference time sequence; for each reference time point in the reference time sequence, performing nearest neighbor timestamp matching in a plurality of data streams respectively, finding a data point which is closest in time distance to the reference time point, and obtaining a matching index and a time deviation corresponding to the plurality of data streams; calculating a maximum value of the time deviation corresponding to each reference time point to obtain a maximum time deviation; if the maximum time deviation is less than or equal to a preset time tolerance window, it is determined that the multi-modal data frame corresponding to the reference time point is successfully aligned; and combining data points corresponding to the reference time point in the plurality of data streams into a multi-modal data frame. The technical effect of the application is that the multi-modal data frame is formed conveniently.

Description

Technical Field

[0001] This invention relates to asynchronous multimodal data fusion, and more particularly to a method for asynchronous multimodal data timestamp alignment, data frame construction, and verification. Background Technology

[0002] With the development of online monitoring and intelligent analysis technologies, multi-parameter and multi-modal collaborative detection has become an important technological direction in fields such as environmental monitoring and industrial process control. Taking water quality monitoring as an example, water quality assessment usually relies on the joint judgment of multiple indicators such as pH, conductivity, turbidity, and spectrum. Sensors with different detection mechanisms characterize the state of water bodies from dimensions such as electrochemistry, optical scattering, and spectral absorption, forming asynchronous multi-modal data streams.

[0003] Chinese invention patent, publication number CN117294376A, publication date 2023-12-26, discloses a method, apparatus, electronic device, and storage medium for multimodal data timestamp synchronization, belonging to the field of communication technology. The method includes: receiving standardized transmission packets encapsulated by a standardized protocol for each modality of multimodal data, wherein the standardized transmission packets carry timestamps; determining the absolute timestamps of each modality of data based on the timestamps carried in the standardized transmission packets; and synchronizing the multimodal data based on the absolute timestamps of each modality of data. This application can improve the compatibility and universality of multimodal data timestamp synchronization and increase synchronization efficiency.

[0004] The shortcoming of this invention is that it lacks a reference time series, which makes it difficult to establish a one-to-one sample index relationship between multimodal data and to form multimodal data frames. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a method for asynchronous multimodal data timestamp alignment, data frame construction and verification, which facilitates the formation of multimodal data frames.

[0006] Technical solution:

[0007] Methods for asynchronous multimodal data timestamp alignment, data frame construction, and verification include:

[0008] Acquire multiple asynchronous data streams;

[0009] Construct a reference time series;

[0010] For each reference time point in the reference time series, nearest neighbor timestamp matching is performed in multiple data streams to find the data point with the closest time distance to the reference time point, thereby obtaining the matching index and time deviation corresponding to the multiple data streams;

[0011] Calculate the maximum time deviation corresponding to each of the reference time points to obtain the maximum time deviation. If the maximum time deviation is less than or equal to a preset time tolerance window, it is determined that the multimodal data frame corresponding to the reference time point is successfully aligned.

[0012] The data points corresponding to the reference time point in multiple data streams are combined into a multimodal data frame;

[0013] Construct a timestamp alignment evaluation index to quantitatively verify the timestamp alignment effect.

[0014] Specifically, the asynchronous multimodal data timestamp alignment, data frame construction and verification method of the present invention facilitates the establishment of a one-to-one corresponding sample index relationship between multiple data streams by constructing a reference time series, thereby facilitating the formation of multimodal data frames.

[0015] Optionally, constructing the reference time series includes:

[0016] A reference time axis is constructed using a fixed sampling interval;

[0017] Alternatively, the timestamp of one of the data streams can be used as a reference time stream.

[0018] Optionally, the plurality of data streams include a pH data stream, a conductivity data stream, a turbidity data stream, and a spectral data stream. When the timestamp of a certain data stream is used as a reference time stream, the timestamp of the conductivity data stream is selected as the reference time stream.

[0019] Optionally, the preset time tolerance window is half the average sampling period of the conductivity data stream.

[0020] Optionally, the nearest neighbor timestamp matching is calculated using the following formula to obtain the matching index:

[0021] ,

[0022] in, This is the matching index for the k-th reference time point corresponding to the m-th type of data stream; This means traversing and searching within the range of values ​​for j, and taking the index corresponding to the minimum value; The timestamp of the j-th data point in the m-th data stream; This is the kth reference time point.

[0023] Optionally, the time deviations corresponding to the multiple data streams are calculated using the following formula:

[0024] ,

[0025] in, The time deviation of the m-th type of data stream at the k-th reference time point; It is the timestamp of the data point that best matches the k-th reference time point in the m-th data stream.

[0026] Optionally, the multimodal data frame can be represented as:

[0027] ,

[0028] ,

[0029] in, This is the k-th multimodal data frame; The pH value after alignment in the k-th frame; This represents the conductivity value after alignment in the k-th frame. This represents the turbidity value after alignment in the k-th frame. This represents the absorbance value at the i-th wavelength after alignment of the k-th frame. This represents the wavelength points of the spectrum, where n is the total number of wavelengths. This represents the measurement that best matches the k-th reference time point in the pH data stream; This represents the measured value of the k-th data point in the conductivity data stream; This represents the measurement that best matches the k-th reference time point in the turbidity data stream; This represents the absorbance vector that best matches the k-th reference time point in the spectral data stream.

[0030] Optionally, the final multimodal dataset can be represented as:

[0031] ,

[0032] in, This represents the entire aligned multimodal dataset; The total number of successfully aligned multimodal data frames.

[0033] Optionally, the timestamp alignment evaluation metrics include the average sampling period of the multiple data streams, the average maximum time deviation, the global maximum time deviation, the time deviation standard deviation, and the alignment success rate.

[0034] Optionally, the alignment success rate is calculated using the following formula:

[0035] ,

[0036] in, Indicates the alignment success rate; This represents the total number of reference time points.

[0037] Beneficial effects:

[0038] (1) The asynchronous multimodal data timestamp alignment and data frame construction and verification method of the present invention facilitates the establishment of a one-to-one sample index relationship between multiple data streams by constructing a reference time series, thereby facilitating the formation of multimodal data frames.

[0039] (2) The asynchronous multimodal data timestamp alignment, data frame construction and verification method of the present invention combines low-dimensional scalar parameters and high-dimensional vector parameters in the same sample in the multimodal data frame, so that subsequent algorithms can directly perform feature-level fusion, decision-level fusion or spectral compensation modeling. Compared with simply exporting multiple CSV files separately, the asynchronous multimodal data timestamp alignment, data frame construction and verification method of the present invention makes it easier to clarify the sample correspondence between different data streams, and transforms multiple asynchronous data streams from "independent sensor results" into "data structures that can be directly used for fusion".

[0040] (3) The asynchronous multimodal data timestamp alignment and data frame construction and verification method of the present invention facilitates quantitative verification of timestamp alignment effect by constructing timestamp alignment evaluation indicators such as average sampling period of multiple data streams, average maximum time deviation, global maximum time deviation, time deviation standard deviation and alignment success rate. Attached Figure Description

[0041] Figure 1 This is the flowchart of the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of the present invention. Figure 1 ;

[0042] Figure 2 This is the flowchart of the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of the present invention. Figure 2 ;

[0043] Figure 3 This is a diagram showing the calibration effect of the pH module of this invention;

[0044] Figure 4 This is a calibration effect diagram of the conductivity module of the present invention;

[0045] Figure 5 This is a diagram showing the calibration effect of the turbidity module of the present invention;

[0046] Figure 6 This is the light source spectrum diagram of the present invention;

[0047] Figure 7 This is the transmittance spectrum of the present invention;

[0048] Figure 8 This is the absorbance spectrum of the present invention;

[0049] Figure 9 This is a timestamp-aligned timeline diagram of the present invention. Detailed Implementation

[0050] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. The terms "first," "second," etc., used in this invention are for the convenience of describing the technical solutions of the invention and have no specific limiting effect; they are all general terms and do not constitute a limitation on the technical solutions of the invention. It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, not to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Multiple technical solutions in the same embodiment, as well as multiple technical solutions in different embodiments, can be arranged and combined to form new technical solutions that do not contradict or conflict, all of which are within the scope of protection claimed by this invention.

[0052] Example 1

[0053] like Figures 1-2 This embodiment provides a method for asynchronous multimodal data timestamp alignment, data frame construction, and verification, including:

[0054] Step S1: Acquire multiple asynchronous data streams; the multiple data streams include pH data stream, conductivity data stream, turbidity data stream and spectral data stream.

[0055] Specifically, pH, conductivity, and turbidity data streams are all low-dimensional scalar parameters, while spectral data streams are high-dimensional vector parameters. The different data streams can be represented as:

[0056] ,

[0057] in, This represents the original sampled dataset of the m-th data stream; This represents the timestamp of the i-th data point in the m-th data stream; Indicates the timestamp of the m-th type of data stream. The collected measurement values; This represents the number of data points in the m-th type of data stream.

[0058] For pH, conductivity, and turbidity data streams, the measured values ​​are scalars and can be expressed as follows:

[0059] ,

[0060] in, This represents the measurement value collected at the corresponding timestamp for the i-th data point in the pH data stream; This represents the i-th pH value; This represents the measured value of the i-th data point in the conductivity data stream at the corresponding timestamp; This represents the i-th conductivity value; This represents the measurement value collected at the corresponding timestamp for the i-th data point in the turbidity data stream; This represents the i-th turbidity value;

[0061] For a spectral data stream, the measured value is an absorbance vector, which can be represented as:

[0062] ,

[0063] in, This represents the measurement value of the i-th data point in the spectral data stream at the corresponding timestamp; This represents the absorbance vector obtained from the i-th spectral acquisition; This represents the wavelength points of the spectrum, where n is the total number of wavelengths.

[0064] Because the sampling periods of different data streams are different, the four types of data streams usually do not generate data points at exactly the same time. Therefore, if the CSV files of each data stream are exported separately, the subsequent fusion algorithm cannot directly determine the sample correspondence between different data streams. To solve this problem, the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of this invention uses timestamps as the basis for data association between different data streams.

[0065] Step S2, construct a reference time series; constructing a reference time series includes: constructing a reference time axis with a fixed sampling interval; or, using the timestamp of a certain data stream as a reference time stream.

[0066] When using the timestamp of a certain data stream as the reference time stream, the timestamp of the conductivity data stream is selected as the reference time stream.

[0067] Specifically, in the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of the present invention, since the sampling period of the conductivity data stream is relatively long and the number of its data points is relatively small, the timestamp of the conductivity data stream is used as the reference time stream. This helps to avoid the problem of some data streams lacking corresponding data points due to an overly dense reference time series. The reference time series is defined as follows:

[0068] ,

[0069] In the experimental implementation:

[0070] ,

[0071] Where T represents the reference time series; This represents the k-th reference time point; This represents the total number of reference time points; This represents the timestamp of the k-th data point in the conductivity data stream.

[0072] It should be noted that choosing conductivity data stream as the reference time stream is not a limitation of the method itself. In subsequent systems, timestamps of spectral data streams, unified software trigger timestamps, or fixed sampling intervals can also be selected as the reference time series, depending on application requirements.

[0073] Step S3: For each reference time point in the reference time series, perform nearest neighbor timestamp matching in multiple data streams to find the data point with the closest time distance to the reference time point, and obtain the matching index and time deviation corresponding to multiple data streams;

[0074] Nearest neighbor timestamp matching calculates the matching index using the following formula:

[0075] ,

[0076] in, This is the matching index for the k-th reference time point corresponding to the m-th type of data stream; This means traversing and searching within the range of values ​​for j, and taking the index corresponding to the minimum value; Let be the timestamp of the j-th data point in the m-th data stream.

[0077] The time deviations of multiple data streams are calculated using the following formula:

[0078] ,

[0079] in, The time deviation of the m-th type of data stream at the k-th reference time point; It is the timestamp of the data point that best matches the k-th reference time point in the m-th data stream.

[0080] Specifically, in a spectral data stream, the matching index is defined as:

[0081] ,

[0082] The corresponding time deviation is:

[0083] ,

[0084] in, This represents the matching index of the k-th reference time point in the spectral data stream; This represents the timestamp of the j-th data point in the spectral data stream; This represents the time deviation of the spectral data stream at the k-th reference time point; This represents the timestamp of the data point in the spectral data stream that best matches the k-th reference time point.

[0085] In the pH data stream, the matching index is defined as:

[0086] ,

[0087] The corresponding time deviation is:

[0088] ,

[0089] in, This represents the matching index of the k-th reference time point in the pH data stream; This represents the timestamp of the j-th data point in the pH data stream; This indicates the time deviation of the pH data stream at the k-th reference time point; This represents the timestamp of the data point in the pH data stream that best matches the k-th reference time point.

[0090] In a turbidity data stream, the matching index is defined as:

[0091] ,

[0092] The corresponding time deviation is:

[0093] ,

[0094] in, This represents the matching index of the k-th reference time point in the turbidity data stream; This represents the timestamp of the j-th data point in the turbidity data stream; This represents the time deviation of the turbidity data stream at the k-th reference time point; This represents the timestamp of the data point in the turbidity data stream that best matches the k-th reference time point.

[0095] Step S4: Calculate the maximum value of the time deviation corresponding to each reference time point to obtain the maximum time deviation. If the maximum time deviation is less than or equal to the preset time tolerance window, it is determined that the multimodal data frame corresponding to the reference time point is successfully aligned.

[0096] The preset time tolerance window is half the average sampling period of the conductivity data stream.

[0097] Specifically, for the k-th reference time point, its maximum time deviation is defined as:

[0098] ,

[0099] To prevent data with large time differences from being incorrectly merged, a preset time tolerance window τ is set, which is determined when: If the multimodal data frame corresponding to the reference time point is successfully aligned, it is determined that the multimodal data frame is aligned successfully; otherwise, the multimodal data frame is marked as invalid or incomplete.

[0100] In the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of the present invention, the preset time tolerance window τ can be determined based on the sampling period of the reference data stream. For example, when the timestamp of the conductivity data stream is used as the reference time stream, half of the average sampling period of the conductivity data stream can be taken as the preset time tolerance window τ.

[0101] ,

[0102] in, This represents the average sampling period of the conductivity data stream.

[0103] Step S5: Combine the data points corresponding to the reference time point in multiple data streams into a multimodal data frame;

[0104] The k-th multimodal data frame can be represented as:

[0105] ,

[0106] ,

[0107] in, This is the k-th multimodal data frame; The pH value after alignment in the k-th frame; This represents the conductivity value after alignment in the k-th frame. This represents the turbidity value after alignment in the k-th frame. This represents the absorbance value at the i-th wavelength after alignment of the k-th frame. This represents the measurement that best matches the k-th reference time point in the pH data stream; This represents the measured value of the k-th data point in the conductivity data stream; This represents the measurement that best matches the k-th reference time point in the turbidity data stream; This represents the absorbance vector that best matches the k-th reference time point in the spectral data stream.

[0108] The final multimodal dataset can be represented as:

[0109] ,

[0110] in, This represents the entire aligned multimodal dataset; The total number of successfully aligned multimodal data frames.

[0111] Specifically, multimodal data frames combine low-dimensional scalar parameters and high-dimensional vector parameters in the same sample, enabling subsequent algorithms to directly perform feature-level fusion, decision-level fusion, or spectral compensation modeling. Compared to simply exporting multiple CSV files separately, the asynchronous multimodal data timestamp alignment, data frame construction, and verification method of this invention facilitates clarifying the sample correspondence between different data streams, transforming multiple asynchronous data streams from "independent sensor results" into "data structures that can be directly used for fusion."

[0112] Step S6: Construct timestamp alignment evaluation indicators to quantitatively verify the timestamp alignment effect.

[0113] Timestamp alignment evaluation metrics include the average sampling period of multiple data streams, the average maximum time deviation, the global maximum time deviation, the standard deviation of time deviation, and the alignment success rate.

[0114] Specifically, the average sampling period is defined as:

[0115] ,

[0116] in, This represents the timestamp of the (i-1)th data point in the m-th data stream;

[0117] The mean maximum time deviation is defined as:

[0118] ;

[0119] The global maximum time deviation is defined as:

[0120] ,

[0121] The standard deviation of time deviation is defined as:

[0122] ;

[0123] Alignment success rate is calculated using the following formula:

[0124] .

[0125] It should be noted that the timestamp alignment method proposed in this embodiment is a data-level synchronization method, not a hardware-level synchronous sampling in the strict sense. Each module still operates independently according to its own sampling cycle, and the host computer software performs post-processing matching on different data streams based on the timestamps. For water quality parameters such as pH, conductivity, and turbidity, which change relatively slowly, this method has good engineering feasibility.

[0126] If subsequent applications require higher time accuracy, such as rapid dynamic water process monitoring or transient pollution event capture, unified hardware triggering, shared clocks, or synchronous acquisition controllers can be introduced to reduce time deviations between modes.

[0127] Example 2

[0128] like Figures 3-5 This embodiment corresponds to the calibration and verification of the scalar detection module. The pH, conductivity, and turbidity data streams are all low-dimensional scalar parameters, representing the fundamental physicochemical characteristics in subsequent multimodal data frames. This embodiment performed system-level calibration and verification experiments on each of the three modules. It should be noted that the evaluation object in this embodiment is not the raw ADC data or raw photoelectric signal from the sensor front-end, but rather the output of the complete measurement chain comprised of the sensor front-end, embedded primary processing, and final calibration by the host computer. In the experiment, the system-level output value after host computer calibration was recorded and compared with the nominal value of the standard solution.

[0129] like Figure 3 As per Table 1, the pH module employed a three-point calibration method. pH values ​​of 4.00, 7.00, and 10.00 were selected as calibration points, and pH values ​​of 6.86 and 9.18 were selected as independent verification points. 100 sets of data were collected for each standard solution, and the device output mean, standard deviation, and error were calculated. Figure 3 In this context, Ideal line y=x represents the ideal reference line y=x; Calibration points represent calibration points; Validation point represents validation points; Device output pH represents the device output pH value; and Reference pH represents the reference standard pH value.

[0130] like Figure 4As shown in Table 1, the conductivity module employed a two-point calibration method. The experiment used 84 μS / cm and 1413 μS / cm as calibration points, and 146.5 μS / cm as an independent verification point. 330 sets of data were repeatedly collected at each measurement point, and the mean and standard deviation were calculated. The results show that the conductivity module exhibits good consistency at both the calibration and independent verification points. The relative error at the 146.5 μS / cm verification point is approximately 0.40%. Figure 4 In this context, Conductivity represents electrical conductivity; Ideal line y=x represents the ideal reference line y=x; Calibration points represent calibration points; Validation point represents validation points; Device output EC represents the device output conductivity; and Reference EC represents the reference standard conductivity.

[0131] like Figure 5 As shown in Table 1, the turbidity module employs a three-point calibration method. The experiment used 20 NTU, 100 NTU, and 800 NTU as calibration points, and 40 NTU and 180 NTU as independent verification points. Experimental results show that the turbidity module exhibits good short-term stability during repeated acquisitions. However, a significant systematic bias exists at the 40 NTU verification point, indicating that the current three-point calibration model still has room for optimization in some low-turbidity ranges. Considering that the focus of this implementation is on building a multi-modal acquisition system for data fusion, rather than solely optimizing the turbidity meter's accuracy, these results still demonstrate that the turbidity module can provide repeatable optical scattering mode data. Further improvements in turbidity measurement accuracy can be achieved by adding low-turbidity calibration points, adopting a segmented calibration model, or optimizing the optical structure. Figure 5 In this context, Turbidity represents turbidity; Ideal line y=x represents the ideal reference line y=x; Calibration points represent calibration points; Validation point represents validation points; Device output turbidity represents the device output turbidity; and Reference turbidity represents the reference standard turbidity.

[0132] Table 1 Summary of Scalar Parameter Measurement Performance

[0133]

[0134] The results demonstrate that the three scalar detection modules can provide calibrable and repeatable low-dimensional water quality characteristics, meeting the basic input requirements for subsequent multimodal data frame construction.

[0135] Example 3

[0136] like Figures 6-8This embodiment corresponds to spectral preprocessing and feature output. The spectral module is used to acquire the high-dimensional optical response of the water sample at different wavelengths. In the experiment, the dark background spectrum and reference spectrum are first acquired, then the sample spectrum is acquired, and the transmittance and absorbance are calculated by software. The spectral intensity after dark background subtraction is expressed as:

[0137] ,

[0138] in, This represents the measured spectral intensity of the sample; This represents the spectral intensity against a dark background.

[0139] Transmittance is expressed as:

[0140] ,

[0141] in, Indicates the reference spectral intensity.

[0142] Absorbance is expressed as:

[0143] .

[0144] like Figures 6-8 The experimental results include the light source spectrum, the transmittance spectrum of the COD standard sample, and the absorbance spectrum of the COD standard sample. The purpose of the spectroscopic experiments in this embodiment is to verify the spectral acquisition link and the absorbance characteristic output capability, and to establish a quantitative prediction model for COD. Figure 6 The light source spectrum is used to verify the effectiveness of the spectral acquisition link; such as Figure 7 Transmittance spectra reflect the transmittance characteristics of a sample to different wavelengths of light; such as Figure 8 The absorbance spectrum can then serve as a high-dimensional input feature for subsequent water quality modeling and multimodal fusion algorithms. Figure 6 In this context, Reference Spectrum represents the reference spectrum; Intensity represents the light intensity; and Wavelength represents the wavelength. Figure 7 In this context, COD Transmittance represents the chemical oxygen demand transmittance spectrum; Transmittance represents transmittance; and Wavelength represents wavelength. Figure 8 In this context, COD Absorbance represents the absorbance spectrum of chemical oxygen demand; Absorbance represents absorbance; and Wavelength represents wavelength.

[0145] From a data fusion perspective, spectral modes differ from pH, conductivity, and turbidity in that their output is not a single scalar, but a high-dimensional vector composed of multiple wavelength points:

[0146] ,

[0147] By subtracting the dark background, calculating transmittance and absorbance, the system can convert the original light intensity into standardized spectral features more suitable for subsequent fusion algorithms. This result demonstrates that the spectral module can output high-dimensional optical features, providing an input basis for subsequent fusion with electrochemical scalar and turbidity scattering features.

[0148] Example 4

[0149] like Figure 9 In this embodiment, to verify the construction method of the present invention, timestamps were recorded for four types of data streams: pH data stream, conductivity data stream, turbidity data stream, and spectral data stream. Since the sampling period of the conductivity data stream is relatively long and the number of its data points is relatively small, the timestamp of the conductivity data stream is used as the reference time stream.

[0150] Timestamps aligned with timeline charts, such as Figure 9 As shown, Figure 9 The diagram illustrates the distribution of four types of data streams along the timeline, as well as the multimodal data frame Xk generated based on the nearest neighbor timestamp matching method. Figure 9 In the diagram, each dashed line represents a nearest neighbor match: using a reference EC time point (e.g., E1) as a baseline, the nearest data point in time is found in the spectral, turbidity, and pH data streams. These four points (EC itself + the nearest data points from the other three data streams) are then packaged into a frame (X1). The horizontal axis represents relative time (seconds). It can be seen that the data points from the four data streams are staggered and asynchronous on the time axis. Through this "nearest neighbor pairing" method, asynchronous data is organized into frames with a unified structure. Figure 9 In this context, "Timestamp-based alignment of multimodal water-quality data" refers to the alignment of multimodal water-quality data based on timestamps; "Reference: EC timestamp" represents the reference baseline: conductivity timestamp; "Alignmentwindow" represents the alignment window; "Spectral data" represents the spectral data stream; "Turbidity data" represents the turbidity data; "pH data" represents the pH data stream; "EC reference packets" represents the conductivity reference data stream; "Aligned frames" represents the aligned data frames; and "Relative time" represents the relative time.

[0151] The experimental statistical results are shown in Table 2. The average sampling periods of the spectral data stream, turbidity data stream, pH data stream, and conductivity data stream are approximately 0.576 s, 1.095 s, 1.447 s, and 2.008 s, respectively, indicating that the four types of data streams have significantly different sampling periods. In the experiment, the preset time tolerance window τ was set to half of the average sampling period of the conductivity data stream, approximately 1.004 s. Under the preset time tolerance window τ, all 77 reference frames successfully generated multimodal data frames, with an alignment success rate of 100%, an average maximum time deviation of 0.437 s, and a global maximum time deviation of 0.747 s.

[0152] Table 2 Timestamp Alignment Statistics

[0153]

[0154] Experimental results show that although the sampling periods of different detection modules are inconsistent, the system can stably construct complete multimodal data frames through timestamp recording and nearest-neighbor timestamp matching. For detection objects such as water quality parameters that change relatively slowly, the aforementioned time deviation is within an acceptable range. The experimental results demonstrate that the system can not only collect multimodal data but also organize asynchronous data streams into a data structure with sample correspondence that can be directly used for fusion.

[0155] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for asynchronous multimodal data timestamp alignment, data frame construction, and verification, characterized in that, include: Acquire multiple asynchronous data streams; Construct a reference time series; For each reference time point in the reference time series, nearest neighbor timestamp matching is performed in multiple data streams to find the data point with the closest time distance to the reference time point, thereby obtaining the matching index and time deviation corresponding to the multiple data streams; Calculate the maximum time deviation corresponding to each of the reference time points to obtain the maximum time deviation. If the maximum time deviation is less than or equal to a preset time tolerance window, it is determined that the multimodal data frame corresponding to the reference time point is successfully aligned. The data points corresponding to the reference time point in multiple data streams are combined into a multimodal data frame; Construct a timestamp alignment evaluation index to quantitatively verify the timestamp alignment effect.

2. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 1, characterized in that, The construction of the reference time series includes: A reference time axis is constructed using a fixed sampling interval; Alternatively, the timestamp of one of the data streams can be used as a reference time stream.

3. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 2, characterized in that, The multiple data streams include a pH data stream, a conductivity data stream, a turbidity data stream, and a spectral data stream. When a timestamp of one of the data streams is used as a reference time stream, the timestamp of the conductivity data stream is selected as the reference time stream.

4. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 3, characterized in that, The preset time tolerance window is half the average sampling period of the conductivity data stream.

5. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 1, characterized in that, The nearest neighbor timestamp matching is calculated using the following formula to obtain the matching index: , in, This is the matching index for the k-th reference time point corresponding to the m-th type of data stream; This means traversing and searching within the range of values ​​for j, and taking the index corresponding to the minimum value; The timestamp of the j-th data point in the m-th data stream; This is the kth reference time point.

6. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 5, characterized in that, The time deviations corresponding to the multiple data streams are calculated using the following formula: ， in, The time deviation of the m-th type of data stream at the k-th reference time point; It is the timestamp of the data point that best matches the k-th reference time point in the m-th data stream.

7. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 1, characterized in that, The multimodal data frame can be represented as: ， ， in, This is the k-th multimodal data frame; The pH value after alignment in the k-th frame; This represents the conductivity value after alignment in the k-th frame. This represents the turbidity value after alignment in the k-th frame. This represents the absorbance value at the i-th wavelength after alignment of the k-th frame. This represents the wavelength points of the spectrum, where n is the total number of wavelengths. This represents the measurement that best matches the k-th reference time point in the pH data stream; This represents the measured value of the k-th data point in the conductivity data stream; This represents the measurement that best matches the k-th reference time point in the turbidity data stream; This represents the absorbance vector that best matches the k-th reference time point in the spectral data stream.

8. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 7, characterized in that, The final multimodal dataset can be represented as: ， in, This represents the entire aligned multimodal dataset; The total number of successfully aligned multimodal data frames.

9. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 1, characterized in that, The timestamp alignment evaluation metrics include the average sampling period of multiple data streams, the average maximum time deviation, the global maximum time deviation, the standard deviation of the time deviation, and the alignment success rate.

10. The asynchronous multimodal data timestamp alignment, data frame construction, and verification method according to claim 9, characterized in that, The alignment success rate is calculated using the following formula: ， in, Indicates the alignment success rate; This represents the total number of reference time points.

Images