Experimental data processing method and device, AI analysis module and computer equipment

Abstract

The invention relates to an experimental data processing method and device, an AI analysis module and computer equipment, and belongs to the field of data processing.The method comprises the steps that multi-dimensional original data are partitioned according to types, and formats are unified; generating a similarity matrix based on time and space neighborhood information, and marking abnormal fluctuation points; effective signals are separated through a time-frequency feature matching noise library; extracting multi-layer features of basic statistics, time sequence correlation and trend change; and dynamically screening core features to update the tracking type experimental model. The matched AI analysis module integrates hardware circuits of data partitioning, similarity calculation, anomaly marking, noise matching, feature extraction and model updating, and whole-process acceleration is achieved. According to the method, through multi-dimensional data compatibility processing, accurate anomaly detection, multilayer feature fusion and model adaptive optimization, the experimental data processing efficiency and conclusion reliability are remarkably improved, and the method is suitable for real-time analysis of multiple scenes such as scientific research and industry.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an experimental data processing method, device, AI analysis module and computer equipment. Background Art

[0002] In scientific research experiments and industrial testing scenarios, efficient processing of multidimensional experimental data is critical to ensuring the reliability of experimental conclusions. Current experimental data is typically collected simultaneously by multiple terminals, including sensors, image acquisition devices, and logging systems. This presents technical challenges such as heterogeneous data formats (e.g., analog signals, digital images, and text logs), complex noise interference (environmental noise, equipment errors, and transmission interference), and hidden feature correlations. Traditional data processing methods often use fixed block rules for initial processing of raw data, which struggles to adapt to the structural characteristics of diverse data types, resulting in low effective signal recognition rates in subsequent analysis. In the outlier detection phase, these methods rely on manually set thresholds or simple statistical methods, failing to dynamically integrate the data's time series correlations with its spatial distribution characteristics, leading to missed or misidentified key fluctuation points. Feature extraction processes often rely solely on basic statistical analysis, lacking in-depth exploration of data fluctuation cycles and multi-scale trends. This results in a single-dimensional input feature for experimental models, making it difficult to capture complex correlations between data. Furthermore, existing experimental model update mechanisms typically use static feature inputs and are unable to dynamically adjust feature importance based on real-time experimental objectives, resulting in insufficient model generalization and difficulties in ensuring the accuracy and reproducibility of experimental results. With the increasing complexity of experimental data, there is an urgent need for a comprehensive data processing method that can take into account data diversity processing, multi-dimensional anomaly detection, multi-level feature extraction and adaptive model optimization to solve the problems of low data processing accuracy and poor model adaptability in existing technologies. Summary of the Invention

[0003] The main purpose of this invention is to provide an experimental data processing method, device, AI analysis module and computer equipment, aiming to improve the processing accuracy and model adaptability of complex experimental data.

[0004] To achieve the above object, the present invention provides an experimental data processing method comprising the following steps:

[0005] Obtaining raw data collected from a multi-dimensional experiment, dividing the raw data into blocks according to a preset data type, and obtaining a block data set with a unified data format;

[0006] Generate a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution in each block data set, and mark the abnormal fluctuation points in the block data set according to the similarity matrix;

[0007] The marked block data set is matched with the preset noise feature library to separate the target data segment containing the valid signal;

[0008] Perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0009] The feature vector set is sorted by importance according to the preset experimental goal, the core feature vectors that have a significant impact on the experimental conclusion are retained, and the preset tracking experimental model is updated through the core feature vectors.

[0010] Furthermore, the step of performing block processing on the original data according to a preset data type to obtain a block data set with a unified data format includes:

[0011] Identify the protocol types and data structures corresponding to different data acquisition devices in the raw data, and label the raw data according to the preset classification standards for sensor data, image data, and text log data;

[0012] For time series sensor data, a sliding time window is set as a block unit according to the data collection frequency, and the continuously collected data stream is cut at fixed time intervals;

[0013] For spatially distributed image data, the image matrix is ​​divided into sub-image blocks with overlapping boundaries according to the preset grid division rules;

[0014] The format of each data unit after block is standardized, including data unit normalization, timestamp alignment and missing value filling, so that different types of block data are converted into a unified structured format containing data identification, collection time and feature parameters.

[0015] Furthermore, for each block data set, the step of generating a similarity matrix between data points based on the local neighborhood information of the corresponding data points in the time series or spatial distribution includes:

[0016] For block datasets of time series type, with the current data point as the center, the k adjacent data points before and after are determined as the local neighborhood according to the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for block datasets of spatial distribution type, the neighboring points within the range of m×m around the current data point are delineated according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated by the cosine similarity of the feature vectors;

[0017] The time dimension similarity and the space dimension similarity are linearly combined according to the preset weights to obtain the comprehensive similarity value between the data points;

[0018] All data points in the block data set are used as matrix nodes, and the comprehensive similarity values ​​of any two data points are used as matrix elements to construct a symmetric similarity matrix. The matrix diagonal elements are set to 1 to indicate that they are completely similar.

[0019] Furthermore, the step of marking abnormal fluctuation points in the block data set according to the similarity matrix includes:

[0020] Calculate the mean of the off-diagonal elements of the corresponding row of each data point in the similarity matrix as the local neighborhood similarity index of the data point;

[0021] Set an abnormal fluctuation determination threshold, which is the average value of the local neighborhood similarity index of all data points in the block data set minus n times the standard deviation, where n is the preset abnormal sensitivity coefficient;

[0022] Data points whose local neighborhood similarity index is lower than the abnormal fluctuation judgment threshold are marked as abnormal fluctuation points, where the marking of time series data points is combined with the fluctuation amplitude change rate in the previous and next time windows, and the marking of spatially distributed data points is combined with the feature gradient change within the neighborhood space range.

[0023] Furthermore, the steps of performing feature matching on the marked block data set and a preset noise feature library to separate target data segments containing valid signals include:

[0024] Extract time domain feature parameters and frequency domain feature parameters from the marked block data set, wherein the time domain feature parameters include waveform peak, rising edge slope, and pulse width, and the frequency domain feature parameters include power spectrum density, center frequency, and proportion of frequency multiplication components;

[0025] The extracted feature parameters are compared with the historical noise feature vectors stored in the noise feature library, and the matching degree between the current block data segment and each noise category is obtained by using a composite measurement method combining Euclidean distance and cosine similarity.

[0026] A noise exclusion threshold is set. When the matching degree between a block data segment and any noise category exceeds the threshold, the data segment is judged to be a pure noise segment. For data segments that are not judged to be pure noise segments and contain abnormal fluctuation point marks, a target data segment containing the complete fluctuation process is generated by extending the data segments before and after by m data points, where m is determined by the data sampling frequency and the signal cycle length.

[0027] Furthermore, the step of performing multi-level feature extraction on the target data segment includes:

[0028] During the first-level feature extraction, the basic statistics of the target data segment are calculated, including the data mean, standard deviation, kurtosis, skewness, range, and percentiles of the data distribution, to form a basic feature vector containing the trend and dispersion of the data;

[0029] The second layer determines the main cycle length of the signal through autocorrelation function analysis based on the data fluctuation period, extracts time series related features such as the consistency of peak intervals within the cycle, the amplitude change rate of adjacent cycles, and the phase difference between the start and end of the cycle, and constructs a time series feature vector based on the waveform distortion coefficient after the sliding average filter;

[0030] The third layer uses a sliding window with a variable window size. The window length is set to 1-3 times the length of the basic period. The basic statistical features and time series related features in adjacent sliding windows are differentially calculated to extract trend features such as the time change rate of feature parameters, the slope of the fluctuation trend, and the migration distance of extreme points. The trend feature vector containing multi-scale change information is generated.

[0031] The three layers of feature vectors are spliced ​​by dimension to form a multi-scale feature vector set that includes basic statistics, time series correlation, and trend changes.

[0032] Furthermore, the steps of sorting the feature vector set by importance according to the preset experimental objectives, retaining the core feature vectors that have a significant impact on the experimental conclusions, and updating the preset tracking experimental model using the core feature vectors include:

[0033] Establish a feature importance evaluation function to calculate the contribution index of each feature vector based on the correlation between the feature and the experimental target. The correlation is quantified using the Spearman correlation coefficient, information gain rate, or a weight coefficient based on a preset regression model.

[0034] Arrange the feature vector set in descending order according to the contribution index, and screen out the core feature vectors whose influence on the experimental conclusion exceeds the critical value based on the preset feature retention ratio or significance P value threshold;

[0035] Inputting the core feature vector into a preset tracking experimental model, which includes a dynamically adjustable parameter layer, and iteratively updating the weight matrix and bias terms in the model parameter layer by minimizing the root mean square error between the current experimental data and the model prediction results;

[0036] During the update process, the historical change curve of feature contribution is recorded synchronously. When the contribution fluctuation of a feature in n consecutive data batches exceeds the preset range, the model structure adaptive adjustment mechanism is triggered to optimize the connection method of the feature input layer.

[0037] The present invention provides an experimental data processing device, comprising:

[0038] An acquisition unit is used to acquire raw data collected from a multi-dimensional experiment, and to perform block processing on the raw data according to a preset data type to obtain a block data set with a unified data format;

[0039] A data unit is used to generate a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution of each block data set, and mark abnormal fluctuation points in the block data set according to the similarity matrix;

[0040] A matching unit is used to perform feature matching between the marked block data set and a preset noise feature library to separate the target data segment containing the valid signal;

[0041] An extraction unit is used to perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0042] An updating unit is used to sort the importance of the feature vector set according to the preset experimental objectives, retain the core feature vectors that have a significant impact on the experimental conclusions, and update the preset tracking experimental model through the core feature vectors

[0043] The present invention also proposes an experimental data AI analysis module, comprising:

[0044] An integrated data processing chip with a built-in data segmentation processing circuit is used to identify and segment the raw data collected from multiple dimensions according to sensor protocol type, image data structure, and text log format, generating segmented data sets in a unified structured format.

[0045] The neighborhood similarity calculation unit includes a time series processing subcircuit and a spatial distribution processing subcircuit. The former constructs a time window register group based on the data sampling frequency to calculate the Euclidean distance between adjacent data points, while the latter completes the cosine similarity calculation of the feature vectors of neighboring points through a spatial grid division logic circuit.

[0046] The abnormal fluctuation marking module is equipped with a statistical computing hardware unit, which can calculate the mean and standard deviation of the local neighborhood similarity of data points in real time and mark abnormal points through a threshold comparator. The time series data processing channel integrates a fluctuation amplitude change rate calculation circuit, and the spatial data channel integrates a feature gradient change detection circuit;

[0047] The noise feature matching chip has a built-in historical noise feature storage array, supports parallel matching calculations of time domain features (peak value, slope, pulse width) and frequency domain features (power spectrum, center frequency, and frequency multiplication components), and implements data segment noise category determination through a composite metric logic circuit.

[0048] The multi-layer feature extraction hardware architecture consists of a basic statistical feature operation unit at the first layer, which includes mean, variance, and percentile calculation modules. The second layer is a time series correlation analysis module, which integrates an autocorrelation function operator and a periodic feature extraction circuit. The third layer is a sliding window trend analysis unit, which has a variable window length control register and feature difference calculation logic.

[0049] The feature screening and model update unit includes a contribution calculation coprocessor that supports hardware-accelerated calculation of the Spearman coefficient and information gain rate, as well as a tracking experimental model parameter adjustment circuit that drives the iterative update of the parameter layer weight matrix and bias terms through a root mean square error calculation module.

[0050] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned experimental data processing method when executing the computer program.

[0051] The experimental data processing method, device, AI analysis module, and computer equipment provided by the present invention have the following beneficial effects:

[0052] (1) Combine the time series neighborhood similarity and spatial distribution feature gradient analysis to mark outliers, and simultaneously introduce the noise feature library for multi-level feature matching to effectively distinguish noise signals from effective fluctuations and improve the accuracy of abnormal data identification;

[0053] (2) Through multi-level feature extraction, a multi-scale feature vector set containing basic statistics, time series correlation, and trend changes is formed, and the core features are dynamically screened based on the experimental objectives to update the model, so that the tracking experimental model can adapt to changes in data characteristics in real time, significantly improving the reliability of experimental conclusions and the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a flow chart of an experimental data processing method according to an embodiment of the present invention;

[0055] Figure 2 is a structural block diagram of an experimental data processing device in one embodiment of the present invention;

[0056] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0057] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Reference Figure 1 The following is a flow chart of the experimental data processing method proposed by the present invention, which includes the following steps:

[0060] S1, obtaining raw data collected from a multi-dimensional experiment, dividing the raw data into blocks according to preset data types, and obtaining a block data set with a unified data format;

[0061] S2, generating a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution in each block data set, and marking abnormal fluctuation points in the block data set according to the similarity matrix;

[0062] S3, feature matching is performed on the marked block data set with the preset noise feature library to separate the target data segment containing the valid signal;

[0063] S4, performing multi-level feature extraction on the target data segment. The first layer extracts basic statistical features of the data, the second layer extracts temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0064] S5, sorting the importance of the feature vector set according to the preset experimental goal, retaining the core feature vectors that have a significant impact on the experimental conclusion, and updating the preset tracking experimental model through the core feature vectors.

[0065] In steps S1 to S5, the experimental data processing method provided aims to solve the technical problems of heterogeneous data compatibility, noise interference elimination, feature depth extraction and model adaptive optimization in multi-dimensional experimental data processing. The specific implementation steps are as follows:

[0066] First, for the raw data collected from multiple sources such as sensors, images, and logs, differentiated block processing is performed according to the data type (time series type, spatial distribution type): the sensor data stream is cut by setting a sliding time window according to the acquisition frequency, and the image data is divided into overlapping sub-blocks according to the grid. Unit normalization, timestamp alignment and other format standardization are completed simultaneously to form a unified structured block data set, solving the compatibility problem of heterogeneous data preprocessing.

[0067] Secondly, a similarity matrix is ​​constructed between data points to identify anomalous fluctuations. For time series data, the Euclidean distance is calculated for the k neighboring points before and after the sampling frequency. For spatial data, the cosine similarity is calculated for an m×m neighborhood with a preset radius. The two are weighted to generate a comprehensive similarity matrix. The anomaly threshold is dynamically set by statistically subtracting the standard deviation. This is combined with the time window fluctuation amplitude change rate and spatial feature gradient changes to accurately mark anomalies, avoiding the omissions and misjudgments that occur with traditional fixed threshold detection.

[0068] Then, effective signal separation is achieved through the noise feature library: the time domain (peak, slope) and frequency domain (power spectrum, center frequency) features of the data segment are extracted, and the historical noise features are matched using the composite measurement method of Euclidean distance and cosine similarity. The pure noise segments with high matching degree are excluded, and the data segments containing abnormal points are expanded to generate target data segments with complete fluctuation processes to ensure that subsequent analysis focuses on effective signals.

[0069] Finally, the experimental model is optimized through a three-level feature extraction and dynamic screening process: the first layer extracts basic statistical features such as mean and variance; the second layer analyzes periodic characteristics and extracts time-series correlation features based on autocorrelation functions; and the third layer calculates trend change characteristics through variable window differencing, forming a multi-scale feature vector set. Feature contributions are assessed using the Spearman correlation coefficient and other methods, and core features are selected for input into the tracking experimental model. Model parameters are iteratively updated by minimizing the root mean square error (RMSE). Fluctuations in feature contributions are simultaneously monitored to trigger structural adaptive adjustments, achieving dynamic adaptation of the model to the experimental objectives.

[0070] This method significantly improves the processing accuracy of complex experimental data and the reliability of model prediction through closed-loop processing of data block standardization, multi-dimensional anomaly detection, multi-layer feature fusion and model adaptive update. It is suitable for efficient data processing needs in multiple scenarios such as scientific research analysis and industrial testing.

[0071] In one embodiment, the step of dividing the original data into blocks according to preset data types to obtain block data sets with a unified data format includes:

[0072] Identify the protocol types and data structures corresponding to different data acquisition devices in the raw data, and label the raw data according to the preset classification standards for sensor data, image data, and text log data;

[0073] For time series sensor data, a sliding time window is set as a block unit according to the data collection frequency, and the continuously collected data stream is cut at fixed time intervals;

[0074] For spatially distributed image data, the image matrix is ​​divided into sub-image blocks with overlapping boundaries according to the preset grid division rules;

[0075] The format of each data unit after block is standardized, including data unit normalization, timestamp alignment and missing value filling, so that different types of block data are converted into a unified structured format containing data identification, collection time and feature parameters.

[0076] In the specific implementation, the steps of dividing the raw data into blocks according to the preset data type and obtaining a unified data format block data set are mainly achieved through multi-source data type identification, differentiated block strategy and format standardization processing to achieve the compatibility integration of multi-dimensional data. First, the protocol type (such as Modbus, TCP / IP, etc.) and data structure characteristics of different acquisition devices in the raw data are identified through the protocol parsing module. For sensor data, metadata such as device type, sampling frequency, data unit, etc. are extracted. For image data, matrix parameters such as file format, resolution, number of channels, etc. are parsed. For text log data, regular expressions are used to match the log entry structure and extract the timestamp format and parameter value range. Then, the type label of sensor data, image data or text log data is marked for each data unit according to the preset classification standard to provide a classification basis for subsequent processing. For time series sensor data, a sliding time window is set as the block unit based on the data acquisition frequency f. The window length T is set to 1-2 times the typical signal period to ensure that the window contains the complete signal fluctuation. The sliding step is set to T / 2 to ensure that adjacent windows overlap by 50% to avoid truncation of signal mutation points. The continuous data stream is then divided into blocks consisting of multiple equally spaced sampling points at fixed time intervals, forming fixed-length data segments in the time dimension. For spatially distributed image data, according to the preset grid partitioning rule of m×m pixel size and k pixel overlap boundary width (usually k=1-2), the grid is slid row and column by row starting from the upper left corner of the image with a step size s=mk. The image matrix is ​​divided into sub-image blocks with overlapping boundaries, preserving image edge features. Spatial metadata such as sub-block coordinate position and pixel value range are extracted and stored together with the pixel matrix data. During the format standardization stage, data unit normalization is performed on each blocked data unit, and the units of different physical quantities are unified by establishing a unit conversion table; timestamp alignment is performed, and the time deviation of synchronously collected data from multiple devices is corrected through linear interpolation based on the timestamp of the master clock device, and the error is controlled within ±1 sampling period; missing value completion is implemented, and the missing points of time series data are filled with the weighted average of the five valid points before and after (the weight decreases with distance), and the missing points of image data are filled with the median of the neighborhood pixel value. Ultimately, different types of blocked data are converted into data identifiers containing the device ID, block timestamp, and spatial coordinates (for image data) that uniquely identify the data block, the acquisition time with a unified time base accurate to microseconds, and a unified structured format of the feature parameters of the standardized numerical matrix or text field, which solves the compatibility problem of heterogeneous data preprocessing.

[0077] In one embodiment, the step of generating a similarity matrix between data points for local neighborhood information of corresponding data points in time series or spatial distribution in each block data set includes:

[0078] For block datasets of time series type, with the current data point as the center, the k adjacent data points before and after are determined as the local neighborhood according to the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for block datasets of spatial distribution type, the neighboring points within the range of m×m around the current data point are delineated according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated by the cosine similarity of the feature vectors;

[0079] The time dimension similarity and the space dimension similarity are linearly combined according to the preset weights to obtain the comprehensive similarity value between the data points;

[0080] All data points in the block data set are used as matrix nodes, and the comprehensive similarity values ​​of any two data points are used as matrix elements to construct a symmetric similarity matrix. The matrix diagonal elements are set to 1 to indicate that they are completely similar.

[0081] In practice, for block-based time series datasets (such as continuous signals collected by sensors), the local neighborhood is defined as the k adjacent data points before and after the current data point, based on the data sampling frequency and a preset time window length. This neighborhood is designed to capture the correlation of short-term fluctuations in the data along the time dimension. The sampling frequency determines the temporal resolution, while the time window length corresponds to the typical fluctuation period of the signal, ensuring that the neighborhood contains sufficient contextual information to reflect the temporal dependence of the data points. By calculating the Euclidean distance between the current point and each point in the neighborhood as the temporal similarity, this distance metric can intuitively reflect the amplitude differences in numerical fluctuations. Smaller distances indicate closer trends in the time series.

[0082] For spatially distributed block datasets (such as pixels in image sub-blocks), neighboring points within an m×m range around the current data point are delineated according to a preset spatial neighborhood radius. This grid range setting takes into account the local continuity of image features (for example, textures and edges generally remain consistent within a small spatial range). The design of overlapping boundaries (the sub-image block division described in claim 2) ensures the neighborhood integrity of edge points. Spatial dimensional similarity is calculated using the cosine similarity of feature vectors. This method focuses on the directional consistency of pixel feature vectors (rather than absolute numerical differences) and is suitable for capturing structural similarities in spatial distributions (for example, pixels of the same material have similar vector directions in RGB color space).

[0083] After obtaining the similarity of the time and space dimensions, they are linearly combined using preset weights to obtain the overall similarity value between the data points. The weight assignment can be dynamically adjusted based on the characteristics of the data type (e.g., sensor data prioritizes the time dimension, while image data prioritizes the spatial dimension), thereby adapting to the core feature requirements of different experimental scenarios. Finally, a symmetric similarity matrix is ​​constructed, using all data points in the block dataset as matrix nodes and the overall similarity value of any two points as matrix elements. Diagonal elements in the matrix are set to 1 to indicate that the data points themselves are completely similar.

[0084] In one embodiment, the step of marking abnormal fluctuation points in the block data set according to the similarity matrix includes:

[0085] Calculate the mean of the off-diagonal elements of the corresponding row of each data point in the similarity matrix as the local neighborhood similarity index of the data point;

[0086] Set an abnormal fluctuation determination threshold, which is the average value of the local neighborhood similarity index of all data points in the block data set minus n times the standard deviation, where n is the preset abnormal sensitivity coefficient;

[0087] Data points whose local neighborhood similarity index is lower than the abnormal fluctuation judgment threshold are marked as abnormal fluctuation points, where the marking of time series data points is combined with the fluctuation amplitude change rate in the previous and next time windows, and the marking of spatially distributed data points is combined with the feature gradient change within the neighborhood space range.

[0088] In practice, for each data point, the mean of the off-diagonal elements in the corresponding row of the similarity matrix is ​​calculated. This serves as the local neighborhood similarity metric—a measure of the average similarity between the current data point and other points in the neighborhood. Higher values ​​indicate a more consistent fluctuation pattern or spatial structure between the data point and its surrounding neighborhood, while lower values ​​indicate a greater deviation from the local data distribution. By excluding diagonal elements (whose self-similarity is always 1), the focus is on the correlation between the data point and its external neighborhood, preventing the interference of self-similarities in anomaly judgment.

[0089] Then, a dynamic threshold setting method based on the overall distribution of the data set is adopted: the local neighborhood similarity indicators of all data points are statistically analyzed, and their average and standard deviation are calculated. The abnormal fluctuation determination threshold is defined as "average minus n times the standard deviation", where n is the preset abnormal sensitivity coefficient (usually 1-3, and the larger n, the lower the threshold and the higher the detection sensitivity). This standard deviation-based threshold setting can adapt to the degree of discreteness of the data distribution. Compared with the fixed threshold method, it can effectively cope with the differences in the fluctuation characteristics of different experimental data and avoid missed or misjudgments due to changes in data variance.

[0090] In the specific labeling process, differentiated auxiliary judgment conditions are introduced for time series and spatial distribution data: for time series data points, based on the similarity index, the fluctuation amplitude change rate in the previous and next time windows (that is, the ratio of the data peak difference in adjacent time windows to the time window length) is synchronously calculated. If the similarity index is lower than the threshold and the fluctuation amplitude change rate exceeds the empirical threshold (such as 20%), it is judged as an anomaly to capture sudden and violent fluctuations; for spatially distributed data points, combined with the feature gradient change within the neighborhood space range (such as the gradient modulus of the pixel point RGB value), when the similarity index is lower than the threshold and the gradient change exceeds 1.5 times the local average value, it is marked as an anomaly to ensure the accurate identification of edge features or texture mutation points.

[0091] This labeling method uses a two-layer mechanism of "statistical threshold initial screening + domain feature precise judgment". It not only uses the global statistical characteristics of the similarity matrix to filter out generally similar regular data points, but also adds specific judgment conditions based on the dynamic volatility of time series and the structural mutation of spatial distribution, effectively solving the judgment limitations of a single statistical method in complex data scenarios.

[0092] In one embodiment, the step of performing feature matching on the marked block data set and a preset noise feature library to separate the target data segment containing the valid signal includes:

[0093] Extract time domain feature parameters and frequency domain feature parameters from the marked block data set, wherein the time domain feature parameters include waveform peak, rising edge slope, and pulse width, and the frequency domain feature parameters include power spectrum density, center frequency, and proportion of frequency multiplication components;

[0094] The extracted feature parameters are compared with the historical noise feature vectors stored in the noise feature library, and the matching degree between the current block data segment and each noise category is obtained by using a composite measurement method combining Euclidean distance and cosine similarity.

[0095] A noise exclusion threshold is set. When the matching degree between a block data segment and any noise category exceeds the threshold, the data segment is judged to be a pure noise segment. For data segments that are not judged to be pure noise segments and contain abnormal fluctuation point marks, a target data segment containing the complete fluctuation process is generated by extending the data segments before and after by m data points, where m is determined by the data sampling frequency and the signal cycle length.

[0096] In specific implementation, time domain and frequency domain feature parameters are extracted synchronously from the marked block data set: time domain analysis focuses on the instantaneous change characteristics of the signal, extracts parameters such as waveform peak (reflecting signal strength), rising edge slope (characterizing mutation speed), and pulse width (reflecting duration). For example, for the pulse signal collected by the sensor, the rising edge slope can effectively distinguish between noise interference and real pulses; frequency domain analysis converts the time domain signal into frequency space through fast Fourier transform, and extracts power spectrum density (reflecting energy distribution), center frequency (characterizing the main vibration mode), and the proportion of frequency harmonic components (reflecting signal periodicity). For example, mechanical vibration noise usually has energy concentration at a specific frequency harmonic.

[0097] Next, a composite metric combining Euclidean distance and cosine similarity is used to calculate the matching degree between the current data segment and the historical feature vectors in the noise feature library. Euclidean distance quantifies the numerical difference in feature parameters (such as peak voltage deviation) and is suitable for measuring the consistency of absolute quantities. Cosine similarity focuses on the directional similarity of feature vectors (such as the power spectrum distribution shape) and is suitable for capturing the matching degree of pattern structure. By weighted fusion of the two metric results (the weights can be dynamically adjusted based on the noise characteristics), a comprehensive matching degree between the data segment and each noise category is obtained. This method overcomes the limitation of a single metric's sensitivity to numerical offsets or morphological differences. For example, it can still accurately identify noise even when the peak amplitude changes but the power spectrum shape remains unchanged.

[0098] In the noise determination phase, a noise exclusion threshold is preset (e.g., a matching degree ≥ 0.8). When a data segment's matching degree with any noise category exceeds the threshold, it is determined to be a pure noise segment and directly removed. For data segments that are not determined to be pure noise and contain abnormal fluctuation point markers, a target data segment is generated by expanding m data points forward and backward, where the value of m follows the empirical formula of "sampling frequency × signal period / 2" (e.g., m = 5 for a sampling frequency of 100 Hz and a period of 0.1 seconds), ensuring that the expanded data segment contains the complete fluctuation process before and after the abnormal point. For example, for abnormal points with sudden voltage changes, expansion can preserve the stable state before the mutation and the recovery process after the mutation, providing a complete signal sample for subsequent feature extraction.

[0099] This noise separation mechanism achieves efficient filtering of noise signals and accurate retention of effective fluctuations through multi-dimensional representation of time-frequency domain features, robust matching of composite metrics, and integrity assurance of dynamic expansion.

[0100] In one embodiment, the step of performing multi-level feature extraction on the target data segment includes:

[0101] During the first-level feature extraction, the basic statistics of the target data segment are calculated, including the data mean, standard deviation, kurtosis, skewness, range, and percentiles of the data distribution, to form a basic feature vector containing the trend and dispersion of the data;

[0102] The second layer determines the main cycle length of the signal through autocorrelation function analysis based on the data fluctuation period, extracts time series related features such as the consistency of peak intervals within the cycle, the amplitude change rate of adjacent cycles, and the phase difference between the start and end of the cycle, and constructs a time series feature vector based on the waveform distortion coefficient after the sliding average filter;

[0103] The third layer uses a sliding window with a variable window size. The window length is set to 1-3 times the length of the basic period. The basic statistical features and time series related features in adjacent sliding windows are differentially calculated to extract trend features such as the time change rate of feature parameters, the slope of the fluctuation trend, and the migration distance of extreme points. The trend feature vector containing multi-scale change information is generated.

[0104] The three layers of feature vectors are spliced ​​by dimension to form a multi-scale feature vector set that includes basic statistics, time series correlation, and trend changes.

[0105] In specific implementation, the first layer of feature extraction is performed. By calculating basic statistics such as the mean, standard deviation, kurtosis, skewness, range, and data distribution percentiles (such as the 25th and 75th percentiles) of the target data segment, a feature vector is formed to characterize the trend and dispersion of the data. The mean reflects the average level of the data, the standard deviation quantifies the fluctuation range, the kurtosis describes the steepness of the distribution shape, the skewness reflects the symmetry, the range represents the data range, and the percentiles assist in determining the distribution of outliers. Together, these parameters constitute a comprehensive characterization of the basic characteristics of the data. For example, for temperature data collected by a sensor, the mean can reflect the ambient baseline temperature, and the standard deviation reflects the temperature fluctuation range.

[0106] Secondly, the second-level feature extraction is based on the data fluctuation period. The length of the signal's main period is determined through autocorrelation function analysis. The autocorrelation function peaks when the lag time is equal to the period. Based on this, the periodic characteristics of the signal (such as the natural frequency period of mechanical vibration) are identified. Then, time-related features such as the consistency of peak intervals within the period (a measure of period stability), the amplitude change rate of adjacent periods (reflecting energy attenuation or enhancement), and the phase difference at the start of the period (reflecting phase stability) are extracted. The waveform distortion coefficient after the sliding average filter (calculated by the mean square error between the ideal waveform and the actual waveform) is combined to construct a time series feature vector. Taking the power system voltage signal as an example, the main period is typically 0.02 seconds (50Hz power frequency). If the consistency of the peak interval within a period is reduced, it may indicate system frequency fluctuations, while an abnormal amplitude change rate may reflect a sudden load change.

[0107] The third-level feature extraction utilizes a sliding window mechanism with a variable window size, setting the window length to 1-3 times the length of the underlying period (e.g., 0.02-0.06 seconds for a primary period of 0.02 seconds). By performing differential calculations on the basic statistical features and time-series-related features within adjacent sliding windows, trend features such as the time rate of change of characteristic parameters (e.g., mean and standard deviation), the slope of the fluctuation trend (increasing or decreasing trends determined through linear fitting), and the migration distance of extreme points (the time difference between extreme points in adjacent windows) are extracted. This generates a trend feature vector containing multi-scale variation information. This variable window design adaptively captures changes at different time scales: short windows (1 time period) focus on local fluctuation details, while long windows (3 times the period) focus on overall trend evolution. For example, in seismic wave data processing, short windows can identify high-frequency pulse characteristics of seismic waves, while long windows are used to analyze low-frequency trends in crustal movement.

[0108] Finally, the three layers of feature vectors are concatenated by dimension to form a multi-scale feature vector set that encompasses basic statistics, time series correlation, and trend changes. This set retains the basic distribution characteristics of the data (the first layer), while also incorporating periodic dynamic characteristics (the second layer) and long- and short-term trend changes (the third layer), achieving a full-dimensional feature representation from microscopic numerical fluctuations to macroscopic trend evolution.

[0109] In one embodiment, the steps of sorting the feature vector set by importance according to a preset experimental objective, retaining core feature vectors that have a significant impact on the experimental conclusion, and updating the preset tracking experimental model using the core feature vectors include:

[0110] Establish a feature importance evaluation function to calculate the contribution index of each feature vector based on the correlation between the feature and the experimental target. The correlation is quantified using the Spearman correlation coefficient, information gain rate, or a weight coefficient based on a preset regression model.

[0111] Arrange the feature vector set in descending order according to the contribution index, and screen out the core feature vectors whose influence on the experimental conclusion exceeds the critical value based on the preset feature retention ratio or significance P value threshold;

[0112] Inputting the core feature vector into a preset tracking experimental model, which includes a dynamically adjustable parameter layer, and iteratively updating the weight matrix and bias terms in the model parameter layer by minimizing the root mean square error between the current experimental data and the model prediction results;

[0113] During the update process, the historical change curve of feature contribution is recorded synchronously. When the contribution fluctuation of a feature in n consecutive data batches exceeds the preset range, the model structure adaptive adjustment mechanism is triggered to optimize the connection method of the feature input layer.

[0114] First, a feature importance evaluation function is established. This function quantifies the contribution of each feature vector based on the correlation between the feature and the experimental objective. If the experimental objective is to analyze the relationship between the data and a continuous variable (such as chemical reaction rate), the Spearman correlation coefficient can be used to measure the nonlinear monotonic correlation between the feature and the objective (for example, a higher correlation coefficient between the temperature standard deviation and the reaction rate indicates a greater impact of temperature fluctuations on the reaction rate). If the objective is a classification task (such as distinguishing normal from abnormal experimental states), the information gain rate is used to evaluate the feature's contribution to class distinction (for example, a high information gain rate for a time series feature indicates a strong ability to distinguish abnormal states). For scenarios where a regression model has already been established (such as predicting the concentration of an experimental product), the weight coefficient obtained from model training is directly used as the correlation coefficient (the larger the absolute value of the weight, the more significant the feature's impact on the prediction result). The contribution index calculated by this method provides a quantitative basis for feature importance.

[0115] Next, the set of eigenvectors is sorted in descending order by contribution index, and a screening strategy is selected based on experimental requirements: If feature simplification is the priority, a retention ratio can be set (e.g., retaining the top 30% of high-contribution features) to reduce model computational complexity; if statistical significance is emphasized, a hypothesis test is performed to calculate the feature's P value (e.g., P < 0.05) to screen out core eigenvectors whose impact on the experimental conclusions exceeds a critical value. For example, in a material strength prediction experiment, if the P value of a basic statistical feature (e.g., mean strain) is 0.03 (< 0.05), while the P value of a trend feature (e.g., rate of change of strain) is 0.12 (> 0.05), only the mean strain is retained as the core feature to avoid interference from redundant information in the model.

[0116] Subsequently, the selected core feature vectors are input into the preset tracking experimental model. The model is designed with a dynamically adjustable parameter layer (such as the fully connected layer of a neural network or the weight matrix of a linear regression). By minimizing the root mean square error (RMSE) between the current experimental data and the model prediction results, the weight matrix and bias terms in the parameter layer are iteratively updated - for example, using the stochastic gradient descent method, the weights are adjusted according to the gradient direction of the RMSE, so that the model prediction value gradually approaches the actual experimental data. This process realizes the real-time adaptation of the model to the current data characteristics. For example, in the battery aging experiment, as the number of cycles increases, the contribution of the key characteristics of capacity attenuation (such as the standard deviation of voltage fluctuation) increases. The model strengthens the response to this feature by updating the weights, thereby improving the accuracy of aging state prediction.

[0117] During the update process, the historical change curve of each feature's contribution is recorded synchronously. When the contribution of a feature in n consecutive data batches (such as n=5) fluctuates beyond a preset range (such as ±20%), the model structure adaptive adjustment mechanism is triggered. For example, the contribution of a certain time series feature was stable in the early stage, but recently due to changes in experimental conditions, its correlation with the target has dropped sharply. At this time, the model will adjust the connection method of the feature input layer - reduce the input weight of the feature or disconnect it from the hidden layer, and may also enhance the connection of other emerging high-contribution features (such as newly added trend features). This dynamic structural adjustment enables the model to actively adapt to changes in experimental objectives or data distribution, avoiding model performance degradation caused by feature importance drift.

[0118] This step solves the poor adaptability problem of traditional models caused by static feature input through a closed-loop mechanism of "association quantification-core screening-parameter update-structure adaptation".

[0119] Reference Attachment Figure 2 This is a structural block diagram of an experimental data processing device proposed by the present invention, comprising:

[0120] An acquisition unit is used to acquire raw data collected from a multi-dimensional experiment, and to perform block processing on the raw data according to a preset data type to obtain a block data set with a unified data format;

[0121] A data unit is used to generate a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution of each block data set, and mark abnormal fluctuation points in the block data set according to the similarity matrix;

[0122] A matching unit is used to perform feature matching between the marked block data set and a preset noise feature library to separate the target data segment containing the valid signal;

[0123] An extraction unit is used to perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0124] An updating unit is used to sort the importance of the feature vector set according to the preset experimental objectives, retain the core feature vectors that have a significant impact on the experimental conclusions, and update the preset tracking experimental model through the core feature vectors

[0125] In one embodiment, the present invention further provides an experimental data AI analysis module, comprising:

[0126] An integrated data processing chip with a built-in data segmentation processing circuit is used to identify and segment the raw data collected from multiple dimensions according to sensor protocol type, image data structure, and text log format, generating segmented data sets in a unified structured format.

[0127] The neighborhood similarity calculation unit includes a time series processing subcircuit and a spatial distribution processing subcircuit. The former constructs a time window register group based on the data sampling frequency to calculate the Euclidean distance between adjacent data points, while the latter completes the cosine similarity calculation of the feature vectors of neighboring points through a spatial grid division logic circuit.

[0128] The abnormal fluctuation marking module is equipped with a statistical computing hardware unit, which can calculate the mean and standard deviation of the local neighborhood similarity of data points in real time and mark abnormal points through a threshold comparator. The time series data processing channel integrates a fluctuation amplitude change rate calculation circuit, and the spatial data channel integrates a feature gradient change detection circuit;

[0129] The noise feature matching chip has a built-in historical noise feature storage array, supports parallel matching calculations of time domain features (peak value, slope, pulse width) and frequency domain features (power spectrum, center frequency, and frequency multiplication components), and implements data segment noise category determination through a composite metric logic circuit.

[0130] The multi-layer feature extraction hardware architecture consists of a basic statistical feature operation unit at the first layer, which includes mean, variance, and percentile calculation modules. The second layer is a time series correlation analysis module, which integrates an autocorrelation function operator and a periodic feature extraction circuit. The third layer is a sliding window trend analysis unit, which has a variable window length control register and feature difference calculation logic.

[0131] The feature screening and model update unit includes a contribution calculation coprocessor that supports hardware-accelerated calculation of the Spearman coefficient and information gain rate, as well as a tracking experimental model parameter adjustment circuit that drives the iterative update of the parameter layer weight matrix and bias terms through a root mean square error calculation module.

[0132] In this embodiment:

[0133] The integrated data processing chip utilizes a heterogeneous multi-core architecture, with internal block processing circuitry comprising a protocol parsing micro-engine and data block logic. The protocol parsing micro-engine uses a hardware state machine to parse sensor protocols (such as the CRC checksum field in Modbus RTU), image data structures (SOI / EOL markers in JPEG files), and text log formats (regular expression matching hardware). The parsing results drive the block logic unit. For sensor data streams, a sampling frequency clock generator controls the time window register bank for sliding segmentation. For image data, a gridding hardware module generates an overlapping sub-block address mapping table. The block data is then processed through a unit normalization lookup table, a timestamp alignment phase-locked loop, and a missing value interpolation circuit, outputting a uniformly structured data stream to the cache bus.

[0134] The neighborhood similarity calculation unit utilizes a time-space parallel processing architecture. The time series processing subcircuit includes a k-order shift register bank (k is dynamically set by the sampling frequency configuration register). The Euclidean distance between the current data point and neighboring points is calculated using a hardware multiply-accumulator array, with the result stored in a similarity temporary register. The spatial distribution processing subcircuit traverses the pixels using an m×m neighborhood coordinate generator. Feature vector cosine similarity is implemented collaboratively by a hardware vector dot product unit and a modulus calculation unit. Overlapping boundary data is processed synchronously via a dual-port cache. The two types of similarity are weighted using a weight configuration register, and a symmetric similarity matrix is ​​generated by matrix construction logic and stored in on-chip high-speed RAM.

[0135] The abnormal fluctuation marking module integrates a statistical calculation ASIC and a dual-channel feature detection circuit: the statistical calculation unit calculates the row mean and standard deviation of the similarity matrix in real time through a pipeline structure, and the threshold comparator uses three-state logic to output abnormality marks; the fluctuation amplitude change rate calculation circuit of the time series channel includes a differential amplifier and a slope comparator, and the feature gradient detection circuit of the spatial channel calculates the gradient modulus through a convolution kernel hardware accelerator. The two types of auxiliary features are fused with the statistical threshold results through an AND gate circuit to generate the final abnormal point marking signal.

[0136] The noise feature matching chip adopts the Harvard architecture. The historical noise feature storage array is divided into a time domain feature area (peak / slope / pulse width lookup table) and a frequency domain feature area (power spectrum template library). The parallel matching calculation unit contains 8 groups of time domain feature comparators and FFT hardware acceleration modules. The composite metric logic circuit outputs the noise category matching degree by weighting the results of the Euclidean distance calculation unit and the cosine similarity multiplier. When the matching degree exceeds the threshold register setting value, the noise judgment state machine outputs a pure noise segment rejection signal.

[0137] The multi-layer feature extraction hardware architecture utilizes a three-stage pipeline design. The first-level basic statistics unit consists of a mean accumulator, a variance calculation unit, and a percentile ranking circuit operating in parallel, storing the results in a feature vector register. The second-level timing analysis module uses an autocorrelation function hardware calculator to identify the primary cycle. The periodic feature extraction circuit includes a peak interval counter, an amplitude change rate multiplier, and a phase difference comparator. The waveform distortion coefficient is generated by a sliding average filter kernel and an error calculation unit. The third-level sliding window unit's variable-length control register supports window configurations of 1 to 3 times the period. The feature differential calculation logic caches adjacent window features in a register file and outputs the trend change rate and extreme value migration distance. The three-layer features are concatenated by a bus arbiter and stored in a feature vector set memory.

[0138] The feature screening and model update unit includes a dedicated coprocessor and a dynamic parameter adjustment circuit: the contribution calculation coprocessor has a built-in Spearman coefficient calculation array and information gain rate lookup table, which supports hardware-accelerated evaluation of feature importance; the model parameter adjustment circuit generates a gradient signal through a root mean square error operator to drive the weight matrix updater and bias term adjuster for iterative optimization; the feature contribution history recorder is implemented by a circular buffer. When the fluctuation of n consecutive batches exceeds the setting range of the threshold comparator, the structural adaptive adjustment state machine is triggered to reconfigure the multiplexer connection relationship of the feature input layer.

[0139] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0140] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0141] In summary, the invention discloses an experimental data processing method and an AI analysis module, which aim to improve the processing accuracy and model adaptability of complex experimental data. The method includes: dividing the multi-dimensional raw data into blocks by type and unifying the format; generating a similarity matrix based on time / space neighborhood information, marking abnormal fluctuation points; separating effective signals by matching the noise library with time-frequency features; extracting multi-layer features of basic statistics, time series correlation, and trend changes; and dynamically screening core features to update the tracking experimental model. The supporting AI analysis module integrates hardware circuits for data segmentation, similarity calculation, anomaly marking, noise matching, feature extraction, and model updating to achieve full-process acceleration. This method significantly improves the efficiency of experimental data processing and the reliability of conclusions through multi-dimensional data compatible processing, precise anomaly detection, multi-layer feature fusion, and model adaptive optimization, and is suitable for real-time analysis in multiple scenarios such as scientific research and industry.

[0142] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Among them, any reference to memory, storage, database, or other media provided by the present invention and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0143] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0144] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for processing experimental data, characterized in that: The following steps are involved: Obtaining raw data collected from a multi-dimensional experiment, dividing the raw data into blocks according to a preset data type, and obtaining a block data set with a unified data format; Generate a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution in each block data set, and mark the abnormal fluctuation points in the block data set according to the similarity matrix; The marked block data set is matched with the preset noise feature library to separate the target data segment containing the valid signal; Perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features. The feature vector set is sorted by importance according to the preset experimental goal, the core feature vectors that have a significant impact on the experimental conclusion are retained, and the preset tracking experimental model is updated through the core feature vectors.

2. The experimental data processing method according to claim 1, characterized in that: The step of dividing the original data into blocks according to preset data types to obtain block data sets with a unified data format includes: Identify the protocol types and data structures corresponding to different data acquisition devices in the raw data, and label the raw data according to the preset classification standards for sensor data, image data, and text log data; For time series sensor data, a sliding time window is set as a block unit according to the data collection frequency, and the continuously collected data stream is cut at fixed time intervals; For spatially distributed image data, the image matrix is ​​divided into sub-image blocks with overlapping boundaries according to the preset grid division rules; The format of each data unit after block is standardized, including data unit normalization, timestamp alignment and missing value filling, so that different types of block data are converted into a unified structured format containing data identification, collection time and feature parameters.

3. The experimental data processing method according to claim 1, characterized in that: The steps of generating a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution in each block data set include: For block datasets of time series type, with the current data point as the center, the k adjacent data points before and after are determined as the local neighborhood according to the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for block datasets of spatial distribution type, the neighboring points within the range of m×m around the current data point are delineated according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated by the cosine similarity of the feature vectors; The time dimension similarity and the space dimension similarity are linearly combined according to the preset weights to obtain the comprehensive similarity value between the data points; All data points in the block data set are used as matrix nodes, and the comprehensive similarity values ​​of any two data points are used as matrix elements to construct a symmetric similarity matrix. The matrix diagonal elements are set to 1 to indicate that they are completely similar.

4. The experimental data processing method according to claim 3, characterized in that: The step of marking abnormal fluctuation points in the block data set according to the similarity matrix includes: Calculate the mean of the off-diagonal elements of the corresponding row of each data point in the similarity matrix as the local neighborhood similarity index of the data point; Set an abnormal fluctuation determination threshold, which is the average value of the local neighborhood similarity index of all data points in the block data set minus n times the standard deviation, where n is the preset abnormal sensitivity coefficient; Data points whose local neighborhood similarity index is lower than the abnormal fluctuation judgment threshold are marked as abnormal fluctuation points, where the marking of time series data points is combined with the fluctuation amplitude change rate in the previous and next time windows, and the marking of spatially distributed data points is combined with the feature gradient change within the neighborhood space range.

5. The experimental data processing method according to claim 1, characterized in that: The steps of performing feature matching on the marked block data set and the preset noise feature library to separate the target data segment containing the valid signal include: Extract time domain feature parameters and frequency domain feature parameters from the marked block data set, wherein the time domain feature parameters include waveform peak, rising edge slope, and pulse width, and the frequency domain feature parameters include power spectrum density, center frequency, and proportion of frequency multiplication components; The extracted feature parameters are compared with the historical noise feature vectors stored in the noise feature library, and the matching degree between the current block data segment and each noise category is obtained by using a composite measurement method combining Euclidean distance and cosine similarity. A noise exclusion threshold is set. When the matching degree between a block data segment and any noise category exceeds the threshold, the data segment is judged to be a pure noise segment. For data segments that are not judged to be pure noise segments and contain abnormal fluctuation point marks, a target data segment containing the complete fluctuation process is generated by extending the data segments before and after by m data points, where m is determined by the data sampling frequency and the signal cycle length.

6. The experimental data processing method according to claim 1, characterized in that: The steps of performing multi-level feature extraction on the target data segment include: During the first-level feature extraction, the basic statistics of the target data segment are calculated, including the data mean, standard deviation, kurtosis, skewness, range, and percentiles of the data distribution, to form a basic feature vector containing the trend and dispersion of the data; The second layer determines the main cycle length of the signal through autocorrelation function analysis based on the data fluctuation period, extracts time series related features such as the consistency of peak intervals within the cycle, the amplitude change rate of adjacent cycles, and the phase difference between the start and end of the cycle, and constructs a time series feature vector based on the waveform distortion coefficient after the sliding average filter; The third layer uses a sliding window with a variable window size. The window length is set to 1-3 times the length of the basic period. The basic statistical features and time series related features in adjacent sliding windows are differentially calculated to extract trend features such as the time change rate of feature parameters, the slope of the fluctuation trend, and the migration distance of extreme points. The trend feature vector containing multi-scale change information is generated. The three layers of feature vectors are spliced ​​by dimension to form a multi-scale feature vector set that includes basic statistics, time series correlation, and trend changes.

7. The experimental data processing method according to claim 1, characterized in that: The steps of sorting the feature vector set by importance according to a preset experimental goal, retaining core feature vectors that have a significant impact on the experimental conclusion, and updating the preset tracking experimental model using the core feature vectors include: Establish a feature importance evaluation function to calculate the contribution index of each feature vector based on the correlation between the feature and the experimental target. The correlation is quantified using the Spearman correlation coefficient, information gain rate, or a weight coefficient based on a preset regression model. Arrange the feature vector set in descending order according to the contribution index, and screen out the core feature vectors whose influence on the experimental conclusion exceeds the critical value based on the preset feature retention ratio or significance P value threshold; Inputting the core feature vector into a preset tracking experimental model, which includes a dynamically adjustable parameter layer, and iteratively updating the weight matrix and bias terms in the model parameter layer by minimizing the root mean square error between the current experimental data and the model prediction results; During the update process, the historical change curve of feature contribution is recorded synchronously. When the contribution fluctuation of a feature in n consecutive data batches exceeds the preset range, the model structure adaptive adjustment mechanism is triggered to optimize the connection method of the feature input layer.

8. An experimental data processing device, characterized in that: include: An acquisition unit is used to acquire raw data collected from a multi-dimensional experiment, and to perform block processing on the raw data according to a preset data type to obtain a block data set with a unified data format; A data unit is used to generate a similarity matrix between data points for the local neighborhood information of the corresponding data points in the time series or spatial distribution of each block data set, and mark abnormal fluctuation points in the block data set according to the similarity matrix; A matching unit is used to perform feature matching between the marked block data set and a preset noise feature library to separate the target data segment containing the valid signal; An extraction unit is used to perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation period, and the third layer analyzes the feature change trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features. The updating unit is used to sort the importance of the feature vector set according to the preset experimental goal, retain the core feature vectors that have a significant impact on the experimental conclusion, and update the preset tracking experimental model through the core feature vectors.

9. An experimental data AI analysis module, characterized in that: include: An integrated data processing chip with a built-in data segmentation processing circuit is used to identify and segment the raw data collected from multiple dimensions according to sensor protocol type, image data structure, and text log format, generating segmented data sets in a unified structured format. The neighborhood similarity calculation unit includes a time series processing subcircuit and a spatial distribution processing subcircuit. The former constructs a time window register group based on the data sampling frequency to calculate the Euclidean distance between adjacent data points, while the latter completes the cosine similarity calculation of the feature vectors of neighboring points through a spatial grid division logic circuit. The abnormal fluctuation marking module is equipped with a statistical computing hardware unit, which can calculate the mean and standard deviation of the local neighborhood similarity of data points in real time and mark abnormal points through a threshold comparator. The time series data processing channel integrates a fluctuation amplitude change rate calculation circuit, and the spatial data channel integrates a feature gradient change detection circuit; The noise feature matching chip has a built-in historical noise feature storage array, supports parallel matching calculations of time domain features (peak value, slope, pulse width) and frequency domain features (power spectrum, center frequency, and frequency multiplication components), and implements data segment noise category determination through a composite metric logic circuit. The multi-layer feature extraction hardware architecture consists of a basic statistical feature operation unit at the first layer, which includes mean, variance, and percentile calculation modules. The second layer is a time series correlation analysis module, which integrates an autocorrelation function operator and a periodic feature extraction circuit. The third layer is the sliding window trend analysis unit, which has a variable window length control register and feature difference calculation logic; The feature screening and model update unit includes a contribution calculation coprocessor that supports hardware-accelerated calculation of the Spearman coefficient and information gain rate, as well as a tracking experimental model parameter adjustment circuit that drives the iterative update of the parameter layer weight matrix and bias terms through a root mean square error calculation module.

10. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the experimental data processing method according to any one of claims 1 to 7 are implemented.

Images