Experimental data processing method and device, AI analysis module and computer equipment

Abstract

This invention relates to an experimental data processing method, apparatus, AI analysis module, and computer equipment, belonging to the field of data processing. The method includes: segmenting multi-dimensional raw data by type and unifying the format; generating a similarity matrix based on temporal and spatial neighborhood information to mark abnormal fluctuation points; separating effective signals by matching a noise library with time-frequency features; extracting multi-layer features of basic statistics, time-series correlation, and trend changes; and dynamically selecting core features to update a tracking experimental model. The accompanying AI analysis module integrates hardware circuits for data segmentation, similarity calculation, anomaly marking, noise matching, feature extraction, and model updating, achieving full-process acceleration. This method significantly improves the efficiency of experimental data processing and the reliability of conclusions through multi-dimensional data compatible processing, accurate anomaly detection, multi-layer feature fusion, and adaptive model optimization, and is suitable for real-time analysis in various scenarios such as scientific research and industry.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an experimental data processing method, apparatus, AI analysis module, and computer equipment. Background Technology

[0002] In scientific research experiments and industrial testing scenarios, efficient processing of multi-dimensional experimental data is crucial for ensuring the reliability of experimental conclusions. Current experimental data is typically collected simultaneously from multiple terminals such as sensors, image acquisition devices, and log systems, presenting technical challenges including heterogeneous data formats (e.g., analog signals, digital images, text logs), complex noise interference (environmental noise, equipment errors, transmission interference), and hidden feature correlations. Traditional data processing methods often employ fixed block rules for preliminary processing of raw data, which is ill-suited to the structural characteristics of different data types, resulting in low effective signal recognition rates in subsequent analyses. In anomaly detection, reliance on manually set thresholds or simple statistical methods fails to dynamically combine the time-series correlation and spatial distribution characteristics of the data, easily leading to missed or misjudged key fluctuation points. Feature extraction often focuses only on basic statistical analysis, lacking in-depth exploration of data fluctuation cycles and multi-scale trends, resulting in single-dimensional input features for experimental models and difficulty in capturing complex relationships between data. Furthermore, existing experimental model update mechanisms typically use static feature inputs, failing to dynamically adjust feature importance based on real-time experimental objectives, leading to insufficient model generalization ability and difficulty in guaranteeing the accuracy and repeatability of experimental results. As the complexity of experimental data continues to increase, there is an urgent need for a comprehensive data processing method that can take into account the processing of diverse data, multi-dimensional anomaly detection, multi-level feature extraction, and adaptive model optimization, in order to solve the problems of low data processing accuracy and poor model adaptability in existing technologies. Summary of the Invention

[0003] The main objective of this invention is to provide an experimental data processing method, apparatus, AI analysis module, and computer equipment, aiming to improve the accuracy of complex experimental data processing and the adaptability of models.

[0004] To achieve the above objectives, the experimental data processing method provided by this invention includes the following steps:

[0005] Obtain raw data collected from multi-dimensional experiments, and process the raw data into blocks according to preset data types to obtain a block dataset with a unified data format;

[0006] For each data point in the segmented dataset, a similarity matrix is ​​generated based on the local neighborhood information of the corresponding data points in the time series or spatial distribution. The abnormal fluctuation points in the segmented dataset are then marked according to the similarity matrix.

[0007] The labeled block dataset is matched with a pre-defined noise feature library to separate the target data segment containing the effective signal.

[0008] Multi-level feature extraction is performed on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation cycle, and the third layer analyzes the feature change trend of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0009] The feature vector set is sorted by importance according to the preset experimental objectives, and the core feature vectors that have a significant impact on the experimental conclusions are retained. The preset tracking experimental model is then updated using the core feature vectors.

[0010] Further, the step of dividing the original data into blocks according to a preset data type to obtain a block dataset with a unified data format includes:

[0011] Identify the protocol type and data structure of different data acquisition devices in the raw data, and label the raw data according to the preset classification standards of sensor data, image data and text log data.

[0012] For time-series sensor data, a sliding time window is set as the block unit according to the data acquisition frequency, and the continuously acquired data stream is divided into segments at fixed time intervals.

[0013] For spatially distributed image data, the image matrix is ​​divided into sub-image blocks with overlapping boundaries according to a preset grid division rule;

[0014] Each data unit after being divided into blocks undergoes format standardization processing, including data unit normalization, timestamp alignment, and missing value completion, so that different types of block data are converted into a unified structured format containing data identifiers, collection time, and feature parameters.

[0015] Furthermore, the step of generating a similarity matrix between data points based on the local neighborhood information of the corresponding data points in the time series or spatial distribution of each data block includes:

[0016] For time series type block datasets, the k neighboring data points before and after the current data point are determined as local neighborhoods based on the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for spatial distribution type block datasets, the neighboring points within an m×m range around the current data point are delineated according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated through the cosine similarity of the feature vectors.

[0017] The temporal and spatial similarities are linearly combined according to preset weights to obtain the comprehensive similarity value between data points.

[0018] Using all data points in the segmented dataset as matrix nodes, and taking the combined similarity value of any two data points as matrix elements, a symmetric similarity matrix is ​​constructed. The diagonal elements of the matrix are set to 1 to indicate that they are completely similar.

[0019] Further, the step of marking anomalous fluctuation points in the segmented dataset according to the similarity matrix includes:

[0020] Calculate the mean of the off-diagonal elements of the corresponding row in the similarity matrix for each data point, and use it as the local neighborhood similarity index for that data point.

[0021] Set an abnormal fluctuation judgment threshold, which is the average value of the local neighborhood similarity index of all data points in the segmented dataset minus n times the standard deviation, where n is a preset abnormal sensitivity coefficient;

[0022] Data points whose local neighborhood similarity index is lower than the abnormal fluctuation judgment threshold are marked as abnormal fluctuation points. The marking of time series data points is combined with the rate of change of fluctuation amplitude within the preceding and following time windows, and the marking of spatially distributed data points is combined with the feature gradient change within the neighborhood space.

[0023] Further, the step of performing feature matching between the labeled segmented dataset and a pre-defined noise feature library to separate the target data segment containing the valid signal includes:

[0024] Time-domain and frequency-domain feature parameters are extracted from the labeled block dataset. The time-domain feature parameters include waveform peak value, rising edge slope, and pulse width. The frequency-domain feature parameters include power spectral density, center frequency, and the proportion of harmonic components.

[0025] The extracted feature parameters are compared with the historical noise feature vectors stored in the noise feature library in a dimension-by-dimensional similarity calculation. A composite metric method combining Euclidean distance and cosine similarity is used to obtain the matching degree between the current data block and each noise category.

[0026] A noise exclusion threshold is set. When the matching degree between a segment of data and any noise category exceeds the threshold, the segment is determined to be a pure noise segment. The data segments that are not determined to be pure noise segments and contain abnormal fluctuation point markers are expanded by m data points before and after to generate a target data segment containing the complete fluctuation process, where m is determined according to the data sampling frequency and the signal period length.

[0027] Furthermore, the steps for multi-level feature extraction of the target data segment include:

[0028] During the first layer of feature extraction, the basic statistics of the target data segment are calculated, including the data mean, standard deviation, kurtosis, skewness, range, and percentiles of the data distribution, forming a basic feature vector that contains the central tendency and dispersion of the data.

[0029] The second layer is based on the data fluctuation period. It determines the main period length of the signal through autocorrelation function analysis, extracts time-related features such as peak interval consistency, amplitude change rate of adjacent periods, and phase difference at the beginning of the period, and constructs a time-series feature vector by combining the waveform distortion coefficient after moving average filtering.

[0030] The third layer uses a sliding window with a variable window size. The window length is set to 1-3 times the length of the basic period. The basic statistical features and time-related features in adjacent sliding windows are differentially calculated to extract trend features such as the time change rate, fluctuation trend slope, and extreme point migration distance of the feature parameters, and to generate a trend feature vector containing multi-scale change information.

[0031] The three layers of feature vectors are concatenated according to their dimensions to form a multi-scale feature vector set that includes basic statistics, time-series correlation, and trend changes.

[0032] Further, the step of ranking the feature vector set by importance according to the preset experimental objectives, retaining the core feature vectors that have a significant impact on the experimental conclusions, and updating the preset tracking experimental model using the core feature vectors includes:

[0033] Establish a feature importance evaluation function, and calculate the contribution index of each feature vector based on the correlation between the feature and the experimental target. The correlation is quantified by Spearman correlation coefficient, information gain ratio or weight coefficient based on a preset regression model.

[0034] The feature vector set is sorted in descending order according to the contribution index, and the core feature vectors that have an impact on the experimental conclusions exceeding the critical value are selected according to the preset feature retention ratio or significance P-value threshold.

[0035] The core feature vector is input into a preset tracking experimental model, which includes a dynamically adjustable parameter layer. The weight matrix and bias term in the model parameter layer are iteratively updated by minimizing the root mean square error between the current experimental data and the model prediction results.

[0036] During the update process, the historical change curve of the feature contribution is recorded synchronously. When the contribution of a certain feature fluctuates beyond the preset range in n consecutive data batches, the adaptive adjustment mechanism of the model structure is triggered to optimize the connection method of the feature input layer.

[0037] The present invention proposes an experimental data processing device, comprising:

[0038] The acquisition unit is used to acquire raw data collected from multi-dimensional experiments, and to process the raw data into blocks according to a preset data type to obtain a block dataset with a unified data format.

[0039] A data unit is used to generate a similarity matrix between data points based on the local neighborhood information of the corresponding data points in the time series or spatial distribution of each block dataset, and to mark abnormal fluctuation points in the block dataset according to the similarity matrix.

[0040] The matching unit is used to perform feature matching between the labeled block dataset and the preset noise feature library to separate the target data segment containing the effective signal;

[0041] The extraction unit is used to perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation cycle, and the third layer analyzes the feature change trend of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0042] The update unit is used to sort the feature vector set by importance according to the preset experimental objectives, retain the core feature vectors that have a significant impact on the experimental conclusions, and update the preset tracking experimental model through the core feature vectors.

[0043] This invention also proposes an experimental data AI analysis module, comprising:

[0044] An integrated data processing chip has a built-in data segmentation processing circuit, which is used to identify and segment raw data collected from multiple dimensions according to sensor protocol type, image data structure, and text log format, and generate segmented datasets in a unified structured format.

[0045] The neighborhood similarity calculation unit includes a time series processing sub-circuit and a spatial distribution processing sub-circuit. The former constructs a time window register group based on the data sampling frequency to realize the Euclidean distance calculation of adjacent data points, while the latter completes the cosine similarity calculation of the feature vectors of neighboring points through a spatial grid partitioning logic circuit.

[0046] The abnormal fluctuation marking module is equipped with a statistical computing hardware unit, which can calculate the mean and standard deviation of the local neighborhood similarity of data points in real time, and mark abnormal points through a threshold comparator. The time series data processing channel integrates a fluctuation amplitude change rate calculation circuit, and the spatial data channel integrates a feature gradient change detection circuit.

[0047] The noise feature matching chip has a built-in historical noise feature storage array, supports parallel matching calculation of time-domain features (peak value, slope, pulse width) and frequency-domain features (power spectrum, center frequency, harmonic components), and realizes the noise category determination of data segments through composite metric logic circuits;

[0048] The multi-layer feature extraction hardware architecture consists of three layers: the first layer is the basic statistical feature operation unit, which includes modules for calculating the mean, variance, and percentiles; the second layer is the time-series correlation analysis module, which integrates an autocorrelation function operator and a periodic feature extraction circuit; and the third layer is the sliding window trend analysis unit, which has a variable window length control register and feature difference calculation logic.

[0049] The feature selection and model update unit includes a contribution calculation coprocessor that supports hardware-accelerated calculation of Spearman coefficients and information gain rate, as well as a tracking experimental model parameter adjustment circuit. This circuit drives the iterative update of the parameter layer weight matrix and bias terms through the root mean square error calculation module.

[0050] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described experimental data processing method.

[0051] The experimental data processing method, apparatus, AI analysis module, and computer equipment provided by this invention have the following beneficial effects:

[0052] (1) Combining time series neighborhood similarity and spatial distribution feature gradient analysis to mark outliers, and simultaneously introducing a noise feature library for multi-level feature matching, effectively distinguishing noise signals from valid fluctuations and improving the accuracy of outlier data identification;

[0053] (2) By extracting features at multiple levels, a set of multi-scale feature vectors containing basic statistics, time-series correlation, and trend changes is formed. The core features are dynamically selected and the model is updated based on the experimental objectives, so that the tracking experimental model can adapt to changes in data characteristics in real time, significantly improving the reliability of experimental conclusions and the accuracy of model prediction. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating an experimental data processing method according to one embodiment of the present invention;

[0055] Figure 2 This is a structural block diagram of an experimental data processing device according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] Reference Figure 1 This is a flowchart illustrating the experimental data processing method proposed in this invention. The method includes the following steps:

[0060] S1, acquire the raw data collected from the multi-dimensional experiment, and process the raw data into blocks according to the preset data type to obtain a block dataset with a unified data format;

[0061] S2, For the local neighborhood information of the corresponding data points in the time series or spatial distribution in each block dataset, generate a similarity matrix between data points, and mark the abnormal fluctuation points in the block dataset according to the similarity matrix;

[0062] S3, perform feature matching between the labeled block dataset and the preset noise feature library to separate the target data segment containing the effective signal;

[0063] S4. Multi-level feature extraction is performed on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation cycle, and the third layer analyzes the feature change trend of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0064] S5. Sort the feature vector set by importance according to the preset experimental objectives, retain the core feature vectors that have a significant impact on the experimental conclusions, and update the preset tracking experimental model through the core feature vectors.

[0065] In steps S1 to S5, the provided experimental data processing method aims to solve the technical problems of heterogeneous data compatibility, noise interference removal, feature depth extraction, and model adaptive optimization in multi-dimensional experimental data processing. The specific implementation steps are as follows:

[0066] First, for raw data collected from multiple sources such as sensors, images, and logs, differentiated block processing is performed according to data type (time series, spatial distribution): sensor data streams are cut into sliding time windows according to the acquisition frequency, and image data is divided into overlapping sub-blocks according to the grid. Simultaneously, unit normalization, timestamp alignment, and other format standardization are completed to form a unified structured block dataset, solving the compatibility problem of heterogeneous data preprocessing.

[0067] Secondly, a similarity matrix is ​​constructed between data points to mark abnormal fluctuation points: for time series data, the Euclidean distance is calculated based on the k nearest neighbors determined by the sampling frequency; for spatial data, the cosine similarity is calculated by defining an m×m neighborhood with a preset radius, and the two are weighted to generate a comprehensive similarity matrix. Anomaly thresholds are dynamically set by subtracting the standard deviation from the statistical mean, and combined with the rate of change of fluctuation amplitude within the time window and the gradient changes of spatial features, anomaly points are accurately marked, avoiding the missed and false positives of traditional fixed threshold detection.

[0068] Then, effective signal separation is achieved through a noise feature library: the time domain (peak value, slope) and frequency domain (power spectrum, center frequency) features of the data segment are extracted, and the historical noise features are matched using a composite metric method of Euclidean distance and cosine similarity. High-matching pure noise segments are excluded, and the data segments containing outliers are expanded to generate target data segments with complete fluctuation processes, ensuring that subsequent analysis focuses on effective signals.

[0069] Finally, the experimental model was optimized through a three-layer feature extraction and dynamic screening process: the first layer extracted basic statistical features such as mean and variance; the second layer analyzed periodic characteristics and extracted time-series correlation features based on the autocorrelation function; and the third layer calculated trend change features using variable window differencing, forming a multi-scale feature vector set. Feature contribution was evaluated based on Spearman correlation coefficient and other metrics, and core features were selected for input into the tracking experimental model. Model parameters were iteratively updated by minimizing the root mean square error, and fluctuations in feature contribution were monitored synchronously to trigger adaptive structural adjustments, achieving dynamic adaptation of the model to the experimental objectives.

[0070] This method significantly improves the processing accuracy and model prediction reliability of complex experimental data through closed-loop processing of data block standardization, multi-dimensional anomaly detection, multi-layer feature fusion, and adaptive model updates. It is suitable for efficient data processing needs in various scenarios such as scientific research analysis and industrial testing.

[0071] In one embodiment, the step of dividing the original data into blocks according to a preset data type to obtain a block dataset with a unified data format includes:

[0072] Identify the protocol type and data structure of different data acquisition devices in the raw data, and label the raw data according to the preset classification standards of sensor data, image data and text log data.

[0073] For time-series sensor data, a sliding time window is set as the block unit according to the data acquisition frequency, and the continuously acquired data stream is divided into segments at fixed time intervals.

[0074] For spatially distributed image data, the image matrix is ​​divided into sub-image blocks with overlapping boundaries according to a preset grid division rule;

[0075] Each data unit after being divided into blocks undergoes format standardization processing, including data unit normalization, timestamp alignment, and missing value completion, so that different types of block data are converted into a unified structured format containing data identifiers, collection time, and feature parameters.

[0076] In practical implementation, the process of dividing the raw data into blocks according to preset data types to obtain a unified data format block dataset mainly achieves the compatible integration of multi-dimensional data through multi-source data type identification, differentiated block strategies, and format standardization. First, the protocol parsing module identifies the protocol types (such as Modbus, TCP / IP, etc.) and data structure characteristics of different acquisition devices in the raw data. For sensor data, metadata such as device type, sampling frequency, and data unit is extracted. For image data, matrix parameters such as file format, resolution, and number of channels are parsed. For text log data, regular expressions are used to match the log entry structure and extract the timestamp format and parameter value range. Then, each data unit is labeled with a type label according to preset classification standards, indicating whether it is sensor data, image data, or text log data, providing a classification basis for subsequent processing. For time-series sensor data, a sliding time window is set as the block unit based on its data acquisition frequency f. The window length T is set to 1-2 times the typical signal period to ensure that the window contains complete signal fluctuations. The sliding step size is set to T / 2 to ensure that adjacent windows overlap by 50% to avoid truncation of signal abrupt changes. Thus, the continuous data stream is cut into multiple block units composed of equally spaced sampling points at fixed time intervals, forming fixed-length data segments in the time dimension. For spatially distributed image data, according to the preset grid division rules of m×m pixel grid size and k pixel overlap boundary width (usually k=1-2), the grid is slid row by row and column by column starting from the upper left corner of the image, with a step size s=mk, to divide the image matrix into sub-image blocks with overlapping boundaries. Image edge features are preserved, and spatial metadata such as sub-block coordinates and pixel value ranges are extracted and bound to the pixel matrix data for storage. In the format standardization stage, data unit normalization is performed on each data unit after segmentation, and the units of different physical quantities are unified by establishing a unit conversion table; timestamp alignment is performed, and the time deviation of data collected synchronously by multiple devices is corrected by linear interpolation based on the master clock device timestamp, and the error is controlled within ±1 sampling period; missing value completion is implemented, and the missing points of time series data are filled by the weighted average of the five valid points before and after (the weight decreases with distance), and the missing points of image data are filled by the median value of the neighboring pixels. Finally, different types of segmented data are converted into data identifiers containing the device ID of the data block, the segment timestamp, and the spatial coordinates (for image data), with the acquisition time of the unified time base accurate to the microsecond level, and the unified structured format of the feature parameters of the standardized numerical matrix or text field, which solves the compatibility problem of heterogeneous data preprocessing.

[0077] In one embodiment, the step of generating a similarity matrix between data points based on the local neighborhood information of the corresponding data points in the time series or spatial distribution of each data block includes:

[0078] For time series type block datasets, the k neighboring data points before and after the current data point are determined as local neighborhoods based on the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for spatial distribution type block datasets, the neighboring points within an m×m range around the current data point are delineated according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated through the cosine similarity of the feature vectors.

[0079] The temporal and spatial similarities are linearly combined according to preset weights to obtain the comprehensive similarity value between data points.

[0080] Using all data points in the segmented dataset as matrix nodes, and taking the combined similarity value of any two data points as matrix elements, a symmetric similarity matrix is ​​constructed. The diagonal elements of the matrix are set to 1 to indicate that they are completely similar.

[0081] In practical implementation, for time-series segmented datasets (such as continuous signals acquired by sensors), the current data point is used as the center, and the k adjacent data points before and after it are determined as local neighborhoods based on the data sampling frequency and the preset time window length. The setting of this neighborhood range aims to capture the short-term fluctuation correlation of the data in the time dimension—the sampling frequency determines the time resolution, and the time window length corresponds to the typical fluctuation period of the signal, ensuring that the neighborhood contains sufficient contextual information to reflect the time dependence of the data points. By calculating the Euclidean distance between the current point and each point in the neighborhood as the time dimension similarity, this distance metric can intuitively reflect the difference in the amplitude of numerical fluctuations; the smaller the distance, the closer the trend of change in the time series.

[0082] For spatially distributed block datasets (such as pixels in image sub-blocks), neighboring points within an m×m range around the current data point are defined according to a preset spatial neighborhood radius. This grid range considers the local continuity of image features (e.g., textures and edges typically maintain consistency within a small spatial area). The design of overlapping boundaries (as described in claim 2, sub-image block partitioning) ensures the integrity of the neighborhood of edge points. Spatial dimensional similarity is calculated using the cosine similarity of feature vectors. This method focuses on the directional consistency of pixel feature vectors (rather than absolute numerical differences), making it suitable for capturing structural similarities in spatial distributions (e.g., pixels of the same material have similar vector directions in the RGB color space).

[0083] After obtaining the similarity in the time and spatial dimensions, the two are linearly combined using preset weights to obtain the comprehensive similarity value between data points. The weight allocation can be dynamically adjusted according to the characteristics of the data type (e.g., sensor data emphasizes the time dimension, while image data emphasizes the spatial dimension), thus adapting to the core feature requirements of different experimental scenarios. Finally, using all data points in the segmented dataset as matrix nodes, a symmetric similarity matrix is ​​constructed by using the comprehensive similarity value of any two points as matrix elements. The diagonal elements of the matrix are set to 1 to indicate that the data points are completely similar.

[0084] In one embodiment, the step of marking anomalous fluctuation points in a segmented dataset according to the similarity matrix includes:

[0085] Calculate the mean of the off-diagonal elements of the corresponding row in the similarity matrix for each data point, and use it as the local neighborhood similarity index for that data point.

[0086] Set an abnormal fluctuation judgment threshold, which is the average value of the local neighborhood similarity index of all data points in the segmented dataset minus n times the standard deviation, where n is a preset abnormal sensitivity coefficient;

[0087] Data points whose local neighborhood similarity index is lower than the abnormal fluctuation judgment threshold are marked as abnormal fluctuation points. The marking of time series data points is combined with the rate of change of fluctuation amplitude within the preceding and following time windows, and the marking of spatially distributed data points is combined with the feature gradient change within the neighborhood space.

[0088] In practice, for each data point, the mean of the off-diagonal elements in the corresponding row of the similarity matrix is ​​calculated. This mean is used as a local neighborhood similarity index—reflecting the average similarity between the current data point and other points in its neighborhood. A higher value indicates that the data point's fluctuation pattern or spatial structure is more consistent with its surrounding neighborhood, while a lower value indicates a greater deviation from the local data distribution characteristics. By excluding diagonal elements (whose own similarity is always 1), the focus is on the correlation between the data point and its external neighborhood, avoiding interference from perfect self-similarity in anomaly detection.

[0089] Subsequently, a dynamic threshold setting method based on the overall distribution of the dataset is adopted: statistical analysis is performed on the local neighborhood similarity index of all data points, and its mean and standard deviation are calculated. The threshold for judging abnormal fluctuations is defined as "mean minus n times the standard deviation", where n is a preset abnormality sensitivity coefficient (usually taken as 1-3, the larger n is, the lower the threshold and the higher the detection sensitivity). This threshold setting based on standard deviation can adapt to the dispersion of data distribution. Compared with the fixed threshold method, it can effectively cope with the differences in the fluctuation characteristics of different experimental data and avoid missed or false judgments caused by changes in data variance.

[0090] In the specific labeling process, differentiated auxiliary judgment conditions are introduced for time series and spatial distribution data: For time series data points, based on the similarity index, the rate of change of fluctuation amplitude within the preceding and following time windows (i.e., the ratio of the difference in data peak values ​​within adjacent time windows to the length of the time window) is calculated simultaneously. If the similarity index is lower than the threshold and the rate of change of fluctuation amplitude exceeds the empirical threshold (e.g., 20%), it is judged as an anomaly, thereby capturing sudden and drastic fluctuations; For spatial distribution data points, the feature gradient changes within the neighborhood space range (e.g., the gradient magnitude of pixel RGB values) are combined. When the similarity index is lower than the threshold and the gradient change exceeds 1.5 times the local average, it is marked as an anomaly, ensuring the accurate identification of edge features or texture abrupt changes.

[0091] This labeling method employs a two-layer mechanism of "initial screening with statistical thresholds + precise judgment based on domain features." It utilizes the global statistical characteristics of the similarity matrix to filter out commonly similar data points, while adding specific judgment conditions to address the dynamic fluctuations of time series and the structural abrupt changes in spatial distribution. This effectively solves the limitations of single statistical methods in complex data scenarios.

[0092] In one embodiment, the step of performing feature matching between the labeled block dataset and a preset noise feature library to separate the target data segment containing the valid signal includes:

[0093] Time-domain and frequency-domain feature parameters are extracted from the labeled block dataset. The time-domain feature parameters include waveform peak value, rising edge slope, and pulse width. The frequency-domain feature parameters include power spectral density, center frequency, and the proportion of harmonic components.

[0094] The extracted feature parameters are compared with the historical noise feature vectors stored in the noise feature library in a dimension-by-dimensional similarity calculation. A composite metric method combining Euclidean distance and cosine similarity is used to obtain the matching degree between the current data block and each noise category.

[0095] A noise exclusion threshold is set. When the matching degree between a segment of data and any noise category exceeds the threshold, the segment is determined to be a pure noise segment. The data segments that are not determined to be pure noise segments and contain abnormal fluctuation point markers are expanded by m data points before and after to generate a target data segment containing the complete fluctuation process, where m is determined according to the data sampling frequency and the signal period length.

[0096] In practical implementation, time-domain and frequency-domain feature parameters are extracted synchronously from the labeled block dataset: time-domain analysis focuses on the instantaneous change characteristics of the signal, extracting parameters such as waveform peak value (reflecting signal strength), rising edge slope (characterizing the rate of change), and pulse width (reflecting duration). For example, for pulse signals acquired by sensors, the rising edge slope can effectively distinguish noise interference from real pulses. Frequency-domain analysis converts the time-domain signal to frequency space through fast Fourier transform, extracting features such as power spectral density (reflecting energy distribution), center frequency (characterizing the main vibration mode), and the proportion of harmonic components (reflecting signal periodicity). For example, mechanical vibration noise usually has energy concentration at specific harmonic frequencies.

[0097] Subsequently, a composite metric combining Euclidean distance and cosine similarity is employed to calculate the matching degree between the current data segment and historical feature vectors in the noise feature database. Euclidean distance quantifies the numerical differences of feature parameters (such as peak voltage deviation) and is suitable for measuring the consistency of absolute quantities; cosine similarity focuses on the directional similarity of feature vectors (such as the power spectrum distribution shape) and is suitable for capturing the matching degree of pattern structure. By weighted fusion of the two metric results (the weights can be dynamically adjusted according to the noise characteristics), the comprehensive matching degree between the data segment and each noise category is obtained. This method overcomes the sensitivity of a single metric to numerical offsets or morphological differences, for example, it can still accurately identify noise even when the peak noise amplitude changes but the power spectrum shape remains the same.

[0098] In the noise assessment stage, a preset noise exclusion threshold (e.g., matching degree ≥ 0.8) is used. When the matching degree of a data segment with any noise category exceeds the threshold, it is judged as a pure noise segment and directly eliminated. For data segments that are not judged as pure noise but contain abnormal fluctuation point markers, a target data segment is generated by expanding the segment by m data points before and after the abnormal point. The value of m follows the empirical formula of "sampling frequency × signal period / 2" (e.g., m = 5 when the sampling frequency is 100Hz and the period is 0.1 seconds), ensuring that the expanded data segment contains the complete fluctuation process before and after the abnormal point. For example, for abnormal points with voltage abrupt changes, the expanded segment can retain the stable state before the abrupt change and the recovery process after the abrupt change, providing a complete signal sample for subsequent feature extraction.

[0099] This noise separation mechanism achieves efficient filtering of noise signals and accurate preservation of effective fluctuations through multi-dimensional characterization of time-frequency domain features, robust matching of composite metrics, and integrity assurance of dynamic expansion.

[0100] In one embodiment, the step of performing multi-level feature extraction on the target data segment includes:

[0101] During the first layer of feature extraction, the basic statistics of the target data segment are calculated, including the data mean, standard deviation, kurtosis, skewness, range, and percentiles of the data distribution, forming a basic feature vector that contains the central tendency and dispersion of the data.

[0102] The second layer is based on the data fluctuation period. It determines the main period length of the signal through autocorrelation function analysis, extracts time-related features such as peak interval consistency, amplitude change rate of adjacent periods, and phase difference at the beginning of the period, and constructs a time-series feature vector by combining the waveform distortion coefficient after moving average filtering.

[0103] The third layer uses a sliding window with a variable window size. The window length is set to 1-3 times the length of the basic period. The basic statistical features and time-related features in adjacent sliding windows are differentially calculated to extract trend features such as the time change rate, fluctuation trend slope, and extreme point migration distance of the feature parameters, and to generate a trend feature vector containing multi-scale change information.

[0104] The three layers of feature vectors are concatenated according to their dimensions to form a multi-scale feature vector set that includes basic statistics, time-series correlation, and trend changes.

[0105] In practical implementation, the first step is to extract the first layer of features. This involves calculating basic statistics such as the mean, standard deviation, kurtosis, skewness, range, and percentiles (e.g., 25th and 75th percentiles) of the target data segment to form a feature vector representing the central tendency and dispersion of the data. The mean reflects the average level of the data, the standard deviation quantifies the fluctuation range, the kurtosis describes the steepness of the distribution, the skewness reflects symmetry, the range characterizes the data range, and the percentiles help identify outlier distributions. These parameters together constitute a comprehensive characterization of the basic properties of the data. For example, for temperature data collected by sensors, the mean reflects the ambient baseline temperature, while the standard deviation reflects the temperature fluctuation range.

[0106] Secondly, the second-layer feature extraction is based on the data fluctuation cycle. The length of the main period of the signal is determined through autocorrelation function analysis—the autocorrelation function peaks when the lag time equals the period. This identifies the periodic characteristics of the signal (such as the natural frequency period of mechanical vibration). Subsequently, time-series related features are extracted, including the consistency of peak intervals within the period (measuring period stability), the rate of change of amplitude between adjacent periods (reflecting energy attenuation or enhancement), and the phase difference at the start of the period (reflecting phase stability). These are then combined with the waveform distortion coefficient after moving average filtering (calculated through the mean square error between the ideal and actual waveforms) to construct a time-series feature vector. Taking a power system voltage signal as an example, the main period is typically 0.02 seconds (50Hz power frequency). If the consistency of peak intervals decreases within a certain period, it may indicate system frequency fluctuations, while abnormal amplitude change rates may reflect sudden load changes.

[0107] The third-layer feature extraction employs a sliding window mechanism with variable window size. The window length is set to 1-3 times the base period length (e.g., 0.02-0.06 seconds when the main period is 0.02 seconds). By performing difference calculations on the basic statistical features and time-related features within adjacent sliding windows, trend features such as the time change rate of feature parameters (e.g., the rate of change of mean and standard deviation), the slope of the fluctuation trend (obtained through linear fitting to determine upward or downward trends), and the migration distance of extreme points (the time difference between extreme points in adjacent windows) are extracted, generating a trend feature vector containing multi-scale change information. This variable window design can adaptively capture changes at different time scales—short windows (1 time period) focus on local fluctuation details, while long windows (3 time periods) focus on the overall trend evolution. For example, in seismic wave data processing, short windows can identify the high-frequency pulse characteristics of seismic waves, while long windows are used to analyze the low-frequency trends of crustal movement.

[0108] Finally, the three layers of feature vectors are concatenated according to their dimensions to form a multi-scale feature vector set that includes basic statistics, time-series correlation, and trend changes. This set retains the basic distribution characteristics of the data (first layer), while incorporating periodic dynamic characteristics (second layer) and long-term and short-term trend changes (third layer), achieving a full-dimensional feature representation from micro-level numerical fluctuations to macro-level trend evolution.

[0109] In one embodiment, the step of ranking the feature vector set by importance according to a preset experimental objective, retaining the core feature vectors that have a significant impact on the experimental conclusions, and updating the preset tracking experimental model using the core feature vectors includes:

[0110] Establish a feature importance evaluation function, and calculate the contribution index of each feature vector based on the correlation between the feature and the experimental target. The correlation is quantified by Spearman correlation coefficient, information gain ratio or weight coefficient based on a preset regression model.

[0111] The feature vector set is sorted in descending order according to the contribution index, and the core feature vectors that have an impact on the experimental conclusions exceeding the critical value are selected according to the preset feature retention ratio or significance P-value threshold.

[0112] The core feature vector is input into a preset tracking experimental model, which includes a dynamically adjustable parameter layer. The weight matrix and bias term in the model parameter layer are iteratively updated by minimizing the root mean square error between the current experimental data and the model prediction results.

[0113] During the update process, the historical change curve of the feature contribution is recorded synchronously. When the contribution of a certain feature fluctuates beyond the preset range in n consecutive data batches, the adaptive adjustment mechanism of the model structure is triggered to optimize the connection method of the feature input layer.

[0114] First, a feature importance evaluation function is established. This function quantifies the contribution index of each feature vector based on the correlation between the feature and the experimental objective. If the experimental objective is to analyze the relationship between data and a continuous variable (such as chemical reaction rate), the Spearman correlation coefficient can be used to measure the nonlinear monotonic correlation between the feature and the objective (for example, the higher the correlation coefficient between temperature standard deviation and reaction rate, the greater the impact of temperature fluctuations on the reaction rate). If the objective is a classification task (such as distinguishing between normal and abnormal experimental states), the information gain ratio is selected to evaluate the contribution of the feature to class distinction (for example, a high information gain ratio of a time-series feature indicates its strong ability to distinguish abnormal states). For scenarios where a regression model has been established (such as predicting the concentration of experimental products), the weight coefficients obtained from model training are directly used as the correlation (the larger the absolute value of the weight, the more significant the impact of the feature on the prediction result). The contribution index calculated by the above methods provides a quantitative basis for feature importance.

[0115] Next, the feature vector set is sorted in descending order according to the contribution index, and a selection strategy is chosen based on experimental needs: if feature simplification is emphasized, a retention ratio can be set (e.g., retaining the top 30% of high-contribution features) to reduce the computational complexity of the model; if statistical significance is emphasized, the p-value of the features is calculated through hypothesis testing (e.g., p < 0.05), and the core feature vectors whose influence on the experimental conclusion exceeds the critical value are selected. For example, in a material strength prediction experiment, if the p-value of a certain basic statistical feature (e.g., mean strain) is 0.03 (< 0.05), while the p-value of a certain trend feature (e.g., strain change rate) is 0.12 (> 0.05), then only the mean strain is retained as the core feature to avoid redundant information interfering with the model.

[0116] Subsequently, the selected core feature vectors are input into a pre-defined tracking experimental model. This model is designed with dynamically adjustable parameter layers (such as the fully connected layer of a neural network or the weight matrix of a linear regression). By minimizing the root mean square error (RMSE) between the current experimental data and the model's predictions, the weight matrix and bias terms in the parameter layers are iteratively updated—for example, using stochastic gradient descent to adjust the weights according to the gradient direction of the RMSE, so that the model's predicted values ​​gradually approach the actual experimental data. This process enables the model to adapt to the characteristics of the current data in real time. For example, in battery aging experiments, as the number of cycles increases, the contribution of key features of capacity decay (such as the standard deviation of voltage fluctuations) increases. The model updates the weights to strengthen the response to this feature, thereby improving the accuracy of aging state prediction.

[0117] During the update process, the historical change curve of the contribution of each feature is recorded synchronously. When the contribution of a certain feature fluctuates beyond a preset range (e.g., ±20%) in n consecutive data batches (e.g., n=5), the adaptive adjustment mechanism of the model structure is triggered. For example, if the contribution of a certain time-series feature was stable in the early stage, but its correlation with the target has dropped sharply recently due to changes in experimental conditions, the model will adjust the connection of the feature input layer—reducing the input weight of the feature or disconnecting its connection with the hidden layer, while possibly strengthening the connection of other emerging high-contribution features (e.g., newly added trend features). This dynamic structural adjustment enables the model to actively adapt to changes in experimental targets or data distribution, avoiding model performance degradation caused by feature importance drift.

[0118] This step solves the problem of poor adaptability caused by static feature input in traditional models through a closed-loop mechanism of "association quantification - core screening - parameter update - structural adaptation".

[0119] Reference Appendix Figure 2 The present invention provides a structural block diagram of an experimental data processing device, comprising:

[0120] The acquisition unit is used to acquire raw data collected from multi-dimensional experiments, and to process the raw data into blocks according to a preset data type to obtain a block dataset with a unified data format.

[0121] A data unit is used to generate a similarity matrix between data points based on the local neighborhood information of the corresponding data points in the time series or spatial distribution of each block dataset, and to mark abnormal fluctuation points in the block dataset according to the similarity matrix.

[0122] The matching unit is used to perform feature matching between the labeled block dataset and the preset noise feature library to separate the target data segment containing the effective signal;

[0123] The extraction unit is used to perform multi-level feature extraction on the target data segment. The first layer extracts the basic statistical features of the data, the second layer extracts the temporal correlation features based on the data fluctuation cycle, and the third layer analyzes the feature change trend of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features.

[0124] The update unit is used to sort the feature vector set by importance according to the preset experimental objectives, retain the core feature vectors that have a significant impact on the experimental conclusions, and update the preset tracking experimental model through the core feature vectors.

[0125] In one embodiment, the present invention also proposes an experimental data AI analysis module, comprising:

[0126] An integrated data processing chip has a built-in data segmentation processing circuit, which is used to identify and segment raw data collected from multiple dimensions according to sensor protocol type, image data structure, and text log format, and generate segmented datasets in a unified structured format.

[0127] The neighborhood similarity calculation unit includes a time series processing sub-circuit and a spatial distribution processing sub-circuit. The former constructs a time window register group based on the data sampling frequency to realize the Euclidean distance calculation of adjacent data points, while the latter completes the cosine similarity calculation of the feature vectors of neighboring points through a spatial grid partitioning logic circuit.

[0128] The abnormal fluctuation marking module is equipped with a statistical computing hardware unit, which can calculate the mean and standard deviation of the local neighborhood similarity of data points in real time, and mark abnormal points through a threshold comparator. The time series data processing channel integrates a fluctuation amplitude change rate calculation circuit, and the spatial data channel integrates a feature gradient change detection circuit.

[0129] The noise feature matching chip has a built-in historical noise feature storage array, supports parallel matching calculation of time-domain features (peak value, slope, pulse width) and frequency-domain features (power spectrum, center frequency, harmonic components), and realizes the noise category determination of data segments through composite metric logic circuits;

[0130] The multi-layer feature extraction hardware architecture consists of three layers: the first layer is the basic statistical feature operation unit, which includes modules for calculating the mean, variance, and percentiles; the second layer is the time-series correlation analysis module, which integrates an autocorrelation function operator and a periodic feature extraction circuit; and the third layer is the sliding window trend analysis unit, which has a variable window length control register and feature difference calculation logic.

[0131] The feature selection and model update unit includes a contribution calculation coprocessor that supports hardware-accelerated calculation of Spearman coefficients and information gain rate, as well as a tracking experimental model parameter adjustment circuit. This circuit drives the iterative update of the parameter layer weight matrix and bias terms through the root mean square error calculation module.

[0132] In this embodiment:

[0133] The integrated data processing chip adopts a heterogeneous multi-core architecture, and its internally embedded block processing circuit includes a protocol parsing micro-engine and a data block logic unit. The protocol parsing micro-engine parses sensor protocols (such as the CRC check field of Modbus RTU), image data structures (SOI / EOL markers in JPEG files), and text log formats (regular expression matching hardware) through a hardware state machine, and drives the block logic unit according to the parsing results: for sensor data streams, the sampling frequency clock generator controls the time window register group to achieve sliding cutting; for image data, the grid partitioning hardware module generates an overlapping sub-block address mapping table; the block-based data is processed by a unit normalized lookup table, a timestamp alignment phase-locked loop, and a missing value completion interpolation circuit, and outputs a data stream with a unified structured format to the buffer bus.

[0134] The neighborhood similarity calculation unit adopts a time-space parallel processing architecture: the time series processing sub-circuit includes a k-order shift register group (k is dynamically set by the sampling frequency configuration register), the Euclidean distance calculation between the current data point and the neighboring points is completed by a hardware multiplier-accumulator array, and the result is stored in the similarity temporary storage register; the spatial distribution processing sub-circuit traverses the pixels through an m×m neighborhood coordinate generator, and the cosine similarity of the feature vector is implemented by the hardware vector dot product unit and the modulus calculation unit in concert, and the overlapping boundary data is processed synchronously through a dual-port cache. After the two types of similarity are weighted by the weight configuration register, a symmetric similarity matrix is ​​generated by the matrix construction logic and stored in the on-chip high-speed RAM.

[0135] The abnormal fluctuation marking module integrates a statistical computing ASIC and a dual-channel feature detection circuit: the statistical computing unit calculates the row mean and standard deviation of the similarity matrix in real time through a pipeline structure, and the threshold comparator uses tri-state logic to output abnormal markings; the fluctuation amplitude change rate calculation circuit of the time series channel includes a differential amplifier and a slope comparator, and the feature gradient detection circuit of the spatial channel calculates the gradient magnitude through a convolution kernel hardware accelerator. The two types of auxiliary features are fused with the statistical threshold results through AND gate circuits to generate the final abnormal point marking signal.

[0136] The noise feature matching chip adopts the Harvard architecture. The historical noise feature storage array is divided into a time-domain feature area (peak / slope / pulse width lookup table) and a frequency-domain feature area (power spectrum template library). The parallel matching calculation unit contains 8 sets of time-domain feature comparators and FFT hardware acceleration modules. The composite metric logic circuit outputs the noise category matching degree by weighting the results of the Euclidean distance calculation unit and the cosine similarity multiplier. When the matching degree exceeds the threshold register setting value, the noise judgment state machine outputs a pure noise segment rejection signal.

[0137] The multi-layer feature extraction hardware architecture adopts a three-stage pipeline design: the first-layer basic statistical unit consists of a mean accumulator, a variance calculation unit, and a percentile sorting circuit working in parallel, with the results stored in a feature vector register; the second-layer time series analysis module identifies the main period through an autocorrelation function hardware calculator, and the periodic feature extraction circuit includes a peak interval counter, an amplitude change rate multiplier, and a phase difference comparator, with waveform distortion coefficients generated by a moving average filter kernel and an error calculation unit; the third-layer sliding window unit's variable-length control register supports 1-3 times the period window configuration, and the feature difference calculation logic caches adjacent window features through a register file, outputting the trend change rate and extreme value migration distance. The three layers of features are concatenated by a bus arbitrator and stored in a feature vector set memory.

[0138] The feature selection and model update unit includes a dedicated coprocessor and a dynamic parameter adjustment circuit: the contribution calculation coprocessor has a built-in Spearman coefficient calculation array and information gain ratio lookup table, which supports hardware-accelerated evaluation of feature importance; the model parameter adjustment circuit generates gradient signals through the root mean square error operator, which drives the weight matrix updater and bias term adjuster to perform iterative optimization; the feature contribution history recorder is implemented by a ring buffer. When the fluctuation of n consecutive batches exceeds the threshold comparator setting range, the structure adaptive adjustment state machine is triggered to reconfigure the multiplexer connection relationship of the feature input layer.

[0139] Reference Figure 3 This invention also provides a computer device, which can be a server, and its internal structure can be as follows: Figure 3 As shown, the computer device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores the data corresponding to this embodiment. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0140] Those skilled in the art will understand that Figure 3 The structures shown are merely block diagrams of some structures related to the present invention and do not constitute a limitation on the computer devices on which the present invention is applied.

[0141] In summary, this invention discloses an experimental data processing method and an AI analysis module, aiming to improve the accuracy and model adaptability of complex experimental data processing. The method includes: segmenting multi-dimensional raw data by type and unifying the format; generating a similarity matrix based on temporal / spatial neighborhood information to mark abnormal fluctuation points; separating effective signals by matching a noise library with time-frequency features; extracting multi-layer features of basic statistics, time-series correlation, and trend changes; and dynamically selecting core features to update the tracking experimental model. The accompanying AI analysis module integrates hardware circuitry for data segmentation, similarity calculation, anomaly marking, noise matching, feature extraction, and model updating, achieving full-process acceleration. This method significantly improves the efficiency of experimental data processing and the reliability of conclusions through multi-dimensional data compatibility processing, accurate anomaly detection, multi-layer feature fusion, and adaptive model optimization, and is suitable for real-time analysis in various scenarios such as scientific research and industry.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the present invention and embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0143] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0144] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An experimental data processing method, characterized by, The method comprises the following steps: obtaining original data collected by multi-dimensional experiments, and performing block processing on the original data according to a preset data type to obtain a block data set with a unified data format; generating a similarity matrix between data points according to local neighborhood information of the data points in each block data set in a time sequence or a spatial distribution, and marking abnormal fluctuation points in the block data set according to the similarity matrix; performing feature matching on the marked block data set and a preset noise feature library to separate a target data segment containing effective signals; performing multi-level feature extraction on the target data segment, wherein a first layer extracts basic statistical features of the data, a second layer extracts time sequence correlation features based on a data fluctuation period, and a third layer analyzes feature variation trends of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features; wherein the multi-level feature extraction on the target data segment comprises: in the first layer feature extraction, basic statistical quantities of the target data segment are calculated, including data mean, standard deviation, kurtosis value, skewness value, range, and percentile of data distribution, to form a basic feature vector containing trend and dispersion degree of the data set; in the second layer, the main period length of the signal is determined through autocorrelation function analysis based on the data fluctuation period, time sequence correlation features such as peak value interval consistency in the period, adjacent period amplitude change rate, and period start phase difference are extracted, and a time sequence feature vector is constructed in combination with a waveform distortion coefficient after moving average filtering; in the third layer, a sliding window with variable window size is used, the window length is set to 1-3 times the basic period length, the basic statistical features and the time sequence correlation features in adjacent sliding windows are differentially calculated, trend features such as time change rate of feature parameters, fluctuation trend slope, and extreme point migration distance are extracted, and a trend feature vector containing multi-scale change information is generated; the three-layer feature vectors are spliced according to dimensions to form a multi-scale feature vector set containing basic statistics, time sequence correlation, and trend change; the feature vector set is sorted according to a preset experimental target, core feature vectors having a significant influence on experimental conclusions are retained, and the preset tracking experimental model is updated through the core feature vectors.

2. The experimental data processing method of claim 1, wherein, The step of performing block processing on the original data according to a preset data type to obtain a block data set with a unified data format comprises: identifying protocol types and data structures corresponding to different data acquisition devices in the original data, and performing type labeling on the original data according to preset classification standards of sensor data, image data, and text log data; for time sequence type sensor data, a sliding time window is set as a block unit according to a data acquisition frequency, and continuously acquired data streams are cut at fixed time intervals; for spatial distribution type image data, the image matrix is divided into sub-image blocks with overlapping boundaries according to a preset grid division rule; each block of data after block processing is subjected to format standardization processing, including data unit normalization, timestamp alignment, and missing value completion, so that different types of block data are converted into a unified structured format containing data identification, acquisition time, and feature parameters.

3. The experimental data processing method of claim 1, wherein, The step of generating a similarity matrix between data points according to local neighborhood information of corresponding data points in each sub-block data set on time series or spatial distribution includes: For the time series type sub-block data set, taking the current data point as the center, the k adjacent data points before and after the current data point are determined as the local neighborhood according to the data sampling frequency and the preset time window length, and the Euclidean distance between the current point and each point in the neighborhood is calculated as the time dimension similarity; for the spatial distribution type sub-block data set, the neighboring points within the m*m range around the current data point are determined according to the preset spatial neighborhood radius, and the spatial dimension similarity is calculated by the cosine similarity of the feature vector; The time dimension similarity and the spatial dimension similarity are linearly combined according to the preset weight to obtain the comprehensive similarity value between the data points; Taking all data points in the sub-block data set as matrix nodes, the comprehensive similarity value between any two data points is taken as the matrix element, and a symmetric similarity matrix is constructed, and the diagonal line elements of the matrix are set to 1 to represent complete similarity.

4. The experimental data processing method of claim 3, wherein, The step of marking the abnormal fluctuation points in the sub-block data set according to the similarity matrix includes: The average value of the non-diagonal line elements of each data point in the corresponding row of the similarity matrix is calculated as the local neighborhood similarity index of the data point; An abnormal fluctuation determination threshold is set, which is the average value of the local neighborhood similarity indexes of all data points in the sub-block data set minus n times the standard deviation, and n is a preset abnormal sensitivity coefficient; The data points with the local neighborhood similarity index lower than the abnormal fluctuation determination threshold are marked as abnormal fluctuation points, wherein the marking of the time series data points is combined with the fluctuation amplitude change rate in the front and rear time windows, and the marking of the spatial distribution data points is combined with the feature gradient change in the neighborhood spatial range.

5. The experimental data processing method of claim 1, wherein, The step of separating the target data segment containing the effective signal by performing feature matching between the marked sub-block data set and the preset noise feature library includes: Time domain feature parameters and frequency domain feature parameters are extracted from the marked sub-block data set, the time domain feature parameters include waveform peak value, rising slope and pulse width, and the frequency domain feature parameters include power spectral density, center frequency and frequency component proportion; The extracted feature parameters and the historical noise feature vectors stored in the noise feature library are calculated for similarity in each dimension, and a composite measurement method combining Euclidean distance and cosine similarity is used to obtain the matching degree of the current sub-block data segment and each noise category; When the matching degree of the sub-block data segment and any noise category exceeds the threshold, the data segment is determined as a pure noise segment; the data segment containing the abnormal fluctuation point mark is generated by extending m data points before and after the data segment to generate a target data segment containing a complete fluctuation process, wherein m is determined according to the data sampling frequency and the signal period length.

6. The experimental data processing method of claim 1, wherein, The step of updating the preset tracking experiment model through the core feature vector sorted according to the preset experimental target includes: A feature importance evaluation function is established, and a contribution degree index of each feature vector is calculated based on the correlation between the feature and the experimental target, which is quantified by using a Spearman correlation coefficient, an information gain rate, or a weight coefficient based on a preset regression model; The feature vector set is arranged in descending order according to the contribution degree index, and the core feature vectors whose influence on the experimental conclusion exceeds a critical value are screened out according to a preset feature retention ratio or a significance P value threshold; the core feature vectors are input into a preset tracking experimental model, which includes a dynamically adjustable parameter layer, and the weight matrix and bias term in the parameter layer are iteratively updated by minimizing the root mean square error between the current experimental data and the model prediction results; During the updating process, the historical change curve of the feature contribution degree is recorded synchronously, and when the contribution degree of a certain feature fluctuates beyond a preset range in consecutive n data batches, a model structure adaptive adjustment mechanism is triggered to optimize the connection mode of the feature input layer.

7. An experimental data processing apparatus, characterized by, Comprise: An acquisition unit is configured to acquire original data collected by a multi-dimensional experiment, block process the original data according to a preset data type, and obtain a block data set with a unified data format; A data unit is configured to generate a similarity matrix between data points according to local neighborhood information of the data points in each block data set in a time sequence or a spatial distribution, and mark abnormal fluctuation points in the block data set according to the similarity matrix; A matching unit is configured to perform feature matching on the marked block data set and a preset noise feature library, and separate a target data segment containing effective signals; An extraction unit is configured to perform multi-level feature extraction on the target data segment, wherein a first layer extracts basic statistical features of the data, a second layer extracts time sequence correlation features based on a data fluctuation period, and a third layer analyzes the feature change trend of adjacent data blocks through a sliding window to form a feature vector set containing multi-scale features; wherein the steps of performing multi-level feature extraction on the target data segment comprise: In the first layer feature extraction, basic statistical quantities of the target data segment are calculated, including data mean, standard deviation, kurtosis value, skewness value, range, and percentile of data distribution, to form a basic feature vector containing the trend and dispersion degree of the data in the set; In the second layer, the main period length of the signal is determined through autocorrelation function analysis based on the data fluctuation period, and time sequence correlation features such as peak value interval consistency in the period, adjacent period amplitude change rate, and period start phase difference are extracted, and a time sequence feature vector is constructed by combining the waveform distortion coefficient after moving average filtering; In the third layer, a sliding window with variable window size is used, and the window length is set to 1-3 times the basic period length, the basic statistical features and time sequence correlation features in adjacent sliding windows are calculated by difference, trend features such as time change rate, fluctuation trend slope, and extreme point migration distance of feature parameters are extracted, and a trend feature vector containing multi-scale change information is generated; The three-layer feature vectors are spliced by dimension to form a multi-scale feature vector set containing basic statistics, time sequence correlation, and trend change. An updating unit is configured to sort the feature vector set according to a preset experiment target, retain core feature vectors having a significant influence on an experiment conclusion, and update a preset tracking experiment model through the core feature vectors 8. An experimental data AI analysis module, characterized by, Comprise: An integrated data processing chip, which internally solidifies a data block processing circuit, is used for identifying and blocking the original data collected in multiple dimensions according to sensor protocol types, image data structures and text log formats, and generating a block data set in a unified structured format. A neighborhood similarity calculation unit includes a time series processing sub-circuit and a spatial distribution processing sub-circuit. The former realizes Euclidean distance calculation of adjacent data points based on a time window register group constructed based on a data sampling frequency, and the latter completes cosine similarity operation of adjacent point feature vectors through a spatial grid division logic circuit. An abnormal fluctuation marking module is configured with a statistical calculation hardware unit, which can calculate the local neighborhood similarity mean and standard deviation of data points in real time, and realize abnormal point marking through a threshold comparator. A time series data processing channel is integrated with a fluctuation amplitude change rate calculation circuit, and a spatial data channel is integrated with a feature gradient change detection circuit. A noise feature matching chip internally stores an array of historical noise features, supports parallel matching calculation of time domain features and frequency domain features, and realizes data segment noise category determination through a composite metric logic circuit. A multi-layer feature extraction hardware architecture, a first layer is a basic statistical feature operation unit, including mean, variance and percentile calculation modules; a second layer is a time series correlation analysis module, integrating an autocorrelation function operator and a period feature extraction circuit. A third layer is a sliding window trend analysis unit, which has a variable window length control register and a feature difference calculation logic. A feature screening and model updating unit includes a contribution calculation coprocessor, which supports hardware accelerated calculation of Spearman coefficients and information gain rates, and a tracking experiment model parameter adjustment circuit. The circuit drives the iterative update of parameter layer weight matrix and bias term through a root mean square error operation module. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8. The processor executes the computer program to realize the steps of the experimental data processing method in any one of claims 1 to 6.

Images