Micro-service-based industrial internet of things data preprocessing method, medium and system

Abstract

The application provides a micro-service-based industrial Internet of Things data preprocessing method, medium and system, belongs to the technical field of micro-service-based industrial Internet of Things data preprocessing method, medium and system, and the application models a multi-protocol heterogeneous data acquisition problem as a sparse signal reconstruction problem, constructs a dictionary matrix by using a sparse Bayesian multi-protocol data reconstruction algorithm, and iteratively solves sparse weight coefficients in a relevant vector machine framework to automatically complete protocol semantic alignment and missing data completion. Then, the standard format data is denoised, filtered and feature-extracted, the inference threshold of a lightweight edge data perception model is adaptively adjusted through a data quality evaluation function, a resource allocation strategy is output by a full-link monitoring module in combination with a game model and a fault-tolerant mechanism is triggered, and the technical problem that protocol semantic automatic alignment and missing data robust completion cannot be realized for multi-protocol heterogeneous industrial Internet of Things data on the edge side is solved.

Description

Technical Field

[0001] This invention belongs to the technical field of industrial IoT data preprocessing methods, media and systems based on microservices. Specifically, it relates to an industrial IoT data preprocessing method, media and system based on microservices. Background Technology

[0002] In the Industrial Internet of Things (IIoT) field, edge gateways need to simultaneously connect to heterogeneous devices using multiple communication protocols such as Modbus, OPC UA, and MQTT, and upload the collected raw data to the cloud or manufacturing execution system after preprocessing. Traditional data preprocessing methods typically rely on manually writing protocol adapters or static mapping rules to convert the raw data from various protocols into a unified format, and then using methods such as mean imputation and linear interpolation to fill in missing data. These methods have certain engineering applicability in scenarios with fixed protocol types and stable acquisition environments, and have been widely used in factory data acquisition and monitoring systems.

[0003] However, in actual industrial settings, there are many types of equipment and frequent protocol version iterations. The engineering cost of manually maintaining protocol mapping rules is extremely high, and static mapping rules cannot adapt to scenarios with dynamic changes in protocol semantics. At the same time, problems such as signal loss and noise pollution are common in industrial settings. When the proportion of missing signals is high or the noise distribution is complex, methods such as mean filling and linear interpolation show a significant decrease in completion accuracy, making it difficult to guarantee the quality consistency of standard format data.

[0004] In current multi-protocol industrial IoT data preprocessing, due to the strong heterogeneity of protocol semantics and the complex distribution of missing data, existing methods based on static rules and simple interpolation cannot automatically infer the semantic attribution of each protocol, nor can they achieve robust missing data completion under noise contamination conditions, resulting in unstable data quality in standard format. In other words, existing technologies suffer from the technical problem of failing to achieve automatic protocol semantic alignment and robust missing data completion for multi-protocol heterogeneous industrial IoT data at the edge. Summary of the Invention

[0005] In view of this, the present invention provides a microservice-based industrial IoT data preprocessing method, medium and system, which can solve the technical problem in the prior art that multi-protocol heterogeneous industrial IoT data cannot achieve automatic alignment of protocol semantics and robust completion of missing data at the edge.

[0006] The present invention is implemented as follows: The first aspect of the present invention provides a microservice-based industrial IoT data preprocessing method, comprising the following steps:

[0007] The upper-layer business microservices register data collection requirements to the service registry. The requirement parser continuously listens to the service registry, translates the data collection requirements into executable device instructions, and completes protocol semantic alignment and missing data completion for heterogeneous original data of multiple protocols through the sparse Bayesian multi-protocol data reconstruction algorithm, and outputs standard format data.

[0008] The preprocessing microservice on the edge gateway performs noise reduction, filtering, feature extraction and format conversion on standard format data, outputs preprocessed data objects, and inputs the preprocessed data objects into the data quality evaluation function to calculate the output quality score. Based on the interval to which the quality score belongs, the inference threshold parameters of the lightweight edge data perception model are adaptively adjusted.

[0009] The lightweight edge data perception model takes preprocessed data objects as input and outputs fault category probability distribution and field mapping confidence matrix. After threshold filtering, the field mapping confidence matrix generates schema evolution transformation rules.

[0010] The context building microservice automatically associates the fault category probability distribution, field mapping confidence matrix with work order information from the manufacturing execution system and equipment information from the asset management system. It enriches the context of multi-source data based on the equipment area subgraph, encapsulates it into semantic data objects, and publishes them to the corresponding topic of the message middleware.

[0011] The preheating instance pool works in conjunction with the predictive elastic scaling mechanism to configure backpressure rate limiting policies on the message middleware side, eliminating message backlog and traffic surges caused by microservice cold starts and ensuring data link continuity;

[0012] The end-to-end monitoring module collects data stream transmission delay, device online status, and message backlog in real time. It inputs the data stream transmission delay and message backlog into a game model, which outputs an edge preprocessing resource allocation strategy and a cloud upload strategy. When the monitoring module detects an anomaly, it triggers a fault tolerance mechanism, automatically switching to a backup link or activating an edge-side cache buffer, thus achieving adaptive self-healing of the data extraction link.

[0013] Specifically, the sparse Bayesian multi-protocol data reconstruction algorithm models the multi-protocol heterogeneous data acquisition problem as a sparse signal reconstruction problem, constructs a dictionary matrix, where the column vectors of the dictionary matrix represent the basis functions of each protocol in the standard feature space, and uses a correlation vector machine framework to iteratively solve for the sparse weight coefficients of each data point with the objective of maximizing the marginal likelihood. The iteration terminates when the sparse weight coefficients of two adjacent rounds are... The norm change is below the convergence threshold .

[0014] Wherein, the convergence threshold Specifically, this was achieved by conducting at least 20 rounds of ablation experiments in a mixed acquisition scenario involving Modbus, OPC UA, and MQTT protocols, and the curve was considered complete when the absolute value of the slope first fell below a certain threshold. The change corresponding to the time is used as the candidate convergence threshold, and the median is taken after repeated experiments on 5 datasets with different noise levels.

[0015] Specifically, the data quality assessment function takes as input data integrity rate. Signal-to-noise ratio and the proportion of outliers Output quality score The calculation formula is: ,in This serves as the baseline value for data integrity. This is the signal-to-noise ratio reference value. This serves as the baseline value for the proportion of outliers. This is the signal-to-noise ratio sensitivity index.

[0016] Specifically, the adaptive adjustment of the inference threshold parameter of the lightweight edge data perception model based on the interval to which the quality score value belongs is when... High-precision inference thresholds are used; when When a medium inference threshold is used, a missing data completion strategy is initiated; when The inference threshold is lowered and the edge-side cache is triggered to wait for resampling.

[0017] Specifically, the lightweight edge data perception model consists of eight depthwise separable one-dimensional convolutional modules stacked sequentially after the input layer, with one skip connection set every two layers. Each skip connection block is connected to a channel attention module. After the eight layers are stacked, a global average pooling layer is connected to output a fixed-length feature vector. The fixed-length feature vector is simultaneously input into two parallel branches: a fault classification head and a field semantic alignment head.

[0018] The fault classification head consists of a fully connected layer and a Softmax activation function, outputting the probability distribution of four fault categories: inner ring fault, outer ring fault, rolling element fault, and normal. The field semantic alignment head adopts a dual-tower structure, with the left and right towers each composed of a lightweight Transformer encoder. Internal field type constraint attention masks are introduced, and the field semantic vectors output by the left and right towers are used to calculate the cross-version field semantic similarity matrix through the InfoNCE contrastive loss function.

[0019] Specifically, the InfoNCE contrastive loss function enables the lightweight edge data awareness model to learn the semantic distance metric between cross-version data contract schema fields by maximizing the cosine similarity of field semantic vectors between positive sample pairs and minimizing the cosine similarity of field semantic vectors between negative sample pairs.

[0020] The game theory model includes an upper-layer model aimed at minimizing end-to-end data transmission latency and a lower-layer model aimed at minimizing the load imbalance at edge nodes. The objective function of the upper-layer model is: The objective function of the lower-level model is The two objective functions are coupled through the term Interrelated.

[0021] Wherein, the coupling coefficient Specifically, this is determined by collecting no fewer than 30 sets of data stream transmission delay and current bandwidth usage data under different network conditions and load levels in the actual deployment environment, and then using multiple linear regression analysis.

[0022] The device region subgraph refers to a local subgraph obtained by cutting the global device asset knowledge graph according to the physical area of ​​the production workshop. Each local subgraph contains only nodes and relationship triples related to the devices in the area managed by the corresponding edge gateway, and is deployed in the lightweight embedded graph database of the corresponding edge gateway.

[0023] The preheating instance pool refers to a group of standby microservice instances that have completed image retrieval, virtual machine preheating, service registration, and health checks in advance. When the predictive elastic scaling mechanism triggers expansion, the instances are directly allocated from the standby microservice instances without having to re-execute the startup process.

[0024] The predictive elastic scaling mechanism refers to predicting the arrival time of peak load based on the message backlog growth rate and consumer processing latency trend collected by the Prometheus monitoring platform, and triggering expansion actions before the peak load arrives.

[0025] The backpressure rate limiting strategy refers to imposing an upper limit constraint on the message retrieval rate of consumers at the message middleware layer. When the consumer's processing capacity has not recovered to the normal level, the message delivery rate is actively limited to avoid backlogged messages being consumed suddenly after the standby microservice instance is ready.

[0026] Specifically, the training dataset for the lightweight edge data perception model consists of collecting the original vibration acceleration time-series signals of no less than 10 industrial rotating devices under four operating conditions. The acquisition time for each operating condition is no less than 48 hours, the sampling frequency is 1kHz, the window length is 2048 sampling points, and the overlap rate between adjacent windows is 50%. The field semantic alignment header collects no less than 50 sets of historical data contract version evolution records, with the ratio of positive sample pairs to negative sample pairs being 1:3.

[0027] A second aspect of the present invention provides a computer-readable storage medium storing program instructions that, when executed in a computer, perform the aforementioned microservice-based industrial IoT data preprocessing method.

[0028] A third aspect of the present invention provides a microservice-based industrial Internet of Things (IoT) data preprocessing system, comprising the aforementioned computer-readable storage medium, wherein the system is a computer, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

[0029] This invention employs a sparse Bayesian multi-protocol data reconstruction algorithm, modeling the problem of acquiring heterogeneous multi-protocol data as a sparse signal reconstruction problem. By constructing a dictionary matrix and a correlation vector machine framework, the algorithm iteratively solves the sparse weight coefficients of each data point with the objective of maximizing marginal likelihood. This enables the algorithm to automatically infer the semantic attribution of each protocol from the observed data without the need for manual specification of protocol mapping rules, thus solving the technical problem that automatic alignment of protocol semantics cannot be achieved for heterogeneous multi-protocol industrial IoT data.

[0030] This invention introduces sparse prior and expectation-maximization algorithms to alternately update the weight prior hyperparameters and posterior mean in a sparse Bayesian framework, enabling the sparse weight coefficients to converge to a stable solution even under noise contamination and signal loss conditions. This achieves robust missing data completion and overcomes the shortcomings of traditional mean filling and linear interpolation methods in terms of accuracy degradation under complex noise environments.

[0031] In summary, this invention solves the technical problem mentioned in the background art that multi-protocol heterogeneous industrial IoT data cannot achieve automatic alignment of protocol semantics and robust completion of missing data at the edge. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention.

[0033] Figure 2 Output a graph showing the change of fault category probability distribution over time for the lightweight edge data perception model during the test period.

[0034] Figure 3 This is a graph showing the dynamic changes in the end-to-end data stream transmission delay as a function of the game model scheduling strategy during the test period. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0036] like Figure 1 The diagram shown is a flowchart of a microservice-based industrial IoT data preprocessing method provided in the first aspect of this invention. This method includes the following steps:

[0037] S01. The upper-layer business microservice registers the data collection requirements to the service registry. The requirement parser continuously listens to the service registry, translates the data collection requirements into executable device instructions, and completes the protocol semantic alignment and missing data completion of the heterogeneous original data of multiple protocols through the sparse Bayesian multi-protocol data reconstruction algorithm, and outputs standard format data.

[0038] S02. The preprocessing microservice on the edge gateway performs noise reduction, filtering, feature extraction and format conversion on the standard format data, outputs a preprocessed data object, and inputs the preprocessed data object into the data quality evaluation function to calculate the output quality score value. Based on the interval to which the quality score value belongs, the inference threshold parameter of the lightweight edge data perception model is adaptively adjusted.

[0039] S03. The lightweight edge data perception model takes the preprocessed data object as input and outputs the fault category probability distribution and field mapping confidence matrix. The field mapping confidence matrix generates Schema evolution transformation rules after threshold filtering.

[0040] S04. The context building microservice automatically associates the fault category probability distribution, the field mapping confidence matrix, and the work order information from the manufacturing execution system and the equipment information from the asset management system. Based on the equipment area subgraph, it completes the context enrichment of multi-source data, encapsulates it into semantic data objects, and publishes them to the corresponding topic of the message middleware.

[0041] S05, the preheating instance pool works in conjunction with the predictive elastic scaling mechanism to configure back pressure rate limiting strategies on the message middleware side, eliminating message backlog and traffic surges caused by microservice cold starts, and ensuring data link continuity;

[0042] S06. The full-link monitoring module collects data stream transmission delay, device online status and message backlog in real time. The data stream transmission delay and message backlog are input into the game model. The game model outputs the edge preprocessing resource allocation strategy and the cloud upload strategy. When the monitoring module detects an anomaly, it triggers the fault tolerance mechanism, automatically switches to the backup link or starts the edge-side cache buffer, realizing the adaptive self-healing of the data extraction link.

[0043] The sparse Bayesian multi-protocol data reconstruction algorithm models the multi-protocol heterogeneous data acquisition problem as a sparse signal reconstruction problem, constructing a dictionary matrix where the column vectors represent the basis functions of each protocol in the standard feature space. Using a relevance vector machine framework, iteratively solves for the sparse weight coefficients of each data point with the objective of maximizing marginal likelihood. In each iteration, the prior hyperparameters and posterior mean of the weights are alternately updated using the expectation-maximization algorithm. Finally, the sparse weight coefficients reflect the degree to which data points belong to each feature basis, automatically completing protocol semantic alignment and missing data completion. The iteration termination condition is the sparse weight coefficients of two adjacent iterations. The norm change is below the convergence threshold The convergence threshold The sparse weight coefficients were obtained by conducting at least 20 rounds of ablation experiments in a mixed acquisition scenario involving Modbus, OPC UA, and MQTT protocols, and the results of each iteration were recorded. The norm change curve is taken when the absolute value of the curve slope first falls below... The change in time was used as a candidate convergence threshold. The above experiment was repeated on 5 datasets with different noise levels, and the median was taken to determine the convergence threshold. The final value. The sparse Bayesian multi-protocol data reconstruction algorithm introduces sparse prior and marginal likelihood maximization mechanisms, enabling the algorithm to automatically infer the semantic attribution of each protocol from the observation data without manually specifying protocol mapping rules when facing scenarios such as mixed protocols, missing signals, and noise pollution in industrial fields. This achieves robust missing data completion and protocol semantic alignment, thereby reducing the engineering cost of multi-protocol integration and improving the quality consistency of standard format data.

[0044] The data quality assessment function is used to comprehensively evaluate the usability of the preprocessed data objects, and its inputs include data integrity rate. Signal-to-noise ratio and the proportion of outliers The output is a dimensionless quality score. The quality score value The calculation formula is expressed as follows:

[0045] ;

[0046] in This serves as the baseline value for data integrity. This is the signal-to-noise ratio reference value. This serves as the baseline value for the proportion of outliers. The signal-to-noise ratio sensitivity index; the data integrity benchmark value. Signal-to-noise ratio reference value Compared with the benchmark value of outlier percentage The signal-to-noise ratio sensitivity index was obtained by collecting data from continuous operation for no less than 72 hours during on-site calibration experiments. The quality of historical data batch labeling results was determined through regression analysis; when At that time, the lightweight edge data perception model uses a high-precision inference threshold to ensure the confidence level of fault classification; when When the lightweight edge data perception model adopts a medium inference threshold and initiates a missing data completion strategy; when At that time, the lightweight edge data perception model lowers the inference threshold and triggers the edge-side cache to wait for resampling.

[0047] The specific structure of the lightweight edge data perception model is as follows: The model input layer receives the original vibration acceleration time-series signal and rotation speed signal from the preprocessed data object. The sampling frequency of the original vibration acceleration time-series signal is 1kHz, and the length of a single inference window is 2048 sampling points. After the input layer, eight depthwise separable one-dimensional convolutional modules are stacked sequentially. Each depthwise separable one-dimensional convolutional module is composed of channel-wise one-dimensional convolution and pointwise one-dimensional convolution connected in series. The channel-wise one-dimensional convolution independently performs one-dimensional temporal convolution on each input channel to extract local temporal features, and the pointwise one-dimensional convolution is performed through... Convolutional kernels perform feature fusion in the channel dimension, and concatenating the two reduces the number of parameters to that of a standard one-dimensional convolution. Every two depthwise separable one-dimensional convolutional modules, a skip connection is set. This skip connection directly adds the input and output of the previous two depthwise separable one-dimensional convolutional modules and passes them to the next depthwise separable one-dimensional convolutional module. This provides a direct gradient path during backpropagation to alleviate gradient vanishing in deep networks. Within each skip connection block, a channel attention module is connected after the second depthwise separable one-dimensional convolutional module. This channel attention module first performs global average pooling on the current feature map to obtain global statistics for each channel. Then, it generates channel attention weight vectors through two fully connected layers and a sigmoid activation function. These channel attention weight vectors are multiplied channel by channel in the feature map to dynamically enhance the response of channels related to the fault frequency band and suppress irrelevant channels. After eight depthwise separable one-dimensional convolutional modules are stacked, a global average pooling layer is connected, outputting a fixed-length feature vector. This fixed-length feature vector is simultaneously input to two parallel branches: the fault classification head and the field semantic alignment head. The fault classification head consists of a fully connected layer and a sigmoid activation function. The model consists of the oftmax activation function, outputting the probability distribution of four fault categories: inner ring fault, outer ring fault, rolling element fault, and normal. The field semantic alignment header employs a dual-tower structure: the left tower receives the name, text, and type description of the source data contract schema fields, while the right tower receives the name, text, and type description of the target data contract schema fields. Each tower is composed of a lightweight Transformer encoder. Internally, the lightweight Transformer encoder introduces a field type constraint attention mask to prevent high attention weights between physically incompatible fields. The field semantic vectors output from the left and right towers are used to calculate a cross-version field semantic similarity matrix using the InfoNCE contrastive loss function. This cross-version field semantic similarity matrix is ​​then filtered by a threshold to automatically generate schema evolution transformation rules, outputting a field mapping confidence matrix. This lightweight edge data awareness model supports single-inference execution on ARM architecture edge processors, with a single-inference latency of less than 10ms. The lightweight edge data perception model decouples the extraction of spatial features from time-series signals and the fusion of channel features through depthwise separable one-dimensional convolution, significantly reducing the number of parameters and computational load, enabling the model to run stably on resource-constrained edge gateways. The channel attention module enables the model to automatically focus on features of fault-related frequency bands during inference, improving the sensitivity to early and subtle faults. The skip connections ensure effective gradient transfer during deep network training. The field semantic alignment header integrates the data contract version evolution problem into the same inference process, enabling fault diagnosis and contract alignment to be completed collaboratively, ensuring both the consistency of data semantics and the accuracy of fault category probability distribution without increasing additional deployment overhead.

[0048] The steps for establishing the training dataset of the lightweight edge data perception model specifically include: collecting vibration acceleration time-series raw signals from no fewer than 10 industrial rotating devices under four operating conditions: inner ring fault, outer ring fault, rolling element fault, and normal operation. The acquisition time for each operating condition is no less than 48 hours, the sampling frequency is 1kHz, the window length is 2048 sampling points, and the overlap rate between adjacent windows is 50%. The corresponding rotational speed signal is recorded synchronously as an auxiliary input. For the field semantic alignment header, no fewer than 50 sets of historical data contract version evolution records are collected. Each set of records includes the name text, field type, field description text, and manually annotated correct mapping relationship of the source data contract schema fields. Positive sample pairs and negative sample pairs are constructed. Negative sample pairs are generated by randomly replacing field names or types. The ratio of positive sample pairs to negative sample pairs is 1:3. Gaussian white noise with a signal-to-noise ratio of 5dB to 30dB is applied to all vibration acceleration time-series raw signal samples for data augmentation.

[0049] The specific steps for training the lightweight edge data perception model include: dividing all samples into training, validation, and test sets in an 8:1:1 ratio; using the cross-entropy loss function for the fault classification head and the InfoNCE contrastive loss function for the field semantic alignment head, jointly optimizing the two loss functions with a weighted summation, and determining the weight coefficients through grid search on the validation set; employing an adaptive moment estimation optimization algorithm for the optimizer, and determining the initial learning rate through learning rate range testing on the training set; using the average accuracy of the validation set fault classification and the average precision of the field mapping as the early stopping criterion during training, terminating training when the validation criterion does not improve after 10 consecutive rounds; performing INT8 quantization on the model weights after training to compress the model size, and verifying on the test set that the accuracy loss does not exceed the quantization accuracy loss tolerance value, which is determined by taking the acceptable upper limit after statistically analyzing the accuracy loss of 5 groups of experiments with different quantization bit widths.

[0050] The InfoNCE contrastive loss function is used to train the field semantic alignment head. By maximizing the cosine similarity of the field semantic vectors between positive sample pairs and minimizing the cosine similarity of the field semantic vectors between negative sample pairs, the lightweight edge data awareness model learns the semantic distance metric between cross-version data contract schema fields, thereby accurately generating the field mapping confidence matrix during the inference phase.

[0051] The game theory model includes an upper-layer model that aims to minimize end-to-end data transmission latency and a lower-layer model that aims to minimize the load imbalance at edge nodes. The objective function of the upper-layer model is expressed as follows:

[0052] ;

[0053] in The data stream transmission delay collected by the end-to-end monitoring module, As the reference value for transmission delay, Current bandwidth usage This represents the maximum available bandwidth. The coupling coefficient is ; the constraint condition of the upper-level model is . and The objective function of the lower-level model is expressed as follows:

[0054] ;

[0055] in For the first The current processing load of each edge node. Process load baseline values ​​for edge nodes. This represents the total number of edge nodes. This is a coupling term shared by the objective function of the upper-layer model and the objective function of the lower-layer model, reflecting the impact of the currently occupied bandwidth on the processing load balancing among edge nodes; the upper-layer model function is used to measure the timeliness and bandwidth efficiency of data upload, and its inputs include data stream transmission latency. With current bandwidth usage The output is a dimensionless transmission efficiency loss value. The lower-level model function is used to measure the deviation of the current processing load of each edge node from the currently occupied bandwidth. The input includes the current processing load of each edge node. With current bandwidth usage The output is a dimensionless load imbalance degree. The coupling coefficient The transmission delay baseline value was determined by collecting no fewer than 30 sets of data stream transmission delay and current bandwidth usage data under different network conditions and load levels in a real deployment environment, and then using multiple linear regression analysis. Available bandwidth limit Edge node processing load baseline The statistical mean value under stable operating conditions is determined by collecting data through on-site no-load calibration experiments; the output edge preprocessing resource allocation strategy and cloud upload strategy of the game model are used by the full-link monitoring module to trigger the execution judgment of the fault tolerance mechanism.

[0056] The service registry is a centralized directory service component used to receive and store data collection requirements declared by upper-layer business microservices. These data collection requirements include the required data type, sampling frequency, accuracy requirements, and push method, which are continuously subscribed to and read by the requirement parser.

[0057] The demand parser is a continuously running service module used to convert data acquisition requirements stored in the service registry into executable device instructions that can be directly executed by the target device. The executable device instructions include protocol type, register address, sampling period, and data encoding format.

[0058] The preheating instance pool refers to a group of standby microservice instances that have completed image retrieval, virtual machine preheating, service registration, and health checks in advance. When the predictive elastic scaling mechanism triggers expansion, it directly allocates from the standby microservice instances without re-executing the startup process, thereby eliminating the impact of new instance readiness delay on data link continuity.

[0059] The predictive elastic scaling mechanism refers to predicting the arrival time of peak load based on the message backlog growth rate and consumer processing latency trend collected by the Prometheus monitoring platform, and triggering expansion before the peak load arrives, so that the standby microservice instances in the preheating instance pool are ready before the traffic surge.

[0060] The backpressure rate limiting strategy refers to imposing an upper limit constraint on the message retrieval rate of consumers at the message middleware layer. When the consumer's processing capacity has not recovered to the normal level, the message delivery rate is actively limited to avoid the backlog of messages being consumed suddenly after the standby microservice instance is ready, which would cause a sudden surge in the write pressure on the downstream database.

[0061] The device region subgraph is a local subgraph obtained by cutting the global device asset knowledge graph according to the physical area of ​​the production workshop. Each local subgraph contains only nodes and relation triples related to the devices in the area managed by the corresponding edge gateway. It is deployed in the lightweight embedded graph database of the corresponding edge gateway and is used to complete context association reasoning on the edge gateway side.

[0062] The Manufacturing Execution System (MES) is a factory-level information system used to manage and track the production process, providing work order information, including work order number, production batch, and process parameters. The Asset Management System (AMS) provides equipment information, including equipment model and equipment number. The message middleware is a middleware component used to implement asynchronous message passing between microservices, adopting a publish-subscribe pattern. The producer of the semantic data object publishes the semantic data object to the corresponding topic of the message middleware, and the data consumer obtains the semantic data object by subscribing to the corresponding topic, realizing loosely coupled communication between the producer and the data consumer.

[0063] The specific implementation of step S01 is as follows: When the upper-layer business microservice starts, it writes its required data collection needs, such as data type, sampling frequency, accuracy requirements, and push method, into the service registry. The service registry, as a centralized directory service component, persistently stores these needs. The requirement parser continuously subscribes to the service registry in a long polling manner. Once a newly registered or updated data collection need is detected, it parses and translates it into executable device instructions that the target device can directly execute. The executable device instructions include elements such as protocol type, register address, sampling period, and data encoding format, and are then sent to the corresponding edge gateway driver device for collection. For the collected multi-protocol heterogeneous raw data, the sparse Bayesian multi-protocol data reconstruction algorithm first constructs a dictionary matrix. Each column vector of the dictionary matrix corresponds to the basis function representation of a certain protocol in the standard feature space. Then, under the framework of the relevance vector machine, with the optimization objective of maximizing the marginal likelihood, the sparse weight prior hyperparameters and posterior means of each data point are alternately updated in each iteration through the expectation maximization algorithm until the sparse weight coefficients of two adjacent rounds are... The norm change is below the convergence threshold Until then, the convergence threshold The reference value was determined through no fewer than 20 rounds of ablation experiments in a mixed acquisition scenario using Modbus, OPC UA, and MQTT protocols. The magnitude of the data points is directly reflected in the degree to which each data point belongs to each protocol feature base, thus automatically completing the protocol semantic alignment. For missing data points, the algorithm uses the learned sparse weight coefficients to reconstruct and complete the data in the standard feature space, outputting standard format data.

[0064] The specific implementation of step S02 is as follows: The preprocessing microservice is deployed on the edge gateway, and performs noise reduction, digital filtering, time-frequency domain feature extraction, and format conversion sequentially on the standard format data output in step S01, transforming the original time-series data into a structured preprocessed data object. The noise reduction stage uses a combination of median filtering and wavelet thresholding. The filtering stage selects a finite impulse response bandpass filter based on the signal's frequency band characteristics, with a cutoff frequency reference value of 10Hz to 500Hz. The feature extraction stage extracts time-frequency domain statistical features such as root mean square value, kurtosis, and spectral peak value. After the preprocessed data object is generated, it is input into a data quality evaluation function, which receives the data integrity rate. Signal-to-noise ratio and the proportion of outliers Three indicators, according to the formula Calculate and output dimensionless quality score. The benchmark value , , The signal-to-noise ratio sensitivity index was obtained through statistical analysis of on-site calibration experiments lasting no less than 72 hours. Based on historical data regression analysis, the reference value is determined to be between 0.5 and 1.5. The interval to which the value belongs, when The lightweight edge data perception model employs a high-precision inference threshold; when When a medium inference threshold is used, a missing data completion strategy is initiated; when The inference threshold is lowered and the edge-side cache is triggered to wait for resampling, thereby achieving adaptive adjustment of the inference threshold parameter.

[0065] The specific implementation of step S03 is as follows: The lightweight edge data perception model receives the original vibration acceleration time series signal and rotation speed signal from the preprocessed data object. The sampling frequency of the original vibration acceleration time series signal is 1kHz, and the length of a single inference window is 2048 sampling points. The model backbone consists of 8 layers of depthwise separable one-dimensional convolutional modules stacked sequentially. Each layer of depthwise separable one-dimensional convolutional module is composed of channel-wise one-dimensional convolution and point-wise one-dimensional convolution in series, and the number of parameters is compressed to that of a standard one-dimensional convolution. A skip connection is set every two layers, where the input and output of the two-layer module are directly added before being fed into the next module, mitigating the gradient vanishing problem in deep networks. Within each skip connection block, after the second depthwise separable one-dimensional convolutional module, a channel attention module is connected. This module performs global average pooling on the feature map, followed by two fully connected layers and a sigmoid activation function to generate channel attention weight vectors, which are then multiplied channel by channel to enhance the response of fault-related channels. After eight layers are stacked, a global average pooling layer is applied, outputting a fixed-length feature vector, which is then fed into the fault classification head and the field semantic alignment head. The fault classification header outputs the probability distribution of four fault categories: inner ring fault, outer ring fault, rolling element fault, and normal. The field semantic alignment header adopts a dual-tower structure, with the left and right towers each composed of a lightweight Transformer encoder. Internally, a field type constraint attention mask is introduced to prevent physically incompatible fields from generating high attention weights. The field semantic vectors output by the dual towers are used to calculate the cross-version field semantic similarity matrix using the InfoNCE contrastive loss function. After threshold filtering, a schema evolution transformation rule is generated and a field mapping confidence matrix is ​​output. The latency of a single inference is less than 10ms.

[0066] The specific implementation of step S04 is as follows: The context construction microservice obtains the fault category probability distribution and field mapping confidence matrix output in step S03 from the message middleware. Simultaneously, it retrieves current work order information (including work order number, production batch, and process parameters) from the manufacturing execution system and equipment information (including equipment model and equipment number) from the asset management system via interface calls. Using the equipment region subgraph in the lightweight embedded graph database deployed on the edge gateway, the above multi-source data is used to complete context association reasoning through graph node matching, thereby enriching the context of the multi-source data. The equipment region subgraph is derived from the global equipment asset knowledge graph by cutting it according to the physical regions of the production workshop. Each local subgraph contains only nodes and relation triples related to the equipment in its own region, reducing the computational overhead of edge graph queries. After context enrichment, all associated information is encapsulated into semantic data objects and published to the corresponding topic in the message middleware using a publish-subscribe model for downstream data consumers to subscribe to, achieving loosely coupled communication between producers and data consumers.

[0067] The specific implementation of step S05 is as follows: A preheating instance pool maintains a group of standby microservice instances that have completed image retrieval, virtual machine preheating, service registration, and health checks. When the predictive elastic scaling mechanism predicts an impending load peak based on the message backlog growth rate and consumer processing latency trend collected by the Prometheus monitoring platform, it triggers expansion in advance, directly allocating standby microservice instances from the preheating instance pool to provide service, eliminating new instance readiness delays. Simultaneously, a backpressure rate limiting strategy is configured on the message middleware side to impose an upper limit constraint on the consumer's message retrieval rate. When the consumer's processing capacity has not yet recovered to normal levels, the message delivery rate is proactively limited to prevent backlogged messages from being suddenly consumed after the standby microservice instances are ready, thus preventing a sudden surge in downstream database write pressure and eliminating traffic peaks, ensuring data link continuity.

[0068] The specific implementation of step S06 is as follows: The end-to-end monitoring module collects data stream transmission delay, device online status, and message backlog at millisecond intervals, and records the data stream transmission delay. A game theory model with input message backlog. The game theory model consists of an upper-level model and a lower-level model. The upper-level model aims to minimize... The objective is to minimize both transmission delay and bandwidth usage, with the constraint that both must not exceed baseline values. The lower-level model aims to minimize... With the goal of [the goal], the two are coupled through [the term]. Interrelated, coupling coefficient The reference value is determined through multiple linear regression analysis of no less than 30 sets of field data, and is approximately between 0.3 and 0.8. After solving the game theory model, the edge preprocessing resource allocation strategy and the cloud upload strategy are output. The end-to-end monitoring module executes resource scheduling according to these strategies. When the monitoring module detects a link anomaly, it triggers a fault tolerance mechanism, automatically switching to a backup link or activating the edge-side cache buffer, realizing the adaptive self-healing of the data extraction link.

[0069] It should be noted that the key technologies of this invention include: a sparse Bayesian multi-protocol data reconstruction algorithm that combines the inherent sparsity of heterogeneous industrial multi-protocol data with a Bayesian inference mechanism. Through the collaborative iteration of a related vector machine framework and an expectation-maximization algorithm, the system can automatically infer the semantic attribution of protocols and complete missing data without manual intervention, fundamentally eliminating the dependence of static mapping rules on engineering maintenance; a lightweight edge data perception model that decouples temporal feature extraction and channel fusion through depthwise separable one-dimensional convolution, and combines a channel attention module, skip connections, and a dual-tower field semantic alignment head to unify fault diagnosis and data contract version evolution alignment into the same inference process, achieving an inference latency of less than 10ms on an ARM architecture edge processor; and a data quality evaluation function and a game theory model that work together to form a closed-loop adaptive control system. The former dynamically adjusts the inference threshold according to data quality, while the latter coordinates edge resource allocation and cloud upload strategies from a full-link perspective. The synergistic effect of the three key technologies enables the system to simultaneously ensure data semantic consistency, fault diagnosis accuracy, and link transmission continuity under complex industrial field conditions such as protocol heterogeneity, data loss, and load fluctuations. Compared with the traditional preprocessing scheme of static rules plus simple interpolation, it achieves full-process self-adaptation and self-healing capabilities without the need for manual annotation of protocol mapping.

[0070] It should be noted that this invention also addresses the following technical problem: In industrial IoT edge computing scenarios, as the data contract schema version continuously evolves, the field mapping relationship between the source data contract and the target data contract changes frequently. Traditional methods rely on manual maintenance of field mapping rules, which is not only costly in terms of engineering but also prone to mapping omissions or semantic mismatches when the version evolution is rapid, resulting in downstream data consumers receiving data with inconsistent field meanings. This invention introduces a field semantic alignment header into a lightweight edge data perception model, uses the InfoNCE contrastive loss function to drive a dual-tower lightweight Transformer encoder to learn the semantic distance metric between cross-version fields, and uses a field type-constrained attention mask to prevent high attention weights between physically incompatible fields. This enables the model to automatically generate a field mapping confidence matrix and schema evolution transformation rules during the inference phase, completing the field semantic alignment of cross-version data contracts without manual intervention, thus solving the technical problem of difficulty in automatically maintaining field mapping relationships due to data contract version evolution.

[0071] A second aspect of the present invention provides a computer-readable storage medium storing program instructions that, when executed in a computer, perform the aforementioned microservice-based industrial IoT data preprocessing method.

[0072] A third aspect of the present invention provides a microservice-based industrial Internet of Things (IoT) data preprocessing system, comprising the aforementioned computer-readable storage medium. The system is any one of a computer, a server, or a microcontroller. The computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor that executes program instructions stored in the computer-readable storage medium.

[0073] Specifically, the principle of this invention is:

[0074] This invention solves the aforementioned technical problems because the mathematical structure of the sparse Bayesian multi-protocol data reconstruction algorithm naturally aligns with the inherent sparsity of industrial multi-protocol data. The projection of the original data of each protocol in the industrial field onto the standard feature space exhibits sparsity, meaning each data point has a significant affiliation with only a few protocol basis functions. This physical prior highly matches the sparse prior assumption in the sparse Bayesian framework, allowing the algorithm to automatically learn the affiliation degree of each data point to each feature basis through marginal likelihood maximization without manual protocol attribution. The correlation vector machine framework, through Bayesian inference, automatically controls model complexity while solving for sparse weight coefficients, avoiding overfitting from interfering with completion accuracy. The alternating iterative mechanism of the expectation-maximization algorithm ensures that the weight prior hyperparameters and posterior mean converge collaboratively under conditions of signal loss and noise contamination, thereby enabling the sparse weight coefficients to accurately reflect protocol semantic attribution and complete missing data completion. A convergence threshold is used as the benchmark. As an iteration termination condition, it ensures a balance between accuracy and efficiency in the algorithm, making the entire preprocessing process feasible even under resource-constrained conditions at the edge gateway.

[0075] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0076] The specific implementation of step S01 is as follows: The upper-layer business microservice registers its data collection requirements with the service registry center. These requirements include the required data type, sampling frequency, accuracy requirements, and push method. The requirement parser continuously subscribes to the service registry center, converting the collection requirements into executable device instructions containing protocol type, register address, sampling period, and data encoding format, and then sends them to the field devices. For multi-protocol heterogeneous raw data generated in a mixed collection scenario using Modbus, OPC UA, and MQTT protocols, a sparse Bayesian multi-protocol data reconstruction algorithm is used to complete protocol semantic alignment and missing data completion. This algorithm models the multi-protocol heterogeneous data collection problem as a sparse signal reconstruction problem, where the observed data vector is... Construct a dictionary matrix ,in The total number of observed data points. This represents the total number of protocol basis functions. The column vector For the first The basis function representation of the protocol in the standard feature space, Indices for protocol basis functions. Sparse weight coefficient vector. satisfy:

[0077] ;

[0078] In the formula, To observe the noise vector, each component follows a pattern with a mean of 0 and a variance of . Gaussian distribution, Let the noise variance be denoted. Using a correlation vector machine framework, for each weight component... Introducing independent Gaussian priors ,in This indicates that the mean is 0 and the variance is 0. Gaussian distribution, For the first Each weight component The corresponding prior accuracy hyperparameter is used to maximize the marginal likelihood. Solve for the sparse weight coefficients for the objective, where Let be a vector composed of all prior precision hyperparameters. Posterior distribution. posterior mean With the posterior covariance matrix The calculation formula is expressed as follows:

[0079] ;

[0080] ;

[0081] In the formula, This is a diagonal matrix formed by using each prior precision hyperparameter as its diagonal element. This represents a diagonal matrix formed by using the elements within the parentheses as diagonal elements. dictionary matrix The transpose of . In each iteration, the hyperparameters are updated alternately using the expectation-maximization algorithm. With posterior mean components , The update formula is expressed as follows:

[0082] ;

[0083] In the formula, This is the [number]th iteration after this round. Updated values ​​of the prior accuracy hyperparameters. For the posterior covariance matrix The The diagonal element, i.e. the th element Each weight component The posterior variance, posterior mean vector The Each component, molecule For a dimensionless quantity, the denominator is... and Since they have the same dimensions, and The units are consistent. The iteration termination condition is the sparse weight coefficients of two adjacent rounds. The norm change is below the convergence threshold ,Right now ,in For the first The sparse weight coefficient vector at the end of the round of iterations. This is the current iteration round number. Representing vectors Norm. Convergence threshold. The sparse weight coefficients were obtained by conducting at least 20 rounds of ablation experiments in a mixed acquisition scenario using Modbus, OPC UA, and MQTT protocols, and the results of each iteration were recorded. The norm change curve is taken when the absolute value of the curve slope first falls below... The change corresponding to the time interval is used as the candidate convergence threshold. After repeating the above experiment on 5 datasets with different noise levels, the median is taken to determine the convergence threshold. The final value. Final sparse weight coefficients. The absolute values ​​of each component reflect the degree to which the data point belongs to each protocol feature base. It automatically completes protocol semantic alignment and missing data completion, and outputs standard format data.

[0084] The specific implementation of step S02 is as follows: The preprocessing microservice on the edge gateway sequentially performs noise reduction, filtering, feature extraction, and format conversion on the standard format data, outputs a preprocessed data object, and then inputs it into a data quality evaluation function. This function comprehensively evaluates the usability of the preprocessed data object, and the inputs include data integrity rate. Signal-to-noise ratio and the proportion of outliers Output dimensionless quality score. The calculation formula is expressed as follows:

[0085] ;

[0086] In the formula, This serves as the baseline value for data integrity. This is the signal-to-noise ratio reference value. This serves as the baseline value for the proportion of outliers. This is the signal-to-noise ratio sensitivity index, used to adjust the impact of the signal-to-noise ratio on the quality score. Influence weight, , and All three are dimensionless ratios. This is a dimensionless quality score. , and The data was obtained by collecting data from continuous operation for no less than 72 hours through on-site calibration experiments. The quality was determined through regression analysis of historical batch data labeling results. The inference threshold parameter of the lightweight edge data perception model is adaptively adjusted according to the value interval: when When high-precision inference thresholds are used to ensure the confidence level of fault classification; when When a medium inference threshold is applied, a missing data completion strategy is initiated; when When this happens, the inference threshold is lowered and the edge-side cache is triggered to wait for resampling.

[0087] The specific implementation of step S03 is as follows: The lightweight edge data perception model takes preprocessed data objects as input. The model input layer receives the original vibration acceleration time series signal and rotation speed signal. The sampling frequency of the original vibration acceleration time series signal is 1kHz, and the length of a single inference window is 2048 sampling points. After the input layer, eight depth-separable one-dimensional convolutional modules are stacked sequentially. Each layer consists of a cascaded one-dimensional convolution and a point-by-point one-dimensional convolution, compressing the number of parameters to that of a standard one-dimensional convolution. A skip connection is set every two layers. The skip connection directly adds the input and output of the two layers before passing it to the next module. Within each skip connection block, after the second layer, a channel attention module is connected. Global average pooling is performed on the current feature map, followed by two fully connected layers and a sigmoid activation function to generate channel attention weight vectors, which are then multiplied channel-by-channel with the feature map. After eight layers are stacked, a global average pooling layer is connected, outputting a fixed-length feature vector, which is simultaneously fed into two parallel branches: the fault classification head and the field semantic alignment head. The fault classification head consists of fully connected layers and a Softmax activation function, outputting the probability distribution of four fault categories: inner ring fault, outer ring fault, rolling body fault, and normal. The field semantic alignment head uses a dual-tower structure. The left tower receives source data contract schema field information, and the right tower receives target data contract schema field information. Each is composed of a lightweight Transformer encoder and incorporates a field type constraint attention mask. The left tower outputs the source field semantic vector. The right tower outputs the semantic vector of the target field. ,in For semantic vector dimension, For sample pair indexing, This represents the number of positive sample pairs in a single batch. The field semantic vectors output by the left and right towers are used to calculate cross-version field semantic similarity using the InfoNCE contrastive loss function. The calculation formula is expressed as follows:

[0088] ;

[0089] In the formula, For the first semantic vector of each source field With the semantic vector of each target field Cosine similarity between them For vectors transpose, and They are respectively and of Norm, This is a temperature coefficient, a dimensionless positive real number with an empirical value of 0.07, used to control the concentration of the similarity distribution. , and All are dimensionless quantities. This loss function maximizes the positive sample pairs through the numerator. The model learns semantic distance metrics between schema fields across versions of a data contract by using cosine similarity and minimizing the cosine similarity of negative sample pairs within the same batch. The cross-version field semantic similarity matrix is ​​then thresholded to automatically generate schema evolution transformation rules, outputting a field mapping confidence matrix. The model supports single-inference operations on ARM architecture edge processors with a latency of less than 10ms.

[0090] The specific implementation of step S04 is as follows: The context construction microservice automatically associates the fault category probability distribution, field mapping confidence matrix with work order information (including work order number, production batch and process parameters) from the manufacturing execution system and equipment information (including equipment model and equipment number) from the asset management system. Based on the equipment area subgraph deployed in the lightweight embedded graph database of the edge gateway, the context of the multi-source data is enriched, encapsulated into semantic data objects, and published to the corresponding topic of the message middleware in a publish-subscribe mode for data consumers to subscribe to and obtain.

[0091] The specific implementation of step S05 is as follows: A preheating instance pool maintains a group of standby microservice instances that have completed image retrieval, virtual machine preheating, service registration, and health checks. The predictive elastic scaling mechanism, based on the message backlog growth rate and consumer processing latency trend collected by the Prometheus monitoring platform, predicts the arrival time of peak loads and triggers expansion actions before the peak, ensuring that standby instances are ready before the traffic surge arrives, thereby eliminating message backlog caused by microservice cold starts. Simultaneously, a backpressure rate limiting strategy is configured on the message middleware side to impose an upper limit constraint on the consumer's message retrieval rate, preventing backlogged messages from being suddenly consumed after the instances are ready, thus avoiding a sudden surge in downstream write pressure and ensuring data link continuity.

[0092] The specific implementation of step S06 is as follows: The end-to-end monitoring module collects data stream transmission delay, device online status, and message backlog in real time, and inputs the data stream transmission delay and message backlog into the game theory model. The game theory model includes an upper-layer model that aims to minimize the end-to-end data transmission delay and a lower-layer model that aims to minimize the load imbalance of edge nodes. The objective function of the upper-layer model is expressed as follows:

[0093] ;

[0094] In the formula, The data stream transmission delay collected by the end-to-end monitoring module As the reference value for transmission delay, Current bandwidth usage This represents the maximum available bandwidth. Let be the coupling coefficient, and be a dimensionless positive real number. and All are dimensionless ratios. This represents the dimensionless transmission efficiency loss value. The constraints of the upper-level model are: and The objective function of the lower-level model is expressed as follows:

[0095] ;

[0096] In the formula, For the first The current processing load of each edge node. Process load baseline values ​​for edge nodes. This represents the total number of edge nodes. For edge node indexing, and All are dimensionless ratios. For dimensionless load imbalance, This is a coupling term shared by the upper and lower layer models, reflecting the impact of the currently occupied bandwidth on the load balancing of processing among edge nodes. Coupling coefficient By collecting no fewer than 30 sets of data under different network conditions and load levels in a real deployment environment and The data were determined using multiple linear regression analysis. , and The statistical mean of data collected under stable operating conditions through on-site no-load calibration experiments is determined. The game theory model outputs edge preprocessing resource allocation strategy and cloud upload strategy. When the monitoring module detects an anomaly, a fault tolerance mechanism is triggered, automatically switching to the backup link or activating the edge-side cache buffer, realizing the adaptive self-healing of the data extraction link.

[0097] To better understand and implement this invention, the following is a specific application scenario of the invention, Example 2: To verify the effectiveness of the invention, technicians built a test environment and deployed a microservice-based industrial IoT data preprocessing system on a heavy equipment manufacturing production line to collect and preprocess data from rotating equipment on the production line that simultaneously uses Modbus, OPC UA, and MQTT protocols, in order to verify the actual effect of the invention in a multi-protocol heterogeneous scenario.

[0098] The test environment connected 12 industrial rotating devices, covering three types of equipment: spindle motors, gearboxes, and conveyor rollers. The edge gateway used an ARM architecture processor, the operating system was Linux, the message middleware was Kafka, and the service registry was Consul. Upper-layer business microservices registered their data collection requests for vibration acceleration, rotational speed, and temperature to the service registry. The request parser then translated these requests into executable device commands corresponding to the respective device protocols and sent them to the corresponding gateways to drive the devices to collect data. During the collection process, the raw data from the three protocols was synchronously fed into the edge gateway. Approximately 18% of data points were artificially simulated to verify the missing data completion capability.

[0099] The sparse Bayesian multi-protocol data reconstruction algorithm completes the dictionary matrix construction in the above multi-protocol hybrid scenario, and iteratively solves the sparse weight coefficients and convergence threshold through a related vector machine framework. Using the method determined in advance through ablation experiments The algorithm converges after an average of approximately 27 iterations, outputting data in a standard format. A data quality assessment function is then called on the output standard-format data to test the data integrity of the dataset. After completion, the average signal-to-noise ratio reached 0.97. The mean is 22dB, and the percentage of outliers is... The mean is 3.2%, and the benchmark is... , , Signal-to-noise ratio sensitivity index The average quality score was calculated. The value is approximately 0.93, which falls within the high-quality range. Therefore, the lightweight edge data perception model automatically adopts a high-precision inference threshold. The distribution and quality score results of the data collected during the test for each operating condition are shown in Table 1.

[0100] Table 1. Statistical Table of Data Quality Scores for Each Operating Condition

[0101]

[0102] The lightweight edge data perception model uses 2048 sampling points as a single inference window to infer the original vibration acceleration time-series signal. Feature extraction is achieved through an 8-layer depthwise separable one-dimensional convolutional module, a channel attention module, and skip connections. The fault classification head outputs the probability distribution of four fault categories, and the field semantic alignment head automatically generates the field mapping confidence matrix and schema evolution transformation rules. During testing, three data contract schema version evolution scenarios were triggered. The field semantic alignment head automatically completed cross-version field mapping without manual intervention, avoiding data parsing errors caused by inconsistent field meanings for downstream data consumers. On the ARM architecture edge processor, the single inference latency was less than 10ms, meeting real-time processing requirements. Figure 2 As shown, the trend of the probability distribution of fault categories over time clearly reflects the evolution of the equipment status from normal to early outer ring faults, verifying the sensitivity of the channel attention module to early weak faults.

[0103] The context-based microservice automatically associates the fault category probability distribution with the field mapping confidence matrix and the work order information from the manufacturing execution system and the equipment information from the asset management system. It uses a device region subgraph to complete graph node matching and context enrichment, encapsulates the information into semantic data objects, and publishes them to the corresponding Kafka topic. During testing, the production line experienced two short-term traffic surges, with the message backlog reaching a peak of 4.3 times the normal level. The predictive elastic scaling mechanism, based on the message backlog growth rate collected by Prometheus, predicted the load peak approximately 15 seconds in advance and triggered expansion. Standby microservice instances in the preheating instance pool were directly put into service, and the backpressure rate limiting strategy synchronously restricted the consumer pull rate. The data link remained uninterrupted during both traffic surges.

[0104] The end-to-end monitoring module detected two link anomalies during the test, caused by network jitter and device disconnection respectively. The game theory model outputs an adjusted edge preprocessing resource allocation strategy based on the real-time data stream transmission delay and message backlog, with a coupling coefficient... Using a value of 0.52 determined through multiple linear regression of 30 sets of field data, the fault tolerance mechanism automatically switched to the backup link and activated the edge-side buffer. Both anomalies resolved themselves within 30 seconds, resulting in no data loss. Figure 3 As shown in the figure, the curves illustrating the dynamic changes in end-to-end data stream transmission latency with the game model's scheduling strategy during the test period demonstrate that the game model can continuously control the transmission latency near the baseline value under load fluctuation conditions, verifying the end-to-end adaptive and self-healing capabilities. A summary of the link anomaly handling results at each stage is shown in Table 2.

[0105] Table 2 Statistical Table of Link Anomaly Handling Results

[0106]

[0107] Compared to traditional preprocessing schemes based on static protocol mapping rules and simple interpolation, this invention incorporates protocol semantic attribution inference into a probabilistic reasoning framework through a sparse Bayesian multi-protocol data reconstruction algorithm. This enables the system to automatically adapt to protocol version changes that have not been explicitly configured, through iterative solutions of the dictionary matrix and sparse weight coefficients, thus eliminating the bottleneck of manual rule maintenance. The lightweight edge data perception model aligns fault diagnosis and data contract evolution into the same reasoning process, avoiding the semantic gap between two independent systems. The closed-loop collaboration between the game theory model and the full-link monitoring module enables resource scheduling to be forward-looking and self-healing, transforming it from passively responding to anomalies to actively predicting and adjusting, thus ensuring the robustness and continuity of the data preprocessing link under complex industrial field conditions from a fundamental perspective.

[0108] It should be noted that the variables involved in this invention are explained in detail in Tables 3 and 4.

[0109] Table 3. Variable Explanation Table (Part 1)

[0110]

[0111] Table 4. Variable Explanation Table (Part Two)

[0112]

[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A microservice-based industrial IoT data preprocessing method, characterized in that, Includes the following steps: The upper-layer business microservices register data collection requirements to the service registry. The requirement parser continuously listens to the service registry, translates the data collection requirements into executable device instructions, and completes protocol semantic alignment and missing data completion for heterogeneous original data of multiple protocols through the sparse Bayesian multi-protocol data reconstruction algorithm, and outputs standard format data. The preprocessing microservice on the edge gateway performs noise reduction, filtering, feature extraction and format conversion on standard format data, outputs preprocessed data objects, and inputs the preprocessed data objects into the data quality evaluation function to calculate the output quality score. Based on the interval to which the quality score belongs, the inference threshold parameters of the lightweight edge data perception model are adaptively adjusted. The lightweight edge data perception model takes preprocessed data objects as input and outputs fault category probability distribution and field mapping confidence matrix. After threshold filtering, the field mapping confidence matrix generates schema evolution transformation rules. The context building microservice automatically associates the fault category probability distribution, field mapping confidence matrix with work order information from the manufacturing execution system and equipment information from the asset management system. It enriches the context of multi-source data based on the equipment area subgraph, encapsulates it into semantic data objects, and publishes them to the corresponding topic of the message middleware. The preheating instance pool works in conjunction with the predictive elastic scaling mechanism to configure backpressure rate limiting policies on the message middleware side, eliminating message backlog and traffic surges caused by microservice cold starts and ensuring data link continuity; The end-to-end monitoring module collects data stream transmission delay, device online status, and message backlog in real time. It inputs the data stream transmission delay and message backlog into a game model, which outputs an edge preprocessing resource allocation strategy and a cloud upload strategy. When the monitoring module detects an anomaly, it triggers a fault tolerance mechanism, automatically switching to a backup link or activating an edge-side cache buffer, thus achieving adaptive self-healing of the data extraction link.

2. The industrial IoT data preprocessing method based on microservices according to claim 1, characterized in that, The sparse Bayesian multi-protocol data reconstruction algorithm specifically models the multi-protocol heterogeneous data acquisition problem as a sparse signal reconstruction problem. It constructs a dictionary matrix, where the column vectors represent the basis functions of each protocol in the standard feature space. Using a correlation vector machine framework, iteratively solving for the sparse weight coefficients of each data point with the objective of maximizing the marginal likelihood, the iteration terminates when the sparse weight coefficients of two adjacent rounds are reached. The norm change is below the convergence threshold .

3. The industrial IoT data preprocessing method based on microservices according to claim 2, characterized in that, The convergence threshold Specifically, this was achieved by conducting at least 20 rounds of ablation experiments in a mixed acquisition scenario involving Modbus, OPC UA, and MQTT protocols, and the curve was considered complete when the absolute value of the slope first fell below a certain threshold. The change corresponding to the time is used as the candidate convergence threshold, and the median is taken after repeated experiments on 5 datasets with different noise levels.

4. The industrial IoT data preprocessing method based on microservices according to claim 3, characterized in that, The device region subgraph refers to a local subgraph obtained by cutting the global device asset knowledge graph according to the physical area of ​​the production workshop. Each local subgraph contains only nodes and relationship triples related to the devices in the area managed by the corresponding edge gateway, and is deployed in the lightweight embedded graph database of the corresponding edge gateway.

5. The industrial IoT data preprocessing method based on microservices according to claim 4, characterized in that, The preheating instance pool refers to a group of standby microservice instances that have completed image retrieval, virtual machine preheating, service registration, and health checks in advance. When the predictive elastic scaling mechanism triggers expansion, they are directly allocated from the standby microservice instances without having to re-execute the startup process.

6. The industrial IoT data preprocessing method based on microservices according to claim 5, characterized in that, The lightweight edge data perception model consists of eight depthwise separable one-dimensional convolutional modules stacked sequentially after the input layer. A skip connection is set every two layers, and a channel attention module is connected inside each skip connection block. After the eight layers are stacked, a global average pooling layer is connected to output a fixed-length feature vector. The fixed-length feature vector is simultaneously input into two parallel branches: a fault classification head and a field semantic alignment head.

7. The industrial IoT data preprocessing method based on microservices according to claim 6, characterized in that, The fault classification head consists of a fully connected layer and a Softmax activation function, outputting the probability distribution of four fault categories: inner ring fault, outer ring fault, rolling element fault, and normal. The field semantic alignment head adopts a dual-tower structure, with the left and right towers each composed of a lightweight Transformer encoder. Internal field type constraint attention masks are introduced, and the field semantic vectors output by the left and right towers are used to calculate the cross-version field semantic similarity matrix through the InfoNCE contrastive loss function.

8. The industrial IoT data preprocessing method based on microservices according to claim 7, characterized in that, The InfoNCE contrastive loss function specifically enables the lightweight edge data perception model to learn the semantic distance metric between cross-version data contract schema fields by maximizing the cosine similarity of field semantic vectors between positive sample pairs and minimizing the cosine similarity of field semantic vectors between negative sample pairs.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, which, when executed in a computer, are used to perform the industrial IoT data preprocessing method based on microservices as described in any one of claims 1-8.

10. A microservice-based industrial IoT data preprocessing system, characterized in that, The system comprises the computer-readable storage medium of claim 9, wherein the system is a computer, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor that executes program instructions stored in the computer-readable storage medium.

Images