A gateway-based multi-protocol environmental data fusion acquisition method

Abstract

The present invention relates to a gateway-based multi-protocol environmental data fusion collection method, comprising the following steps: obtaining the source protocol and target protocol of an environmental sensor gateway, building a field mapping relationship library between the source protocol and the target protocol; identifying contaminated data that does not meet the target protocol data quality requirements according to the target protocol; obtaining the source protocol type identifier and protocol version number of the contaminated data; determining the key nodes of the contaminated data in the protocol conversion process by analyzing the protocol conversion log of each contaminated data in the contaminated data isolation area; optimizing the numerical length restriction condition and data format conversion logic in the protocol conversion function; extracting the features of the contaminated data according to the protocol exception handling method; and redeploying the optimized protocol to the environmental sensor gateway. The present invention can effectively identify and handle data contamination problems in a multi-protocol environment, improve data quality, and provide reliable data support for environmental monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things environmental data collection, and in particular to a gateway-based multi-protocol environmental data fusion collection method. Background Art

[0002] In IoT systems, gateways serve as bridges connecting devices with different protocols, carrying out the crucial tasks of protocol conversion and data fusion. However, during multi-protocol data fusion, differences between protocols can lead to data contamination. For example, incompatible data types can cause format errors and missing information in the converted data. If data contamination is not addressed promptly, it can spread to subsequent data processing stages, impacting aggregation results, model training, and report generation.

[0003] Therefore, there is an urgent need for a fine-grained data pollution identification and isolation method that can go deep into the single record level, use the verification rules of the source protocol and the target protocol to detect illegal data in real time and prevent it from spreading downstream; at the same time, in order to trace the source and find the root cause of data pollution, it is also necessary to establish a data lineage mechanism to track the steps that each data goes through during the conversion process, so as to locate the conversion node where the problem occurs and debug and optimize it. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, the present invention aims to provide a gateway-based multi-protocol environmental data fusion and collection method. This gateway-based multi-protocol environmental data fusion and collection method can effectively identify and address data contamination issues in multi-protocol environments, improve data quality, and provide reliable data support for environmental monitoring.

[0005] The present invention provides a gateway-based multi-protocol environment data fusion collection method, comprising the following steps:

[0006] S1. Obtain the source protocol and target protocol of the environmental sensor gateway, build a field mapping relationship library between the source protocol and the target protocol, and verify the data of the environmental sensor;

[0007] S2. According to the target protocol, perform compliance judgment on the environmental sensor data processed by the protocol adapter plug-in, and identify contaminated data that does not meet the target protocol data quality requirements;

[0008] S3. Obtain the source protocol type identifier and protocol version number of the contaminated data, and store the contaminated data whose protocol is incompatible with the target protocol in a contaminated data isolation area;

[0009] S4. Analyze the protocol conversion log of each piece of contaminated data in the contaminated data isolation zone to determine the key nodes in the protocol conversion process of the contaminated data and locate the specific protocol field that introduces the contamination;

[0010] S5. Optimize the value length restriction and data format conversion logic in the protocol conversion function according to the specific protocol field, and update the conversion rules in the protocol adapter plug-in;

[0011] S6. Extracting features of the contaminated data according to the protocol exception handling method, and adding the extracted features to the field mapping relationship library and the source protocol data verification rule library;

[0012] S7. Redeploy the optimized protocol to the environmental sensor gateway, and improve the field mapping relationship library, source protocol data verification rules and the protocol adapter plug-in.

[0013] Preferably, the step S1 specifically includes:

[0014] Acquire the source protocol and target protocol of the environmental sensor gateway from the gateway database, and build a field mapping relationship library between the source protocol and the target protocol;

[0015] According to the field mapping relationship library, a mapping table is used to record the corresponding relationship between the source field and the target field and the parameter conversion ratio to obtain a conversion rule library of the source and target protocol fields;

[0016] A source protocol data verification rule base is constructed using a random forest algorithm according to the numerical range threshold, change threshold, and effective digit threshold set for the numerical field parameters in the conversion rule base;

[0017] The original data of the environmental sensor is read from the source protocol data buffer area, and the original data is verified according to the verification standard in the source protocol data verification rule library, and the verified source protocol data is obtained by using a threshold replacement method.

[0018] Preferably, the step S2 specifically includes:

[0019] Obtain the temperature, humidity, air pressure, wind speed, and radiation field data processed by the environmental sensor from the source protocol data buffer area, and use the naive Bayesian discriminator to obtain the abnormal data set based on the field value range judgment standard in the target protocol;

[0020] According to the field sampling time interval of the abnormal data set, a linear interpolation function is used to supplement the data points that exceed the sampling period, and the sliding variance is used to calculate the change trend of the supplemented data and the surrounding data to obtain the data set to be verified;

[0021] Based on the dataset to be verified, a convolutional neural network is used to learn the variation pattern of the numerical field sequence, the data anomaly level is determined by comparing the deviation between the sequence prediction value and the actual value, and a data anomaly level table is generated according to the anomaly level threshold;

[0022] Obtain the resolution requirement of the numerical field in the target protocol, and determine the number of significant digits of the numerical values ​​for the data of different levels in the data anomaly level table;

[0023] If the number of significant digits of the value exceeds the accuracy requirement threshold, a contaminated data record table is generated for the abnormally marked data that appears in the continuous sampling period, and contaminated data that does not meet the target protocol data quality requirements is obtained.

[0024] Preferably, the step S3 specifically includes:

[0025] Read the header identification segment of the contaminated data from the source protocol data buffer, obtain the source protocol identifier through the feature matcher, and use the naive Bayes classifier to compare the source protocol identifier with the standard features in the protocol feature library to obtain the source protocol identification result;

[0026] Based on the source protocol identification result, the corresponding relationship of the target protocol is searched in a pre-established protocol compatibility matrix, the feature similarity between the source protocol identifier and the target protocol is calculated using a feature vector, and the feature similarity and the conversion success rate are weighted to obtain the source-target protocol compatibility level;

[0027] According to the source-target protocol compatibility level, for the contaminated data below the compatibility level threshold, an isolation flag is added to the contaminated data header identification segment, and protocol information is extracted from the contaminated data according to the isolation flag to establish a contaminated data isolation zone;

[0028] According to the pollution data isolation area, the timestamp, geographic location mark and sensor identification of the read pollution data are written into the alarm record.

[0029] Preferably, the step S4 specifically includes:

[0030] Acquire a protocol conversion log record from the contaminated data isolation area, extract a conversion function identifier, an execution timestamp, a field identifier, a field value, and a conversion rule according to the protocol conversion log record, and obtain a conversion function execution record;

[0031] According to the conversion function execution record, a string matcher is used to extract input fields and output fields, and a conversion function linked list is established according to the conversion function execution sequence, wherein the conversion function linked list records the field value range, field precision and conversion rules;

[0032] According to the conversion function linked list, a directed graph builder is used to generate a field lineage dependency graph, wherein nodes in the field lineage dependency graph record function identifiers and timestamps, and lines between nodes record field identifiers and conversion rules;

[0033] For the field lineage dependency graph, obtaining a value of the source protocol field before conversion and a value of the target protocol field after conversion;

[0034] If the numerical deviation after field conversion exceeds the deviation threshold, the forward input field and the backward output field of the node are extracted from the field lineage dependency graph, and a field conversion path record is established.

[0035] Preferably, the step S5 specifically includes:

[0036] Obtain field value length limits, value range constraints, and value precision requirements from the source protocol field table and the target protocol field table, and use a string matcher to compare field attributes to obtain a field attribute difference table;

[0037] Extracting the value conversion rules of the source and target protocols according to the field attribute difference table, and classifying the value conversion rules using a naive Bayes classifier to obtain a value rule optimization table;

[0038] According to the numerical rule optimization table, the frequency histogram is used to statistically analyze the distribution of conversion data in the isolation area, and the data points that exceed the numerical limit range are marked to obtain an abnormal conversion mapping table;

[0039] The numerical length restriction parameter, the numerical range constraint parameter and the numerical precision requirement parameter in the numerical conversion rule are modified according to the abnormal conversion mapping table, and the conversion rule in the protocol adapter plug-in is updated.

[0040] Preferably, the step S6 specifically includes:

[0041] Obtaining the contaminated data in the contaminated data isolation zone, calculating three characteristic parameters of the contaminated data, namely, numerical value exceeding limit amplitude, numerical precision deviation, and sampling period offset, using a sliding window, and generating a contaminated data characteristic record according to a preset characteristic classification rule;

[0042] According to the contaminated data feature record, the rules corresponding to the source and target protocol fields are obtained from the field mapping relationship library, the field value range and the number of valid digits are extracted according to the rules corresponding to the source and target protocol fields, and field restriction rules are generated;

[0043] According to the field restriction rule, a field check condition is obtained from the source protocol data verification rule, and a value range constraint and a valid digit constraint are added to the field check condition to generate a field check rule;

[0044] Extract inspection conditions and threshold parameters from the field inspection rules, expand the field attributes in the field mapping relationship library, add value range parameters, valid digit parameters and sampling interval parameters to the numerical field attributes, supplement constraint parameters to the inspection conditions in the source protocol data verification rule library, and record the update results of the source protocol data verification rule library.

[0045] Preferably, the step S7 specifically includes:

[0046] The data collected by the environmental sensors is obtained from the environmental sensor gateway, and the naive Bayes classifier is used to identify abnormal data in the numerical range, numerical precision, and sampling period to obtain the pollution rate curve;

[0047] Obtaining pollution rate records before protocol optimization according to the pollution rate curve, and smoothing the pollution rate records using exponential weighted average to obtain a pollution rate reduction ratio;

[0048] According to the pollution rate reduction ratio, obtain the field name in the pollution rate non-compliant protocol, and obtain the field mapping rules, check parameters and conversion parameters from the field mapping relationship library, source protocol data verification rules and the protocol adapter plug-in according to the field name to obtain a rule optimization table;

[0049] According to the restriction parameters in the rule optimization table, the corresponding field rules in the field mapping relationship library, the corresponding field parameters in the source protocol data verification rules and the corresponding field conversion parameters in the protocol adapter plug-in are updated, and the update timestamp of the source protocol data verification rule library is recorded.

[0050] The gateway-based multi-protocol environment data fusion collection method described in the present invention has the following advantages:

[0051] The invention discloses a gateway-based multi-protocol environmental data fusion collection method. By constructing a field mapping relationship library between the source protocol and the target protocol, the method helps to efficiently fuse data under different protocols at the gateway level, thereby improving data compatibility. Through a protocol adapter plug-in, the method can realize the processing of environmental sensor data of multiple protocols, thereby enhancing the flexibility and scalability of the system. Through compliance judgment, the method can identify polluted data that does not meet the data quality requirements of the target protocol, thereby avoiding the impact of low-quality data on subsequent analysis and decision-making. By analyzing the protocol conversion log in the polluted data isolation area, the key nodes of the polluted data in the protocol conversion process can be accurately determined, and the specific protocol fields that introduce pollution can be located, thereby improving the efficiency of problem repair. By adding the extracted polluted data features to the field mapping relationship library and the source protocol data verification rule library, the method helps to better process and verify similar data in the future. The perfect field mapping relationship library, source protocol data verification rules and protocol adapter plug-in help to reduce the complexity of system maintenance and improve the stability and reliability of the system. The gateway-based multi-protocol environmental data fusion collection method can effectively identify and process data pollution problems in a multi-protocol environment, improve data quality, and provide reliable data support for environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a gateway-based multi-protocol environment data fusion collection method described in the present invention. DETAILED DESCRIPTION

[0053] like Figure 1 As shown, the gateway-based multi-protocol environment data fusion collection method described in the present invention includes the following steps:

[0054] S1. Obtain the source protocol and target protocol of the environmental sensor gateway, build a field mapping relationship library between the source protocol and the target protocol, and verify the data of the environmental sensor;

[0055] S2. According to the target protocol, the compliance of the environmental sensor data processed by the protocol adapter plug-in is judged, and the contaminated data that does not meet the data quality requirements of the target protocol is identified;

[0056] S3. Obtain the source protocol type identifier and protocol version number of the contaminated data, and store the contaminated data whose protocol is incompatible with the target protocol in a contaminated data isolation area;

[0057] S4. Analyze the protocol conversion log of each piece of contaminated data in the contaminated data isolation zone to determine the key nodes in the protocol conversion process of the contaminated data and locate the specific protocol fields that introduced the contamination;

[0058] S5. Optimize the value length restriction and data format conversion logic in the protocol conversion function according to the specific protocol field, and update the conversion rules in the protocol adapter plug-in;

[0059] S6. Extract features of the contaminated data according to the protocol exception handling method, and add the extracted features to the field mapping relationship library and the source protocol data verification rule library;

[0060] S7. Redeploy the optimized protocol to the environmental sensor gateway, and improve the field mapping relationship library, source protocol data verification rules, and protocol adapter plug-in.

[0061] Furthermore, in this embodiment, step S1 specifically includes:

[0062] Obtain the source protocol and target protocol of the environmental sensor gateway from the gateway database, and build a field mapping relationship library between the source protocol and the target protocol;

[0063] According to the field mapping relationship library, a mapping table is used to record the corresponding relationship between the source field and the target field, and the parameter conversion ratio, to obtain the conversion rule library of the source and target protocol fields;

[0064] According to the numerical range threshold, change threshold and effective digit threshold set by the numerical field parameters in the conversion rule base, the random forest algorithm is used to build the source protocol data verification rule base;

[0065] Read the original data of the environmental sensor from the source protocol data buffer, verify the original data according to the verification standard in the source protocol data verification rule library, and use the threshold replacement method to obtain the verified source protocol data;

[0066] Specifically, the source protocol field name, data type, numerical range, resolution, and precision parameters are obtained from the gateway database, and the target protocol field name, data type, numerical range, resolution, and precision parameters are obtained. A source-target protocol field mapping table is constructed. For the temperature, humidity, and pressure numerical fields, the mapping table is used to record the corresponding relationship between the source field and the target field, and the parameter conversion ratio, to obtain a conversion rule library for the source-target protocol field.

[0067] According to the numerical field parameters in the rule base for source-target protocol field conversion, set upper and lower limit verification thresholds for the source protocol field value range, set a minimum change verification threshold for the source protocol field resolution, and set a valid digit verification threshold for the source protocol field precision. A random forest algorithm is used to build a source protocol data verification rule base to generate three data verification standards for field value range, change, and valid digits.

[0068] Read the raw data of the environmental sensor from the source protocol data buffer, and execute the verification standards in the source protocol data verification rule library on the numeric fields in sequence;

[0069] If the field value exceeds the range threshold, the limit value is replaced. If the change exceeds the resolution threshold, the mean value is replaced. If the number of significant digits exceeds the precision threshold, truncation is performed and verified source protocol data is generated.

[0070] According to the conversion rule library of the source and target protocol fields, the parameters of the numerical fields in the verified source protocol data are converted. A convolutional neural network is used to detect anomalies in the converted numerical fields. The detected abnormal data is marked and recorded. A second verification is performed using the source protocol data verification rule library, and the verified data is encapsulated according to the target protocol format.

[0071] The environmental sensor gateway receives data from multiple sensor protocols. The source protocol uses a frequently changing protocol format, while the target protocol uses a periodic reporting protocol format. The source and target protocols differ in data structure and parameter settings.

[0072] Here is an example:

[0073] Taking temperature data as an example, the source protocol temperature value range is -40 to 80 degrees Celsius, with a resolution of 0.1 degrees Celsius and an accuracy of 0.5 degrees Celsius. The target protocol temperature value range is -50 to 100 degrees Celsius, with a resolution of 0.01 degrees Celsius and an accuracy of 0.1 degrees Celsius.

[0074] The humidity data source protocol value range is 0 to 100%, with a resolution of 1% and an accuracy of 2%. The target protocol value range is 0 to 100%, with a resolution of 0.1% and an accuracy of 1%. The pressure data source protocol value range is 800 to 1100 hPa, with a resolution of 1 hPa and an accuracy of 2 hPa. The target protocol value range is 700 to 1300 hPa, with a resolution of 0.1 hPa and an accuracy of 1 hPa.

[0075] The source-target protocol field mapping table records the mapping relationship between the three numerical fields of temperature, humidity, and air pressure. The conversion rule base has a conversion ratio of 10:1 for the temperature field, 10:1 for the humidity field, and 10:1 for the pressure field, to achieve conversion of the source protocol data to the target protocol data format.

[0076] When the source protocol temperature value of 81 degrees Celsius exceeds the upper limit of the value range of 80 degrees Celsius, 80 degrees Celsius is used as the replacement;

[0077] When the change in the source protocol humidity value by 2% exceeds the resolution threshold of 1%, the mean of the source data and the change data is used for replacement;

[0078] When the source protocol pressure value of 1056.789 hPa exceeds the accuracy threshold of 2 hPa, 1057 hPa is used instead;

[0079] The source protocol data verification rule base includes three standards: numerical field range verification, change verification, and accuracy verification;

[0080] Taking temperature data as an example, the range test determines whether the temperature is within the range of -40 to 80 degrees Celsius, the change test determines whether the temperature change is less than 0.1 degrees Celsius, and the precision test determines whether the number of digits after the decimal point of the temperature value exceeds 1;

[0081] The humidity data range test determines whether the humidity is within the range of 0 to 100%, the change test determines whether the humidity change is less than 1%, and the accuracy test determines whether the number of digits after the decimal point of the humidity value exceeds 0;

[0082] The pressure data range test determines whether the pressure is within the range of 800 to 1100 hPa, the change test determines whether the pressure change is less than 1 hPa, and the accuracy test determines whether the number of decimal places of the pressure value exceeds 0;

[0083] After verification, the source protocol data is converted according to the source-target protocol field conversion rules. After conversion, the temperature, humidity, and air pressure numerical field data are detected for anomalies, and the detected abnormal data is marked and recorded;

[0084] Taking temperature data as an example, according to the target protocol field order, the temperature field value 800 corresponds to 80.0 degrees Celsius and is encapsulated in the 3rd to 6th bytes of the target protocol;

[0085] The humidity field value 678 corresponds to 67.8% and is encapsulated in bytes 7 to 10 of the target protocol. The pressure field value 10203 corresponds to 1020.3 hPa and is encapsulated in bytes 11 to 14 of the target protocol.

[0086] The target protocol data consists of four parts: protocol header, data length, field data, and check code. The data packet is assembled according to the target protocol format requirements.

[0087] Furthermore, in this embodiment, step S2 specifically includes:

[0088] Obtain the temperature, humidity, air pressure, wind speed, and radiation field data processed by the environmental sensor from the source protocol data buffer. Based on the field value range judgment criteria in the target protocol, a naive Bayesian discriminator is used to obtain the abnormal data set.

[0089] According to the sampling time interval of the field in the abnormal data set, the linear interpolation function is used to supplement the data points that exceed the sampling period, and the sliding variance is used to calculate the change trend of the supplemented data and the surrounding data to obtain the data set to be verified;

[0090] Based on the dataset to be verified, a convolutional neural network is used to learn the variation patterns of numerical field sequences. The data anomaly level is determined by comparing the deviation between the sequence prediction value and the actual value, and a data anomaly level table is generated based on the anomaly level threshold.

[0091] Obtain the resolution requirements of the numerical fields in the target protocol and determine the number of significant digits for the different levels of data in the data anomaly level table;

[0092] If the number of significant digits of the value exceeds the accuracy requirement threshold, a contaminated data record table is generated for the abnormally marked data in the continuous sampling period, and contaminated data that does not meet the target protocol data quality requirements is obtained;

[0093] Specifically, the temperature, humidity, air pressure, wind speed, and radiation field data processed by the environmental sensor protocol adapter plug-in are obtained from the source data cache area, and the field data quality judgment standard in the target protocol is obtained. According to the three indicators of the target protocol field value range, value change interval, and value occurrence frequency, a naive Bayesian discriminator is used to identify anomalies of a single numerical field. The data that exceeds the value range, has an abnormal change interval, and has an abnormal occurrence frequency are marked and stored, and the abnormal data set is obtained according to the mark position index;

[0094] According to the sampling period requirements of the numerical fields in the target protocol, the sampling time intervals of the fields in the abnormal data set are counted. If the time interval exceeds the sampling period, a linear interpolation function is used to supplement the missing data points. The numerical compliance of the supplemented data is judged. The sliding variance is used to calculate the change trend of the supplemented data and the surrounding data. The data with a change amplitude exceeding the threshold specified by the target protocol is recorded and marked to obtain the data set to be verified;

[0095] For the validation dataset, we obtained the numerical range limits of the temperature, humidity, air pressure, wind speed, and radiation fields from the target protocol quality judgment standard. We used a convolutional neural network to learn the variation patterns of the numerical field sequences. We compared the deviation between the sequence prediction value and the actual value to determine the data anomaly level. We then graded and labeled the data according to the anomaly level threshold and generated a data anomaly level table.

[0096] Obtain the resolution and accuracy requirements of the numerical fields in the target protocol, determine the number of significant digits for data of different levels in the data anomaly level table, mark the data if the number of significant digits exceeds the accuracy requirement limit, mark the data simultaneously for multiple fields, and generate a contaminated data record table for abnormally marked data within a continuous sampling period. The record contains the abnormal field name, abnormal value, abnormal level, and associated fields.

[0097] Environmental sensor data quality assessment involves multiple dimensions including numerical range, sampling period, and data accuracy;

[0098] Here is an example:

[0099] The temperature field value range is limited to -40 to 80 degrees Celsius. Any value outside the range is considered abnormal. The temperature change interval shall not exceed 2 degrees Celsius every 5 seconds. The frequency of the value occurrence shall not exceed 10% of the total sampling times within 24 hours.

[0100] The humidity field value is limited to between 0 and 100%, the humidity change within 5 seconds shall not exceed 5%, and the number of repetitions of the same humidity value shall not exceed 15% of the total sampling times;

[0101] The pressure field value is limited to 800 to 1100 hPa. The pressure change within 5 seconds must not exceed 10 hPa, and the same pressure value must not appear more than 20% of the total sampling time. The sampling period for numeric fields is set to 5 seconds. The temperature data at 15:00:05 shows 25.6 degrees Celsius, and at 15:00:15 shows 26.1 degrees Celsius. The intermediate sampling point at 15:00:10 is missing. Linear interpolation is used to calculate the temperature value at 15:00:10 as 25.85 degrees Celsius.

[0102] The humidity data at 15:00:05 showed 67.8%, and at 15:00:15 it showed 68.9%. The interpolated humidity value at 15:00:10 was 68.35%.

[0103] The air pressure data at 15:00:05 shows 1013.2 hPa, and at 15:00:15 shows 1013.8 hPa. The interpolated air pressure value at 15:00:10 is 1013.5 hPa.

[0104] Field value anomalies are classified into four levels: temperature data exceeding -40 to 80 degrees Celsius is considered a level 4 anomaly, a temperature change exceeding 2 degrees Celsius within 5 seconds is considered a level 3 anomaly, a temperature value exceeding the limit for repeated occurrences is considered a level 2 anomaly, and a temperature accuracy exceeding 0.1 degrees Celsius is considered a level 1 anomaly.

[0105] If the humidity data exceeds 0 to 100%, it is considered a level 4 abnormality; if the humidity changes by more than 5% in 5 seconds, it is considered a level 3 abnormality; if the humidity value appears repeatedly more than the standard, it is considered a level 2 abnormality; if the humidity accuracy exceeds 0.1%, it is considered a level 1 abnormality;

[0106] If the air pressure data exceeds 800 to 1100 hPa, it is considered a Level 4 abnormality. If the air pressure changes by more than 10 hPa in 5 seconds, it is considered a Level 3 abnormality. If the air pressure value is repeated more than the standard number of times, it is considered a Level 2 abnormality. If the air pressure accuracy exceeds 0.1 hPa, it is considered a Level 1 abnormality.

[0107] The pollution data record table records the specific information of abnormal data. The temperature data of three consecutive sampling points from 15:00:05 to 15:00:15, 85.6, 86.1, and 86.3 degrees Celsius, exceeded the upper limit, and was determined to be a level 4 abnormality;

[0108] Humidity data from 15:00:05 to 15:00:15 showed 65.5%, 72.8%, and 79.2% for three consecutive sampling points. The change exceeded 5% in 5 seconds, which was considered a Level 3 abnormality.

[0109] The air pressure data at three consecutive sampling points from 15:00:05 to 15:00:15 showed 1013.256, 1013.852, and 1013.534 hPa, with an accuracy exceeding 0.1 hPa, and was determined to be Level 1 abnormality;

[0110] The temperature level 4 anomaly and the humidity level 3 anomaly overlap on the time axis, and the anomaly correlation is recorded.

[0111] Furthermore, in this embodiment, step S3 specifically includes:

[0112] Read the header identification segment of the contaminated data from the source protocol data buffer, obtain the source protocol identifier through the feature matcher, and use the naive Bayes classifier to compare the source protocol identifier with the standard features in the protocol feature library to obtain the source protocol identification result;

[0113] Based on the source protocol identification result, the corresponding relationship of the target protocol is searched in the pre-established protocol compatibility matrix. The feature similarity between the source protocol identifier and the target protocol is calculated using the feature vector. The feature similarity and conversion success rate are weighted to obtain the source and target protocol compatibility level.

[0114] According to the source-target protocol compatibility level, for contaminated data below the compatibility level threshold, an isolation flag is added to the contaminated data header identification segment, and protocol information is extracted from the contaminated data based on the isolation flag to establish a contaminated data isolation zone;

[0115] According to the pollution data isolation zone, the timestamp, geographic location tag and sensor identification of the pollution data read are written into the alarm record;

[0116] Specifically, the contaminated data header identification segment is read from the source protocol data buffer, the source protocol identifier, protocol version number and sensor identification number are extracted through the feature matcher, matching items are searched in the protocol feature library according to the source protocol identifier, and the source protocol identifier is compared with the standard features in the protocol feature library using the naive Bayes classifier. The source protocol identification result is obtained by comparing the source protocol feature sequence with the standard features.

[0117] For the source protocol identification result, the corresponding relationship with the target protocol is found in the compatibility matrix. The characteristic vector is used to calculate the similarity between the source protocol identifier and the target protocol feature, and the source-target protocol data conversion success rate in the compatibility matrix is ​​obtained. The source-target protocol feature similarity and conversion success rate are weighted to generate the source-target protocol compatibility level.

[0118] If the compatibility level is lower than the compatibility level threshold, an isolation flag is added to the contaminated data header identification segment;

[0119] According to the isolation identification bit of the contaminated data header, the protocol header identification, timestamp, location identification and sensor identification are extracted from the contaminated data. The location identification data is corrected using a Kalman filter to generate a geographical location tag for the contaminated data. An isolation area for the contaminated data is established. The isolation area is divided into storage blocks according to the source protocol identification, and the contaminated data is stored in the corresponding storage blocks.

[0120] For the contaminated data in the isolated storage block, read the contaminated data timestamp, geographic location tag, and sensor identifier, establish an alarm record index table, generate a unique identification number for the alarm record, write the contaminated data timestamp, geographic location tag, sensor identifier, source protocol identifier, and isolated block number into the alarm record, and establish a time dimension index and a space dimension index for the alarm record;

[0121] Here is an example:

[0122] Environmental sensor protocol adaptation involves multiple source protocol formats. The source protocol identifier uses 16-bit binary encoding, with the upper 8 bits indicating the protocol type and the lower 8 bits indicating the version number. The sensor identifier uses 32-bit encoding, with the upper 16 bits indicating the sensor type and the lower 16 bits indicating the device serial number.

[0123] The protocol feature library records the source protocol standard feature sequence, including the message header identifier, length field position, data field start position, check field structure and other feature items;

[0124] The source protocol identifier 1010010100000011 indicates the temperature sensor data acquisition protocol version 3.0, 1101001000000010 indicates the humidity sensor data acquisition protocol version 2.0, and 1011001100000001 indicates the pressure sensor data acquisition protocol version 1.0.

[0125] The protocol compatibility matrix uses a weighted directed graph structure to record the compatibility relationship between the source protocol and the target protocol. The rows of the protocol compatibility matrix represent the source protocol identifiers, the columns represent the target protocol identifiers, and the weights represent the protocol compatibility levels.

[0126] The temperature sensor data acquisition protocol version 3.0 has a compatibility level of 0.95 with the target protocol and a data conversion success rate of 0.98;

[0127] The humidity sensor data acquisition protocol version 2.0 has a compatibility level of 0.85 with the target protocol and a data conversion success rate of 0.92;

[0128] The compatibility level of the air pressure sensor data acquisition protocol version 1.0 with the target protocol is 0.75, and the data conversion success rate is 0.88;

[0129] The preset compatibility level threshold is 0.8, and data isolation mechanism is triggered when the threshold is lower than the threshold;

[0130] The contaminated data isolation area is divided into storage blocks according to the source protocol identifier. Each storage block contains a data cache area and an index area.

[0131] The temperature sensor pollution data is stored in block number 001. The pollution data location is 40.5 degrees north latitude and 116.3 degrees east longitude, with a timestamp of 2024011513254 and a sensor identification number of TS00000123.

[0132] The humidity sensor pollution data is stored in block number 002, with the pollution data location marked as 40.6 degrees north latitude and 116.4 degrees east longitude, timestamp 2024011513255, and sensor identification number HS00000234;

[0133] The air pressure sensor pollution data is stored in block number 003. The pollution data location is 40.7 degrees north latitude and 116.5 degrees east longitude, with a timestamp of 2024011513256 and a sensor identification number of PS00000345.

[0134] The alarm record index table adopts a bidirectional linked list structure and establishes indexes according to the time dimension and space dimension;

[0135] The alarm record unique identification number AL202401151325001 records the temperature sensor pollution data, including the timestamp 2024011513254, the location mark N40.5E116.3, the sensor identifier TS00000123, the source protocol identifier 0xA503, and the isolation block number 001;

[0136] Alarm record AL202401151325002 records humidity sensor pollution data, including timestamp 2024011513255, location mark N40.6E116.4, sensor identifier HS00000234, source protocol identifier 0xD402, and isolation block number 002;

[0137] Alarm record AL202401151325003 records air pressure sensor pollution data, including timestamp 2024011513256, location mark N40.7E116.5, sensor identifier PS00000345, source protocol identifier 0xB301, and isolation block number 003.

[0138] Furthermore, in this embodiment, step S4 specifically includes:

[0139] Acquire the protocol conversion log record from the contaminated data isolation area, extract the conversion function identifier, execution timestamp, field identifier, field value and conversion rule according to the protocol conversion log record, and obtain the conversion function execution record;

[0140] According to the conversion function execution record, a string matcher is used to extract the input field and the output field, and a conversion function linked list is established according to the conversion function execution sequence. The conversion function linked list records the field value range, field precision and conversion rules;

[0141] Based on the conversion function linked list, a directed graph builder is used to generate a field lineage dependency graph. The nodes in the field lineage dependency graph record function identifiers and timestamps, and the connections between nodes record field identifiers and conversion rules.

[0142] For the field lineage dependency graph, obtain the value of the source protocol field before conversion and the value of the target protocol field after conversion;

[0143] If the numerical deviation after field conversion exceeds the deviation threshold, the forward input field and backward output field of the node are extracted from the field lineage dependency graph, and a field conversion path record is established;

[0144] Specifically, the contaminated data protocol conversion log is obtained from the contaminated data isolation area. The conversion function identifier, execution timestamp, field identifier, field value and conversion rule are extracted according to the protocol conversion log record format. A string matcher is used to obtain the function execution order and field mapping relationship in the protocol conversion process from the log.

[0145] Extract the input and output fields from each conversion function execution record, build a conversion function linked list according to the function execution order, and record the field value range, field precision and conversion rules during the function processing;

[0146] Based on the conversion function linked list, a directed graph builder is used to generate a field lineage dependency graph. The nodes in the graph record function identifiers and timestamps, and the connections between nodes record field identifiers and conversion rules. The lineage dependency graph nodes are labeled according to the data flow from the source protocol field to the target protocol field. A naive Bayes classifier is used to classify the conversion function nodes, identify the correspondence between the function input and output fields, and generate a field lineage relationship table.

[0147] According to the field lineage relationship table, obtain the value of the source protocol field before conversion and the value of the target protocol field after conversion, calculate the standard conversion value according to the protocol conversion rules, compare the difference between the converted value and the standard conversion value, and use the Kalman filter to calculate the degree of deviation of the converted value;

[0148] If the numerical deviation of the field after conversion exceeds the deviation threshold, the corresponding conversion function node is marked in the field lineage relationship table;

[0149] According to the marked function node, the forward input field and backward output field of the node are extracted from the lineage dependency graph, and the function node context association path is constructed. The conversion function identifier, field identifier, and value change on the path are sorted according to the execution time sequence to establish the field conversion path record;

[0150] The conversion path record contains the function execution time, field identifier, field value, conversion rule and function identifier;

[0151] The protocol conversion log records the data change information during the conversion function processing. The log contains the conversion function identifier, execution time, field information, and value changes;

[0152] Here is an example:

[0153] Taking the temperature field conversion as an example, the conversion function identifier F001 is executed at the timestamp 20240115132501, the input field name is temp_raw, the input value is 256, the conversion rule is converted at a rate of 0.1, the output field name is temperature, and the output value is 25.6;

[0154] The conversion function F002 is executed at timestamp 20240115132502, with the input field temp_raw, the input value 825, the conversion rule 0.1 multiplier, the output field temperature, and the output value 85.6.

[0155] The field lineage dependency graph records the data flow during the field conversion process and uses a directed graph structure to represent the relationship between function nodes and fields;

[0156] For example, node F001 records the conversion function identifier and execution time 20240115132501, the input edge record field temp_raw has a value of 256, the output edge record field temperature has a value of 25.6, and the conversion rule records a multiplier of 0.1;

[0157] Node F002 records the function ID and time 20240115132502, the input edge temp_raw value is 825, the output edge temperature value is 85.6, and the conversion rule is 0.1 multiplier;

[0158] The field lineage relationship table records the correspondence between fields before and after conversion, including the source field name, source field value, conversion function identifier, target field name, and target field value.

[0159] In the temperature field record, the source field temp_raw value 256 is converted by function F001 to the target field temperature value 25.6, and the source field temp_raw value 825 is converted by function F002 to the target field temperature value 85.6.

[0160] The temperature field has a value range of -40 to 80 degrees Celsius. The temperature value of 85.6 is out of range, so the corresponding function node F002 is marked as abnormal.

[0161] The conversion path record table stores the entire process of field format conversion. The record content includes function execution time, field name, field value, conversion rule and function identifier;

[0162] The temperature data conversion path shows that at time 20240115132501, the input field temp_raw has a value of 256, and the conversion rule of 0.1 is used. The function F001 processes the output field temperature to obtain a value of 25.6. At time 20240115132502, the input field temp_raw has a value of 825, and the conversion rule of 0.1 is used. The function F002 processes the output field temperature to obtain a value of 85.6.

[0163] Through the path record, it is determined that the abnormal function node F002 introduces temperature data pollution, and the value of the polluted field temperature exceeds the range limit.

[0164] Furthermore, in this embodiment, step S5 specifically includes:

[0165] Obtain field value length limits, value range constraints, and value precision requirements from the source protocol field table and the target protocol field table, and use a string matcher to compare field attributes to obtain a field attribute difference table;

[0166] The numerical conversion rules of the source and target protocols are extracted based on the field attribute difference table, and the numerical conversion rules are classified using the naive Bayes classifier to obtain the numerical rule optimization table;

[0167] According to the numerical rule optimization table, the frequency histogram statistical data is used to analyze the distribution of conversion data in the isolation area, and the data points that exceed the numerical limit range are marked to obtain the abnormal conversion mapping table;

[0168] Modify the numerical length restriction parameter, numerical range constraint parameter, and numerical precision requirement parameter in the numerical conversion rule according to the abnormal conversion mapping table, and update the conversion rule in the protocol adapter plug-in;

[0169] Specifically, the upper limit of the value length, the upper and lower limits of the value range, and the number of digits of the value precision of the introduced contaminated field are obtained from the source protocol field table. The field value length limit, value range constraint, and value precision requirement are obtained from the target protocol field table. A string matcher is used to compare the source and target protocol field attributes item by item, and the differences in the three attributes of the source and target protocol fields in terms of value length, value range, and value precision are recorded to generate a field attribute difference table.

[0170] Extract source-target protocol value conversion rules based on the records in the field attribute difference table. Obtain the conversion ratio, truncation digits, and format standard of the source protocol value to the target protocol value. Use a naive Bayesian classifier to classify the value conversion rules. Mark the places where the value length limit, value range constraint, and value accuracy requirements do not meet the conversion rules. Establish a value rule optimization table.

[0171] According to the numerical rule optimization table, the original and converted values ​​of the contaminated data are obtained from the data isolation area. The frequency histogram is used to statistically analyze the distribution of the converted data. The positions of data points that exceed the numerical length limit, numerical range constraint, and numerical accuracy requirements are marked. The corresponding relationship between the pre-conversion value and the post-conversion value of the marked data points is extracted to establish an abnormal conversion mapping table.

[0172] Obtain the defect location of the numerical conversion rule from the abnormal conversion mapping table, modify the conversion rule according to the source and target protocol field attribute difference table, modify the numerical length limit parameter, numerical range constraint parameter, and numerical precision requirement parameter, add numerical limit judgment conditions to the conversion rule, update the conversion rule in the protocol adapter plug-in, and record the rule modification results. The source and target protocol field attributes include three indicators: numerical length, numerical range, and numerical precision;

[0173] Here is an example:

[0174] The source protocol temperature field temp_raw has a value length of 4 bytes, a value range of -400 to 800, and a value precision of 1. The mapped target protocol temperature field temperature has a value length of 2 bytes, a value range of -40.0 to 80.0, and a value precision of 0.1.

[0175] The source protocol humidity field hum_raw has a value length of 4 bytes, a value range of 0 to 1000, and a numerical precision of 1. The mapped target protocol humidity field humidity has a value length of 2 bytes, a value range of 0 to 100.0, and a numerical precision of 0.1.

[0176] The source protocol pressure field pres_raw has a length of 4 bytes, a range of 8000 to 11000, and a precision of 1. The mapped target protocol pressure field pressure has a length of 2 bytes, a range of 800.0 to 1100.0, and a precision of 0.1.

[0177] In the numerical conversion rule, the temperature field conversion ratio is 0.1. The value 825 is converted to 82.5, which exceeds the upper limit of the target protocol range of 80.0. The pressure field conversion ratio is 0.1. The value 11256 is converted to 1125.6, which exceeds the upper limit of the target protocol range of 1100.0.

[0178] The value rule optimization table records that the temp_raw to temperature conversion rule limits the source value to an upper limit of 800, and the pres_raw to pressure conversion rule limits the source value to an upper limit of 11000;

[0179] The conversion rule judgment condition for the temperature field is that the source value does not exceed 800 and the converted value does not exceed 80.0. The conversion rule judgment condition for the pressure field is that the source value does not exceed 11000 and the converted value does not exceed 1100.0.

[0180] The abnormal conversion data statistics show that at the temperature field conversion abnormal data point 20, the source value range is 825 to 856, and the converted value range is 82.5 to 85.6;

[0181] At the abnormal data point 15 of the pressure field conversion, the source value range is 11256 to 11389, and the converted value range is 1125.6 to 1138.9;

[0182] The abnormal conversion mapping table records that the source value 825 of the temperature field is mapped to the conversion value 82.5, the source value 856 is mapped to the conversion value 85.6, the source value 11256 of the pressure field is mapped to the conversion value 1125.6, and the source value 11389 is mapped to the conversion value 1138.9;

[0183] Updates to the protocol adapter plug-in conversion rules include limiting the temperature field length to 2 bytes, limiting the source value range to -400 to 800, setting a conversion factor of 0.1, and limiting the converted value range to -40.0 to 80.0.

[0184] The pressure field value is limited to 2 bytes in length, the source value range is limited to 8000 to 11000, the conversion ratio is 0.1, and the converted value range is limited to 800.0 to 1100.0;

[0185] After the update, the source value range judgment and the converted value range judgment are added during the execution of the conversion rules, and the out-of-range values ​​are limited.

[0186] Furthermore, in this embodiment, step S6 specifically includes:

[0187] Obtaining contaminated data in the contaminated data isolation zone, calculating three characteristic parameters of the contaminated data, namely, numerical value exceeding limit amplitude, numerical precision deviation, and sampling period offset, using a sliding window, and generating contaminated data feature records according to preset feature classification rules;

[0188] According to the contaminated data feature records, the rules corresponding to the source and target protocol fields are obtained from the field mapping relationship library. The field value range and valid digits are extracted according to the rules corresponding to the source and target protocol fields, and field restriction rules are generated;

[0189] According to the field restriction rules, the field check conditions are obtained from the source protocol data verification rules. The value range constraints and valid digit constraints are added to the field check conditions to generate the field check rules.

[0190] Extract check conditions and threshold parameters from field check rules, expand field attributes in the field mapping relationship library, add value range parameters, valid digit parameters, and sampling interval parameters to numeric field attributes, add constraint parameters to the check conditions in the source protocol data verification rule library, and record the update results of the source protocol data verification rule library;

[0191] Specifically, the contaminated data is read from the data isolation area, and the characteristic parameters of the contaminated data are extracted according to the protocol exception handling mechanism, including the numerical value exceeding the limit, numerical precision deviation and sampling period offset;

[0192] A sliding window is used to calculate the statistical distribution of pollution data characteristic parameters. According to the preset feature classification rules, the three features of value exceeding the limit, precision deviation and period offset are divided. The feature type, feature value, occurrence time and duration are recorded to generate pollution data feature records.

[0193] According to the contaminated data feature records, the source-target protocol field correspondence rules are obtained from the protocol mapping relationship library. According to the source-target protocol field correspondence rules, the field value range in the numerical limit crossing feature is extracted, the number of valid digits in the field in the precision deviation feature is extracted, and the field sampling interval in the period offset feature is extracted. The feature parameters are written into the source-target protocol field mapping table to generate field restriction rules.

[0194] Based on the field restriction rules, the field check conditions are obtained from the data validity check rules. The field check items are divided according to the value type. The value range constraint is added for the value limit violation feature, the effective digit constraint is added for the precision deviation feature, and the sampling interval constraint is added for the period offset feature to generate the field check rules.

[0195] Extract check conditions and threshold parameters from field check rules, expand field attributes in the source and target protocol field mapping libraries according to field types, add value range, valid digits, and sampling interval parameters to numeric field attributes, add constraint parameters to the check conditions in the source protocol data verification rule library, and record the rule library update results;

[0196] According to the added source-target protocol field mapping library and the supplemented source protocol data verification rule library, the source protocol data is verified, and the optimized protocol is obtained based on the verification results;

[0197] Here is an example:

[0198] The pollution data characteristics of environmental sensors include three categories: value exceeding the limit, accuracy deviation, and period offset;

[0199] The temperature field value exceeds the limit feature display. The value 82.5 degrees Celsius exceeds the upper limit of 80.0 degrees Celsius. The feature appears at 20240115132501 and lasts for 300 seconds.

[0200] The humidity field precision deviation feature shows that the value 67.856% exceeds the precision requirement by 0.1%. The feature appeared at 20240115132502 and lasted for 180 seconds.

[0201] The pressure field periodic offset feature shows that the sampling interval of 12 seconds exceeds the specified interval by 5 seconds. The feature appears at 20240115132503 and lasts for 240 seconds.

[0202] The source-target protocol field mapping rules record the correspondence between field names and value conversion rules. The temperature field is mapped from temp_raw to temperature, the source value 825 is mapped to 82.5, the value range of -400 to 800 is mapped to -40.0 to 80.0, and the value multiplication factor is 0.1. The humidity field is mapped from hum_raw to humidity, the source value 67856 is mapped to 67.856, the value range of 0 to 1000 is mapped to 0 to 100.0, and the value multiplication factor is 0.1.

[0203] The pressure field is mapped from pres_raw to pressure, and the sampling interval is mapped from 12 seconds in the source protocol to 5 seconds in the target protocol.

[0204] Data validity check rules set check conditions according to field type. The temperature field check rules include a value range check of -40.0 to 80.0 degrees Celsius, a value precision check of 0.1 degrees Celsius, and a sampling period check of 5 seconds.

[0205] The humidity field check rules include a value range check of 0 to 100.0%, a value precision check of 0.1%, and a sampling period check of 5 seconds;

[0206] The pressure field check rules include a value range check of 800.0 to 1100.0 hPa, a value precision check of 0.1 hPa, and a sampling period check of 5 seconds.

[0207] The source-target protocol field mapping library extends the field attributes. The temperature field attribute now has a value range of -40.0 to 80.0 degrees Celsius, 1 decimal place, and a sampling interval of 5 seconds.

[0208] The humidity field property has been added with a value range of 0 to 100.0%, 1 decimal place for significant digits, and a sampling interval of 5 seconds;

[0209] The pressure field attributes have been increased to include a value range of 800.0 to 1100.0 hPa, 1 decimal place for significant digits, and a sampling interval of 5 seconds.

[0210] The data validity check rule library updates the check conditions and constraint parameters. The rule update record includes the field name, check type, constraint parameters and update time.

[0211] Furthermore, in this embodiment, step S7 specifically includes:

[0212] The data collected by the environmental sensors is obtained from the environmental sensor gateway, and the naive Bayes classifier is used to identify abnormal data in the numerical range, numerical precision, and sampling period to obtain the pollution rate curve;

[0213] The pollution rate records before protocol optimization are obtained according to the pollution rate curve, and the pollution rate records are smoothed using exponential weighted average to obtain the pollution rate reduction ratio;

[0214] According to the pollution rate reduction ratio, the field names in the pollution rate non-compliant protocol are obtained. Based on the field names, field mapping rules, check parameters, and conversion parameters are obtained from the field mapping relationship library, source protocol data verification rules, and protocol adapter plug-in to obtain a rule optimization table.

[0215] According to the restriction parameters in the rule optimization table, update the corresponding field rules in the field mapping relationship library, the corresponding field parameters in the source protocol data verification rules, and the corresponding field conversion parameters in the protocol adapter plug-in, and record the source protocol data verification rule library update timestamp;

[0216] Specifically, the data collected by environmental sensors is extracted from the environmental sensor gateway according to the collection cycle. The data is checked one by one according to the data legitimacy check rules. The naive Bayes classifier is used to identify three types of data: abnormal value range, abnormal value precision, and abnormal sampling period. The ratio of the number of abnormal data in a single protocol to the total number of data is calculated to obtain the pollution rate. A fixed time window is set to record the pollution rate change curve.

[0217] Obtain the pollution rate records before protocol optimization from the historical database, compare the pollution rate curves of the same protocol in the same time period before and after optimization, smooth the pollution rate curves using exponential weighted average, calculate the pollution rate reduction ratio after optimization, and generate pollution rate comparison records based on protocol type, field name, and time period. If the pollution rate reduction ratio does not reach the preset threshold, trigger rule optimization;

[0218] Based on the pollution rate comparison records, the field names in the pollution rate non-compliant protocol are extracted. The field mapping rules are obtained from the field mapping relationship library, the check parameters are obtained from the data legitimacy check rules, and the conversion parameters are obtained from the protocol adapter plug-in. The three restriction parameters of field value range, numerical precision, and sampling period are supplemented by comparing the optimized pollution data characteristics to generate a rule optimization table.

[0219] Obtain restriction parameters from the rule optimization table, update the corresponding field rules in the protocol mapping relationship library according to the field name, update the corresponding field parameters in the data legitimacy check rule, update the corresponding field conversion parameters in the protocol adapter plug-in, record the rule library update timestamp, and repeat the data quality assessment according to the fixed assessment cycle;

[0220] Here is an example:

[0221] The data collection cycle of the environmental sensor is set to 5 seconds. Within the time window from 20240115132500 to 20240115133000, the temperature protocol collects 720 pieces of data, the humidity protocol collects 720 pieces of data, and the pressure protocol collects 720 pieces of data.

[0222] There were 36 abnormal temperature data values, with values ​​ranging from 82.5 to 85.6 degrees Celsius exceeding the upper limit of 80.0 degrees Celsius;

[0223] There were 24 numerical precision anomalies, with the number of digits after the decimal point exceeding 1; there were 12 sampling period anomalies, with the sampling interval exceeding 5 seconds;

[0224] There were 28 abnormal values ​​in the humidity data, with values ​​ranging from 102.5 to 105.8% exceeding the upper limit of 100.0%; there were 32 abnormal values ​​in the precision, with more than one decimal place.

[0225] There were 16 abnormal sampling periods, with the sampling interval exceeding 5 seconds, and 42 abnormal pressure ranges, with values ​​ranging from 1125.6 to 1138.9 hPa exceeding the upper limit of 1100.0 hPa.

[0226] There were 38 numerical precision anomalies, with the number of digits after the decimal point exceeding 1; there were 18 sampling period anomalies, with the sampling interval exceeding 5 seconds;

[0227] Before optimization, the temperature protocol contamination rate in the same period was 15%, the humidity protocol contamination rate was 12%, and the pressure protocol contamination rate was 13%. After optimization, the temperature protocol contamination rate was 10%, and the contamination rate was reduced by 33%.

[0228] The humidity agreement pollution rate is 8%, the pollution rate reduction ratio is 33%, the air pressure agreement pollution rate is 13%, the pollution rate reduction ratio is 0%;

[0229] The air pressure protocol contamination rate does not reach the preset reduction ratio threshold of 30%, triggering rule optimization;

[0230] The original parameter settings of the pressure protocol field mapping rules include: pres_raw maps pressure, the value range 8000 to 11000 maps 800.0 to 1100.0, and the conversion ratio is 0.1;

[0231] The original parameter settings for the data validity check rule include: the pressure field value range is 800.0 to 1100.0, the precision requirement is 0.1, and the sampling period is 5 seconds;

[0232] Supplement the restriction parameters based on the characteristics of the pollution data: the source value range is limited to 8000 to 10000, the target value range is limited to 800.0 to 1000.0, the precision is limited to 1 decimal place, and the sampling interval is limited to 5 seconds;

[0233] The rule base update record shows that the air pressure protocol field mapping rule was updated on 20240115133000. The update content includes narrowing the source value range to 8000 to 10000 and the target value range to 800.0 to 1000.0.

[0234] The data validity check rules were updated on 20240115133000. The updated content includes narrowing the value range of the pressure field to 800.0 to 1000.0.

[0235] The protocol adapter plug-in was updated on 20240115133000, including the pressure conversion limit parameters. The evaluation period is set to 300 seconds, triggering the next round of data quality evaluation on 20240115133500.

[0236] In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention.

[0237] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of the present invention.

Claims

1. A gateway-based multi-protocol environment data fusion collection method, characterized in that: The following steps are involved: S1. Obtain the source protocol and target protocol of the environmental sensor gateway, build a field mapping relationship library between the source protocol and the target protocol, and verify the data of the environmental sensor; S2. According to the target protocol, perform compliance judgment on the environmental sensor data processed by the protocol adapter plug-in, and identify contaminated data that does not meet the target protocol data quality requirements; S3. Obtain the source protocol type identifier and protocol version number of the contaminated data, and store the contaminated data whose protocol is incompatible with the target protocol in a contaminated data isolation area; S4. Analyze the protocol conversion log of each piece of contaminated data in the contaminated data isolation zone to determine the key nodes in the protocol conversion process of the contaminated data and locate the specific protocol field that introduces the contamination; S5. Optimize the value length restriction and data format conversion logic in the protocol conversion function according to the specific protocol field, and update the conversion rules in the protocol adapter plug-in; S6. Extracting features of the contaminated data according to the protocol exception handling method, and adding the extracted features to the field mapping relationship library and the source protocol data verification rule library; S7. Redeploy the optimized protocol to the environmental sensor gateway, and improve the field mapping relationship library, source protocol data verification rules and the protocol adapter plug-in.

2. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S1 specifically includes: Acquire the source protocol and target protocol of the environmental sensor gateway from the gateway database, and build a field mapping relationship library between the source protocol and the target protocol; According to the field mapping relationship library, a mapping table is used to record the corresponding relationship between the source field and the target field and the parameter conversion ratio to obtain a conversion rule library of the source and target protocol fields; A source protocol data verification rule base is constructed using a random forest algorithm according to the numerical range threshold, change threshold, and effective digit threshold set for the numerical field parameters in the conversion rule base; The original data of the environmental sensor is read from the source protocol data buffer area, and the original data is verified according to the verification standard in the source protocol data verification rule library, and the verified source protocol data is obtained by using a threshold replacement method.

3. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S2 specifically includes: Obtain the temperature, humidity, air pressure, wind speed, and radiation field data processed by the environmental sensor from the source protocol data buffer area, and use the naive Bayesian discriminator to obtain the abnormal data set based on the field value range judgment standard in the target protocol; According to the field sampling time interval of the abnormal data set, a linear interpolation function is used to supplement the data points that exceed the sampling period, and the sliding variance is used to calculate the change trend of the supplemented data and the surrounding data to obtain the data set to be verified; Based on the dataset to be verified, a convolutional neural network is used to learn the variation pattern of the numerical field sequence, the data anomaly level is determined by comparing the deviation between the sequence prediction value and the actual value, and a data anomaly level table is generated according to the anomaly level threshold; Obtain the resolution requirement of the numerical field in the target protocol, and determine the number of significant digits of the numerical values ​​for the data of different levels in the data anomaly level table; If the number of significant digits of the value exceeds the accuracy requirement threshold, a contaminated data record table is generated for the abnormally marked data that appears in the continuous sampling period, and contaminated data that does not meet the target protocol data quality requirements is obtained.

4. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S3 specifically includes: Read the header identification segment of the contaminated data from the source protocol data buffer, obtain the source protocol identifier through the feature matcher, and use the naive Bayes classifier to compare the source protocol identifier with the standard features in the protocol feature library to obtain the source protocol identification result; Based on the source protocol identification result, the corresponding relationship of the target protocol is searched in a pre-established protocol compatibility matrix, the feature similarity between the source protocol identifier and the target protocol is calculated using a feature vector, and the feature similarity and the conversion success rate are weighted to obtain the source-target protocol compatibility level; According to the source-target protocol compatibility level, for the contaminated data below the compatibility level threshold, an isolation flag is added to the contaminated data header identification segment, and protocol information is extracted from the contaminated data according to the isolation flag to establish a contaminated data isolation zone; According to the pollution data isolation area, the timestamp, geographic location mark and sensor identification of the read pollution data are written into the alarm record.

5. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S4 specifically includes: Acquire a protocol conversion log record from the contaminated data isolation area, extract a conversion function identifier, an execution timestamp, a field identifier, a field value, and a conversion rule according to the protocol conversion log record, and obtain a conversion function execution record; According to the conversion function execution record, a string matcher is used to extract input fields and output fields, and a conversion function linked list is established according to the conversion function execution sequence, wherein the conversion function linked list records the field value range, field precision and conversion rules; According to the conversion function linked list, a directed graph builder is used to generate a field lineage dependency graph, wherein nodes in the field lineage dependency graph record function identifiers and timestamps, and lines between nodes record field identifiers and conversion rules; For the field lineage dependency graph, obtaining a value of the source protocol field before conversion and a value of the target protocol field after conversion; If the numerical deviation after field conversion exceeds the deviation threshold, the forward input field and the backward output field of the node are extracted from the field lineage dependency graph, and a field conversion path record is established.

6. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S5 specifically includes: Obtain field value length limits, value range constraints, and value precision requirements from the source protocol field table and the target protocol field table, and use a string matcher to compare field attributes to obtain a field attribute difference table; Extracting the value conversion rules of the source and target protocols according to the field attribute difference table, and classifying the value conversion rules using a naive Bayes classifier to obtain a value rule optimization table; According to the numerical rule optimization table, the frequency histogram is used to statistically analyze the distribution of conversion data in the isolation area, and the data points that exceed the numerical limit range are marked to obtain an abnormal conversion mapping table; The numerical length restriction parameter, the numerical range constraint parameter and the numerical precision requirement parameter in the numerical conversion rule are modified according to the abnormal conversion mapping table, and the conversion rule in the protocol adapter plug-in is updated.

7. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S6 specifically includes: Obtaining the contaminated data in the contaminated data isolation zone, calculating three characteristic parameters of the contaminated data, namely, numerical value exceeding limit amplitude, numerical precision deviation, and sampling period offset, using a sliding window, and generating a contaminated data characteristic record according to a preset characteristic classification rule; According to the contaminated data feature record, the rules corresponding to the source and target protocol fields are obtained from the field mapping relationship library, the field value range and the number of valid digits are extracted according to the rules corresponding to the source and target protocol fields, and field restriction rules are generated; According to the field restriction rule, a field check condition is obtained from the source protocol data verification rule, and a value range constraint and a valid digit constraint are added to the field check condition to generate a field check rule; Extract inspection conditions and threshold parameters from the field inspection rules, expand the field attributes in the field mapping relationship library, add value range parameters, valid digit parameters and sampling interval parameters to the numerical field attributes, supplement constraint parameters to the inspection conditions in the source protocol data verification rule library, and record the update results of the source protocol data verification rule library.

8. The gateway-based multi-protocol environment data fusion collection method according to claim 1, characterized in that: The step S7 specifically includes: The data collected by the environmental sensors is obtained from the environmental sensor gateway, and the naive Bayes classifier is used to identify abnormal data in the numerical range, numerical precision, and sampling period to obtain the pollution rate curve; Obtaining pollution rate records before protocol optimization according to the pollution rate curve, and smoothing the pollution rate records using exponential weighted average to obtain a pollution rate reduction ratio; According to the pollution rate reduction ratio, obtain the field name in the pollution rate non-compliant protocol, and obtain the field mapping rules, check parameters and conversion parameters from the field mapping relationship library, source protocol data verification rules and the protocol adapter plug-in according to the field name to obtain a rule optimization table; According to the restriction parameters in the rule optimization table, the corresponding field rules in the field mapping relationship library, the corresponding field parameters in the source protocol data verification rules and the corresponding field conversion parameters in the protocol adapter plug-in are updated, and the update timestamp of the source protocol data verification rule library is recorded.

Images