An integrated storage and use management method and system for an Internet of Things platform

Abstract

The present invention relates to the field of Internet of Things management technology at oil and gas pipeline construction sites, and specifically, to an integrated storage and use management method and system for an Internet of Things platform. Business logic is embedded in the data storage stage to achieve storage and use at the same time, ensuring real-time and compliance. The present invention is based on real-time importance scoring, dynamically determines data priority, and triggers a high-response channel, so as to meet the immediate response requirements of oil and gas pipeline construction site monitoring equipment to abnormal events, shorten the abnormal response time, and thus significantly reduce the security risks caused by delays. At the same time, through dynamic compression of redundant data and adaptive encryption tags, efficient storage and compliance localized management and control are achieved, and edge resources can be dynamically scheduled in combination with resource occupancy coefficients, reducing the CPU occupancy rate of key equipment and improving bandwidth utilization. In addition, by introducing a risk prediction model, the accuracy of anomaly detection is improved, realizing an upgrade from passive response to active prevention.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things management at oil and gas pipeline construction sites, and in particular to an integrated storage and use management method and system for an Internet of Things platform. Background Art

[0002] The IoT platform is a core component of the IoT ecosystem. It integrates multiple technologies and services to achieve efficient connection and intelligent management between devices and the cloud.

[0003] Current IoT platforms generally adopt an architecture that separates data storage from business applications. This architecture centrally stores device data in the cloud or on edge servers, then uses independent business modules for data analysis and decision-making. However, in oil and gas pipeline construction site management scenarios (such as pressure monitoring and leak detection), this architecture presents the following key issues:

[0004] 1. Safety risks caused by insufficient real-time performance: Monitoring equipment (such as pressure sensors and leak detectors) at oil and gas pipeline construction sites must respond to abnormal events (such as pipeline leaks and abnormal pressure) within milliseconds. Traditional platforms first store device data in a database before triggering the analysis process, resulting in delayed alerts and potentially delaying emergency response.

[0005] 2. Massive redundant data increases the storage burden: Equipment at oil and gas pipeline construction sites continuously generates high-frequency monitoring data (for example, pressure sensors collect data multiple times per second), but the majority of this data is normal. Existing technologies still store all raw data, resulting in a surge in storage costs. Furthermore, redundant data interferes with the rapid retrieval of critical anomaly information.

[0006] 3. Conflicts between edge resources and compliance: Oil and gas pipeline construction site equipment is often located in remote areas or complex environments, relying on low-power edge nodes (such as field gateways). Traditional platforms cannot simultaneously implement high-strength data encryption storage and real-time business logic processing (such as dynamic adjustment of safety valve status) at the resource-constrained edge. Relying on cloud computing would violate industry data security standards (such as specific pipeline safety regulations).

[0007] Therefore, there is an urgent need for an integrated storage and use solution for oil and gas pipeline construction site management that can embed lightweight business logic (such as anomaly detection and instruction generation) during the data storage phase to achieve storage and use simultaneously. While ensuring data compliance, it can improve the exception response speed and reduce redundant storage costs. Summary of the Invention

[0008] The purpose of the present invention is to provide an integrated storage and use management method and system for an Internet of Things platform, so as to solve the problems proposed in the above background technology, such as the security risks caused by the lack of real-time performance of traditional Internet of Things platforms in the oil and gas pipeline construction site management scenario, the storage burden increased by massive redundant data, and the contradiction between edge resources and compliance.

[0009] To solve the above technical problems, the present invention adopts a technical solution: providing an integrated storage and usage management method for an Internet of Things platform, comprising the following steps:

[0010] S1. Real-time data collection: Real-time collection of equipment status data and environmental monitoring data at the oil and gas pipeline construction site. The equipment status data includes equipment operating parameters. , data generation frequency and network latency , the environmental monitoring data includes occupancy, memory usage, and network bandwidth;

[0011] S2. Real-time importance score calculation and priority determination: According to the formula , calculate the real-time importance score of oil and gas pipeline construction site data , the real-time importance score With preset threshold Compare, if ≥ , it is marked as a key data stream, and triggers the high-priority storage and real-time response channel, directly generates control instructions and sends them to the construction site equipment. < , it is marked as a normal data stream and performs lightweight storage;

[0012] S3, edge storage and business logic embedded processing: Dynamically compresses redundant data for common data streams, performs high-intensity encryption for key data streams, and simultaneously detects abnormal data and generates risk prediction values;

[0013] S4. Dynamic resource scheduling and compliance verification: Evaluate node load, dynamically reduce the frequency of data collection on non-critical devices, verify encryption tag strength, block illegal data transmission, and enforce local storage.

[0014] S5. Control instruction issuance and feedback optimization: High-priority instructions are sent directly to devices through edge nodes, and thresholds are dynamically adjusted based on execution results. and , optimize the scoring model and periodically update the risk prediction value .

[0015] As a further improvement of the present technical solution, the specific implementation method of dynamically compressing redundant data for the normal data stream in step S3 is as follows:

[0016] According to the formula , calculate storage redundancy, dynamically compress non-critical data, discard redundant data, and only retain complete critical data and abnormal fragment timestamps.

[0017] As a further improvement of this technical solution, the specific implementation method of performing high-intensity encryption on the key data stream in step S3 is:

[0018] Generate data encryption tags at the edge node and use the formula , calculate and generate the encryption tag strength , the encryption tag strength Dynamically adjust based on data sensitivity.

[0019] As a further improvement of this technical solution, the specific implementation method of synchronously detecting abnormal data and generating risk prediction values ​​in step S3 is:

[0020] Real-time analysis of equipment parameters If the critical data identification threshold is exceeded , it is marked as abnormal;

[0021] According to the formula , and calculate the risk prediction value by combining historical abnormal data ;

[0022] like Exceeding the preset threshold , then improve the device Calculate weights and trigger alerts first.

[0023] As a further improvement of this technical solution, the specific implementation method of evaluating the node load and dynamically reducing the frequency of non-critical equipment data in step S4 is as follows:

[0024] Calculate the resource occupancy coefficient according to the formula , evaluate edge node load ;

[0025] like Exceeding the threshold , then reduce the data collection frequency of non-critical equipment.

[0026] As a further improvement of this technical solution, the specific implementation method of verifying the encryption tag strength, blocking illegal data transmission, and forcing local storage in step S4 is:

[0027] Verify whether the encryption tag strength L meets the industry security standards. If the verification fails, Does not meet the minimum value required by regulations , the transmission is blocked and a local alarm is triggered;

[0028] If the verification passes, Meet the minimum value required by regulations , the data is only stored in the local edge node and is prohibited from being transmitted to the cloud.

[0029] To solve the above technical problems, another technical solution provided by the present invention is: providing an integrated storage and usage management system for an Internet of Things platform, the system comprising a data acquisition module, an edge storage processing module, a business logic embedding module, a dynamic priority scheduling module, and a compliance verification module;

[0030] The data acquisition module is used to collect equipment status data and environmental monitoring data at the oil and gas pipeline construction site in real time;

[0031] The edge storage processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection and encryption tag generation;

[0032] The business logic embedding module is used to dynamically adjust the data processing strategy according to preset rules;

[0033] The dynamic priority scheduling module is used to score according to real-time importance Sort data from multiple devices and prioritize the highest-scoring data streams;

[0034] The compliance verification module is used to ensure that data storage and processing comply with industry data security standards and prohibit the transmission of unencrypted data to the cloud.

[0035] As a further improvement of the present technical solution, the system further includes a processor, a memory and instructions stored therein, and the processor implements the steps of the above method when executing the instructions.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention calculates real-time importance scores , dynamically determine the data priority, directly trigger the high-priority channel to generate control instructions, bypassing the cloud processing link, thereby meeting the immediate response needs of oil and gas pipeline construction site monitoring equipment to abnormal events, shortening the abnormal response time, and significantly reducing the safety risks caused by delays.

[0038] 2. The present invention dynamically compresses non-critical data based on storage redundancy, retaining only complete exception fragments and timestamps, thereby avoiding the serious waste of resources caused by the traditional solution of storing high-frequency oil and gas pipeline data in full, thereby improving the steady-state data compression rate, reducing the overall storage cost, and improving the retrieval efficiency of key exception data.

[0039] 3. The present invention calculates the resource occupancy coefficient , evaluate the node load in real time, and dynamically adjust the data collection frequency of non-critical equipment, thereby reducing the CPU occupancy rate of critical equipment, while improving the network bandwidth utilization rate, ensuring stable operation of the edge end, and then dynamically scheduling edge resources to avoid equipment overload.

[0040] 4. The present invention calculates the dynamic encryption tag strength , adaptively adjust the encryption level and force local storage, thus avoiding the situation in which traditional solutions rely on cloud processing to violate industry data security standards and the difficulty of achieving high-strength encryption on the low-power edge. The encryption strength is adaptively matched to the sensitivity of the data, so that it can pass industry security audits and reduce illegal transmission incidents to zero.

[0041] 5. The present invention is based on risk prediction value , predict the probability of equipment failure and dynamically adjust the scoring weight, thereby improving the accuracy of equipment anomaly detection and reducing the false alarm rate, thereby achieving an upgrade from passive response to active prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the steps of the integrated storage and usage management method for the Internet of Things platform of the present invention.

[0043] Figure 2 This is a schematic diagram of the operation flow of the integrated storage and utilization management method for the Internet of Things platform of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] In a specific embodiment, Figure 1 Figure 2 As shown, a storage and use integrated management method of an Internet of Things platform includes the following steps:

[0046] The first step is real-time data collection.

[0047] Real-time collection of equipment status data and environmental monitoring data at oil and gas pipeline construction sites, including equipment operating parameters , data generation frequency and network latency , the environmental monitoring data includes occupancy, memory usage, and network bandwidth.

[0048] Step 2: Real-time importance score calculation and priority determination.

[0049] According to the formula , calculate real-time importance score .in, It is the normalized value of the device operating parameters. Generate frequency logarithmic transformation values ​​for the data to suppress the excessive influence of high-frequency data on the score. Used to amplify the priority of low-latency data.

[0050] Score real-time importance With preset threshold Compare, if ≥ , it is marked as a key data stream, and triggers the high-priority storage and real-time response channel, directly generates control instructions and sends them to the construction site equipment. < , it is marked as a normal data stream and lightweight storage is performed (such as only retaining abnormal fragments).

[0051] Step 3: Edge storage and business logic embedded processing.

[0052] 1. For ordinary data streams, dynamically compress redundant data. The specific operations are as follows:

[0053] According to the formula , calculate storage redundancy, dynamically compress non-critical data, discard redundant data, and only retain complete critical data and abnormal fragment timestamps.

[0054] 2. Perform high-intensity encryption on key data streams. The specific operations are as follows:

[0055] Generate data encryption tags at the edge node and use the formula , calculate and generate the encryption tag strength , the encryption tag strength Dynamically adjust based on data sensitivity.

[0056] 3. Synchronously detect abnormal data and generate risk prediction values. The specific operations are as follows:

[0057] Real-time analysis of equipment parameters If the critical data identification threshold is exceeded , it is marked as abnormal;

[0058] According to the formula , and calculate the risk prediction value by combining historical abnormal data ;

[0059] like Exceeding the preset threshold , then improve the device Calculate weights and trigger alarms first.

[0060] Step 4: Dynamic resource scheduling and compliance verification.

[0061] 1. Evaluate node load and dynamically reduce the frequency of non-critical equipment data collection. The specific operations are as follows:

[0062] Calculate the resource occupancy coefficient according to the formula , evaluate edge node load ;

[0063] like Exceeding the threshold , then reduce the data collection frequency of non-critical equipment.

[0064] 2. Verify the encryption tag strength, block illegal data transmission, and force local storage. The specific operations are as follows:

[0065] Verify whether the encryption tag strength L meets the industry security standards. If the verification fails, Does not meet the minimum value required by regulations , the transmission is blocked and a local alarm is triggered;

[0066] If the verification passes, Meet the minimum value required by regulations , the data is only stored in the local edge node and is prohibited from being transmitted to the cloud.

[0067] The present invention also provides an integrated storage and usage management system for an Internet of Things platform, the system comprising a data acquisition module, an edge storage processing module, a business logic embedding module, a dynamic priority scheduling module, and a compliance verification module;

[0068] The data acquisition module is used to collect equipment status data and environmental monitoring data at the oil and gas pipeline construction site in real time;

[0069] The edge storage processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection and encryption tag generation;

[0070] The business logic embedding module is used to dynamically adjust the data processing strategy according to preset rules;

[0071] The dynamic priority scheduling module is used to score according to real-time importance Sort data from multiple devices and prioritize the highest-scoring data streams;

[0072] The compliance verification module is used to ensure that data storage and processing comply with industry data security standards and prohibit the transmission of unencrypted data to the cloud. For example, the original image of weld X-ray inspection has the following risks: if the image contains the coordinates of weak points in the pipeline, it may be maliciously used for targeted destruction; its compliance requirements include failure to meet the encryption requirements of "Level 3 Sensitive Data" in GB / T 35678-2017 "Guidelines for Classification and Grading of Industrial Data".

[0073] The system further comprises a processor, a memory and instructions stored therein, and the steps of the above method are implemented when the processor executes the instructions.

[0074] The beneficial effects of the present invention are described below by means of specific embodiments:

[0075] Example 1: Real-time response to abnormal oil and gas pipeline pressure.

[0076] Scenario description: At an oil and gas pipeline construction site, the pipeline pressure suddenly becomes abnormal during operation, and construction parameters need to be adjusted in time to avoid safety accidents.

[0077] Step 1: Real-time data collection.

[0078] Equipment parameters: pipeline pressure value =130kPa (rated value =120kPa); data generation frequency =10H (data is generated 10 times per second); network delay time =50ms.

[0079] Step 2: Real-time importance score calculation.

[0080] (Preset threshold =2000).

[0081] Judgment result: > , marked as critical data flow, triggering the high priority channel.

[0082] Step 3: Generate and issue control instructions.

[0083] Directly generate the "reduce construction pressure" instruction and send it to the on-site control system through the edge node, shortening the instruction delay and avoiding safety accidents caused by overpressure in the pipeline.

[0084] Example 2: Optimization of oil and gas pipeline data storage.

[0085] Scenario description: Oil and gas pipeline construction sites continuously generate high-frequency data, requiring optimized storage efficiency to reduce storage costs.

[0086] Step 1: Data classification and compression.

[0087] Key data: Abnormal pipeline pressure fragments were detected ( =180bpm, threshold =150bpm);

[0088] Normal data: Steady-state pressure data ( =75bpm).

[0089] Step 2: Calculate storage redundancy.

[0090] Calculate storage redundancy for common data: .

[0091] Compression operation: discards 69% of redundant data and retains only complete exception fragments and timestamps.

[0092] It reduces storage costs, shortens the retrieval time of key abnormal data, and improves data analysis efficiency.

[0093] Example 3: Dynamic scheduling of marginal resources at an oil and gas pipeline construction site.

[0094] Scenario description: When multiple devices are running in parallel at an oil and gas pipeline construction site, edge node resources are limited.

[0095] Step 1: Calculate resource usage coefficient (threshold =0.005).

[0096] Occupancy = 85%, memory usage = 70%, network bandwidth = 100Mbps.

[0097] Calculate the resource occupancy coefficient according to the formula : .

[0098] Step 2: Dynamically adjust the data collection frequency of non-critical equipment.

[0099] because Exceeding the threshold ,Therefore, the acquisition frequency of on-site environmental monitoring sensors (such as temperature and humidity sensors) is reduced from 10Hz to 5Hz.

[0100] Reduced The occupancy rate reduces the resource usage of key equipment (such as pipeline pressure monitors) while improving bandwidth utilization, ensuring stable data transmission at the construction site.

[0101] Example 4: Compliance verification of oil and gas pipeline data.

[0102] Scenario description: Data packets at oil and gas pipeline construction sites must comply with data security regulations.

[0103] Step 1: Calculate the encryption tag strength (regulatory requirements =1500).

[0104] Parameter: Real-time importance scoring =1800, equipment safety level =0.9.

[0105] Generate encryption tag strength : .

[0106] Step 2: Local storage and transmission control.

[0107] Verification passed ( > ), data is only stored in local edge nodes and cannot be uploaded to unsafe cloud servers.

[0108] The encryption strength adaptively matches the data sensitivity to ensure the secure transmission and storage of data at the construction site, and to reduce illegal transmission incidents to zero.

[0109] Example 5: Oil and gas pipeline risk prediction and active warning.

[0110] Scenario description: Predict oil and gas pipeline failure risks based on historical data.

[0111] Step 1: Calculation of risk prediction value (threshold =0.3).

[0112] Historical data: There were 5 abnormalities in the past 24 hours, with severity levels of 2, 1, 3, 2, and 1 respectively.

[0113] Calculate the risk prediction value according to the formula : .

[0114] Step 2: Dynamically adjust the scoring weights.

[0115] because Exceeding the preset threshold , therefore, improving the real-time importance score The calculation weight is 20%, and the alarm is triggered first.

[0116] The accuracy of anomaly detection is improved and the false alarm rate is reduced, achieving an upgrade from passive response to active prevention, providing early warning of potential failures and thus avoiding the occurrence of safety accidents.

[0117] In summary, in Example 1, the present invention calculates the real-time importance score , dynamically determine the data priority, directly trigger the high-priority channel to generate control instructions, bypassing the cloud processing link, thereby meeting the immediate response needs of oil and gas pipeline construction site monitoring equipment to abnormal events, shortening the abnormal response time, and significantly reducing the safety risks caused by delays.

[0118] In Example 2, the present invention dynamically compresses non-critical data based on storage redundancy, retaining only complete exception fragments and timestamps, thereby avoiding the serious waste of resources caused by full storage of high-frequency oil and gas pipeline data in traditional solutions, thereby improving the steady-state data compression rate, reducing overall storage costs, and improving the retrieval efficiency of key exception data.

[0119] In embodiment 3, the present invention calculates the resource occupancy coefficient , evaluate the node load in real time, and dynamically adjust the data collection frequency of non-critical equipment, thereby reducing the CPU occupancy rate of critical equipment, while improving the network bandwidth utilization rate, ensuring stable operation of the edge end, and then dynamically scheduling edge resources to avoid equipment overload.

[0120] In embodiment 4, the present invention calculates the dynamic encryption tag strength , adaptively adjust the encryption level and force local storage, thus avoiding the situation in which traditional solutions rely on cloud processing to violate industry data security standards and the difficulty of achieving high-strength encryption on the low-power edge. The encryption strength is adaptively matched to the sensitivity of the data, so that it can pass industry security audits and reduce illegal transmission incidents to zero.

[0121] In Example 5, the present invention is based on the risk prediction value , predict the probability of equipment failure and dynamically adjust the scoring weight, thereby improving the accuracy of equipment anomaly detection and reducing the false alarm rate, thereby achieving an upgrade from passive response to active prevention.

[0122] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A storage and use integrated management method for an Internet of Things platform, characterized in that: The following steps are involved: S1. Real-time data collection: Real-time collection of equipment status data and environmental monitoring data at the oil and gas pipeline construction site. The equipment status data includes equipment operating parameters. , data generation frequency and network latency , the environmental monitoring data includes occupancy, memory usage, and network bandwidth; S2. Real-time importance score calculation and priority determination: According to the formula , calculate the real-time importance score of oil and gas pipeline construction site data , the real-time importance score With preset threshold Compare, if ≥ , it is marked as a key data stream, and triggers the high-priority storage and real-time response channel, directly generates control instructions and sends them to the construction site equipment. < , it is marked as a normal data stream and performs lightweight storage; S3, edge storage and business logic embedded processing: Dynamically compresses redundant data for common data streams, performs high-intensity encryption for key data streams, and simultaneously detects abnormal data and generates risk prediction values; The specific implementation method of synchronously detecting abnormal data and generating risk prediction values ​​in step S3 is as follows: Real-time analysis of equipment parameters If the critical data identification threshold is exceeded , it is marked as abnormal; According to the formula , and calculate the risk prediction value by combining historical abnormal data ; like Exceeding the preset threshold , then improve the device Calculate weights and trigger alarms first; S4. Dynamic resource scheduling and compliance verification: Evaluate node load, dynamically reduce the frequency of data collection on non-critical devices, verify encryption tag strength, block illegal data transmission, and enforce local storage. The specific implementation method of evaluating the node load and dynamically reducing the frequency of non-critical equipment data in step S4 is as follows: Calculate the resource occupancy coefficient according to the formula , evaluate edge node load ; like Exceeding the threshold , then reduce the data collection frequency of non-critical equipment; S5. Control instruction issuance and feedback optimization: High-priority instructions are sent directly to devices through edge nodes, and thresholds are dynamically adjusted based on execution results. and , optimize the scoring model and periodically update the risk prediction value .

2. The method for integrated storage and usage management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of dynamically compressing redundant data for the normal data stream in step S3 is: According to the formula , calculate storage redundancy, dynamically compress non-critical data, discard redundant data, and only retain complete critical data and abnormal fragment timestamps.

3. The method for integrated storage and usage management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of performing high-intensity encryption on the key data stream in step S3 is: Generate data encryption tags at the edge node and use the formula , calculate and generate the encryption tag strength , the encryption tag strength Dynamically adjust based on data sensitivity.

4. The method for integrated storage and usage management of the Internet of Things platform according to claim 3, characterized in that: The specific implementation method of verifying the encryption tag strength, blocking illegal data transmission, and forcing local storage in step S4 is: Verify whether the encryption tag strength L meets the industry security standards. If the verification fails, Does not meet the minimum value required by regulations , the transmission is blocked and a local alarm is triggered; If the verification passes, Meet the minimum value required by regulations , the data is only stored in the local edge node and is prohibited from being transmitted to the cloud.

5. An Internet of Things platform storage and utilization integrated management system using the Internet of Things platform storage and utilization integrated management method according to claim 1, characterized in that: The system includes a data acquisition module, an edge storage processing module, a business logic embedding module, a dynamic priority scheduling module and a compliance verification module; The data acquisition module is used to collect equipment status data and environmental monitoring data at the oil and gas pipeline construction site in real time; The edge storage processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection and encryption tag generation; The business logic embedding module is used to dynamically adjust the data processing strategy according to preset rules; The dynamic priority scheduling module is used to score according to real-time importance Sort data from multiple devices and prioritize the highest-scoring data streams; The compliance verification module is used to ensure that data storage and processing comply with industry data security standards and prohibit the transmission of unencrypted data to the cloud.

Images