Storage and use integrated management method and system for Internet of Things platform

Abstract

The invention relates to the technical field of internet-of-things management of oil and gas pipeline construction sites, in particular to an internet-of-things platform storage and use integrated management method and an internet-of-things platform storage and use integrated management system. Business logic is embedded in the data storage stage, storage and use are achieved at the same time, and real-time performance and compliance are guaranteed. Based on real-time importance scoring, dynamic judgment of data priority and triggering of a high-response channel, the instant response requirement of monitoring equipment on an oil and gas pipeline construction site on an abnormal event can be met, the abnormal response time is shortened, the safety risk caused by delay is remarkably reduced, and the safety of the oil and gas pipeline construction site is improved. Meanwhile, efficient storage and compliance localization management and control are achieved by dynamically compressing redundant data and adaptively encrypting labels, edge resources can be dynamically scheduled in combination with resource occupancy coefficients, the CPU occupancy rate of key equipment is reduced, the bandwidth utilization rate is improved, in addition, by introducing a risk prediction model, the accuracy of anomaly detection is improved, and the reliability of anomaly detection is improved. And the upgrade from passive response to active prevention is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things management at the construction site of oil and gas pipelines. Specifically, it relates to a method and system for integrated management of storage and usage on an Internet of Things platform. Background Art

[0002] The Internet of Things platform is a core component of the Internet of Things ecosystem. It enables efficient connection and intelligent management between devices and the cloud by integrating various technologies and services.

[0003] Currently, Internet of Things platforms generally adopt an architecture that separates data storage from business applications. That is, device data is centrally stored in the cloud or edge servers, and then independent business modules are called for data analysis and decision-making. However, in the management scenarios of oil and gas pipeline construction sites (such as pressure monitoring and leak detection), the following key problems exist in such an architecture: 1. Safety risks caused by insufficient real-time performance: Monitoring devices at oil and gas pipeline construction sites (such as pressure sensors and leak detectors) need to respond to abnormal events (such as pipeline leaks and abnormal pressures) in milliseconds. Traditional platforms need to first store device data in a database and then trigger the analysis process, resulting in alarm delays and potentially delaying the emergency response time.

[0004] 2. The storage burden is exacerbated by a large amount of redundant data: Devices at oil and gas pipeline construction sites continuously generate high-frequency monitoring data (such as pressure sensors collecting data multiple times per second), but most of it is normal state data. Existing technologies still store all original data in full, resulting in a sharp increase in storage costs, and redundant data interfering with the rapid retrieval of key abnormal information.

[0005] 3. The contradiction between edge resources and compliance: Devices at oil and gas pipeline construction sites are mostly distributed in remote areas or complex environments and rely on low-power edge nodes (such as on-site gateways). Traditional platforms cannot synchronously achieve high-strength data encrypted storage and real-time business logic processing (such as dynamically adjusting the state of safety valves) at the resource-constrained edge. If relying on cloud computing, it will violate industry data security standards (such as specific pipeline safety regulations).

[0006] Therefore, there is an urgent need for an integrated storage and usage solution for oil and gas pipeline construction site management, which can embed lightweight business logic (such as anomaly detection and instruction generation) during the data storage stage to achieve storage and usage simultaneously, and improve the anomaly response speed and reduce redundant storage costs while ensuring data compliance. Summary of the Invention

[0007] The object of the present invention is to provide an integrated management method and system for storage and usage of an Internet of Things platform, so as to solve the problems of safety risks caused by insufficient real-time performance, increased storage burden due to massive redundant data, and contradictions between edge resources and compliance in the management scenario of oil and gas pipeline construction sites as mentioned in the above background technology.

[0008] To solve the above technical problems, a technical solution adopted by the present invention is: to provide an integrated management method for storage and usage of an Internet of Things platform, including the following steps: S1. Real-time data collection: Real-time collect the device status data and environmental monitoring data of the oil and gas pipeline construction site, where the device status data includes device operation parameters , data generation frequency and network latency , and the environmental monitoring data includes occupancy rate, memory occupancy rate, and network bandwidth; S2. Real-time importance score calculation and priority determination: According to the formula , calculate the real-time importance score of the data at the oil and gas pipeline construction site , and compare the real-time importance score with a preset threshold . If ≥ , then mark it as a critical data stream, trigger a high-priority storage and real-time response channel, directly generate a control instruction and send it to the construction site equipment. If < , then mark it as an ordinary data stream and perform lightweight storage; S3. Edge storage and business logic embedding processing: For ordinary data streams, dynamically compress redundant data. For critical data streams, perform high-strength encryption, and at the same time synchronously detect abnormal data and generate a risk prediction value; S4. Dynamic resource scheduling and compliance verification: Evaluate the node load, dynamically reduce the data collection frequency of non-critical devices, and at the same time verify the encryption label strength, block illegal data transmission, and enforce local storage; S5. Control instruction issuance and feedback optimization: High-priority instructions are directly sent to the equipment through the edge node, and the threshold and are dynamically adjusted based on the execution result, the scoring model is optimized, and the risk prediction value is periodically updated .

[0009] As a further improvement of this technical solution, the specific implementation method of dynamically compressing redundant data for ordinary data streams in step S3 is: According to the formula , calculate the storage redundancy, dynamically compress non-critical data, discard redundant data, and only retain the complete critical data and abnormal segment timestamps.

[0010] As a further improvement of this technical solution, the specific implementation method of performing high-strength encryption on the critical data stream in step S3 is: Generate a data encryption label at the edge node, and according to the formula , calculate and generate the encryption label strength , the encryption label strength is dynamically adjusted according to data sensitivity.

[0011] As a further improvement of this technical solution, the specific implementation method of synchronously detecting abnormal data and generating a risk prediction value in step S3 is: Analyze device parameters in real time , if it exceeds the critical data identification threshold , then mark it as abnormal; According to the formula , and calculate the risk prediction value in combination with historical abnormal data ; If exceeds the preset threshold , then increase the computing weight of this device and give priority to triggering an alarm.

[0012] As a further improvement of this technical solution, the specific implementation method of evaluating the node load and dynamically reducing the data collection frequency of non-critical devices in step S4 is: Calculate the resource occupancy coefficient, according to the formula , evaluate the edge node load ; If exceeds the threshold , then reduce the data collection frequency of non-critical devices.

[0013] As a further improvement of this technical solution, the specific implementation method of verifying the encryption label strength, blocking illegal data transmission, and forcing local storage in step S4 is: Verify whether the encryption label strength L meets the industry security standard. If the verification fails, that is it does not reach the minimum value required by regulations , then block the transmission and trigger a local alarm; If the verification passes, it reaches the minimum value required by regulations , then the data is only stored in the local edge node and prohibited from being transmitted to the cloud.

[0014] To solve the above technical problems, another technical solution provided by the present invention is: to provide an integrated management system for storage and use of an Internet of Things platform, the system including a data acquisition module, an edge storage and processing module, a business logic embedding module, a dynamic priority scheduling module, and a compliance verification module; The data acquisition module is used to collect device status data and environmental monitoring data at the construction site of the oil and gas pipeline in real time; The edge storage and processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection, and encrypted tag generation; The business logic embedding module is used to dynamically adjust the data processing strategy according to preset rules; The dynamic priority scheduling module is used to sort multi-device data according to the real-time importance score and preferentially process the data stream with the highest score; The compliance verification module is used to ensure that data storage and processing comply with industry data security standards and prohibit unencrypted data from being transmitted to the cloud.

[0015] As a further improvement of this technical solution, the system further includes a processor, a memory, and instructions stored therein, and when the processor executes the instructions, the steps of the above method are implemented.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention calculates the real-time importance score , dynamically determines the data priority, directly triggers the generation of control instructions for the high-priority channel, bypasses the cloud processing link, so as to meet the immediate response requirements of the monitoring equipment at the construction site of the oil and gas pipeline for abnormal events, shortens the abnormal response time, and further significantly reduces the safety risks caused by delays.

[0017] 2. The present invention dynamically compresses non-critical data based on the storage redundancy, and only retains the complete abnormal fragments and timestamps, thus avoiding the serious waste of resources caused by the full-volume storage of high-frequency oil and gas pipeline data in the traditional scheme, further improving the steady-state data compression rate, reducing the overall storage cost, and at the same time improving the retrieval efficiency of key abnormal data.

[0018] 3. The present invention calculates the resource occupancy coefficient , evaluates the node load in real time, dynamically adjusts the data acquisition frequency of non-critical devices, thereby reducing the CPU occupancy rate of critical devices, improving the network bandwidth utilization rate at the same time, ensuring the stable operation of the edge side, and further dynamically scheduling the edge resources to avoid the occurrence of device overload.

[0019] 4. The present invention calculates the dynamic encryption tag strength , adaptively adjust the encryption level and enforce local storage, thus avoiding the situation in the traditional solution where relying on cloud processing violates industry data security standards and it is difficult for low-power edge devices to achieve high-intensity encryption. This enables the encryption intensity to adaptively match the data sensitivity, thereby being able to pass industry security audits and reducing the number of illegal transmission events to zero.

[0020] 5. The present invention is based on the risk prediction value , predict the probability of equipment failure, and dynamically adjust the scoring weight, so as to improve the accuracy of equipment anomaly detection, while reducing the false alarm rate, and then achieve the upgrade from passive response to active prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the steps of the integrated storage and usage management method for the Internet of Things platform of the present invention.

[0022] Figure 2 It is a schematic diagram of the operation process of the integrated storage and usage management method for the Internet of Things platform of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0024] In a specific embodiment, as Figure 1 Figure 2 shown, an integrated storage and usage management method for an Internet of Things platform includes the following steps: The first step: Real-time data collection.

[0025] Real-time collect the equipment status data and environmental monitoring data at the oil and gas pipeline construction site. The equipment status data includes equipment operation parameters , data generation frequency and network latency time , and the environmental monitoring data includes occupancy rate, memory occupancy rate, and network bandwidth.

[0026] The second step: Calculation of real-time importance score and priority determination.

[0027] According to the formula , calculate the real-time importance score . Among them, is the normalized value of the equipment operation parameters. It is the logarithmic conversion value of the data generation frequency, which suppresses the excessive influence of high-frequency data on scoring. Used to amplify the priority of low-latency data.

[0028] The real-time importance score is compared with a preset threshold . If ≥ , it is marked as a critical data stream, and a high-priority storage and real-time response channel is triggered to directly generate control instructions and send them to the construction site equipment. If < , it is marked as an ordinary data stream and lightweight storage is performed (such as only retaining abnormal segments).

[0029] Step 3: Edge storage and business logic embedding processing.

[0030] 1. For ordinary data streams, redundant data is dynamically compressed. The specific operation is as follows: According to the formula , the storage redundancy is calculated, non-critical data is dynamically compressed, redundant data is discarded, and only complete critical data and abnormal segment timestamps are retained.

[0031] 2. For critical data streams, high-strength encryption is performed. The specific operation is as follows: A data encryption label is generated at the edge node, and according to the formula , the encryption label strength is calculated. The encryption label strength is dynamically adjusted according to data sensitivity.

[0032] 3. Abnormal data is synchronously detected and a risk prediction value is generated. The specific operation is as follows: Device parameters are analyzed in real time . If it exceeds the critical data identification threshold , it is marked as abnormal; According to the formula , and combined with historical abnormal data, the risk prediction value is calculated; If exceeds the preset threshold , the computing weight of this device is increased, and an alarm is triggered preferentially.

[0033] Fourth step: Dynamic resource scheduling and compliance verification.

[0034] 1. Evaluate the node load and dynamically reduce the data collection frequency of non-critical devices. The specific operation is as follows: Calculate the resource occupancy coefficient. According to the formula , evaluate the edge node load ; If exceeds the threshold , the data acquisition frequency of non-critical devices is reduced.

[0035] 2. Verify the encryption label strength, block illegal data transmission, and enforce local storage. The specific operations are as follows: Verify whether the encryption label strength L meets the industry security standard. If the verification fails, that is it does not reach the minimum value required by regulations , transmission is blocked and a local alarm is triggered; If the verification passes, it reaches the minimum value required by regulations , the data is only stored in the local edge node, and transmission to the cloud is prohibited.

[0036] The present invention also provides an integrated storage and usage management system for the Internet of Things platform. The system includes a data acquisition module, an edge storage and processing module, a business logic embedding module, a dynamic priority scheduling module, and a compliance verification module; The data acquisition module is used to collect device status data and environmental monitoring data at the oil and gas pipeline construction site in real time; The edge storage and processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection, and encryption label generation; The business logic embedding module is used to dynamically adjust data processing strategies according to preset rules; The dynamic priority scheduling module is used to sort multi-device data according to real-time importance scores and give priority to processing the data stream with the highest score; The compliance verification module is used to ensure that data storage and processing comply with industry data security standards, and prohibit the transmission of unencrypted data to the cloud. For example, the original X-ray inspection image of the weld seam, the risk is: if the image contains the coordinates of the pipeline weak point, it may be maliciously exploited for targeted destruction; the compliance requirement: it does not meet the encryption requirements for "level 3 sensitive data" in GB / T 35678-2017 "Guidelines for Classification and Grading of Industrial Data".

[0037] The system also includes a processor, a memory, and instructions stored therein. When the processor executes the instructions, the steps of the above method are implemented.

[0038] The beneficial effects of the present invention are illustrated by the following specific embodiments: Embodiment 1: Real-time response to abnormal oil and gas pipeline pressure.

[0039] Scenario description: At an oil and gas pipeline construction site, the pipeline pressure suddenly becomes abnormal during operation, and construction parameters need to be adjusted in time to avoid safety accidents.

[0040] Step 1: Real-time data acquisition.

[0041] Equipment parameters: Pipeline pressure value = 130 kPa (rated value = 120 kPa); Data generation frequency = 10 Hz (generate 10 times of data per second); Network latency = 50 ms.

[0042] Step 2: Calculation of real-time importance score.

[0043] (Preset threshold = 2000).

[0044] Judgment result: > , Marked as critical data stream, triggering high-priority channel.

[0045] Step 3: Generation and distribution of control instructions.

[0046] Directly generate the instruction of "reducing construction pressure" and distribute it to the on-site control system through the edge node, so as to shorten the instruction delay and avoid safety accidents caused by overpressure of the pipeline.

[0047] Example 2: Optimization of oil and gas pipeline data storage.

[0048] Scenario description: High-frequency data is continuously generated at the construction site of the oil and gas pipeline, and the storage efficiency needs to be optimized to reduce the storage cost.

[0049] Step 1: Data classification and compression.

[0050] Critical data: Detected abnormal pipeline pressure segments ( = 180 bpm, threshold = 150 bpm); Ordinary data: Steady-state pressure data ( = 75 bpm).

[0051] Step 2: Calculation of storage redundancy.

[0052] Calculate the storage redundancy for ordinary data: .

[0053] Compression operation: Discard 69% of the redundant data and only retain the complete abnormal segments and timestamps.

[0054] Reduces the storage cost, shortens the retrieval time of critical abnormal data, and improves the data analysis efficiency.

[0055] Example 3: Dynamic scheduling of edge resources at the oil and gas pipeline construction site.

[0056] Scenario description: When multiple devices are running in parallel at the oil and gas pipeline construction site, the resources of the edge nodes are strained.

[0057] Step 1: Calculate the resource occupancy factor (threshold = 0.005).

[0058] Occupancy rate = 85%, memory occupancy rate = 70%, network bandwidth = 100 Mbps.

[0059] Calculate the resource occupancy factor according to the formula : .

[0060] Step 2: Dynamically adjust the data collection frequency of non-critical devices.

[0061] Since exceeds the threshold , the data collection frequency of on-site environmental monitoring sensors (such as temperature and humidity sensors) is reduced from 10 Hz to 5 Hz.

[0062] The occupancy rate is reduced, resulting in reduced resource occupancy of critical devices (such as pipeline pressure monitors) and increased bandwidth utilization, ensuring stable data transmission at the construction site.

[0063] Example 4: Oil and gas pipeline data compliance verification.

[0064] Scenario description: The data packets at the oil and gas pipeline construction site need to ensure compliance with data security regulations.

[0065] Step 1: Calculate the encryption label strength (regulatory requirement = 1500).

[0066] Parameters: Real-time importance score = 1800, device security level = 0.9.

[0067] Generate the encryption label strength : .

[0068] Step 2: Localized storage and transmission control.

[0069] The verification passes ( > ), and the data is only stored in the local edge node and is prohibited from being uploaded to an insecure cloud server.

[0070] The encryption strength adaptively matches the data sensitivity to ensure the security of data transmission and storage at the construction site, and the number of illegal transmission events is zero.

[0071] Example 5: Risk Prediction and Active Warning for Oil and Gas Pipelines.

[0072] Scenario Description: Predict the failure risk of oil and gas pipelines based on historical data.

[0073] Step 1: Calculate the risk prediction value (threshold = 0.3).

[0074] Historical data: There were 5 anomalies in the past 24 hours, with severity levels of 2, 1, 3, 2, and 1 respectively.

[0075] Calculate the risk prediction value according to the formula : .

[0076] Step 2: Dynamically adjust the scoring weight.

[0077] Since exceeds the preset threshold , therefore, increase the calculation weight of the real-time importance score by 20% and trigger the alarm preferentially.

[0078] The accuracy of anomaly detection is improved, and the false alarm rate is reduced, realizing the upgrade from passive response to active prevention, early warning of potential failures, and thus avoiding the occurrence of safety accidents.

[0079] In summary, in Example 1, the present invention calculates the real-time importance score , dynamically determines the data priority, directly triggers the high-priority channel to generate control instructions, bypasses the cloud processing link, and thus can meet the immediate response requirements of the monitoring equipment for abnormal events at the oil and gas pipeline construction site, shorten the abnormal response time, and further significantly reduce the safety risks caused by delays.

[0080] In Example 2, the present invention dynamically compresses non-critical data based on the storage redundancy, and only retains the complete abnormal fragments and timestamps, thus avoiding the serious waste of resources caused by the full storage of high-frequency oil and gas pipeline data in the traditional scheme, further improving the steady-state data compression rate, reducing the overall storage cost, and at the same time enhancing the retrieval efficiency of key abnormal data.

[0081] In Example 3, the present invention calculates the resource occupancy coefficient , evaluates the node load in real time, and dynamically adjusts the data acquisition frequency of non-critical devices, thereby reducing the CPU occupancy rate of critical devices, while enhancing the network bandwidth utilization rate, ensuring the stable operation of the edge side, and further dynamically scheduling the edge resources to avoid the occurrence of device overload.

[0082] In Example 4, the present invention calculates the dynamic encryption label strength , adaptively adjust the encryption level and enforce local storage, thus avoiding the situation in the traditional solution where relying on cloud processing violates industry data security standards and it is difficult for low-power edge devices to achieve high-strength encryption. This enables the encryption strength to adaptively match the data sensitivity, thereby being able to pass industry security audits and reducing the number of illegal transmission events to zero.

[0083] In Embodiment 5, the present invention is based on the risk prediction value , predict the probability of device failure, and dynamically adjust the scoring weight, so as to improve the accuracy of device anomaly detection, and at the same time reduce the false alarm rate, and then achieve the upgrade from passive response to active prevention.

[0084] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and do not limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An integrated storage and use management method for an Internet of Things platform, characterized in that: The following steps are involved: S1. Real-time data collection: Real-time collection of equipment status data and environmental monitoring data at the oil and gas pipeline construction site. The equipment status data includes equipment operating parameters. , data generation frequency and network latency , the environmental monitoring data includes occupancy, memory usage, and network bandwidth; S2. Real-time importance score calculation and priority determination: According to the formula , calculate real-time importance scores for oil and gas pipeline construction site data , the real-time importance score With preset threshold Compare, if ≥ , it is marked as a critical data stream, and triggers a high-priority storage and real-time response channel to directly generate control instructions and send them to the construction site equipment. < , it is marked as a normal data stream and performs lightweight storage; S3, edge storage and business logic embedded processing: for common data streams, dynamically compress redundant data, perform high-intensity encryption on key data streams, and simultaneously detect abnormal data and generate risk prediction values; S4, Dynamic resource scheduling and compliance verification: Evaluate node load, dynamically reduce the frequency of non-critical equipment data collection, verify encryption tag strength, block illegal data transmission, and force local storage; S5. Control instruction delivery and feedback optimization: High-priority instructions are sent directly to devices through edge nodes, and thresholds are dynamically adjusted based on execution results. and , optimize the scoring model and periodically update the risk prediction value .

2. The method for integrated storage and use management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of dynamically compressing redundant data for the common data stream in step S3 is: According to the formula , calculate storage redundancy, dynamically compress non-critical data, discard redundant data, and only retain complete critical data and abnormal fragment timestamps.

3. The method for integrated storage and use management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of performing high-intensity encryption on the key data stream in step S3 is: Generate data encryption tags at the edge node and use the formula , calculate and generate the encryption tag strength , the encryption tag strength Dynamically adjust based on data sensitivity.

4. The method for integrated storage and use management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of synchronously detecting abnormal data and generating risk prediction values ​​in step S3 is: Real-time analysis of equipment parameters If the critical data identification threshold is exceeded , it is marked as abnormal; According to the formula , and calculate the risk prediction value by combining historical abnormal data ; like Exceeding the preset threshold , then increase the device's Calculate weights and trigger alarms first.

5. The method for integrated storage and use management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of evaluating the node load and dynamically reducing the frequency of non-critical equipment data in step S4 is: Calculate the resource occupancy coefficient according to the formula , evaluate edge node load ; like Exceeding the threshold , then reduce the data collection frequency of non-critical equipment.

6. The method for integrated storage and use management of the Internet of Things platform according to claim 1, characterized in that: The specific implementation method of verifying the encryption tag strength, blocking illegal data transmission, and forcing local storage in step S4 is: Verify whether the encryption tag strength L meets the industry security standards. If the verification fails, Does not meet the minimum value required by regulations , the transmission is blocked and a local alarm is triggered; If the verification passes, Meet the minimum value required by regulations , the data is only stored in the local edge node and is prohibited from being transmitted to the cloud.

7. An integrated storage and management system for an Internet of Things platform, characterized in that: The system includes a data acquisition module, an edge storage processing module, a business logic embedding module, a dynamic priority scheduling module and a compliance verification module; The data acquisition module is used to collect equipment status data and environmental monitoring data at the oil and gas pipeline construction site in real time; The edge storage processing module is used to synchronously execute lightweight business logic during data storage, including data compression, anomaly detection and encryption tag generation; The business logic embedding module is used to dynamically adjust the data processing strategy according to preset rules; The dynamic priority scheduling module is used to score according to real-time importance. Sort multi-device data and prioritize the highest-scoring data streams; The compliance verification module is used to ensure that data storage and processing comply with industry data security standards and prohibit the transmission of unencrypted data to the cloud.

8. The IoT platform storage and use integrated management system according to claim 7, characterized in that: The system also includes a processor, a memory and instructions stored therein, and the processor implements the steps of the method according to any one of claims 1 to 6 when executing the instructions.

Images