An analysis and early warning system and method based on pipeline network big data

Abstract

The present invention relates to the technical field of data processing, and discloses an analysis and early warning system and method based on pipeline network big data, wherein the system comprises: a data acquisition module, a model building module and a risk early warning module; the data acquisition module is configured to collect real-time data and crack images of each node of an oil and gas pipeline network; the model building module is configured to build a big data model of the oil and gas pipeline network, and the big data model is used to set an early warning threshold according to historical data; the risk early warning module is configured to analyze and evaluate the real-time data and crack images according to the big data model, and automatically generate an early warning notification when the real-time data exceeds the threshold, collect data of each node of the oil and gas pipeline network in real time through the data acquisition module, including crack images, equipment status and environmental factors, and use the big data model and risk assessment indicators to perform risk assessment and early warning.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an analysis and early warning system and method based on pipeline network big data. Background Art

[0002] After long-term use, pipelines are corroded by humid air, internal electrolytes and other substances, resulting in frequent accidents such as energy leakage and oil and gas pipeline explosions caused by defects such as corrosion, cracks or deformation, posing a huge threat to people's lives and property. Oil and gas pipeline leakage and explosion accidents not only seriously pollute the environment, but also cause casualties, causing immeasurable double harm. At present, many oil and gas pipelines in service in my country have entered a stage of fatigue and frequent accidents. In order to ensure the long-term safe operation of oil and gas pipelines, pipelines must be regularly inspected and maintained to ensure safe and effective operation of oil and gas pipelines during transportation. The determination of the degree of hazard of pipeline structural abnormalities and defects and the prediction of the remaining life of the pipeline are of great significance to the country's economic development and industrialization construction.

[0003] Traditional pipeline network analysis is expected to rely mainly on manual inspections and empirical judgments. This method is not only inefficient, but also difficult to monitor pipeline status in real time, and it is easy to miss potential risks. In addition, with the increase in data volume, manual analysis alone can no longer meet the processing needs of complex data. In-pipeline detection technology refers to the use of intelligent pipe cleaners equipped with non-destructive testing equipment and signal acquisition, processing and storage systems without affecting the normal and safe operation of the pipeline. The medium transported in the pipeline is used as the driving force to perform online detection of pipeline corrosion, deformation, and cracking, and to conduct risk analysis and early warning of the pipeline through in-pipeline detection data. Summary of the invention

[0004] In view of this, the present invention proposes an analysis and early warning system and method based on pipeline network big data, aiming to solve the problems in current technology of low efficiency of analysis and early warning, difficulty in real-time monitoring of pipeline status, and easy omission of potential risks.

[0005] On the one hand, the present invention proposes an analysis and early warning system based on pipeline network big data, comprising:

[0006] Data collection module, model building module and risk warning module;

[0007] The data acquisition module is configured to collect real-time data and crack images of each node of the oil and gas pipeline network;

[0008] The model building module is configured to build a big data model of the oil and gas pipeline network, and the big data model sets an early warning threshold based on historical data;

[0009] The risk warning module is configured to analyze and evaluate the real-time data and crack images according to the big data model, and automatically generate a warning notification when the real-time data exceeds a threshold.

[0010] Optionally, when collecting real-time data and crack images of each node of the oil and gas pipeline network, including:

[0011] Preprocessing the collected real-time data and crack image as original data and original image respectively;

[0012] The preprocessing of the real-time data includes data denoising and / or outlier detection and correction and / or data interpolation; the data denoising process is based on a filtering algorithm to remove noise interference in the data; the outlier detection and correction detects and corrects outliers in the data through statistical methods or machine learning algorithms; the data interpolation is to fill in missing data through an interpolation algorithm if there are missing values ​​in the real-time data;

[0013] The preprocessing of the crack image is to convert the original image into a grayscale image, and perform shape recognition and feature extraction on the grayscale image;

[0014] The preprocessed data and grayscale images are verified and calibrated to obtain the final data.

[0015] Optionally, when building a big data model of an oil and gas pipeline network, include:

[0016] The collected raw data and raw images as well as the final data and grayscale images obtained after preprocessing are stored in a database, wherein the raw data and raw images are stored for future use, and the final data and grayscale images are used to construct an oil and gas pipeline network database;

[0017] Select a big data framework to perform secondary processing on the final data and grayscale image, and send the secondary processed final data and grayscale image to the risk warning module;

[0018] The secondary processing includes final data processing and grayscale image processing, wherein the final data processing includes extracting feature information, and the feature information includes the number of historical failures of the equipment, the number of maintenance records and the use time, the soil pH value, the number of annual rainfalls and the magnitude of earthquakes;

[0019] The grayscale image processing includes determining the shadow area on the real-time grayscale image, and dividing all the shadow areas into regions to obtain a plurality of separate shadow areas, wherein there is no overlapping area between each separate shadow area;

[0020] Determine the pixel points of each individual shadow area, and calculate the shadow area factor of the individual shadow area based on all the pixel points;

[0021] All shadow area factors are extracted, and the shadow area degree values ​​of the real-time grayscale image are calculated according to the shadow area factors.

[0022] Optional, when the analysis is based on real-time data and crack images, including:

[0023] Determine a risk assessment index and a pipeline crack assessment index, determine the risk level according to the risk assessment index, and determine the pipeline cracking level according to the pipeline crack assessment index.

[0024] Optionally, when determining the risk level according to the risk assessment indicator, it includes:

[0025] Determine risk assessment indicators based on the final data, wherein the risk assessment indicators include health status indicators, equipment reliability indicators and environmental risk factors;

[0026] Assume R as risk level, H as health status index, C as equipment reliability index, and E as environmental factor index, and calculate the risk using the following formula:

[0027] ;

[0028] in, , , is the weight of the corresponding target;

[0029] The risk level standards are set in advance. After the risk level is determined according to the risk level, the warning threshold is set according to the historical data. When the real-time data exceeds the threshold, a warning notification is automatically generated.

[0030] Optionally, the equipment reliability index is determined by the number of historical failures, the number of maintenance records and the usage time of the equipment;

[0031] The environmental factor index is determined by soil pH value, annual rainfall number and earthquake magnitude;

[0032] The health status indicator is determined by the pipeline cracking level;

[0033] Among them, the environmental factor index E satisfies the following relationship:

[0034] ;

[0035] The equipment reliability index C satisfies the following relationship:

[0036] ;

[0037] Among them, R is the annual rainfall number, Q is the earthquake magnitude, F is the number of historical failures, M is the number of maintenance records, and U is the usage time. , , , , is the weight of the corresponding target.

[0038] Optionally, when determining the pixel points of each individual shadow area and calculating the shadow area factor of the individual shadow area based on all the pixel points, it includes:

[0039] Extract all pixel points and construct a pixel value set based on the pixel values ​​corresponding to all pixel points;

[0040] Determine the cluster center of the pixel value set based on the K-Means clustering algorithm;

[0041] Determine the pixel distance from each pixel value in the pixel value set to the cluster center respectively, and construct a pixel distance set;

[0042] Calculate the shadow area factor of the individual shadow areas based on the set of pixel distances:

[0043] ;

[0044] Among them, G is the shadow area factor of the individual shadow area, c1 is the weight of the individual shadow area, n is the number of pixel distances in the pixel distance set, e i is the i-th pixel distance in the pixel distance set,

[0045] e 均 is the average value of all pixel distances in the pixel distance set, h i is the weight corresponding to the i-th pixel distance.

[0046] Optionally, when all shadow area factors are extracted and the shadow area degree value of the real-time grayscale image is calculated according to the shadow area factors, it includes:

[0047] Obtaining a preset shadow area factor, and generating a first coding mark for all shadow area factors that are smaller than the preset shadow area factor;

[0048] Generating a second coding mark from all shadow area factors that are equal to the preset shadow area factor;

[0049] Generating a third coding mark for all shadow area factors greater than the preset shadow area factor;

[0050] Count the number of the first coding marks, the number of the second coding marks, and the number of the third coding marks, and calculate the shadow area degree value of the real-time grayscale image:

[0051] ;

[0052] Wherein, K is the shadow area degree value of the real-time grayscale image, L1 is the number of first coding marks, L2 is the number of second coding marks, and L3 is the number of third coding marks.

[0053] Optionally, when judging the cracking level of the pipeline according to the degree value of the shadow area, it includes:

[0054] A first shadow area size K1, a second shadow area size K2, and a third shadow area size K3 are preset, and K1 is smaller than K2 and smaller than K3;

[0055] When the shadow area degree value K is less than K1, it is judged that there is no cracking risk and is determined to be a fourth-level cracking level;

[0056] When the shadow area degree value K is greater than K1 and less than K2, it is judged as a micro-cracking risk and determined to be a third-level cracking level;

[0057] When the shadow area degree value K is greater than K2 and less than K3, it is judged to be a medium cracking risk and determined to be a secondary cracking level;

[0058] When the shadow area degree value K is greater than K3, it is judged as a strong cracking risk and is determined to be a first-level cracking level.

[0059] On the other hand, the present invention also proposes a method for applying to the above-mentioned analysis and early warning system based on pipeline network big data, the method comprising:

[0060] Collect real-time data and crack images from each node of the oil and gas pipeline network;

[0061] Building a big data model of the oil and gas pipeline network, wherein the big data model sets warning thresholds based on historical data;

[0062] The real-time data and crack images are analyzed and evaluated according to the big data model, and when the real-time data exceeds a threshold, an early warning notification is automatically generated.

[0063] Compared with the prior art, the beneficial effect of the present invention is that the analysis and early warning system based on pipeline network big data of the present invention collects data of each node of the oil and gas pipeline network in real time through the data acquisition module, including crack images, equipment status and environmental factors, and uses big data models and risk assessment indicators to perform risk assessment and early warning. The system can dynamically adjust the threshold according to real-time data and historical data, and quickly issue early warning notifications when potential risks occur in the pipeline, which greatly improves the accuracy and timeliness of risk detection and reduces the probability of potential pipeline accidents. The system comprehensively considers the health status of the pipeline, equipment reliability and environmental risk factors, and integrates and analyzes these multi-dimensional data through big data technology. Compared with traditional single data sources or manual detection methods, the system can provide more comprehensive risk assessment and prediction, accurately judge the degree of pipeline cracking and potential hidden dangers, and effectively improve the scientificity and effectiveness of pipeline safety management. Moreover, the design of the system takes into account the automated operation of data acquisition and preprocessing, including data denoising, outlier detection and correction, data interpolation and image grayscale processing, which greatly reduces the workload of manual processing and improves the efficiency of data processing. At the same time, the system's data integration, secondary processing and calibration ensure the accuracy and completeness of the final data, providing a reliable data basis for subsequent risk assessment and decision support. The system has a flexible architecture and can adjust risk assessment indicators and warning thresholds according to the actual needs of different oil and gas pipelines. It can also easily integrate new data sources or analysis modules to adapt to the complex and changing pipeline operation environment. This scalability enables the system to be continuously optimized and always maintain efficient management and early warning capabilities for pipeline risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0065] Figure 1 A schematic diagram of an analysis and early warning system based on pipeline network big data provided by an embodiment of the present invention;

[0066] Figure 2 A flowchart of an analysis and early warning method based on pipeline network big data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0068] like Figure 1 As shown, it is a schematic diagram of an analysis and early warning system based on pipeline network big data provided by an embodiment of the present invention, wherein the system includes:

[0069] Data collection module, model building module and risk warning module;

[0070] The data acquisition module is configured to collect real-time data and crack images of each node of the oil and gas pipeline network;

[0071] The model building module is configured to build a big data model of the oil and gas pipeline network, and the big data model sets an early warning threshold based on historical data;

[0072] The risk warning module is configured to analyze and evaluate the real-time data and crack images according to the big data model, and automatically generate a warning notification when the real-time data exceeds a threshold.

[0073] In this preferred embodiment, when collecting real-time data and crack images of each node of the oil and gas pipeline network, it includes:

[0074] Preprocessing the collected real-time data and crack image as original data and original image respectively;

[0075] The preprocessing of the real-time data includes data denoising and / or outlier detection and correction and / or data interpolation; the data denoising process is based on a filtering algorithm to remove noise interference in the data; the outlier detection and correction detects and corrects outliers in the data through statistical methods or machine learning algorithms; the data interpolation is to fill in missing data through an interpolation algorithm if there are missing values ​​in the real-time data;

[0076] The preprocessing of the crack image is to convert the original image into a grayscale image, and perform shape recognition and feature extraction on the grayscale image;

[0077] The preprocessed data and grayscale images are verified and calibrated to obtain the final data.

[0078] It can be understood that the data acquisition module is responsible for collecting real-time data and crack images from various nodes of the oil and gas pipeline network. These data are the basis for subsequent analysis and model building. The model building module is used to build a big data model of the oil and gas pipeline network. The model can integrate multi-source data to provide support for risk assessment; the risk warning module performs risk detection and assessment on real-time data and crack images. This module also sets warning thresholds and automatically generates warning notifications when the risk exceeds the threshold.

[0079] In this preferred embodiment, the big data model for constructing the oil and gas pipeline network specifically includes:

[0080] The collected raw data and raw images as well as the final data and grayscale images obtained after preprocessing are stored in a database, wherein the raw data and raw images are stored for future use, and the final data and grayscale images are used to construct an oil and gas pipeline network database;

[0081] Select a big data framework to perform secondary processing on the final data and grayscale image, and send the secondary processed final data and grayscale image to the risk warning module;

[0082] The secondary processing includes final data processing and grayscale image processing, wherein the final data processing includes extracting feature information, and the feature information includes the number of historical failures of the equipment, the number of maintenance records and the use time, the soil pH value, the number of annual rainfalls and the magnitude of earthquakes;

[0083] The grayscale image processing includes determining the shadow area on the real-time grayscale image, and dividing all the shadow areas into regions to obtain a plurality of separate shadow areas, wherein there is no overlapping area between each separate shadow area;

[0084] Determine the pixel points of each individual shadow area, and calculate the shadow area factor of the individual shadow area based on all the pixel points;

[0085] All shadow area factors are extracted, and the shadow area degree values ​​of the real-time grayscale image are calculated according to the shadow area factors.

[0086] In this preferred embodiment, when analyzing and evaluating based on real-time data and crack images, it includes:

[0087] Determine a risk assessment index and a pipeline crack assessment index, determine the risk level according to the risk assessment index, and determine the pipeline cracking level according to the pipeline crack assessment index.

[0088] It can be understood that in this preferred embodiment, a big data model of an oil and gas pipeline network is constructed to achieve effective monitoring, risk detection and assessment of oil and gas pipelines. The data after the initial processing and analysis of the original data and the original image include data cleaning, outlier detection, etc. The image obtained after image processing technology (such as denoising, enhancement, etc.) is used for subsequent analysis. The above-mentioned various data are stored in a centralized database to form an oil and gas pipeline network database. This database will become the basis for subsequent analysis and decision-making, ensuring unified management and access to data. Select a suitable big data framework (such as Hadoop, Spark, etc.) to perform secondary processing on the integrated data. These frameworks have the ability to process large-scale data and can efficiently complete data calculation and analysis tasks. A more in-depth analysis is performed on the final data and grayscale images, such as feature extraction, pattern recognition, etc., to extract useful information and further improve the availability and accuracy of the data. The final data and grayscale images obtained after the secondary processing will be sent to the risk warning module to provide basic data for subsequent risk detection and assessment. In risk detection, a series of risk assessment indicators need to be determined, which involve the health status of the pipeline, equipment reliability, and environmental factors, etc., aiming to fully reflect the risk level of the pipeline. Pipeline crack assessment indicators focus on assessing the physical state of the pipeline, usually including the detection and analysis of cracks. These indicators provide a direct basis for judging the safety of the pipeline. Based on the determined risk assessment indicators, the system will calculate a comprehensive risk level. This level can reflect the current safety status and potential risks of the pipeline. Relying on the pipeline crack assessment indicators, the system assesses the cracking level of the pipeline. This assessment helps to identify and monitor the physical defects of the pipeline, thereby ensuring the safe operation of the pipeline.

[0089] In this way, by building a big data model of the oil and gas pipeline network, real-time monitoring and risk assessment of pipeline status are achieved. By centrally storing raw data and processed data and using the big data framework for effective processing, the system can identify potential risks in a timely manner. In addition, the combination of risk assessment and pipeline crack assessment provides a scientific basis for pipeline management and ensures the safety and reliability of oil and gas pipelines. This comprehensive management model improves the maintenance efficiency and emergency response capabilities of oil and gas pipelines.

[0090] Furthermore, a first risk level standard, a second risk level standard and a third risk level standard are pre-set, and when the risk level is less than or equal to the first risk level standard, the risk level is determined to be a first risk level;

[0091] When the risk level is less than or equal to the second risk level standard and greater than the first risk level standard, the risk level is determined to be the second risk level;

[0092] When the risk level is less than or equal to the third risk level standard and greater than the second risk level standard, the risk level is determined to be the third risk level;

[0093] When the risk level is level 3, if the real-time data is judged to be ≥ the warning threshold, a warning notification is automatically generated;

[0094] The warning threshold is set according to the historical data, and 90% of the maximum value in the historical data is selected as the warning threshold.

[0095] In this preferred embodiment, determining the risk level according to the risk assessment indicator specifically includes:

[0096] Determine risk assessment indicators based on the final data, wherein the risk assessment indicators include health status indicators, equipment reliability indicators and environmental risk factors;

[0097] Assume R as risk level, H as health status index, C as equipment reliability index, and E as environmental factor index, and calculate the risk using the following formula:

[0098] ;

[0099] in, , , is the weight of the corresponding target;

[0100] The risk level standards are set in advance. After the risk level is determined according to the risk level, the warning threshold is set according to the historical data. When the real-time data exceeds the threshold, a warning notification is automatically generated.

[0101] In this preferred embodiment, the equipment reliability index is determined by the number of historical failures, maintenance records and usage time of the equipment;

[0102] The environmental factor index is determined by soil pH value, annual rainfall number and earthquake magnitude;

[0103] The health status indicator is determined by the pipeline cracking level;

[0104] Among them, the environmental factor index E satisfies the following relationship:

[0105] ;

[0106] The equipment reliability index C satisfies the following relationship:

[0107] ;

[0108] Among them, R is the annual rainfall number, Q is the earthquake magnitude, F is the number of historical failures, M is the number of maintenance records, and U is the usage time. , , , , is the weight of the corresponding target.

[0109] It is understandable that the key to risk assessment is to integrate multiple indicators to determine the risk level of the pipeline or equipment. The final data is obtained through various monitoring and evaluation methods, including real-time monitoring data, historical data, equipment status data, etc. The health status indicator (H) reflects the current health of the pipeline, which is usually obtained by evaluating factors such as the cracking level of the pipeline. The equipment reliability indicator (C) indicates the reliability of the equipment under certain conditions, which is determined based on data such as the historical failure record, maintenance record and service life of the equipment. Environmental risk factors (E) involve the impact of the external environment on the safety of equipment or pipelines, including soil type, weather conditions (such as rainfall, temperature changes) and seismic activity. According to the calculated risk level R, the system will define different risk levels (such as low risk, medium risk, high risk, etc.), which helps managers quickly understand the risk status. The system will also set warning thresholds based on historical data and industry standards. The setting of thresholds can be based on past event analysis to ensure timely response to potential risks. When the real-time monitoring data exceeds the set warning threshold, the system will automatically generate a warning notification to remind relevant personnel to conduct necessary inspections and responses. This mechanism can effectively reduce the occurrence of risk events and ensure the safety of pipelines and equipment. Equipment reliability data includes historical failure data, maintenance records, and service life information. These data are key factors in determining the long-term stability of equipment and can reveal potential risks of equipment. Environmental factor data: including soil type, weather conditions, and seismic activity information. The impact of environmental factors on pipelines cannot be ignored, especially when extreme weather or natural disasters occur. Understanding these factors helps to fully assess risks.

[0110] By comprehensively considering health status, equipment reliability and environmental risk factors, a comprehensive risk assessment of pipelines or equipment is achieved. The calculated risk level provides a basis for risk management, and the set early warning mechanism enhances the system's responsiveness and enables appropriate preventive measures to be taken before potential risks occur.

[0111] It is understood that the cracking level assessment of the pipeline is based on the shadow area and its characteristics in the real-time grayscale image. The image is captured by the monitoring equipment and reflects the current state of the pipeline surface, where the shadow area may refer to cracks or other defects. The shadow area in the image is divided into multiple non-overlapping individual shadow areas. This step ensures that each shadow area is independent and convenient for subsequent analysis and calculation. For each individual shadow area, the system extracts all the pixels in it. The gray value of each pixel reflects the characteristics of the area. Combining all the extracted pixels, the system calculates the shadow area factor of each shadow area. This factor is a quantitative indicator that reflects the intensity or severity of the shadow area and can reflect the potential cracking risk. The factors of all individual shadow areas are collected to prepare for the subsequent comprehensive assessment. Based on the extracted shadow area factors, the system calculates the overall shadow area degree value of the real-time grayscale image. This value can be regarded as a comprehensive assessment of the surface state of the pipeline. A higher degree value may indicate a greater potential cracking risk. Finally, based on the calculated shadow area degree value, the system judges the cracking level of the pipeline. This judgment is usually divided into multiple levels (such as no risk, micro-cracking, moderate cracking, severe cracking, etc.), which helps managers quickly understand the safety status of the pipeline.

[0112] In this way, the cracking level of the pipeline is evaluated by systematically analyzing the shadow area in the real-time grayscale image using pixel information and statistical methods. This process ensures the accuracy and timeliness of the evaluation, allowing relevant managers to respond quickly and take necessary maintenance measures to ensure the safe operation of the pipeline. Overall, the use of shadow area factors as evaluation indicators provides a scientific basis for pipeline condition monitoring and risk management.

[0113] In this preferred embodiment, the step of determining the pixel points of each separate shadow area and calculating the shadow area factor of the separate shadow area based on all the pixel points includes:

[0114] Extract all pixel points and construct a pixel value set based on the pixel values ​​corresponding to all pixel points;

[0115] Determine the cluster center of the pixel value set based on the K-Means clustering algorithm;

[0116] Determine the pixel distance from each pixel value in the pixel value set to the cluster center respectively, and construct a pixel distance set;

[0117] Calculate the shadow area factor of the individual shadow areas based on the set of pixel distances:

[0118] ;

[0119] Among them, G is the shadow area factor of the individual shadow area, c1 is the weight of the individual shadow area, n is the number of pixel distances in the pixel distance set, e i is the i-th pixel distance in the pixel distance set,

[0120] e 均 is the average value of all pixel distances in the pixel distance set, h i is the weight corresponding to the i-th pixel distance;

[0121] When all shadow area factors are extracted and the shadow area degree value of the real-time grayscale image is calculated according to the shadow area factors, it includes:

[0122] Obtaining a preset shadow area factor, and generating a first coding mark for all shadow area factors that are smaller than the preset shadow area factor;

[0123] Generating a second coding mark from all shadow area factors that are equal to the preset shadow area factor;

[0124] Generating a third coding mark for all shadow area factors greater than the preset shadow area factor;

[0125] Count the number of the first coding marks, the number of the second coding marks, and the number of the third coding marks, and calculate the shadow area degree value of the real-time grayscale image:

[0126] ;

[0127] Wherein, K is the shadow area degree value of the real-time grayscale image, L1 is the number of first coding marks, L2 is the number of second coding marks, and L3 is the number of third coding marks.

[0128] It can be understood that the system accurately analyzes the shadow area in the image, calculates the shadow area factors using pixels and clustering algorithms, and evaluates the shadow area degree value of the real-time grayscale image based on these factors. The following is an in-depth understanding of this process: Through image processing technology, the system extracts all relevant pixels in the shadow area of ​​the image. These pixels contain various grayscale values, representing the intensity and characteristics of the shadow area. The extracted pixels are sorted into a set for subsequent analysis. This set contains all pixel values ​​related to the shadow area. The K-Means clustering algorithm is used to analyze the set of pixel values ​​to determine the cluster center. The cluster center represents the most representative value in the pixel value, helping to identify different shadow intensities or characteristics. The system calculates the distance from each pixel value to the cluster center to form a set of pixel distances. These distances reflect the similarities and differences between each pixel value and the cluster center.

[0129] In this preferred embodiment, judging the cracking level of the pipeline according to the degree value of the shadow area specifically includes:

[0130] A first shadow area size K1, a second shadow area size K2, and a third shadow area size K3 are preset, and K1 is smaller than K2 and smaller than K3;

[0131] When the shadow area degree value K is less than K1, it is judged that there is no cracking risk and is determined to be a fourth-level cracking level;

[0132] When the shadow area degree value K is greater than K1 and less than K2, it is judged as a micro-cracking risk and determined to be a third-level cracking level;

[0133] When the shadow area degree value K is greater than K2 and less than K3, it is judged to be a medium cracking risk and determined to be a secondary cracking level;

[0134] When the shadow area degree value K is greater than K3, it is judged as a strong cracking risk and is determined to be a first-level cracking level.

[0135] It can be understood that, in this embodiment, the judgment of the pipeline cracking level is classified based on the size of the shadow area degree value (K), and three preset shadow area sizes (K1, K2 and K3) are specifically defined, and correspond to different cracking risk levels respectively.

[0136] The degree value (K) of the shaded area is calculated by analyzing the pipeline monitoring image, which reflects the severity of possible cracks or openings on the pipeline. The shaded area usually refers to the area or intensity occupied by the crack in the image. The larger the value, the higher the possibility or severity of the opening. Preset the size of the shaded area (K1, K2, K3): K1 represents a smaller threshold for judging whether there is a risk of cracking in the pipeline. If the degree value of the shaded area is less than K1, it indicates that the pipeline is in a safe state. K2 represents a medium threshold, which is between no risk and higher risk, and is used to judge the risk of micro-cracking. K3 represents a larger threshold, which is used as the standard for judging the risk of severe cracking. The purpose of setting these three thresholds is to form a progressive risk assessment system, so that different degree values of the shaded area can correspond to different cracking levels. Four levels of cracking: When K < K1, it is judged that there is no risk of cracking, indicating that the condition of the pipeline is good and no special maintenance or inspection is required. Level 3 cracking: When K is between K1 and K2, it is judged that there is a risk of micro-cracking, indicating that the pipeline may have slight cracks and needs to be monitored regularly to observe whether the situation deteriorates. Level 2 cracking: When K is between K2 and K3, it is judged that there is a risk of medium cracking, which means that the pipeline has medium-level cracks and may require repair or more detailed assessment. Level 1 cracking: When K > K3, it is judged that there is a risk of strong cracking, indicating that the pipeline may have serious safety hazards and immediate measures need to be taken to prevent accidents.

[0137] In summary, the analysis and early warning system based on pipeline network big data of the present invention collects data of each node of the oil and gas pipeline network in real time through the data acquisition module, including crack images, equipment status and environmental factors, and uses big data models and risk assessment indicators for risk assessment and early warning. The system can dynamically adjust the threshold according to real-time data and historical data, and quickly issue early warning notifications when potential risks occur in the pipeline, which greatly improves the accuracy and timeliness of risk detection and reduces the probability of potential pipeline accidents. The system comprehensively considers the health status of the pipeline, equipment reliability and environmental risk factors, and integrates and analyzes these multi-dimensional data through big data technology. Compared with traditional single data sources or manual detection methods, the system can provide more comprehensive risk assessment and prediction, accurately judge the degree of pipeline cracking and potential hidden dangers, and effectively improve the scientificity and effectiveness of pipeline safety management. Using data visualization technology, the system provides managers with an intuitive comprehensive dashboard that displays real-time pipeline status, risk level, alarm status and historical data trends, helping managers to quickly understand complex data and make scientific decisions. Moreover, the system design takes into account the automation of data collection and preprocessing, including data denoising, outlier detection and correction, data interpolation, and image grayscale processing, which greatly reduces the workload of manual processing and improves the efficiency of data processing. At the same time, the system's integration, secondary processing, and calibration of data ensure the accuracy and completeness of the final data, providing a reliable data foundation for subsequent risk assessment and decision support. The system has a flexible architecture and can adjust risk assessment indicators and warning thresholds according to the actual needs of different oil and gas pipelines. It can also easily integrate new data sources or analysis modules to adapt to the complex and changing pipeline operating environment. This scalability enables the system to be continuously optimized and always maintain efficient management and early warning capabilities for pipeline risks.

[0138] like Figure 2 As shown, an embodiment of the present invention provides an analysis and early warning method based on pipeline network big data, including:

[0139] Collect real-time data and crack images from each node of the oil and gas pipeline network;

[0140] Building a big data model of the oil and gas pipeline network, wherein the big data model sets warning thresholds based on historical data;

[0141] The real-time data and crack images are analyzed and evaluated according to the big data model, and when the real-time data exceeds a threshold, an early warning notification is automatically generated.

[0142] It is understandable that the above method collects real-time data from various nodes of the oil and gas pipeline network, including node status, equipment reliability data, environmental factor data and original images. The collected data is preprocessed by noise removal, outlier correction and missing value filling to ensure that the data is accurate and reliable, and finally forms an integrated data set. The preprocessed data and images are stored in the oil and gas pipeline network database, and the big data framework is selected for secondary processing of the data. This processing ensures the quality and availability of the data, and finally the processed data is sent to the risk warning module for subsequent analysis as needed. Based on the integrated data, the system calculates health status indicators, equipment reliability indicators and environmental risk factors to determine the comprehensive risk level and set the risk level. The system also generates risk reports to achieve early warning, and automatically sends warning notifications when real-time data exceeds the preset threshold, prompting relevant personnel to take countermeasures.

[0143] In summary, the analysis and early warning system based on pipeline network big data of the present invention collects data of each node of the oil and gas pipeline network in real time through the data acquisition module, including crack images, equipment status and environmental factors, and uses big data models and risk assessment indicators for risk assessment and early warning. The system can dynamically adjust the threshold according to real-time data and historical data, and quickly issue early warning notifications when potential risks occur in the pipeline, which greatly improves the accuracy and timeliness of risk detection and reduces the probability of potential pipeline accidents. The system comprehensively considers the health status of the pipeline, equipment reliability and environmental risk factors, and integrates and analyzes these multi-dimensional data through big data technology. Compared with traditional single data sources or manual detection methods, the system can provide more comprehensive risk assessment and prediction, accurately judge the degree of pipeline cracking and potential hidden dangers, and effectively improve the scientificity and effectiveness of pipeline safety management. Using data visualization technology, the system provides managers with an intuitive comprehensive dashboard that displays real-time pipeline status, risk level, alarm status and historical data trends, helping managers to quickly understand complex data and make scientific decisions. Moreover, the system design takes into account the automation of data collection and preprocessing, including data denoising, outlier detection and correction, data interpolation, and image grayscale processing, which greatly reduces the workload of manual processing and improves the efficiency of data processing. At the same time, the system's integration, secondary processing, and calibration of data ensure the accuracy and completeness of the final data, providing a reliable data foundation for subsequent risk assessment and decision support. The system has a flexible architecture and can adjust risk assessment indicators and warning thresholds according to the actual needs of different oil and gas pipelines. It can also easily integrate new data sources or analysis modules to adapt to the complex and changing pipeline operating environment. This scalability enables the system to be continuously optimized and always maintain efficient management and early warning capabilities for pipeline risks.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An analysis and early warning system based on pipeline network big data, characterized in that: include: Data collection module, model building module and risk warning module; The data acquisition module is configured to collect real-time data and crack images of each node of the oil and gas pipeline network; The model building module is configured to build a big data model of the oil and gas pipeline network, and the big data model is used to set an early warning threshold based on historical data; The risk warning module is configured to analyze and evaluate the real-time data and crack images according to the big data model, and automatically generate a warning notification when the real-time data exceeds a threshold; When collecting real-time data and crack images from various nodes in the oil and gas pipeline network, including: Preprocessing the collected real-time data and crack image as original data and original image respectively; The preprocessing of the real-time data includes data denoising and / or outlier detection and correction and / or data interpolation; the data denoising process is based on a filtering algorithm to remove noise interference in the data; the outlier detection and correction detects and corrects outliers in the data through statistical methods or machine learning algorithms; the data interpolation is to fill in missing data through an interpolation algorithm if there are missing values ​​in the real-time data; The preprocessing of the crack image is to convert the original image into a grayscale image, and perform shape recognition and feature extraction on the grayscale image; Verify and calibrate the preprocessed data and grayscale images to obtain the final data; When building a big data model for an oil and gas pipeline network, it includes: The collected raw data and raw images as well as the final data and grayscale images obtained after preprocessing are stored in a database, wherein the raw data and raw images are stored for future use, and the final data and grayscale images are used to construct an oil and gas pipeline network database; Select a big data framework to perform secondary processing on the final data and grayscale image, and send the secondary processed final data and grayscale image to the risk warning module; The secondary processing includes final data processing and grayscale image processing, wherein the final data processing includes extracting feature information, and the feature information includes the number of historical failures of the equipment, the number of maintenance records and the use time, the soil pH value, the number of annual rainfalls and the magnitude of earthquakes; The grayscale image processing includes determining the shadow area on the real-time grayscale image, and dividing all the shadow areas into regions to obtain a plurality of separate shadow areas, wherein there is no overlapping area between each separate shadow area; Determine the pixel points of each individual shadow area, and calculate the shadow area factor of the individual shadow area based on all the pixel points; All shadow area factors are extracted, and the shadow area degree values ​​of the real-time grayscale image are calculated according to the shadow area factors.

2. The analysis and early warning system based on pipeline network big data according to claim 1 is characterized in that: When analyzing and evaluating based on real-time data and crack images, it includes: Determine a risk assessment index and a pipeline crack assessment index, determine the risk level according to the risk assessment index, and determine the pipeline cracking level according to the pipeline crack assessment index.

3. The analysis and early warning system based on pipeline network big data according to claim 2 is characterized in that: When determining the risk level based on the risk assessment indicators, it includes: Determine risk assessment indicators based on the final data, wherein the risk assessment indicators include health status indicators, equipment reliability indicators and environmental risk factors; Assume R as risk level, H as health status index, C as equipment reliability index, and E as environmental factor index, and calculate the risk using the following formula: ； in, , , is the weight of the corresponding target; The risk level standards are set in advance. After the risk level is determined according to the risk level, the warning threshold is set according to the historical data. When the real-time data exceeds the warning threshold, a warning notification is automatically generated.

4. The analysis and early warning system based on pipeline network big data according to claim 3 is characterized in that: The equipment reliability index is determined by the number of historical failures, maintenance records and usage time of the equipment; The environmental factor index is determined by soil pH value, annual rainfall number and earthquake magnitude; The health status indicator is determined by the pipeline cracking level; Among them, the environmental factor index E satisfies the following relationship: ； Among them, R is the annual rainfall number, Q is the earthquake magnitude, pH is the soil pH value, , , is the weight of the corresponding target; The equipment reliability index C satisfies the following relationship: ； Among them, F is the number of historical failures, M is the number of maintenance records, and U is the usage time. , is the weight of the corresponding target.

5. The analysis and early warning system based on pipeline network big data according to claim 4 is characterized in that: When determining the pixel points of each individual shadow area and calculating the shadow area factor of the individual shadow area based on all the pixel points, including: Extract all pixel points and construct a pixel value set based on the pixel values ​​corresponding to all pixel points; Determine the cluster center of the pixel value set based on the K-Means clustering algorithm; Determine the pixel distance from each pixel value in the pixel value set to the cluster center respectively, and construct a pixel distance set; Calculate the shadow area factor of the individual shadow areas based on the set of pixel distances: ； Where G is the shadow area factor of the individual shadow area, c1 is the weight of the individual shadow area, n is the number of pixel distances in the pixel distance set, e i is the i-th pixel distance in the pixel distance set, e 均 is the average value of all pixel distances in the pixel distance set, h i is the weight corresponding to the i-th pixel distance.

6. The analysis and early warning system based on pipeline network big data according to claim 5 is characterized in that: When all shadow area factors are extracted and the shadow area degree value of the real-time grayscale image is calculated according to the shadow area factors, it includes: Obtaining a preset shadow area factor, and generating a first coding mark for all shadow area factors that are smaller than the preset shadow area factor; Generating a second coding mark from all shadow area factors that are equal to the preset shadow area factor; Generating a third coding mark for all shadow area factors greater than the preset shadow area factor; Count the number of the first coding marks, the number of the second coding marks, and the number of the third coding marks, and calculate the shadow area degree value of the real-time grayscale image: ； Wherein, K is the shadow area degree value of the real-time grayscale image, L1 is the number of first coding marks, L2 is the number of second coding marks, and L3 is the number of third coding marks.

7. The analysis and early warning system based on pipeline network big data according to claim 6 is characterized in that: When judging the cracking level of the pipeline according to the degree value of the shadow area, it includes: A first shadow area size K1, a second shadow area size K2, and a third shadow area size K3 are preset, and K1 is smaller than K2 and smaller than K3; When the shadow area degree value K is less than K1, it is judged that there is no cracking risk and is determined to be a fourth-level cracking level; When the shadow area degree value K is greater than K1 and less than K2, it is judged as a micro-cracking risk and determined to be a third-level cracking level; When the shadow area degree value K is greater than K2 and less than K3, it is judged to be a medium cracking risk and determined to be a secondary cracking level; When the shadow area degree value K is greater than K3, it is judged as a strong cracking risk and is determined to be a first-level cracking level.

8. An analysis and early warning method based on pipeline network big data, applied to the analysis and early warning system based on pipeline network big data according to any one of claims 1 to 7, characterized in that: The method comprises: Collect real-time data and crack images from each node of the oil and gas pipeline network; Building a big data model of the oil and gas pipeline network, wherein the big data model sets warning thresholds based on historical data; The real-time data and crack images are analyzed and evaluated according to the big data model, and when the real-time data exceeds a threshold, an early warning notification is automatically generated.

Images