Processing method and system for pipeline full life cycle data based on data lake

Abstract

The present invention discloses a method and system for processing pipeline life cycle data based on a data lake, wherein the method comprises: establishing the data lake based on a cloud platform, storing the full life cycle data of girth welds of oil and gas pipelines in the data lake; classifying and marking the full life cycle data of girth welds of oil and gas pipelines; purifying and verifying the collected girth weld data to ensure the integrity and accuracy of the data; using blockchain technology to store and verify the integrity and credibility of girth weld data; using smart contracts to manage and verify girth weld data; establishing a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data. The present invention realizes the digitization of business objects, business rules, and business processes; realizes digital guidance of business processes, improves work efficiency, and ensures risk prediction and pre-control.

Description

Technical Field

[0001] The present invention belongs to the field of production and operation and maintenance of oil and gas storage and transportation industry, and in particular, to a method and system for processing pipeline full life cycle data based on a data lake. Background Art

[0002] In all aspects of the entire life cycle of oil and gas pipelines, such as procurement, construction, operation and maintenance, the pipeline data involved has the characteristics of huge data volume, diverse data types, and various data forms, covering multiple stages of the entire life cycle of oil and gas pipeline design, procurement, construction and operation. The missing, misplaced, and inaccurate data have brought great impacts and difficulties to the safety management of oil and gas pipelines. Therefore, there is an urgent need for a new data management model to establish an oil and gas pipeline data lake and improve the utilization of data to achieve comprehensive, accurate, and efficient data management. Summary of the invention

[0003] In view of the defects existing in the above-mentioned prior art, the present invention provides a processing and evaluation method for pipeline full life cycle data based on a data lake, comprising the following steps:

[0004] Step S101, establishing the data lake based on the cloud platform, wherein the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form;

[0005] Step S103, classifying and marking the oil and gas pipeline girth weld full life cycle data to identify and distinguish different types of data, and adding corresponding tags or metadata to each data category;

[0006] Step S105, purifying and verifying the collected girth weld data to ensure the integrity and accuracy of the data;

[0007] Step SS107: Using blockchain technology to store and verify the integrity and credibility of girth weld data;

[0008] Step S109: Using smart contracts to manage and verify girth weld data;

[0009] Step S1011: Establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data.

[0010] Wherein, the step S101 includes:

[0011] Create data lake storage on the cloud platform;

[0012] Creating a data table or collection in the data lake to store the full life cycle data of the girth weld;

[0013] Create a form interface for inputting girth weld life cycle data;

[0014] Integrate the form input interface with the data lake to ensure that the data entered in the form can be directly stored in the corresponding table or collection in the data lake.

[0015] Wherein, creating a data table or a collection in the data lake includes:

[0016] Use postgreSQL to establish a database, construct a data table according to the logical model and data dictionary, and determine the name of the data table, the field name and data type contained in the data table, and the data subset of the data lake includes a feature class data subset, a relationship class data subset and a geometry class subset;

[0017] With the relevant data fields of pile data, coordinates and weld numbers as the main line and the girth weld as the base, the relationship between the data is established.

[0018] The data fields of pile data, coordinates and weld numbers are taken as the main line, and the girth weld is taken as the base to establish the association relationship between the data, including:

[0019] Establish the connection between pile data and coordinates, which determines the specific location of the pile by associating the coordinate fields in the pile data with the geographic coordinates, so as to analyze the distribution, distance relationship and spatial area of ​​the piles in the geographic space;

[0020] Establishing a connection between pile data and weld numbers, by associating the weld numbers with pile data, determining the pile corresponding to each weld, so as to query, retrieve and update the pile data related to a specific weld according to the weld numbers;

[0021] Establishing a connection between the weld number and the weld data, by using the weld number as a primary key to associate the weld data with the weld, so as to query, retrieve and update the weld data related to a specific weld according to the weld number;

[0022] Establish a connection between weld data and pile data by matching the location field in the weld data with the pile data to determine the specific location of the weld, and perform geospatial analysis and location query.

[0023] The cloud platform includes Amazon Web Services (AWS), Microsoft Azure or Google Cloud Platform, and the cloud storage service includes Amazon S3, Azure Blob Storage or Google Cloud Storage.

[0024] Wherein, the step S103 includes:

[0025] Data classification includes welding parameter data, material data, inspection data, welding process monitoring data and welding quality assessment data;

[0026] For welding parameter data, the parameter values ​​of welding current, welding voltage and welding speed are marked; for material data, the pipe material type, material batch and specification are marked;

[0027] Add metadata for each data category, including relevant information and properties of the data.

[0028] Wherein, the step S105 includes: data cleaning, data deduplication, and data format verification to remove duplicate, incomplete or erroneous data and ensure that the data complies with the agreed format and specifications.

[0029] Specifically, step S105 may be data cleaning and preprocessing to ensure the quality and consistency of the data, which may include processing missing values, removing duplicate data, correcting format errors, etc. Data cleaning techniques and algorithms are used to clean and purify the data.

[0030] Wherein, the step S107 includes: each girth weld data can be used as a transaction record, stored in the form of a blockchain, and the consistency and non-tamperability of the data are ensured through a distributed consensus algorithm.

[0031] The method also includes establishing a data audit and monitoring mechanism to monitor and record the access and use of girth weld data.

[0032] The present invention also proposes a pipeline full life cycle data processing system based on a data lake, which includes:

[0033] A data lake establishment module, which is used to establish the data lake based on the cloud platform, wherein the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form;

[0034] A tagging module, which is used to classify and tag the oil and gas pipeline girth weld full life cycle data, so as to identify and distinguish different types of data, and add corresponding tags or metadata to each data category;

[0035] A purification module, which is used to purify and verify the collected girth weld data to ensure the integrity and accuracy of the data;

[0036] A blockchain module for storing and verifying the integrity and authenticity of girth weld data using blockchain technology;

[0037] Smart contract module, which is used to manage and verify girth weld data using smart contracts;

[0038] The access control module is used to establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data.

[0039] Compared with the prior art, the present invention has the following significant advantages:

[0040] Realize the digitization of business objects, business rules, and business processes;

[0041] The data of the entire life cycle is aggregated in the data lake, and the data is processed through the digital algorithm of business rules, and the data is loaded accurately;

[0042] Realize the association and display of full life cycle data based on girth welds;

[0043] It enables managers to query and analyze pipeline weld information retrospectively. It fully implements digital guidance of business processes, improves work efficiency, and ensures risk prediction and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0045] Figure 1 is a flow chart showing a method for processing pipeline full life cycle data based on a data lake according to an embodiment of the present invention;

[0046] Figure 2 is a data node diagram showing data management of a girth weld throughout its life cycle according to an embodiment of the present invention; and

[0047] Figure 3 The invention is a schematic diagram showing a system for processing pipeline full life cycle data based on a data lake according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0050] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0051] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0052] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0053] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0054] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] Embodiment 1

[0056] like Figure 1 As shown, the present invention discloses a method for processing and evaluating pipeline life cycle data based on a data lake, comprising the following steps:

[0057] Step S101, establishing the data lake based on the cloud platform, wherein the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form;

[0058] Step S103, classifying and marking the oil and gas pipeline girth weld full life cycle data to identify and distinguish different types of data, and adding corresponding tags or metadata to each data category;

[0059] Step S105, purifying and verifying the collected girth weld data to ensure the integrity and accuracy of the data;

[0060] Step SS107: Using blockchain technology to store and verify the integrity and credibility of girth weld data;

[0061] Step S109: Using smart contracts to manage and verify girth weld data;

[0062] Step S1011: Establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data.

[0063] Embodiment 2

[0064] The present invention proposes a method for processing and evaluating pipeline life cycle data based on a data lake, comprising the following steps:

[0065] Step S101, establishing the data lake based on the cloud platform, wherein the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form;

[0066] Step S103, classifying and marking the oil and gas pipeline girth weld full life cycle data to identify and distinguish different types of data, and adding corresponding tags or metadata to each data category;

[0067] Step S105, purifying and verifying the collected girth weld data to ensure the integrity and accuracy of the data;

[0068] Step SS107: Using blockchain technology to store and verify the integrity and credibility of girth weld data;

[0069] Step S109: Using smart contracts to manage and verify girth weld data;

[0070] Step S1011: Establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data.

[0071] Wherein, the step S101 includes:

[0072] Create data lake storage on the cloud platform;

[0073] Creating a data table or collection in the data lake to store the full life cycle data of the girth weld;

[0074] Create a form interface for inputting girth weld life cycle data;

[0075] Integrate the form input interface with the data lake to ensure that the data entered in the form can be directly stored in the corresponding table or collection in the data lake.

[0076] Wherein, creating a data table or a collection in the data lake includes:

[0077] Use postgreSQL to establish a database, construct a data table according to the logical model and data dictionary, and determine the name of the data table, the field name and data type contained in the data table, and the data subset of the data lake includes a feature class data subset, a relationship class data subset and a geometry class subset;

[0078] With the relevant data fields of pile data, coordinates and weld numbers as the main line and the girth weld as the base, the relationship between the data is established.

[0079] The steps may include:

[0080] Based on the concepts of feature class, relationship class and geometry class of the whole life cycle data of girth weld, as well as the design goals and usage requirements of the data lake, the logical model is determined to determine the data entities, attributes and the relationships between them. This can help you sort out the organizational structure and association relationships of the data.

[0081] Create a data dictionary to clarify the name of each data table, field name, data type, constraints, etc. This helps ensure the consistency and standardization of data.

[0082] Based on the logical model and data dictionary, start designing the data table structure. In PostgreSQL, you can use the CREATE TABLE statement to create a data table and specify the name, data type, constraints, etc. for each field.

[0083] Determine the primary key and foreign key as required by the relationship class. A primary key is a field that uniquely identifies each record in a table, while a foreign key is a field that relates to other tables. You can define primary and foreign keys using PRIMARY KEY and FOREIGN KEY constraints.

[0084] Consider creating indexes for certain fields based on the access pattern and query requirements of your data. Indexes can improve query performance, but they also increase the cost of inserting and updating data. You can create indexes using the CREATE INDEX statement.

[0085] If you need to store and manipulate geometric data, PostgreSQL provides the PostGIS extension, which supports geometric data types and related spatial operations. You can install and configure the PostGIS extension, and then use the geometry type field in the data table.

[0086] Consider optimizing the data table structure based on data access patterns and performance requirements. This may include partitioning, sharding, horizontal or vertical splitting, etc. to improve query performance and data management efficiency.

[0087] To maintain consistency and readability, develop a naming convention for your data tables. Make sure the naming is clear, concise, and reflects the content and purpose of the data table.

[0088] Record the design decisions, field descriptions, constraints, etc. of each data table. This will help with future maintenance and understanding of the meaning and structure of the data table.

[0089] The data fields of pile data, coordinates and weld numbers are taken as the main line, and the girth weld is taken as the base to establish the association relationship between the data, including:

[0090] Establish the connection between pile data and coordinates, which determines the specific location of the pile by associating the coordinate fields in the pile data with the geographic coordinates, so as to analyze the distribution, distance relationship and spatial area of ​​the piles in the geographic space;

[0091] Establishing a connection between pile data and weld numbers, by associating the weld numbers with pile data, determining the pile corresponding to each weld, so as to query, retrieve and update the pile data related to a specific weld according to the weld numbers;

[0092] Establishing a connection between the weld number and the weld data, by using the weld number as a primary key to associate the weld data with the weld, so as to query, retrieve and update the weld data related to a specific weld according to the weld number;

[0093] Establish a connection between weld data and pile data by matching the location field in the weld data with the pile data to determine the specific location of the weld, and perform geospatial analysis and location query.

[0094] Here are some examples:

[0095] Girth weld data sheet (WeldSeam):

[0096] WeldSeamID (girth weld ID, primary key)

[0097] WeldType

[0098] WeldDate (welding date)

[0099] Pile data table (Pile):

[0100] PileID (pile number, primary key)

[0101] PileType

[0102] PileLength

[0103] Coordinate data table (Coordinate):

[0104] CoordinateID (coordinate number, primary key)

[0105] X (X coordinate)

[0106] Y(Y coordinate)

[0107] Z (Z coordinate)

[0108] WeldJoint data table (WeldJoint):

[0109] WeldJointID (weld joint number, primary key)

[0110] WeldSeamID (girth weld ID, foreign key associated to the girth weld data table)

[0111] PileID (pile ID, foreign key associated to the pile data table)

[0112] CoordinateID (coordinate number, foreign key associated with the coordinate data table)

[0113] In this relational model, the girth weld data table is the core table, and other tables are associated with the girth weld data table through foreign keys. For example, the WeldSeamID field in the weld data table is associated with the WeldSeamID field in the girth weld data table, and the pile data table and coordinate data table are also associated with the girth weld data table through corresponding foreign keys.

[0114] In one embodiment, the data subsets of the data lake include feature class data subsets, relationship class data subsets, and geometry class data subsets, as shown below:

[0115] Feature class data subset: Table name: feature_table

[0116] Fields:

[0117] feature_id: integer type, unique identifier of feature

[0118] feature_name: text type, feature name

[0119] feature_type: text type, feature type

[0120] feature_properties: JSONB type, feature attributes, stores additional attribute information Relationship class data subset:

[0121] Table name: relation_table

[0122] Fields:

[0123] relation_id: integer type, unique identifier of the relation

[0124] relation_name: text type, relation name

[0125] relation_type: text type, relation type

[0126] related_features: integer array type, list of related feature IDs

[0127] Geometry data subset:

[0128] Table name: geometry_table

[0129] Fields:

[0130] geometry_id: integer type, unique identifier of geometric object

[0131] geometry_type: text type, geometric object type

[0132] geometry_data: geometric object data type, storing geometric object data

[0133] PostgreSQL provides rich data type support. You can choose the appropriate data type according to actual needs, such as integer type, text type, JSONB type, and geometric object data type.

[0134] The cloud platform includes Amazon Web Services (AWS), Microsoft Azure or Google Cloud Platform, and the cloud storage service includes Amazon S3, Azure Blob Storage or Google Cloud Storage.

[0135] Wherein, the step S103 includes:

[0136] Data classification includes welding parameter data, material data, inspection data, welding process monitoring data and welding quality assessment data;

[0137] For welding parameter data, the parameter values ​​of welding current, welding voltage and welding speed are marked; for material data, the pipe material type, material batch and specification are marked;

[0138] Add metadata for each data category, including relevant information and attributes of the data. For example, for welding parameter data, you can add metadata such as data collection time, collection equipment, and operator; for inspection data, you can add metadata such as inspection method, inspection equipment, and inspection date.

[0139] Here are some examples of possible data categories and corresponding tags or metadata:

[0140] Welding parameter data:

[0141] Marking: welding current, welding voltage, welding speed and other parameter values.

[0142] Metadata: information such as data collection time, collection equipment, and operator.

[0143] Material Data:

[0144] Marking: Pipe material type, material batch, specifications, etc.

[0145] Metadata: information such as material supplier, purchase date, quality inspection results, etc.

[0146] Test data:

[0147] Marking: defect type, defect location, defect size, etc.

[0148] Metadata: information such as test method, test equipment, test date, etc.

[0149] Welding process monitoring data:

[0150] Mark: numerical values ​​of monitoring parameters such as temperature, stress, deformation, etc.

[0151] Metadata: information such as monitoring equipment, monitoring location, monitoring time, etc.

[0152] Welding quality assessment data:

[0153] Marking: welding defect grade, quality score, etc.

[0154] Metadata: information such as evaluation criteria, evaluators, and evaluation date.

[0155] Wherein, the step S105 includes: data cleaning, data deduplication, and data format verification to remove duplicate, incomplete or erroneous data and ensure that the data complies with the agreed format and specifications.

[0156] Specifically, step S105 may be data cleaning and preprocessing to ensure the quality and consistency of the data, which may include processing missing values, removing duplicate data, correcting format errors, etc. Data cleaning techniques and algorithms are used to clean and purify the data.

[0157] Subsequently, business rules and targeted purification rules are defined based on the requirements of the girth weld life cycle data. These rules can include aspects such as data integrity, scope limitation, association, format specification, quality assessment and compliance.

[0158] The specific targeted purification implementation needs to implement the corresponding algorithms and logic according to the defined targeted purification rules. This may involve the use of data processing tools and programming languages ​​to filter, transform and repair data to make it comply with the specified business rules.

[0159] Among them, the targeted purification rules include:

[0160] Data integrity rules: Ensure the integrity of girth weld data, such as requiring that all girth weld data must contain welding date, welder information, and welding parameter information. The directed cleansing algorithm can check whether each girth weld data contains these necessary fields, and if missing, mark it as incomplete data or perform data filling.

[0161] Data range restriction rules: restrict the value range of certain fields of girth weld data. For example, welding current must be within a specific range, welding speed must be greater than zero, etc. The directional purification algorithm can filter out or correct out-of-range data based on predefined range rules.

[0162] Data association rules: Ensure that the associations between girth weld data are correct. For example, each girth weld data should be associated with the corresponding welder information and pipeline component information. The directional cleansing algorithm can check the associations between data and exclude missing or incorrectly associated data.

[0163] Data format rules: Ensure that the girth weld data is in the required format. For example, the date and time fields must be in a specific format, welder information must include name and employee number, etc. Targeted cleansing algorithms can convert or standardize the data according to predefined format rules.

[0164] Data quality rules: Evaluate the quality of girth weld data according to quality standards. For example, check the data accuracy, repeatability and consistency of girth weld data. Directed purification algorithms can identify and process low-quality data through methods such as data comparison, deduplication and logical verification.

[0165] Compliance rules: Ensure that girth weld data complies with relevant regulations and industry standards. For example, privacy protection of personally identifiable information is performed to ensure data security and compliance. Targeted purification algorithms can desensitize or encrypt personally identifiable information according to privacy protection rules.

[0166] For example, a specific formula is used to perform targeted purification of business rule definitions:

[0167] Assume there is a business rule that the welding speed of the girth weld must be within a specific range (between 10mm / min and 50mm / min). To meet this rule, the following formula can be used for directional purification:

[0168] Assume that the welding speed field is "WeldingSpeed" and the unit is mm / min.

[0169] Set the minimum welding speed value to 10 and the maximum welding speed value to 50.

[0170] In the directional cleaning algorithm, each girth weld data is checked:

[0171] If "WeldingSpeed" is less than the minimum welding speed value (10), update the value to the minimum welding speed value.

[0172] If "WeldingSpeed" is greater than the maximum welding speed value (50), then update the value to the maximum welding speed value.

[0173] If "WeldingSpeed" is between the minimum and maximum values, the original value is retained.

[0174] Through the above content, the welding speed field in the girth weld data will be cleaned in a targeted manner to ensure that its value is within the specified range (between 10mm / min and 50mm / min). This can ensure the accuracy, consistency and compliance of the data, so that the welding speed data meets the predefined business rules.

[0175] The above is a simple example. If we consider all the rules:

[0176] Data integrity rules:

[0177] Ensure that all girth weld data contains necessary fields such as welding date, welder information, and welding parameter information.

[0178] The following steps are used for directional purification: check whether each girth weld data contains these necessary fields. If missing, mark it as incomplete data or perform data filling.

[0179] Data range restriction rules:

[0180] Limit the value range of certain fields in the girth weld data, such as welding current, welding speed, etc.

[0181] The following steps are used for directional cleansing: According to the predefined range rules, the data that exceeds the range is filtered out or corrected.

[0182] Data association rules:

[0183] Ensure that the association between girth weld data is correct, such as welder information and pipe component information.

[0184] The following steps are used for targeted purification: check the association between data and exclude missing or incorrectly associated data.

[0185] Data format rules:

[0186] Ensure that the girth weld data format meets the requirements, such as date and time fields, welder information, etc.

[0187] The following steps are used for targeted cleansing: data is formatted or standardized according to predefined format rules.

[0188] Data quality rules:

[0189] Evaluate the quality of girth weld data for accuracy, repeatability, and consistency.

[0190] The following steps are used for targeted purification: Identify and process low-quality data through methods such as data comparison, deduplication and logical verification.

[0191] Compliance rules:

[0192] Ensure girth weld data complies with relevant regulations and industry standards, such as privacy protection of personally identifiable information.

[0193] The following steps are used for targeted purification: Desensitize personal identity information or encrypt data according to privacy protection rules.

[0194] Wherein, the step S107 includes: each girth weld data can be used as a transaction record, stored in the form of a blockchain, and the consistency and non-tamperability of the data are ensured through a distributed consensus algorithm.

[0195] The method also includes establishing a data audit and monitoring mechanism to monitor and record the access and use of girth weld data.

[0196] Embodiment 3

[0197] The present invention also proposes a pipeline full life cycle data processing system based on a data lake, which includes:

[0198] A data lake establishment module, which is used to establish the data lake based on the cloud platform, wherein the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form;

[0199] A tagging module, which is used to classify and tag the oil and gas pipeline girth weld full life cycle data, so as to identify and distinguish different types of data, and add corresponding tags or metadata to each data category;

[0200] A purification module, which is used to purify and verify the collected girth weld data to ensure the integrity and accuracy of the data;

[0201] A blockchain module for storing and verifying the integrity and authenticity of girth weld data using blockchain technology;

[0202] Smart contract module, which is used to manage and verify girth weld data using smart contracts;

[0203] The access control module is used to establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data.

[0204] Embodiment 4:

[0205] Based on the safety production technical standard system of the oil and gas storage and transportation industry, and in accordance with the technical specifications for oil and gas pipeline protection management and operation management procedures, a girth weld full life cycle data management model is developed. The intelligent capture, exchange, and transfer of pipeline full life cycle data is realized; data standardization and normalization are realized; the acquisition channels of weld and steel pipe information are controlled from the five aspects of manufacturing, procurement, construction, monitoring, and operation to realize data flow control; the automatic growth function of girth weld data is realized; and the "collection, management, update, and maintenance" of girth weld data is realized.

[0206] By systematically sorting out the generation process, type and attribute values ​​of girth weld data throughout its life cycle, combined with the girth weld management process during pipeline construction, operation and maintenance, data standards for girth welds are established, and a new data connection method that relies on girth welds as data hard reference points is established. The goals of digitizing girth weld rules, objects and processes are achieved, and a virtuous dual cycle from business to data and data to business is promoted, laying a reliable data foundation for realizing girth weld quality risk assessment and truly creating data value.

[0207] The system of the present invention is deployed on a mobile terminal and a WEB terminal respectively.

[0208] 1) Mobile terminal: It mainly serves the personnel in the operation area and other grassroots units of oil and gas pipelines, facilitates the collection and storage of data, and centrally displays data, which provides convenience for the development of pipeline-related work. If an emergency occurs, the system can be used to track and query data, quickly grasp all relevant information of the girth weld, and respond in time.

[0209] 2) Web terminal: mainly serves the managers of the National Pipeline Network Corporation and its branches, meeting their needs for the use, analysis and subsequent maintenance management of pipeline life cycle data, and assisting managers in making pipeline-related decisions.

[0210] (2) Business Process

[0211] 1) Basic information architecture (operated and executed by the system administrator or person in charge of the branch (management office) and implemented on the web)

[0212] Line information: information on line network structure, test piles, stations, and pipe sections;

[0213] Data set information: building frameworks for various data template tables;

[0214] Personnel information organizational information.

[0215] 2) Various data management (implemented by different data source units and mobile applets)

[0216] Mainly includes: database management, data maintenance, data verification, contractor management, material management, and data review.

[0217] 3) Data analysis (by professional technicians, based on system functions and implemented on the web)

[0218] Mainly includes: data statistics, tracking query.

[0219] 4) Data normalization (web-based implementation)

[0220] It mainly includes automatic association and matching of data such as weld alignment and defect alignment.

[0221] 5) 2D and 3D visualization (web-based implementation)

[0222] Automatically visualize the data in two or three dimensions based on the standardized data.

[0223] 6) Document management (web-based implementation)

[0224] After cleaning various types of data, they are stored according to standardized templates. They can be used as standardized data results and can also provide data support for other subsequent analyses.

[0225] (3) Tasks, activities, roles and positions

[0226] 1) Company administrator: Visualize the test progress and test results of each branch company; analyze the data in the data lake at any time.

[0227] Maintenance administrator (pipeline professional): input pipeline information, personnel information and other data to complete system initialization; update the subsequent pipeline related information in real time if there are changes;

[0228] 2) Regional company and branch auditors (pipeline professionals): pipeline data management for the entire life cycle. After collecting, exchanging and transferring data for the entire life cycle of the pipeline, the system automatically analyzes abnormal data and guides data processing. Company auditors can review the test results.

[0229] 3) Data entry personnel (various data source units): Different data entry roles are assigned according to the source of different data, and the data stakeholders enter the data into the system.

[0230] (4) Girth weld full life cycle data management module

[0231] like Figure 2 As shown, it is a data node diagram in the data management of the whole life cycle of the girth weld consistent with a certain embodiment of the present invention. The system of the present invention can include a physical module, a data flow control module, and a data module in terms of process.

[0232] Physics Module

[0233] The physical module of the girth weld life cycle data management model mainly considers the following rules:

[0234] The data foundation for module design is based on the digital restoration of pipelines / digital handover results during construction period;

[0235] Taking girth welds and steel pipes as management objects, the information of manufacturing and production stages is expanded upwards, and the information of construction monitoring, excavation re-inspection, internal and external monitoring is expanded downwards;

[0236] Taking virtual mileage as the main line of model management, data at different stages are aligned to standardize mileage, so as to achieve sequential management of data at different stages and improve the relevance of previous and next data objects;

[0237] Control Module

[0238] The acquisition channels of weld and steel pipe information are controlled from five aspects: manufacturing, procurement, construction, monitoring and operation. The method of accurate data collection by all participating units is adopted to ensure the continuity and accuracy of data related to weld materials, construction technology, physical and chemical properties, operation detection, excavation verification, etc. throughout the life cycle.

[0239] Data Module

[0240] The data module involves three sub-libraries: steel material library, pipe material library, and weld defect library; two types of key codes: weld code and steel pipe code.

[0241] Three types of users: procurement period, construction period, and operation period.

[0242] Consistent with an embodiment of the present invention, the software architecture of the present invention is consistent with the national pipeline network IMS system, and mainly follows the following rules:

[0243] (1) User login authentication is consistent with IMS;

[0244] (2) GIS visualization is consistent with IMS geographic information software;

[0245] (3) Business data uses the domestic GBase 8s distributed database;

[0246] (4) Use Huawei CSE Registration Center to register services;

[0247] (5) The software service mode is REST service of https protocol;

[0248] (6) The front-end application uses the cross-platform Vue structure to ensure cross-browser support;

[0249] Flutter is used on the mobile terminal to achieve cross-platform mobile development.

[0250] like Figure 3 As shown, it is a system block diagram consistent with a certain embodiment of the present invention, which is composed of an APP end and a Web end. The main functions include operation and maintenance management (authority management, collection process management), data management (database management, data maintenance, data verification, contractor management, material management, data review), data analysis (data statistics, tracking query), data normalization (weld alignment, defect alignment), two- and three-dimensional visualization, and document management. Six modules.

[0251] Embodiment 5

[0252] An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.

[0253] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0254] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0255] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (AN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0256] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0257] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0258] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A method for processing pipeline life cycle data based on a data lake, characterized in that: The method comprises the following steps: Step S101, establishing the data lake based on the cloud platform, wherein the front-end application adopts a cross-platform Vue structure to ensure that the browser is cross-platform, and the data lake stores the full life cycle data of the girth weld of the oil and gas pipeline, wherein the full life cycle data of the girth weld of the oil and gas pipeline enters the data lake in the form of a form; Step S103, classifying and marking the oil and gas pipeline girth weld full life cycle data to identify and distinguish different types of data, and adding corresponding tags or metadata to each data category; Step S105, purifying and verifying the collected girth weld data to ensure the integrity and accuracy of the data; Step S107: Using blockchain technology to store and verify the integrity and credibility of girth weld data; Step S109: Using smart contracts to manage and verify girth weld data; Step S1011, establish a strict access control and authority management mechanism to ensure that only authorized users can access and operate girth weld data; Wherein, the step S105 includes: data cleaning, data deduplication, and data format verification to remove duplicate, incomplete or erroneous data and ensure that the data complies with the agreed format and specifications; Among them, business rules and directional purification rules are defined according to the requirements of the full life cycle data of the girth weld; Among them, the implementation of directional purification needs to implement the corresponding algorithms and logic according to the defined directional purification rules; Among them, the targeted purification rules include: Data integrity rules: Ensure the integrity of girth weld data, including requiring that all girth weld data must contain welding date, welder information and welding parameter information, and checking whether each girth weld data contains these necessary fields. If missing, mark it as incomplete data or fill in the data; Data range restriction rules: restrict the value range of certain fields of girth weld data, including filtering out data that exceeds the range or correcting it according to predefined range rules; Data association rules: ensure that the association between girth weld data is correct, which includes checking the association between data and excluding missing or incorrectly associated data; Data format rules: Ensure that the format of girth weld data meets the requirements, including format conversion or standardization of data according to predefined format rules; Data quality rules: Evaluate the quality of girth weld data according to quality standards, including identifying and processing low-quality data through data comparison, deduplication and logical verification; Compliance rules: Ensure that girth weld data complies with relevant regulations and industry standards, including desensitizing personally identifiable information or encrypting data in accordance with privacy protection rules; Wherein, the step S101 includes: Create data lake storage on the cloud platform; Creating a data table or collection in the data lake to store the full life cycle data of the girth weld; Create a form interface for inputting girth weld life cycle data; Integrate the form input interface with the data lake to ensure that the data entered in the form can be directly stored in the corresponding table or collection in the data lake; Wherein, creating a data table or a collection in the data lake includes: Use postgreSQL to establish a database, construct a data table according to the logical model and data dictionary, and determine the name of the data table, the field name and data type contained in the data table, and the data subset of the data lake includes a feature class data subset, a relationship class data subset and a geometry class subset; With the relevant data fields of pile data, coordinates and weld numbers as the main line and the girth weld as the base, the relationship between the data is established; Among them, using postgreSQL to establish a database, constructing a data table according to the logical model and data dictionary, and determining the name of the data table, including: Based on the feature class, relationship class and geometry class of the girth weld life cycle data, determine the logical model, data entities, attributes and the relationships between them; Create a data dictionary to clarify the name, field name, data type, and constraints of each data table; Design data table structure according to logical model and data dictionary; According to the requirements of the relationship class, determine the primary key and foreign key. The primary key is a field that uniquely identifies each record in the table, and the foreign key is a field associated with other tables. Use PRIMARY KEY and FOREIGN KEY constraints to define the primary key and foreign key. Create indexes based on data access patterns and query requirements; The data fields of pile data, coordinates and weld numbers are used as the main line, and the girth weld is used as the base to establish the relationship between the data, including: Establish the connection between pile data and coordinates, which determines the specific location of the pile by associating the coordinate fields in the pile data with the geographic coordinates, so as to analyze the distribution, distance relationship and spatial area of ​​the piles in the geographic space; Establishing a connection between pile data and weld numbers, by associating the weld numbers with pile data, determining the pile corresponding to each weld, so as to query, retrieve and update the pile data related to a specific weld according to the weld numbers; Establishing a connection between the weld number and the weld data, by using the weld number as a primary key to associate the weld data with the weld, so as to query, retrieve and update the weld data related to a specific weld according to the weld number; Establishing a connection between weld data and pile data by matching the location field in the weld data with the pile data to determine the specific location of the weld and perform geospatial analysis and location query; The cloud platform includes Amazon Web Services, Microsoft Azure or Google Cloud Platform, and the cloud storage service includes Amazon S3, Azure Blob Storage or Google Cloud Storage; Wherein the step S103 comprises: Data classification includes welding parameter data, material data, inspection data, welding process monitoring data and welding quality assessment data; For welding parameter data, the parameter values ​​of welding current, welding voltage and welding speed are marked; for material data, the pipe material type, material batch and specification are marked; Add metadata to each data category, including relevant information and attributes of the data; The step S107 includes: each girth weld data can be used as a transaction record, stored in the form of a blockchain, and the consistency and non-tamperability of the data are ensured through a distributed consensus algorithm; The method also includes establishing a data audit and monitoring mechanism to monitor and record the access and use of girth weld data.

Images