Data structure and method for organizing and storing magnetic flux leakage analysis data

Abstract

The invention discloses a data structure and method for organizing and storing magnetic flux leakage analysis data, and the data structure comprises a storage mode of organizing data through a mode of constructing a group, the storage mode is started from a root directory, and a plurality of data nodes are established under the root directory to store data contents. Different data contents are classified and stored in a grouping mode to form storage nodes; by means of data classification, grouping and data row combination, data can be read and written concurrently, and the efficiency problem during data processing is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of magnetic flux leakage analysis data processing, and in particular to a data structure and method for organizing and storing magnetic flux leakage analysis data. Background Art

[0002] Magnetic flux leakage detection technology for oil and gas pipelines is a non-destructive testing method that uses magnetic field changes to detect pipeline defects. Its basic principles are: 1. Magnetic field generation: In magnetic flux leakage detection, it is first necessary to apply a magnetic field to the object being tested through external means. This magnetic field is usually provided by a permanent magnet or an electromagnet. 2. Defect detection: When there are defects in the pipeline, these defects will change the originally uniform magnetic field distribution and form the so-called "leakage magnetic field". Through highly sensitive magnetic sensors, these tiny magnetic field changes can be detected to determine the location and size of the defects.

[0003] Therefore, collecting and storing this data can monitor and analyze the status of oil and gas pipelines in real time, greatly improving the efficiency and accuracy of pipeline safety management. The current market generally stores data in segments, with each file storing complete inspection data for a section of the pipeline. For example, each data file stores 1,000 meters of data. The existing segmented storage technology is not conducive to concurrent data processing. Summary of the invention

[0004] In order to solve the problems of the prior art, the present invention provides a data structure and method for organizing and storing magnetic flux leakage analysis data.

[0005] Its main technical solutions are as follows:

[0006] The present invention provides a data structure for organizing and storing magnetic flux leakage analysis data, including:

[0007] The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; including:

[0008] A description file module, which is used to record the file information of all storage nodes under the root directory through a file information recording model by setting a description file under the root directory;

[0009] The file recording module is configured to store the original data of the received magnetic flux leakage analysis data by setting an original file storage node under the root directory, and the original file storage node is configured to: classify and store different data contents by grouping to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content;

[0010] The storage subnode is configured to store data files of different views according to a set file format, configure the data information of the data files by setting a configuration program, and implement the setting of the data file channel by configuring the data information.

[0011] Furthermore, the file information recording model is provided with a field structure with set rules, and the file information of all storage nodes under the root directory is recorded through the field structure.

[0012] Furthermore, the field structure at least includes:

[0013] The first field area for describing and recording the file path relative to the root path;

[0014] A second field area of ​​the key file for describing and recording data files under all storage nodes under the root directory;

[0015] A third field area for describing and recording the file offsets of data files under all storage nodes under the root directory;

[0016] A fourth field area for describing and recording the key type used by the key file;

[0017] And a fifth field area for describing and recording the field structure type.

[0018] Furthermore, it also includes an index unit, which is set under the file recording module. The index unit is used to set an index file under the original file storage node, and to construct different data content index information for classified storage through the index file.

[0019] Furthermore, the index file includes:

[0020] The data header area is used to describe the summary information of the current index file;

[0021] and a data block area for recording data, wherein the data block area is used to record the data amount of the data block, and the data block is a group classified as different data.

[0022] Furthermore, the configuration procedure at least includes:

[0023] According to the set field rules, the header information in the header information area, the data file information and data unit in the data information area, and the extended information in the extended information area are configured;

[0024] The data information at least includes:

[0025] Used to describe the header information area including at least the data file channel number;

[0026] Data information area for recording data file information and data units;

[0027] And an extended information area for recording extended information.

[0028] Furthermore, it also includes an extended information data management module for preprocessing non-standard raw data;

[0029] Preferably, the extended information data management module pre-processes the non-standard original data by annotating the data.

[0030] Further preferably, the data annotation method is to process the non-standard original data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtain the index file and field structure of the new data.

[0031] Furthermore, the same data content is classified and stored by constructing groups, including: grouping the magnetic flux leakage data according to the data attributes of the detection probe, and grouping the data under the same probe and the same data attribute into one group.

[0032] The present invention also provides a method for organizing and storing magnetic flux leakage analysis data, including:

[0033] The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; including:

[0034] By setting a description file in the root directory, the file information of all storage nodes in the root directory is recorded through a file information recording model;

[0035] An original file storage node is set under the root directory to store the original data of the received magnetic flux leakage analysis data, and the original file storage node is configured to: classify and store different data contents in a grouping manner to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content;

[0036] The data files of different views are stored according to the set file format, and the data information of the data files is configured by setting the configuration program, and the setting of the data file channel is realized through the configuration of the data information.

[0037] Furthermore, it also includes: setting an index file under the original file storage node, and constructing different data content index information for classified storage through the index file;

[0038] And / or, for non-standard raw data that cannot construct index files and field structures; process them by annotating data, and the annotating data method is to process the non-standard raw data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtain the index file and field structure of the new data.

[0039] The beneficial effect of the present invention is that in the method for organizing and storing magnetic flux leakage analysis data provided by the present invention, the detection data is changed from traditional segmented storage to classified and grouped storage, which is convenient for display processing and can realize concurrent reading and writing of data, significantly improving the analysis efficiency. At the same time, by classifying, grouping and arranging data, data can be read and written concurrently, and the efficiency problem of processing data is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] One of the key technologies is the file format technology for storing magnetic flux leakage detection data. Because the detection data is usually large, taking the 406 type device as an example, if the device parameters are set to 2k during detection, the amount of data collected per millisecond is: main probe: 44×7×2=616 bytes, sub-probe: 44×2=88 bytes.

[0044] The main and auxiliary probes generate a total of (616+88)×2=1,408 bytes. The amount of data collected per minute is: 1408 bytes×1000×60=80M, and the amount of data collected per hour is: 80M×60=4800M. It takes about 100 hours to detect a 120-kilometer pipeline, and the total amount of data generated is about: 4800M×100=468G. Therefore, the amount of data is very large, so the traditional segmented storage technology is not conducive to concurrent processing of data.

[0045] Reference Figure 1 Therefore, the present application discloses a data structure for organizing and storing magnetic flux leakage analysis data, including:

[0046] The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; including:

[0047] A description file module, which is used to record the file information of all storage nodes under the root directory through a file information recording model by setting a description file under the root directory;

[0048] The file recording module is configured to store the original data of the received magnetic flux leakage analysis data by setting an original file storage node under the root directory, and the original file storage node is configured to: classify and store different data contents by grouping to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content;

[0049] An index unit is provided under the file recording module, and is used to set an index file under the original file storage node, and to construct index information of different data contents for classified storage through the index file;

[0050] The storage subnode is configured to: store data files of different views according to a set file format, configure the data information of the data files by setting a configuration program, and implement the setting of the data file channel by configuring the data information;

[0051] The extended information data management module is used to extend non-standard original data.

[0052] Furthermore, the file information recording model is provided with a field structure with set rules, and the file information of all storage nodes under the root directory is recorded through the field structure.

[0053] Furthermore, the field structure at least includes:

[0054] The first field area for describing and recording the file path relative to the root path;

[0055] A second field area of ​​the key file for describing and recording data files under all storage nodes under the root directory;

[0056] A third field area for describing and recording the file offsets of data files under all storage nodes under the root directory;

[0057] A fourth field area for describing and recording the key type used by the key file;

[0058] And a fifth field area for describing and recording the field structure type.

[0059] Furthermore, the index file includes:

[0060] The data header area is used to describe the summary information of the current index file;

[0061] and a data block area for recording data, wherein the data block area is used to record the data amount of the data block, and the data block is a group classified as different data.

[0062] Furthermore, the configuration procedure at least includes:

[0063] According to the set field rules, the header information in the header information area, the data file information and data unit in the data information area, and the extended information in the extended information area are configured;

[0064] The data information at least includes:

[0065] Used to describe the header information area including at least the data file channel number;

[0066] Data information area for recording data file information and data units;

[0067] And an extended information area for recording extended information.

[0068] Furthermore, the extended information data management module manages non-standard original data by marking data.

[0069] Furthermore, the data annotation method is to process the non-standard original data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtain the index file and field structure of the new data.

[0070] The present invention also provides a method for organizing and storing magnetic flux leakage analysis data, including:

[0071] The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; including:

[0072] By setting a description file in the root directory, the file information of all storage nodes in the root directory is recorded through a file information recording model;

[0073] An original file storage node is set under the root directory to store the original data of the received magnetic flux leakage analysis data, and the original file storage node is configured to: classify and store different data contents in a grouping manner to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content;

[0074] Set up index files under the original file storage node, and use the index files to build index information of different data contents for classified storage;

[0075] The data files of different views are stored according to the set file format, and the data information of the data files is configured by setting the configuration program, and the setting of the data file channel is realized by the configuration of the data information;

[0076] For non-standard raw data that cannot construct index files and field structures, it is processed by annotated data. The annotated data method is to process the non-standard raw data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtain the index file and field structure of the new data.

[0077] In the present invention, the detection data is changed from traditional segmented storage to classified and grouped storage, which has the advantages of facilitating display processing and enabling concurrent reading and writing of data, significantly improving analysis efficiency. At the same time, by classifying, grouping and arranging data in a combined manner, data can be read and written concurrently, and the efficiency problem of processing data can be solved.

[0078] The present application is described in detail below with reference to examples.

[0079] The basic information of data sources is as follows:

[0080] 1.1 main probe contains 7 channel sensors, each channel sensor has 3-axis data (x / y / z);

[0081] 2.1 sub-probe contains 2 channels (deformation + IDOD);

[0082] 3.1016mm detector has 112 main probes and 112 auxiliary probes; the distance between two adjacent points is 1mm;

[0083] 4. The MFL file is named PRJ0001_20230402-101924_00001.MFL, where PRJ0001 is the project number, 20230402-101924 is the serve synchronization time, and 00001 is the file sequence number; the single file size is 5164B×10000+32=51640,032≈50MB, and the mileage length is about 10m;

[0084] 5. The IMU file is named PRJ0001_20230402-101924_00001.IMU, where PRJ0001 is the project number, 20230402-101924 is the serve synchronization time, 00001 is the file sequence number, the single file size is 40B×10000=400kB, and the mileage length is about 10m;

[0085] The traditional magnetic flux leakage MFL file format is as follows:

[0086]

[0087]

[0088] The data formats include the following:

[0089] name type bit15~8 bit7~4 bit3~0 Conversion formula Magnetic flux leakage data Int16 Axis_MSB Axis_LSB 0000 Data ÷ 32, in Gauss Deformation / IDOD Int16 Chx_MSB Chx_LSB 0000 Data÷(2^16)×5, unit is volt

[0090] The inertial navigation IMU file format is as follows:

[0091]

[0092]

[0093] In the present invention, there is a file in the root directory ending with Description, which is used to describe the information of the entire current storage structure to ensure the integrity and correctness of the storage; wherein, the root directory of the file storage is the starting point of the entire file structure system.

[0094] The file storage content is Json, and the field structure is as follows:

[0095]

[0096] The DataFileInfo field structure is as follows:

[0097]

[0098] There is a RawDatas folder in the root directory, which stores the most original data. The data does not contain any extensions or operations. The root directory of the RawDatas folder contains a fixed name<Idx.Data> This is the index file of the current data. The main contents of the file are used to explain the following:

[0099]

[0100]

[0101] The data header is used to describe the summary information of the current index file. At the same time, the subsequent data blocks will be stored repeatedly, and each block is a group, totaling a total number of data.

[0102] There are five different folders under the Rawdatas directory: X, Y, Z, IDOD, and Deformation, which store data files of different views respectively. The data file of each view ends with .Data.

[0103] All raw data files end with .Data, and each data file corresponds to only one channel.

[0104] The specific definitions are as follows:

[0105]

[0106] Channel repair data is used when the channel is repaired but not merged. The total index uses Json to store the index data in the following format:

[0107]

[0108]

[0109] FixDataFileInfo data information:

[0110]

[0111] There is a folder called Datas under the root folder of the channel repair data. This folder is used to store the created repair data. The file ends with the .Data extension. The file name is random and corresponds to the index file. The corresponding data structure is:

[0112]

[0113] Labeled data If the expert has adjusted the data, the adjustment includes but is not limited to: outlier filtering, odd-even alignment, IDOD alignment, etc. When the expert confirms that it is complete, click Output New Data, then a new folder will appear in the Extends data, and the folder structure is exactly the same as RawDatas, and the data structure is also the same.

[0114] In the data storage, the present invention classifies the leakage magnetic data according to the probe coordinate attributes, classifies the data under the same data attribute under different probes into one category, and classifies the data under the same probe and the same data attribute into one group, so that the relevant data can be quickly retrieved in the process of data visualization, which greatly improves the data reading efficiency. For example, in traditional data storage, if the data of the x-axis 1000m of probe A needs to be visualized, then when acquiring the data, it is necessary to traverse the data of the x-axis, y-axis, z-axis and IDOD attributes within 1000m of the probe, so as to completely obtain and read the data of the x-axis 1000m before the probe, and form a visual graph of the x-axis of the pipeline; after adopting the data structure of the present invention, it is only necessary to directly retrieve the data group under the x-axis classification file.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data structure for organizing and storing magnetic flux leakage analysis data, characterized in that: include: The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; including: A description file module, which is used to record the file information of all storage nodes under the root directory through a file information recording model by setting a description file under the root directory; The file recording module is configured to store the original data of the received magnetic flux leakage analysis data by setting an original file storage node under the root directory, and the original file storage node is configured to: classify and store different data contents by grouping to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content; The storage subnode is configured to store data files of different views according to a set file format, configure the data information of the data files by setting a configuration program, and implement the setting of the data file channel by configuring the data information.

2. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 1, characterized in that: The file information recording model is provided with a field structure with set rules, and the file information of all storage nodes under the root directory is recorded through the field structure.

3. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 2, characterized in that: The field structure at least includes: The first field area for describing and recording the file path relative to the root path; A second field area of ​​the key file for describing and recording data files under all storage nodes under the root directory; A third field area for describing and recording the file offsets of data files under all storage nodes under the root directory; A fourth field area for describing and recording the key type used by the key file; And a fifth field area for describing and recording the field structure type.

4. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 1, characterized in that: It also includes an index unit, which is set under the file recording module. The index unit is used to set an index file under the original file storage node, and to construct different data content index information for classified storage through the index file.

5. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 4, characterized in that: The index file contains: The data header area is used to describe the summary information of the current index file; and a data block area for recording data, wherein the data block area is used to record the data amount of the data block, and the data block is a group classified as different data.

6. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 1, characterized in that: The configuration procedure at least includes: According to the set field rules, the header information in the header information area, the data file information and data unit in the data information area, and the extended information in the extended information area are configured; The data information at least includes: Used to describe the header information area including at least the data file channel number; Data information area for recording data file information and data units; And an extended information area for recording extended information.

7. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 4, characterized in that: It also includes an extended information data management module for preprocessing non-standard raw data; Preferably, the extended information data management module pre-processes the non-standard original data by annotating the data. Further preferably, the data annotation method processes the non-standard original data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtains the index file and field structure of the new data.

8. The data structure for organizing and storing magnetic flux leakage analysis data according to claim 1, characterized in that: Different data contents are classified and stored by constructing groups, including: classifying the magnetic flux leakage data according to the data attributes of the detection probe, and grouping the data under the same probe and the same data attribute into one group.

9. A method for organizing and storing magnetic flux leakage analysis data, characterized in that: include: The storage method of organizing data is to build groups, and the storage method starts from a root directory. Several data nodes are established under the root directory to store data content. Different data contents are classified and stored by building groups to form storage nodes; Among them, include: By setting a description file in the root directory, the file information of all storage nodes in the root directory is recorded through a file information recording model; An original file storage node is set under the root directory to store the original data of the received magnetic flux leakage analysis data, and the original file storage node is configured to: classify and store different data contents in a grouping manner to obtain corresponding storage child nodes, and set the read and write attributes of the original file storage node so that the original data in the original file storage node does not contain any extension and operation content; The data files of different views are stored according to the set file format, and the data information of the data files is configured by setting the configuration program, and the setting of the data file channel is realized through the configuration of the data information.

10. The method for organizing and storing magnetic flux leakage analysis data according to claim 7, characterized in that: Also includes: Set up index files under the original file storage node, and use the index files to build index information of different data contents for classified storage; And / or, for non-standard raw data that cannot construct index files and field structures; process them through labeled data, which is to process non-standard raw data through outlier filtering, parity alignment, and IDOD alignment to obtain new data, and obtain index files and field structures for the new data.

Images