Data alignment method, apparatus, device, and storage medium

Abstract

The application discloses a data alignment method and device, equipment and a storage medium, relates to the technical field of data processing, and can realize automatic data alignment of pipeline internal detection data of different detection periods, improve data matching efficiency and matching accuracy. The method comprises the following steps: obtaining a first data set and a second data set to be subjected to data alignment. Based on the data amount of the data in the first data set and the second data set, a core data set and a candidate data set are determined. The data in the core data set and the data in the candidate data set are subjected to data matching, and matching result information is obtained.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data alignment method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous expansion of my country's natural gas pipeline network, periodic inspections using pipeline inspection technology are necessary to assess pipeline integrity. However, inspection data from different inspection cycles is often acquired by different manufacturers using different equipment, leading to significant differences in inspection standards, data formats, and feature naming across different cycles. This poses a significant challenge to data alignment across different inspection cycles.

[0003] In related technologies, data alignment is mainly achieved by manually comparing limited features such as circumferential weld numbers and pipe section lengths. This data alignment method is inefficient and prone to errors due to human factors. Summary of the Invention

[0004] The purpose of this application is to provide a data alignment method, apparatus, device, and storage medium that can achieve automated data alignment of pipeline inspection data in different inspection cycles, thereby improving data matching efficiency and accuracy.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a data alignment method, the method comprising: Obtain a first dataset and a second dataset for data alignment. The first dataset indicates the set of detection data within a first detection period, and the second dataset indicates the detection data within a second period. Based on the data volume in the first and second datasets, determine the core dataset and candidate datasets. The core dataset is the dataset used as a standard for alignment with data in other datasets, and the candidate datasets are the datasets compared during data alignment. Perform data matching between the data in the core dataset and the data in the candidate datasets to obtain matching result information, which describes the data matching situation between the data in the first and second datasets.

[0006] The technical solution provided in this application determines a core dataset and a candidate dataset based on a first dataset and a second dataset, and then performs data matching between the data in the core dataset and the data in the candidate dataset to obtain matching result information. This achieves automated data alignment between the data in the first dataset and the second dataset, which is more efficient and accurate than manual matching.

[0007] In some embodiments, the above-mentioned data matching between the core dataset and the candidate dataset to obtain matching results can be specifically implemented as follows: Based on the data volume ratio of the core dataset and the candidate dataset, a target candidate data subset corresponding to each data point in the core dataset is obtained in the candidate dataset, wherein the target candidate data subset is a subset of the candidate dataset. Data matching is then performed between each data point in the core dataset and the corresponding data in the target candidate data subset to obtain matching results. By determining the target candidate data subset based on the data volume ratio of the core dataset and the candidate dataset, and then performing data matching between each data point in the core dataset and the corresponding data in the target candidate data subset, the search scope during data matching is narrowed, improving the efficiency of data matching.

[0008] In some embodiments, based on the data volume ratio of the core dataset and the candidate dataset, the target candidate data subset corresponding to each data point in the core dataset in the candidate dataset is obtained. Specifically, this can be implemented as follows: Based on the data volume ratio of the core dataset and the candidate dataset, the candidate subset size of the target candidate data subset corresponding to each data point in the core dataset in the candidate dataset is obtained using a first formula, where the candidate subset size indicates the number of data points in the target candidate data subset. The Euclidean distance between each data point in the core dataset and the data points in the candidate dataset is calculated. The target candidate data subset is determined based on the Euclidean distance and the candidate subset size of the target candidate data subset.

[0009] In some embodiments, the first formula may be expressed as:

[0010] in, This refers to the size of the candidate subset of the target candidate data subset. This refers to the amount of data in the candidate dataset, ceil is the rounding function, and ratio refers to the ratio of the amount of data in the core dataset to the amount of data in the candidate dataset.

[0011] In some embodiments, data matching is performed between each data point in the core dataset and the corresponding target candidate data subset to obtain matching result information. Specifically, this can be implemented by comparing the detection mileage of each data point in the core dataset with that of the corresponding target candidate data subset to obtain a detection mileage difference, and then obtaining matching result information based on this difference. By comparing the detection mileage of each data point in the core dataset with that of the corresponding target candidate data subset to obtain a detection mileage difference, and then obtaining matching result information based on this difference, complex conditions such as detection mileage drift can be effectively addressed.

[0012] In some embodiments, data matching is performed between each data in the core dataset and the data in the corresponding target candidate data subset to obtain matching result information. Specifically, this can be implemented by comparing the wall thickness of each data in the core dataset with the data in the corresponding target candidate data subset to obtain the wall thickness difference, and obtaining the matching result information based on the wall thickness difference.

[0013] In some embodiments, data matching is performed between each data point in the core dataset and the corresponding target candidate data subset to obtain matching result information. Specifically, this can be implemented by comparing the pipe segment lengths of each data point in the core dataset with those in the corresponding target candidate data subset to obtain pipe segment length differences, and then obtaining matching result information based on these differences. By comparing the pipe segment lengths of each data point in the core dataset with those in the corresponding target candidate data subset to obtain pipe segment length differences, and then obtaining matching result information based on these differences, complex operating conditions such as variations in pipe segment lengths can be effectively addressed.

[0014] In some embodiments, a core dataset and candidate datasets are determined based on the amount of data in the first dataset and the second dataset. Specifically, this can be achieved by comparing the amount of data in the first dataset and the second dataset, identifying the dataset with less data as the core dataset, and identifying the dataset with more data as the candidate dataset. Dynamically determining the core dataset and candidate datasets based on the amount of data in the first dataset and the second dataset for data matching improves the efficiency of data matching.

[0015] In some embodiments, a core dataset and candidate datasets are determined based on the amount of data in the first dataset and the second dataset. Specifically, this can be achieved by comparing the amount of data in the first dataset and the second dataset, identifying the dataset with more data as the core dataset, and identifying the dataset with less data as the candidate dataset. Dynamically determining the core dataset and candidate datasets based on the amount of data in the first dataset and the second dataset for data matching improves the efficiency of data matching.

[0016] In some embodiments, the data alignment method provided in this application further includes: performing data preprocessing on the first dataset and the second dataset; the data preprocessing includes one or more of the following: data cleaning and standardization. By performing data preprocessing on the first dataset and the second dataset, the data matching problem caused by differences in detection standards, equipment updates, or manufacturer changes in different detection cycles can be solved.

[0017] In some embodiments, the data alignment method provided in this application further includes: obtaining manually labeled information, which indicates the correct matching relationship between data in a first dataset and a second dataset that have been manually verified; comparing the matching result information with the manually labeled information to obtain the amount of correctly matched data; calculating the ratio of the amount of correctly matched data to the total amount of matched data to obtain the matching accuracy, where the total amount of matched data indicates all the matched data included in the matching result information. By comparing the matching result information with the manually labeled information to obtain the amount of correctly matched data, and then obtaining the matching accuracy, the accuracy of data matching when using the data alignment method provided in this application can be evaluated.

[0018] In a second aspect, a data alignment device is provided, comprising: an acquisition module, a processing module, and a matching module.

[0019] The aforementioned acquisition module is used to acquire a first dataset and a second dataset to be aligned. The first dataset indicates the set of detection data within a first detection period, and the second dataset indicates the detection data within a second period.

[0020] The aforementioned processing module is used to determine the core dataset and candidate dataset based on the amount of data in the first dataset and the second dataset. The core dataset refers to the dataset that is used as a standard to align with the data in other datasets during data alignment, while the candidate dataset refers to the dataset that is compared during data alignment.

[0021] The matching module described above is used to match the data in the core dataset with the data in the candidate dataset to obtain matching result information. The matching result information is used to describe the data matching situation of the data in the first dataset and the data in the second dataset.

[0022] In some embodiments, the matching module is further configured to: obtain a target candidate data subset corresponding to each data point in the core dataset in the candidate dataset based on the data volume ratio between the core dataset and the candidate dataset, wherein the target candidate data subset is a subset of the candidate dataset; and perform data matching between each data point in the core dataset and the data in the corresponding target candidate data subset to obtain matching result information.

[0023] In some embodiments, the matching module is further configured to: based on the data volume ratio of the core dataset and the candidate dataset, obtain the candidate subset size of the target candidate data subset corresponding to each data point in the core dataset in the candidate dataset using a first formula, wherein the candidate subset size is used to indicate the number of data points in the target candidate data subset; calculate the Euclidean distance between each data point in the core dataset and the data points in the candidate dataset; and determine the target candidate data subset based on the Euclidean distance and the candidate subset size of the target candidate data subset.

[0024] In some embodiments, the first formula may be expressed as:

[0025] in, This refers to the size of the candidate subset of the target candidate data subset. This refers to the amount of data in the candidate dataset, ceil is the rounding function, and ratio refers to the ratio of the amount of data in the core dataset to the amount of data in the candidate dataset.

[0026] In some embodiments, the matching module is further configured to: compare the detection mileage of each data in the core dataset with the data in the target candidate data subset corresponding to each data to obtain the detection mileage difference, and obtain matching result information based on the detection mileage difference.

[0027] In some embodiments, the matching module is further configured to: compare the wall thickness of each data in the core dataset with the wall thickness of the data in the target candidate data subset corresponding to each data, obtain the wall thickness difference, and obtain matching result information based on the wall thickness difference.

[0028] In some embodiments, the matching module is further configured to: compare the pipe segment lengths of each data in the core dataset with the pipe segment lengths of the data in the target candidate data subset corresponding to each data, obtain the pipe segment length difference, and obtain matching result information based on the pipe segment length difference.

[0029] In some embodiments, the above-described processing module is further configured to: compare the amount of data in the first dataset and the second dataset, determine the dataset with less data as the core dataset, and determine the dataset with more data as the candidate dataset.

[0030] In some embodiments, the above-described processing module is further configured to: compare the amount of data in the first dataset and the second dataset, determine the dataset with more data as the core dataset, and determine the dataset with less data as the candidate dataset.

[0031] In some embodiments, the above-described processing module is further configured to: perform data preprocessing on the first dataset and the second dataset; the data preprocessing includes one or more of the following: data cleaning and standardization.

[0032] In some embodiments, the acquisition module is further configured to: acquire manually labeled information, which indicates the correct matching relationship between the data in the first dataset and the second dataset after manual verification. The processing module is further configured to: compare the matching result information with the manually labeled information to obtain the amount of correctly matched data; calculate the ratio of the amount of correctly matched data to the total amount of matched data to obtain the matching accuracy, where the total amount of matched data indicates the total amount of matched data included in the matching result information.

[0033] The technical effects of any implementation method in the second aspect can be found in the technical effects of any implementation method in the first aspect mentioned above, and will not be repeated here.

[0034] Thirdly, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the data alignment method described above.

[0035] Fourthly, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the data alignment method of the above.

[0036] The solutions provided in the third and fourth aspects above are used to implement the method provided in the first aspect above, and their specific implementations will not be elaborated further. The technical effects corresponding to any implementation method in the solutions provided in the third and fourth aspects above can be found in the technical effects corresponding to any implementation method in the first aspect above, and will not be elaborated further here.

[0037] It should be noted that any of the possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic diagram of a system architecture for a data alignment method provided as an exemplary embodiment; Figure 2 A flowchart illustrating a data alignment method provided for an exemplary embodiment; Figure 3 A flowchart illustrating another data alignment method provided for an exemplary embodiment; Figure 4 A flowchart illustrating another data alignment method provided for an exemplary embodiment; Figure 5 A schematic diagram of a data alignment device provided for an exemplary embodiment; Figure 6 A schematic diagram of the structure of a computer device provided for an exemplary embodiment. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] In the description of this application, it should be understood that the terms "upper," "lower," "left," "right," "front," "rear," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or relative positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Unless otherwise specified, the above-mentioned orientational descriptions can be flexibly set in practical applications, provided that the relative positional relationships shown in the accompanying drawings are satisfied.

[0042] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0043] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "communication" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection. They can refer to a direct connection or an indirect connection through an intermediate medium, or a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0044] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.

[0045] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0046] In the embodiments of this application, at least one can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any restrictions.

[0047] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0048] To facilitate understanding, the terms used in the embodiments of this application will be explained first.

[0049] Piezoelectric ultrasonic testing: A piezoelectric crystal on the detector generates high-frequency mechanical vibrations (ultrasound) under the action of an alternating electric field. The ultrasound waves propagate within the pipe wall. When encountering changes in wall thickness (such as corrosion thinning) or defects, some of the sound waves are reflected back, received by the receiving crystal, and converted into electrical signals. By calculating the time difference between the emission and reception of the sound waves, the remaining thickness of the pipe wall can be accurately measured.

[0050] Electromagnetic ultrasonic testing: A detector generates a high-frequency alternating magnetic field that acts on the pipe wall, inducing eddy currents on the pipe wall surface; simultaneously, a static bias magnetic field is applied to the detector. The eddy currents are subjected to Lorentz force in the magnetic field, exciting the pipe wall material to produce mechanical vibrations (ultrasound waves). The receiving process is the reverse process, converting the returned sound wave vibrations into electrical signals.

[0051] Magnetic flux leakage detection: A permanent magnet on the detector magnetizes a section of the pipe wall to saturation. If there are volumetric defects in the pipe wall (such as corrosion pits or mechanical damage), the magnetic permeability at that location will change, causing some magnetic field lines to leak into the air around the defect, forming a "leakage magnetic field." The sensor array detects these leakage magnetic field signals, thereby identifying the defect.

[0052] K-Nearest Neighbor (KNN) Algorithm: The KNN algorithm is essentially an instance-based learning or lazy learning algorithm. During the "training" phase, no explicit model is built; instead, all training data is stored, and all computation and decisions occur when predicting new data points. The first and only "training" step of the KNN algorithm is to memorize all training data points with labels (classification) or numerical values ​​(regression). Each data point is represented as a vector in the feature space. Next, the value of K is determined: K is the only hyperparameter in the KNN algorithm and is crucial; the choice of K significantly impacts the algorithm's performance. The third step is similarity calculation: When an unlabeled data point appears, the KNN algorithm calculates the distance between this new data point and other data points in the training set. This distance measures their similarity in the feature space. Then, the K nearest neighbors are identified: After calculating all distances, the algorithm sorts these distances in ascending order and selects the K training data points with the smallest distance to the new data point. These are the new data point's "K nearest neighbors". Finally, make a prediction (either by voting or by averaging).

[0053] It should be noted that all information (including but not limited to device information, personal information of the subject), data (including but not limited to data used for analysis, stored data, and displayed data), and signals involved in this application have been authorized by the subject or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, the first dataset, the second dataset, etc., involved in this application were all obtained with full authorization.

[0054] With the continuous expansion of my country's natural gas pipeline network, pipeline internal inspection technology, as an important means to ensure the safe operation of pipelines, is being used more and more widely. Currently, the main pipeline internal inspection technologies include piezoelectric ultrasonic testing, electromagnetic ultrasonic testing, and magnetic flux leakage testing, which are used to periodically inspect pipelines to assess their integrity.

[0055] However, in practical engineering applications, pipeline inspection data from different inspection cycles are often obtained by different manufacturers using different inspection equipment, resulting in significant differences in inspection standards, data formats, and feature naming. Therefore, it is extremely difficult to align pipeline inspection data from different inspection cycles.

[0056] The industry-standard data alignment method mainly achieves this by manually comparing limited features such as circumferential weld numbers and pipe section lengths, which will be briefly explained below.

[0057] For example, pipeline inspection data from different inspection cycles can be obtained, such as two pipeline inspection data tables from different years. By manually comparing pipeline characteristics such as circumferential weld numbers and pipe section lengths in the inspection data tables, the data alignment results of the inspection data from different inspection cycles can be obtained.

[0058] However, the data alignment methods described above are inefficient and prone to matching errors due to human factors. Especially when dealing with complex situations such as variations in pipe section length and drift in detection mileage, the accuracy of manual alignment is difficult to guarantee.

[0059] Furthermore, existing commercial software can be used to align pipeline inspection data. However, this software typically employs fixed matching rules, making it difficult to adapt to data differences between different inspection cycles. For example, when pipe section length varies by ±25%, conventional matching algorithms often fail to accurately identify the corresponding relationships. Additionally, existing methods lack effective mechanisms to handle situations where defects are missed or new defects are added due to changes in inspection thresholds. In summary, existing commercial software struggles to adapt to complex operating conditions such as variations in pipe section length and drift in inspection mileage.

[0060] In defect development trend analysis, the lack of a reliable data alignment foundation makes it difficult for engineers to accurately assess the rate of defect expansion and the degree of danger. This not only affects the accuracy of pipeline integrity assessment but also introduces uncertainty into pipeline maintenance decisions. Especially in the application scenarios of high-grade steel pipelines (such as X80), even minor defect changes can have a significant impact on pipeline safety, thus placing higher demands on data alignment accuracy.

[0061] Based on this, this application provides a data alignment method, which determines a core dataset and a candidate dataset based on a first dataset and a second dataset, and then performs data matching between the data in the core dataset and the data in the candidate dataset to obtain matching result information. This achieves automated data alignment between the data in the first dataset and the second dataset, improving data matching efficiency and accuracy.

[0062] Figure 1 A schematic diagram of a system architecture for a data alignment method provided as an exemplary embodiment, such as... Figure 1 As shown, the system includes: a computer device 101, which can be implemented as a terminal or a server.

[0063] When the computer device 101 is implemented as a server, the computer device 101 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0064] When the computer device 101 is implemented as a terminal, the computer device 101 can be a smartphone, tablet computer, laptop computer, desktop computer, etc.

[0065] Optionally, the system described above includes one or more computer devices 101. This application embodiment does not limit the number of computer devices 101.

[0066] Specifically, computer device 101 acquires a first dataset and a second dataset, and then performs data alignment on the acquired first dataset and second dataset to obtain matching result information of the first dataset and the second dataset.

[0067] Figure 2 This is a flowchart illustrating a data alignment method as provided in an exemplary embodiment. The method can be applied to a data alignment system installed in a computer device, which can be... Figure 1 Computer equipment 101 in the middle.

[0068] like Figure 2 As shown, the data alignment method provided in this application embodiment may include: Step S202: The computer device acquires the first dataset and the second dataset to be aligned.

[0069] The first dataset indicates the set of detection data within the first detection period, and the second dataset indicates the set of detection data within the second detection period. These datasets can be sets of detection data obtained from pipeline inspections. For example, the first dataset contains pipeline inspection data obtained in 2017, and the second dataset contains pipeline inspection data obtained in 2023.

[0070] Inspection cycle refers to the time period during which pipelines are inspected.

[0071] Data alignment refers to comparing data belonging to the same object in two datasets. For example, aligning pipeline inspection data from 2017 and 2023 to the same physical location allows us to obtain changes in parameters such as the depth and length of defects at the same physical location across different years. This enables us to calculate corrosion rates, defect growth patterns, and assess pipeline integrity.

[0072] Data alignment is used to achieve high-precision cross-year automatic correlation of pipeline inspection features, providing reliable data support for pipeline integrity assessment, defect development trend prediction, and preventive maintenance decisions.

[0073] For example, a computer device can obtain a first dataset and a second dataset by calling a data acquisition interface.

[0074] The data acquisition interface can be a pre-configured interface for acquiring pipeline detection data (such as the first dataset and the second dataset) for different detection cycles.

[0075] Specifically, after acquiring the first and second datasets, the computer device performs preliminary grouping processing on the data in the first and second datasets according to the pipeline feature type, resulting in multiple groups of the first and second datasets. Each group represents a type of pipeline structure feature with similar physical properties and geometric shapes. Based on this grouping, subsequent data alignment operations are performed on the first and second datasets within each group.

[0076] Among them, pipeline characteristics refer to components in a pipeline system that have specific functions, structures, or identification roles, such as circumferential welds, elbows, and flanges.

[0077] For example, data in the first and second datasets whose pipe feature is a circumferential weld are grouped into a circumferential weld group; data in the first and second datasets whose pipe feature is an elbow are grouped into an elbow group; and data in the first and second datasets whose pipe feature is a flange are grouped into a flange group.

[0078] By classifying the first and second datasets based on pipeline features, the search space for subsequent data matching is effectively reduced, making the matching process more focused on local feature sets, thus improving matching efficiency and accuracy.

[0079] Step S204: The computer device determines the core dataset and candidate dataset based on the amount of data in the first dataset and the second dataset.

[0080] It should be noted that the datasets are composed of data points. The first dataset consists of multiple first data points, and the second dataset consists of multiple second data points. The first dataset is a collection of first data points, and the second dataset is a collection of second data points. A data point can be understood as the collection of all characteristic data of a specified pipe segment with a circumferential weld number. For example, a data point may include data such as the circumferential weld number, inspection mileage, wall thickness, and pipe section length.

[0081] The amount of data in the first dataset is the number of data points in the first dataset; the amount of data in the second dataset is the number of data points in the second dataset.

[0082] For example, suppose the first dataset contains first data point 1, first data point 2, and first data point 3; and the second dataset contains second data point 1, second data point 2, second data point 3, second data point 4, second data point 5, and second data point 6. Then the data volume in the first dataset is 3, and the data volume in the second dataset is 6.

[0083] The core dataset refers to the dataset used as a standard for data alignment with other datasets. The core dataset can be either the first dataset or the second dataset.

[0084] A candidate dataset is a dataset that is compared during data alignment. A candidate dataset can be either the first dataset or the second dataset.

[0085] When the core dataset is the first dataset, the candidate dataset is the second dataset; conversely, when the core dataset is the second dataset, the candidate dataset is the first dataset.

[0086] Optionally, the computer device compares the amount of data in the first dataset and the second dataset, and determines the dataset with less data as the core dataset and the dataset with more data as the candidate dataset; or, the computer device compares the amount of data in the first dataset and the second dataset, and determines the dataset with more data as the core dataset and the dataset with less data as the candidate dataset.

[0087] Step S206: The computer device performs data matching between the data in the core dataset and the data in the candidate dataset to obtain matching result information.

[0088] The matching result information describes the data matching situation between the data in the first dataset and the data in the second dataset.

[0089] Matching results can include both matches and no matches.

[0090] In this context, a matching pair refers to a data point in the first dataset that successfully matches data in the second dataset. For example, if the core dataset is a dataset of pipeline inspection data from 2017 and the candidate dataset is a dataset of pipeline inspection data from 2023, a matching pair can be data points in the 2017 and 2023 inspection data that correspond to the same physical location (such as the same bend or the same pipe segment).

[0091] Unmatched items include unmatched core points and unmatched candidate points. For example, unmatched items may be misaligned data caused by pipeline rerouting or replacement between two detections.

[0092] Unmatched core points refer to core data points in the core dataset that do not successfully match candidate data points in the candidate dataset. For example, if the core dataset is a dataset of pipeline inspection data from 2017 and the candidate dataset is a dataset of pipeline inspection data from 2023, unmatched core points could be old defects that "disappeared" between the 2017 and 2023 inspections due to pipeline repair, replacement, or improved inspection accuracy; or, unmatched core points could be defects identified in the 2017 inspection but not detected in the 2023 inspection.

[0093] Unmatched candidate points refer to candidate data points in the candidate dataset that do not successfully match core data points in the core dataset. For example, if the core dataset is a dataset of pipeline inspection data from 2017 and the candidate dataset is a dataset of pipeline inspection data from 2023, unmatched candidate points could be defects that arise between the two inspections in 2017 and 2023.

[0094] It should be noted that the handling strategy for unmatched items can be flexibly configured according to business needs, employing either a strict or lenient mode. The technical solution presented in this application has good scalability and can be upgraded in the future to support fuzzy matching evaluation. For example, allowing a certain range of detection mileage deviation or introducing advanced judgment conditions such as feature similarity can make the evaluation system more closely aligned with actual engineering needs.

[0095] Data matching refers to comparing and identifying data points in a first dataset and a second dataset.

[0096] In some embodiments, during the data matching process, the computer device obtains the target candidate data subset corresponding to each data point in the core dataset in the candidate dataset based on the data volume ratio between the core dataset and the candidate dataset. Then, it performs data matching between each data point in the core dataset and the corresponding data in the target candidate data subset to obtain matching result information.

[0097] The target candidate data subset is a subset of the candidate dataset.

[0098] The ratio of data size between the core dataset and the candidate dataset is the ratio of the data size in the core dataset to the data size in the candidate dataset. This ratio can be calculated using the following formula: .

[0099] in, This refers to the ratio of the core dataset to the candidate dataset. data1 refers to the candidate dataset, and data2 refers to the core dataset. The len function is used to return the length or number of elements of an object (such as a string, list, tuple, dictionary, and set).

[0100] For example, if the core dataset has 6 data points and the candidate dataset has 15 data points, the ratio of the core dataset to the candidate dataset can be calculated using the formula above. It is 2.5.

[0101] Based on the data volume ratio of the core dataset and the candidate dataset, the size of the target candidate data subset corresponding to each data point in the core dataset is obtained using the first formula. Then, the Euclidean distance between each data point in the core dataset and the data points in the candidate dataset is calculated. Finally, the target candidate data subset is determined based on the Euclidean distance and the size of the target candidate data subset.

[0102] The candidate subset size is used to indicate the number of data points in the target candidate data subset.

[0103] In some embodiments, the first formula can be expressed as: .

[0104] in, This refers to the size of the candidate subset of the target candidate data subset. This refers to the amount of data in the candidate dataset, ceil is the rounding function, and ratio refers to the ratio of the amount of data in the core dataset to the amount of data in the candidate dataset.

[0105] For example, when the core dataset contains 6 data points and the candidate dataset contains 15 data points, the ratio of the core dataset to the candidate dataset is 2.5, i.e., ratio = 2.5. Then, using the first formula, the size of the candidate subset is calculated to be 6. The specific calculation process is as follows: .

[0106] Determining the size of the candidate subset based on the ratio of data volume between the core dataset and the candidate dataset avoids missed matches and prevents invalid searches, thus ensuring a reasonable number of candidate matches are obtained under different data densities and avoiding omissions or false matches. Furthermore, the size of the candidate subset is dynamically determined based on the ratio of data volume between the first and second datasets, breaking through the limitation of the traditional fixed K value in the K-nearest neighbors algorithm.

[0107] After obtaining the size of the candidate subset, calculate the Euclidean distance between the detection mileage of the data points in the core dataset and the detection mileage of all data points in the candidate dataset. Then, sort the calculated Euclidean distances in ascending order to obtain the sorting results. Select the data points with the smallest Euclidean distance in the sorting results to determine the target candidate data subset.

[0108] For example, when the size of the candidate subset is 6, the 6 data points with the smallest Euclidean distance in the permutation result are selected and set together to obtain the target candidate data subset.

[0109] For example, the Euclidean distance between detection mileages can be calculated using the following formula: .

[0110] Where p refers to a data point in the core dataset, and q refers to a data point in the candidate dataset. This refers to the detection mileage of data points in the core dataset. It refers to the detection mileage of data points in the candidate dataset.

[0111] It should be noted that the target candidate data subset can also be understood as the dynamic candidate set that is closest to the core data point.

[0112] In one possible implementation, the computer device compares the detection mileage of each data point in the core dataset with the detection mileage of the corresponding target candidate data subset, obtaining a detection mileage difference. Then, based on the detection mileage difference, matching result information is obtained.

[0113] The individual data points in the core dataset can be referred to as core data points.

[0114] Specifically, the difference in detection mileage between the core data point and the data in the target candidate data subset can be calculated. If the difference in detection mileage between the core data point and the data in the target candidate data subset is less than the detection mileage difference threshold, then the data in the target candidate data subset matches the core data point, and the matching result information is obtained.

[0115] The detection mileage difference threshold can be set according to the actual situation. For example, the detection mileage difference threshold can be ±5 meters. There is no limit to the detection mileage difference threshold here.

[0116] For example, when comparing pipeline inspection data from 2017 and 2023, inspection mileage is a key benchmark for achieving data alignment. Inspection mileage is used to ensure that the two inspection results are analyzed at the same location on the pipeline, thereby accurately calculating the rate of defect change.

[0117] In one possible implementation, the computer device compares the wall thickness of each data point in the core dataset with the corresponding target candidate data subset to obtain a wall thickness difference. Then, based on this wall thickness difference, matching result information is obtained.

[0118] Specifically, the wall thickness difference between the core data point and the data in the target candidate data subset can be obtained by calculating the difference between the wall thickness of the core data point and the data in the target candidate data subset. If the wall thickness difference corresponding to the data in the target candidate data subset is less than the wall thickness difference threshold, then the data in the target candidate data subset matches the core data point, and the matching result information is obtained.

[0119] The wall thickness difference threshold can be set according to the actual situation. For example, the wall thickness difference threshold can be ±5%. There is no limit to the wall thickness difference threshold here.

[0120] In one possible implementation, the computer device compares the pipe segment lengths of each data point in the core dataset with the pipe segment lengths of the corresponding target candidate data subsets to obtain pipe segment length differences; based on the pipe segment length differences, matching result information is obtained.

[0121] Specifically, the difference in pipe segment length between the core data point and the data in the target candidate data subset can be calculated to obtain the pipe segment length difference between the core data point and the data in the target candidate data subset. If the pipe segment length difference corresponding to the data in the target candidate data subset is less than the pipe segment length difference threshold, then the data in the target candidate data subset matches the core data point, and the matching result information is obtained.

[0122] The threshold for the difference in pipe section length can be set according to the actual situation. For example, the threshold for the difference in pipe section length can be ±20%. There is no limit to the threshold for the difference in pipe section length here.

[0123] It should be noted that the above-mentioned methods of comparing by detecting mileage, wall thickness, or pipe section length can achieve data matching results individually or in combination. This application does not specifically limit this method.

[0124] For example, after determining the core dataset and candidate datasets, this application employs a dynamic candidate set K-nearest neighbor algorithm to perform preliminary matching (initial screening) on ​​the candidate datasets, identifying a target candidate data subset. Then, based on the target candidate data subset, independent matching is performed on each core data point. During the independent matching process, two levels of engineering verification are implemented: first, coarse matching is performed with an absolute mileage deviation of ±5 meters; then, fine matching is performed using dual relative tolerance thresholds of ±20% for pipe section length and ±5% for wall thickness, obtaining the matching results. After the matching process is completed, the matching results are fully recorded.

[0125] For example, the core dataset includes core data point 1, core data point 2, and core data point 3; the candidate dataset includes candidate data point 1, candidate data point 2, candidate data point 3, candidate data point 4, candidate data point 5, candidate data point 6, candidate data point 7, candidate data point 8, and candidate data point 9.

[0126] First, through the formula: The ratio of data volume between the core dataset and the candidate dataset was calculated. The value is 3, then the first formula is used: The calculated size of the candidate subset is 6.

[0127] The following describes the process of performing independent matching for each core data point, taking core data point 1 as an example.

[0128] After obtaining the size of the candidate subset, calculate the Euclidean distance of the detection mileage between core data point 1 and candidate data points 1, 2, 3, 4, 5, 6, 7, 8, and 9, and then sort the calculated Euclidean distances.

[0129] For example, the sorting result according to Euclidean distance in ascending order is as follows: Candidate data point 1, Candidate data point 3, Candidate data point 5, Candidate data point 6, Candidate data point 8, Candidate data point 9, Candidate data point 2, Candidate data point 3, Candidate data point 7. Based on the candidate subset size of 6, it can be determined that the target candidate data subset includes candidate data point 1, candidate data point 3, candidate data point 5, candidate data point 6, candidate data point 8, and candidate data point 9.

[0130] Then, based on the target candidate data subset, a coarse matching is performed with an absolute mileage deviation of ±5 meters to obtain the first candidate data subset. For example, the first candidate data subset includes candidate data point 1, candidate data point 3, candidate data point 6, and candidate data point 8.

[0131] Then, based on the first candidate data subset, a fine-matching process is performed using a relative tolerance threshold of ±20% of the pipe section length to obtain the second candidate data subset. For example, the second candidate data subset includes candidate data point 3 and candidate data point 6.

[0132] Then, based on the second candidate data subset, a fine-matching process is performed using a relative tolerance threshold of ±5% for wall thickness to obtain the third candidate data subset. For example, the third candidate data subset includes candidate data point 3.

[0133] The candidate data point that matches the core data point 1 is then the candidate data point 3, and the matching result information is obtained.

[0134] Furthermore, after obtaining the matching results, the candidate dataset and the core dataset are swapped, that is, the candidate dataset is used as the "core dataset" and the core dataset is used as the "candidate dataset", and the matching process is executed again to obtain secondary matching results, thus forming a two-way verification mechanism.

[0135] The secondary matching result information is used to indicate the matching result information obtained after the candidate dataset and the core dataset have swapped roles and the matching process has been executed.

[0136] After obtaining the secondary matching results, the initial matching results and the secondary matching results are cross-compared to retain the optimal matching combination to resolve one-to-many or many-to-one conflicts. At the same time, suspicious matching items are marked for manual review.

[0137] Among them, one-to-many conflict refers to the situation in the matching result information where one core data point matches multiple candidate data points, and many-to-one conflict refers to the situation in the matching result information where multiple core data points match one candidate data point.

[0138] The data alignment method provided in this application achieves high-precision automatic alignment of cross-cycle detection data by dynamically switching the matching core direction and adaptive candidate set size, combined with the dual engineering constraints of wall thickness and pipe section length. It can effectively cope with complex actual engineering conditions such as pipe section length variation, detection mileage drift, and uneven data volume of detection data in different detection cycles to be aligned.

[0139] In addition, to evaluate the accuracy of the matching results, the precise matching accuracy can be calculated by comparing the matching results with manually labeled matching data.

[0140] In some embodiments, manually labeled information is obtained. The matching results are compared with the manually labeled information to obtain the amount of correctly matched data. The ratio of the amount of correctly matched data to the total amount of matched data is calculated to obtain the matching accuracy.

[0141] The manually labeled information indicates the correct matching relationship between the data in the first dataset and the second dataset, which have been manually verified. The total matched data volume indicates the total amount of matched data included in the matching results. The correctly matched data volume refers to the amount of data points that are correctly matched in the matching results.

[0142] Specifically, the process involves acquiring both the automatic matching result file and the manually labeled file, extracting their core fields, and then mapping the matching information in the automatic matching result file to the manually labeled information in the manually labeled file based on a unique identifier (such as the circumferential weld number). By comparing the matching relationships in the matching result information with the correct matching relationships in the manually labeled information, the amount of correctly matched data in the matching result information is obtained. Finally, based on the amount of correctly matched data and the total amount of matched data, the matching accuracy is calculated using a formula.

[0143] The automatic matching results file contains the first dataset, the second dataset, and the matching results information. The manually labeled file stores the manually labeled information.

[0144] Specifically, the matching accuracy can be calculated using the following formula: .

[0145] in, This refers to the matching accuracy. This refers to the amount of data that is correctly matched. This refers to the total amount of matched data.

[0146] When calculating the matching accuracy, the total amount of data in the first and second datasets, the amount of correctly matched data in the matching results information, and all values ​​in the matching accuracy are accurate to two decimal places.

[0147] It should be noted that data that fails to match is marked as "unmatched". Data that is completely identical to the manually labeled data is marked as "correct match", and data that fails to match in the automatic matching data but matches successfully in the manually labeled data is marked as "match error". Only when the data in the automatic matching data and the manually labeled data are completely identical is it counted as a correct match. This requires that the core identifiers correspond completely in both files, thus avoiding misalignment issues when merging data from the automatic matching result file and the manually labeled file.

[0148] Matching accuracy provides crucial data support for algorithm optimization, helping to improve the reliability of automatic matching. Furthermore, the calculation of matching accuracy plays a vital role in practical engineering, guiding algorithm parameter optimization and monitoring system stability. When the matching rate falls below a preset threshold, an automatic optimization process is triggered to ensure the system maintains optimal performance, providing reliable assurance for the automated processing of pipeline inspection data.

[0149] Furthermore, based on the matching results and matching accuracy, a structured alignment report is generated. The structured alignment report may include a list of matching pairs, details of unmatched items, and matching rates under each pipeline feature category.

[0150] The matching pair list refers to the list of data points that are successfully matched in the first and second datasets.

[0151] The details of unmatched items refer to the information related to data points that did not match successfully in the first and second datasets.

[0152] The structured alignment report provides a high-quality time-series data foundation for subsequent defect development analysis and trend prediction. It also records complete matching logs and discrepancy information, facilitating later model optimization and manual intervention.

[0153] In summary, the technical solution provided in this application determines a core dataset and a candidate dataset based on a first dataset and a second dataset, and then matches the data in the core dataset and the data in the candidate dataset to obtain matching result information. This achieves automated data alignment of the data in the first dataset and the second dataset, improving data matching efficiency and accuracy. Furthermore, after acquiring the first dataset and the second dataset, the computer device needs to perform data preprocessing to provide a reliable data foundation for subsequent data alignment operations. That is, step S203 is included between steps S202 and S204.

[0154] Figure 3 This is a flowchart illustrating another data alignment method provided as an exemplary embodiment. The method can be executed by a computer device.

[0155] Step S203: The computer device performs data preprocessing on the data in the first dataset and the second dataset.

[0156] Specifically, after obtaining the first dataset and the second dataset, data preprocessing is performed on the data in the first dataset and the second dataset.

[0157] Since the data in the first and second datasets are raw detection data from different detection periods, and the detection methods and equipment used in different detection periods may be different, there are differences in the detection standards, data formats, and feature names of the data in the first and second datasets. Therefore, data preprocessing is required before aligning the data in the first and second datasets.

[0158] Data preprocessing refers to the processing of data before data alignment to improve data quality and make it more suitable for subsequent processing and analysis. Data preprocessing includes one or more of the following: data cleaning and standardization.

[0159] Data cleaning can include steps such as handling outliers, filling in missing values, removing duplicate data, and converting the data type of numerical data.

[0160] For example, missing values ​​can be filled using the mean, median, or interpolation; outliers can be detected and corrected using the standard score (Z-score) or interquartile range (IQR) method.

[0161] Standardization refers to the process of uniformly naming the key fields of data in datasets from different testing periods.

[0162] Specifically, to address the issue of inconsistent field naming across different testing periods, key fields are extracted from the first and second datasets by pre-establishing field mapping rules. Based on these key fields, their corresponding field names are matched within the field mapping rules, and the key fields are then named accordingly, thus standardizing them. Furthermore, while unifying field names, the original numerical information is fully preserved, providing a complete data traceability foundation for subsequent manual review.

[0163] Among them, the field mapping rule refers to the correspondence between key fields and field names. For example, "circumferential weld number" and "circumferential weld ID" both correspond to the name "code"; "detection mileage" corresponds to "mileage"; and "feature recognition" corresponds to "feature_name".

[0164] Key fields refer to fields in the data points that have a significant impact on data alignment, such as circumferential weld number, inspection mileage, wall thickness, and feature identification (such as "crack", "corrosion", "weld anomaly" etc.).

[0165] Feature recognition can be understood as the detection of defects.

[0166] This application effectively solves the problem of inconsistent field naming between different testing manufacturers through field mapping rules, achieving data alignment capabilities across testing cycles and platforms. By preprocessing the data in the first and second datasets, it resolves the data matching challenges caused by differences in testing standards, equipment updates, or manufacturer changes across different testing cycles.

[0167] In summary, the technical solution provided in this application determines the core dataset and candidate dataset based on the first dataset and the second dataset, and then performs data matching between the data in the core dataset and the data in the candidate dataset to obtain matching result information. This achieves automated data alignment between the data in the first dataset and the second dataset, improving data matching efficiency and accuracy.

[0168] This application utilizes a dynamic grouping mechanism based on pipeline feature names, combined with a two-stage algorithm of K-nearest neighbor coarse matching in mileage space and multi-dimensional fine matching of engineering features, to achieve high-precision cross-year automatic association of pipeline detection features. This provides reliable data support for pipeline integrity assessment, defect development trend prediction, prediction of pipeline remaining life, and preventive maintenance decisions. Ultimately, it achieves comprehensive benefits such as ensuring safe pipeline operation, optimizing maintenance resource allocation, and reducing operational risks. Furthermore, this strategy significantly improves the robustness of cross-period heterogeneous data matching, with actual measurements showing a 12.7% increase in algorithm accuracy, providing key technical support for the overall level of intelligence.

[0169] Furthermore, the data alignment method provided in this application can also be extended to the analysis of detection data of other pressure pipeline systems such as urban gas pipeline networks and process pipelines in chemical industrial parks. It has wide engineering applicability and meets the technical requirements of TSGD7003-2020 "Periodic Inspection Rules for Pressure Pipelines" and GB 32167-2015 "Integrity Management Standard for Oil and Gas Transmission Pipelines".

[0170] Compared to traditional manual comparison methods, the data alignment method of this application can reduce the alignment time of two internal inspection data tables from 30-60 minutes to 5 minutes, while improving the matching accuracy to over 95%, significantly reducing the risk of human error. Furthermore, the technical solution provided in this application is particularly suitable for the periodic internal inspection data comparison and analysis of long-distance pipelines (such as the West-East Gas Pipeline). It achieves automatic data alignment, improving the accuracy and efficiency of data alignment.

[0171] For example, Figure 4 A flowchart illustrating another data alignment method provided for an exemplary embodiment. This method is performed by a computer device.

[0172] Step S401: The computer device acquires the first dataset and the second dataset to be aligned.

[0173] For a description of this step, please refer to step S202; it will not be elaborated further here.

[0174] In step S402, the computer device performs data preprocessing on the data in the first dataset and the second dataset.

[0175] For a description of this step, please refer to step S203; it will not be elaborated further here.

[0176] In step S403, the computer device groups the first dataset and the second dataset according to the pipeline feature name.

[0177] For a description of this step, please refer to step S202; it will not be elaborated further here.

[0178] Step S404: The computer device compares the amount of data in the first dataset and the second dataset.

[0179] Step S405: The computer device determines whether the amount of data in the first dataset is greater than the amount of data in the second dataset.

[0180] The computer device determines whether the amount of data in the first dataset is greater than the amount of data in the second dataset. If the amount of data in the first dataset is greater than the amount of data in the second dataset, then step S407 is executed; otherwise, step S406 is executed.

[0181] In step S406, the computer device determines the first dataset as the core dataset and the second dataset as the candidate dataset.

[0182] In step S407, the computer device determines the second dataset as the core dataset and the first dataset as the candidate dataset.

[0183] In step S408, the computer device performs data matching between the core dataset and the candidate dataset to obtain matching result information.

[0184] The matching results include the best match, unmatched core data points, and unmatched candidate data points.

[0185] Specifically, KNN is used to search for neighboring candidate points of the core data points to obtain a target candidate data subset. Based on the target candidate data subset, data matching is performed to obtain matching results. The matching results are saved to a comma-separated values ​​(CSV) file.

[0186] A CSV file is a plain text file used to store tabular data. CSV files use commas to separate different data fields.

[0187] The specific process of matching the core dataset and the candidate dataset can be found in step S206, and will not be elaborated further here.

[0188] Step S409: The computer device calculates the matching accuracy.

[0189] For a description of this step, please refer to step S206; it will not be elaborated upon here.

[0190] The foregoing mainly describes the solution provided in this application. Accordingly, this application also provides a data alignment device for implementing the above-described method embodiments.

[0191] like Figure 5 The diagram shows the structure of a data alignment device, which may include an acquisition module 501, a processing module 502, and a matching module 503. The acquisition module 501 is used to perform... Figure 2 or Figure 3 The illustrated method includes step S202; the processing module 502 is used to execute... Figure 2 or Figure 3 The illustrated method includes step S204; the matching module 503 is used to perform... Figure 2 or Figure 3 The operation of step S206.

[0192] In some embodiments, the data alignment device includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] This application embodiment can divide the data alignment device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one data alignment module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0194] like Figure 6As shown, an exemplary embodiment of this application provides a computer device that may include a processor 601, a bus 602, a communication interface 603, and a memory 604. The processor 601, the memory 604, and the communication interface 603 communicate with each other via the bus 602. It should be understood that this application does not limit the number of processors and memories in the network device.

[0195] Bus 602 can be a PCI bus, an Extended Industry Standard Architecture (EISA) bus, or a UB bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 602 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 602 may include a path for transmitting information between various components of the network device (e.g., memory 604, processor 601, communication interface 603).

[0196] Processor 601 may include any one or more processors such as CPU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0197] Memory 604 may include volatile memory, such as random access memory (RAM). Processor 601 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0198] The communication interface 603 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between network devices and other devices or communication networks.

[0199] The memory 604 stores executable program code, which the processor 601 executes to implement the functions of the aforementioned method embodiments. That is, the memory 604 stores instructions for executing the aforementioned data alignment method.

[0200] In another aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the data alignment method provided in the above-described method embodiments.

[0201] In another aspect, a computer program product is provided, which includes a computer program or instructions that, when executed by a processor, implement the data alignment method provided in the above-described method embodiments.

[0202] Through the above description of the implementation methods, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the module can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, modules, and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0203] Since the data alignment module, computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above method, the technical effects obtained can also be referred to the above method embodiments. The embodiments of the present invention will not be described again here.

[0204] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a network device. Of course, the processor and storage medium can also exist as discrete components in the network device.

[0205] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer programs or instructions. When a computer program or instruction is loaded and executed on a computer, the processes or functions of the embodiments of this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable module. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0206] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data alignment method, characterized in that, The method includes: Obtain a first dataset and a second dataset to be aligned, wherein the first dataset is used to indicate the set of detection data within a first detection period, and the second dataset is used to indicate the detection data within a second period. Based on the amount of data in the first dataset and the second dataset, a core dataset and a candidate dataset are determined. The core dataset refers to the dataset that is used as a standard to align with data in other datasets during data alignment, and the candidate dataset refers to the dataset that is compared during data alignment. The data in the core dataset and the data in the candidate dataset are matched to obtain matching result information, which is used to describe the data matching situation between the data in the first dataset and the data in the second dataset.

2. The method according to claim 1, characterized in that, The step of matching the data in the core dataset and the data in the candidate dataset to obtain matching result information includes: Based on the data volume ratio of the core dataset and the candidate dataset, a target candidate data subset corresponding to each data in the core dataset is obtained in the candidate dataset, and the target candidate data subset is a subset of the candidate dataset; The data in the core dataset is matched with the data in the target candidate data subset corresponding to each data to obtain the matching result information.

3. The method according to claim 2, characterized in that, The step of obtaining the target candidate data subset corresponding to each data point in the core dataset in the candidate dataset based on the data volume ratio of the core dataset and the candidate dataset includes: Based on the data volume ratio of the core dataset and the candidate dataset, the candidate subset size of the target candidate data subset corresponding to each data in the core dataset in the candidate dataset is obtained by the first formula. The candidate subset size is used to indicate the number of data in the target candidate data subset. Calculate the Euclidean distance between each data point in the core dataset and the data points in the candidate dataset; The target candidate data subset is determined based on the Euclidean distance and the size of the candidate subset of the target candidate data subset.

4. The method according to claim 3, characterized in that, The first formula is: in, This refers to the size of the candidate subset of the target candidate data subset. This refers to the amount of data in the candidate dataset, ceil is the rounding function, and ratio refers to the ratio of the amount of data in the core dataset to the amount of data in the candidate dataset.

5. The method according to claim 2, characterized in that, The step of matching each data point in the core dataset with the data in the corresponding target candidate data subset to obtain the matching result information includes: The detection mileage of each data point in the core dataset is compared with that of the data points in the corresponding target candidate data subset to obtain the detection mileage difference; based on the detection mileage difference, the matching result information is obtained. And / or, The wall thickness of each data point in the core dataset is compared with the wall thickness of the data in the corresponding target candidate data subset to obtain the wall thickness difference; based on the wall thickness difference, the matching result information is obtained. And / or, The pipe segment lengths of each data point in the core dataset are compared with the pipe segment length differences of the corresponding target candidate data subsets to obtain the matching result information.

6. The method according to any one of claims 1 to 5, characterized in that, The determination of the core dataset and candidate dataset based on the data volume of the first dataset and the second dataset includes: By comparing the amount of data in the first dataset and the second dataset, the dataset with less data is identified as the core dataset, and the dataset with more data is identified as the candidate dataset. or, By comparing the amount of data in the first dataset and the second dataset, the dataset with more data is determined as the core dataset, and the dataset with less data is determined as the candidate dataset.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Data preprocessing is performed on the data in the first dataset and the second dataset; the data preprocessing includes one or more of the following: data cleaning, standardization processing.

8. A data alignment device, characterized in that, The device includes: The acquisition module is used to acquire a first dataset and a second dataset to be aligned, wherein the first dataset is used to indicate the set of detection data in a first detection period, and the second dataset is used to indicate the detection data in a second period. The processing module is used to determine a core dataset and a candidate dataset based on the amount of data in the first dataset and the second dataset. The core dataset refers to the dataset that is used as a standard to align with data in other datasets when performing data alignment. The candidate dataset refers to the dataset that is compared when performing data alignment. The matching module is used to match the data in the core dataset and the data in the candidate dataset to obtain matching result information. The matching result information is used to describe the data matching situation of the data in the first dataset and the data in the second dataset.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the data alignment method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the data alignment method as described in any one of claims 1-7.

Images