A tubing detection method and device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for oil pipe inspection, relating to the field of oil pipe inspection. The method includes: acquiring an image of an oil pipe to be inspected, and extracting features from the image to obtain a corresponding feature map; using a preset key point detection model to predict key points in the feature map, thereby predicting the positions of a preset number of first key points representing the position and shape of the oil pipe in the image of the oil pipe to be inspected; determining the position and shape of the oil pipe in the image of the oil pipe to be inspected based on the positions of the first key points; the key point detection method can be used to detect flexible targets such as oil pipes, thereby enabling effective detection of oil pipes in various environments and postures, and improving the robustness of oil pipe inspection.

Description

Technical Field

[0001] This invention relates to the field of oil pipe inspection, and in particular to an oil pipe inspection method, apparatus, electronic device and storage medium. Background Technology

[0002] In the petrochemical industry, target identification and posture detection of oil pipes are crucial for better standardizing the execution of oil stabilization and unloading operations. Among related technologies, traditional machine vision methods that use gradient calculation to identify the shape of oil pipes are easily affected by complex backgrounds. If target detection algorithms are used to identify oil pipe targets, considering that oil pipes are flexible objects with varying shapes, it is not only difficult to locate the oil pipe using target detection boxes, but the complexity of the oil pipe's shape also makes it difficult for the model to converge, posing significant challenges to model training. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for oil pipe inspection, which can detect flexible targets such as oil pipes by means of key point detection, and can effectively detect oil pipes in various environments and postures, thereby improving the robustness of oil pipe inspection.

[0004] To solve the above-mentioned technical problems, the present invention provides a method for inspecting oil pipes, comprising:

[0005] Acquire an image of the oil pipe to be detected, and extract features from the image to obtain the corresponding feature map.

[0006] A preset key point detection model is used to predict key points in the feature map, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be tested.

[0007] The location and shape of the oil pipe are determined in the image of the oil pipe to be detected based on the location of the first key point.

[0008] Optionally, the step of predicting key points in the feature map using a preset key point detection model includes:

[0009] The feature map is input into the preset key point detection model to generate a confidence map corresponding to each first key point; the confidence map is used to record the confidence level of each pixel in the oil pipe image to be detected as the first key point corresponding to the confidence map;

[0010] The location of each of the first key points in the image of the oil pipe to be detected is determined based on the confidence map.

[0011] Optionally, the training process of the preset keypoint detection model includes:

[0012] Acquire training tubing images and extract features from the training tubing images to obtain corresponding training feature maps; the training tubing images are labeled with the preset number of tubing key points, and the positions of each tubing key point and the affinity between each tubing key point are recorded;

[0013] The first network branch of the preset key point detection model is used to predict key points in the training feature map to predict the position of the second key point corresponding to each of the oil pipe key points in the training oil pipe image. The second network branch of the preset key point detection model is used to predict key point affinity in the training feature map to predict the affinity between each of the second key points.

[0014] A first loss value is generated by using the position of each second key point and the position of the corresponding oil pipe key point, and a second loss value is generated by using the affinity between each second key point and the affinity between the corresponding oil pipe key points.

[0015] A total loss value is generated using the first loss value and the second loss value, and it is determined whether the total loss value is greater than a preset threshold.

[0016] If so, the total loss value is used to update the parameters of the prediction model, and based on the updated prediction model, the steps of using the first network branch of the preset key point detection model to predict key points in the training feature map, and using the second network branch of the preset key point detection model to predict key point affinity in the training feature map are entered.

[0017] If not, then the training of the preset key point detection model is complete.

[0018] Optionally, the key points of the pipeline include turning key points and fitting key points. The turning key points are marked at the starting position and the bending position where the bending angle is greater than a preset value. The fitting key points are marked between adjacent turning key points. The step of using the second network branch of the preset key point detection model to predict the key point affinity of the training feature map to predict the affinity between each second key point includes:

[0019] Based on the arrangement order of the key points of the oil pipe, add all the second key points to the key point sequence;

[0020] Set the first second key point in the key point sequence as the key point to be calculated, and determine the type of the oil pipe key point corresponding to the key point to be calculated;

[0021] If the type is a turning point, then in the key point sequence, find the next target second key point that is adjacent to the key point to be calculated and corresponds to the turning point, and set the vector from the key point to be calculated to the target second key point as the affinity value corresponding to the key point to be calculated.

[0022] If the type is fitting key point, determine two target second key points in the key point sequence that are adjacent to the key point to be calculated and both correspond to turning key points; determine the straight line passing through the two target second key points, and set the distance between the key point to be calculated and the straight line as the affinity value corresponding to the key point to be calculated;

[0023] When calculating the affinity value of the current key point to be calculated, if it is determined that there is a second key point in the key point sequence for which the affinity value has not been calculated, the key point to be calculated is updated to the next second key point, and the process proceeds to the step of determining the type of the oil pipe key point corresponding to the key point to be calculated.

[0024] If it is determined that there is no second keypoint in the keypoint sequence for which the affinity value has not been calculated, then the keypoint affinity prediction is completed.

[0025] Optionally, generating the second loss value using the affinity between each second key point and the corresponding affinity between tubing key points includes:

[0026] Set the first second key point in the key point sequence as the key point to be processed, and determine the type of the oil pipe key point corresponding to the key point to be processed;

[0027] If the type is a turning point, then determine the cosine angle between the vector corresponding to the key point to be processed and the vector corresponding to the oil pipe key point to be processed, and set the cosine angle as the affinity loss value corresponding to the key point to be processed.

[0028] If the type is fitting key point, then determine the distance loss value between the distance corresponding to the key point to be processed and the distance corresponding to the oil pipe key point corresponding to the key point to be processed, and set the distance loss value as the affinity loss value corresponding to the key point to be processed.

[0029] When calculating the affinity loss value of the current key point to be processed, if it is determined that there is a second key point in the key point sequence for which the affinity loss value has not been calculated, the key point to be processed is updated to the next second key point, and the type of the tubing key point corresponding to the key point to be processed is determined.

[0030] If it is determined that there are no second keypoints in the keypoint sequence for which the affinity loss value has not been calculated, then the second loss value is generated using the affinity loss values ​​corresponding to all second keypoints.

[0031] Optionally, the step of using the first network branch of the preset keypoint detection model to predict keypoints in the training feature map to predict the position of the second keypoint corresponding to each of the pipeline keypoints in the training pipeline image, and using the second network branch of the preset keypoint detection model to predict keypoint affinity in the training feature map to predict the affinity between each of the second keypoints, includes:

[0032] Set the training feature map as the feature to be processed, and initialize the current stage to 1;

[0033] The features to be processed are input into the first network branch of the preset key point detection model corresponding to the current stage to predict the position of each second key point in the training oil pipe image at the current stage, and the features to be processed are input into the second network branch of the preset key point detection model corresponding to the current stage to predict the affinity between each second key point at the current stage.

[0034] The first sub-loss value is generated by using the position of each second key point and the position of the corresponding oil pipe key point in the current stage, and the second sub-loss value is generated by using the affinity between each second key point and the affinity between the corresponding oil pipe key points in the current stage.

[0035] The intermediate features are obtained by fusing the training feature map with the positions of each second key point in the current stage and the affinity between each second key point in the current stage.

[0036] If it is determined that the current stage is less than the preset stage value, then the current stage is incremented by one, the intermediate feature is set as the feature to be processed, and the process proceeds to the step of inputting the feature to be processed into the first network branch of the preset key point detection model corresponding to the current stage.

[0037] Accordingly, the step of generating a first loss value using the positions of each of the second key points and the corresponding positions of the tubing key points, and generating a second loss value using the affinity between each of the second key points and the corresponding affinity between the tubing key points, includes:

[0038] If it is determined that the current stage value is equal to the preset stage value, then the first loss value is calculated using the first sub-loss value of each stage, and the second loss value is calculated using the second sub-loss value of each stage.

[0039] The present invention also provides an oil pipe inspection device, comprising:

[0040] The feature extraction module is used to acquire the image of the oil pipe to be detected and to extract features from the image of the oil pipe to be detected to obtain the corresponding feature map.

[0041] The key point prediction module is used to predict key points in the feature map using a preset key point detection model, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be tested.

[0042] The oil pipe determination module is used to determine the position and shape of the oil pipe in the oil pipe image to be detected based on the position of the first key point.

[0043] Optionally, the key point prediction module includes:

[0044] The confidence map generation submodule is used to input the feature map into the preset key point detection model, so as to generate a confidence map corresponding to each predicted key point using the preset key point detection model; the confidence map is used to record the confidence level of each pixel in the pipeline image to be detected as the first key point corresponding to the confidence map;

[0045] The key point prediction submodule is used to determine the position of each of the first key points in the image of the oil pipe to be detected based on the confidence map.

[0046] The present invention also provides an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor is used to implement the tubing detection method as described above when executing the computer program.

[0049] The present invention also provides a storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the oil pipe detection method described above.

[0050] This invention provides a method for detecting oil pipes, comprising: acquiring an image of an oil pipe to be detected, and extracting features from the image of the oil pipe to be detected to obtain a corresponding feature map; using a preset key point detection model to predict key points in the feature map to predict the positions of a preset number of first key points representing the position and shape of the oil pipe in the image of the oil pipe to be detected; and determining the position and shape of the oil pipe in the image of the oil pipe to be detected based on the positions of the first key points.

[0051] As can be seen, after acquiring the image of the oil pipe to be detected, the present invention first extracts features to obtain a corresponding feature map. Subsequently, the present invention uses a preset keypoint detection model to predict keypoints in this feature map, predicting the positions of a preset number of first keypoints representing the position and shape of the oil pipe in the image. Based on the positions of these first keypoints, the position and shape of the oil pipe can be determined in the image. In short, the present invention can use keypoint detection to detect flexible targets such as oil pipes, effectively detecting oil pipes in various environments and postures, thereby improving the robustness of oil pipe detection. The present invention also provides an oil pipe detection device, electronic device, and storage medium, which have the aforementioned beneficial effects. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 A flowchart of an oil pipe inspection method provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of an oil pipe inspection method provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a preset key point detection model provided in an embodiment of the present invention;

[0056] Figure 4 This is a structural block diagram of an oil pipe inspection device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] In related technologies, if traditional machine vision methods are used to identify the shape of oil pipes through gradient calculation, the identification process is easily affected by complex backgrounds. If object detection algorithms are used to identify oil pipe targets, considering that oil pipes are flexible objects with varying shapes, it is not only difficult to locate the oil pipe using object detection boxes, but the complexity of the oil pipe's shape also makes it difficult for the model to converge, posing significant challenges to model training. In view of this, the present invention provides an oil pipe detection method that uses keypoint detection to detect flexible targets such as oil pipes. This method can effectively detect oil pipes in various environments and postures, thereby improving the robustness of oil pipe detection. Please refer to [reference needed]. Figure 1 , Figure 1 A flowchart of an oil pipe inspection method provided in an embodiment of the present invention is shown. The method may include:

[0059] S101. Obtain the image of the oil pipe to be detected, and extract features from the image to obtain the corresponding feature map.

[0060] It is understood that the image of the oil pipe to be detected is an image containing the oil pipe. After obtaining the image of the oil pipe to be detected, this embodiment of the invention first extracts features from the image to obtain the corresponding feature map, and then the oil pipe can be detected based on the feature map. It should be noted that this embodiment of the invention does not limit the specific method of feature extraction, nor does it limit the neural network model used for feature extraction. It can be set according to the actual application requirements. For example, feature extraction can use a VGG network (Visual Geometry Group).

[0061] S102. Use a preset key point detection model to predict key points in the feature map, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be tested.

[0062] It should be noted that the "first key point" is a predicted key point generated by the preset key point detection model, and the "first" is only used to distinguish it from other predicted key points appearing in the embodiments of the present invention.

[0063] It is worth noting that the preset keypoint detection model belongs to the pose estimation network. Such networks typically use keypoints to mark the position and pose of the target to be detected. Because this type of network is well-adapted to complex background environments and can perform pose detection on targets of various shapes, this embodiment of the invention, by introducing this model into the field of pipe detection, can significantly improve the accuracy and reliability of pipe detection, thereby effectively solving the problem that the accuracy and reliability of existing pipe detection methods are greatly affected by complex background environments and complex pipe poses. It should be noted that pose estimation networks are typically used to identify human poses, and can usually predict the position of keypoints representing various joints of the human body in the image by generating confidence maps. Therefore, the preset keypoint detection model in this embodiment of the invention can also predict the position of keypoints representing the position and shape of the pipe in the image of the pipe to be tested by generating confidence maps. Specifically, the confidence map in this embodiment of the invention is used to record the confidence level of each pixel in the image of the pipe to be tested as the corresponding keypoint. Furthermore, this embodiment of the invention can set the pixels with higher confidence levels in the confidence map as the corresponding keypoints, thus completing the determination of the keypoint positions.

[0064] Of course, it should also be noted that since pose estimation networks can typically only predict the positions of a fixed number of keypoints, the number of the first keypoints is also fixed in this embodiment of the invention. It should be noted that the number of the first keypoints is a preset number and can be adjusted according to actual application requirements; for example, it can be 8.

[0065] Based on this, using a pre-defined keypoint detection model to predict keypoints in the feature map can include:

[0066] Step 11: Input the feature map into the preset key point detection model to generate a confidence map corresponding to each first key point using the preset key point detection model; the confidence map is used to record the confidence level of each pixel in the oil pipe image to be detected as the first key point corresponding to the confidence map;

[0067] Step 12: Determine the position of each first key point in the image of the oil pipe to be detected based on the confidence map.

[0068] It should be noted that the embodiments of the present invention do not limit the specific form of the confidence map, and related technologies can be referred to. It should be pointed out that in the process of generating the confidence map, the preset key point detection model has taken into account the arrangement between each key point, and can write the positions of each first key point into a key point sequence in an orderly manner according to the arrangement. Therefore, the position and orientation of the oil pipe image to be detected can be determined based on the sequence.

[0069] It should be noted that the embodiments of the present invention do not limit the generation method of the confidence maps; for specific details, please refer to relevant technologies of pose estimation networks such as OpenPose. It is understood that the number of confidence maps generated by the preset keypoint detection model should be at least the same as the number of keypoints, i.e., both should be the preset number. Of course, in practical applications, the preset keypoint detection model may generate an additional confidence map as a background.

[0070] S103. Determine the position and shape of the oil pipe in the image of the oil pipe to be detected based on the position of the first key point.

[0071] Please refer to Figure 2 , Figure 2 This is a schematic diagram of an oil pipe detection method provided in an embodiment of the present invention, wherein kt1, kt2, kn1, kn2, kn3, kt3, kn4, and kt4 represent key points predicted by a preset key point detection model. Therefore, in this embodiment of the present invention, the position and shape of the oil pipe in the image of the oil pipe to be detected can be determined based on these key points.

[0072] Based on the above embodiments, after acquiring the image of the oil pipe to be detected, the present invention can first extract features from it to obtain the corresponding feature map. Subsequently, the present invention can use a preset key point detection model to predict key points in this feature map, so as to predict the position of a preset number of first key points representing the position and shape of the oil pipe in the image of the oil pipe to be detected. Then, the position and shape of the oil pipe can be determined in the image based on the position of the first key points. In short, the present invention can use key point detection to detect flexible targets such as oil pipes, and can effectively detect oil pipes of various postures in various environments, thereby improving the robustness of oil pipe detection.

[0073] Based on the above embodiments, the training process of the preset keypoint detection model will be described in detail below. In one possible case, the training process of the preset keypoint detection model may include:

[0074] S201. Obtain training pipeline images and extract features from the training pipeline images to obtain corresponding training feature maps; the training pipeline images are labeled with a preset number of pipeline key points, and the positions of each pipeline key point and the affinity between each pipeline key point are recorded.

[0075] The training pipeline images are images containing pipelines used to train the preset keypoint detection model. These images are all labeled with a preset number of pipeline keypoints, and the location of each keypoint and the affinity between them are recorded. Affinity describes the relationship between the pipeline keypoints; the use of affinity in this invention will be described in detail later in this embodiment. It should be noted that this embodiment does not limit the number of training pipeline images or their parameters, and can be set according to actual application needs. This embodiment also does not limit the keypoint annotation format used for the training pipeline images; for example, it can be annotated using the COCO dataset.

[0076] Furthermore, it is understood that training pipeline images need to be converted into training feature maps before they can be used for model training. This embodiment of the invention does not limit the specific method of feature extraction, nor does it limit the neural network model used for feature extraction; it can be set according to actual application requirements.

[0077] S202. Use the first network branch of the preset key point detection model to predict key points in the training feature map, so as to predict the position of the second key point corresponding to each pipeline key point in the training pipeline image. Use the second network branch of the preset key point detection model to predict the key point affinity in the training feature map, so as to predict the affinity between each second key point.

[0078] The preset keypoint detection model in this embodiment of the invention comprises two network branches. The input to both branches is a training feature map. The first network branch is the primary branch for keypoint prediction, used to generate confidence maps corresponding to each keypoint and determine the location of each keypoint based on the confidence maps. The second network branch is a secondary branch that assists in training the first branch, primarily used to predict the affinity between keypoints. It is worth noting that this embodiment of the invention mainly improves the second network branch. The affinity prediction method will be described in detail below.

[0079] Existing pose estimation networks are mainly used to predict the positions of a fixed number of human joints; that is, in human pose estimation, keypoints correspond to various joints of the human body. However, oil pipes are flexible objects and do not have a fixed number of joints; even the number of bends is not fixed. If, based solely on the approach of human pose estimation, the connection points and bends of the oil pipe are treated as joints, and a pre-defined keypoint detection model is used to predict the positions of the keypoints (hereinafter referred to as turning keypoints) corresponding to these connection points and bends, the actual connection points and bends of the oil pipe may not fill the output of the pre-defined keypoint detection model, thus causing the model to fail to predict accurately. For example, if the pre-defined keypoint detection model can predict a fixed number of 8 keypoints, ... Figure 2If the oil pipe has sections kt1, kt2, kt3, and kt4 that can be considered joints, then the preset key point detection model will have difficulty accurately locating each key point. Therefore, this embodiment of the invention adds a new type of key point, namely the fitting key point. The fitting key point is usually located on the straight section of the oil pipe between a pair of adjacent turning key points, for example, in... Figure 2 In the diagram, kn1, kn2, kn3, and kn4 are the key points of the tubing marked on the aforementioned straight section. kn1, kn2, and kn3 are located between kt2 and kt3, and kn4 is located between kt3 and kt4. In other words, in this embodiment of the invention, there are two types of key points: turning key points marked at the starting position of the tubing and at bends with angles greater than a preset value, and fitting key points marked between adjacent turning key points. It should be noted that this embodiment of the invention does not limit the specific value of the preset value and can be set according to actual application needs; this embodiment of the invention does not limit the number of fitting key points that can be marked between adjacent turning key points and can be set according to actual application needs. For example, if the straight section is long, more fitting key points can be marked; if it is short, fewer fitting key points can be marked, or even no fitting key points may be marked.

[0080] Furthermore, for key turning points, embodiments of the present invention can use the directional relationship between the key turning point and its adjacent subsequent key turning point as its corresponding affinity. For example, for Figure 2 In the equations kt1 and kt2, the vector pointing from kt1 to kt2 represents the affinity of kt1; similarly, for kt2 and kt3, the vector pointing from kt2 to kt3 represents the affinity of kt2. For fitted keypoints, this embodiment of the invention uses the distance between the keypoint and a straight line, where the straight line passes through two adjacent turning keypoints. For example, the affinity of kn1 is its distance to the straight line passing through kt2 and kt3; similarly, the affinity of kn4 is its distance to the straight line passing through kt3 and kt4.

[0081] Based on this, the key points of the pipeline include turning key points and fitting key points. Turning key points are marked at the starting position of the pipeline and the bending position where the bending angle is greater than a preset value. Fitting key points are marked between adjacent turning key points. The second network branch of the preset key point detection model is used to predict the affinity of the training feature map to predict the affinity between each second key point. This can include:

[0082] Step 21: Add all the second key points to the key point sequence according to the arrangement order of the key points of the pipeline.

[0083] It should be noted that the second key point corresponds one-to-one with the pipeline key point, so the second key point can be added to the key point sequence in an orderly manner based on the arrangement order of the pipeline key points.

[0084] Step 22: Set the first second key point in the key point sequence as the key point to be calculated, and determine the type of the tubing key point corresponding to the key point to be calculated.

[0085] Step 23: If the type is a turning point key point, find the next target second key point adjacent to the key point to be calculated and corresponding to the turning point key point in the key point sequence, and set the vector from the key point to be calculated to the target second key point as the affinity value corresponding to the key point to be calculated.

[0086] For example, given the second critical point p1 and its corresponding tubing critical point g1, if g1 is determined to be a turning point critical point, it is necessary to find the next adjacent second critical point p2 that is also a turning point critical point, and to find the next adjacent tubing critical point g2 that is also a turning point critical point. Subsequently, the affinity corresponding to p1 can be represented as V. p1 =(p p1 ,p p2 ), where V p1 Let p1 be a vector pointing to p2. p1 Indicates the position of p1, p p2 This indicates the position of p2; the affinity corresponding to g1 can be represented as V. g1 =(p g1 ,p g2 ), where V g1 p represents the vector pointing from g1 to g2. g1 p represents the position of g1. g2 This indicates the position of g2.

[0087] Step 24: If the type is fitting keypoint, find two target second keypoints in the keypoint sequence that are adjacent to the keypoint to be calculated and are both corresponding to the turning keypoints; determine the straight line passing through the two target second keypoints, and set the distance between the keypoint to be calculated and the straight line as the affinity value corresponding to the keypoint to be calculated.

[0088] For example, given the second key point p3 and its corresponding tubing key point g3, if g3 is determined to be the fitting key point, it is necessary to find a pair of second key points p1 and p2 adjacent to p3 that are also turning key points, and to find a pair of tubing key points g1 and g2 adjacent to g3 that are also turning key points. Subsequently, the affinity corresponding to p3 can be expressed as D. p3 D p3 The distance between p3 and the line passing through p1 and p2 is represented by D; the affinity corresponding to g3 can be expressed as D. g3 Dg3 This represents the distance between g3 and the line passing through g1 and g2.

[0089] Furthermore, for ease of querying, the key turning points, fitted key points, the equations of the lines between key turning points, and the distances between fitted key points and relevant lines can be recorded in the following form:

[0090]

[0091] Among them, H n Let P represent the keypoint affinity matrix; P represents the point set, which records a pair of adjacent turning keypoints and all fitted keypoints between these pairs; f(x,y) represents the equation of the straight line passing through the turning keypoints recorded in point set P; D represents the distance set, which records the distance between each fitted keypoint in point set P and the line corresponding to f(x,y). For keypoints (x0,y0) and (x1,y1), f(x,y) can be expressed as:

[0092] (y1-y0)x+(x1-x0)y+(y1x0-y0x1)=0;

[0093] The distance between f(x,y) and the keypoint (x2,y2) can be calculated using the following formula:

[0094]

[0095] Where d represents the distance, A = x1 - x0, B = y1 - y0, C = y1x0 - y0x1.

[0096] Step 25: When the affinity value of the current key point to be calculated is completed, if it is determined that there is a second key point in the key point sequence for which the affinity value has not been calculated, the key point to be calculated is updated to the next second key point, and the step of determining the type of the tubing key point corresponding to the key point to be calculated is entered.

[0097] Step 26: If it is determined that there is no second keypoint in the keypoint sequence for which affinity value has not been calculated, then the keypoint affinity prediction is completed.

[0098] S203. Generate a first loss value using the location of each second key point and the location of the corresponding oil pipe key point, and generate a second loss value using the affinity between each second key point and the affinity between the corresponding oil pipe key points.

[0099] Understandably, the first loss value can be calculated using the positions of each second key point and the corresponding positions of the tubing key points. The calculation of the second loss value involves two cases. For turning key points, this embodiment of the invention uses the cosine of the angle between the vector of the second key point and the vector of the corresponding tubing key point as the affinity loss value for that second key point. For fitting key points, this embodiment of the invention generates the affinity loss value for the second key point based on the difference between the distance corresponding to the second key point and the distance corresponding to the corresponding tubing key point. Finally, the second loss value can be calculated using the affinity loss values ​​of all second key points.

[0100] Based on this, a second loss value is generated using the affinity between each second key point and the corresponding affinity between tubing key points, including:

[0101] Step 31: Set the first and second key points in the key point sequence as the key points to be processed, and determine the type of the tubing key points corresponding to the key points to be processed;

[0102] Step 32: If the type is a turning point key point, determine the cosine angle between the vector corresponding to the key point to be processed and the vector corresponding to the oil pipe key point to be processed, and set the cosine angle value as the affinity loss value corresponding to the key point to be processed.

[0103] Step 33: If the type is fitting key point, then determine the distance loss value between the distance corresponding to the key point to be processed and the distance corresponding to the oil pipe key point to be processed, and set the distance loss value as the affinity loss value corresponding to the key point to be processed.

[0104] Step 34: When calculating the affinity loss value of the current key point to be processed, if it is determined that there is a second key point in the key point sequence for which the affinity loss value has not been calculated, then update the key point to be processed to the next second key point and determine the type of the tubing key point corresponding to the key point to be processed.

[0105] Step 35: If it is determined that there are no second keypoints in the keypoint sequence for which affinity loss values ​​have not been calculated, then generate a second loss value using the affinity loss values ​​corresponding to all second keypoints.

[0106] Specifically, the affinity loss value can be expressed as:

[0107]

[0108] Among them, L t (p) represents the affinity loss value corresponding to pixel p, and t represents the stage, the specific meaning of which will be introduced in subsequent embodiments; if pixel p corresponds to a second key point, and the key point type of the second key point is a turning key point, then V g1V represents the vector corresponding to the key point g1 of the oil pipe. p1 This represents the vector corresponding to the second keypoint p1. If pixel p corresponds to a second keypoint, and the keypoint type of this second keypoint is a fitted keypoint, then D... g1 V represents the distance corresponding to the critical point g1 of the oil pipe. p1 This represents the distance to the corresponding second keypoint p1. `sigmoid` represents the sigmoid function (activation function).

[0109] S204. Generate a total loss value using the first loss value and the second loss value, and determine whether the total loss value is greater than a preset threshold.

[0110] It should be noted that the embodiments of the present invention do not limit the specific value of the preset threshold, and can be set according to actual application needs.

[0111] S205. If so, the parameters of the prediction model are updated using the total loss value, and based on the updated prediction model, the steps of using the first network branch of the preset key point detection model to predict key points in the training feature map and using the second network branch of the preset key point detection model to predict key point affinity in the training feature map are entered.

[0112] S206. If not, then complete the training of the preset key point detection model.

[0113] Based on the above embodiments, the preset keypoint detection model can also improve prediction accuracy through a staged processing approach. Please refer to... Figure 3 , Figure 3 This diagram illustrates a preset keypoint detection model provided in an embodiment of the present invention. "Stage" represents a stage, "Branch 1" represents a first network branch, and "Branch 2" represents a second network branch. The model comprises multiple stages, each with the aforementioned first and second network branches. The two network branches in each stage function identically: the first network branch predicts keypoint locations, and the second network branch predicts affinity between keypoints. Furthermore, after processing by the two network branches in each stage, a corresponding sub-loss value is generated; for example, the first network branch corresponds to the first sub-loss value, and the second network branch corresponds to the second sub-loss value. Therefore, when updating model parameters, the model parameters can be updated in stages based on the sub-loss values ​​of each stage. Additionally, after processing, each stage's network branch needs to fuse its output with the original training feature map to obtain intermediate features, which are then input into the two network branches of the next stage.

[0114] Based on the above description, the first network branch of the preset keypoint detection model is used to predict keypoints in the training feature map to predict the position of the second keypoint corresponding to each pipeline keypoint in the training pipeline image. The second network branch of the preset keypoint detection model is also used to predict keypoint affinity in the training feature map to predict the affinity between each second keypoint. This can include:

[0115] S301. Set the training feature map as the feature to be processed, and initialize the current stage to 1;

[0116] S302. Input the features to be processed into the first network branch of the preset key point detection model at the current stage to predict the position of each second key point in the training pipeline image at the current stage, and input the features to be processed into the second network branch of the preset key point detection model at the current stage to predict the affinity between each second key point at the current stage.

[0117] S303. Generate a first sub-loss value by using the position of each second key point and the position of the corresponding oil pipe key point in the current stage, and generate a second sub-loss value by using the affinity between each second key point and the affinity between the corresponding oil pipe key points in the current stage.

[0118] S304. The intermediate features are obtained by fusing the training feature map with the positions of the second key points in the current stage and the affinity between the second key points in the current stage.

[0119] S305. If it is determined that the current stage is less than the preset stage value, then add one to the current stage, set the intermediate feature as the feature to be processed, and proceed to the step of inputting the feature to be processed into the first network branch of the preset key point detection model corresponding to the current stage.

[0120] It should be noted that the embodiments of the present invention do not limit the specific value of the preset stage value. For example, it can be 4, that is, the preset key point prediction model includes four stages of network branches.

[0121] Accordingly, a first loss value is generated using the location of each second key point and the location of the corresponding tubing key point, and a second loss value is generated using the affinity between each second key point and the affinity between the corresponding tubing key points, including:

[0122] S306. If it is determined that the current stage value is equal to the preset stage value, then the first loss value is calculated using the first sub-loss value of each stage, and the second loss value is calculated using the second sub-loss value of each stage.

[0123] Specifically, the first sub-loss value can be expressed as:

[0124]

[0125] in, This represents the first sub-loss value corresponding to stage t; The predicted value representing the location of key points; W(p) represents the true value of the key point location; W(p) is used to characterize whether the pixel p is occluded, 0 if occluded and 1 if not occluded.

[0126] The second sub-loss value can be expressed as:

[0127]

[0128] in This represents the affinity loss value based on the c-th predicted key point in stage t.

[0129] The total loss value can be expressed as:

[0130]

[0131] The following describes the tubing inspection device, electronic device, and storage medium provided in the embodiments of the present invention. The tubing inspection device, electronic device, and storage medium described below can be referred to in correspondence with the tubing inspection method described above.

[0132] Please refer to Figure 4 , Figure 4 This is a structural block diagram of an oil pipe inspection device provided in an embodiment of the present invention. The device may include:

[0133] The feature extraction module 401 is used to acquire the image of the oil pipe to be detected and to extract features from the image of the oil pipe to be detected to obtain the corresponding feature map.

[0134] The key point prediction module 402 is used to predict key points in the feature map using a preset key point detection model, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be tested.

[0135] The oil pipe determination module 403 is used to determine the position and shape of the oil pipe in the image to be detected based on the position of the first key point.

[0136] Optionally, the keypoint prediction module 402 may include:

[0137] The confidence map generation submodule is used to input the feature map into the preset key point detection model, so as to generate a confidence map corresponding to each first key point using the preset key point detection model; the confidence map is used to record the confidence level of each pixel in the oil pipe image to be detected as the first key point corresponding to the confidence map;

[0138] The key point prediction submodule is used to determine the position of each first key point in the pipeline image to be detected based on the confidence map.

[0139] Optionally, the device may further include:

[0140] The training pipeline image feature extraction module is used to acquire training pipeline images and extract features from the training pipeline images to obtain the corresponding training feature maps; the training pipeline images are labeled with a preset number of pipeline key points, and the position of each pipeline key point and the affinity between each pipeline key point are recorded.

[0141] The training module is used to predict key points in the training feature map using the first network branch of the preset key point detection model, so as to predict the position of the second key point corresponding to each pipeline key point in the training pipeline image, and to predict the affinity between each second key point by using the second network branch of the preset key point detection model.

[0142] The loss value calculation module is used to generate a first loss value by using the position of each second key point and the position of the corresponding oil pipe key point, and to generate a second loss value by using the affinity between each second key point and the affinity between the corresponding oil pipe key points.

[0143] The parameter update module is used to generate a total loss value using the first loss value and the second loss value, and to determine whether the total loss value is greater than a preset threshold. If so, the module updates the parameters of the prediction model using the total loss value, and then proceeds to the steps of using the first network branch of the preset keypoint detection model to predict key points in the training feature map, and using the second network branch of the preset keypoint detection model to predict key point affinity in the training feature map. If not, the module completes the training of the preset keypoint detection model.

[0144] Optionally, the pipeline key points include turning key points and fitting key points. Turning key points are marked at the starting position of the pipeline and the bend position where the bend angle is greater than a preset value. Fitting key points are marked between adjacent turning key points. The training module may include:

[0145] Add a submodule to add all secondary keypoints to the keypoint sequence according to the arrangement order of the tubing keypoints;

[0146] The first type determination submodule is used to set the first second key point in the key point sequence as the key point to be calculated, and to determine the type of the tubing key point corresponding to the key point to be calculated.

[0147] The first affinity calculation submodule is used to find the next target second key point that is adjacent to the key point to be calculated and corresponds to the turning key point in the key point sequence if the type is a turning key point, and set the vector from the key point to be calculated to the target second key point as the affinity value corresponding to the key point to be calculated.

[0148] The second affinity calculation submodule is used to determine, if the type is fitting keypoint, two target second keypoints that are adjacent to the keypoint to be calculated and both correspond to turning keypoints in the keypoint sequence; determine the straight line passing through the two target second keypoints; and set the distance between the keypoint to be calculated and the straight line as the affinity value corresponding to the keypoint to be calculated.

[0149] The first update submodule is used to update the key point to be calculated to the next second key point if it is determined that there is a second key point in the key point sequence whose affinity value has not been calculated when the affinity value of the current key point to be calculated is completed; and then proceed to the step of determining the type of the oil pipe key point corresponding to the key point to be calculated.

[0150] The first termination submodule is used to complete the keypoint affinity prediction if it is determined that there is no second keypoint in the keypoint sequence for which affinity values ​​have not been calculated.

[0151] Optionally, the loss calculation module may include:

[0152] The second type determination submodule is used to set the first second key point in the key point sequence as the key point to be processed, and to determine the type of the tubing key point corresponding to the key point to be processed.

[0153] The first affinity loss value calculation submodule is used to determine the cosine angle between the vector corresponding to the key point to be processed and the vector corresponding to the oil pipe key point to be processed if the type is a turning key point, and set the cosine angle value as the affinity loss value corresponding to the key point to be processed.

[0154] The second affinity loss value calculation submodule is used to determine the distance loss value between the distance corresponding to the key point to be processed and the distance corresponding to the oil pipe key point to be processed if the type is fitting key point, and set the distance loss value as the affinity loss value corresponding to the key point to be processed.

[0155] The second update submodule is used to update the key point to be processed to the next second key point if it is determined that there is a second key point in the key point sequence whose affinity loss value has not been calculated when the affinity loss value of the current key point to be processed is completed; and to determine the type of the tubing key point corresponding to the key point to be processed.

[0156] The second termination submodule is used to generate a second loss value by utilizing the affinity loss values ​​corresponding to all second keypoints if it is determined that there are no second keypoints in the keypoint sequence for which the affinity loss value has not been calculated.

[0157] Optionally, the training module includes:

[0158] The initialization submodule is used to set the training feature map as the feature to be processed and initialize the current stage to 1;

[0159] The training submodule is used to input the features to be processed into the first network branch of the preset key point detection model at the current stage to predict the position of each second key point in the training pipeline image at the current stage, and to input the features to be processed into the second network branch of the preset key point detection model at the current stage to predict the affinity between each second key point at the current stage.

[0160] The loss value calculation submodule is used to generate a first sub-loss value by using the position of each second key point and the position of the corresponding oil pipe key point in the current stage, and to generate a second sub-loss value by using the affinity between each second key point and the affinity between the corresponding oil pipe key points in the current stage.

[0161] The fusion submodule is used to fuse the training feature map with the positions of each second key point in the current stage and the affinity between the second key points in the current stage to obtain intermediate features;

[0162] The update submodule is used to increment the current stage by one if it is determined that the current stage is less than the preset stage value, set the intermediate feature as the feature to be processed, and enter the step of inputting the feature to be processed into the first network branch of the preset key point detection model corresponding to the current stage.

[0163] Correspondingly, the loss value calculation module includes:

[0164] The loss value fusion submodule is used to calculate the first loss value using the first sub-loss value of each stage and the second loss value using the second sub-loss value of each stage if the current stage value is determined to be equal to the preset stage value.

[0165] This invention also provides an electronic device, comprising:

[0166] Memory, used to store computer programs;

[0167] A processor is used to execute computer programs to implement the steps of the oil pipe detection method described above.

[0168] Since the embodiments of the electronic equipment part correspond to the embodiments of the oil pipe detection method part, please refer to the description of the embodiments of the oil pipe detection method part for the embodiments of the electronic equipment part, and will not be repeated here.

[0169] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the tubing detection method described in any of the above embodiments.

[0170] Since the embodiments of the storage medium section correspond to the embodiments of the tubing detection method section, please refer to the description of the embodiments of the tubing detection method section for the embodiments of the storage medium section, and will not be repeated here.

[0171] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0172] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0173] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0174] The present invention has provided a detailed description of an oil pipe inspection method, apparatus, electronic device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely illustrative of the method and its core concepts. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A method for inspecting oil pipes, characterized in that, include: Acquire an image of the oil pipe to be detected, and extract features from the image to obtain the corresponding feature map. A preset key point detection model is used to predict key points in the feature map, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be detected. The position and shape of the oil pipe are determined in the image of the oil pipe to be detected based on the position of the first key point. The training process of the preset key point detection model includes: A training pipeline image is acquired, and features are extracted from the training pipeline image to obtain a corresponding training feature map. The training pipeline image is labeled with a preset number of pipeline key points, and the position of each pipeline key point and the affinity between each pipeline key point are recorded. The first network branch of the preset key point detection model is used to predict key points in the training feature map to predict the position of the second key point corresponding to each pipeline key point in the training pipeline image, and the second network branch of the preset key point detection model is used to predict the affinity between each second key point in the training feature map. The key points of the pipeline include turning key points and fitting key points. Turning key points are marked at the starting position and bending positions where the bending angle is greater than a preset value. Fitting key points are marked between adjacent turning key points. The step of using the second network branch of the preset key point detection model to predict the key point affinity of the trained feature map, in order to predict the affinity between each of the second key points, includes: According to the arrangement order of the tubing key points, all second key points are added to the key point sequence; the first second key point in the key point sequence is set as the key point to be calculated, and the type of tubing key point corresponding to the key point to be calculated is determined; if the type is a turning key point, the next target second key point adjacent to the key point to be calculated and corresponding to the turning key point is found in the key point sequence, and the vector from the key point to be calculated to the target second key point is set as the affinity value corresponding to the key point to be calculated; if the type is a fitting key point, two target second key points adjacent to the key point to be calculated and both corresponding to turning key points are found in the key point sequence; a straight line passing through the two target second key points is determined, and the distance between the key point to be calculated and the straight line is set as the affinity value corresponding to the key point to be calculated.

2. The tubing inspection method according to claim 1, characterized in that, The step of predicting key points in the feature map using a preset key point detection model includes: The feature map is input into the preset key point detection model to generate a confidence map corresponding to each first key point; the confidence map is used to record the confidence level of each pixel in the oil pipe image to be detected as the first key point corresponding to the confidence map; The location of each of the first key points in the image of the oil pipe to be detected is determined based on the confidence map.

3. The tubing inspection method according to claim 1 or 2, characterized in that, The training process of the preset key point detection model also includes: A first loss value is generated by using the position of each second key point and the position of the corresponding oil pipe key point, and a second loss value is generated by using the affinity between each second key point and the affinity between the corresponding oil pipe key points. A total loss value is generated using the first loss value and the second loss value, and it is determined whether the total loss value is greater than a preset threshold. If so, the parameters of the preset key point detection model are updated using the total loss value, and the updated preset key point detection model is used to perform key point prediction on the training feature map using the first network branch of the preset key point detection model, and key point affinity prediction on the training feature map using the second network branch of the preset key point detection model. If not, then the training of the preset key point detection model is complete.

4. The oil pipe inspection method according to claim 1, characterized in that, The step of using the second network branch of the preset keypoint detection model to predict the keypoint affinity of the training feature map, so as to predict the affinity between each second keypoint, further includes: When calculating the affinity value of the current key point to be calculated, if it is determined that there is a second key point in the key point sequence for which the affinity value has not been calculated, the key point to be calculated is updated to the next second key point, and the process proceeds to the step of determining the type of the oil pipe key point corresponding to the key point to be calculated. If it is determined that there is no second keypoint in the keypoint sequence for which the affinity value has not been calculated, then the keypoint affinity prediction is completed.

5. The tubing inspection method according to claim 3, characterized in that, The process of generating a second loss value by utilizing the affinity between each second key point and the affinity between the corresponding tubing key points includes: Set the first second key point in the key point sequence as the key point to be processed, and determine the type of the oil pipe key point corresponding to the key point to be processed; If the type is a turning point, then determine the cosine angle between the vector corresponding to the key point to be processed and the vector corresponding to the oil pipe key point to be processed, and set the cosine angle value as the affinity loss value corresponding to the key point to be processed. If the type is fitting key point, then determine the distance loss value between the distance corresponding to the key point to be processed and the distance corresponding to the oil pipe key point corresponding to the key point to be processed, and set the distance loss value as the affinity loss value corresponding to the key point to be processed; When calculating the affinity loss value of the current key point to be processed, if it is determined that there is a second key point in the key point sequence for which the affinity loss value has not been calculated, the key point to be processed is updated to the next second key point, and the type of the tubing key point corresponding to the key point to be processed is determined. If it is determined that there are no second keypoints in the keypoint sequence for which the affinity loss value has not been calculated, then the second loss value is generated using the affinity loss values ​​corresponding to all second keypoints.

6. The method for detecting oil pipes according to claim 1, characterized in that, The step of using the first network branch of the preset keypoint detection model to predict keypoints in the training feature map, thereby predicting the position of the second keypoint corresponding to each of the pipeline keypoints in the training pipeline image, and using the second network branch of the preset keypoint detection model to predict keypoint affinity in the training feature map, thereby predicting the affinity between each of the second keypoints, includes: Set the training feature map as the feature to be processed, and initialize the current stage to 1; The features to be processed are input into the first network branch of the preset key point detection model corresponding to the current stage to predict the position of each second key point in the training oil pipe image at the current stage, and the features to be processed are input into the second network branch of the preset key point detection model corresponding to the current stage to predict the affinity between each second key point at the current stage. The first sub-loss value is generated by using the position of each second key point and the position of the corresponding oil pipe key point in the current stage, and the second sub-loss value is generated by using the affinity between each second key point and the affinity between the corresponding oil pipe key points in the current stage. The intermediate features are obtained by fusing the training feature map with the positions of each second key point in the current stage and the affinity between each second key point in the current stage. If it is determined that the current stage value is less than the preset stage value, then the current stage is incremented by one, the intermediate feature is set as the feature to be processed, and the process proceeds to the step of inputting the feature to be processed into the first network branch of the preset key point detection model corresponding to the current stage. Accordingly, the step of generating a first loss value using the positions of each of the second key points and the corresponding positions of the tubing key points, and generating a second loss value using the affinity between each of the second key points and the corresponding affinity between the tubing key points, includes: If it is determined that the current stage value is equal to the preset stage value, then the first loss value is calculated using the first sub-loss value of each stage, and the second loss value is calculated using the second sub-loss value of each stage.

7. An oil pipe inspection device, characterized in that, include: The feature extraction module is used to acquire the image of the oil pipe to be detected and to extract features from the image of the oil pipe to be detected to obtain the corresponding feature map. The key point prediction module is used to predict key points in the feature map using a preset key point detection model, so as to predict the position of a preset number of first key points representing the location and shape of the oil pipe in the image of the oil pipe to be detected. The oil pipe determination module is used to determine the position and shape of the oil pipe in the oil pipe image to be detected based on the position of the first key point. The oil pipe inspection device also includes: The training pipeline image feature extraction module is used to acquire training pipeline images and extract features from the training pipeline images to obtain the corresponding training feature maps; the training pipeline images are labeled with a preset number of pipeline key points, and the position of each pipeline key point and the affinity between each pipeline key point are recorded. The training module is used to predict key points in the training feature map using the first network branch of the preset key point detection model, so as to predict the position of the second key point corresponding to each pipeline key point in the training pipeline image, and to predict the affinity between each second key point by using the second network branch of the preset key point detection model. The tubing key points include turning key points and fitting key points. Turning key points are marked at the starting position and bending positions where the bending angle is greater than a preset value. Fitting key points are marked between adjacent turning key points. The training module includes: Add a submodule to add all secondary keypoints to the keypoint sequence according to the arrangement order of the tubing keypoints; The first type determination submodule is used to set the first second key point in the key point sequence as the key point to be calculated, and to determine the type of the tubing key point corresponding to the key point to be calculated. The first affinity calculation submodule is used to find the next target second key point that is adjacent to the key point to be calculated and corresponds to the turning key point in the key point sequence if the type is a turning key point, and set the vector from the key point to be calculated to the target second key point as the affinity value corresponding to the key point to be calculated. The second affinity calculation submodule is used to determine, if the type is fitting keypoint, two target second keypoints that are adjacent to the keypoint to be calculated and both correspond to turning keypoints in the keypoint sequence; determine the straight line passing through the two target second keypoints, and set the distance between the keypoint to be calculated and the straight line as the affinity value corresponding to the keypoint to be calculated.

8. The oil pipe inspection device according to claim 7, characterized in that, The key point prediction module includes: The confidence map generation submodule is used to input the feature map into the preset key point detection model, so as to generate a confidence map corresponding to each preset key point using the preset key point detection model; the confidence map is used to record the confidence level of each pixel in the oil pipe image to be detected as the first key point corresponding to the confidence map; The key point prediction submodule is used to determine the position of each of the first key points in the image of the oil pipe to be detected based on the confidence map.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the tubing detection method as described in any one of claims 1 to 6.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the tubing inspection method as described in any one of claims 1 to 6.

Images