A deep learning iron roughneck coupling detection method and system

Abstract

The present application relates to the technical field of oil drilling and exploitation, and particularly relates to a method and system for detecting a tool joint of an iron roughneck based on deep learning, wherein a recognition device inputs a first original image into a tool joint detection model to obtain coordinates of the tool joint in a pixel coordinate system; a backbone network of the tool joint detection model is used to extract image features of the original image, and the backbone network comprises joint attention modules added after two down-sampling modules of the backbone network respectively; the joint attention modules are used to perform feature fusion on the first original feature image after assigning attention weight values to the first original feature image along a channel dimension and a spatial dimension respectively, so as to obtain a feature fusion image; the recognition device generates a control instruction according to the coordinates of the tool joint in the pixel coordinate system and sends the control instruction to an iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform a make-up or break-out operation. The method has the beneficial effect that full intelligentization and automation of iron roughneck control are realized through a high-precision tool joint detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of oil drilling and exploitation, and particularly relates to a deep learning iron roughneck coupling detection method and system. BACKGROUND

[0002] With the rapid development of automation technology, the work efficiency in the field of oil drilling and exploitation has been greatly improved. In the drilling operation process, as the drilling depth gradually increases, the tripping operation occupies a large part of the time.

[0003] The iron roughneck is one of the main devices for attention operation system and is widely used at present. At present, the iron roughneck can automatically make up and break out, but in the process of use, the operator still needs to control the iron roughneck to align the position of the coupling, and then realize the make up / break out operation. This has a high requirement on the operation specification of the operator, and once an error operation occurs, it will cause huge property loss. In recent years, with the gradual maturity of machine vision technology, this technology has been applied to automatic driving, visual auxiliary gripper and many other fields. Computer vision assisted iron roughneck coupling height positioning has become one of the most effective solutions.

[0004] The YOLO series algorithm can better balance speed and accuracy and is widely used in various fields. However, due to the more complex task scene in the process of oil drilling and exploitation and the more severe field environment, the original YOLO series model cannot meet the accuracy requirement in the process of iron roughneck make up / break out. SUMMARY

[0005] (I) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a deep learning iron roughneck coupling detection method and system, which solves the technical problems that the operation specification of personnel is high when manually aligning the coupling position, and the original YOLO series model cannot meet the accuracy requirement in the process of iron roughneck make up / break out due to the complex task scene and severe environment on site.

[0007] (II) Technical solutions

[0008] In order to achieve the above-mentioned purposes, the main technical solutions adopted by the present application include:

[0009] In a first aspect, the present application provides a deep learning-based iron roughneck coupling detection method, comprising:

[0010] S10, the recognition device inputs the received first original image of the drilling platform surface into the trained coupling detection model to obtain the coordinates of the coupling in the pixel coordinate system in the first original image; the first original image is an image of the wellhead and / or mouse hole position.

[0011] The coupling detection model comprises a trunk network, a neck network and a head network;

[0012] The trunk network is used for extracting image features of the original image, and comprises joint attention modules added after two down-sampling modules of the trunk network respectively;

[0013] Any joint attention module is used for assigning attention weight to a received first original feature image along a channel dimension and a spatial dimension respectively, and performing feature fusion on the first original feature image after the attention weight assignment along the channel dimension and the spatial dimension, to obtain a feature fusion image focusing on a coupling at a wellhead and / or a mouse hole position, i.e., an output result of the joint attention module;

[0014] The first original feature image is semantic information of the first original image after preliminary feature extraction and down-sampling, and the first original feature image comprises features of the coupling at the wellhead and / or the mouse hole position, and features of a background; and the feature fusion image is used for the trunk network, the neck network and the head network to obtain coordinates of the coupling in the first original image in a pixel coordinate system based on the feature fusion image;

[0015] S20, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system, and sends the control instruction to the iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform a make-up or break-out operation.

[0016] Optionally, any joint attention module comprises a spatial attention unit, a channel attention unit and a feature fusion unit;

[0017] The channel attention unit is used for, after feature compression in a spatial direction by global average pooling according to the features of the coupling at the wellhead and / or the mouse hole position and the features of the background in the first original feature image, performing feature extraction by a 1x1 convolution layer, then performing normalization processing by a Sigmoid layer, obtaining a channel attention weight matrix, and finally assigning attention weight to the first original feature image based on the channel attention weight matrix to obtain a channel attention map; the channel attention weight matrix comprises channel weights of the coupling at the wellhead and / or the mouse hole position, and channel weights of the background;

[0018] The spatial attention unit is configured to perform feature extraction on the first original feature map based on the features of the wellhead and / or the rat-hole position collar and the features of the background through a 1×1 convolution layer to obtain spatial context weights corresponding to the first original feature map, then reshape the spatial context weights, and normalize the reshaped spatial context weights through a Softmax layer to obtain a spatial attention weight matrix; and finally assign attention weights to the first original feature map based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix includes spatial weights of the wellhead and / or the rat-hole position collar and spatial weights of the background.

[0019] The feature fusion unit is configured to perform feature fusion on the channel attention map and the spatial attention map based on the pre-set channel attention weights and spatial attention weights to obtain a feature fusion image focusing on the wellhead and / or the rat-hole position collar; the channel attention weights and the spatial attention weights are obtained by pre-training a collar detection model.

[0020] Optionally, the channel attention unit is configured to perform feature compression in a spatial direction through global average pooling in an input feature channel direction after obtaining the features of the wellhead and / or the rat-hole position collar and the features of the background in the first original feature map, perform feature extraction on the first original feature map through a 1×1 convolution layer, then perform normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assign attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map, including:

[0021] The received first original feature map is compressed in a spatial direction through global average pooling in an input feature channel direction and a pre-set formula one; the formula one is:

[0022]

[0023] wherein, X c is a matrix in the input feature channel direction, i and j represent position indexes of corresponding elements of the matrix in the input feature channel direction, GAP c is an output of the global average pooling, H is a height of the first original feature map, and W is a width of the first original feature map.

[0024] Then, the first original feature map after the global average pooling is input into a 1×1 convolution layer for feature extraction.

[0025] X1=Conv 1×1 (GAP c );

[0026] wherein, Conv 1×1 is a 1×1 grouped convolution, and GAPc X1 is the output feature after the 1x1 convolution layer;

[0027] Then, the channel attention weight matrix is obtained by normalization processing through the Sigmoid layer;

[0028] Out = σ (X1) ;

[0029] wherein, Out is the channel attention weight matrix, σ is the Sigmoid activation function, and X1 is the output feature after the 1x1 convolution layer;

[0030] Finally, the channel attention weight matrix is multiplied with each channel dimension of the first original feature map to obtain the channel attention map.

[0031] Optionally, the spatial attention unit is configured to: based on the features of the wellhead and / or mouse hole position collar and the features of the background in the first original feature map, perform feature extraction through a 1x1 convolution layer to obtain spatial context weights corresponding to the first original feature map; then reshape the spatial context weights and perform normalization processing on the reshaped spatial context weights through a Softmax layer to obtain a spatial attention weight matrix; and finally, assign attention weights to the first original feature map based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix comprises spatial weights of the wellhead and / or mouse hole position collar and spatial weights of the background, and comprises:

[0032] The first original feature map with the size of CxHxW is input into the 1x1 convolution layer to generate spatial context weights corresponding to the first original feature map;

[0033] Then, the spatial context weights are reshaped into the size of HWx1x1;

[0034] The reshaped spatial context weights are normalized through the Softmax layer to obtain a spatial attention weight matrix;

[0035] Then, the first original feature map with the size of CxHxW is reshaped into the size of CxHW, the reshaped first original feature is multiplied with the attention weight matrix, and then the first original feature map is spliced to obtain a spatial attention map;

[0036] The overall processing formula of the spatial attention unit is:

[0037]

[0038] wherein, W k is a linear transformation matrix, which is implemented by a 1x1 convolution layer, q is an enumeration of all possible positions, and N pH x W, i.e., the number of positions in the feature map, o is the index of the query position, x o is the first original feature map, z o is the spatial attention map output by the spatial attention unit.

[0039] Optionally, all up-sampling modules in the neck network of the collar detection model are DUpsampling modules.

[0040] The DUpsampling module is configured to perform channel dimension increasing on the received second original feature map through a 1x1 deep dimension increasing convolutional layer, then expand the second original feature map after dimension increasing according to the channel dimension, and then combine elements with the same position and different channels to obtain an up-sampling image of the edge features of the collar of the refined wellhead and / or mouse hole position. The whole process formula of the DUpsampling module is:

[0041] Out DUp = R(DConv 1×1 (X));

[0042] wherein X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1x1 deep dimension increasing convolutional layer, R is an element rearrangement operation, and Out DUp is the up-sampling image output by the DUpsampling module.

[0043] The second original feature map is semantic information obtained by performing image feature extraction on the first original image through the backbone network.

[0044] The up-sampling image is used for the neck network and the head network to obtain the coordinates of the collar in the pixel coordinate system in the first original image based on the up-sampling image.

[0045] Optionally, the S10 further comprises, before the S10:

[0046] S00, the recognition module determines the mapping relationship between the image coordinate system and the world coordinate system according to the coordinates of the three calibration points in the image coordinate system in the second original image sent by the industrial camera, the coordinates of the three calibration points in the world coordinate system, and a pre-set formula one.

[0047] The formula one is:

[0048]

[0049] wherein λ is a mapping coefficient of the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1 and y2 are respectively the coordinates of the three calibration points in the image coordinate system in the second original image; y00 , y 11 and y 22 are respectively coordinates of three calibration points in the world coordinate system;

[0050] The S20 comprises:

[0051] The identification device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and the mapping relationship between the image coordinate system and the world coordinate system, and sends the control instruction to the iron roughneck control system, so that the iron roughneck control system controls the iron roughneck to perform the make-up / demolition operation; the control instruction comprises the coordinates of the coupling in the world coordinate system.

[0052] Optionally, the S10 further comprises, before the S10:

[0053] S01, the identification module inputs the third original image sent by the industrial camera into a pre-set ROI model to generate a first original image; the ROI model is used for region of interest cropping on the received second original image to obtain a first original image comprising only a region of interest; the region of interest is a wellhead and / or a mouse hole position.

[0054] Optionally, the S10 further comprises, before the S10:

[0055] S02, the identification module acquires training images at different time points by using a pre-set industrial camera, and labels the coupling of the wellhead and / or the mouse hole position in all training images and the coordinates of the coupling of the wellhead and / or the mouse hole position in the pixel coordinate system corresponding to each training image by using a labeling tool to obtain a training set.

[0056] The training set is input into a pre-constructed coupling detection model to obtain a trained coupling detection model.

[0057] During the period, a loss function MPDIoU is calculated to optimize the coupling detection model.

[0058] Optionally, the neck network is used for feature fusion on the image features extracted by the backbone network, and the head network is used for coupling coordinate detection according to the fused features of the neck network.

[0059] The backbone network comprises, in sequence, a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first down-sampling module, a first joint attention module, a third C2f module, a second down-sampling module, a second joint attention module, a first C2fCIB module, an SPPF module and a PSA module.

[0060] The neck network comprises, in sequence, a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third downsampling module, a fifth Concat module, a third C2fCIB module, a fourth downsampling module, a sixth Concat module, and a fourth C2fCIB module.

[0061] The head network comprises four Detct modules, and the four Detct modules are connected with the fifth C2f module, the sixth C2f module, the third C2fCIB module, and the fourth C2fCIB module, respectively.

[0062] The PSA module is connected with the first DUpsampling module.

[0063] The first Concat module is further connected with the third C2f module.

[0064] The second Concat module is further connected with the second C2f module.

[0065] The third Concat module is further connected with the first C2f module.

[0066] The fourth Concat module is further connected with the fourth C2f module.

[0067] The fifth Concat module is further connected with the first C2fCIB module.

[0068] The sixth Concat module is further connected with the PSA module.

[0069] In a second aspect, an embodiment of the present application provides a deep learning iron roughneck coupling detection system, comprising an industrial camera, a recognition device, and an iron roughneck central control device.

[0070] The industrial camera is configured to collect a first original image of a drilling platform surface and send the first original image to the recognition device.

[0071] The recognition device is configured to input the received first original image into a trained coupling detection model, obtain the coordinates of the coupling in the pixel coordinate system in the first original image, and generate a control instruction according to the coordinates of the coupling in the pixel coordinate system and send the control instruction to the iron roughneck central control device.

[0072] The iron roughneck central control device is configured to control the iron roughneck to make up or break out a coupling based on the received control instruction.

[0073] The first original image is an image of a wellhead and / or a mouse hole position.

[0074] The coupling detection model comprises a trunk network, a neck network and a head network;

[0075] The trunk network is used for extracting image features of the original image, and comprises joint attention modules added after two down-sampling modules of the trunk network respectively;

[0076] Any joint attention module is used for assigning attention weight to a received first original feature image along a channel dimension and a spatial dimension respectively, and performing feature fusion on the first original feature image after the attention weight assignment along the channel dimension and the spatial dimension, to obtain a feature fusion image focusing on a coupling at a wellhead and / or a mouse hole position, that is, an output result of the joint attention module;

[0077] The first original feature image is semantic information of the first original image after preliminary feature extraction and down-sampling, and the first original feature image comprises features of the coupling at the wellhead and / or the mouse hole position, and features of a background.

[0078] (Three) beneficial effects

[0079] The beneficial effects of the present application are: the deep learning iron roughneck coupling detection method of the present application uses a coupling detection model based on YOLOv10 to improve the visual recognition of the height information of the drilling tool coupling, compared with the prior art, the full intelligentization and automation of the iron roughneck are realized, the operation efficiency of the pipe string operation system is further improved, and the exposure risk of the drilling platform operator is reduced; the joint attention module is introduced into the coupling detection model, so that the coupling detection model focuses better on the coupling at the wellhead and / or the mouse hole position, the influence of background interference is reduced, and the accuracy of position detection is improved. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 It is a deep learning-based iron roughneck coupling detection method flowchart;

[0081] Figure 2 It is a joint attention module structure diagram;

[0082] Figure 3 It is a channel attention unit structure diagram;

[0083] Figure 4 It is a spatial attention unit structure diagram;

[0084] Figure 5 It is a DUpsampling module feature processing process diagram;

[0085] Figure 6 It is a coupling detection model structure diagram;

[0086] Figure 7 a to-be-detected original image of a drilling platform surface;

[0087] Figure 8 a detection effect image of a YOLOv8 model;

[0088] Figure 9 a detection effect image of a YOLOv10 model;

[0089] Figure 10 a detection effect image of a coupling detection model. DETAILED DESCRIPTION

[0090] In order to better explain the present application, so as to be understood, the present application is described in detail below through specific embodiments in combination with the accompanying drawings.

[0091] The deep learning-based iron roughneck coupling detection method provided in the embodiment of the present application uses a coupling detection model improved based on YOLOv10 to visually recognize the height information of the drilling tool coupling, compared with the prior art, realizes full intelligentization and automation of the iron roughneck, further improves the operation efficiency of the pipe string operation system, and reduces the exposure risk of the drilling platform operator; the joint attention module is introduced in the coupling detection model, so that the coupling detection model focuses better on the wellhead and / or mouse hole position coupling, reduces the influence of background interference, and improves the accuracy of position detection.

[0092] In order to better understand the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present application can be more clearly, thoroughly understood, and the scope of the present application can be completely conveyed to those skilled in the art.

[0093] Embodiment 1

[0094] The embodiment provides a deep learning-based iron roughneck coupling detection method, which is applied to the iron roughneck in the make-up / demolition of the drilling tool (drill pipe) in the tripping process, and is realized based on a pre-set recognition device, an industrial camera, an iron roughneck central control device and the iron roughneck; the industrial camera is used for collecting a first original image of a drilling platform surface and sending the first original image to the recognition device; the recognition device is used for inputting the received first original image into a trained coupling detection model, obtaining the coordinates of the coupling in the pixel coordinate system in the first original image; and generating a control instruction according to the coordinates of the coupling in the pixel coordinate system and sending the control instruction to the iron roughneck central control device; the iron roughneck central control device is used for controlling the iron roughneck to make up / demolish based on the received control instruction.

[0095] The method comprises the following steps as shown in the method Figure 1

[0096] S00, the recognition module determines the mapping relationship between the image coordinate system and the world coordinate system according to the coordinates of the three calibration points in the second original image sent by the industrial camera and the coordinates of the three calibration points in the world coordinate system, and a pre-set formula one.

[0097] The formula one is as follows:

[0098]

[0099] wherein λ is the mapping coefficient of the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1 and y2 are the coordinates of the three calibration points in the image coordinate system in the second original image; y 00 , y 11 and y 22 are the coordinates of the three calibration points in the world coordinate system. (In general, only the mapping relationship of the longitudinal coordinate, i.e. the height, is established, therefore, the selected calibration points only need to measure the pixel height and the actual height)

[0100] S01, the recognition module inputs the third original image sent by the industrial camera into a pre-set ROI model to generate a first original image; the ROI model is used for region of interest cropping on the received second original image to obtain a first original image including only the region of interest; the region of interest is the wellhead and / or the mouse hole position.

[0101] S02, the recognition module collects training images at different time points by using the pre-set industrial camera, and labels the coupling of the wellhead and / or the mouse hole position in all training images by using a labeling tool, and obtains the coordinates of the coupling of the wellhead and / or the mouse hole position corresponding to each training image in the pixel coordinate system, to obtain a training set.

[0102] The training set is input into a pre-constructed coupling detection model to obtain a trained coupling detection model, and the channel attention weight and the spatial attention weight are obtained based on the model parameters of the trained coupling detection model.

[0103] During the period, a loss function MPDIoU is calculated to optimize the coupling detection model.

[0104] S10, the recognition device inputs the received first original image of the drilling platform surface into the trained coupling detection model to obtain the coordinates of the coupling in the pixel coordinate system in the first original image.

[0105] ​S20, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and sends the control instruction to the iron roughneck control device, so that the iron roughneck control device controls the iron roughneck to perform the make-up / demolition operation;

[0106] Specifically, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and the mapping relationship between the image coordinate system and the world coordinate system, and sends the control instruction to the iron roughneck control system, so that the iron roughneck control system controls the iron roughneck to perform the make-up / demolition operation; the control instruction includes the coordinates of the coupling in the world coordinate system.

[0107] The first original image is an image of the wellhead and / or mouse hole position; the coupling detection model includes a backbone network, a neck network and a head network; wherein the backbone network is used to extract image features of the original image, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to detect the coupling coordinates according to the fused features of the neck network. Further, the backbone network includes a joint attention module added after each of the two down-sampling modules of the backbone network, and all the up-sampling modules in the neck network are replaced by DUpsampling modules.

[0108] The joint attention module is used to assign attention weight values to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weight values along the channel dimension and the spatial dimension are assigned, to obtain a feature fusion image focusing on the coupling of the wellhead and / or mouse hole position, i.e. the output result of the joint attention module; the first original feature image is semantic information of the first original image after preliminary feature extraction and down-sampling, and the first original feature image includes features of the coupling of the wellhead and / or mouse hole position, and features of the background. The feature fusion image is used for the backbone network, the neck network and the head network to obtain the coordinates of the coupling in the pixel coordinate system in the first original image based on the feature fusion image;

[0109] Any joint attention module structure is as shown in Figure 2 including a channel attention unit, a spatial attention unit and a feature fusion unit.

[0110] The channel attention unit has a structure as shown in Figure 3As shown, according to the features of the wellhead and / or rat hole position collar and the features of the background in the first original feature map, global average pooling is adopted in the input feature channel direction to compress the features in the spatial direction, then a 1×1 convolution layer is used to extract features, followed by a Sigmoid layer for normalization processing to obtain a channel attention weight matrix, and finally the first original feature map is assigned attention weights based on the channel attention weight matrix to obtain a channel attention map; the channel attention weight matrix includes the channel weight of the wellhead and / or rat hole position collar and the channel weight of the background;

[0111] Specifically: the received first original feature map is subjected to global average pooling in its input feature channel direction and a pre-set formula one to compress the features in the spatial direction; the formula one is:

[0112]

[0113] wherein, X c is the matrix in the input feature channel direction, i and j represent the position indexes of the corresponding elements of the matrix in the input feature channel direction, GAP c is the output of global average pooling, H is the height of the first original feature map, and W is the width of the first original feature map;

[0114] Then the first original feature map after global average pooling is input into a 1×1 convolution layer for feature extraction;

[0115] X1=Conv 1×1 (GAP c );

[0116] wherein, Conv 1×1 is a 1×1 grouped convolution, GAP c is the output of global average pooling, and X1 is the output feature after the 1×1 convolution layer;

[0117] Then a Sigmoid layer is used for normalization processing to obtain a channel attention weight matrix;

[0118] Out=σ(X1);

[0119] wherein, Out is the channel attention weight matrix, σ is a Sigmoid activation function, and X1 is the output feature after the 1×1 convolution layer;

[0120] Finally, the channel attention weight matrix is multiplied by each channel dimension of the first original feature map to obtain a channel attention map.

[0121] The structure of the spatial attention unit is as shown in Figure 4As shown, based on the features of the wellhead and / or rat hole position coupling and the features of the background in the first original feature map, feature extraction is performed through a 1x1 convolution layer to obtain the spatial context weight corresponding to the first original feature map, then the spatial context weight is reshaped, and the reshaped spatial context weight is normalized through a Softmax layer to obtain a spatial attention weight matrix; finally, the first original feature map is assigned an attention weight based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix includes spatial weights of the wellhead and / or rat hole position coupling and spatial weights of the background;

[0122] Specifically, the CxHxW first original feature map is input into a 1x1 convolution layer to generate spatial context weight corresponding to the first original feature map;

[0123] Then the spatial context weight is reshaped into HWx1x1;

[0124] And the reshaped spatial context weight is normalized through a Softmax layer to obtain a spatial attention weight matrix;

[0125] Then the CxHxW first original feature map is reshaped into CxHW, and after the reshaped first original feature is multiplied by the attention weight matrix and then spliced with the first original feature map, a spatial attention map is obtained;

[0126] The formula for the whole process of the spatial attention unit is:

[0127]

[0128] Where, W k is a linear transformation matrix, realized by a 1x1 convolution layer, q is an enumeration of all possible positions, N p is HxW, i.e. the number of positions in feature mapping, o is the index of the query position, x o is the first original feature map, and z o is the spatial attention map output by the spatial attention unit.

[0129] The feature fusion unit is configured to perform feature fusion on the channel attention map and the spatial attention map based on the pre-set channel attention weight and spatial attention weight to obtain a feature fusion image focusing on the wellhead and / or rat hole position coupling; the channel attention weight and the spatial attention weight are obtained by pre-training the coupling detection model.

[0130] The DUpsampling module is configured to perform channel dimension increasing on the received second original feature map through a 1*1 deep dimension increasing convolution layer, and then expand the second original feature map in the channel dimension, and then combine elements with the same position and different channels to obtain an up-sampling image of the edge features of the refined wellhead and / or mouse hole position coupling; the whole process of the DUpsampling module is as follows:

[0131] Out DUp =R(DConv 1×1 (X));

[0132] wherein, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1*1 deep dimension increasing convolution layer, R is an element rearrangement operation, and Out DUp is the up-sampling image output by the DUpsampling module;

[0133] The second original feature map is semantic information obtained by performing image feature extraction on the first original image through a backbone network.

[0134] The up-sampling image is used for the neck network and the head network to obtain the coordinates of the coupling in the pixel coordinate system in the first original image based on the up-sampling image.

[0135] The embodiment uses a coupling detection model based on the improved YOLOv10 to visually recognize the height information of the drill coupling. Compared with the prior art, the full intelligentization and automation of the iron driller are realized, the operation efficiency of the pipe string operation system is further improved, and the exposure risk of the drilling floor operator is reduced. The joint attention module is introduced into the coupling detection model, so that the coupling detection model focuses better on the wellhead and / or mouse hole position coupling, reduces the influence of background interference, and improves the accuracy of position detection.

[0136] Embodiment 2

[0137] The embodiment provides an iron driller coupling detection method based on deep learning, which comprises the following steps:

[0138] The picture information of the drilling floor is collected by an industrial camera, the collected image is cut in an ROI mode, and then the processed image is transmitted to the coupling detection model deployed on the recognition device. After inference, the coordinates of the coupling in the pixel coordinate system are obtained. Finally, the mapping relationship between the pixel coordinate system and the world coordinate system is converted to obtain the actual coordinates (usually the longitudinal coordinate) of the drill coupling. Finally, the actual coordinates are transmitted to the iron driller control device, so that the iron driller can perform make-up and break-out operations. The coupling detection model usually needs to be constructed in advance, and the coupling detection model is improved based on YOLOv10, and the structure thereof is as follows:Figure 6 As shown, the network includes: a trunk network, a neck network and a head network;

[0139] The trunk network includes: a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first down-sampling module, a first joint attention module, a third C2f module, a second down-sampling module, a second joint attention module, a first C2fCIB module, a SPPF module and a PSA module connected in sequence;

[0140] The neck network includes: a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third down-sampling module, a fifth Concat module, a third C2fCIB module, a fourth down-sampling module, a sixth Concat module and a fourth C2fCIB module connected in sequence;

[0141] The head network includes four Detct modules, and the four Detct modules are connected with the fifth C2f module, the sixth C2f module, the third C2fCIB module and the fourth C2fCIB module respectively;

[0142] The PSA module is connected with the first DUpsampling module;

[0143] The first Concat module is further connected with the third C2f module;

[0144] The second Concat module is further connected with the second C2f module;

[0145] The third Concat module is further connected with the first C2f module;

[0146] The fourth Concat module is further connected with the fourth C2f module;

[0147] The fifth Concat module is further connected with the first C2fCIB module;

[0148] The sixth Concat module is further connected with the PSA module.

[0149] Due to the complex situation of the drilling floor, especially the existence of a large number of drilling tools, the collar height of the drilling tool in the background is easily fed back to the iron driller, and then the lifting height is inaccurate. In order to focus the attention of the detection algorithm on the collar at the wellhead and the mouse hole position, the embodiment adds a joint attention module (SCAM module) after the down-sampling module of the YOLOv10 model backbone network. The joint attention module mainly has two branches of space and channel, which respectively perform attention calculation on the input features in the space and channel dimensions.

[0150] For the channel dimension, the embodiment proposes a channel attention unit without significantly increasing the parameters. In existing attention mechanisms, such as Squeeze-and-Excitation Networks (SE), additional parameters are added, which reduces the inference speed of the neural network. Specifically, the main parameters of SE are concentrated in the fully connected layer of the excitation compression stage. If multiple attention modules are embedded in the neural network, the inference speed of the detection algorithm will be greatly reduced. The channel attention unit proposed in the embodiment performs lightweight design on the original SE mechanism, inputs the features, first adopts global average pooling according to the channel direction of the input features to compress the spatial direction features, then adopts a group convolution with a kernel size of to perform feature extraction operation, converts the sampled features from to, then adopts Sigmoid to normalize the C dimension to obtain the attention weight matrix, and finally multiplies the attention weight matrix with each channel dimension of the original features to obtain the final channel attention branch result.

[0151] For the spatial dimension, unlike the existing global context attention network, the spatial attention unit of the embodiment generates a spatial context weight using a convolution, and then reshapes the generated weight to, and normalizes the attention weight matrix to 0-1 by Softmax after reshaping the spatial attention weight. Then, the original input feature is reshaped to. Finally, the reshaped original feature is multiplied by the normalized weight matrix to obtain the output of the final spatial attention branch. The specific calculation process (the whole process formula of the spatial attention unit) is as follows:

[0152]

[0153] where W k is a linear transformation matrix, implemented by a 1x1 convolution layer, q is an enumeration of all possible positions, N p is HxW, that is, the number of positions in the feature map, o is the index of the query position, x o is the first original feature map, z o is the spatial attention map output by the spatial attention unit.

[0154] Finally, the feature fusion is performed based on the preset spatial attention weight and channel attention weight by combining the outputs of the spatial attention unit and the channel attention unit, so as to obtain an output result of the joint attention module.

[0155] In addition, the position information of the target is particularly important. If there is a deviation of 1 pixel at the pixel level, an error of 5 mm or even higher will be reflected at the spatial level. In the existing target detection algorithm, multiple downsampling and upsampling operations are often required. Upsampling plays a crucial role in the target positioning task in the target detection task. Although the existing upsampling method, such as bilinear interpolation, has a significant advantage in computational efficiency, the high requirements of the target positioning task are difficult to achieve optimal results. The embodiment proposes a high-efficiency upsampling module, DUpsampling module, to replace all upsampling modules in the neck network of the YOLOv10 model.

[0156] Taking double upsampling as an example, the processing process of the DUpsampling module is as shown in Figure 5 For the input feature, the DUpsampling module first performs depth convolution with an output channel of 4 times and a kernel size of 1x1 in the channel direction to perform dimensionality increasing calculation in the channel direction. The output after dimensionality increasing is unfolded in the channel dimension in turn, and finally the elements with the same space position and different channels are spliced and combined to form the final output result. The calculation process of the DUpsampling module (the whole process formula of the DUpsampling module) is as follows: Out DUp = R(DConv 1×1 (X));

[0157] Wherein, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1x1 depth dimension increasing convolution layer, R is an element rearrangement operation, and Out DUp is the upsampling image output by the DUpsampling module.

[0158] The baseline model used in the embodiment is YOLOv10. Then, the joint attention module and the DUpsampling module are introduced, and the regression frame loss is optimized to design a joint attention model suitable for the existing task, as shown in Figure 6As shown, after the second and third stage down-sampling modules of the backbone network (i.e., the first down-sampling module and the second down-sampling module), a joint attention module is added; in the deep features of the backbone network, there is more semantic information, but since the task needs to identify the drill collar in the wellhead and the mouse hole position, the features of the collar in the background are ignored. The joint attention module can make the network reduce the influence of background interference and better focus on the corresponding position. In the neck network part, the original up-sampling module is replaced with a DUpsampling module, which can refine the edge features of the object and is more conducive to the regression task of drill collar detection. Finally, the original regression loss is replaced with MPDIoU to optimize the error of the anchor box regression task and improve the positioning accuracy of the object.

[0159] The embodiment provides a performance comparison between models, where FLOPs represents the number of floating-point operations, which can be used to measure the complexity of the model. mAP represents the average precision mean, which reflects the average precision of all classes. As shown in Table 1, a good balance between the precision and FLOPs of the collar detection model is achieved. Specifically, in the case of close parameter quantity, the collar detection model is compared with existing mainstream target detection algorithms (such as YOLOv8, YOLOv7 and YOLOv6, etc.), and the specific performance comparison is shown in Table 1:

[0160] Table 1: Model performance comparison table

[0161]

[0162] Compared with other existing target detection models, the mAP of the collar detection model is improved by 2.5% compared with YOLOv8-S, and by 2.9% compared with YOLOv6-S, and the parameter quantity is only that of YOLOv6-S. Although the parameter quantity of the collar detection model is higher than that of YOLOv7-tiny, the mAP is improved by 3.6%, and the high-precision detection algorithm is more advantageous for the judgment of the height of the joint. These results show that the collar detection model proposed in the embodiment is more suitable for the detection task in the field of oil drilling and production, and is an effective target detection algorithm.

[0163] In order to further demonstrate the effectiveness of the algorithm proposed in the embodiment, visual display is carried out according to the data collected on site, and comparison is made with existing algorithms, and the results are shown in Figure 7 to Figure 10 . Figure 7 The original image to be detected on the drilling platform surface collected by the industrial camera is shown in Figure 8 to Figure 10 , and the detection effects of YOLOv8, YOLOv10 and the collar detection model are shown in Figure 8 , respectively. As shown in the detection results, there is a large gap between the pin and the box, and in the actual situation Figure 7 , there is no obvious gap between the drill pipe.Figure 9 In the middle, the gap between the pin and the box is obviously reduced, but there is still a small amount of offset, which is still not conducive to the judgment of the high-precision drill tool joint height. And the method of the present application is adopted Figure 10 As can be seen in the middle, the joint detection model proposed in the embodiment has basically no gap between the detection results, basically meeting the high-precision labeling frame requirement.

[0164] Embodiment 3

[0165] The embodiment proposes a deep learning iron roughneck joint detection system, including an industrial camera, an identification device and an iron roughneck central control device;

[0166] The industrial camera is used to collect the first original image of the drilling platform surface and send it to the identification device;

[0167] The identification device is used to input the received first original image into the trained joint detection model to obtain the coordinates of the joint in the pixel coordinate system in the first original image; and according to the coordinates of the joint in the pixel coordinate system, generate a control instruction and send it to the iron roughneck central control device;

[0168] The iron roughneck central control device is used to control the iron roughneck to make up or break out based on the received control instruction.

[0169] The joint detection model is the joint detection model proposed in embodiment 1 or embodiment 2.

[0170] The deep learning iron roughneck joint detection system proposed in the embodiment is used to realize the deep learning iron roughneck joint detection method in embodiment 1 or embodiment 2.

[0171] The deep learning iron roughneck joint detection system proposed in the embodiment provides structural support for the deep learning iron roughneck joint detection method of embodiment 1 or embodiment 2, realizes the full intelligentization and automation of the iron roughneck, further improves the operation efficiency of the pipe string operation system, and reduces the exposure risk of the drilling platform operator.

[0172] In the description of the present application, it should be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0173] In the present application, unless specifically defined otherwise, the terms "mounting", "connected", "connection", "fixed", "unfixed", and the like should be construed broadly, for example, can be fixed connection, can be detachable connection, or integral; can be mechanical connection, can be electrical connection; can be direct connection, can be indirect connection through an intermediate medium; can be internal communication of two elements or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0174] In the present application, unless specifically defined otherwise, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. Moreover, the first feature is "above", "over" and "on" the second feature, which can be directly above or obliquely above the first feature, or only indicates that the first feature is higher than the second feature in horizontal height. The first feature is "below", "under" and "under" the second feature, which can be directly below or obliquely below the first feature, or only indicates that the first feature is lower than the second feature in horizontal height.

[0175] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0176] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can modify, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for detecting iron drill couplings based on deep learning, characterized in that, include: S10. The recognition device inputs the received first original image of the drilling platform into the trained coupling detection model to obtain the coordinates of the coupling in the pixel coordinate system in the first original image; the first original image is an image of the wellhead and / or the location of the mouse hole; The coupling detection model includes a backbone network, a neck network, and a head network; The backbone network is used to extract image features from the original image, including a joint attention module added after each of the two downsampling modules in the backbone network; Any of the aforementioned joint attention modules are used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension, respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned along the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or mouse hole location coupling, which is the output result of the joint attention module; The first original feature image is the semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes the features of the wellhead and / or mouse hole location coupling, as well as the background features; The feature fusion image is used by the subsequent backbone network, neck network, and head network to obtain the coordinates of the collar in the pixel coordinate system in the first original image based on the feature fusion image; S20. The identification device generates a control command based on the coordinates of the coupling in the pixel coordinate system and sends it to the iron driller's central control device so that the iron driller's central control device can control the iron driller to perform the coupling / uncoupling operation.

2. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, Any of the aforementioned joint attention modules includes a spatial attention unit, a channel attention unit, and a feature fusion unit; The channel attention unit is used to perform spatial feature compression by global average pooling according to the input feature channel direction based on the features of the wellhead and / or mouse hole location coupling in the first original feature map and the background features, then extract features through a 1×1 convolutional layer, and then normalize them through a Sigmoid layer to obtain the channel attention weight matrix. Finally, attention weights are assigned to the first original feature map based on the channel attention weight matrix to obtain the channel attention map. The channel attention weight matrix includes the channel weights of the wellhead and / or mouse hole location couplings, as well as the channel weights of the background; The spatial attention unit is used to extract features based on the features of the wellhead and / or mouse hole location coupling in the first original feature map, as well as the background features, through a 1×1 convolutional layer to obtain the spatial context weights corresponding to the first original feature map. Then, the spatial context weights are reshaped and normalized through a Softmax layer to obtain the spatial attention weight matrix. Finally, attention weights are assigned to the first original feature map based on the spatial attention weight matrix to obtain the spatial attention map. The spatial attention weight matrix includes the spatial weights of the wellhead and / or mouse hole location couplings, as well as the spatial weights of the background; The feature fusion unit is used to fuse the channel attention map and the spatial attention map based on the pre-set channel attention weight and spatial attention weight to obtain a feature fusion image focusing on the wellhead and / or mouse hole location coupling; the channel attention weight and spatial attention weight are both obtained by pre-training the coupling detection model.

3. The deep learning-based method for detecting iron drill couplings according to claim 2, characterized in that, The channel attention unit, based on the features of the wellhead and / or mouse hole locations in the first original feature map, and the background features, performs spatial feature compression using global average pooling according to the input feature channel direction. Then, it extracts features through a 1×1 convolutional layer, followed by normalization through a Sigmoid layer to obtain the channel attention weight matrix. Finally, it assigns attention weights to the first original feature map based on the channel attention weight matrix to obtain the channel attention map, including: The received first raw feature map is subjected to global average pooling according to its input feature channel direction and a pre-set formula (Formula 1) to compress the spatial features; the formula (Formula 1) is: Among them, X c Let i and j be the matrix representing the input feature channel directions, respectively, and represent the indexes of the corresponding elements in the matrix representing the input feature channel directions. c The output of global average pooling is H, where H is the height of the first original feature map and W is the width of the first original feature map. The first original feature map after global average pooling is then input into a 1×1 convolutional layer for feature extraction. X1=Conv 1×1 (GAP c ); Among them, Conv 1×1 For 1×1 grouped convolution, GAP c X1 is the output of global average pooling, and X2 is the output feature after passing through a 1×1 convolutional layer. Then, the channel attention weight matrix is ​​obtained by normalization through a Sigmoid layer. Out = σ(X1); Where Out is the channel attention weight matrix, σ is the Sigmoid activation function, and X1 is the output feature after a 1×1 convolutional layer; Finally, the channel attention weight matrix is ​​multiplied by each channel dimension of the first original feature map to obtain the channel attention map.

4. The deep learning-based method for detecting iron drill couplings according to claim 2, characterized in that, The spatial attention unit is used to extract features based on the features of the wellhead and / or mouse hole location coupling in the first original feature map, as well as the background features, through a 1×1 convolutional layer to obtain the spatial context weights corresponding to the first original feature map. Then, the spatial context weights are reshaped and normalized through a Softmax layer to obtain the spatial attention weight matrix. Finally, attention weights are assigned to the first original feature map based on the spatial attention weight matrix to obtain the spatial attention map. The spatial attention weight matrix includes spatial weights for wellhead and / or mouse hole location couplings, and spatial weights for the background, including: The first original feature map, which receives C×H×W, is input into a 1×1 convolutional layer to generate the spatial context weights corresponding to the first original feature map. The spatial context weights were then reshaped to HW×1×1; The reshaped spatial context weights are then normalized using a Softmax layer to obtain the spatial attention weight matrix. Subsequently, the first original feature map of C×H×W is reshaped into C×HW, and the reshaped first original feature map is multiplied by the attention weight matrix and then concatenated with the first original feature map to obtain the spatial attention map. The complete processing formula for the spatial attention unit is as follows: Among them, W k The transformation matrix is ​​a linear transformation matrix, implemented using a 1×1 convolutional layer, where q represents the number of possible positions, and N is the number of possible positions. p H×W represents the number of positions in the feature map, o is the index of the query position, and x o For the first original feature map, z o The spatial attention map output by the spatial attention unit.

5. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, All upsampling modules in the neck network of the coupling detection model are DUpsampling modules; The DUpsampling module is used to increase the dimensionality of the received second original feature map through a 1×1 depth-up convolutional layer. Then, the increased dimensionality second original feature map is unfolded according to the channel dimension, and elements with the same empty positions but different channels are concatenated and combined to obtain an upsampled image with refined edge features of the wellhead and / or mouse hole location couplings. The entire processing formula of the DUpsampling module is as follows: Out DUp =R(DConv 1×1 (X)); Where X is the second raw feature map received by the DUpsampling module, and DConv 1×1 It is a 1×1 depthwise up-dimensional convolutional layer, R is the element rearrangement operation, and Out is... DUp The upsampled image output by the DUpsampling module; The second original feature map is the semantic information of the first original image after image feature extraction through the backbone network; The upsampled image is used by the subsequent neck and head networks to obtain the coordinates of the collar in the pixel coordinate system in the first original image based on the upsampled image.

6. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, Before S10, the following is also included: S00. The recognition module determines the mapping relationship between the image coordinate system and the world coordinate system based on the coordinates of three pre-calibrated calibration points in the image coordinate system and the coordinates of three pre-set calibration points in the world coordinate system in the second raw image sent by the industrial camera, as well as the pre-set formula one. Formula 1 is: Where λ is the mapping coefficient between the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1, and y2 are the coordinates of the three calibration points in the second original image in the image coordinate system, respectively; y 00 y 11 and y 22 These are the coordinates of the three calibration points in the world coordinate system; Then S20 includes: The identification device generates control commands based on the coordinates of the coupling in the pixel coordinate system and the pre-set mapping relationship between the image coordinate system and the world coordinate system, and sends them to the iron driller's central control system so that the iron driller's central control system can control the iron driller to perform coupling / uncoupling operations; the control commands include the coordinates of the coupling in the world coordinate system.

7. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, Before S10, the following is also included: S01. The recognition module inputs the third raw image sent by the industrial camera into a pre-set ROI model to generate a first raw image; the ROI model is used to crop the region of interest of the received second raw image to obtain a first raw image that only includes the region of interest; the region of interest is the location of the wellhead and / or the mouse hole.

8. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, Before S10, the following is also included: S02. The recognition module acquires training images at different times using a pre-set industrial camera, and uses a labeling tool to label the couplings at the wellhead and / or mouse hole locations in all training images, as well as the coordinates of the couplings at the wellhead and / or mouse hole locations corresponding to each training image in the pixel coordinate system, to obtain the training set. Input the training set into the pre-built coupling detection model to obtain the trained coupling detection model; During this process, the loss function MPDIoU was calculated, and the hoop detection model was optimized.

9. The deep learning-based method for detecting iron drill couplings according to claim 1, characterized in that, The neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform hoop coordinate detection based on the features fused by the neck network. The backbone network includes: a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first downsampling module, a first joint attention module, a third C2f module, a second downsampling module, a second joint attention module, a first C2fCIB module, an SPPF module, and a PSA module connected in sequence. The neck network includes: a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third downsampling module, a fifth Concat module, a third C2fCIB module, a fourth downsampling module, a sixth Concat module, and a fourth C2fCIB module connected in sequence. The head network includes four Dectt modules, which are connected to the fifth C2f module, the sixth C2f module, the third C2fCIB module, and the fourth C2fCIB module, respectively. The PSA module is connected to the first DUpsampling module. The first Concat module is then connected to the third C2f module; The second Concat module is then connected to the second C2f module; The third Concat module is then connected to the first C2f module; The fourth Concat module is connected to the fourth C2f module; The fifth Concat module is then connected to the first C2fCIB module; The sixth Concat module is connected to the PSA module.

10. A deep learning-based system for detecting iron drill couplings, characterized in that, This includes industrial cameras, identification devices, and central control devices for iron drills; The industrial camera is used to capture a first raw image of the drill table surface and send it to the recognition device. The recognition device is used to input the received first original image into the trained coupling detection model to obtain the coordinates of the coupling in the pixel coordinate system in the first original image; and generate control commands based on the coordinates of the coupling in the pixel coordinate system and send them to the iron driller central control device. The central control device for the iron drill operator is used to control the iron drill operator to engage / disengage the hook based on the received control commands. The first original image is an image of the location of the wellhead and / or mouse hole; The coupling detection model includes a backbone network, a neck network, and a head network; The backbone network is used to extract image features from the original image, including a joint attention module added after each of the two downsampling modules in the backbone network; Any of the aforementioned joint attention modules are used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension, respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned along the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or mouse hole location coupling, which is the output result of the joint attention module; The first original feature image is the semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes features of the wellhead and / or mouse hole location coupling, as well as background features.

Images