Deep learning iron roughneck coupling detection method and system

Abstract

The invention relates to the technical field of petroleum drilling and production, in particular to an iron roughneck coupling detection method and system based on deep learning, an identification device inputs a first original image into a coupling detection model to obtain coordinates of a coupling in a pixel coordinate system; the backbone network of the coupling detection model is used for extracting image features of an original image, and comprises joint attention modules respectively added behind two down-sampling modules of the backbone network; the joint attention module is used for carrying out attention weight assignment on the first original feature image along a channel dimension and a space dimension and then carrying out feature fusion to obtain a feature fusion image; and the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and sends the control instruction to the iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform screwing-on / screwing-off operation. The system has the beneficial effects that full intelligence and automation of iron roughneck control are realized through the high-precision coupling detection model.

Description

Technical Field

[0001] The present invention relates to the field of oil drilling technology, and in particular to a deep learning iron roughneck collar detection method and system. Background Art

[0002] With the rapid development of automation technology, the work efficiency in the field of oil drilling has been greatly improved. During the drilling operation, as the drilling depth gradually increases, the tripping operation occupies most of the time.

[0003] As one of the main devices concerned with operating systems, the iron drill is currently widely used. At present, although the iron drill can automatically make and break the operation, the operator still needs to control the iron drill to align the position of the coupling during use to achieve the make / break operation. This places high demands on the operator's operating specifications. Once an incorrect operation occurs, it will cause huge property losses. In recent years, with the increasing maturity of machine vision technology, this technology has been applied to many fields such as autonomous driving and visual-assisted clamping. Computer vision-assisted iron drill coupling height positioning has become one of the most effective solutions.

[0004] The YOLO series of algorithms can better balance speed and accuracy and are widely used in various fields. However, due to the fact that the mission scenarios in the oil drilling process are more complex than in other fields and the on-site environment is more severe, the original YOLO series model cannot meet the accuracy requirements of the iron roughneck during the process of making / breaking the buckle. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a deep learning method and system for detecting couplings for roughnecks, which solves the technical problems that when manually aligning the coupling position, high requirements are placed on the operating specifications of personnel and the risks are relatively high, and the original YOLO series model cannot meet the technical problem of the accuracy requirements of the roughnecks during the making / breaking process of the roughnecks due to the complex on-site task scenes and harsh environment.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for detecting an iron roughneck collar based on deep learning, comprising:

[0010] S10, the recognition device inputs the received first original image of the drilling floor into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system; the first original image is an image of the wellhead and / or rathole position;

[0011] The collar detection model includes a backbone network, a neck network and a head network;

[0012] The backbone network is used to extract image features of the original image, including a joint attention module added after two downsampling modules of the backbone network respectively;

[0013] Any of the joint attention modules is used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned in the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or the rathole position collar, i.e., the output result of the joint attention module;

[0014] The first original feature image is the semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes the features of the collar at the wellhead and / or rathole position, and the features of the background; the feature fusion image is used by the subsequent backbone network, neck network and head network to obtain the coordinates of the collar in the first original image in the pixel coordinate system based on the feature fusion image;

[0015] S20, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and sends the control instruction to the iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform the make / break operation.

[0016] The deep learning iron roughneck coupling detection method according to claim 1, characterized in that any of the joint attention modules includes a spatial attention unit, a channel attention unit and a feature fusion unit;

[0017] The channel attention unit is used to, according to the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, perform feature compression in the spatial direction by using global average pooling according to the input feature channel direction, perform feature extraction through a 1×1 convolution layer, and then perform normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assign attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map; the channel attention weight matrix includes channel weights of the collar at the wellhead and / or rathole position, and channel weights of the background;

[0018] The spatial attention unit is used to extract features through a 1×1 convolution layer based on the features of the collar at the wellhead and / or rathole position in the first original feature map, as well as the features of the background, to obtain the spatial context weight corresponding to the first original feature map, and then reshape the spatial context weight, and normalize the reshaped spatial context weight through a Softmax layer to obtain a spatial attention weight matrix; finally, the first original feature map is assigned attention weights based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix includes the spatial weights of the collar at the wellhead and / or rathole position, as well as the spatial weights of the background;

[0019] The feature fusion unit is used to perform feature fusion of the channel attention map and the spatial attention map based on the pre-set channel attention weight and spatial attention weight, so as to obtain a feature fusion image focusing on the collar at the wellhead and / or rathole position; the channel attention weight and the spatial attention weight are both obtained by pre-training the collar detection model.

[0020] Optionally, the channel attention unit, according to the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, uses global average pooling to perform feature compression in the spatial direction according to the input feature channel direction, extracts features through a 1×1 convolution layer, then performs normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assigns attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map, including:

[0021] The received first original feature map is subjected to global average pooling according to its input feature channel direction and the preset formula 1 to compress the features in the spatial direction; the formula 1 is:

[0022]

[0023] Among them, X c is the matrix of the input feature channel direction, i and j represent the position index of the corresponding element of the matrix of the input feature channel direction, GAP c is the output of global average pooling, H is the height of the first original feature map, and W is the width of the first original feature map;

[0024] Then the first original feature map after global average pooling is input into the 1×1 convolution layer for feature extraction;

[0025] X1=Conv 1×1 (GAP c );

[0026] Among them, Conv 1×1 is a 1×1 group convolution, GAPc is the output of global average pooling, X1 is the output feature after the 1×1 convolution layer;

[0027] Then normalize it through the Sigmoid layer to obtain the channel attention weight matrix;

[0028] Out = σ(X1);

[0029] Among them, Out is the channel attention weight matrix, σ is the Sigmoid activation function, and X1 is the output feature after the 1×1 convolution layer;

[0030] Finally, the channel attention weight matrix is ​​multiplied by each channel dimension of the first original feature map to obtain the channel attention map.

[0031] Optionally, the spatial attention unit is used to extract features through a 1×1 convolution layer based on the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, to obtain the spatial context weight corresponding to the first original feature map, and then reshape the spatial context weight, and normalize the reshaped spatial context weight through a Softmax layer to obtain a spatial attention weight matrix; finally, the first original feature map is assigned attention weights based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix includes the spatial weights of the collar at the wellhead and / or rathole position, and the spatial weights of the background, including:

[0032] Input the received C×H×W first original feature map into a 1×1 convolutional layer to generate a spatial context weight corresponding to the first original feature map;

[0033] The spatial context weights are then reshaped to HW×1×1;

[0034] The reshaped spatial context weights are normalized through the Softmax layer to obtain the spatial attention weight matrix;

[0035] Then, after reshaping the first original feature map of C×H×W into C×HW, the reshaped first original feature is multiplied by the attention weight matrix and then concatenated with the first original feature map to obtain a spatial attention map;

[0036] The whole process processing formula of the spatial attention unit is:

[0037]

[0038] Among them, W k is a linear transformation matrix, implemented with a 1×1 convolutional layer, q is the enumeration of all possible positions, N pis H×W, the number of locations in the feature map, o is the index of the query location, and x o is the first original feature map, z o The spatial attention map output by the spatial attention unit.

[0039] Optionally, all upsampling modules in the neck network of the collar detection model are DUpsampling modules;

[0040] The DUpsampling module is used to perform channel dimension upscaling on the received second original feature map through a 1×1 deep dimension upscaling convolution layer, and then after the dimension upscaling of the second original feature map is expanded according to the channel dimension, the elements with the same vacancy position and different channels are spliced ​​and combined to obtain an upsampled image of the edge features of the collar at the refined wellhead and / or rathole position; the whole process processing formula of the DUpsampling module is:

[0041] Out DUp =R(DConv 1×1 (X));

[0042] Among them, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1×1 depth-dimensional convolutional layer, R is an element rearrangement operation, Out DUp The upsampled image output by the DUpsampling module;

[0043] The second original feature map is the semantic information of the first original image after image features are extracted through the backbone network;

[0044] The upsampled image is used by a subsequent neck network and a head network to obtain the coordinates of the collar in the first original image in a pixel coordinate system based on the upsampled image.

[0045] Optionally, before S10, the step further includes:

[0046] S00, the recognition module determines the mapping relationship between the image coordinate system and the world coordinate system according to the coordinates of the three pre-calibrated calibration points in the second original image sent by the industrial camera received in the image coordinate system and the coordinates of the three pre-set calibration points in the world coordinate system, and the pre-set formula 1;

[0047] The formula 1 is:

[0048]

[0049] Where λ is the mapping coefficient between the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1 and y2 are the coordinates of the three calibration points in the second original image in the image coordinate system; y00 ,y 11 and 22 are the coordinates of the three calibration points in the world coordinate system;

[0050] Then the S20 includes:

[0051] The recognition device generates a control instruction based on the coordinates of the coupling in the pixel coordinate system and the mapping relationship between the preset image coordinate system and the world coordinate system, and sends it to the iron roughneck central control system, so that the iron roughneck central control system controls the iron roughneck to perform the bolting / breaking operation; the control instruction includes the coordinates of the coupling in the world coordinate system.

[0052] Optionally, before S10, the step further includes:

[0053] S01. The recognition module inputs the third original image received from the industrial camera into a preset ROI model to generate a first original image; the ROI model is used to crop the region of interest of the received second original image to obtain a first original image including only the region of interest; the region of interest is the wellhead and / or rathole location.

[0054] Optionally, before S10, the step further includes:

[0055] S02, the recognition module collects training images at different times through a pre-set industrial camera, and uses a labeling tool to label the collars at the wellhead and / or rathole positions in all training images, as well as the coordinates of the collars at the wellhead and / or rathole positions corresponding to each training image in a pixel coordinate system, to obtain a training set;

[0056] Input the training set into a pre-built collar detection model to obtain a trained collar detection model;

[0057] During this period, the loss function MPDIoU is calculated to optimize the hoop detection model.

[0058] Optionally, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform collar coordinate detection based on the fused features of the neck network;

[0059] The backbone network includes: a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first downsampling module, a first joint attention module, a third C2f module, a second downsampling module, a second joint attention module, a first C2fCIB module, an SPPF module and a PSA module connected in sequence;

[0060] The neck network includes: a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third down-sampling module, a fifth Concat module, a third C2fCIB module, a fourth down-sampling module, a sixth Concat module and a fourth C2fCIB module connected in sequence;

[0061] The head network includes four Detct modules, and the four Detct modules are respectively connected to the fifth C2f module, the sixth C2f module, the third C2fCIB module and the fourth C2fCIB module;

[0062] Wherein, the PSA module is connected to the first DUpsampling module;

[0063] The first Concat module is in turn connected to the third C2f module;

[0064] The second Concat module is connected to the second C2f module;

[0065] The third Concat module is connected to the first C2f module;

[0066] The fourth Concat module is connected to the fourth C2f module;

[0067] The fifth Concat module is connected to the first C2fCIB module;

[0068] The sixth Concat module is in turn connected to the PSA module.

[0069] In a second aspect, an embodiment of the present invention provides a deep learning iron roughneck collar detection system, including an industrial camera, a recognition device, and an iron roughneck central control device;

[0070] The industrial camera is used to collect a first original image of the drilling floor surface and send it to the recognition device;

[0071] The recognition device is used to input the received first original image into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system; and generate a control instruction according to the coordinates of the coupling in the pixel coordinate system and send it to the iron roughneck central control device;

[0072] The iron roughneck central control device is used to control the iron roughneck to make / break out the hook based on the received control instructions;

[0073] The first original image is an image of the wellhead and / or rathole location;

[0074] The collar detection model includes a backbone network, a neck network and a head network;

[0075] The backbone network is used to extract image features of the original image, including a joint attention module added after two downsampling modules of the backbone network respectively;

[0076] Any of the joint attention modules is used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned in the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or the rathole position collar, i.e., the output result of the joint attention module;

[0077] The first original feature image is semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes features of the collar at the wellhead and / or rathole position, and features of the background.

[0078] (III) Beneficial effects

[0079] The beneficial effects of the present invention are as follows: a deep learning method for detecting couplings for iron roughnecks of the present invention adopts an improved coupling detection model based on YOLOv10 to visually identify the height information of the drilling tool coupling. Compared with the prior art, the method realizes full intelligence and automation of the iron roughneck, while further improving the operating efficiency of the pipe string operating system and reducing the exposure risk of operators on the drilling floor; and a joint attention module is introduced into the coupling detection model, so that the coupling detection model can better focus on the couplings at the wellhead and / or rathole position, reduce the influence of background interference, and improve the accuracy of position detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of an iron roughneck collar detection method based on deep learning;

[0081] Figure 2 Schematic diagram of the joint attention module structure;

[0082] Figure 3 Schematic diagram of the channel attention unit structure;

[0083] Figure 4 Schematic diagram of the spatial attention unit structure;

[0084] Figure 5 It is a schematic diagram of the feature processing process of the DUpsampling module;

[0085] Figure 6 It is a schematic diagram of the structure of the coupling detection model;

[0086] Figure 7 is the original image of the drilling table to be detected;

[0087] Figure 8 This is the detection effect image of the YOLOv8 model;

[0088] Fig. 9 This is the detection effect image of the YOLOv10 model;

[0089] Fig.10 This is the detection effect image of the collar detection model. DETAILED DESCRIPTION

[0090] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0091] A deep learning method for detecting couplings for iron roughnecks proposed in an embodiment of the present invention adopts an improved coupling detection model based on YOLOv10 to visually identify the height information of the drilling tool coupling. Compared with the prior art, this method realizes full intelligence and automation of the iron roughneck, further improves the operating efficiency of the pipe string operating system, and reduces the exposure risk of operators on the drilling floor. A joint attention module is introduced into the coupling detection model, so that the coupling detection model can better focus on the couplings at the wellhead and / or rathole position, reduce the influence of background interference, and improve the accuracy of position detection.

[0092] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0093] Example 1

[0094] The present embodiment provides a method for detecting couplings for an iron roughneck based on deep learning, which is applied to the process of the iron roughneck performing coupling-on / off-coupling operations on a drill tool (drill rod) by the iron roughneck during the process of pulling out / down drilling, and is implemented based on a pre-set recognition device, an industrial camera, a central control device for the iron roughneck and the iron roughneck; the industrial camera is used to collect a first original image of the drill floor and send it to the recognition device; the recognition device is used to input the received first original image into a trained coupling detection model to obtain the coordinates of the coupling in the pixel coordinate system in the first original image; and according to the coordinates of the coupling in the pixel coordinate system, a control instruction is generated and sent to the central control device for the iron roughneck; the central control device for the iron roughneck is used to control the iron roughneck to perform coupling-on / off-coupling based on the received control instruction.

[0095] This method is Figure 1 As shown, including:

[0096] S00, the recognition module determines the mapping relationship between the image coordinate system and the world coordinate system according to the coordinates of the three pre-calibrated calibration points in the second original image sent by the industrial camera received in the image coordinate system and the coordinates of the three pre-set calibration points in the world coordinate system, and the pre-set formula 1;

[0097] The formula 1 is:

[0098]

[0099] Where λ is the mapping coefficient between the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1 and y2 are the coordinates of the three calibration points in the second original image in the image coordinate system; y 00 ,y 11 and 22 These are the coordinates of the three calibration points in the world coordinate system. (Usually, only the vertical coordinate, i.e. the mapping relationship of the height, is established. Therefore, the selected calibration point only needs to measure its pixel height and actual height)

[0100] S01. The recognition module inputs the third original image received from the industrial camera into a preset ROI model to generate a first original image; the ROI model is used to crop the region of interest of the received second original image to obtain a first original image including only the region of interest; the region of interest is the wellhead and / or rathole location.

[0101] S02, the recognition module collects training images at different times through a pre-set industrial camera, and uses a labeling tool to label the collars at the wellhead and / or rathole positions in all training images, as well as the coordinates of the collars at the wellhead and / or rathole positions corresponding to each training image in a pixel coordinate system, to obtain a training set;

[0102] The training set is input into a pre-built hoop detection model to obtain a trained hoop detection model, and the channel attention weight and the spatial attention weight are obtained based on the model parameters of the trained hoop detection model.

[0103] During this period, the loss function MPDIoU is calculated to optimize the hoop detection model.

[0104] S10, the recognition device inputs the received first original image of the drill floor into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system.

[0105] S20, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and sends the control instruction to the iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform the hook-up / hooking-out operation;

[0106] Specifically, the recognition device generates a control instruction based on the coordinates of the coupling in the pixel coordinate system and the pre-set mapping relationship between the image coordinate system and the world coordinate system, and sends it to the iron roughneck central control system, so that the iron roughneck central control system controls the iron roughneck to perform the buckle-on / off operation; the control instruction includes the coordinates of the coupling in the world coordinate system.

[0107] The first original image is an image of the wellhead and / or rathole location; the collar detection model includes a backbone network, a neck network and a head network; wherein the backbone network is used to extract image features of the original image, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform collar coordinate detection based on the features fused by the neck network. Furthermore, the backbone network includes a joint attention module added after two downsampling modules of the backbone network, and all upsampling modules in the neck network are replaced by DUpsampling modules.

[0108] The joint attention module is used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned in the channel dimension and the spatial dimension, to obtain a feature fusion image focused on the collar at the wellhead and / or rathole position, i.e., the output result of the joint attention module; the first original feature image is the semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes the features of the collar at the wellhead and / or rathole position, as well as the features of the background. The feature fusion image is used by the subsequent backbone network, neck network, and head network to obtain the coordinates of the collar in the first original image in the pixel coordinate system based on the feature fusion image;

[0109] Any joint attention module structure such as Figure 2 As shown, it includes a channel attention unit, a spatial attention unit, and a feature fusion unit.

[0110] The channel attention unit has the following structure: Figure 3As shown, it is used to, according to the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, after using global average pooling to perform feature compression in the spatial direction according to the input feature channel direction, perform feature extraction through a 1×1 convolution layer, and then perform normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assign attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map; the channel attention weight matrix includes channel weights of the collar at the wellhead and / or rathole position, and channel weights of the background;

[0111] Specifically, the received first original feature map is subjected to global average pooling according to the input feature channel direction and the preset formula 1 to compress the features in the spatial direction; the formula 1 is:

[0112]

[0113] Among them, X c is the matrix of the input feature channel direction, i and j represent the position index of the corresponding element of the matrix of the input feature channel direction, GAP c is the output of global average pooling, H is the height of the first original feature map, and W is the width of the first original feature map;

[0114] Then the first original feature map after global average pooling is input into the 1×1 convolution layer for feature extraction;

[0115] X1=Conv 1×1 (GAP c );

[0116] Among them, Conv 1×1 is a 1×1 group convolution, GAP c is the output of global average pooling, X1 is the output feature after the 1×1 convolution layer;

[0117] Then normalize it through the Sigmoid layer to obtain the channel attention weight matrix;

[0118] Out = σ(X1);

[0119] Among them, Out is the channel attention weight matrix, σ is the Sigmoid activation function, and X1 is the output feature after the 1×1 convolution layer;

[0120] Finally, the channel attention weight matrix is ​​multiplied by each channel dimension of the first original feature map to obtain the channel attention map.

[0121] The spatial attention unit has the following structure: Figure 4As shown, it is used to extract features through a 1×1 convolution layer based on the features of the collar at the wellhead and / or rathole position in the first original feature map, as well as the features of the background, to obtain the spatial context weight corresponding to the first original feature map, and then reshape the spatial context weight, and normalize the reshaped spatial context weight through a Softmax layer to obtain a spatial attention weight matrix; finally, the first original feature map is assigned attention weights based on the spatial attention weight matrix to obtain a spatial attention map; the spatial attention weight matrix includes the spatial weights of the collar at the wellhead and / or rathole position, as well as the spatial weights of the background;

[0122] Specifically, the first original feature map of C×H×W is input into a 1×1 convolution layer to generate a spatial context weight corresponding to the first original feature map;

[0123] The spatial context weights are then reshaped to HW×1×1;

[0124] The reshaped spatial context weights are normalized through the Softmax layer to obtain the spatial attention weight matrix;

[0125] Then, after reshaping the first original feature map of C×H×W into C×HW, the reshaped first original feature is multiplied by the attention weight matrix and then concatenated with the first original feature map to obtain a spatial attention map;

[0126] The whole process processing formula of the spatial attention unit is:

[0127]

[0128] Among them, W k is a linear transformation matrix, implemented with a 1×1 convolutional layer, q is the enumeration of all possible positions, N p is H×W, the number of locations in the feature map, o is the index of the query location, and x o is the first original feature map, z o The spatial attention map output by the spatial attention unit.

[0129] The feature fusion unit is used to perform feature fusion of the channel attention map and the spatial attention map based on the pre-set channel attention weight and spatial attention weight, so as to obtain a feature fusion image focusing on the collar at the wellhead and / or rathole position; the channel attention weight and the spatial attention weight are both obtained by pre-training the collar detection model.

[0130] The DUpsampling module is used to perform channel dimension upsampling on the received second original feature map through a 1×1 deep dimension upsampling convolution layer, and then expands the second original feature map after dimension upsampling according to the channel dimension, and then combines the elements with the same vacancy position and different channels to obtain an upsampled image of the edge features of the collar at the refined wellhead and / or rathole position; the whole process processing formula of the DUpsampling module is:

[0131] Out DUp =R(DConv 1×1 (X));

[0132] Among them, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1×1 depth-dimensional convolutional layer, R is an element rearrangement operation, Out DUp The upsampled image output by the DUpsampling module;

[0133] The second original feature map is the semantic information of the first original image after image features are extracted through the backbone network;

[0134] The upsampled image is used by a subsequent neck network and a head network to obtain the coordinates of the collar in the first original image in a pixel coordinate system based on the upsampled image.

[0135] This embodiment adopts the improved coupling detection model based on YOLOv10 to visually identify the height information of the drilling tool coupling. Compared with the existing technology, it realizes the full intelligence and automation of the iron roughneck, further improves the operating efficiency of the pipe string operating system, and reduces the exposure risk of operators on the drilling floor; and the coupling detection model introduces a joint attention module, so that the coupling detection model can better focus on the couplings at the wellhead and / or rathole position, reduce the impact of background interference, and improve the accuracy of position detection.

[0136] Example 2

[0137] This embodiment proposes a method for detecting an iron roughneck collar based on deep learning, comprising:

[0138] The image information of the drilling table is collected by an industrial camera, and the collected image is cropped using the ROI method. Then the processed image is passed to the coupling detection model deployed in the recognition device. After reasoning, the coordinates of the coupling in the pixel coordinate system are obtained. Finally, through the mapping relationship between the pixel coordinate system and the world coordinate system, the actual coordinates of the drill coupling (usually the vertical coordinate) can be obtained, and finally passed to the iron driller's central control device to enable the iron driller to perform the buckle-on / off operation. The coupling detection model usually needs to be constructed in advance, and the coupling detection model is improved based on YOLOv10, and its structure is as follows: Figure 6 As shown, it includes: a backbone network, a neck network and a head network;

[0139] The backbone network includes: a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first downsampling module, a first joint attention module, a third C2f module, a second downsampling module, a second joint attention module, a first C2fCIB module, an SPPF module and a PSA module connected in sequence;

[0140] The neck network includes: a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third down-sampling module, a fifth Concat module, a third C2fCIB module, a fourth down-sampling module, a sixth Concat module and a fourth C2fCIB module connected in sequence;

[0141] The head network includes four Detct modules, and the four Detct modules are respectively connected to the fifth C2f module, the sixth C2f module, the third C2fCIB module and the fourth C2fCIB module;

[0142] Wherein, the PSA module is connected to the first DUpsampling module;

[0143] The first Concat module is in turn connected to the third C2f module;

[0144] The second Concat module is connected to the second C2f module;

[0145] The third Concat module is connected to the first C2f module;

[0146] The fourth Concat module is connected to the fourth C2f module;

[0147] The fifth Concat module is connected to the first C2fCIB module;

[0148] The sixth Concat module is in turn connected to the PSA module.

[0149] Due to the complex situation on the drilling floor, especially the presence of a large number of drilling tools, it is easy to feed back the height of the drilling tool collar in the background to the iron driller, resulting in inaccurate lifting height. In order to focus the detection algorithm on the collar at the wellhead and rathole location, this embodiment adds a joint attention module (SCAM module) after the downsampling module of the YOLOv10 model backbone network. The joint attention module is mainly divided into two branches, space and channel, which perform attention calculations on the input features in two dimensions, space and channel.

[0150] For the channel dimension, this embodiment proposes a channel attention unit that does not require a significant increase in parameters. In the existing attention mechanism, such as the Squeeze-and-Excitation Networks (SE), additional parameters are added, which reduces the reasoning speed of the neural network. Specifically, the main parameters of SE are concentrated in the fully connected layer of the excitation compression stage, and for the fully connected layer, if multiple attention modules are embedded in the neural network, the reasoning speed of the detection algorithm will be greatly reduced. The channel attention unit proposed in this embodiment is a lightweight design of the original SE mechanism, and the input features are added. First, global average pooling is used according to the channel direction of the input feature to compress the features in the spatial direction, and then a group convolution with a kernel size of is used to perform feature extraction operations on it, and the sampled features are converted from , and then the output is normalized by Sigmoid on the C dimension to obtain the attention weight matrix, and finally the attention weight matrix is ​​multiplied by each channel dimension of the original feature to obtain the final channel attention branch result.

[0151] For the spatial dimension, unlike the existing global context attention network, the spatial attention unit of this embodiment uses a convolution to generate spatial context weights, and then reshapes the generated weights to , and the reshaped spatial attention weights are normalized to between 0 and 1 through Softmax. Then the original input features are reshaped to . Finally, the reshaped original features are multiplied by the normalized weight matrix to obtain the output of the final spatial attention branch. The specific calculation process (the full process processing formula of the spatial attention unit) is shown in the following formula:

[0152]

[0153] Among them, W k is a linear transformation matrix, implemented with a 1×1 convolutional layer, q is the enumeration of all possible positions, N p is H×W, the number of locations in the feature map, o is the index of the query location, and x o is the first original feature map, z o The spatial attention map output by the spatial attention unit.

[0154] Finally, the output results of the joint attention module are obtained by fusion of the output results of the spatial attention unit and the channel attention unit based on the preset spatial attention weights and channel attention weights.

[0155] In addition, the location information of the target is particularly important. If there is a deviation of 1 pixel at the pixel level, the error reflected in the spatial level will reach 5 mm or even higher. In the existing target detection algorithms, multiple downsampling and upsampling operations are often required. Upsampling plays a vital role in the target positioning task in the target detection task. Although the existing upsampling methods, such as bilinear interpolation, have obvious advantages in computational efficiency, it is difficult to achieve the optimal effect due to the high requirements of target positioning in this task. This embodiment proposes an efficient upsampling module, the DUpsampling module, which is used to replace all upsampling modules in the neck network of the YOLOv10 model.

[0156] Taking double upsampling as an example, the processing process of the DUpsampling module is as follows Figure 5 As shown in the figure, for the input features, the DUpsampling module first uses a deep convolution with a kernel size of 4 times the output channel to perform dimensionality increase calculation in the channel direction, expands the output after dimensionality increase in accordance with the channel dimension, and finally concatenates the elements of the same vacant position and different channels to form the final output result. The calculation process of the DUpsampling module (the whole process processing formula of the DUpsampling module) is as follows: Out DUp =R(DConv 1×1 (X));

[0157] Among them, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1×1 depth-dimensional convolutional layer, R is an element rearrangement operation, Out DUp It is the upsampled image output by the DUpsampling module.

[0158] The baseline model used in this embodiment is YOLOv10, and then the joint attention module and DUpsampling module are introduced, and the regression box loss is optimized to design a collar detection model suitable for the existing task, such as Figure 6As shown in the figure, a joint attention module is added after the second and third stage downsampling modules of the backbone network (i.e., the first downsampling module and the second downsampling module); in the deep features of the backbone network, there is more semantic information, but because the task needs to identify the drill collars at the wellhead and rathole positions in the entire feature map, the collar features in the background are ignored. The joint attention module can reduce the impact of background interference on the network and better focus on the corresponding position. In the neck network part, the original upsampling module is replaced with the DUPsampling module, which can refine the edge features of the object and is more conducive to the regression task of drill collar detection. Finally, the original regression loss is replaced with MPDIoU to optimize the error of the anchor frame regression task and improve the positioning accuracy of the object.

[0159] This embodiment provides a performance comparison between models, where FLOPs represents the number of floating-point operations and can be used to measure the complexity of the model. mAP represents mean average precision, which reflects the average precision of all classes. As shown in Table 1, a good balance is achieved between the accuracy and FLOPs of the hoop detection model. Specifically, when the number of parameters is close, the hoop detection model is compared with the existing mainstream target detection algorithms (such as YOLOv8, YOLOv7, and YOLOv6, etc.), and the specific performance comparison is shown in Table 1:

[0160] Table 1: Model performance comparison table

[0161]

[0162] Compared with other current target detection models, the collar detection model improves mAP by 2.5% over YOLOv8-S, and by 2.9% over YOLOv6-S, with only half the number of parameters of YOLOv6-S. Although the number of parameters of the collar detection model has increased compared to YOLOv7-tiny, the mAP has increased by 3.6%, and the high-precision detection algorithm is more beneficial for judging the height of the collar. These results show that the collar detection model proposed in this embodiment is more suitable for detection tasks in the field of oil drilling and is an effective target detection algorithm.

[0163] In order to further demonstrate the effectiveness of the algorithm proposed in this paper, a visualization display is carried out based on the data collected on site and compared with the existing algorithms. The results are as follows: Figures 7 to 10 shown. Figure 7 The original image of the drilling table to be detected is collected by the industrial camera. Figures 8 to 10 They are the detection effects of YOLOv8, YOLOv10 and the collar detection model. Figure 8 As shown in the test results, there is a large gap between the male buckle and the female buckle. In actual situations ( Figure 7 There is no obvious gap in the drilling tool. Fig. 9 In the process, the gap between the male buckle and the female buckle is significantly reduced, but there is still a small amount of offset, which is still unfavorable for the high-precision judgment of the drill bit height. Fig.10 It can be seen that there is basically no gap between the detection results of the coupling detection model proposed in this embodiment, which basically meets the requirements of high-precision annotation boxes.

[0164] Example 3

[0165] This embodiment proposes a deep learning iron roughneck collar detection system, including an industrial camera, a recognition device, and an iron roughneck central control device;

[0166] The industrial camera is used to collect a first original image of the drilling table surface and send it to the recognition device;

[0167] The recognition device is used to input the received first original image into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system; and generate a control instruction according to the coordinates of the coupling in the pixel coordinate system and send it to the iron roughneck central control device;

[0168] The iron roughneck central control device is used to control the iron roughneck to make / break out the hook based on the received control instructions.

[0169] The coupling detection model is the coupling detection model proposed in Example 1 or Example 2.

[0170] A deep learning iron roughneck coupling detection system proposed in this embodiment is used to implement a deep learning iron roughneck coupling detection method in Example 1 or Example 2.

[0171] The deep learning iron roughneck coupling detection system proposed in this embodiment provides structural support for implementing the deep learning iron roughneck coupling detection method of Example 1 or Example 2, while realizing full intelligence and automation of the iron roughneck, further improving the operating efficiency of the pipe string operating system and reducing the exposure risk of operators on the drilling floor.

[0172] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0173] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0174] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0175] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0176] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for detecting iron roughneck collars based on deep learning, characterized in that: include: S10, the recognition device inputs the received first original image of the drilling floor into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system; the first original image is an image of the wellhead and / or rathole position; The collar detection model includes a backbone network, a neck network and a head network; The backbone network is used to extract image features of the original image, including a joint attention module added after two downsampling modules of the backbone network respectively; Any of the joint attention modules is used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned in the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or the rathole position collar, i.e., the output result of the joint attention module; The first original feature image is semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes features of the collar at the wellhead and / or rathole position, and features of the background; The feature fusion image is used by the subsequent backbone network, neck network and head network to obtain the coordinates of the collar in the first original image in the pixel coordinate system based on the feature fusion image; S20, the recognition device generates a control instruction according to the coordinates of the coupling in the pixel coordinate system and sends the control instruction to the iron roughneck central control device, so that the iron roughneck central control device controls the iron roughneck to perform the make / break operation.

2. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: Any of the joint attention modules includes a spatial attention unit, a channel attention unit and a feature fusion unit; The channel attention unit is used to, according to the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, perform feature compression in the spatial direction by using global average pooling according to the input feature channel direction, perform feature extraction through a 1×1 convolution layer, and then perform normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assign attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map; The channel attention weight matrix includes channel weights of the wellhead and / or rathole location collar, and channel weights of the background; The spatial attention unit is used to extract features through a 1×1 convolution layer based on the features of the collar at the wellhead and / or rathole position in the first original feature map, as well as the features of the background, to obtain the spatial context weight corresponding to the first original feature map, and then reshape the spatial context weight, and normalize the reshaped spatial context weight through a Softmax layer to obtain a spatial attention weight matrix; Finally, the attention weight is assigned to the first original feature map based on the spatial attention weight matrix to obtain the spatial attention map; The spatial attention weight matrix includes the spatial weights of the collars at the wellhead and / or rathole locations, and the spatial weights of the background; The feature fusion unit is used to perform feature fusion of the channel attention map and the spatial attention map based on the pre-set channel attention weight and spatial attention weight, so as to obtain a feature fusion image focusing on the collar at the wellhead and / or rathole position; the channel attention weight and the spatial attention weight are both obtained by pre-training the collar detection model.

3. The deep learning iron roughneck coupling detection method according to claim 2, characterized in that: The channel attention unit, according to the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, uses global average pooling to perform feature compression in the spatial direction according to the input feature channel direction, extracts features through a 1×1 convolution layer, and then performs normalization processing through a Sigmoid layer to obtain a channel attention weight matrix, and finally assigns attention weights to the first original feature map based on the channel attention weight matrix to obtain a channel attention map, including: The received first original feature map is subjected to global average pooling according to its input feature channel direction and the preset formula 1 to compress the features in the spatial direction; the formula 1 is: Among them, X c is the matrix of the input feature channel direction, i and j represent the position index of the corresponding element of the matrix of the input feature channel direction, GAP c is the output of global average pooling, H is the height of the first original feature map, and W is the width of the first original feature map; Then the first original feature map after global average pooling is input into the 1×1 convolution layer for feature extraction; X1=Conv 1×1 (GAP c ); Among them, Conv 1×1 is a 1×1 group convolution, GAP c is the output of global average pooling, X1 is the output feature after the 1×1 convolution layer; Then normalize it through the Sigmoid layer to obtain the channel attention weight matrix; Out = σ(X1); Among them, Out is the channel attention weight matrix, σ is the Sigmoid activation function, and X1 is the output feature after the 1×1 convolution layer; Finally, the channel attention weight matrix is ​​multiplied by each channel dimension of the first original feature map to obtain the channel attention map.

4. The deep learning iron roughneck coupling detection method according to claim 2, characterized in that: The spatial attention unit is used to extract features through a 1×1 convolution layer based on the features of the collar at the wellhead and / or rathole position in the first original feature map, and the features of the background, to obtain the spatial context weight corresponding to the first original feature map, and then reshape the spatial context weight, and normalize the reshaped spatial context weight through a Softmax layer to obtain a spatial attention weight matrix; Finally, the attention weight is assigned to the first original feature map based on the spatial attention weight matrix to obtain the spatial attention map; The spatial attention weight matrix includes the spatial weights of the collars at the wellhead and / or rathole locations, and the spatial weights of the background, including: Input the received C×H×W first original feature map into a 1×1 convolutional layer to generate a spatial context weight corresponding to the first original feature map; The spatial context weights are then reshaped to HW×1×1; The reshaped spatial context weights are normalized through the Softmax layer to obtain the spatial attention weight matrix; Then, after reshaping the first original feature map of C×H×W into C×HW, the reshaped first original feature is multiplied by the attention weight matrix and then concatenated with the first original feature map to obtain a spatial attention map; The whole process processing formula of the spatial attention unit is: Among them, W k is a linear transformation matrix, implemented with a 1×1 convolutional layer, q is the enumeration of all possible positions, N p is H×W, the number of locations in the feature map, o is the index of the query location, and x o is the first original feature map, z o The spatial attention map output by the spatial attention unit.

5. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: All upsampling modules in the neck network of the collar detection model are DUpsampling modules; The DUpsampling module is used to perform channel dimension upscaling on the received second original feature map through a 1×1 deep dimension upscaling convolution layer, and then after the dimension upscaling of the second original feature map is expanded according to the channel dimension, the elements with the same vacancy position and different channels are spliced ​​and combined to obtain an upsampled image of the edge features of the collar at the refined wellhead and / or rathole position; the whole process processing formula of the DUpsampling module is: Out DUp =R(DConv 1×1 (X)); Among them, X is the second original feature map received by the DUpsampling module, DConv 1×1 is a 1×1 depth-dimensional convolutional layer, R is an element rearrangement operation, Out DUp The upsampled image output by the DUpsampling module; The second original feature map is the semantic information of the first original image after image features are extracted through the backbone network; The upsampled image is used by a subsequent neck network and a head network to obtain the coordinates of the collar in the first original image in a pixel coordinate system based on the upsampled image.

6. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: The S10 and above also include: S00, the recognition module determines the mapping relationship between the image coordinate system and the world coordinate system according to the coordinates of the three pre-calibrated calibration points in the second original image sent by the industrial camera received in the image coordinate system and the coordinates of the three pre-set calibration points in the world coordinate system, and the pre-set formula 1; The formula 1 is: Where λ is the mapping coefficient between the coordinates in the image coordinate system and the coordinates in the world coordinate system; y0, y1 and y2 are the coordinates of the three calibration points in the second original image in the image coordinate system; y 00 ,y 11 and 22 are the coordinates of the three calibration points in the world coordinate system; Then the S20 includes: The recognition device generates a control instruction based on the coordinates of the coupling in the pixel coordinate system and the mapping relationship between the preset image coordinate system and the world coordinate system, and sends it to the iron roughneck central control system, so that the iron roughneck central control system controls the iron roughneck to perform the bolting / breaking operation; the control instruction includes the coordinates of the coupling in the world coordinate system.

7. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: The S10 and above also include: S01. The recognition module inputs the third original image received from the industrial camera into a preset ROI model to generate a first original image; the ROI model is used to crop the region of interest of the received second original image to obtain a first original image including only the region of interest; the region of interest is the wellhead and / or rathole location.

8. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: The S10 and above also include: S02, the recognition module collects training images at different times through a pre-set industrial camera, and uses a labeling tool to label the collars at the wellhead and / or rathole positions in all training images, as well as the coordinates of the collars at the wellhead and / or rathole positions corresponding to each training image in a pixel coordinate system, to obtain a training set; Input the training set into a pre-built collar detection model to obtain a trained collar detection model; During this period, the loss function MPDIoU is calculated to optimize the hoop detection model.

9. The deep learning iron roughneck coupling detection method according to claim 1, characterized in that: The neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform collar coordinate detection based on the fused features of the neck network; The backbone network includes: a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a first downsampling module, a first joint attention module, a third C2f module, a second downsampling module, a second joint attention module, a first C2fCIB module, an SPPF module and a PSA module connected in sequence; The neck network includes: a first DUpsampling module, a first Concat module, a second C2fCIB module, a second DUpsampling module, a second Concat module, a fourth C2f module, a third DUpsampling module, a third Concat module, a fifth C2f module, a fourth Conv module, a fourth Concat module, a sixth C2f module, a third down-sampling module, a fifth Concat module, a third C2fCIB module, a fourth down-sampling module, a sixth Concat module and a fourth C2fCIB module connected in sequence; The head network includes four Detct modules, and the four Detct modules are respectively connected to the fifth C2f module, the sixth C2f module, the third C2fCIB module and the fourth C2fCIB module; Wherein, the PSA module is connected to the first DUpsampling module; The first Concat module is in turn connected to the third C2f module; The second Concat module is connected to the second C2f module; The third Concat module is connected to the first C2f module; The fourth Concat module is connected to the fourth C2f module; The fifth Concat module is connected to the first C2fCIB module; The sixth Concat module is in turn connected to the PSA module.

10. A deep learning iron roughneck collar detection system, characterized in that: Includes industrial cameras, identification devices and iron roughneck central control devices; The industrial camera is used to collect a first original image of the drilling floor surface and send it to the recognition device; The recognition device is used to input the received first original image into the trained coupling detection model to obtain the coordinates of the coupling in the first original image in the pixel coordinate system; and generate a control instruction according to the coordinates of the coupling in the pixel coordinate system and send it to the iron roughneck central control device; The iron roughneck central control device is used to control the iron roughneck to make / break out the hook based on the received control instructions; The first original image is an image of the wellhead and / or rathole location; The collar detection model includes a backbone network, a neck network and a head network; The backbone network is used to extract image features of the original image, including a joint attention module added after two downsampling modules of the backbone network respectively; Any of the joint attention modules is used to assign attention weights to the received first original feature image along the channel dimension and the spatial dimension respectively, and to perform feature fusion on the first original feature image after the attention weights are assigned in the channel dimension and the spatial dimension, so as to obtain a feature fusion image focused on the wellhead and / or the rathole position collar, i.e., the output result of the joint attention module; The first original feature image is semantic information of the first original image after preliminary feature extraction and downsampling, and the first original feature image includes features of the collar at the wellhead and / or rathole position, and features of the background.

Images