Thermal imaging enhanced binocular iron roughneck coupling height identification system

Abstract

The invention relates to a thermal imaging enhanced binocular iron roughneck coupling height identification system, and the system comprises a visual device which is used for carrying out the shooting of a coupling operation region, and obtaining the image information of a wellhead region; the recognition system is used for judging whether a drill rod exists in the wellhead area in the coupling operation area or not based on the wellhead area image information, and a first judgment result is obtained; the central control system is used for controlling the visual device to shoot a hooping operation area according to a specified frequency through an identification system after an operation instruction triggered by a user is received under the condition that the first judgment result is that the drill rod exists in the wellhead area, so as to obtain iron roughneck image information; the recognition system is further used for acquiring first movement information of the iron roughneck corresponding to the operation instruction based on the image information of the iron roughneck; and the central control system is further used for controlling the iron roughneck to carry out moving operation according to the first moving information of the iron roughneck.

Description

Technical Field

[0001] The invention relates to the technical field of oil drilling, and in particular to a thermal imaging enhanced binocular iron roughneck collar height recognition system. Background Art

[0002] At present, the recognition of the height of the iron roughneck collar mainly relies on the monocular 2D image stream captured by the industrial camera and is recognized by traditional image processing technology. The Chinese patent application "CN202311025853.6 A method and device for drilling tool detection based on monocular vision" provides a wellhead rathole dual-position drill pipe detection method based on monocular vision without the need for an auxiliary light source. It uses a high-dynamic camera compatible with day and night scenes to obtain a clearer drill pipe image; then the algorithm is run on the computing device to combine deep learning with traditional algorithms to complete the drill pipe collar height detection and drill pipe partial diameter measurement. The technical solution of this patent application is greatly affected by the target detection image quality environment. If the target image quality is not obvious in the image (the picture is overexposed, the mud covers the seam, etc.), it will lead to a decrease in confidence or even recognition errors.

[0003] Although 2D image-based coupling recognition methods can achieve high accuracy under standard lighting conditions, their performance is greatly reduced under extreme lighting conditions, such as direct sunlight, searchlight lighting, sunset, and low-light environments. In particular, strong lighting can cause light spots and local overexposure in the image. The image is overexposed, resulting in loss of details on the edge of the coupling, which interferes with the accurate recognition of the coupling height. In addition, these traditional methods also face challenges in dealing with reflections and shadows in the image, because these phenomena change the visual characteristics of the coupling, increasing the complexity and uncertainty of the recognition algorithm. Moreover, in the field of oil drilling, the mud coverage of the drill pipe is relatively serious. Although the well team will scrape off the mud at the drill pipe coupling and joints during the process of raising and lowering the column, the traditional monocular vision will not be able to identify the type of drill tool and the accurate coupling height due to the mud coverage, which will cause the iron driller to make mistakes when operating the coupling. Summary of the invention

[0004] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a thermal imaging enhanced binocular iron roughneck coupling height recognition system.

[0005] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0006] The embodiment of the present invention provides a thermal imaging enhanced binocular iron roughneck collar height recognition system, the system comprising: an iron roughneck, a central control system, a recognition system and a visual device connected in sequence;

[0007] The visual device is used to take pictures of the coupling operation area to obtain image information of the wellhead area;

[0008] The collar operation area includes the wellhead area and the area where the iron roughneck is located;

[0009] The visual device comprises: a 2D industrial camera, a thermal imaging camera and an internal calibration card slot; the 2D industrial camera and the thermal imaging camera are respectively arranged on the internal calibration card slot;

[0010] The recognition system is used to determine whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information to obtain a first judgment result;

[0011] The central control system is used to control the visual device to shoot the butt collar operation area at a specified frequency through the recognition system after receiving the operation instruction triggered by the user when the first judgment result is that there is a drill pipe in the wellhead area, so as to obtain the iron roughneck image information;

[0012] The recognition system is further configured to obtain first movement information of the iron roughneck corresponding to the operation instruction based on the iron roughneck image information;

[0013] The first movement information is a movement distance of the drill rod pin that makes the distance between the drill rod pin and the drill rod box in the iron roughneck meet the first distance corresponding to the operation instruction;

[0014] The central control system is further used to control the iron roughneck to perform movement operations according to the first movement information of the iron roughneck.

[0015] Preferably,

[0016] The central control system is further configured to control the visual device to photograph the iron roughneck at a specified frequency again through the recognition system when the iron roughneck completes the moving operation according to the first movement information of the iron roughneck, so as to obtain new image information of the iron roughneck;

[0017] The recognition system is further used to obtain second movement information of the iron roughneck based on the new iron roughneck image information;

[0018] The second movement information is the distance between the drill pipe pin and the drill pipe box in the iron roughneck;

[0019] The central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result.

[0020] Preferably,

[0021] The wellhead area image information is a 3D point cloud file constructed based on the first image, the second image, the internal parameters of the two cameras acquired in advance, and the external parameters corresponding to the relative positions and directions of the two cameras acquired in advance through calibration;

[0022] The first image is an image taken by a 2D industrial camera in the visual device for the coupling operation area;

[0023] The second image is an image taken by the thermal imaging camera in the visual device with respect to the coupling operation area.

[0024] Preferably,

[0025] The recognition system determines whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information, and obtains a first judgment result, specifically including:

[0026] Inputting the wellhead area image information into a trained 3D point cloud model, and the trained 3D point cloud model outputs a first judgment result;

[0027] Using the first training data set to train a 3D point cloud model to obtain a trained 3D point cloud model; the 3D point cloud model is an MMdet3D model;

[0028] The first training data set includes: image information of the wellhead area corresponding to a plurality of historical moments respectively provided with a first label;

[0029] The first label is a label indicating that the image information of the wellhead area contains a drill rod or a label indicating that the image information of the wellhead area does not contain a drill rod.

[0030] Preferably,

[0031] The iron roughneck image information includes iron roughneck images captured by the visual device at designated frequencies;

[0032] Wherein, the iron roughneck image includes: a third image and a fourth image;

[0033] The third image is an image taken by a 2D industrial camera in the visual device;

[0034] The fourth image is an image taken by a thermal imaging camera in the visual device.

[0035] Preferably,

[0036] The recognition system, based on the iron roughneck image information, acquires first movement information of the iron roughneck corresponding to the operation instruction, specifically comprising:

[0037] Inputting a third image of the iron roughneck image captured for the first time in the iron roughneck image information into the trained 2D detection model, wherein the trained 2D detection model outputs a 2D detection result;

[0038] The 2D detection model is trained using the second training data set to obtain a trained 2D detection model; the 2D point cloud model is a YOLOv8 model;

[0039] The second training data set includes: third images taken by a 2D industrial camera at multiple historical moments after the target object is labeled;

[0040] The target objects include a drill pipe pin, a drill pipe box and a drill pipe in the image;

[0041] The 2D detection result includes: pixel information of the target object identified in the third image; inputting the fourth image in the first iron roughneck image captured in the iron roughneck image information into the trained infrared thermal imaging detection model, and the trained infrared thermal imaging detection model outputs the infrared detection result;

[0042] Using the third training data set to train the infrared thermal imaging detection model to obtain a trained infrared thermal imaging detection model;

[0043] The infrared thermal imaging detection model is a YOLOv8 model;

[0044] The third training data set includes: fourth images taken by the thermal imaging camera at multiple historical moments after the target object is labeled;

[0045] The infrared detection result includes: pixel information of the target object identified in the fourth image;

[0046] Based on the 2D detection result and the infrared detection result, first movement information of the iron roughneck corresponding to the operation instruction is acquired.

[0047] Preferably,

[0048] Acquiring first movement information of the iron roughneck corresponding to the operation instruction based on the 2D detection result and the infrared detection result specifically includes:

[0049] Determine whether the 2D detection result and the infrared detection result meet the preset conditions. If the 2D detection result and the infrared detection result meet the preset conditions, obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the 2D detection result according to the 2D detection result and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, and obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the infrared detection result according to the conversion relationship between the infrared detection result and the pre-acquired pixel coordinates and the actual coordinates;

[0050] The preset condition is that the 2D detection result and the infrared detection result both include pixel information of the drill rod male buckle, pixel information of the drill rod female buckle, and pixel information of the drill rod;

[0051] If the 2D detection result shows that the target object identified in the third image lacks the first object, and the target object identified in the fourth image in the infrared detection result includes the first object, the pixel information corresponding to the first object in the fourth image in the infrared detection result is used as the pixel information corresponding to the first object in the 2D detection result;

[0052] The first object is at least one of a drill rod pin, a drill rod box and a drill rod;

[0053] If the infrared detection result shows that the target object identified in the fourth image lacks the first object, and the target object identified in the third image in the 2D detection result includes the first object, the pixel information corresponding to the first object in the third image in the 2D detection result is used as the pixel information corresponding to the first object in the infrared detection result;

[0054] According to the infrared detection results, the 2D detection results and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection results and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection results are obtained respectively;

[0055] Determine whether the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, and obtain a third judgment result;

[0056] If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is less than or equal to a preset value, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection result and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection result are respectively merged to obtain the final actual height of the male buckle, the final actual height of the female buckle and the final actual width of the drill rod;

[0057] According to the final actual height of the male buckle, the final actual height of the female buckle and the first distance corresponding to the operation instruction acquired in advance, the moving distance of the drill rod male buckle is determined, and the moving distance of the drill rod male buckle is used as the first movement information.

[0058] Preferably,

[0059] If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, then obtaining an average actual height value and an average pixel height value, and determining a new conversion relationship between pixel coordinates and actual coordinates based on the average actual height value and the average pixel height value;

[0060] The new conversion relationship between pixel coordinates and actual coordinates = actual height average / pixel height average;

[0061] The actual height average value is: the average value of the actual height of the female buckle in the 2D detection result and the actual height of the female buckle in the infrared detection result respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information;

[0062] The average pixel height is: the average value of the female buckle pixel information in the 2D detection results and the female buckle pixel information in the infrared detection results respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information.

[0063] Preferably,

[0064] The actual height of the pin buckle, the actual height of the box buckle, and the actual width of the drill rod corresponding to the 2D detection results are fused with the actual height of the pin buckle, the actual height of the box buckle, and the actual width of the drill rod corresponding to the infrared detection results to obtain the final actual height of the pin buckle, the final actual height of the box buckle, and the final actual width of the drill rod, including:

[0065] Formula (1) is used to fuse the actual height of the male buckle corresponding to the 2D detection result and the actual height of the male buckle corresponding to the infrared detection result to obtain the final actual height of the male buckle;

[0066] The formula (1) is:

[0067] The final actual height of the male buckle = the height of the male buckle × W (L) + the height of the male buckle (1-W (L));

[0068] Formula (2) is used to fuse the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result to obtain the final actual height of the female buckle;

[0069] The formula (2) is:

[0070] The final actual height of the female buckle = the height of the female buckle × W (L) + the height of the female buckle (1-W (L));

[0071] Formula (3) is used to fuse the actual width of the drill rod corresponding to the 2D detection result and the actual width of the drill rod corresponding to the infrared detection result to obtain the final actual width of the drill rod;

[0072] The formula (3) is:

[0073] The final actual width of the drill pipe = actual width of the drill pipe × W (L) + actual width of the drill pipe (1-W (L));

[0074] in,

[0075] L is the brightness parameter of the 2D industrial camera;

[0076] L0 is the characteristic brightness of the wellhead area collected by the 2D industrial camera;

[0077] k is a preset coefficient.

[0078] Preferably,

[0079] The central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result, specifically including:

[0080] If the second movement information is equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified;

[0081] If the second movement information is not equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is unqualified.

[0082] The beneficial effect of the present invention is that a thermal imaging enhanced binocular iron roughneck coupling height recognition system of the present invention, because the visual device includes a thermal imaging camera and a 2D industrial camera, can overcome the problems of light spots, local overexposure, shadows and mud coverage through the thermal imaging camera, thereby improving the accuracy of recognition.

[0083] In addition, thermal imaging cameras can adapt to various working environments from strong light to low light, maintain high-accuracy recognition performance, and enhance environmental adaptability.

[0084] The thermal imaging enhanced binocular iron roughneck coupling height recognition system of the present invention uses a binocular system composed of a 2D industrial camera and a thermal imaging camera, utilizes image information captured by the 2D industrial camera and the thermal imaging camera, and integrates information under different spectra through data fusion technology, so that the system can be more accurate and stable when processing data in complex environments.

[0085] The thermal imaging enhanced binocular iron roughneck coupling height recognition system of the present invention uses a binocular system composed of a 2D industrial camera and a thermal imaging camera to obtain spatial information through stereo vision technology to accurately calculate the three-dimensional position and height of the coupling. By capturing a target object of known size and identifying the target object, the iron roughneck is made safer and more stable to use, and the occurrence of failures in making and breaking out the coupling is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a schematic structural diagram of a thermal imaging enhanced binocular iron roughneck coupling height recognition system of the present invention;

[0087] Figure 2 It is a schematic diagram of the structure of a visual device in a thermal imaging enhanced binocular iron roughneck coupling height recognition system of the present invention;

[0088] Figure 3 It is a schematic diagram of the process of making and breaking out the iron roughneck in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0090] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0091] See also Figure 1 ,This embodiment provides a thermal imaging enhanced binocular iron roughneck collar height recognition system, the system comprising: an iron roughneck, a central control system, a recognition system and a visual device connected in sequence;

[0092] The visual device is used to take pictures of the coupling operation area to obtain image information of the wellhead area;

[0093] The collar operation area includes the wellhead area and the area where the iron roughneck is located;

[0094] See also Figure 2 , the visual device described in this embodiment includes: a 2D industrial camera, a thermal imaging camera and an internal calibration card slot; the 2D industrial camera and the thermal imaging camera are respectively arranged on the internal calibration card slot;

[0095] The recognition system is used to determine whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information to obtain a first judgment result;

[0096] The central control system is used to control the visual device to shoot the butt collar operation area at a specified frequency through the recognition system after receiving the operation instruction triggered by the user when the first judgment result is that there is a drill pipe in the wellhead area, so as to obtain the iron roughneck image information;

[0097] In this embodiment, the operation instruction triggered by the user is an operation instruction for buckling the drill rod pin buckle and the drill rod box buckle or an operation instruction for buckling the drill rod pin buckle and the drill rod box buckle. Assuming that the operation instruction triggered by the current user is an operation instruction for buckling the drill rod pin buckle and the drill rod box buckle, then at this time, the drill rod pin buckle and the drill rod box buckle are separated. After the drill rod pin buckle and the drill rod box buckle are buckled, the distance between the drill rod pin buckle and the drill rod box buckle should be 0, that is, the drill rod pin buckle and the drill rod box buckle are in contact after being buckled. Therefore, the first distance corresponding to the operation instruction for buckling the drill rod pin buckle and the drill rod box buckle is 0. Assume that the operation instruction triggered by the current user is the operation instruction of detaching the drill pipe pin and the drill pipe box. At this time, the drill pipe pin and the drill pipe box are in contact. After the drill pipe pin and the drill pipe box are detached, the distance between the drill pipe pin and the drill pipe box should be h (cm). That is to say, the distance between the drill pipe pin and the drill pipe box after the detachment is h. Therefore, the first distance corresponding to the operation instruction of detaching the drill pipe pin and the drill pipe box is h.

[0098] The recognition system is further configured to obtain first movement information of the iron roughneck corresponding to the operation instruction based on the iron roughneck image information;

[0099] The first movement information is a movement distance of the drill rod pin that makes the distance between the drill rod pin and the drill rod box in the iron roughneck meet the first distance corresponding to the operation instruction;

[0100] The central control system is further used to control the iron roughneck to perform movement operations according to the first movement information of the iron roughneck.

[0101] In the practical application of this embodiment, the central control system is further used to control the visual device to shoot the iron roughneck at a specified frequency again through the recognition system when the iron roughneck completes the moving operation according to the first movement information of the iron roughneck, so as to obtain new image information of the iron roughneck;

[0102] The recognition system is further used to obtain second movement information of the iron roughneck based on the new iron roughneck image information;

[0103] The second movement information is the distance between the drill pipe pin and the drill pipe box in the iron roughneck;

[0104] The central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result.

[0105] For example, when the iron driller completes the moving operation according to the first movement information of the iron driller, the recognition system controls the visual device to shoot the iron driller at a specified frequency again to obtain new image information of the iron driller; the recognition system obtains the distance between the drill rod pin and the drill rod box in the iron driller based on the new image information of the iron driller; the main purpose is to check whether the drill rod pin and the drill rod box meet the first distance corresponding to the operation instruction after the operation instruction is executed. Specifically, after the execution of the make-up operation instruction is completed, the second movement information should be 0, and after the execution of the make-up operation instruction is completed, the second movement information should be h (cm), so that it is qualified.

[0106] This embodiment provides a thermal imaging enhanced binocular iron roughneck coupling height recognition system, which obtains image information of the coupling operation area through a 2D industrial camera and a thermal imaging camera, and can accurately determine whether there is a drill pipe in the wellhead area, and perform operations based on this information. This precise monitoring can reduce manual intervention and improve the automation and efficiency of operations.

[0107] This embodiment provides a thermal imaging enhanced binocular iron roughneck coupling height recognition system, which can obtain iron roughneck image information by shooting at a specified frequency after the user triggers an operation instruction, and calculate the actual distance between the drill pipe male buckle and the drill pipe female buckle by comparing the operation instruction. In this way, the system can provide real-time feedback and adjust the movement operation of the iron roughneck to ensure the accuracy and correctness of the operation.

[0108] The present embodiment provides a thermal imaging enhanced binocular iron roughneck coupling height recognition system, which monitors and verifies the distance between the drill pipe male buckle and the drill pipe female buckle by the system, and can effectively prevent operation errors or equipment damage caused by improper operation, thereby improving the safety and overall quality of the operation.

[0109] This embodiment provides a thermal imaging enhanced binocular iron roughneck collar height recognition system. After the iron roughneck moves, the system takes pictures again to obtain new image information and compares it with the standard distance corresponding to the operation instruction. This process can automatically check whether the operation result is qualified, reduce the workload of manual inspection, and improve the accuracy of inspection.

[0110] Specifically, the wellhead area image information is a 3D point cloud file constructed based on the first image, the second image, the internal parameters of the two cameras acquired in advance, and the external parameters corresponding to the relative positions and directions of the two cameras acquired in advance through calibration;

[0111] The first image is an image taken by a 2D industrial camera in the visual device for the coupling operation area;

[0112] The second image is an image taken by the thermal imaging camera in the visual device with respect to the coupling operation area.

[0113] Preferably, the recognition system determines whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information, and obtains a first judgment result, which specifically includes:

[0114] Inputting the wellhead area image information into a trained 3D point cloud model, and the trained 3D point cloud model outputs a first judgment result;

[0115] Using the first training data set to train a 3D point cloud model to obtain a trained 3D point cloud model; the 3D point cloud model is an MMdet3D model;

[0116] The first training data set includes: image information of the wellhead area corresponding to a plurality of historical moments respectively provided with a first label;

[0117] The first label is a label indicating that the image information of the wellhead area contains a drill rod or a label indicating that the image information of the wellhead area does not contain a drill rod.

[0118] The iron roughneck image information includes iron roughneck images captured by the visual device at designated frequencies;

[0119] Wherein, the iron roughneck image includes: a third image and a fourth image;

[0120] The third image is an image taken by a 2D industrial camera in the visual device;

[0121] The fourth image is an image taken by a thermal imaging camera in the visual device.

[0122] The recognition system, based on the iron roughneck image information, acquires first movement information of the iron roughneck corresponding to the operation instruction, specifically comprising:

[0123] Inputting a third image of the iron roughneck image captured for the first time in the iron roughneck image information into the trained 2D detection model, wherein the trained 2D detection model outputs a 2D detection result;

[0124] The 2D detection model is trained using the second training data set to obtain a trained 2D detection model; the 2D point cloud model is a YOLOv8 model;

[0125] The second training data set includes: third images taken by a 2D industrial camera at multiple historical moments after the target object is labeled;

[0126] See also Figure 3 , the target objects include the drill pipe pin, drill pipe box and drill pipe in the image;

[0127] The 2D detection result includes: pixel information of the target object identified in the third image; inputting the fourth image in the first iron roughneck image captured in the iron roughneck image information into the trained infrared thermal imaging detection model, and the trained infrared thermal imaging detection model outputs the infrared detection result;

[0128] Using the third training data set to train the infrared thermal imaging detection model to obtain a trained infrared thermal imaging detection model;

[0129] The infrared thermal imaging detection model is a YOLOv8 model;

[0130] The third training data set includes: fourth images taken by the thermal imaging camera at multiple historical moments after the target object is labeled;

[0131] The infrared detection result includes: pixel information of the target object identified in the fourth image;

[0132] Based on the 2D detection result and the infrared detection result, first movement information of the iron roughneck corresponding to the operation instruction is acquired.

[0133] In this embodiment, based on the 2D detection result and the infrared detection result, obtaining the first movement information of the iron roughneck corresponding to the operation instruction specifically includes:

[0134] Determine whether the 2D detection result and the infrared detection result meet the preset conditions. If the 2D detection result and the infrared detection result meet the preset conditions, obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the 2D detection result according to the 2D detection result and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, and obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the infrared detection result according to the conversion relationship between the infrared detection result and the pre-acquired pixel coordinates and the actual coordinates;

[0135] The preset condition is that the 2D detection result and the infrared detection result both include pixel information of the drill rod male buckle, pixel information of the drill rod female buckle, and pixel information of the drill rod;

[0136] The pixel information in this embodiment includes pixel coordinates.

[0137] In this embodiment, the conversion relationship between pixel coordinates and actual coordinates is obtained in advance by calibration based on three clear pixel points in the camera field of view. The pixel information of the three clear pixel points is known and the actual heights corresponding to the pixel points are also known.

[0138] For example, a binocular camera system is used to measure the height of an iron drill. First, three known landmarks in the image are selected, such as three different marker bars (such as the scale on a height ruler). In the image, the ordinates (pixel heights) of these marker bars are recorded, and at the same time, their actual heights in the real world are measured and recorded. Next, through the geometric principle of similar triangles, the pixel widths of these marker bars are calculated and converted to actual widths. Then, this information and the known height scale factor are used to estimate the actual height of the iron drill. If some parts of the iron drill are beyond the range of these three marker bars, the system will apply a linear interpolation algorithm to infer the actual height of the out-of-range part based on the data of the marker bars, thereby ensuring accurate measurement under various conditions.

[0139] If the 2D detection result shows that the target object identified in the third image lacks the first object, and the target object identified in the fourth image in the infrared detection result includes the first object, the pixel information corresponding to the first object in the fourth image in the infrared detection result is used as the pixel information corresponding to the first object in the 2D detection result;

[0140] The first object is at least one of a drill rod pin, a drill rod box and a drill rod;

[0141] If the infrared detection result shows that the target object identified in the fourth image lacks the first object, and the target object identified in the third image in the 2D detection result includes the first object, the pixel information corresponding to the first object in the third image in the 2D detection result is used as the pixel information corresponding to the first object in the infrared detection result;

[0142] According to the infrared detection results, the 2D detection results and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection results and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection results are obtained respectively;

[0143] Determine whether the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, and obtain a third judgment result;

[0144] In this embodiment, the preset value is 15 cm.

[0145] If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is less than or equal to a preset value, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection result and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection result are respectively merged to obtain the final actual height of the male buckle, the final actual height of the female buckle and the final actual width of the drill rod;

[0146] According to the final actual height of the male buckle, the final actual height of the female buckle and the first distance corresponding to the operation instruction acquired in advance, the moving distance of the drill rod male buckle is determined, and the moving distance of the drill rod male buckle is used as the first movement information.

[0147] If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, then obtaining an average actual height value and an average pixel height value, and determining a new conversion relationship between pixel coordinates and actual coordinates based on the average actual height value and the average pixel height value;

[0148] The new conversion relationship between pixel coordinates and actual coordinates = actual height average / pixel height average;

[0149] The actual height average value is: the average value of the actual height of the female buckle in the 2D detection result and the actual height of the female buckle in the infrared detection result respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information;

[0150] The average pixel height is: the average value of the female buckle pixel information in the 2D detection results and the female buckle pixel information in the infrared detection results respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information.

[0151] In this embodiment, the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the 2D detection result and the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the infrared detection result are respectively fused to obtain the final actual height of the male buckle, the final actual height of the female buckle, and the final actual width of the drill rod, which specifically includes:

[0152] Formula (1) is used to fuse the actual height of the male buckle corresponding to the 2D detection result and the actual height of the male buckle corresponding to the infrared detection result to obtain the final actual height of the male buckle;

[0153] The formula (1) is:

[0154] The final actual height of the male buckle = the height of the male buckle × W (L) + the height of the male buckle (1-W (L));

[0155] Formula (2) is used to fuse the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result to obtain the final actual height of the female buckle;

[0156] The formula (2) is:

[0157] The final actual height of the female buckle = the height of the female buckle × W (L) + the height of the female buckle (1-W (L));

[0158] Formula (3) is used to fuse the actual width of the drill rod corresponding to the 2D detection result and the actual width of the drill rod corresponding to the infrared detection result to obtain the final actual width of the drill rod;

[0159] The formula (3) is:

[0160] The final actual width of the drill pipe = actual width of the drill pipe × W (L) + actual width of the drill pipe (1-W (L));

[0161] in,

[0162] L is the brightness parameter of the 2D industrial camera;

[0163] L0 is the characteristic brightness of the wellhead area collected by the 2D industrial camera;

[0164] k is a preset coefficient.

[0165] In this embodiment, the central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result, specifically including:

[0166] If the second movement information is equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified;

[0167] If the second movement information is not equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is unqualified.

[0168] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0169] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0170] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0171] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0172] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A thermal imaging enhanced binocular iron roughneck coupling height recognition system, characterized in that: The system comprises: an iron roughneck, a central control system, an identification system and a visual device connected in sequence; The visual device is used to take pictures of the coupling operation area to obtain image information of the wellhead area; The collar operation area includes the wellhead area and the area where the iron roughneck is located; The visual device comprises: a 2D industrial camera, a thermal imaging camera and an internal calibration card slot; the 2D industrial camera and the thermal imaging camera are respectively arranged on the internal calibration card slot; The recognition system is used to determine whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information to obtain a first judgment result; The central control system is used to control the visual device to shoot the butt collar operation area at a specified frequency through the recognition system after receiving the operation instruction triggered by the user when the first judgment result is that there is a drill pipe in the wellhead area, so as to obtain the iron roughneck image information; The recognition system is further configured to obtain first movement information of the iron roughneck corresponding to the operation instruction based on the iron roughneck image information; The first movement information is a movement distance of the drill rod pin that makes the distance between the drill rod pin and the drill rod box in the iron roughneck meet the first distance corresponding to the operation instruction; The central control system is further used to control the iron roughneck to perform movement operations according to the first movement information of the iron roughneck.

2. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 1, characterized in that: The central control system is further configured to control the visual device to photograph the iron roughneck at a specified frequency again through the recognition system when the iron roughneck completes the moving operation according to the first movement information of the iron roughneck, so as to obtain new image information of the iron roughneck; The recognition system is further used to obtain second movement information of the iron roughneck based on the new iron roughneck image information; The second movement information is the distance between the drill pipe pin and the drill pipe box in the iron roughneck; The central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result.

3. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 2, characterized in that: The wellhead area image information is a 3D point cloud file constructed based on the first image, the second image, the internal parameters of the two cameras acquired in advance, and the external parameters corresponding to the relative positions and directions of the two cameras acquired in advance through calibration; The first image is an image taken by a 2D industrial camera in the visual device for the coupling operation area; The second image is an image taken by the thermal imaging camera in the visual device with respect to the coupling operation area.

4. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 3, characterized in that: The recognition system determines whether there is a drill pipe in the wellhead area in the coupling operation area based on the wellhead area image information, and obtains a first judgment result, specifically including: Inputting the wellhead area image information into a trained 3D point cloud model, and the trained 3D point cloud model outputs a first judgment result; Using the first training data set to train a 3D point cloud model to obtain a trained 3D point cloud model; the 3D point cloud model is an MMdet3D model; The first training data set includes: image information of the wellhead area corresponding to a plurality of historical moments respectively provided with a first label; The first label is a label indicating that the image information of the wellhead area contains a drill rod or a label indicating that the image information of the wellhead area does not contain a drill rod.

5. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 4, characterized in that: The iron roughneck image information includes iron roughneck images captured by the visual device at designated frequencies; Wherein, the iron roughneck image includes: a third image and a fourth image; The third image is an image taken by a 2D industrial camera in the visual device; The fourth image is an image taken by a thermal imaging camera in the visual device.

6. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 5, characterized in that: The recognition system, based on the iron roughneck image information, acquires first movement information of the iron roughneck corresponding to the operation instruction, specifically comprising: Inputting a third image of the iron roughneck image captured for the first time in the iron roughneck image information into the trained 2D detection model, wherein the trained 2D detection model outputs a 2D detection result; The 2D detection model is trained using the second training data set to obtain a trained 2D detection model; the 2D point cloud model is a YOLOv8 model; The second training data set includes: third images taken by a 2D industrial camera at multiple historical moments after the target object is labeled; The target objects include a drill pipe pin, a drill pipe box and a drill pipe in the image; The 2D detection result includes: pixel information of the target object identified in the third image; inputting the fourth image in the first iron roughneck image captured in the iron roughneck image information into the trained infrared thermal imaging detection model, and the trained infrared thermal imaging detection model outputs the infrared detection result; Using the third training data set to train the infrared thermal imaging detection model to obtain a trained infrared thermal imaging detection model; The infrared thermal imaging detection model is a YOLOv8 model; The third training data set includes: fourth images taken by the thermal imaging camera at multiple historical moments after the target object is labeled; The infrared detection result includes: pixel information of the target object identified in the fourth image; Based on the 2D detection result and the infrared detection result, first movement information of the iron roughneck corresponding to the operation instruction is acquired.

7. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 6, characterized in that: Acquiring first movement information of the iron roughneck corresponding to the operation instruction based on the 2D detection result and the infrared detection result specifically includes: Determine whether the 2D detection result and the infrared detection result meet the preset conditions. If the 2D detection result and the infrared detection result meet the preset conditions, obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the 2D detection result according to the 2D detection result and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, and obtain the actual height of the male buckle, the actual height of the female buckle, and the actual width of the drill rod corresponding to the infrared detection result according to the conversion relationship between the infrared detection result and the pre-acquired pixel coordinates and the actual coordinates; The preset condition is that the 2D detection result and the infrared detection result both include pixel information of the drill rod male buckle, pixel information of the drill rod female buckle, and pixel information of the drill rod; If the 2D detection result shows that the target object identified in the third image lacks the first object, and the target object identified in the fourth image in the infrared detection result includes the first object, the pixel information corresponding to the first object in the fourth image in the infrared detection result is used as the pixel information corresponding to the first object in the 2D detection result; The first object is at least one of a drill rod pin, a drill rod box and a drill rod; If the infrared detection result shows that the target object identified in the fourth image lacks the first object, and the target object identified in the third image in the 2D detection result includes the first object, the pixel information corresponding to the first object in the third image in the 2D detection result is used as the pixel information corresponding to the first object in the infrared detection result; According to the infrared detection results, the 2D detection results and the conversion relationship between the pre-acquired pixel coordinates and the actual coordinates, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection results and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection results are obtained respectively; Determine whether the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, and obtain a third judgment result; If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is less than or equal to a preset value, the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the 2D detection result and the actual height of the male buckle, the actual height of the female buckle and the actual width of the drill rod corresponding to the infrared detection result are respectively merged to obtain the final actual height of the male buckle, the final actual height of the female buckle and the final actual width of the drill rod; According to the final actual height of the male buckle, the final actual height of the female buckle and the first distance corresponding to the operation instruction acquired in advance, the moving distance of the drill rod male buckle is determined, and the moving distance of the drill rod male buckle is used as the first movement information.

8. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 7, characterized in that: If the third judgment result is that the difference between the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result is greater than a preset value, then obtaining an average actual height value and an average pixel height value, and determining a new conversion relationship between pixel coordinates and actual coordinates based on the average actual height value and the average pixel height value; The new conversion relationship between pixel coordinates and actual coordinates = actual height average / pixel height average; The actual height average value is: the average value of the actual height of the female buckle in the 2D detection result and the actual height of the female buckle in the infrared detection result respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information; The average pixel height is: the average value of the female buckle pixel information in the 2D detection results and the female buckle pixel information in the infrared detection results respectively corresponding to the iron roughneck images at the first three adjacent shooting moments in the iron roughneck image information.

9. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 8, characterized in that: The actual height of the pin buckle, the actual height of the box buckle, and the actual width of the drill rod corresponding to the 2D detection results are fused with the actual height of the pin buckle, the actual height of the box buckle, and the actual width of the drill rod corresponding to the infrared detection results to obtain the final actual height of the pin buckle, the final actual height of the box buckle, and the final actual width of the drill rod, including: Formula (1) is used to fuse the actual height of the male buckle corresponding to the 2D detection result and the actual height of the male buckle corresponding to the infrared detection result to obtain the final actual height of the male buckle; The formula (1) is: The final actual height of the male buckle = the height of the male buckle × W (L) + the height of the male buckle (1-W (L)); Formula (2) is used to fuse the actual height of the female buckle corresponding to the 2D detection result and the actual height of the female buckle corresponding to the infrared detection result to obtain the final actual height of the female buckle; The formula (2) is: The final actual height of the female buckle = the height of the female buckle × W (L) + the height of the female buckle (1-W (L)); Formula (3) is used to fuse the actual width of the drill rod corresponding to the 2D detection result and the actual width of the drill rod corresponding to the infrared detection result to obtain the final actual width of the drill rod; The formula (3) is: The final actual width of the drill pipe = actual width of the drill pipe × W (L) + actual width of the drill pipe (1-W (L)); in, L is the brightness parameter of the 2D industrial camera; L0 is the characteristic brightness of the wellhead area collected by the 2D industrial camera; k is a preset coefficient.

10. The thermal imaging enhanced binocular iron roughneck coupling height recognition system according to claim 9, characterized in that: The central control system is further used to determine whether the moving operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified based on the first distance and the second movement information corresponding to the operation instruction, and obtain a second judgment result, specifically including: If the second movement information is equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is qualified; If the second movement information is not equal to the first distance corresponding to the operation instruction, it is determined that the movement operation performed by the iron roughneck according to the first movement information of the iron roughneck is unqualified.

Images