3D vision guidance system for oilfield robotic arm grasping based on deep learning

Abstract

The present invention discloses a 3D visual guidance system for oilfield manipulator grasping based on deep learning, comprising: an oilfield manipulator equipped with a motion control system and a data acquisition module, a data processing module, an initialization calibration module, a target detection module and a grasping detection module that are communicatively connected thereto. During oilfield operations, when the system is initially operated or moved, the initialization calibration module obtains parameter initialization. The data processing module calculates the acquired image in real time, and after ensuring that the drill pipe is stationary, the image and point cloud data are transmitted to the target detection module to determine the position of the object, and then transmitted to the grasping detection module as the initial posture estimation result, and finally the result is transmitted to the manipulator motion control system for grasping operation. In summary, the present invention realizes efficient, intelligent and accurate identification and grasping of drill pipes, provides new possibilities for the application of oilfield manipulators in complex tasks, and has a profound impact on oilfield automation and intelligent production.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a 3D vision guidance system for oilfield manipulator grasping based on deep learning. Background Art

[0002] Today's oilfield production operations place increasing demands on the precision, efficiency, and intelligence of robotic arm grasping operations. Traditional oilfield robotic arm grasping methods typically rely on preset programs and fixed parameters, often struggling with the complex and ever-changing oilfield working environment and diverse equipment and workpiece characteristics. In actual oilfield operations, due to uncertainties in the shape, size, position, and posture of drill pipes, robotic arms may be unable to accurately grasp the target drill pipe, resulting in low production efficiency, unstable operation quality, and even operational failure.

[0003] The continuous development of 3D vision technology is enabling oilfield robotic arms to provide rich spatial information, creating new opportunities for solving robotic gripping challenges. However, effectively integrating 3D vision technology with oilfield robotic gripping systems is challenging, requiring solutions to a range of technical challenges, including data acquisition, processing, and transmission, as well as coordination with the robotic arm's control system.

[0004] Furthermore, with the trend toward automation and intelligent production in oilfields, increasing the degree of automation and intelligence on production lines is crucial. As a key actuator, improving the gripping performance of robotic arms is crucial for optimizing and upgrading the entire oilfield production process.

[0005] In summary, in order to meet the higher requirements of the oil field for robotic arm grasping operations and achieve more efficient, intelligent and precise grasping, the development of an oil field robotic arm grasping system based on 3D vision guidance has become a hot topic and urgent need in current technical research. Summary of the Invention

[0006] This invention aims to achieve intelligent, efficient, and precise grasping of drill pipe by oilfield manipulators, meeting the diverse needs of oilfield automation and intelligent production. By incorporating 3D vision-guided technology, the system can perceive and identify the drill pipe's state in real time within the oilfield environment and automatically plan a grasping path for grasping, significantly improving the efficiency and accuracy of the oilfield manipulator's grasping capabilities.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The system is equipped with an oilfield robotic arm with a motion control system and a data acquisition module, a data processing module, an initialization and calibration module, a target detection module and a grasping detection module connected to the arm for communication.

[0009] (1) When the system device is first operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0010] (2) The data acquisition module collects data and sends it to the data processing module after preprocessing;

[0011] (3) The data processing module performs real-time calculations on the images transmitted by the data acquisition module, ensuring that the drill rod to be captured is in a stationary state before transmitting the image data and point cloud data collected by the data acquisition module to the target detection module;

[0012] (4) The object detection module determines the position, orientation, and approximate range of the object of interest and passes it as the initial pose estimation result and constraints to the grasp detection module;

[0013] (5) The grasping detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield manipulator;

[0014] (6) The oilfield robotic arm equipped with a motion control system controls its grasping operation according to the transmitted information.

[0015] A further improvement of the present invention is that the specific implementation steps of step (1) are:

[0016] (101) The initialization module contains two parts, namely camera calibration and hand-eye calibration. When the system is first run or moves, the initialization calibration module is run to obtain the necessary parameters for initialization, such as the internal and external parameters of the camera, the conversion relationship between the posture (position) of the oilfield robot arm end effector and the camera coordinate system, etc.

[0017] A further improvement of the present invention is that the specific implementation steps of step (2) are:

[0018] (201) Using a high-definition 3D vision camera to capture depth images and color images of the scene in real time;

[0019] (202) The depth image is converted into point cloud data through depth image processing technology, where each point contains the position information and corresponding color information in three-dimensional space;

[0020] (203) Preprocessing the collected color images to remove noise and unnecessary background information and improve image quality;

[0021] (204) The processed image and point cloud data are packaged and transmitted to the data processing module in real time for subsequent processing.

[0022] A further improvement of the present invention is that the specific implementation steps of step (3) are:

[0023] (301) The data acquisition module uses an RGB camera and a lidar or RGBD camera;

[0024] (302) The data processing module uses a background difference method on the input multiple continuous color images to determine whether the drill rod to be grabbed remains stationary.

[0025] A further improvement of the present invention is that the specific implementation steps of step (4) include:

[0026] (401) The target detection module performs 3D target detection on the input multimodal data;

[0027] (402) 3D object detection mainly includes heterogeneous feature alignment and multimodal data fusion problems;

[0028] (403) Perform image feature extraction and voxel feature extraction on RGB data and point cloud data respectively;

[0029] (404) Using 3D-2D coordinate projection, the voxel features are mapped to the corresponding image features to obtain the corresponding image feature index. The specific formula of 3D-2D coordinate projection is as follows:

[0030] (405) ;

[0031] (406) RC is the combination of the camera correction rotation matrix and the calibration matrix, T cam-lidar is the camera projection matrix, V i is the voxel feature center, R i is the corresponding image feature index;

[0032] (407) Find the corresponding index pixel feature, obtain the adjacent pixel features by index translation, use Manhattan distance in pixel query, and sample K pixels within the distance threshold;

[0033] (408) These K pixels are combined with a cross-attention mechanism to dynamically capture the correlation between the two modalities and achieve subsequent fusion.

[0034] (409) The fused features are input into 3D Backbone for further processing and analysis, and then prediction is performed.

[0035] A further improvement of the present invention is that the specific implementation steps of step (5) are:

[0036] (501) The grasping detection module uses the 3D target detection results as the initial posture and constraint conditions of the drill rod and performs posture refinement on the fused features.

[0037] (502) Using the ICP algorithm, the point cloud data is matched with the preset object model, and the object pose hypothesis is continuously adjusted through iterative optimization;

[0038] (503) When the matching error reaches a preset threshold or the number of iterations reaches a maximum value, the iteration is stopped and the final drill rod posture estimation result is output;

[0039] (504) The final drill rod posture estimation results such as the drill rod spatial coordinates (x, y, z), rotation angle (θ x ,θ y , θ z ), gripping width w, confidence score c, etc. are transmitted to the motion control system of the oilfield robot arm to guide the oilfield robot arm to perform precise grasping operations.

[0040] The present invention has at least the following beneficial technical effects:

[0041] First, this invention provides a completely new solution that effectively addresses the problems and limitations of traditional technologies. Through innovative design concepts and advanced technical means, this invention not only optimizes operational processes and improves work efficiency, but also significantly reduces energy consumption and reduces environmental impact.

[0042] Secondly, the present invention is simple to understand, easy to operate, and maintain. Its modular design clarifies the functions of each component, making it easy for users to quickly master and use it. Furthermore, the present invention is highly scalable and customizable, allowing for flexible adjustments based on user needs to meet the demands of diverse scenarios.

[0043] Furthermore, the present invention boasts a high degree of security and stability. Through rigorous quality control and safety testing, it is ensured to operate stably in a variety of complex environments and effectively protect user data security. This security and stability are crucial for both businesses and individual users, as it can prevent significant losses caused by system crashes or data leaks.

[0044] Finally, the present invention has broad application prospects. With the continuous development of science and technology and the continuous progress of society, people's demand for efficient, convenient, and environmentally friendly solutions is growing. The present invention was born out of this demand. It not only meets the needs of the current market, but also has huge potential and room for development, and is expected to be widely applied and promoted in more fields.

[0045] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a flow chart of the data processing module in the deep learning-based 3D vision guidance system for oilfield robotic arm grasping described in the present invention.

[0048] Figure 2 This is the overall structural diagram of the deep learning-based 3D vision guidance system for oilfield robotic arm grasping described in the present invention. DETAILED DESCRIPTION

[0049] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0050] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0051] like Figure 2 As shown, the present invention provides a 3D visual guidance system for oilfield manipulator grasping based on deep learning, comprising the following steps:

[0052] (1) When the system device is first operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0053] (2) The data acquisition module collects data and sends it to the data processing module after preprocessing;

[0054] (3) The data processing module performs real-time calculations on the images transmitted by the data acquisition module, ensuring that the drill rod to be captured is in a stationary state before transmitting the image data and point cloud data collected by the data acquisition module to the target detection module;

[0055] (4) The object detection module determines the position, orientation, and approximate range of the object of interest and passes it as the initial pose estimation result and constraints to the grasp detection module;

[0056] (5) The grasping detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield manipulator;

[0057] (6) The oilfield robotic arm equipped with a motion control system controls the robotic arm to perform grasping operations based on the transmitted information.

[0058] Example 1

[0059] like Figure 1 As shown, Example 1 of the present invention provides a flowchart of a data processing module in a 3D visual guidance system for oilfield manipulator grasping based on deep learning. The specific operation steps are as follows:

[0060] (1) First, perform background subtraction based on a mixed Gaussian model.

[0061] (2) Grayscale, binarize, and erode the image to obtain the foreground contour and draw the circumscribed rectangle of the foreground contour;

[0062] (3) Determine the difference in the center position of the rectangle. If it is less than the set threshold, the target is considered to be stationary.

[0063] (4) Send the RGB image and corresponding point cloud data at this time to the target detection module.

[0064] Example 2

[0065] like Figure 2 As shown, embodiment 2 of the present invention provides an overall structural diagram of the oilfield manipulator grasping 3D vision guidance system based on deep learning, including:

[0066] Initialization and calibration module: When the system device is first operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0067] Data acquisition module, collecting RGB image data and point cloud data;

[0068] The data processing module calculates the images transmitted by the data acquisition module in real time to ensure that the drill rod to be grabbed is

[0069] After the stationary state, the image data and point cloud data collected by the data acquisition module are transmitted to the target detection module;

[0070] The object detection module determines the position, orientation, and approximate range of the object of interest and passes this as the initial pose estimation result and constraints to the grasp detection module;

[0071] The grab detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield robot arm;

[0072] The motion control system controls the oilfield robotic arm to perform grasping operations based on the transmitted information.

[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A 3D visual guidance system for oilfield manipulator grasping based on deep learning, characterized by: The following steps are involved: a. When the system device is initially operated or the system device is moved, run the initialization and calibration module to obtain and initialize the required parameters; b. The data processing module calculates the image transmitted by the data acquisition module in real time, ensures that the drill rod to be grabbed is in a stationary state, and transmits the image data and point cloud data collected by the data acquisition module to the target detection module; c. The object detection module determines the position, orientation, and approximate range of the object of interest and passes this as the initial pose estimation result and constraints to the grasp detection module; d. The grasping detection module estimates the drill pipe posture and transmits the estimation result to the motion control system of the oilfield manipulator; e. The oilfield manipulator equipped with a motion control system controls the oilfield manipulator to perform a grabbing operation according to the received information; The data acquisition module uses an RGB camera and a laser radar or an RGBD camera. The data processing module uses a background difference method on multiple continuous color images to determine whether the drill rod to be grabbed remains stationary. The data processing module includes the following steps: a. Perform background subtraction based on Gaussian mixture modeling and background subtraction; b. Perform grayscale, binarization, and erosion and dilation processing on the image to obtain the foreground contour and draw the circumscribed rectangle of the foreground contour; c. Determine the difference in the center position of the rectangle. If the difference is less than the set threshold, the target is considered to be stationary. d. Send the RGB image and corresponding point cloud data at this time to the target detection module; The initialization calibration module includes the following steps: a. Start the system and perform preliminary visual calibration using the built-in visual sensor and external calibration plate; b. Use the calibration tool at the end of the oilfield manipulator to perform more precise calibration of the vision sensor to ensure the precise spatial correspondence between the oilfield manipulator and the vision sensor; c. Set up multiple calibration points within the reach of the oilfield manipulator and record the position information of these points relative to the vision sensor; d. Based on the collected calibration point data, calculate and generate the kinematic model of the oilfield manipulator and the internal and external parameter matrices of the visual sensor to complete the initialization calibration; e. After the initialization and calibration are completed, save the relevant parameters for subsequent operation; Step c includes the following steps: The object detection module performs 3D object detection on the input multimodal data; 3D object detection mainly includes heterogeneous feature alignment problems and multimodal data fusion problems; Perform image feature extraction and voxel feature extraction on RGB data and point cloud data respectively; Use 3D-2D coordinate projection to map voxel features to corresponding image features and obtain the corresponding image feature index. The specific formula of 3D-2D coordinate projection is as follows: R i =RC·T cam-lidar ·V i ; RC is the combination of the camera correction rotation matrix and the calibration matrix, T cam-lidar is the camera projection matrix, V i is the voxel feature center, R i is the corresponding image feature index; Find the corresponding index pixel feature, obtain the adjacent pixel features by index translation, use Manhattan distance in pixel query, and sample K pixels within the distance threshold; Combine these K pixels with the cross-attention mechanism to dynamically capture the correlation between the two modalities and achieve subsequent fusion; The fused features are input into 3D Backbone for further processing and analysis, and then prediction is performed.

2. The deep learning-based 3D visual guidance system for oilfield manipulator grasping according to claim 1 is characterized in that: The data acquisition module collects image and point cloud data in the following ways: a. Use a high-definition 3D vision camera to capture depth images and color images of the scene in real time; b. Using depth image processing technology, the depth image is converted into point cloud data, where each point contains the position information and corresponding color information in three-dimensional space; c. Preprocess the collected color images to remove noise and unnecessary background information and improve image quality; d. Package the processed image and point cloud data and transmit them to the data processing module in real time for subsequent processing.

3. The deep learning-based 3D visual guidance system for oilfield manipulator grasping according to claim 1 is characterized in that: The target detection module uses deep learning algorithms to perform target recognition and positioning: a. Pre-train a deep neural network model to identify and locate objects of interest in the image; b. When receiving image data transmitted by the data processing module, the image is input into the pre-trained deep neural network model for processing; c. The model outputs the position, orientation, and approximate range of the object of interest as the initial pose estimate; d. Transfer the initial pose estimation results and related constraints to the grasp detection module for subsequent processing.

4. The deep learning-based 3D visual guidance system for oilfield manipulator grasping according to claim 1 is characterized in that: The grasp detection module uses the Iterative Closest Point (ICP) algorithm to estimate the object pose: a. Determine an initial object pose hypothesis based on the initial pose estimation results and constraints provided by the target detection module; b. Use the ICP algorithm to match the point cloud data with the preset object model and continuously adjust the object pose hypothesis through iterative optimization; c. When the matching error reaches a preset threshold or the number of iterations reaches a maximum, the iteration is stopped and the final drill rod posture estimation result is output; d. Transmit the final drill pipe posture estimation result to the motion control system of the oilfield manipulator to guide the oilfield manipulator to perform precise grasping operations.

Images