Oil field mechanical arm grabbing 3D visual guidance system based on deep learning

Abstract

The invention discloses an oil field mechanical arm grabbing 3D visual guidance system based on deep learning. The system comprises an oil field mechanical arm loaded with a motion control system, and a data acquisition module, a data processing module, an initialization calibration module, a target detection module and a grabbing detection module which are in communication connection with the oil field mechanical arm. In oil field operation, when the system initially operates or moves, the initialization calibration module obtains parameter initialization. The data processing module calculates collected images in real time, after it is ensured that the drill rod is static, the images and point cloud data are transmitted to the target detection module to determine the position of an object and then serve as an initial posture estimation result to be transmitted to the grabbing detection module, and finally the result is transmitted to the mechanical arm motion control system for grabbing operation. In conclusion, efficient, intelligent and accurate recognition and grabbing of the drill rod are achieved, new possibility is provided for application of the oil field mechanical arm in complex tasks, and far-reaching influences are generated on oil field automation and intelligent production.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a 3D vision guidance system for oilfield manipulator grasping based on deep learning. Background Art

[0002] In today's oilfield production operations, the accuracy, efficiency and intelligence level of the robot arm grasping operation are also put forward higher and higher requirements. Traditional oilfield robot arm grasping methods usually rely on preset programs and fixed parameters, which are often unable to cope with the complex and changeable oilfield working environment and the diverse equipment and workpiece characteristics. In actual oilfield operations, due to the uncertainty of the shape, size, position and posture of the drill pipe, the robot arm may not be able to accurately grasp the target drill pipe, which leads to low production efficiency, unstable operation quality and even operation failure.

[0003] With the continuous development of 3D vision technology, it can provide rich spatial information for oilfield manipulators, bringing new opportunities to solve the problem of manipulator grasping. However, it is not easy to effectively integrate 3D vision technology with oilfield manipulator grasping system, which requires solving a series of technical problems such as data collection, processing, transmission and coordination with manipulator control system.

[0004] In addition, under the general trend of oilfield automation and intelligent production, improving the degree of automation and intelligence on the production line has become the key. As an important actuator, the improvement of the gripping performance of the robotic arm is of great significance to the optimization and upgrading of the entire oilfield production process.

[0005] In summary, in order to meet the higher requirements of robotic arm grasping operations in the oilfield field and achieve more efficient, intelligent and precise grasping, the development of a deep learning-based oilfield robotic arm grasping 3D visual guidance system has become a hot topic and urgent need in current technical research. Summary of the invention

[0006] The purpose of this invention is to realize the intelligent, efficient and precise grasping of drill pipes by oilfield manipulators to meet the diverse needs of oilfield automation and intelligent production. By introducing 3D visual guidance technology, the system can perceive and identify the state of drill pipes in the oilfield environment in real time, and automatically plan the grasping path for grasping, thereby greatly improving the grasping efficiency and accuracy of oilfield manipulators.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The system is equipped with an oilfield manipulator arm with a motion control system and a data acquisition module, a data processing module, an initialization calibration module, a target detection module and a grasping detection module which are connected to the oilfield manipulator arm in communication.

[0009] (1) When the system is first operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0010] (2) The data acquisition module collects data and sends it to the data processing module after preprocessing;

[0011] (3) The data processing module performs real-time calculations on the images transmitted by the data acquisition module, ensuring that the drill rod to be grasped is in a stationary state before transmitting the image data and point cloud data collected by the data acquisition module to the target detection module;

[0012] (4) The object detection module determines the position, orientation, and approximate range of the object of interest and passes it to the grasp detection module as the initial pose estimation result and constraints;

[0013] (5) The grasping detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield manipulator;

[0014] (6) The oilfield robot arm equipped with a motion control system controls its grasping operation according to the transmitted information.

[0015] A further improvement of the present invention is that the specific implementation steps of step (1) are:

[0016] (101) The initialization module contains two parts, namely camera calibration and hand-eye calibration. When the system is running for the first time or moves, the initialization calibration module is run to obtain the necessary parameters for initialization, such as the internal and external parameters of the camera, the conversion relationship between the posture (position) of the end effector of the oilfield robot arm and the camera coordinate system, etc.

[0017] A further improvement of the present invention is that the specific implementation steps of step (2) are:

[0018] (201) Using a high-definition 3D vision camera to capture depth images and color images of the scene in real time;

[0019] (202) The depth image is converted into point cloud data by using depth image processing technology, wherein each point contains position information and corresponding color information in three-dimensional space;

[0020] (203) Preprocessing the collected color images to remove noise and unnecessary background information and improve image quality;

[0021] (204) The processed image and point cloud data are packaged and transmitted to the data processing module in real time for subsequent processing.

[0022] A further improvement of the present invention is that the specific implementation steps of step (3) are:

[0023] (301) The data acquisition module uses an RGB camera and a LiDAR or RGBD camera;

[0024] (302) The data processing module uses a background difference method to determine whether the drill rod to be grasped remains stationary for the input multiple continuous color images.

[0025] A further improvement of the present invention is that the specific implementation steps of step (4) include:

[0026] (401) The target detection module performs 3D target detection on the input multimodal data;

[0027] (402) 3D object detection mainly includes heterogeneous feature alignment problems and multimodal data fusion problems;

[0028] (403) performing image feature extraction and voxel feature extraction on the RGB data and the point cloud data respectively;

[0029] (404) Use 3D-2D coordinate projection to map the voxel features to the corresponding image features and obtain the corresponding image feature index. The specific formula of 3D-2D coordinate projection is as follows:

[0030] (405) ;

[0031] (406) RC is the combination of the camera correction rotation matrix and the calibration matrix, is the camera projection matrix, V i is the voxel feature center, R i is the corresponding image feature index;

[0032] (407) Find the corresponding index pixel feature, obtain the adjacent pixel features by index translation with effective distance, use Manhattan distance in pixel query, and sample K pixels within the distance threshold;

[0033] (408) These K pixels are combined with a cross-attention mechanism to dynamically capture the correlation between the two modalities and achieve subsequent fusion.

[0034] (409) The fused features are input into 3D Backbone for further processing and analysis, and then prediction is performed.

[0035] A further improvement of the present invention is that the specific implementation steps of step (5) are:

[0036] (501) The grasping detection module uses the 3D target detection results as the initial posture and constraint conditions of the drill rod and refines the posture of the fused features.

[0037] (502) Using the ICP algorithm, the point cloud data is matched with the preset object model, and the object posture hypothesis is continuously adjusted through iterative optimization;

[0038] (503) When the matching error reaches a preset threshold or the number of iterations reaches a maximum value, the iteration is stopped and the final drill rod posture estimation result is output;

[0039] (504) The final drill rod posture estimation results such as the drill rod spatial coordinates (x, y, z), rotation angle (θ x ,θ y , θ z ), gripping width w, confidence score c, etc. are transmitted to the motion control system of the oilfield robot arm to guide the oilfield robot arm to perform precise grasping operations.

[0040] The present invention has at least the following beneficial technical effects:

[0041] First of all, the present invention provides a brand-new solution, which effectively solves the problems and limitations of traditional technologies. Through innovative design concepts and advanced technical means, the present invention not only optimizes the operation process and improves work efficiency, but also significantly reduces energy consumption and reduces environmental burden.

[0042] Secondly, the implementation process of the present invention is simple and easy to understand, easy to operate and maintain. The modular design adopted makes the functions of each part clear, which is convenient for users to quickly master and use. At the same time, the present invention also has a high degree of scalability and customizability, and can be flexibly adjusted according to the actual needs of users to meet the application needs in different scenarios.

[0043] In addition, the present invention is highly secure and stable. Through strict quality control and safety testing, it is ensured that the present invention can operate stably in various complex environments and effectively protect the data security of users. Such security and stability are crucial for both enterprises and individual users, and can avoid significant losses caused by system crashes or data leaks.

[0044] Finally, the application prospects of the present invention are broad. With the continuous development of science and technology and the continuous progress of society, people's demand for efficient, convenient and environmentally friendly solutions is growing. The present invention is born based on this demand. It can not only meet the needs of the current market, but also has huge potential and development space, and is expected to be widely used and promoted in more fields.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0047] FIG1 is a flow chart of a data processing module in a deep learning-based 3D vision guidance system for oilfield manipulator grasping according to the present invention.

[0048] FIG2 is an overall structural diagram of the deep learning-based 3D vision guidance system for oilfield manipulator grasping according to the present invention. DETAILED DESCRIPTION

[0049] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0050] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0051] As shown in FIG2 , the present invention provides a 3D visual guidance system for oilfield manipulator grasping based on deep learning, comprising the following steps:

[0052] (1) When the system is first operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0053] (2) The data acquisition module collects data and sends it to the data processing module after preprocessing;

[0054] (3) The data processing module performs real-time calculations on the images transmitted by the data acquisition module, ensuring that the drill rod to be grasped is in a stationary state before transmitting the image data and point cloud data collected by the data acquisition module to the target detection module;

[0055] (4) The object detection module determines the position, orientation, and approximate range of the object of interest and passes it to the grasp detection module as the initial pose estimation result and constraints;

[0056] (5) The grasping detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield manipulator;

[0057] (6) The oilfield robot arm equipped with a motion control system controls the robot arm to perform grasping operations based on the transmitted information.

[0058] Example 1

[0059] As shown in FIG1 , Embodiment 1 of the present invention provides a flow chart of a data processing module in a 3D visual guidance system for grabbing oilfield manipulators based on deep learning, and the specific operation steps are as follows:

[0060] (1) First, perform background subtraction based on the Gaussian mixture model.

[0061] (2) Grayscale, binarize, and erode the image to obtain the foreground contour and draw the circumscribed rectangle of the foreground contour;

[0062] (3) Determine the difference in the center position of the rectangle. If it is less than the set threshold, the target is considered to be stationary.

[0063] (4) The RGB image and corresponding point cloud data at this time are sent to the target detection module.

[0064] Example 2

[0065] As shown in FIG2 , Embodiment 2 of the present invention provides an overall structure diagram of the oilfield manipulator grasping 3D vision guidance system based on deep learning, including:

[0066] Initialization and calibration module: when the system device is initially operated or moved, the initialization and calibration module needs to be run to obtain the required parameters for initialization;

[0067] Data acquisition module, collecting RGB image data and point cloud data;

[0068] The data processing module performs real-time calculations on the images transmitted by the data acquisition module to ensure that the drill rod to be grabbed is

[0069] After reaching the stationary state, the image data and point cloud data collected by the data acquisition module are transmitted to the target detection module;

[0070] The object detection module determines the position, orientation, and approximate range of the object of interest and passes it to the grasp detection module as the initial pose estimation result and constraints;

[0071] The grab detection module estimates the drill pipe posture and transmits the obtained results to the motion control system of the oilfield robot arm;

[0072] The motion control system controls the oilfield robot arm to perform grasping operations based on the transmitted information.

[0073] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0075] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing device, and a series of operating steps may be executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0077] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A 3D visual guidance system for oilfield manipulator grabbing based on deep learning, characterized by: The following steps are involved: a. When the system device is initially operated or moved, run the initialization calibration module to obtain and initialize the required parameters; b. The data processing module calculates the image transmitted by the data acquisition module in real time, ensures that the drill rod to be grabbed is in a stationary state, and transmits the image data and point cloud data collected by the data acquisition module to the target detection module; c. The object detection module determines the position, orientation, and approximate range of the object of interest and passes it to the grasp detection module as the initial pose estimation result and constraints; d. The grab detection module estimates the drill pipe posture and transmits the estimation result to the motion control system of the oilfield manipulator; e. The oilfield manipulator equipped with a motion control system controls the oilfield manipulator to perform a grabbing operation according to the received information.

2. The oilfield manipulator grasping 3D vision guidance system based on deep learning according to claim 1 is characterized in that: The initialization calibration module comprises the following steps: a. Start the system device and perform preliminary visual calibration using the built-in visual sensor and external calibration board; b. Use the calibration tool at the end of the oilfield robot arm to calibrate the visual sensor more finely to ensure the precise spatial correspondence between the oilfield robot arm and the visual sensor; c. Set multiple calibration points within the reach of the oilfield robot arm and record the position information of these points relative to the visual sensor; d. Based on the collected calibration point data, calculate and generate the kinematic model of the oilfield manipulator and the internal and external parameter matrix of the visual sensor to complete the initialization calibration; e. After the initialization calibration is completed, save the relevant parameters for subsequent operation.

3. The oilfield manipulator grasping 3D visual guidance system based on deep learning according to claim 1 is characterized in that: The data acquisition module acquires image and point cloud data in the following ways: a. Use a high-definition 3D vision camera to capture depth images and color images of the scene in real time; b. Through the depth image processing technology, the depth image is converted into point cloud data, in which each point contains the position information and corresponding color information in the three-dimensional space; c. Preprocess the collected color images to remove noise and unnecessary background information and improve image quality; d. Package the processed image and point cloud data and transmit them to the data processing module in real time for subsequent processing.

4. The oilfield manipulator grasping 3D visual guidance system based on deep learning according to claim 1 is characterized in that: The data processing module comprises the following steps: a. Perform background subtraction based on mixed Gaussian model modeling and use background subtraction; b. Grayscale, binarize, and corrode and dilate the image to obtain the foreground contour, and draw the circumscribed rectangle of the foreground contour; c. Determine the difference in the center position of the rectangle. If the difference is less than the set threshold, the target is considered to remain stationary. d. Send the RGB image and corresponding point cloud data at this time to the target detection module.

5. The oilfield manipulator grasping 3D visual guidance system based on deep learning according to claim 1 is characterized in that: The target detection module uses a deep learning algorithm for target recognition and positioning: a. Pre-train a deep neural network model to identify and locate objects of interest in the image; b. When receiving the image data transmitted by the data processing module, the image is input into the pre-trained deep neural network model for processing; c. The model outputs the position, orientation, and approximate range of the object of interest as the initial pose estimation result; d. The initial pose estimation results and related constraints are transmitted to the grasp detection module for subsequent processing.

6. The oilfield manipulator grasping 3D visual guidance system based on deep learning according to claim 1, characterized in that: The grasp detection module uses the iterative closest point (ICP) algorithm to estimate the object posture: a. Determine an initial object pose hypothesis based on the initial pose estimation results and constraints provided by the target detection module; b. Use the ICP algorithm to match the point cloud data with the preset object model, and continuously adjust the object posture hypothesis through iterative optimization; c. When the matching error reaches a preset threshold or the number of iterations reaches a maximum value, the iteration is stopped and the final drill rod posture estimation result is output; d. The final drill pipe posture estimation result is transmitted to the motion control system of the oilfield robot arm to guide the oilfield robot arm to perform precise grasping operations.

Images