Target pose progressive optimization identification method and system in robot operation process

Abstract

The invention belongs to the field of autonomous operation of power robots, and provides a progressive optimization recognition method and system for a target pose in the operation process of a robot, and the technical scheme is as follows: the pose recognition is divided into three stages of pose adjustment, coarse detection and fine detection, at a first operation position, a template matching algorithm is adopted to carry out template matching on a target object, and at a second operation position, the template matching algorithm is adopted to carry out template matching on the target object; obtaining the position and attitude of the target object in the image, and constructing a bounding box of the target object based on the position and attitude of the target object in the image; a second target operation position is determined based on the surrounding frame of the target object, and the mechanical arm is controlled to reach the second operation position; and based on the second operation position, a third target operation position is obtained according to pixel information of the target object in the image, the coordinate position of the mechanical arm is adjusted according to the third target operation position to reach the final operation position, and operation is executed. The influence of a complex environment is overcome, the attitude estimation precision is improved, and the autonomous operation precision of the power robot is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous operation of electric robots, and in particular relates to a method and system for progressive optimization and recognition of target pose during robot operation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Electrical equipment frequently requires maintenance and repair, and traditional methods necessitate manual intervention, which is time-consuming and labor-intensive. Autonomous maintenance and repair robots can perform simple tasks such as pressing buttons and tightening nuts. These robots are typically equipped with robotic arms, tools, and sensors, enabling them to operate autonomously based on the equipment's status and feedback. Target pose recognition is a crucial task during robotic operations, involving the robot's ability to accurately identify and locate target objects or positions.

[0004] The inventors discovered that current pose recognition methods during operation mainly suffer from the following problems: 1) Insufficient pose estimation accuracy. In some cases, the accuracy of pose estimation may be limited. For example, when the target object has a complex shape or texture, traditional feature extraction and matching methods may not be able to accurately estimate its pose, resulting in insufficient accuracy.

[0005] 2) Inability to handle complex environments. In complex environments, such as those with varying lighting, occlusion, or reflections, target pose recognition becomes more difficult. These environmental factors can cause noise or distortion in sensor data, thus affecting the accuracy and stability of target pose. Summary of the Invention

[0006] To address at least one of the technical problems mentioned above, this invention provides a progressive optimization and recognition method and system for target pose during robot operation. The method decomposes pose recognition into three stages: pose adjustment, coarse detection, and fine detection. By progressively operating these three stages, the influence of complex environments is overcome, the accuracy of pose estimation is improved, and the accuracy of autonomous operation of electric robots is effectively enhanced.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for progressive optimization and recognition of target pose during robot operation, comprising the following steps: Acquire image data; Target detection is performed based on image data to obtain the target area for the task. Extract point cloud data of the target area, fit the point cloud using a plane fitting algorithm to obtain the first target operation position, and control the robotic arm to reach the first operation position based on the first target operation position; At the first working position, a template matching algorithm is used to perform template matching on the target object to obtain the position and pose of the target object in the image. Based on the position and pose of the target object in the image, a bounding box of the target object is constructed. Based on the bounding box of the target object, the second target working position is determined, and the robotic arm is controlled to move to the second working position. Based on the second working position, the third target working position is obtained according to the pixel information of the target object in the image. The coordinate position of the robotic arm is adjusted according to the third target working position to reach the final working position and perform the operation.

[0008] Further, the step of extracting point cloud data of the target area and fitting the point cloud using a plane fitting algorithm to obtain the first target operation location includes: assuming that the plane to be fitted is represented by an equation, i.e. ax j + by j + cz j + d =0. Where, a , b , c These are the components of the plane normal vector, and d It is the negative of the distance from the plane to the origin; the point cloud is represented as P={( x j , y j , z j )},in( x j , y j ) is the first in the point cloud j Image coordinates of points, z j This corresponds to the depth value; By combining point cloud data and the equations of the plane to be fitted, a system of linear equations is constructed. By using the least squares method, solving this system of linear equations yields the parameters of the plane equations, thus determining the equations of the plane model. The normal vector of the plane is calculated based on the equations of the plane model. The fixed distance from the center of the plane and perpendicular to the plane is calculated based on the normal vector of the plane and used as the first target operation position.

[0009] Furthermore, when the robotic arm reaches the first target working position, the normal vector of the point cloud is compared with the reference direction of the robotic arm coordinate system to calculate the angle between the point cloud normal vector and the three axes of the robotic arm coordinate system. θ Based on the included angle θ Control the movement of the robotic arm; Here, the reference direction of the robotic arm coordinate system is assumed to be represented as R=( r x , r y , r z ), r x , r y , r z express x , y , z Reference directions and included angles of the three-directional robot arm coordinate system θ We obtain the following by calculating the dot product of the point cloud normal vector and the reference direction: θ =arccos((N·R) / (||N|| ||R||)), where · represents the dot product operation, and || || represents the magnitude of the vector. N This indicates the number of point clouds.

[0010] Further, in the first working position, a template matching algorithm is used to perform template matching on the target object to obtain the position and pose of the target object in the image, including: Using a pre-acquired set of 2D feature lines of the target object, it is matched with the input image. Through the matching process, the position of the target object in the image is found. Based on the matching results of the linemod algorithm, the pose information of the target object is obtained, including the position of the target object in the image and its rotation angle relative to the image plane.

[0011] Furthermore, constructing the bounding box of the target object based on its position and pose in the image includes: Based on the position and pose of the target object in the image, select the corresponding point set from the pre-acquired 3D template of the target object; Map the selected 3D template point set from 3D space to the 2D image space of the image; The bounding box of the target object is determined based on the template point set mapped to the 2D image space. Based on the actual size and shape of the target object, the bounding box is further adjusted to obtain the final bounding box of the target object.

[0012] Furthermore, when determining the bounding box of the target object based on the template point set mapped to the 2D image space, the minimum bounding rectangle or minimum bounding circle is used to include all template point sets and minimize the area of ​​the bounding box.

[0013] A second aspect of the present invention provides a target pose progressive optimization recognition system during robot operation, comprising: The target region detection module is used to acquire image data, perform target detection based on the image data, and obtain the target region for the task. The initial work position determination module is used to extract point cloud data of the work target area, fit the point cloud with a plane fitting algorithm to obtain the first target work position, and control the robotic arm to reach the first work position according to the first target work position. The target work position coarse detection module performs template matching on the target object at the first work position to obtain the position and pose of the target object in the image. Based on the position and pose of the target object in the image, it constructs the bounding box of the target object. Based on the bounding box of the target object, it determines the second target work position and controls the robotic arm to reach the second work position. The target operation position precision detection module obtains the third target operation position based on the second operation position and the pixel information of the target object in the image. Based on the third target operation position, the coordinate position of the robotic arm is adjusted to reach the final operation position and the operation is performed.

[0014] A third aspect of the present invention provides a computer-readable storage medium.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the target pose progressive optimization recognition method for robot operations as described above.

[0016] A fourth aspect of the present invention provides a computer device.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the target pose progressive optimization recognition method for robot operation as described above.

[0018] A fourth aspect of the present invention provides a program product.

[0019] A program product, which is a computer program product, includes a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps in the target pose progressive optimization recognition method for robot operation as described above.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a progressive optimization algorithm for target pose recognition during robot operations. The algorithm breaks down pose recognition into three stages: pose adjustment, coarse detection, and fine detection. By progressively executing these three stages, it avoids errors from single pose recognition, overcomes the influence of complex environments, improves the accuracy of pose estimation, and effectively enhances the autonomous operation accuracy of electric robots.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of a progressive optimization and recognition method for target pose during robot operation provided by an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] To address the technical problems mentioned in the background section of this invention, such as insufficient attitude estimation accuracy and inability to cope with complex environments, this invention decomposes pose recognition into three stages: attitude adjustment, coarse detection, and fine detection. Through the sequential execution of these three stages, the influence of complex environments is overcome, the accuracy of attitude estimation is improved, and the autonomous operation accuracy of electric robots is effectively enhanced.

[0028] Example 1 like Figure 1 As shown, this embodiment provides a progressive optimization and recognition method for target pose during robot operation, including the following steps: Step 1: Posture Adjustment Step 101: Acquire depth camera data: Assume the depth camera data is represented as D={ d i},in d i The depth camera acquires the first... i The depth value of each pixel.

[0029] Step 102: Obtain the target detection region: The target detection algorithm can obtain the position and size of the bounding box of the target object, represented as... R =( x , y , w , h ),in( x , y () represents the coordinates of the top-left corner of the bounding box. w and h These are the width and height of the bounding box, respectively.

[0030] Step 103: Construct point cloud: Extract point cloud of target detection area from depth camera data.

[0031] A point cloud consists of multiple three-dimensional coordinate points. Let the point cloud be represented as P = {( x j , y j , z j )},in( x j , y j ) is the first in the point cloud j Image coordinates of points, z j That is the corresponding depth value.

[0032] Step 104: Plane Fitting: Use a plane fitting algorithm to fit the point cloud to obtain the equation of the planar model. This includes the following steps: Step 1041: Assume the plane to be fitted is represented by an equation, i.e. ax j + by j + cz j + d =0. Where, a , b , c These are the components of the plane normal vector, and d It is the negative of the distance from the plane to the origin; Step 1042: Combine the point cloud data and the equations of the plane to be fitted to construct a system of linear equations; To find the best-fitting plane, the least squares method is used, which fits the model by minimizing the sum of squared errors. Solving this system of linear equations yields the parameters a, b, and c of the plane equation. The obtained parameters a, b, and c... d The equations of the planar model were determined; Step 105: Determine the first target operation position based on the equations of the plane model; specifically, calculate the normal vector of the plane based on the equations of the plane model, and calculate a fixed distance from the center of the plane and perpendicular to the plane based on the normal vector of the plane as the first target operation position; Step 106: Control the robotic arm to reach the first working position according to the first target working position; When controlling the movement of the robotic arm, the normal vector of the point cloud is compared with the reference direction of the robotic arm's coordinate system. The angle between the point cloud normal vector and the three axes of the robotic arm's coordinate system is calculated, and the angle is then used to determine the direction of the movement. θ Control the movement of the robotic arm.

[0033] Here, the reference direction of the robotic arm coordinate system is assumed to be represented as R=( r x , r y , r z The included angle can be obtained by calculating the dot product of the point cloud normal vector and the reference direction: θ =arccos((N·R) / (||N|| ||R||)). Where · represents the dot product operation, and || || represents the magnitude of the vector. r x , r y , r z express x , y , z Reference directions of the robot arm coordinate system in three directions. N This indicates the number of point clouds.

[0034] Step 2: Coarse Inspection Step 201: Perform template matching on the target object using a template matching algorithm. This algorithm uses a pre-acquired set of 2D feature lines of the target object and matches it with the input image. Through the matching process, the location of the target object in the image can be found.

[0035] Step 202: Obtain the matching results: Based on the matching results of the linemod algorithm, obtain the pose information of the target object. This includes the position of the target object in the image ( x ,y ) and its rotation angle (θ) relative to the image plane.

[0036] Step 203: Construct the bounding box: Based on the matching results, construct the bounding box of the target object. This includes the following steps: Step 2031: Obtain the template point set of the target object: Based on the matching results, select the corresponding point set from the pre-obtained 3D template of the target object. These point sets can be feature points or boundary points representing the target object.

[0037] Step 2032: Map the template point set from 3D space to 2D image space: Based on the pose information of the target object, the selected template point set is mapped from 3D space to the 2D image space of the input image by using the camera's intrinsic and extrinsic parameters and pose information to perform perspective projection calculation.

[0038] Step 2033: Determine the bounding box boundary: Based on the template point set mapped to 2D image space, determine the bounding box of the target object. Geometry such as a minimum bounding rectangle or minimum bounding circle can be used to include all template point sets and minimize the area of ​​the bounding box.

[0039] Step 2034: Adjust the size and position of the bounding box: As needed, the bounding box can be further adjusted according to the actual size and shape of the target object. This can be achieved by calculating the appropriate bounding box size and position based on the position and distribution of the template point set.

[0040] Step 3: Fine-tuning At this point, distance information is not calculated because the distance is relatively short and the target object has a large number of pixels in the field of view. Instead, sub-pixel information is obtained from the image's coordinates, and adjustments are made in both the X and Y directions to gradually refine the accuracy. Based on the extracted sub-pixel information, the system is adjusted in the X and Y directions. Utilizing real-time image feedback from the camera, the displacement in the X and Y directions is continuously adjusted. By constantly acquiring new sub-pixel level position information, the system can correct deviations in real time, ensuring continuous improvement in accuracy.

[0041] Smaller adjustments are made using subpixel information to gradually approach the ideal position. After each adjustment, the image is reacquired and subpixel information is extracted again to evaluate the effect of the current adjustment.

[0042] Example 2 This embodiment provides a target pose progressive optimization recognition system during robot operations, including: The target region detection module is used to acquire image data, perform target detection based on the image data, and obtain the target region for the task. The initial work position determination module is used to extract point cloud data of the work target area, fit the point cloud with a plane fitting algorithm to obtain the first target work position, and control the robotic arm to reach the first work position according to the first target work position. The target work position coarse detection module is used to perform template matching algorithm on the target object at the first work position to obtain the position and pose of the target object in the image. Based on the position and pose of the target object in the image, the bounding box of the target object is constructed. Based on the bounding box of the target object, the second target work position is determined and the robotic arm is controlled to reach the second work position. The target operation position precision detection module is used to obtain the third target operation position based on the second operation position and the pixel information of the target object in the image. Based on the third target operation position, the coordinate position of the robotic arm is adjusted to reach the final operation position and the operation is performed.

[0043] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the target pose progressive optimization recognition method for robot operations as described above.

[0044] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the target pose progressive optimization recognition method for robot operation as described above.

[0045] Example 5 This embodiment provides a program product, which is a computer program product including a computer program. The computer program is characterized in that, when executed by a processor, it implements the steps in the target pose progressive optimization recognition method for robot operation as described above.

[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying a target pose progression optimization in a robot job process, characterized in that, The method comprises the following steps: Obtaining image data; Performing target detection based on the image data to obtain a work target region; Extracting point cloud data of the work target region, fitting the point cloud by using a plane fitting algorithm to obtain a first target work position, and controlling a mechanical arm to reach the first work position according to the first target work position; At the first work position, performing template matching on the target object by using a template matching algorithm to obtain a position and a pose of the target object in an image, and constructing a bounding box of the target object based on the position and the pose of the target object in the image; Determining a second target work position based on the bounding box of the target object, and controlling the mechanical arm to reach the second target work position; Based on the second work position, obtaining a third target work position according to pixel information of the target object in the image, adjusting a coordinate position of the mechanical arm to reach a final work position according to the third target work position, and performing a work operation.

2. The method of claim 1, wherein the method further comprises: determining a pose of the target object in the robot work cell; and determining a pose of the target object in the robot work cell based on the determined pose of the target object in the robot work cell. The step of extracting point cloud data of the target area and fitting the point cloud with a plane fitting algorithm to obtain the first target operation location includes: assuming that the plane to be fitted is represented by an equation, i.e. ax j + by j + cz j + d =0. Where, a , b , c These are the components of the plane normal vector, and d It is the negative of the distance from the plane to the origin; the point cloud is represented as P={( x j , y j , z j )},in( x j , y j ) is the first in the point cloud j Image coordinates of points, z j This corresponds to the depth value; Joint point cloud data and the equation of the plane to be fitted to construct a linear equation set; The parameters of the plane equation can be obtained by solving the linear equation set by using the least square method, and the equation of the plane model is determined; The normal vector of the plane is calculated based on the equation of the plane model, and a fixed distance from the center of the plane in the direction perpendicular to the plane is calculated as the first target work position.

3. The method of claim 1, wherein the method further comprises: determining a pose of the target object in the robot work cell; and determining a pose of the target object in the robot work cell based on the determined pose of the target object in the robot work cell. According to the first target working position, when the mechanical arm reaches the first working position, the normal vector of the point cloud is compared with the reference direction of the mechanical arm coordinate system, the included angle between the normal vector of the point cloud and the three axes of the mechanical arm coordinate system is calculated θ , and the mechanical arm is controlled to move based on the included angle θ ; wherein the reference direction of the robot coordinate system is denoted as R = (R x, R y, R z)T, r x , r y , r z , r x , r y , r z denote the reference direction of the robot coordinate system in the three directions, the included angle x , y , z between the reference direction of the robot coordinate system and the normal vector of the point cloud is denoted as θ The dot product of the normal vector of the point cloud and the reference direction is calculated as follows: θ = arccos((N · R) / (||N|| ||R||)), wherein · denotes the dot product operation, || || denotes the modulus of the vector, N denotes the number of the point cloud.

4. The method of claim 1, wherein, The template matching algorithm is used to match the target object at the first work position to obtain the position and the pose of the target object in the image, comprising: Using a pre-acquired 2D feature line set of the target object to match with the input image, and finding the position of the target object in the image through the matching process; According to the matching result of the linemod algorithm, the pose information of the target object is obtained, including the position of the target object in the image and the rotation angle of the target object relative to the image plane.

5. The method of claim 1, wherein, The bounding box of the target object is constructed based on the position and the pose of the target object in the image, comprising: According to the position and the pose of the target object in the image, a corresponding point set is selected from a pre-acquired 3D template of the target object; The selected 3D template point set is mapped from the 3D space to the 2D image space of the image; Based on the template point set mapped to the 2D image space, the bounding box of the target object is determined; According to the actual size and shape of the target object, the bounding box is further adjusted to obtain the final bounding box of the target object.

6. The method of claim 1, wherein, When determining the bounding box of the target object based on the template point set mapped to the 2D image space, the minimum enclosing rectangle or the minimum enclosing circle is used to contain all the template point sets and minimize the area of the bounding box.

7. A target pose progressive optimization and recognition system for robot operations, characterized in that, Comprise: A target region detection module for obtaining image data, performing target detection based on the image data to obtain a work target region; An initial work position determination module for extracting point cloud data of the work target region, fitting the point cloud by using a plane fitting algorithm to obtain a first target work position, and controlling a mechanical arm to reach the first work position according to the first target work position; a target work position coarse detection module, configured to perform template matching on the target object at the first work position by using a template matching algorithm to obtain a position and an attitude of the target object in the image, and construct a bounding box of the target object based on the position and the attitude of the target object in the image; determine a second target work position based on the bounding box of the target object, and control the robot arm to reach the second target work position; a target work position fine detection module, configured to obtain a third target work position based on pixel information of the target object in the image based on the second target work position, adjust a coordinate position of the robot arm to a final work position according to the third target work position, and perform a work operation.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the target pose progressive optimization identification method in the robot work process according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the target pose progressive optimization identification method in the robot work process according to any one of claims 1-6.

10. A program product, the program product being a computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the target pose progressive optimization identification method in the robot work process according to any one of claims 1-6.

Images