Object pose refinement method, device and medium for industrial robot

Abstract

The application belongs to the technical field of industrial internet, and particularly relates to an object pose refinement method for an industrial robot, a device and a medium. After a depth image and a color image of a target object are acquired through an image acquisition device, a first point cloud of the target object is determined according to the depth image, the color image and an intrinsic matrix of the image acquisition device; an initial coarse pose transformation matrix of the target object is determined according to the color image, the depth image and / or the first point cloud; a virtual camera of the target object is constructed based on the initial coarse pose transformation matrix, and a second point cloud of the target object is determined based on the virtual camera; the initial coarse pose transformation matrix is refined based on the first point cloud and the second point cloud, and a pose of the target object is obtained. The method constructs a virtual physical imaging space by introducing a virtual camera, directly generates a high-fidelity local visible point cloud consistent with a real scene, and effectively solves the problem that the prior art cannot accurately position a workpiece pose.

Description

Technical Field

[0001] This application belongs to the field of industrial internet technology, specifically relating to a method, device, and medium for object pose refinement for industrial robots. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, disordered grasping operations are gradually becoming a new trend in the interaction between industrial robots and objects. In this process, industrial robots need to possess the ability to perceive objects in order to identify their position and orientation. However, relying solely on preliminary object perception is insufficient to meet the requirements; therefore, further refinement of the pose is necessary.

[0003] In existing technologies, pose refinement is often achieved using the following two methods: The first is global geometric registration, which directly performs ICP registration between the complete 3D model point cloud of the object and the single-view visible surface point cloud acquired by the depth camera; the second is a rendering-based alignment method, which uses depth buffering technology to render the 3D model into a virtual depth map, back-projects the virtual depth map into a virtual point cloud, and then matches and registers it with the real point cloud acquired by the depth camera.

[0004] However, both of the above methods have their own drawbacks: the first global geometric registration algorithm incorrectly pulls the back face of the model toward the visible surface of the camera, resulting in large translational errors in the depth direction (Z-axis) and systematic deviations in rotational attitude, making it difficult to meet the requirements of high-precision pose; the second method based on rendering alignment not only produces step-like quantization errors when back-projecting virtual data back to the point cloud, but also fails to obtain continuous and high-precision surface normal vectors, which makes the point-to-surface registration process converge slowly and even easily get trapped in local optima, thus failing to guarantee high-precision pose. Summary of the Invention

[0005] This application provides a method, device, and medium for refining the pose of objects for industrial robots, which solves the problem that existing technologies struggle to accurately position workpieces.

[0006] In a first aspect, this application provides a method for refining the pose of an object for an industrial robot, including:

[0007] The depth and color images of the target object are acquired using an image acquisition device;

[0008] Based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device, the first point cloud of the target object is determined;

[0009] The initial coarse pose transformation matrix of the target object is determined based on the color image, the depth image, and / or the first point cloud.

[0010] Based on the initial coarse pose transformation matrix, a virtual camera for the target object is constructed, and based on the virtual camera, the second point cloud of the target object is determined.

[0011] Based on the first point cloud and the second point cloud, the initial coarse pose transformation matrix is ​​refined to obtain the pose of the target object.

[0012] Optionally, determining the first point cloud of the target object based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device includes:

[0013] The color image is masked to obtain a masked image;

[0014] The depth image and the mask image are multiplied to obtain the mask depth image;

[0015] The first point cloud is obtained by performing three-dimensional back projection processing on the mask depth image and the intrinsic parameter matrix.

[0016] Optionally, determining the initial coarse pose transformation matrix of the target object based on the color image, depth image, and / or the first point cloud includes:

[0017] Obtain the three-dimensional model data of the target object, wherein the three-dimensional model data includes at least one of a three-dimensional template model and a three-dimensional template point cloud;

[0018] The initial coarse pose transformation matrix is ​​determined based on at least one of the color image, depth image, or first point cloud, and the three-dimensional model data.

[0019] Optionally, the virtual camera includes: an optical center and an extrinsic parameter matrix; the construction of the virtual camera of the target object based on the initial coarse pose transformation matrix includes:

[0020] Obtain a 3D model of the target object and fix the 3D model;

[0021] The initial coarse pose transformation matrix is ​​used as the extrinsic parameter matrix of the virtual camera.

[0022] The optical center of the virtual camera is determined based on the extrinsic parameter matrix of the virtual camera.

[0023] Optionally, determining the second point cloud of the target object based on the virtual camera includes:

[0024] Based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, the beam emission set of the virtual camera is determined, wherein the beam emission set includes multiple rays, and the beam direction and beam path corresponding to each ray;

[0025] The virtual camera is controlled to emit each ray from its optical center, according to the beam direction and beam path corresponding to each ray;

[0026] The second point cloud is determined based on the three-dimensional model and the emitted rays.

[0027] Optionally, determining the second point cloud based on the three-dimensional modeling model and the emitted rays includes:

[0028] The three-dimensional modeling model is triangulated to obtain a triangular mesh model, which includes multiple triangular facets and a facet normal vector corresponding to each triangular facet.

[0029] The second point cloud is determined based on the plurality of said triangular facets, the facet normal vector corresponding to each of said triangular facets, and each of said emitted rays.

[0030] Optionally, determining the second point cloud based on the plurality of triangular facets, the facet normal vector corresponding to each triangular facet, and each emitted ray includes:

[0031] For any emitted ray, spatial intersection of the ray with each of the triangular facets is determined to obtain a set of collision points corresponding to the ray; the set of collision points includes at least one candidate collision point, and each candidate collision point corresponds to a collision depth and the facet normal vector of the triangular facet to which it belongs;

[0032] According to the collision depth corresponding to the candidate collision point, the candidate collision points corresponding to the ray are sorted in ascending order to obtain the sorted candidate collision points.

[0033] The sorted candidate collision points are filtered to obtain the target collision point, and the collision depth of the target collision point is less than the collision depth of other candidate collision points on the same ray.

[0034] The target collision point and the normal vector of the surface to which the target collision point belongs are combined to determine the second point cloud of the target object.

[0035] Optionally, the step of refining the initial coarse pose transformation matrix based on the first point cloud and the second point cloud to obtain the pose of the target object includes:

[0036] The initial coarse pose transformation matrix is ​​used as the initial pose of the target object;

[0037] Based on the initial pose, the first point cloud and the second point cloud are refined to obtain the pose.

[0038] Secondly, this application provides an object pose refinement device for industrial robots, comprising:

[0039] The acquisition module is used to acquire depth and color images of the target object through an image acquisition device;

[0040] The determination module is used to determine the first point cloud of the target object based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device;

[0041] The determining module is further configured to determine the initial coarse pose transformation matrix of the target object based on the color image, the depth image, and / or the first point cloud;

[0042] A construction module is used to construct a virtual camera for the target object based on the initial coarse pose transformation matrix;

[0043] The determining module is further configured to determine a second point cloud of the target object based on the virtual camera;

[0044] The processing module is used to refine the initial coarse pose transformation matrix based on the first point cloud and the second point cloud to obtain the pose of the target object.

[0045] Optionally, the processing module is further configured to perform masking processing on the color image to obtain a masked image;

[0046] The processing module is further configured to multiply the depth image and the mask image to obtain a mask depth image;

[0047] The processing module is specifically used to perform three-dimensional back projection processing on the mask depth image and the intrinsic parameter matrix to obtain the first point cloud.

[0048] Optionally, the acquisition module is further configured to acquire three-dimensional model data of the target object, wherein the three-dimensional model data includes at least one of a three-dimensional template model and a three-dimensional template point cloud;

[0049] The determining module is specifically used to determine the initial coarse pose transformation matrix based on at least one of the color image, depth image, or first point cloud, and the three-dimensional model data.

[0050] Optionally, the acquisition module is further configured to acquire a three-dimensional modeling model of the target object and fix the three-dimensional modeling model;

[0051] The determining module is further configured to use the initial coarse pose transformation matrix as the extrinsic parameter matrix of the virtual camera;

[0052] The determining module is further configured to determine the optical center of the virtual camera based on the extrinsic parameter matrix of the virtual camera.

[0053] Optionally, the determining module is further configured to determine the beam emission set of the virtual camera based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, wherein the beam emission set includes multiple rays, and the beam direction and beam path corresponding to each ray;

[0054] The device further includes: a control module;

[0055] The control module is used to control the virtual camera to emit each ray from the optical center of the virtual camera, according to the beam direction and beam path corresponding to each ray;

[0056] The determining module is specifically used to determine the second point cloud based on the three-dimensional modeling model and each of the emitted rays.

[0057] Optionally, the processing module is further configured to triangulate the three-dimensional modeling model to obtain a triangular mesh model, wherein the triangular mesh model includes multiple triangular facets and a facet normal vector corresponding to each triangular facet.

[0058] The determining module is specifically used to determine the second point cloud based on the plurality of triangular facets, the facet normal vector corresponding to each of the triangular facets, and the emitted rays.

[0059] Optionally, the device further includes: a determination module;

[0060] The judgment module is used to determine the spatial intersection of any emitted ray with each of the triangular facets to obtain a set of collision points corresponding to the ray; the set of collision points includes at least one candidate collision point, and each candidate collision point corresponds to a collision depth and the facet normal vector of the triangular facet to which it belongs.

[0061] The processing module is further configured to sort the candidate collision points corresponding to the ray in ascending order according to the collision depth corresponding to the candidate collision points, so as to obtain sorted candidate collision points.

[0062] The processing module is also used to filter the sorted candidate collision points to obtain a target collision point, wherein the collision depth of the target collision point is less than the collision depth of other candidate collision points on the same ray.

[0063] The processing module is further configured to combine the target collision point and the surface normal vector to which the target collision point belongs to determine the second point cloud of the target object.

[0064] Optionally, the determining module is further configured to use the initial coarse pose transformation matrix as the initial pose of the target object;

[0065] The processing module is specifically used to refine the first point cloud and the second point cloud based on the initial pose to obtain the pose.

[0066] Thirdly, this application provides an object pose refinement device for industrial robots, comprising:

[0067] Memory;

[0068] processor;

[0069] The memory stores computer-executed instructions;

[0070] The processor executes computer execution instructions stored in the memory to implement the object pose refinement method for industrial robots as described in the first aspect and various possible implementations of the first aspect above.

[0071] Fourthly, this application provides a computer storage medium storing computer execution instructions thereon, which are executed by a processor to implement the object pose refinement method for industrial robots as described in the first aspect and various possible implementations of the first aspect above.

[0072] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the object pose refinement method for industrial robots as described above.

[0073] The object pose refinement method for industrial robots provided in this application first acquires a depth image of the target object using an image acquisition device. Based on the depth image, color image, and the intrinsic parameter matrix of the image acquisition device, a first point cloud of the target object is determined. Then, an initial coarse pose transformation matrix of the target object is obtained based on the color image, depth image, and / or the first point cloud. A virtual camera is constructed to observe the target object based on this initial coarse pose transformation matrix, and a second point cloud of the target object is generated using the virtual camera. Finally, the initial coarse pose transformation matrix is ​​refined based on the first and second point clouds to obtain the pose of the target object. This method constructs a virtual camera using the initial coarse pose transformation matrix, builds a virtual imaging space using the virtual camera, and generates a second point cloud with a corresponding viewpoint. The second point cloud is then matched with the first point cloud acquired by the image acquisition device to achieve accurate alignment with the real depth data. This eliminates the need for traditional global geometric registration methods that pull the model's back face towards the camera's visible surface, and avoids the problems of staircase artifacts and matching deviations easily generated by conventional rendering alignment methods. Therefore, it solves the problem of accurately locating the workpiece pose in existing technologies. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0075] Figure 1 A schematic diagram of an existing scenario for an object pose refinement method for industrial robots provided in this application;

[0076] Figure 2 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 1 ;

[0077] Figure 3 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 2 ;

[0078] Figure 4 A schematic diagram of point cloud generation by ray projection for an object pose refinement method for industrial robots provided in this application;

[0079] Figure 5 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 3 ;

[0080] Figure 6 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 4 ;

[0081] Figure 7This is a schematic diagram of the object pose refinement device for industrial robots provided in this application.

[0082] Figure 8 This is a structural schematic diagram of the object pose refinement device for industrial robots provided in this application.

[0083] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0085] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0086] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0087] First, the terms used in this application will be explained.

[0088] Pose, also known as 6D pose, is a complete description of the position and orientation of a target object in a three-dimensional coordinate system. It consists of three translational dimensions and three rotational dimensions. The three translational dimensions correspond to the object's position along the X, Y, and Z axes, while the three rotational dimensions correspond to the object's orientation around the X, Y, and Z axes. The combination of these two dimensions uniquely determines the object's spatial state.

[0089] With the rapid development of industrial automation and intelligent manufacturing, disordered grasping operations are gradually becoming a new trend in the interaction between industrial robots and objects. In this process, industrial robots need to possess the ability to perceive objects in order to identify their position and orientation. However, relying solely on preliminary object perception is insufficient to meet the requirements; therefore, further refinement of the pose is necessary.

[0090] In existing technologies, pose refinement is often achieved using the following two methods: The first is global geometric registration, which directly performs ICP registration between the complete 3D model point cloud of the object and the single-view visible surface point cloud acquired by the depth camera; the second is a rendering-based alignment method, which uses depth buffering technology to render the 3D model into a virtual depth map, back-projects the virtual depth map into a virtual point cloud, and then matches and registers it with the real point cloud acquired by the depth camera.

[0091] However, both of the above methods have their own drawbacks: the first global geometric registration algorithm incorrectly pulls the back face of the model toward the visible surface of the camera, resulting in a large translation error in the depth direction (Z-axis) and a systematic deviation in the rotational attitude, making it difficult to meet the requirements of high accuracy; the second method based on rendering alignment not only produces a stepped quantization error when back-projecting virtual data back to the point cloud, but also fails to obtain continuous and high-precision surface normal vectors, which makes the point-to-surface registration process converge slowly and even easily get trapped in a local optimum, thus failing to guarantee high accuracy.

[0092] For example, this application provides a schematic diagram of an existing scenario for an object pose refinement method for industrial robots, such as... Figure 1 As shown, the diagram includes: the left image is a complete 3D model point cloud of the target object, which can present the complete geometric structure of the target object; the middle image is a single-view local observation point cloud (2.5D point cloud) acquired by a depth camera, which only presents the visible surface information of the target object and cannot obtain the back view data of the target object; the right image is a schematic diagram of iterative nearest point registration (ICP registration) directly using the complete 3D model point cloud and the single-view point cloud. This method will cause translational errors and attitude shifts along the depth direction (Z-axis) due to the incorrect matching between the back view points of the model and the single-view points.

[0093] To address the aforementioned problems, this application provides a method for refining the pose of industrial robots. First, a depth image, a color image, and the corresponding intrinsic parameter matrix of the target object are acquired by an image acquisition device. Based on the depth image, color image, and intrinsic parameter matrix, a first point cloud of the target object is calculated. Then, based on the color image, depth image, and / or the first point cloud, an initial coarse pose transformation matrix of the target object is determined. Subsequently, a virtual camera is constructed to observe the target object using the initial coarse pose transformation matrix. A second point cloud reflecting the truly visible area of ​​the target object is generated through the virtual camera. Finally, based on the first and second point clouds, the initial coarse pose transformation matrix is ​​refined to obtain the pose of the target object. This method, by introducing a virtual camera to construct a virtual physical imaging space, directly generates a high-fidelity locally visible point cloud consistent with the real scene, avoiding the discrete errors and depth direction offsets caused by traditional registration, and effectively solving the problem of accurately locating the workpiece pose in existing technologies.

[0094] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0095] Figure 2 A flowchart illustrating the object pose refinement method for industrial robots provided in this embodiment. Figure 1 The execution subject in this application embodiment can be, for example, an industrial robot, such as... Figure 2 As shown, the object pose refinement method for industrial robots provided in this embodiment includes:

[0096] S101: Acquire depth and color images of the target object through an image acquisition device.

[0097] Among them, the image acquisition device refers to an industrial camera with depth perception and color imaging capabilities, such as an RGB-D depth camera, a 3D structured light camera, a laser depth camera, a binocular stereo camera, or a Time-of-Flight (TOF) camera.

[0098] A color image is a two-dimensional image directly acquired by an image acquisition device, which can reflect the color, texture, or appearance characteristics of a target object.

[0099] Depth images are used to reflect the distance values ​​between each pixel on the surface of a target object and the image acquisition device. The larger the distance value of a pixel, the farther away the point is from the image acquisition device; the smaller the distance value, the closer the point is to the image acquisition device.

[0100] Understandably, by acquiring images of a target object through an image acquisition device, corresponding depth images and color images can be obtained; the depth image is used to reflect the actual distance from the surface of the target object to the image acquisition device, and the color image is used to reflect the color, texture or appearance features of the target object.

[0101] S102: Determine the first point cloud of the target object based on the depth image, color image, and the intrinsic parameter matrix of the image acquisition device.

[0102] The first point cloud is a set of three-dimensional points calculated based on depth and color images and the intrinsic parameter matrix of the image acquisition device. It can reflect the spatial shape and positional distribution of the target object from the perspective of the image acquisition device.

[0103] The intrinsic parameter matrix consists of inherent parameters that have been calibrated at the factory of the image acquisition device, including but not limited to focal length and optical center coordinates.

[0104] In this step, for example, the intrinsic parameter matrix of the image acquisition device can be obtained first, and then the first point cloud of the target object can be determined based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device.

[0105] The intrinsic parameter matrix can be obtained, for example, from a cloud database associated with the industrial robot, from the industrial robot's local parameter database, or by reading the driver parameters of the image acquisition device.

[0106] The purpose of this step is to obtain three-dimensional point cloud data of the target object from the perspective of the image acquisition device.

[0107] S103: Determine the initial coarse pose transformation matrix of the target object based on the color image, depth image, and / or first point cloud.

[0108] The initial coarse pose transformation matrix is ​​a transformation matrix used to initially describe the position and orientation of the target object in the three-dimensional space where the image acquisition device is located.

[0109] Understandably, color images can reflect the color, texture, and appearance features of a target object, depth images can reflect the distance information between various points on the target object's surface and the acquisition device, and the first point cloud can reflect the three-dimensional spatial structure information of the target object. Therefore, by using any one or more combinations of the above information for calculation, the initial coarse pose transformation matrix of the target object can be obtained.

[0110] S104: Based on the initial coarse pose transformation matrix, construct a virtual camera for the target object, and based on the virtual camera, determine the second point cloud of the target object.

[0111] Among them, the virtual camera is a virtual imaging unit constructed based on an initial coarse pose transformation matrix in the industrial robot's working environment, which simulates the imaging principle of a real image acquisition device.

[0112] The second point cloud is used to reflect the spatial shape and positional distribution of the target object from the perspective of the virtual camera.

[0113] The purpose of this step is to obtain 3D point cloud data of the target object from the perspective of a virtual camera.

[0114] Understandably, since the initial coarse pose transformation matrix has preliminarily determined the approximate spatial position and orientation of the target object, constructing a virtual camera through this matrix can keep the observation angle of the virtual camera consistent with that of the image acquisition device, thereby generating three-dimensional point cloud data under the corresponding angle, providing a valid reference for subsequent registration and comparison with the first point cloud acquired in reality.

[0115] S105: Based on the first and second point clouds, the initial coarse pose transformation matrix is ​​refined to obtain the pose of the target object.

[0116] The pose of the target object is used to fully describe its spatial position and orientation in three-dimensional space. It consists of translational components along the three coordinate axes X, Y, and Z, as well as rotational components around the three coordinate axes, which can uniquely determine the object's placement and specific position in space.

[0117] The purpose of this step is to obtain a precise and complete target object pose by registering and optimizing the first point cloud obtained from actual data and the second point cloud generated by simulation, based on the initial coarse pose, so as to provide the final pose basis for the industrial robot to perform precise grasping operations.

[0118] This embodiment provides a method for refining the object pose of industrial robots. First, a depth image and a color image of the target object are acquired using an image acquisition device. Next, a first point cloud of the target object is determined based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device. Then, an initial coarse pose transformation matrix of the target object is calculated based on the color image, the depth image, and / or the first point cloud. Based on this initial coarse pose transformation matrix, a virtual camera for the target object is constructed, and a second point cloud of the target object is generated using this virtual camera. Finally, based on the first and second point clouds, the initial coarse pose transformation matrix is ​​refined to obtain the pose of the target object. This method improves the refinement accuracy of the object pose, solves the problem of accurately locating the workpiece pose in existing technologies, and provides more reliable technical support for industrial automation and intelligent manufacturing.

[0119] Figure 3 A flowchart illustrating the object pose refinement method for industrial robots provided in this embodiment. Figure 2 .like Figure 3 As shown. This embodiment is in Figure 2 Based on the embodiments, the implementation process of object pose refinement for industrial robots is described in detail. The object pose refinement method for industrial robots provided in this embodiment includes:

[0120] S201: Acquire depth and color images of the target object using an image acquisition device.

[0121] The explanation of step S201 is the same as that in the above embodiments, and will not be repeated here.

[0122] S202: Perform masking on the color image to obtain a masked image.

[0123] Among them, a mask image is a binary or labeled image used to distinguish between target objects and background regions. It marks the regions in a color image that belong to the target object and the background region by different pixel values, thereby achieving effective segmentation and localization of the target object.

[0124] The purpose of this step is to accurately segment the target object region in the color image.

[0125] Understandably, in masking color images, a threshold segmentation method based on color features can be used to perform pixel-level filtering of the color image to distinguish the target object from the background and obtain a mask image; a deep learning semantic segmentation model can be used to directly output the mask image corresponding to the target object; or edge detection and contour extraction algorithms can be used to locate the target contour and fill the internal region to generate a mask image.

[0126] S203: Multiply the depth image and the mask image to obtain the mask depth image.

[0127] The mask depth image is a depth image obtained by multiplying the original depth image and the mask image by corresponding pixels. It retains only the depth values ​​of the area where the target object is located and removes values ​​in the depth image that are irrelevant to the target object.

[0128] The purpose of this step is to remove data that is irrelevant to the target object from the depth image, retaining only the mask depth information corresponding to the target object.

[0129] Understandably, when multiplying a depth image with a mask image, one can, for example, use the mask image as a weight matrix and multiply it with the depth values ​​at corresponding positions in the depth image to extract the mask depth image corresponding to the target object; or one can perform a pixel-by-pixel multiplication operation on the two images, retain the depth values ​​of the target region and set irrelevant background data to zero, thereby obtaining the mask depth image.

[0130] S204: Perform 3D backprojection processing on the mask depth image and intrinsic parameter matrix to obtain the first point cloud.

[0131] The purpose of this step is to convert the masked and purified depth data into a three-dimensional spatial point set, thereby obtaining a first point cloud that is free from background interference and only represents the shape of the target object from the perspective of the image acquisition device.

[0132] Understandably, in the process of performing 3D backprojection processing on the mask depth image and the intrinsic parameter matrix, for example, a projection model can be established based on the intrinsic parameter matrix, and the depth value of the mask depth image can be back-mapped to 3D spatial coordinates pixel by pixel to generate the first point cloud. Alternatively, the pixels and depth values ​​of the mask depth image can be combined with the intrinsic parameter matrix to perform homogeneous coordinate transformation to obtain the first point cloud. Or, based on the principle of ray projection, the projection direction can be determined by the intrinsic parameter matrix, and each effective depth pixel can be extended along the corresponding ray to the corresponding spatial position to form the first point cloud.

[0133] S205: Obtain the three-dimensional model data of the target object, wherein the three-dimensional model data includes at least one of a three-dimensional template model and a three-dimensional template point cloud.

[0134] Among them, the three-dimensional template model is used to represent the standardized three-dimensional geometric model of the target object.

[0135] A 3D template point cloud is a set of 3D points sampled from a 3D template model, used to represent the standard spatial shape of a target object.

[0136] The purpose of this step is to obtain standard three-dimensional reference data for the target object.

[0137] This step involves obtaining the 3D model data of the target object, for example, by calling a 3D model library interface or by reading a local storage file. This application does not impose any special restrictions on this.

[0138] S206: Determine the initial coarse pose transformation matrix based on at least one of a color image, a depth image, or a first point cloud, and 3D model data.

[0139] Understandably, the pose of the target object in the actual scene changes in real time, while the color image, depth image and first point cloud are all real-time acquired data, and the 3D model data is a pre-standardized and fixed template.

[0140] Therefore, the three-dimensional model data can be used as a reference benchmark, and at least one type of data from the color image, depth image, and first point cloud can be selected to match it, thereby obtaining the spatial position and shape information of the target object.

[0141] S207: Obtain the 3D model of the target object and fix the 3D model.

[0142] Among them, the 3D modeling model is a digital model used to describe the standard geometric shape and spatial structure of the target object.

[0143] Understandably, by acquiring and fixing the 3D model, the spatial position of the model can be locked, allowing the 3D model to establish its own object coordinate system based on the target object itself. The 3D model is then firmly placed at the origin of the object coordinate system, maintaining the model's shape and relative spatial position constant, and completing spatial anchoring based on the target object itself.

[0144] The method for obtaining the 3D model of the target object in this step can be, for example, from a local database, from the cloud, or by staff manually entering and parsing the modeling-related data on the corresponding operating platform. This application does not impose any special restrictions on this.

[0145] S208: Use the initial coarse pose transformation matrix as the extrinsic parameter matrix of the virtual camera. The virtual camera includes: optical center and extrinsic parameter matrix.

[0146] Among them, the extrinsic parameter matrix is ​​used to characterize the spatial position and attitude of the virtual camera relative to the target object, and can determine the spatial orientation of the virtual camera and the shooting observation angle.

[0147] The purpose of this step is to directly set the initial coarse pose transformation matrix as the extrinsic parameter matrix corresponding to the virtual camera, complete the preliminary parameter configuration of the virtual camera's imaging space pose and position, and determine the overall spatial placement and relative transformation relationship of the virtual camera.

[0148] Understandably, the initial coarse pose transformation matrix is ​​calculated based on the color image, depth image, or first point cloud data acquired by the image acquisition device, and can reflect the coarse spatial position and orientation of the image acquisition device relative to the target object.

[0149] Therefore, by configuring the initial coarse pose transformation matrix to the virtual camera while keeping the 3D model fixed, the position and shooting orientation of the virtual camera in the 3D model space can be basically consistent with the image acquisition device, thus achieving a preliminary matching and alignment between the virtual observation perspective and the actual shooting perspective.

[0150] S209: Determine the optical center of the virtual camera based on the extrinsic parameter matrix of the virtual camera.

[0151] The optical center is the projection center point inside the virtual camera, representing the observation position of the virtual viewpoint. All observation rays are emitted outward from this point.

[0152] The purpose of this step is to determine the spatial coordinates of the optical center of the virtual camera based on the configured virtual camera extrinsic parameter matrix.

[0153] Understandably, the extrinsic parameter matrix of a virtual camera can only reflect the spatial transformation relationship between the virtual camera and the target object, and cannot directly determine the specific coordinates of the optical center. Therefore, by using the extrinsic parameter matrix of the virtual camera as a resolving factor to determine the optical center of the virtual camera, all spatial composition parameters of the virtual camera can be completed, providing a reliable data basis for the subsequent generation of the second point cloud.

[0154] In this step, for example, the spatial coordinates of the camera coordinate system origin in the object coordinate system can be obtained by performing an inverse transformation on the extrinsic parameter matrix, and this coordinate can be used as the optical center position of the virtual camera; alternatively, the translation vector of the virtual camera extrinsic parameter matrix can be used as the origin of the homogeneous coordinate system, and the position of the origin of the homogeneous coordinate system can be defined as the optical center of the virtual camera.

[0155] S210: Determine the second point cloud of the target object based on a virtual camera.

[0156] The virtual camera used in this method can completely replicate the intrinsic parameters, spatial pose, and shooting angle of a real image acquisition device. It can simulate the areas that the image acquisition device can and cannot see, achieving equivalent simulation of the visible range. This solves the topological mismatch problem that occurs when registering a single-view 2D semi-real point cloud with a complete 3D model in a global geometric registration algorithm. It eliminates pose stretching deviations caused by non-overlapping areas, effectively maintains the geometric consistency of the point cloud registration process, and improves the accuracy of solving the pose of the target object.

[0157] For example, this application provides a schematic diagram of point cloud generation via ray projection for an object pose refinement method for industrial robots, such as... Figure 4 As shown, the process includes: first, determining the 3D model of the target object and fixing it so that it remains stationary; second, constructing a virtual camera, determining the initial coarse pose transformation matrix of the target object as the extrinsic parameter matrix of the virtual camera, and determining the optical center of the virtual camera based on the extrinsic parameter matrix; then, determining multiple rays from the optical center of the virtual camera toward the object based on the extrinsic parameter matrix of the virtual camera and the intrinsic parameter matrix of the image acquisition device; then, calculating the intersection points (target collision points) between the rays and the surface of the target object, and determining the normal vector of the triangular facet where the target collision point is located; finally, combining all intersection points and the corresponding normal vectors to obtain the second point cloud.

[0158] S211: Use the initial coarse pose transformation matrix as the initial pose of the target object.

[0159] The purpose of this step is to provide an initial iterative value that is close to the real pose for the subsequent point cloud registration and pose optimization process, so as to avoid the optimization algorithm starting the search from a random pose.

[0160] Understandably, since the initial coarse pose has roughly reflected the true spatial pose of the target object, the initial coarse pose transformation matrix can be used as the initial pose of the target object, thereby shortening the iteration time of pose optimization and improving the matching success rate and positioning accuracy.

[0161] S212: Based on the initial pose, refine the first point cloud and the second point cloud to obtain the pose.

[0162] The purpose of this step is to continuously adjust the object pose based on the initial coarse pose, so that the first point cloud and the second point cloud have the highest degree of overlap, thereby obtaining the pose.

[0163] Understandably, since the precise surface normal vectors of the corresponding triangular facets can be obtained simultaneously during the generation of the second point cloud, the resulting point cloud can simultaneously carry surface geometric orientation and spatial gradient information.

[0164] Therefore, by using the initial coarse pose as a reference, the second point cloud with accompanying normal vector information can be registered and optimized without performing a large-scale global traversal search, and can quickly converge to the true pose. At the same time, the normal vector is used to complete the iterative solution along the precise surface gradient direction, which effectively avoids the registration algorithm from getting stuck in local optima.

[0165] In this step, for example, based on the initial pose, a normal distribution transformation algorithm can be used to perform registration optimization on the first and second point clouds, and iterative optimization of the pose can be achieved through probability distribution matching to finally obtain the pose of the target object; alternatively, based on the initial pose, a point-to-plane iterative nearest point algorithm can be used to perform registration optimization on the first and second point clouds to obtain the pose of the target object; or, based on the initial pose, a generalized iterative nearest point algorithm can be used to perform registration optimization on the first and second point clouds, combined with point-to-point and point-to-plane constraints for joint optimization to finally obtain the pose of the target object.

[0166] The object pose refinement method for industrial robots provided in this embodiment first acquires a depth image and a color image of the target object through an image acquisition device. The color image is then masked to obtain a mask image. This mask image is multiplied by the depth image to obtain a masked depth image. A first point cloud is then obtained by performing 3D back-projection processing using the intrinsic parameter matrix of the image acquisition device. Next, 3D model data of the target object is acquired. An initial coarse pose transformation matrix is ​​determined based on at least one of the color image, depth image, or the first point cloud, along with the 3D model data. Then, the 3D model of the target object is acquired and fixed. This coarse pose transformation matrix is ​​used as the extrinsic parameter matrix of the virtual camera, and the optical center of the virtual camera is determined. A second point cloud of the target object is obtained based on the virtual camera. Finally, using the initial coarse pose transformation matrix as the initial pose, the first and second point clouds are refined to obtain the pose of the target object.

[0167] This method introduces a virtual camera and constructs a virtual physical imaging space based on the principle of relative motion. It simulates the real acquisition perspective by fixing the target object and adjusting only the pose of the virtual camera. It generates a second point cloud without quantization error by analyzing geometric ray projection. Then, it combines the initial coarse pose transformation matrix, the first point cloud and the second point cloud for registration optimization, and finally outputs the pose of the target object, which solves the problems of high rendering latency and inaccurate pose positioning in the existing technology.

[0168] Figure 5 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 3 .like Figure 5 As shown. This embodiment, based on the above embodiments, provides a detailed explanation of the implementation process for determining the second point cloud of a target object using a virtual camera. The object pose refinement method for industrial robots provided in this embodiment includes:

[0169] S301: Based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, determine the beam emission set of the virtual camera, wherein the beam emission set includes multiple rays, and the beam direction and beam path corresponding to each ray.

[0170] Among them, a ray refers to a projected straight line that originates from the optical center of the virtual camera and extends outward in a specific direction. It is used to simulate the propagation of light and to calculate its intersection with the 3D model.

[0171] The beam direction is the direction in which each ray extends in three-dimensional space. It is calculated by both the intrinsic and extrinsic parameter matrices and determines the propagation angle of the ray.

[0172] The beam path is the complete spatial trajectory of a ray that starts from the optical center of the virtual camera, extends along the beam direction, and may intersect with the 3D model. It is used to determine the intersection point on the model surface.

[0173] Understandably, the intrinsic parameter matrix determines the imaging projection method of the image acquisition device, while the extrinsic parameter matrix determines the position and orientation of the virtual camera in space. Therefore, by comprehensively considering the intrinsic and extrinsic parameter matrices, all rays originating from the virtual optical center and passing through each pixel can be calculated, forming a complete set of ray beams.

[0174] S302: Control the virtual camera to emit each ray from the optical center of the virtual camera, according to the beam direction and beam path corresponding to each ray.

[0175] The purpose of this step is to allow the virtual camera to emit rays from the optical center along the direction and path of each ray, simulating the propagation process of the imaging light from the image acquisition device.

[0176] Understandably, emitting rays from the optical center and strictly following the corresponding direction and path can ensure that the virtual imaging geometry is completely consistent with the image acquisition device, so that the attitude and viewpoint of the second point cloud obtained later strictly correspond to the actual acquired point cloud, laying the foundation for subsequent accurate registration.

[0177] This method fixes the 3D model of the target object at the origin of the object coordinate system throughout the process, so that its position and orientation remain constant. Only the initial coarse pose transformation matrix is ​​set as the virtual camera extrinsic matrix, and combined with the intrinsic matrix of the image acquisition device, rays are emitted from the optical center of the virtual camera into the 3D space in a predetermined direction and path.

[0178] Furthermore, this method employs a relative motion mechanism where the 3D model remains stationary while the virtual camera simulates spatial motion. The 3D model is fixed at the origin of the coordinate system, and ray emission is completed solely based on pre-defined virtual camera extrinsic parameters. This effectively recreates the relative positional transformation between the image acquisition device and the target object in a real scene. The point cloud obtained from ray intersection is directly within the object's coordinate system, completely eliminating the lengthy processing steps of traditional methods that involve converting the model to the camera coordinate system, rendering the image, and then inversely transforming it back to the object coordinate system. It eliminates the need for complex graphics rendering architectures and avoids tedious matrix inversions and multiple point cloud coordinate transformations, effectively reducing computational overhead and suppressing the accumulation of floating-point operation errors. Simultaneously, it ensures that the virtual imaging geometry remains consistent with the real image acquisition device, allowing the subsequently generated second point cloud to accurately correspond to the point cloud data acquired in the field in terms of spatial attitude and observation perspective.

[0179] S303: Determine the second point cloud based on the 3D model and the emitted rays.

[0180] The purpose of this step is to calculate the intersection points of each ray with the 3D model, and then collect these intersection points to form a virtual observation point cloud corresponding to the current pose.

[0181] Optionally, this application provides a possible implementation method, including:

[0182] The first step is to triangulate the 3D model to obtain a triangular mesh model. The triangular mesh model includes multiple triangular faces and the face normal vector corresponding to each triangular face.

[0183] Among them, triangular facets are the basic triangular units that make up the surface of a three-dimensional model. Multiple triangular facets are spliced ​​together to represent the curved surface and outline of the target object.

[0184] The normal vector of a facet is a direction vector perpendicular to the plane containing the triangular facet, used to indicate the orientation of the triangular facet.

[0185] The purpose of this step is to break down the 3D model into a series of small triangular facets and calculate the facet normal vector for each facet.

[0186] Understandably, since the 3D model remains fixed and does not shift, triangulation of the 3D model to obtain a triangular mesh model can restore the surface geometry of the target object and the orientation information of each region, providing a reliable basis for the subsequent generation of the second point cloud.

[0187] This method directly triangulates the 3D model and solves for the normal vectors of the facets, avoiding the rendering alignment methods commonly used in existing technologies. Traditional rendering alignment relies on depth buffering technology to render the 3D model to generate a virtual depth map, and then backprojects the virtual depth map to obtain a virtual point cloud, thereby completing the registration operation with the real-world point cloud. Furthermore, this method does not perform image rendering or pixel conversion processing throughout the process, completely eliminating the discrete errors caused by pixel rasterization, and directly obtaining analytical normal vectors in pure 3D geometric space. The obtained normal vectors can represent the spatial orientation of the corresponding triangular facets.

[0188] In this step, for example, a surface triangulation method can be used. First, the surface curvature of the 3D model is divided into different regions. Then, vertices are placed on the surface of each region and connected to generate triangular patches. At the same time, the corresponding patch normal vectors are calculated to finally form a triangular mesh model. Alternatively, a layer-by-layer triangulation method can be used. Triangular patches are divided sequentially from the top layer to the bottom layer of the 3D model, and the patch normal vectors corresponding to each patch are calculated simultaneously. A vertex clustering triangulation method can also be used. First, densely distributed vertices on the model are clustered and merged. Then, triangular patches are generated by connecting the clustered key points, and the corresponding patch normal vectors are calculated.

[0189] The second step is to determine the second point cloud based on multiple triangular facets, the facet normal vector corresponding to each triangular facet, and the emitted rays.

[0190] Understandably, firstly, the multiple triangular facets and their corresponding facet normals are all obtained by triangulation of a fixed-position 3D model. This 3D model is fixed at the origin of the object's coordinate system and does not shift throughout the process, so its overall position remains unchanged. Secondly, the emitted observation rays are obtained by a virtual camera with a determined optical center.

[0191] Therefore, by utilizing triangular facets, facet normals, and the emitted rays, the intersection of each ray with a triangular facet is detected. The facet normal is then used to determine whether the ray effectively penetrates the triangular facet. All valid intersection points are then aggregated to generate the second point cloud. Because the 3D model remains fixed and without displacement, the resulting second point cloud is naturally situated in the object's coordinate system, eliminating the need for any inverse coordinate transformation calculations.

[0192] The object pose refinement method for industrial robots provided in this embodiment determines a virtual camera beam emission set consisting of multiple rays based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, and clarifies the beam direction and path corresponding to each ray. Then, the virtual camera is controlled to emit rays from its own optical center according to the direction and path of each ray. Finally, the second point cloud is determined by combining the interaction results between the 3D model of the target object and the ray projection. This method, by constructing a standardized ray projection system based on intrinsic and extrinsic parameters, achieves accurate simulation of the real imaging process in virtual space. It can efficiently generate point clouds containing only the visible area of ​​the target object, providing a reliable data foundation for subsequent point cloud registration and pose refinement.

[0193] Figure 6 A flowchart illustrating the object pose refinement method for industrial robots provided in this application. Figure 4 .like Figure 6 As shown. This embodiment, based on the above embodiments, provides a detailed explanation of the process for determining the second point cloud based on multiple triangular facets, the facet normal vector corresponding to each triangular facet, and the emitted rays. The object pose refinement method for industrial robots provided in this embodiment includes:

[0194] S401: For any emitted ray, determine the spatial intersection of the ray with each triangular facet to obtain the set of collision points corresponding to the ray; the set of collision points includes at least one candidate collision point, and each candidate collision point corresponds to a collision depth and the facet normal vector of the triangular facet to which it belongs.

[0195] The collision depth refers to the straight-line distance from the optical center of the virtual camera along the ray direction to the point where the ray intersects with the triangular facet. It is used to indicate the distance of the candidate collision point relative to the observation point. For example, if a ray originates from the optical center of the virtual camera, travels 50 mm in space, intersects the first triangular facet, and intersects the second triangular facet at 120 mm, then the collision depths of these two candidate collision points are 50 mm and 120 mm, respectively.

[0196] The purpose of this step is to perform spatial intersection detection on each emitted ray, find its intersection positions with all triangular facets, and form a set of candidate collision points containing the collision depth and the facet normal vector.

[0197] Understandably, for any ray emitted from the virtual camera, all the triangular faces of the model are traversed sequentially to determine whether the ray intersects with the triangular face. Each intersection position is recorded as a candidate collision point, and the distance from the point to the camera and the orientation of the corresponding face are marked. Finally, all intersection points are summarized into a collision point set for the ray.

[0198] S402: Sort the candidate collision points corresponding to the ray in ascending order according to the collision depths of the candidate collision points to obtain the sorted candidate collision points.

[0199] The purpose of this step is to sort all candidate collision points corresponding to the same ray from smallest to largest according to their distance from the virtual camera.

[0200] Understandably, since a ray may pass through multiple triangular facets and generate multiple candidate collision points, in order to distinguish the front and back occlusion relationships of each candidate collision point and facilitate the subsequent selection of the nearest and unoccluded effective collision point, the candidate collision points corresponding to the ray can be sorted in ascending order according to the collision depth corresponding to the candidate collision point.

[0201] S403: Filter the sorted candidate collision points to obtain the target collision point. The collision depth of the target collision point is less than the collision depth of other candidate collision points on the same ray.

[0202] The purpose of this step is to select the candidate collision point closest to the virtual camera from multiple candidate collision points corresponding to the same ray, and use it as the target collision point.

[0203] This is understandable, because although a ray may intersect with multiple triangular facets, only the collision point closest to the camera and not obstructed can represent the visible surface of the target object.

[0204] Therefore, by selecting the point with the smallest collision depth, we can ensure that the obtained point is a real and visible valid collision point, and avoid using occluded invalid collision points.

[0205] S404: Combine the target collision point and the normal vector of the surface to which the target collision point belongs to determine the second point cloud of the target object.

[0206] Understandably, the target collision point is the precise location where the ray intersects the object's surface, directly corresponding to the object's surface coordinates in three-dimensional space, and can be used to represent the spatial position of the object's surface; the facet normal vector is the perpendicular direction of the triangular facet at that intersection point, reflecting the tilt and orientation of the object's surface at that point, and can be used to represent the surface orientation at that location.

[0207] Therefore, by combining the target collision point corresponding to each ray and the normal vector of its corresponding surface, the second point cloud of the target object can be obtained, which provides support for subsequent accurate registration with the first point cloud collected in reality, thereby improving the accuracy of pose calculation.

[0208] The object pose refinement method for industrial robots provided in this embodiment first performs spatial intersection analysis on any ray emitted by a virtual camera with each triangular facet of the 3D model to obtain a set of collision points containing candidate collision points, corresponding collision depths, and facet normal vectors. Then, the candidate collision points are sorted in ascending order by collision depth, and the target collision point with the smallest depth is selected. Finally, the target collision point and its corresponding facet normal vector are combined to determine the second point cloud of the target object. This method accurately extracts point cloud data from the visible surface of the model by precisely finding the intersection of rays and triangular facets and selecting effective collision points by depth. It effectively eliminates occlusion and redundant points, providing a high-precision, high-purity local point cloud for subsequent pose refinement, significantly improving registration accuracy and robustness.

[0209] Figure 7 This is a schematic diagram of the object pose refinement device for industrial robots provided in this application. Figure 7 As shown, this application provides an object pose refinement device for industrial robots. The object pose refinement device 500 for industrial robots includes:

[0210] The acquisition module 501 is used to acquire depth images and color images of the target object through the image acquisition device;

[0211] The determination module 502 is used to determine the first point cloud of the target object based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device.

[0212] The determination module 502 is also used to determine the initial coarse pose transformation matrix of the target object based on the color image, the depth image and / or the first point cloud;

[0213] Module 503 is used to construct a virtual camera for the target object based on the initial coarse pose transformation matrix;

[0214] The determination module 502 is also used to determine the second point cloud of the target object based on the virtual camera;

[0215] The processing module 504 is used to refine the initial coarse pose transformation matrix based on the first point cloud and the second point cloud to obtain the pose of the target object.

[0216] Optionally, the processing module 504 is also used to perform masking processing on the color image to obtain a mask image;

[0217] The processing module 504 is also used to multiply the depth image and the mask image to obtain the mask depth image;

[0218] The processing module 504 is specifically used to perform three-dimensional back projection processing on the mask depth image and the intrinsic parameter matrix to obtain the first point cloud.

[0219] Optionally, the acquisition module 501 is also used to acquire the three-dimensional model data of the target object, wherein the three-dimensional model data includes at least one of a three-dimensional template model and a three-dimensional template point cloud;

[0220] The determination module 502 is specifically used to determine the initial coarse pose transformation matrix based on at least one of a color image, a depth image, or a first point cloud, and three-dimensional model data.

[0221] Optionally, the acquisition module 501 is also used to acquire the three-dimensional modeling model of the target object and fix the three-dimensional modeling model;

[0222] The determination module 502 is also used to use the initial coarse pose transformation matrix as the extrinsic parameter matrix of the virtual camera;

[0223] The determination module 502 is also used to determine the optical center of the virtual camera based on the extrinsic parameter matrix of the virtual camera.

[0224] Optionally, the determining module 502 is further configured to determine the beam emission set of the virtual camera based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, wherein the beam emission set includes multiple rays, and the beam direction and beam path corresponding to each ray;

[0225] The device also includes: a control module 505;

[0226] The control module 505 is used to control the virtual camera to emit each ray from the optical center of the virtual camera, according to the beam direction and beam path corresponding to each ray;

[0227] The determination module 502 is specifically used to determine the second point cloud based on the 3D model and the emitted rays.

[0228] Optionally, the processing module 504 is also used to triangulate the 3D model to obtain a triangular mesh model, which includes multiple triangular faces and a face normal vector corresponding to each triangular face.

[0229] The determination module 502 is specifically used to determine the second point cloud based on multiple triangular facets, the facet normal vector corresponding to each triangular facet, and the emitted rays.

[0230] Optionally, the device may also include: a judgment module 506;

[0231] The judgment module 506 is used to determine the spatial intersection of any emitted ray with each triangular facet to obtain the collision point set corresponding to the ray. The collision point set includes at least one candidate collision point, and each candidate collision point corresponds to a collision depth and the facet normal vector of the triangular facet to which it belongs.

[0232] The processing module 504 is also used to sort the candidate collision points corresponding to the ray in ascending order according to the collision depth corresponding to the candidate collision points, so as to obtain the sorted candidate collision points.

[0233] The processing module 504 is also used to filter the sorted candidate collision points to obtain the target collision point, and the collision depth of the target collision point is less than the collision depth of other candidate collision points on the same ray.

[0234] The processing module 504 is also used to combine the target collision point and the normal vector of the surface to which the target collision point belongs to determine the second point cloud of the target object.

[0235] Optionally, the determining module 502 is also used to use the initial coarse pose transformation matrix as the initial pose of the target object;

[0236] The processing module 504 is specifically used to refine the first point cloud and the second point cloud based on the initial pose to obtain the pose.

[0237] Figure 8 This is a structural schematic diagram of the object pose refinement device for industrial robots provided in this application. Figure 8 As shown, this application provides an object pose refinement device for industrial robots. The object pose refinement device 600 for industrial robots includes: a receiver 601, a transmitter 602, a processor 603, and a memory 604.

[0238] Receiver 601 is used to receive instructions and data;

[0239] Transmitter 602 is used to send commands and data;

[0240] Memory 604 is used to store instructions executed by the computer;

[0241] The processor 603 is used to execute computer execution instructions stored in the memory 604 to implement the various steps of the object pose refinement method for industrial robots in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the object pose refinement method for industrial robots.

[0242] Optionally, the memory 604 can be either standalone or integrated with the processor 603.

[0243] When the memory 604 is set up independently, the electronic device also includes a bus for connecting the memory 604 and the processor 603.

[0244] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the object pose refinement method for industrial robots as described above, executed by the object pose refinement device for industrial robots.

[0245] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0246] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for refining the pose of an object for industrial robots, characterized in that, include: The depth and color images of the target object are acquired using an image acquisition device; Based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device, the first point cloud of the target object is determined; The initial coarse pose transformation matrix of the target object is determined based on the color image, the depth image, and / or the first point cloud. Based on the initial coarse pose transformation matrix, a virtual camera for the target object is constructed, and based on the virtual camera, the second point cloud of the target object is determined. Based on the first point cloud and the second point cloud, the initial coarse pose transformation matrix is ​​refined to obtain the pose of the target object.

2. The method according to claim 1, characterized in that, Determining the first point cloud of the target object based on the depth image, the color image, and the intrinsic parameter matrix of the image acquisition device includes: The color image is masked to obtain a masked image; The depth image and the mask image are multiplied to obtain the mask depth image; The first point cloud is obtained by performing three-dimensional back projection processing on the mask depth image and the intrinsic parameter matrix.

3. The method according to claim 1, characterized in that, Determining the initial coarse pose transformation matrix of the target object based on the color image, depth image, and / or the first point cloud includes: Obtain the three-dimensional model data of the target object, wherein the three-dimensional model data includes at least one of a three-dimensional template model and a three-dimensional template point cloud; The initial coarse pose transformation matrix is ​​determined based on at least one of the color image, depth image, or first point cloud, and the three-dimensional model data.

4. The method according to claim 1, characterized in that, The virtual camera includes an optical center and an extrinsic parameter matrix. The construction of the virtual camera for the target object based on the initial coarse pose transformation matrix includes: Obtain a 3D model of the target object and fix the 3D model; The initial coarse pose transformation matrix is ​​used as the extrinsic parameter matrix of the virtual camera. The optical center of the virtual camera is determined based on the extrinsic parameter matrix of the virtual camera.

5. The method according to claim 4, characterized in that, The step of determining the second point cloud of the target object based on the virtual camera includes: Based on the intrinsic parameter matrix of the image acquisition device and the extrinsic parameter matrix of the virtual camera, the beam emission set of the virtual camera is determined, wherein the beam emission set includes multiple rays, and the beam direction and beam path corresponding to each ray; The virtual camera is controlled to emit each ray from its optical center, according to the beam direction and beam path corresponding to each ray; The second point cloud is determined based on the three-dimensional model and the emitted rays.

6. The method according to claim 5, characterized in that, Determining the second point cloud based on the three-dimensional modeling model and the emitted rays includes: The three-dimensional modeling model is triangulated to obtain a triangular mesh model, which includes multiple triangular facets and a facet normal vector corresponding to each triangular facet. The second point cloud is determined based on the plurality of said triangular facets, the facet normal vector corresponding to each of said triangular facets, and each of said emitted rays.

7. The method according to claim 6, characterized in that, Determining the second point cloud based on the plurality of triangular facets, the facet normal vector corresponding to each triangular facet, and the emitted rays includes: For any emitted ray, spatial intersection of the ray with each of the triangular facets is determined to obtain a set of collision points corresponding to the ray; the set of collision points includes at least one candidate collision point, and each candidate collision point corresponds to a collision depth and the facet normal vector of the triangular facet to which it belongs; According to the collision depth corresponding to the candidate collision point, the candidate collision points corresponding to the ray are sorted in ascending order to obtain the sorted candidate collision points. The sorted candidate collision points are filtered to obtain the target collision point, and the collision depth of the target collision point is less than the collision depth of other candidate collision points on the same ray. The target collision point and the normal vector of the surface to which the target collision point belongs are combined to determine the second point cloud of the target object.

8. The method according to claim 1, characterized in that, The step of refining the initial coarse pose transformation matrix based on the first point cloud and the second point cloud to obtain the pose of the target object includes: The initial coarse pose transformation matrix is ​​used as the initial pose of the target object; Based on the initial pose, the first point cloud and the second point cloud are refined to obtain the pose.

9. An object pose refinement device for industrial robots, characterized in that, include: Memory; processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the object pose refinement method for industrial robots as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the object pose refinement method for industrial robots as described in any one of claims 1-8.

Images