Mechanical arm rapid three-dimensional reconstruction and grabbing method and system based on Nerf

Abstract

The invention discloses a mechanical arm rapid three-dimensional reconstruction and grabbing method and system based on Nerf. The method comprises the following steps that a working area image is obtained; performing object three-dimensional reconstruction according to the image; estimating the pose of the object; and according to the current pose of the object, the movement track of the mechanical arm is planned, and the object is grabbed. According to the method and the system disclosed by the invention, the flexibility and the accuracy of operation are greatly improved, and the processing speed is remarkably improved and the hardware requirement is reduced especially in a scene with a limited view field or complex control.

Description

Technical Field

[0001] The present invention relates to a method and system for rapid three-dimensional reconstruction and grasping of a robotic arm based on Nerf, belonging to the technical field of robotic arms. Background Art

[0002] Traditional single-arm or simple robotic arms mostly rely on preset programs and basic sensor feedback, which makes them perform well in simple or repetitive operation tasks. However, when facing unknown or complex environments, their performance is often limited. In addition, although existing point-cloud-based 3D vision technologies can provide certain spatial data for robotic arms, these data often cannot meet higher-level operation requirements in terms of accuracy and details, such as precision assembly or delicate object manipulation, especially in situations with large changes in lighting conditions or visual occlusion.

[0003] Due to the above reasons, there is a need for more in-depth research on existing robotic arms to solve the above problems. Summary of the Invention

[0004] In order to overcome the above problems, the inventors of the present invention have conducted in-depth research and designed a method for rapid three-dimensional reconstruction and grasping of a robotic arm based on Nerf, including the following steps:

[0005] S1. Obtain an image of the working area.

[0006] S2. Perform three-dimensional reconstruction of the object according to the image.

[0007] S3. Estimate the pose of the object.

[0008] S4. According to the current pose of the object, plan the motion trajectory of the robotic arm and grasp the object.

[0009] In a preferred embodiment, in S2, the three-dimensional reconstruction method includes the following steps:

[0010] S21. Construct a neural radiance field network combining explicit and implicit forms.

[0011] S22. Generate sampling points.

[0012] S23. Based on the sampling points, use the constructed neural radiance field network combining explicit and implicit forms to predict the volume density and color.

[0013] S24. Perform volume rendering to obtain the three-dimensional object.

[0014] In a preferred embodiment, in S21, a voxel grid and a color feature encoding are set in the neural radiance field network combining explicit and implicit forms, and the voxel grid explicitly stores the volume density of the spatial position.

[0015] The volumetric density represents the probability of the presence of matter at that position, and the color feature encoding is used to store color features.

[0016] In a preferred embodiment, the voxel grid is set through the following sub-steps:

[0017] S211. Obtain the ray origin and direction of each pixel according to the size of the input image, the internal parameters of the camera, and the transformation matrix;

[0018] S222. Select the rays at the positions marked as 1 in the segmentation mask, calculate the nearest and farthest coordinates of the intersection points of the pixels and the scene according to the rays, update the coordinates of the bounding box with this, and obtain the region of interest;

[0019] S223. Establish a voxel grid within the region of interest.

[0020] In a preferred embodiment, in S22, a coordinate system transformation is also performed to convert the sampling points from the camera coordinate system to the object coordinate system.

[0021] In a preferred embodiment, in S23, the transformed sampling points (x′ 3D , d′) and the corresponding position encodings are input into the implicit-explicit combined neural radiance field network, and the original volumetric density σ′ and color c of the sampling points are output by the implicit-explicit combined neural radiance field network, expressed as:

[0022] σ′ = interp(x′ 3D , V σ )

[0023] c = D(interp(x′ 3D , V c ), x′ 3D , d)

[0024] where interp() is an interpolation operation, x′ 3D is the position coordinate of the transformed sampling point, d is the line-of-sight direction of the sampling point in the camera coordinate system, V σ is the voxel grid storing volumetric density information, V c is the voxel grid storing color features, and D is the color feature decoder;

[0025] The original volumetric density is activated through an activation function to obtain the volumetric density σ.

[0026] In a preferred embodiment, in S24, the volume rendering is expressed as:

[0027]

[0028] α i = alpha(σi , δ i ) = 1 - exp(-σ i δ i )

[0029]

[0030] δ i = t i+1 - t i

[0031]

[0032] wherein, represents the color of the two - dimensional pixel position u, K is the number of sampling points, T i is the transmittance from the near - plane to the i - th point, representing the probability that the light is not completely absorbed before passing through the i - th point, c i is the color of the i - th sampling point, δ i is the distance between adjacent sampling points, t i is the distance between the i - th sampling point and the origin, represents the mask of the two - dimensional pixel position u, is the estimated value of the three - dimensional point corresponding to the two - dimensional image pixel.

[0033] In a preferred embodiment, the loss function L of the neural radiance field R is set as:

[0034]

[0035] wherein, L c represents the color loss, L M represents the mask loss, L P represents the reprojection loss;

[0036] The color loss is used to train the implicit - explicit combined neural radiance field network by minimizing the squared error between the predicted color and the real image. The mask loss is used to train the implicit - explicit combined neural radiance field network by minimizing the squared error between the predicted segmentation mask by the model and the real mask. The reprojection loss is used to train the implicit - explicit combined neural radiance field network by minimizing the squared error between the 3D center - point projection of the target object and the center - point of the 2D image bounding box.

[0037] In a preferred embodiment, in S3, the loss function L of the pose estimation network P′ is set as:

[0038] L P′ = L M′ + L d′ + Lg′ +L n′

[0039] Among them, L M′ represents the segmentation loss, and L d′ represents the distance loss, and L g′ represents the gradient loss, and L n′ represents the normal vector loss.

[0040] The present invention also discloses a dual-arm vision fusion robotic arm system, including a robotic arm, a camera and a processor arranged on the robotic arm,

[0041] The processor uses the above-mentioned Nerf-based rapid 3D reconstruction and grasping method for robotic arms to plan the motion trajectory of the robotic arm and control the robotic arm to grasp an object.

[0042] The beneficial effects of the present invention include:

[0043] (1) Greatly improves the flexibility and accuracy of operations, especially in scenarios with limited vision or complex manipulations;

[0044] (2) Quickly processes images and reconstructs fine 3D models without sacrificing accuracy, significantly improving the processing speed and reducing the hardware requirements;

[0045] (3) Has low hardware requirements, effectively reduces costs, and improves the adaptability and operation accuracy in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Shows a schematic flow chart of a Nerf-based rapid 3D reconstruction and grasping method for robotic arms according to a preferred embodiment of the present invention;

[0047] Figure 2 Shows the 3D reconstruction result in Example 1;

[0048] Figure 3 Shows the pose estimation result in Example 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be further described in detail below with reference to the drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become more clearly defined.

[0050] The special term "exemplary" here means "serving as an example, embodiment or illustration". Any embodiment described as "exemplary" here does not have to be construed as superior or better than other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0051] The method for rapid 3D reconstruction and grasping of a robotic arm based on Nerf provided by the present invention, as Figure 1 shown, includes the following steps:

[0052] S1. Obtain the working area image;

[0053] S2. Perform 3D reconstruction of the object based on the image;

[0054] S3. Estimate the pose of the object;

[0055] S4. According to the current pose of the object, plan the motion trajectory of the robotic arm to grasp the object.

[0056] In the present invention, by performing 3D reconstruction of the object as a prior basis, high-quality scene information is provided for the subsequent object positioning, object pose estimation, and grasping estimation processes, so that the robotic arm can achieve precise grasping when facing unknown or complex environments.

[0057] In S1, the working area image is obtained by setting up a camera to take pictures.

[0058] Preferably, the camera is set on the robotic arm.

[0059] Preferably, the camera takes pictures of the working area from multiple angles to provide sufficient information for subsequent 3D reconstruction of the object.

[0060] In a preferred embodiment, there are multiple cameras, which divide the entire shooting area into multiple layers, preferably three layers. The shooting between each layer is relatively independent. In the shooting of each layer, the method of circular arc planning is used to define the movement path of the robotic arm.

[0061] This method can ensure smooth and continuous movement within the layer, is suitable for capturing images at specific angles for subsequent effective 3D reconstruction. Circular arc planning provides precise control and can maintain a stable camera pose during shooting, thereby reducing image distortion and improving image quality.

[0062] The movement between layers is achieved by the method of polynomial interpolation. This method can effectively connect the shooting points between different layers and ensure smooth and efficient transition of the robotic arm between each layer. Specifically, using polynomial interpolation can describe the angular change of each joint over time and ensure that there are no sudden speed changes or acceleration mutations during the inter-layer movement, thereby improving the coherence of the movement.

[0063] Through the above method, by using the polynomial interpolation formula, the displacement, velocity, and acceleration of each robotic arm joint at different time points can be calculated to achieve precise control, ensuring clear images can be obtained within each layer, providing high-quality data support for subsequent 3D reconstruction. Such a design not only improves the shooting efficiency but also guarantees the quantity and quality of the required images, thereby enhancing the effect of 3D reconstruction.

[0064] Before S2, preferably, preprocess the image.

[0065] The preprocessing can be one or more of image denoising, contrast enhancement, normalization, and cropping, so as to improve the image quality and ensure the image size consistency, providing more reliable data input for subsequent 3D reconstruction.

[0066] Furthermore, during the preprocessing, obtain the bounding box and segmentation mask of the target object through a visual recognition network.

[0067] The visual recognition network is preferably the YOLOv8 network.

[0068] Although any 3D reconstruction method can be implemented, however, traditional 3D reconstruction methods, such as SFM and MVS methods, not only face problems such as slow speed, inability to process in real time, and limited effects in complex environments, but also have high requirements for hardware, such as stereo cameras and RGB-D cameras, are limited in external environment use, and are easily affected by object surface characteristics.

[0069] In a preferred embodiment, the 3D reconstruction method is based on Neural Radiance Fields (NeRF). NeRF maps points in continuous 3D space to colors and densities through a fully connected neural network, achieving high-precision scene rendering. This method does not rely on traditional depth data acquisition but optimizes the continuous volume scene function to synthesize new views, effectively overcoming many drawbacks in traditional technologies. The hierarchical sampling and blending equation of NeRF enable it to effectively handle transparent objects and occlusion relationships, greatly improving the rendering efficiency and photometric consistency of images, demonstrating great potential for high-quality 3D reconstruction in complex scenes. However, the original NeRF also has some drawbacks, such as high requirements for hardware, large computational volume, slow training and rendering speeds, and inability to meet the real-time requirements.

[0070] In a preferred embodiment, in S2, the 3D reconstruction method includes the following steps:

[0071] S21. Construct a neural radiance field network that combines explicit and implicit forms;

[0072] S22. Generate sampling points;

[0073] S23. Based on the sampling points, use the constructed neural radiance field network combining explicit and implicit methods to predict the volume density σ and color c;

[0074] S24. Perform volume rendering to obtain the three-dimensional object.

[0075] In S21, different from the traditional neural radiance field network, the neural radiance field network combining explicit and implicit methods is provided with a voxel grid and color feature encoding on the basis of the traditional neural radiance field network. The voxel grid explicitly stores the volume density of the spatial position.

[0076] Among them, the volume density represents the probability of the existence of matter at this position.

[0077] The color feature encoding is used to store color features, which can reduce the need to predict colors through the deep network during each query and speed up the rendering speed.

[0078] Such a design not only maintains the flexibility and high rendering quality of NeRF, but also increases the computational efficiency, and the volume density value can be quickly obtained through linear interpolation.

[0079] Furthermore, the voxel grid is set through the following sub-steps:

[0080] S211. Obtain the starting point and direction of each pixel ray according to the size of the input image, the internal parameters of the camera, and the transformation matrix;

[0081] S222. Select the rays at the positions marked as 1 in the segmentation mask, calculate the nearest and farthest coordinates of the intersection points of the pixels and the scene according to the rays, and update the coordinates of the bounding box to obtain the region of interest;

[0082] S223. Establish a voxel grid within the region of interest.

[0083] The voxel grid is expressed as:

[0084]

[0085] Among them, (n x , n y , n z ) is the number of voxels divided, s voxel is the size of the voxel, n voxel is the total number of voxels, (x max , y max , z max ) represents the maximum boundary coordinates of the region of interest, and (x min , y min , z min ) represents the minimum boundary coordinates of the region of interest.

[0086] Further, the color feature decoder in the explicit-implicit combined neural radiance field network consists of 3 hidden layers. Preferably, 512, 256, and 128 neurons are respectively set.

[0087] In S22, during sampling, centered on the object, uniform sampling is performed near the object to generate N sampling points.

[0088] Further, the sampling interval is (|d 3D |-s, |d 3D |+s), where |d 3D | is the distance from the object center to the camera center, and s is a value greater than the object scale, which can be freely set by those skilled in the art according to needs.

[0089] Preferably, the distance |d 3D | from the object center to the camera center is expressed as:

[0090] |d 3D | = ||t 3D -C||

[0091] t 3D = K -1 (t 2D , z)

[0092] where t 3D is the 3D coordinate of the object center, C is the coordinate of the camera center, t 2D is the 2D coordinate of the object center obtained according to the target bounding box, z is the depth value, and K -1 is the inverse matrix of the camera intrinsic matrix.

[0093] Further, the 3D coordinate x 3D is obtained by mapping the 2D pixel coordinate x 2D in the input image to 3D space, and is expressed as:

[0094] x 3D = K -1 (x 2D , z)

[0095] where K -1 is the inverse matrix of the camera intrinsic matrix, and z is the depth value, representing the distance along the line of sight.

[0096] Further, in S22, coordinate transformation is also performed to convert the sampling points from the camera coordinate system to the object coordinate system.

[0097] Specifically, the conversion from the camera coordinate system to the object coordinate system is achieved through the conversion matrix , and the conversion matrix is expressed as:

[0098]

[0099] Among them, is the captured image I i relative to the captured image I 1 is the relative pose of the camera when capturing the image I, is the transformation matrix from the camera coordinate system to the object coordinate system when capturing the image I 1 at the time of capturing the image I.

[0100] In S23, based on the sampling points, the constructed implicit-explicit combined neural radiance field network is used to predict the volume density σ and the color c;

[0101] The transformed sampling points (x′ 3D , d′) and the corresponding position encodings are input into the implicit-explicit combined neural radiance field network, and the original volume density σ′ and color c of the sampling points are output by the implicit-explicit combined neural radiance field network, expressed as:

[0102] σ′ = interp(x′ 3D , V σ )

[0103] c = D(interp(x′ 3D , V c ), x′ 3D , d)

[0104] Among them, interp() is the interpolation operation, x′ 3D is the position coordinate of the transformed sampling point, d is the line-of-sight direction of the sampling point in the camera coordinate system, V σ is the voxel grid storing volume density information, V c is the voxel grid storing color features, and D is the color feature decoder.

[0105] Furthermore, the original volume density is activated through an activation function to obtain the volume density σ, expressed as:

[0106] σ = softplus(σ′) = log(1 + exp(σ′ + b)

[0107] Among them, softplus() is the activation function and b is the bias term.

[0108] In S24, the volume rendering is expressed as:

[0109]

[0110] α i = alpha(σ i , δ i) = 1 - exp(-σ i δ i )

[0111]

[0112] δ i = t u+1 - t i

[0113]

[0114] wherein, represents the color of the two-dimensional pixel position u, K is the number of sampling points, T i is the transmittance from the near plane to the i-th point, indicating the probability that the light is not completely absorbed before passing through the i-th point, c i is the color of the i-th sampling point, δ i is the distance between adjacent sampling points, t i is the distance between the i-th sampling point and the origin, represents the mask of the two-dimensional pixel position u, is the estimated value of the three-dimensional point corresponding to the two-dimensional image pixel.

[0115] Preferably, the loss function L of the neural radiance field R is set to:

[0116]

[0117] wherein, L c represents the color loss, L M represents the mask loss, L P represents the reprojection loss.

[0118] The color loss trains the implicit-explicit combined neural radiance field network by minimizing the squared error between the predicted color and the real image, making the predicted color of the target object as close as possible to the real color C(u) of the target object.

[0119] Preferably, the color loss function L c is set to:

[0120]

[0121] wherein, M(u) ∈ {0, 1} is the segmentation mask of the target object, which indicates whether the pixel belongs to the target object. M(u) = 1 means that the pixel u belongs to the target object, and the color differences of these pixels will be included in the loss calculation. M(u) = 0 means that the pixel u does not belong to the target object, and the color differences of these pixels will not participate in the loss calculation;

[0122] C(u) is the color value of pixel u in the real image, is the predicted color value of pixel u, represents the square of the L2 norm.

[0123] The mask loss trains the implicit and explicit combined neural radiance field network by minimizing the squared error between the segmentation mask predicted by the model and the real mask, enabling the model to better distinguish the foreground and background and accurately segment the target object.

[0124] Preferably, the mask loss is expressed as:

[0125]

[0126] The reprojection loss trains the implicit and explicit combined neural radiance field network by minimizing the squared error between the 3D center point projection of the target object and the center point of the 2D image bounding box, enabling the projection of the 3D center point of the target object after reconstruction in each view to be as close as possible to the 2D bounding box center in the image and improving the accuracy of target object pose estimation.

[0127] Preferably, the reprojection loss function is expressed as:

[0128]

[0129] where K is the intrinsic matrix of the camera, is the transformation function from the object coordinate system to the camera coordinate system, is the 3D center point coordinate of the object, obtained by weighted average; is the relative pose of the camera, is the center of the 2D bounding box in view i.

[0130] Preferably, during the training process, the implicit and explicit combined neural radiance field network is optimized based on the Adam optimizer.

[0131] Preferably, during the training process, the learning rate r is set to:

[0132]

[0133] where n i represents the number of visible viewpoints of the i-th voxel, and n max represents the maximum value of the visible viewpoints of all voxels.

[0134] In S3, the three-dimensional coordinates of the object are estimated by setting up a pose estimation network.

[0135] By estimating the object pose, the spatial pose of the object in the robotic arm coordinate system is identified to achieve precise grasping and operation. At the same time, through the neural network training of the postures of different objects, the system can identify the posture of similar objects faster and more accurately in the future.

[0136] Preferably, the pose estimation network is based on the U-Net network.

[0137] Preferably, the U-Net network is improved by replacing its encoder with ResNet to enhance the feature extraction ability.

[0138] Furthermore, the input of the pose estimation network is the predicted segmentation mask and the 3D object coordinates

[0139] In a preferred embodiment, the loss function L P′ of the pose estimation network is set as:

[0140] L P′ = L M′ + L d′ + L g′ + L n′

[0141] where, L M′ represents the segmentation loss, L d′ represents the distance loss, L g′ represents the gradient loss, L n′ represents the normal vector loss.

[0142] Among them, the segmentation loss uses the mean square error to minimize the difference between the segmentation mask predicted by the pose estimation network and the true segmentation mask M. Preferably, it is expressed as:

[0143]

[0144] The distance loss is used to minimize the element-wise distance difference between the normalized object coordinates predicted by the pose estimation network and the true value O. Preferably, it is expressed as:

[0145]

[0146] The gradient loss is used to minimize the gradient difference between the normalized object coordinates predicted by the pose estimation network and the true value O, and improve the sensitivity of the network to subtle geometric features. Preferably, it is expressed as:

[0147]

[0148] The normal vector loss is used to measure the cosine similarity between normal vectors, enhancing the model's understanding of the object's geometry, and is preferably expressed as:

[0149]

[0150] According to the present invention, based on the estimated three-dimensional object coordinates, the PnP+RANSAC method is used for pose estimation.

[0151] The PnP (Perspective-n-Point) is a commonly used method for estimating the camera pose and will not be elaborated in the present invention.

[0152] The RANSAC method (Random Sample Consensus) is a random sampling consensus algorithm proposed by Fischler and Bolles in 1981, which is particularly suitable for problems such as line and curve fitting in computer vision and has good robustness. In the present invention, its specific process will not be elaborated.

[0153] The PnP+RANSAC method includes the following steps:

[0154] S31. Randomly select a small number of matching points from the two-dimensional to three-dimensional point set for PnP solution to generate a pose hypothesis;

[0155] S32. Substitute all the remaining points into the pose hypothesis, calculate the error of their projection onto the image plane, and the points with an error less than the set threshold are considered inliers;

[0156] S33. Repeat S31 to S32 multiple times to generate multiple pose hypotheses;

[0157] S34. Select the pose hypothesis with the largest number of inliers as the final object pose.

[0158] In S4, the method for planning the motion trajectory of the robotic arm is not limited, and those skilled in the art can adopt any existing planning method.

[0159] Furthermore, during the grasping process of the robotic arm, the camera continuously acquires real-time images to monitor the pose change of the object. If a pose error occurs during the grasping process, adjustments are made according to the real-time feedback to ensure the accuracy of the grasping action.

[0160] The present invention also discloses a two-arm vision fusion robotic arm system, including a robotic arm, a camera and a processor arranged on the robotic arm,

[0161] The processor uses the above-mentioned Nerf-based rapid three-dimensional reconstruction and grasping method of the robotic arm to plan the motion trajectory of the robotic arm and control the robotic arm to grasp the object.

[0162] In various embodiments of the method described above in the present invention, it can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] It should be understood that various forms of the processes shown above can be used, reordering, adding or deleting steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved, and no limitations are imposed herein.

[0164] Embodiment

[0165] Embodiment 1

[0166] A robotic arm is used to perform a grasping experiment on a certain packaging box, including the following steps:

[0167] S1. Obtain an image of the working area.

[0168] S2. Perform three-dimensional reconstruction of the object based on the image.

[0169] S3. Estimate the pose of the object.

[0170] In S1, there are multiple cameras, which are set on the robotic arm. The entire shooting area is divided into three layers, and each layer is independently shot. In the shooting of each layer, the arc planning method is used to define the movement path of the robotic arm, and the movement between layers is achieved by the polynomial interpolation method.

[0171] Furthermore, the bounding box and segmentation mask of the target object are obtained through the YOLOv8 network.

[0172] In S2, the three-dimensional reconstruction method includes the following steps:

[0173] S21. Construct a neural radiance field network that combines explicit and implicit forms.

[0174] S22. Generate sampling points.

[0175] S23. Based on the sampling points, use the constructed implicit and explicit neural radiance field network to predict the volume density and color;

[0176] S24. Perform volume rendering to obtain the three-dimensional object.

[0177] In S21, on the basis of the traditional neural radiance field network, the implicit and explicit neural radiance field network is provided with a voxel grid and a color feature encoding. The voxel grid explicitly stores the volume density of the spatial position. The color feature decoder consists of 3 hidden layers, with 512, 256, and 128 neurons respectively. The voxel grid is set through the following sub-steps:

[0178] S211. Obtain the ray origin and direction of each pixel according to the size of the input image, the internal parameters of the camera, and the transformation matrix;

[0179] S222. Select the rays at the positions marked as 1 in the segmentation mask, calculate the nearest and farthest coordinates of the intersection points of the pixels and the scene according to the rays, and update the coordinates of the bounding box with this to obtain the region of interest;

[0180] S223. Establish a voxel grid within the region of interest;

[0181] The voxel grid is expressed as:

[0182]

[0183] In S21, during the training process, the Labelme annotation tool is used to perform detailed annotation on the images captured by the robotic arm camera.

[0184] In S22, during sampling, centered on the object, uniform sampling is performed near the object to generate sampling points, and coordinate transformation is also performed to convert the sampling points from the camera coordinate system to the object coordinate system.

[0185] In S23, the original volume density σ′ and color c of the sampling points are output by the implicit and explicit neural radiance field network, expressed as:

[0186] σ′ = interp(x′ 3D , V σ )

[0187] c = D(interp(x′ 3D , V c ), x′ 3D , d)

[0188] The original volume density is activated through an activation function to obtain the volume density σ, expressed as:

[0189] σ = softplus(σ′) = log(1 + exp(σ′ + b))

[0190] where b = 0.01.

[0191] In S24, the volume rendering is expressed as:

[0192]

[0193] α i = alpha(σ i , δ i ) = 1 - exp(-σ i δ i )

[0194]

[0195] δ i = t i+1 - t i

[0196]

[0197] The loss function L of the neural radiance field R is set to:

[0198]

[0199] During the training process, the explicit - implicit combined neural radiance field network optimizes the network based on the Adam optimizer, and the learning rate r is set to:

[0200]

[0201] In S3, the three - dimensional coordinates of the object are estimated by setting up a pose estimation network. The pose estimation network is based on the U - Net network and is improved by replacing its encoder with ResNet.

[0202] The loss function L of the pose estimation network P′ is set to:

[0203] L P′ = L M′ + L d′ + L g′ + L n′

[0204]

[0205] Based on the estimated three - dimensional object coordinates, the PnP + RANSAC method is used for pose estimation, including the following steps:

[0206] S31. Randomly select a small number of matching points from the two-dimensional to three-dimensional point set for PnP solution to generate a pose hypothesis;

[0207] S32. Substitute all the remaining points into the pose hypothesis and calculate the error of their projection onto the image plane. Points with an error less than the set threshold are considered inliers;

[0208] S33. Repeat S31 - S32 multiple times to generate multiple pose hypotheses;

[0209] S34. Select the pose hypothesis with the largest number of inliers as the final object pose.

[0210] In experiment S2, the three-dimensional reconstruction results are as shown in Figure 2 ((a), (b), (c)), and the pose estimation results are as shown in Figure 3 ((a), (b), (c)). It can be seen from Figure 2-3 that the object pose can be accurately estimated, enabling the robotic arm to be adjusted according to real-time feedback to ensure the accuracy of the grasping action.

[0211] The present invention has been described in combination with preferred embodiments above. However, these embodiments are merely exemplary and only serve an illustrative purpose. On this basis, various substitutions and improvements can be made to the present invention, and all of these fall within the protection scope of the present invention.

Claims

1. A Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method, characterized in that: The following steps are involved: S1, obtaining the working area image; S2, reconstructing the object in three dimensions according to the image; S3, estimate the object pose; S4. According to the current position of the object, plan the motion trajectory of the robot arm and grasp the object.

2. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 1, characterized in that: In S2, the three-dimensional reconstruction method comprises the following steps: S21. Constructing a neural radiation field network combining explicit and implicit methods; S22, generating sampling points; S23, based on the sampling points, the volume density and color are predicted using the constructed explicit and implicit combined neural radiation field network; S24, performing volume rendering to obtain a three-dimensional object.

3. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 2, characterized in that: In S21, the explicit and implicit neural radiation field network is provided with a voxel grid and a color feature encoding, wherein the voxel grid explicitly stores the volume density of the spatial position, The volume density represents the existence probability of a substance at the position, and the color feature code is used to store color features.

4. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 3, characterized in that: The voxel grid is set up via the following sub-steps: S211, obtaining the ray starting point and direction of each pixel according to the size of the input image, the intrinsic parameters of the camera and the transformation matrix; S222, selecting the ray at the position marked as 1 in the segmentation mask, calculating the nearest and farthest coordinates of the intersection of the pixel and the scene according to the ray, and updating the coordinates of the bounding box to obtain the region of interest; S223. Create a voxel grid in the region of interest.

5. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 2, characterized in that: In S22 , a coordinate system transformation is also performed to transform the sampling points from the camera coordinate system to the object coordinate system.

6. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 5, characterized in that: In S23, the converted sampling point (x′ 3D ,d′) and the corresponding position encoding input is an implicit and explicit neural radiation field network, which outputs the original volume density σ′ and color c of the sampling point, expressed as: σ′=interp(x′ 3D ,V σ ) c=D(interp(x′ 3D ,V c ),x′ 3D ,d) Among them, interp() is the interpolation operation, x′ 3D is the position coordinate of the converted sampling point, d is the sight direction of the sampling point in the camera coordinate system, V σ is the voxel grid that stores volume density information, V c is the voxel grid storing color features, and D is the color feature decoder; The original volume density is activated by the activation function to obtain the volume density σ.

7. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 2, characterized in that: In S24, the volume rendering is expressed as: a i =alpha(σ i ,d i )=1-exp(-σ i d i ) δ i =t i+1 -t i in, represents the color of the two-dimensional pixel position u, K is the number of sampling points, T i is the transmittance from the near plane to the i-th point, indicating the probability that the light is not completely absorbed before passing through the i-th point, c i is the color of the i-th sampling point, δ i is the distance between adjacent sampling points, t i is the distance between the ith sampling point and the origin, represents the mask at the 2D pixel position u, is the estimated value of the 3D point corresponding to the 2D image pixel.

8. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 2, characterized in that: Loss function L of the neural radiation field combined with explicit and implicit R Set to: Among them, L c Indicates color loss, L M represents the mask loss, L P represents the reprojection loss; The color loss trains the explicit and implicit neural radiance field network by minimizing the square error between the predicted color and the real image, the mask loss trains the explicit and implicit neural radiance field network by minimizing the square error between the segmentation mask predicted by the model and the real mask, and the reprojection loss trains the explicit and implicit neural radiance field network by minimizing the square error between the 3D center point projection of the target object and the center point of the 2D image bounding box.

9. The Nerf-based robotic arm rapid three-dimensional reconstruction and grasping method according to claim 1, characterized in that: In S3, the loss function L of the posture estimation network is P′ Set to: L P′ =L M′ +L d′ +L g′ +L n′ Among them, L M′ represents the segmentation loss, L d′ Represents the distance loss, L g′ represents the gradient loss, L n′ Denotes the normal vector loss.

10. A dual-arm vision fusion robotic arm system, characterized in that: It includes a robotic arm, a camera and a processor arranged on the robotic arm, The processor plans the motion trajectory of the robotic arm using the Nerf-based robotic arm fast three-dimensional reconstruction and grasping method, and controls the robotic arm to grasp the object.

Images