Target grabbing three-dimensional modeling method based on dynamic vision

Abstract

The invention discloses a target grabbing three-dimensional modeling method based on dynamic vision, and a system used in the method comprises a first mechanical arm used for grabbing a target object and placing the target object in a transfer position, the tail end of the first mechanical arm is a suction cup provided with an IMU sensor, and the surface of the suction cup is provided with an ArUco code identifier; the second mechanical arm is used for grabbing the target object on the transfer position; the industrial camera can capture an ArUco code identifier; the method comprises the following steps: recording the timestamp of the action of the sucker of the mechanical arm, performing time synchronization with camera and IMU data, continuously acquiring RGB images according to the starting timestamp and the ending timestamp, calculating the posture of the sucker in real time, and mapping a body coordinate system of the sucker to a camera coordinate system; segmenting a target object from a dynamic region by using a NeuFlow algorithm; carrying out three-dimensional reconstruction by adopting a NeRF technology, and carrying out geometric constraint by combining the foreground mask and the pose of the mechanical arm; and extracting the point cloud and carrying out size measurement. According to the method, high-precision and rapid three-dimensional modeling of the target object under the condition of limited computing resources is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent grabbing of mechanical arms, and particularly relates to a three-dimensional modeling method for a grabbed target based on dynamic vision. BACKGROUND

[0002] In the field of advanced automation production, the application of multi-robot collaborative target grabbing is increasing, and in these practical applications, it is crucial to accurately obtain the three-dimensional size and placement position information of the target, which is the basis for multi-robot coordination and accurate grabbing, and can provide accurate data support for path planning and motion control of the robot, thereby improving production efficiency and operation quality.

[0003] Traditional three-dimensional modeling methods for targets are mostly designed based on static scenes, and these methods often rely on complex mathematical models and a large amount of computing resources to construct the three-dimensional model of the target. In a dynamic environment, due to the rapid changes in the position and posture of the target, it is difficult for traditional modeling methods to timely and stably track and extract the accurate three-dimensional parameters (such as length, width, height, etc.) and coordinate positions of the target.

[0004] In existing solutions, although laser point cloud sensors can obtain high-precision three-dimensional data, the devices are bulky and have a huge amount of calculation, which is not suitable for real-time deployment on edge computing hardware; the field of view of an RGBD camera is limited; and pure visual depth estimation often lacks accuracy. More and more application scenarios require direct identification and three-dimensional modeling of the target on an edge computing board to reduce data transmission delay and improve the real-time performance and autonomy of the system. Due to the limited computing resources of the edge computing board, the existing three-dimensional modeling methods for targets cannot meet the requirements. Therefore, it is urgent to develop a fast and accurate multi-modal data fusion three-dimensional modeling method in a computing resource limited environment to meet the requirements of collaborative robots for fast positioning and grabbing of targets. SUMMARY

[0005] To solve the problems in the prior art, the application provides a three-dimensional modeling method for a grabbed target based on dynamic vision, which can realize efficient modeling and accurate positioning of the target in a resource-limited edge computing environment.

[0006] The technical scheme adopted by the application is as follows: a three-dimensional modeling method for a grabbed target based on dynamic vision, the system used in the method comprises:

[0007] a first robot arm, the first robot arm is used for grabbing and placing a target in a transfer position, the end grabbing execution unit of the first robot arm is a suction cup, an IMU sensor is installed on the suction cup, and an ArUco code mark is arranged on the surface of the suction cup;

[0008] a second robot arm for picking up the target object placed on the transit position by the first robot arm;

[0009] an industrial camera fixedly installed in the working area and having a field of view covering the whole process of picking up the target object by the first robot arm and transferring to the transit position, so that the industrial camera can capture the ArUco code mark;

[0010] The method comprises the following steps:

[0011] S1, recording a start time stamp of a start of a picking action of a suction cup of the first robot arm , and an end time stamp of a completion of a releasing action ;

[0012] S2, extracting a multi-view RGB image sequence {I t} and IMU data in a corresponding time interval , from a data stream synchronized with an image acquisition time stamp of the industrial camera and an IMU sensor data time stamp, according to the start time stamp , ; the image sequence {I t} is obtained by the industrial camera continuously acquiring a movement process of the target object in the time interval;

[0013] S3, solving a pose of the suction cup in a coordinate system of the industrial camera at each image frame time stamp by a PnP algorithm, by using the acquired image containing the ArUco code;

[0014] S4, interpolating or filtering the pose obtained in S3 by using high-frequency attitude data provided by the IMU sensor, to obtain a higher time resolution or a smoother movement track of the suction cup;

[0015] S5, calculating an optical flow field by using a NeuFlow algorithm on the acquired continuous multi-view RGB image sequence {I t}, separating a moving foreground region and a static background region based on a change feature of an optical flow gradient, and then separating the foreground region of the picked target object from the complex background region to obtain a foreground mask M_fg;

[0016] S6, adopt NeRF technology to carry out three-dimensional reconstruction, utilize foreground mask M_fg to limit the space range of NeRF modeling, only the foreground mask area is reconstructed, and the reconstruction process of NeRF is combined with the pose and normal vector information of the center of the first mechanical arm suction cup, and the geometric constraint is that according to the accurate pose of the suction cup in the industrial camera coordinate system obtained in step S4, a virtual plane is uniquely determined from the position vector P_t of the center point of the suction cup and the normal vector N_t of the suction cup surface at this moment, in the training process of the NeRF model, a regularization loss term is added to punish the body density of the space region above the virtual plane, so as to constrain the reconstructed model to be below the suction cup;

[0017] S7, after completing the NeRF modeling, the model is processed to extract the size and final position information of the target object, and the step includes:

[0018] a) extract high-precision three-dimensional point cloud data from the NeRF model, and obtain the three-dimensional geometric size information of the target object by analyzing the point cloud data;

[0019] b) calculate the center point local coordinates of the point cloud data in the suction cup coordinate system, and transform the center point local coordinates to the industrial camera coordinate system by using the suction cup pose obtained in step S4 at the end timestamp, to determine the final center point coordinates of the target object on the transfer site.

[0020] Extracting the center point coordinates and three-dimensional geometric size information of the target object provides accurate auxiliary parameters for the second mechanical arm grabbing operation.

[0021] Further, it also includes an air pressure sensor, which is used to detect the internal negative pressure state of the suction cup during the suction process of the first mechanical arm; when the air pressure sensor detects that the internal pressure of the suction cup drops below the set negative pressure threshold and stabilizes in a short time, it indicates that the suction cup has effectively adhered to the surface of the target object, that is, it is determined that the suction process starts, and the system records this time as the timestamp of the start of the grabbing action When the suction cup releases the pressure, the air pressure sensor detects that the internal pressure of the suction cup quickly rises close to atmospheric pressure and stabilizes in a short time, and the system records this time as the timestamp of the completion of the release action .

[0022] Further, in step S6, Instant-NGP technology is used to accelerate the NeRF training process.

[0023] Further, in step S5, the specific method of generating the foreground mask M_fg is: for the continuous image sequence {I tAdjacent frames of the optical flow calculation is carried out, for the pixel point in the image, if its optical flow amplitude and gradient is greater than the preset threshold, it is marked as foreground, otherwise it is marked as background.

[0024] The beneficial effects of the present application are:

[0025] The present application is aimed at multi-robot cooperative grasping, which realizes fast and accurate three-dimensional modeling of the target object under edge computing conditions by combining NeuFlow optical flow segmentation and NeRF implicit representation technology. Compared with traditional laser point cloud or RGBD scheme, the present application effectively reduces the computational and device burden through pure visual fusion and high-speed optimized NeRF modeling method, and meets the resource limitation requirements of edge computing platform. The NeuFlow foreground segmentation method ensures fast extraction of target object area in complex scenes, avoids relying on large-scale depth sensors, and improves the adaptability and flexibility of modeling. The Instant-NGP-based NeRF optimization process greatly shortens the reconstruction time, and improves the spatial consistency and accuracy of the three-dimensional model through the constraint of the sucker normal vector, providing reliable coordinate and size data for subsequent multi-robot cooperative grasping planning. The present application realizes high-precision and fast three-dimensional modeling of the target object under limited computing resources, providing reliable parameter support for subsequent robot grasping, and improving the accuracy and efficiency of grasping. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the flow chart of the data acquisition and synchronization steps of the present application.

[0027] Figure 2 is the flow chart of the foreground target segmentation step of the present application.

[0028] Figure 3 is the flow chart of the NeRF modeling process of the present application.

[0029] Figure 4 is the flow chart of the three-dimensional information extraction step of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] The present embodiment takes the practical application of multi-robot cooperative grasping of road stones as an example. The present embodiment provides a three-dimensional modeling method for grasping target based on dynamic vision, which uses a system including:

[0032] The first mechanical arm is used for grabbing and placing the curbstone on the transfer position, and the end grabbing execution unit of the first mechanical arm is a suction cup.

[0033] The second mechanical arm is used for grabbing the curbstone placed on the transfer position by the first mechanical arm.

[0034] The industrial camera is fixedly installed in the working area, and the field of view thereof can cover the whole process of grabbing and transferring the curbstone to the transfer position by the first mechanical arm, so that the industrial camera can capture the ArUco code mark on the surface of the suction cup.

[0035] The mechanical device related to the present application only completes identification and three-dimensional modeling on the edge computing board.

[0036] The three-dimensional modeling method for grabbing a target based on dynamic vision provided by the embodiment comprises the following steps:

[0037] S1, recording the start time stamp of the start of the grabbing action of the suction cup of the first mechanical arm , and the end time stamp of the completion of the releasing action ;

[0038] In the embodiment, a gas pressure sensor is installed on the suction cup of the first mechanical arm, which is used for detecting the internal negative pressure state of the suction cup during the suction process of the first mechanical arm on the curbstone. The time stamp of the start of the grabbing action and the time stamp of the completion of the releasing action : when the gas pressure sensor detects that the internal pressure of the suction cup drops below the set negative pressure threshold and stabilizes in a short time (such as 0.5 seconds), it indicates that the suction cup has effectively adhered to the surface of the curbstone, that is, it is determined that the suction process starts, and the system records this time as the time stamp of the start of the grabbing action ; when the suction cup releases the pressure, the gas pressure sensor detects that the internal pressure of the suction cup quickly rises close to atmospheric pressure and stabilizes in a short time, and the system records this time as the time stamp of the completion of the releasing action

[0039] S2, according to the start time stamp and the end time stamp , from the data stream synchronized with the image acquisition time stamp of the industrial camera and the IMU sensor data time stamp, extract the multi-view RGB image sequence {I , } and IMU data in the corresponding time interval t} and IMU data in the corresponding time interval tThe motion of the target object is continuously captured by an industrial camera within the stated time interval. Time synchronization can be achieved through various methods known in the art. For example, a unified clock synchronization program can be run on an edge computing board (such as Jetson AGX Orin) that periodically broadcasts synchronization signals to each sensor module (IMU sensor, barometric pressure sensor, and industrial camera). Alternatively, clock alignment can be achieved through a unified time protocol (such as PTP or an embedded clock source), ensuring that each data sample (image frame, IMU reading, barometric pressure value) has a timestamp corresponding to the system's unified time reference. This allows for precise alignment of the image sequence to be reconstructed, the corresponding IMU data, and the barometric pressure information during subsequent data processing. Image frames can be cached in system memory or written to high-speed storage media (such as NVMe SSD) to provide raw data support for subsequent NeuFlow segmentation, NeRF modeling, and point cloud extraction steps.

[0040] S3. Using the acquired images containing ArUco codes, the PnP algorithm is used to solve the pose of the suction cup in the industrial camera coordinate system at each image frame timestamp. The pose includes the position vector P_t of the suction cup center point and the normal vector N_t of the suction cup surface.

[0041] S4. Using the high-frequency attitude data provided by the IMU sensor, the pose obtained in S3 is interpolated or filtered to obtain a higher time resolution or a smoother suction cup motion trajectory. The IMU sensor provides attitude information such as tilt angle, roll, and yaw at the end of the suction cup, usually given in the form of quaternions or Euler angles. Noise is suppressed in the real-time streaming data through data fusion algorithms (such as Madgwick or Mahony filters) to obtain a smooth and stable suction cup attitude.

[0042] Through the above process, even with only auxiliary information from IMU sensors and barometric pressure sensors, and with missing or distorted feedback from the robotic arm joints, the system can still achieve high-precision, multi-source data time synchronization and attitude determination. This process provides a rigorous temporal and spatial reference basis for subsequent 3D modeling of curb stones, making the generated 3D data and curb stone size and position information more accurate and reliable.

[0043] S5, in acquiring continuous multi-view RGB image sequences {I t Afterwards, the NeuFlow algorithm is used to calculate the optical flow field of the acquired image sequence {It}. Based on the variation characteristics of the optical flow gradient, the moving foreground region and the stationary background region in the scene are separated, and finally the foreground mask M_fg is obtained. This separates the foreground region of the captured curbstone from the complex background region, creating conditions for subsequent 3D modeling to focus within the effective spatial range. The specific method for generating the foreground mask M_fg is as follows: for the continuous image sequence {I}... tThe optical flow calculation is performed on adjacent frames of the image, and for a pixel point in the image, if the amplitude of the optical flow and the gradient of the pixel point are greater than a preset threshold, the pixel point is marked as foreground, otherwise, the pixel point is marked as background.

[0044] S6, three-dimensional reconstruction is performed by using the NeRF technology, in the process of training the NeRF parameters, only the rays meeting the foreground mask M_fg are emphatically sampled, so that the optimization is focused on the curbstone area. In order to enhance the spatial consistency and precision of three-dimensional modeling, it is assumed that the top surface of the curbstone is approximately parallel to the end surface of the suction cup, and the embodiment utilizes the pose and normal vector information of the center of the suction cup to constrain the NeRF on the plane, and the specific implementation manner is as follows:

[0045] 1. Definition of a virtual constraint plane:

[0046] One of the core assumptions of the present application is that, in the process of grabbing, the target object (such as a curbstone) is firmly adsorbed under the first robotic arm suction cup. Therefore, the plane where the suction cup is located can be regarded as a physical boundary, and the space above the plane should theoretically not exist any part of the target object (i.e. should be a low-density or zero-density area).

[0047] According to the suction cup pose corresponding to the timestamp of each frame image It obtained in step S4, a corresponding virtual constraint plane Π_t can be defined for the frame image. The plane Π_t is uniquely determined by the position vector P_t (obtained by the PnP algorithm) of the center point of the suction cup and the normal vector N_t (also obtained by pose solving) of the suction cup surface at the time.

[0048] 2. Mathematical model of constraint and loss function design:

[0049] In order to integrate the above-mentioned physical assumption into the training process of the NeRF model, the present application introduces a plane regularization loss term, denoted as L_plane. The loss term is used to punish the volume density of the three-dimensional space points predicted to be located above the virtual constraint plane Π_t.

[0050] Specifically, for any three-dimensional point p = (x, y, z) sampled by the NeRF model on the light ray r, whether it is located in the area that needs to be punished can be judged by the relationship between the point and the plane Π_t. The directed distance d of the point p to the plane Π_t can be obtained by dot product operation:

[0051] ,

[0052] wherein, represents vector dot product.

[0053] If d > 0, it indicates that point p is located on the side pointed by normal vector N_t, i.e. the "upper" space of the chuck. If d≤ 0, it indicates that point p is located in the "lower" space of the chuck or exactly on the plane.

[0054] Based on this, The loss function is defined as the sum of predicted density σ(p) of all sampling points located in the "upper" space. To increase robustness in implementation, a small tolerance ε (e.g. a very small positive number representing half of the chuck thickness or a safety distance) can be set to avoid over-penalizing points exactly on the boundary. In this embodiment, considering the physical thickness of the chuck and possible measurement errors, ε can be set to a small positive value, for example, ε can be set in the range of 0.5mm to 2mm, and a typical value is 1mm. Therefore, the loss function can be formalized as:

[0055] for all sampling points p_i that satisfy .

[0056] 3. Integrated into the NeRF training process:

[0057] This newly designed L_plane loss term will be combined with the original rendering loss L_rgb (i.e. the difference between predicted color and real image color) of NeRF to form the final total loss function L_total:

[0058]

[0059] where λ is a hyperparameter as a weight coefficient to balance the intensity of rendering fidelity and geometric constraints. The value of λ can be adjusted through experiments, and the value of λ is related to the specific implementation of the scene and NeRF model, which needs to be adjusted through experiments to achieve the best effect. In this embodiment for the road curb grasping scene, the initial value of λ can be set between 0.01 and 0.5. For example, when the order of magnitude of the rendering loss L_rgb is around 1.0, a setting of λ = 0.1 usually achieves good constraint effect, which can effectively suppress the artifacts in the upper space, and also does not excessively affect the reconstruction quality of the target object itself. Those skilled in the art should understand that the weight coefficient λ is not fixed and can be adjusted adaptively according to factors such as the material of the target object and the lighting conditions.

[0060] In the backpropagation and gradient descent optimization process of the NeRF model, will guide the update of network parameters at the same time. In this way, the network not only learns how to render realistic images, but also is "taught" the geometric prior knowledge about the scene, i.e. "objects cannot appear above the chuck".

[0061] In this embodiment, Instant-NGP technology is used to accelerate the NeRF training process, thus meeting the processing capability requirements of edge computing boards.

[0062] S7, after completing the NeRF modeling, the model is processed to extract the final position and size information of the target object in the industrial camera coordinate system, which specifically includes:

[0063] a) Three-dimensional point cloud extraction and size measurement: First, the high-density area is extracted as a three-dimensional point cloud by voxelizing the implicit scene of NeRF or using methods such as marching cubes. Then, the point cloud is analyzed, such as determining its principal axis direction using principal component analysis (PCA), or fitting a minimum volume bounding box, to calculate the three-dimensional geometric size information of the target object (such as a curbstone), such as length, width, and height.

[0064] b) Determination of final position coordinates: Calculate the local coordinates of the center point of the three-dimensional point cloud extracted in step a) in its own reference coordinate system (i.e. aligned with the suction cup coordinate system). Finally, from the smooth motion trajectory generated in step S4, query the suction cup pose transformation matrix corresponding to the end timestamp tend. Using this pose matrix, transform the center point local coordinates from the suction cup coordinate system to the industrial camera coordinate system, to obtain the precise global center point coordinates of the target object placed on the transfer site.

[0065] The extracted center point coordinates of the target object and its three-dimensional geometric size information provide precise positioning parameters for the grabbing operation of the second mechanical arm. The implementation of this patent stops at the size and position measurement output, and does not involve subsequent grabbing control and implementation details.

[0066] Through the above steps, the present application uses IMU and ArUco markers and other sensor information to obtain the suction cup pose without the need for mechanical arm joint pose information, uses NeRF implicit modeling technology as the core, identifies the foreground target through NeuFlow segmentation and introduces plane constraints, improves the accuracy and speed of three-dimensional reconstruction of curbstones, and thus provides solid three-dimensional data support for subsequent automated grabbing planning of the mechanical arm.

[0067] The above is only an example and description of the structure of the present application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, as long as they do not deviate from the structure of the present application or exceed the scope defined by the present claims, and should belong to the protection scope of the present application.

Claims

1. A dynamic vision-based three-dimensional modeling method for grasping objects, characterized in that, The system used in the method comprises: A first mechanical arm for grabbing and placing a target object in a transfer position, the end of the first mechanical arm being a suction cup, an IMU sensor being installed on the suction cup, and an ArUco code marker being arranged on the surface of the suction cup; A second mechanical arm for grabbing the target object placed in the transfer position by the first mechanical arm; An industrial camera fixedly installed in a work area, and having a field of view covering the entire process of the first mechanical arm grabbing the target object and transferring it to the transfer position, so that the industrial camera can capture the ArUco code marker; The method comprises the following steps: S1, record a start time stamp of a start of a grabbing action of the first robot arm suction pad , and an end time stamp of a completion of a releasing action ; S2, according to the start timestamp and end timestamp , from a data stream synchronized with the industrial camera image acquisition timestamp and the IMU sensor data timestamp, extract a multi-view RGB image sequence {I t} and IMU data within a corresponding time interval [ , ]; the image sequence {I t} is obtained by the industrial camera during the time interval of the target object movement process; S3. Using the collected image containing the ArUco code, the pose of the suction cup in the industrial camera coordinate system at each image frame timestamp is solved by a PnP algorithm; S4. The pose obtained in S3 is interpolated or filtered using high-frequency attitude data provided by the IMU sensor to obtain a higher time resolution or smoother suction cup motion trajectory; S5, calculating the optical flow field of the acquired continuous multi-view RGB image sequence {I t} using the NeuFlow algorithm, separating the moving foreground region and the static background region based on the change characteristics of the optical flow gradient, and then separating the foreground region of the captured target object from the complex background region to obtain a foreground mask M_fg; S6. Three-dimensional reconstruction is performed using the NeRF technology, the range of the NeRF modeling space is limited using the foreground mask M_fg, only the foreground mask area is reconstructed, and the reconstruction process of the NeRF is geometrically constrained by combining the pose and normal vector information of the center of the first mechanical arm suction cup, the geometric constraint being that according to the accurate pose of the suction cup in the industrial camera coordinate system obtained in step S4, a virtual plane is uniquely determined by the position vector P_t of the center of the suction cup and the normal vector N_t of the surface of the suction cup at that time, and a regularization loss term is added in the training process of the NeRF model to punish the volume density of the space region above the virtual plane, so as to constrain the reconstructed model to be below the suction cup; S7. After completing the NeRF modeling, high-precision three-dimensional point cloud data of the target object is extracted therefrom, the three-dimensional point cloud data is processed, and the center point coordinates and three-dimensional geometric size information of the target object are extracted to provide accurate auxiliary parameters for the grabbing operation of the second mechanical arm.

2. The method of claim 1, wherein, A barometric pressure sensor is further included for detecting the internal negative pressure state of the suction cup during the suction process of the first mechanical arm on the target object; When the air pressure sensor detects that the internal pressure of the suction cup drops below the set negative pressure threshold and stabilizes in a short time, it indicates that the suction cup has effectively adhered to the surface of the target object, that is, it is determined that the suction process has started, and the system records this time as the timestamp of the start of the grabbing action When the suction cup releases the pressure, the air pressure sensor detects that the internal pressure of the suction cup quickly rises to close to atmospheric pressure and stabilizes in a short time, and the system records this time as the timestamp of the completion of the release action .

3. The method of claim 1, wherein the method further comprises: In step S6, the Instant-NGP technology is used to accelerate the NeRF training process.

4. The method of claim 1, wherein the method further comprises: In step S5, the specific method for generating the foreground mask M_fg is: performing optical flow calculation on adjacent frames of the continuous image sequence {I t} and, for a pixel point in an image, if the amplitude of the optical flow and the gradient of the pixel point are greater than a preset threshold, marking the pixel point as foreground, otherwise marking the pixel point as background.

Images