Three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision

Abstract

The present application relates to the technical field of robot vision, and more particularly to a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, comprising: S1: visual calibration is performed on a 3D camera on a collaborative robot; S2: it is judged whether there is a workpiece standard model, if yes, a candidate viewpoint set is constructed according to the workpiece standard model, if not, a workpiece standard model is established according to a preset fixed multi-viewpoint, and then a candidate viewpoint set is constructed; S3: a deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, controls the actual six-axis collaborative robot motion according to the planning result, carries out data acquisition, obtains 3D point cloud data of the workpiece under the viewpoint, and carries out preprocessing including removing noise points and filtering, and generates a three-dimensional model of the workpiece based on the fused point cloud data. The present application has high automation, almost no manual operation is needed throughout the process, and can effectively and efficiently reconstruct three-dimensional models for a large number of different types of non-standard customized small-batch workpieces.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision, and in particular to a three-dimensional reconstruction viewpoint planning method, device and storage medium based on collaborative robot 3D vision. BACKGROUND

[0002] In the wave of rapid development of modern industry, digital twin technology has increasingly become an indispensable part of manufacturing and intelligent manufacturing systems, and has been widely applied in many fields such as industrial manufacturing, urban management, aerospace, medical treatment, etc. Among the many key technologies involved in digital twin technology, three-dimensional reconstruction technology undoubtedly occupies a pivotal position.

[0003] With the development of digital twin technology, there are more and more diversified and high-standard demands for three-dimensional reconstruction technology. The increase in the complexity of the industrial manufacturing process has led to the need for three-dimensional reconstruction technology to develop in the direction of automation in order to improve efficiency and reduce human intervention. Traditional manual data acquisition methods have been unable to meet the efficient and accurate needs, so automated viewpoint planning has become a key technology for improving the quality of three-dimensional reconstruction. The core of viewpoint planning is to automatically select the best shooting position and angle to ensure that the acquired point cloud data can cover the workpiece surface as much as possible. Reasonable viewpoint planning can significantly improve the reconstruction integrity and accuracy of complex workpieces and improve production efficiency, which is particularly important for the application of three-dimensional reconstruction technology in digital twins. SUMMARY

[0004] The purpose of the present application is to solve the shortcomings in the prior art, and to provide a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, comprising the following steps:

[0005] S1: visual calibration of a 3D camera on a collaborative robot, the visual calibration including internal parameter calibration of determining camera internal parameters according to a calibration board, hand-eye calibration, and obtaining an effective reconstruction viewing angle threshold range, wherein the hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the six-axis collaborative robot end effector and the pose relationship of the calibration board coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to obtain the effective viewing angle threshold range;

[0006] S2: determining whether there is a workpiece standard model, if there is the workpiece standard model, a candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, if there is no workpiece standard model, establishing the workpiece standard model according to the camera internal parameters, the conversion relationship and the viewing angle threshold range, and after the establishment, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model;

[0007] S3: The deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, controls the actual six-axis collaborative robot to move according to the planning result, and the 3D camera collects data at each viewpoint position to obtain 3D point cloud data of the workpiece under the viewpoint, and the multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data;

[0008] Unless otherwise specified, all references to viewpoints are generalized representation viewpoints including position and attitude.

[0009] Preferably, in step S2, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, including:

[0010] The model upper surface of the workpiece standard model is obtained through modeling software, the model upper surface is uniformly triangulated by mesh division software to obtain a division grid, and the point cloud of the division grid is obtained;

[0011] The normal direction of each vertex in the point cloud is calculated, and the normal direction is adjusted to ensure that the normal vector direction is perpendicular to the model upper surface and outward, Gaussian distribution sampling is performed along the vertex normal direction, and a candidate viewpoint is generated;

[0012] The current coordinate system of the candidate viewpoint is converted to the robot coordinate system according to the conversion relationship, positions of the converted candidate viewpoints that are unreachable or close to singular points are filtered, the camera optical axis direction is calculated after filtering, whether the camera angle is within the viewpoint threshold range is verified according to the camera optical axis direction, and if yes, the final viewpoint set is added.

[0013] Preferably, in step S2, the workpiece standard model is established according to the camera intrinsic parameter, the conversion relationship and the viewpoint threshold range, including:

[0014] Within the viewpoint threshold range, the six-axis collaborative robot moves the camera according to a preset path to collect multi-viewpoint point cloud data, the multi-viewpoint point cloud data is preprocessed according to the camera intrinsic parameter, and the multi-viewpoint point cloud data is unified to the robot coordinate system through the conversion relationship to obtain converted point cloud;

[0015] The position alignment point cloud between the converted point cloud is adjusted by using iterative closest point (ICP) or feature matching to obtain spliced point cloud data;

[0016] The spliced point cloud data is converted into a triangular mesh model, and the triangular mesh model is used as the workpiece standard model.

[0017] Preferably, in step S3, the deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, including:

[0018] The deep reinforcement learning mechanism determines the number of iterations for obtaining a stable initial viewpoint and an acceptable viewpoint reward value through Monte-Carlo initialization evaluation experiments, and obtains the initial viewpoint;

[0019] The deep reinforcement learning mechanism searches for a next viewpoint with the highest reward within a given number of iterations from the candidate viewpoint set according to an ε-greedy strategy and a PPO algorithm, judges whether to add the obtained next viewpoint to the current viewpoint list according to a viewpoint judgment mechanism, and obtains a final viewpoint list from the obtained current viewpoint list according to a viewpoint list judgment mechanism.

[0020] The planning result is obtained according to the final viewpoint list.

[0021] Preferably, searching for a next viewpoint with the highest reward within a given number of iterations from the candidate viewpoint set according to an ε-greedy strategy and a PPO algorithm comprises:

[0022] The policy network uses the ε-greedy strategy to decide which direction to explore from the current viewpoint, selects a direction with the highest current estimated value from all directions with a probability of 1-ε, and selects a direction randomly from all directions with a probability of ε, and determines a next viewpoint from the candidate viewpoint set according to the exploration direction;

[0023] After the next viewpoint is selected, the PPO algorithm fine-tunes the specific position and pose of the viewpoint, generates a series of actions using the existing strategy, and adjusts the strategy according to the rewards of these actions.

[0024] The reward calculation and action selection in the ε-greedy strategy and the PPO algorithm are both constrained by constraint conditions including field of view constraints, occlusion constraints, viewing angle constraints, data redundancy constraints, and sensor accessibility constraints.

[0025] Preferably, judging whether to add the obtained next viewpoint to the current viewpoint list according to a viewpoint judgment mechanism comprises:

[0026] The viewpoint judgment mechanism judges whether the obtained next viewpoint meets the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations, and if the obtained next viewpoint meets the viewpoint judgment mechanism, it is added to the current viewpoint list.

[0027] If the viewpoint judgment mechanism is not met, the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0028] Preferably, the final viewpoint list is obtained from the obtained current viewpoint list according to a viewpoint list judgment mechanism, and further comprises:

[0029] The view point column judgment mechanism judges whether the current view point column meets the given three-dimensional reconstruction requirement and whether the number of view points is the smallest within the given iteration number, and if the view angle column meets the view point column judgment mechanism, it is determined that the final view point column has been obtained.

[0030] If the constraint condition is not met, the current view point column is recorded, the view points in the view point column except the initial view point are emptied, and the view points are iteratively searched again through the deep reinforcement learning mechanism before the iteration number reaches the given value.

[0031] Preferably, the planning result is obtained according to the final view point column, comprising:

[0032] Taking each view point in the final view point column as a target position, a six-axis collaborative robot is calculated to start from an initial pose, and a feasible path from the current pose to the first view point is calculated using a preset path planning;

[0033] When planning the path to the next view point, the pose of the previous view point is taken as a new starting pose, and the above path planning process is repeated until a complete path connecting all view points is planned;

[0034] Collision detection is performed on the planned path to check whether the robot will collide with obstacles or its own components in the workspace when moving along the path, and if a collision is detected, the path is optimized and adjusted.

[0035] Based on the same concept, the present application also provides a computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor to make the processor execute the steps of the three-dimensional reconstruction view point planning method based on the collaborative robot 3D vision as described in any of the embodiments.

[0036] Based on the same concept, the present application also provides a storage medium storing computer readable instructions, the computer readable instructions being executed by one or more processors to make one or more processors execute the steps of the three-dimensional reconstruction view point planning method based on the collaborative robot 3D vision as described in any of the embodiments.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] The application judges whether there is a workpiece standard model, if there is a workpiece standard model, the candidate view point set construction module constructs a view point set according to the workpiece standard model, if there is no workpiece standard model, a workpiece standard model is established according to the camera internal parameter, the conversion relationship and the view angle threshold range, and the candidate view point set construction module constructs a view point set according to the workpiece standard model after the establishment, by considering two cases of having and not having a workpiece standard model, the flexibility is high, and different working scenes can be coped with.

[0039] The application adopts a deep reinforcement learning method mechanism to obtain a planning result according to a candidate viewpoint set, wherein a strategy network module can be replaced and changed for different types of tasks, and the expansibility is strong.

[0040] The application obtains a planning result according to a candidate viewpoint set through a deep reinforcement learning mechanism, controls the actual six-axis collaborative robot motion according to the planning result, and the 3D camera collects data at each viewpoint position to obtain 3D point cloud data of the workpiece under the perspective, the multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data, which is high in automation, almost no manual operation is needed throughout the process, and can effectively and efficiently reconstruct the three-dimensional model for a large number of different types of non-standard customized small batches of workpieces.

[0041] The application adjusts the target of three-dimensional reconstruction by changing the reward function, selectively realizes global reconstruction or key local reconstruction, and has high flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments and are not meant to limit the present application.

[0043] Figure 1 A flowchart of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the application;

[0044] Figure 2 Another flowchart of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the application;

[0045] Figure 3 A six-axis collaborative robot structure diagram of the application;

[0046] Figure 4 A workpiece upper surface grid diagram of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the application;

[0047] Figure 5 A schematic diagram of constructing a candidate viewpoint set by Gaussian random sampling of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the application;

[0048] Figure 6 A field of view visualization schematic diagram of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the application;

[0049] Figure 7The schematic diagram of the occlusion judgment of the three-dimensional reconstruction view point planning method based on the collaborative robot 3D vision of the present application.

[0050] Reference signs:

[0051] 1: six-axis collaborative robot, 2: 3D camera, 3: tool, 4: target workpiece. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0053] Those skilled in the art can understand that, unless specifically stated, when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for circuit communication.

[0054] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0055] In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0056] First embodiment

[0057] Referring to FIGS. 1-4, the three-dimensional reconstruction view point planning method based on the collaborative robot 3D vision provided by the present embodiment comprises the following steps: Figure 1 and Figure 2 The three-dimensional reconstruction view point planning method based on the collaborative robot 3D vision provided by the present embodiment comprises the following steps:

[0058] Unless otherwise specified, all the viewpoints mentioned in the embodiments are generalized representation viewpoints containing position and posture.

[0059] S1: visual calibration is performed on the 3D camera on the collaborative robot, the visual calibration includes internal parameter calibration for determining the camera internal parameters according to the calibration board, hand-eye calibration and obtaining the effective reconstruction visual angle threshold range, wherein the hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the six-axis collaborative robot end effector and the pose relationship of the calibration board coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to obtain the effective visual angle threshold range.

[0060] The internal parameter calibration for determining the camera internal parameters according to the calibration board is to select a suitable calibration board, transform a variety of positions and angles in front of the 3D camera, collect a large number of images containing the calibration board, detect the feature points of the calibration board by using Zhang Zhengyou calibration algorithm and the like, and accurately calculate the internal parameters of the camera including focal length, principal point coordinates and distortion coefficients according to the corresponding relationship between the image coordinates of the feature points and the actual world coordinates.

[0061] The hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the six-axis collaborative robot end effector and the pose relationship of the calibration board coordinate system relative to the camera coordinate system, which further includes: stably installing the 3D camera on the robot hand, controlling the robot hand to move at different poses, and synchronously recording the robot hand pose information (such as joint angle) and the images captured by the 3D camera, specifically, in the embodiments, please refer to Figure 6 The provided 3D camera has a best working distance of 500 mm and a field of view distance range of 250 mm to 750 mm, Figure 6 The current viewpoint position and line of sight direction are represented by the ray, and the prism geometry constructed by the point and the line around the ray is the field of view of the 3D camera.

[0062] S2: it is judged whether there is a workpiece standard model, if there is a workpiece standard model, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model, if there is no workpiece standard model, the workpiece standard model is established according to the camera internal parameters, the conversion relationship and the visual angle threshold range, and after the establishment, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model.

[0063] Preferably, in step S2, the workpiece standard model is established according to the camera internal parameters, the conversion relationship and the visual angle threshold range, which includes:

[0064] Within the range of the threshold of the view angle, the six-axis collaborative robot moves the camera according to the preset path to collect multi-view point cloud data, the multi-view point cloud data is preprocessed according to the camera internal parameter, and the multi-view point cloud data is unified to the robot coordinate system through the conversion relationship to obtain converted point cloud;

[0065] The position alignment of the converted point cloud is adjusted by using an iterative closest point (ICP) or feature matching to obtain spliced point cloud data.

[0066] The spliced point cloud data is converted into a triangular mesh model, and the triangular mesh model is used as a standard model of the workpiece.

[0067] Preferably, in step S2, the candidate view point set construction module constructs the candidate view point set according to the standard model of the workpiece, including:

[0068] As shown in Figure 4 , the model upper surface of the standard model of the workpiece is obtained through modeling software (SolidWorks or other software capable of operating the standard model), the model upper surface is uniformly triangularly meshed by meshing software (Abaqus or other software capable of meshing on the model), and the point cloud of the meshed grid is obtained.

[0069] As shown in Figure 5 , the normal direction of each vertex in the point cloud is calculated, and the normal direction is adjusted to ensure that the normal vector direction is perpendicular to the model upper surface and outward, Gaussian distribution sampling is performed along the normal direction of the vertex to generate candidate view points. Specifically, in this embodiment, for each point in the point cloud, the normal direction of the point is calculated using the points in the local neighborhood. During the calculation process, the size of the neighborhood and the calculation method may need to be adjusted according to the density and distribution of the point cloud to ensure the accuracy of the normal direction. The center of the point cloud is taken as the mean value, and the variance is determined according to the range of the point cloud and the distribution of the normal. A series of candidate view points are sampled in space, i.e. Gaussian random sampling is performed to construct candidate view points within the range that meets the field of view distance of the 3D camera near the best working distance of the 3D camera, and the specific sampling method is not limited in this embodiment, nor are the related parameters of Gaussian random sampling or other sampling methods limited. In actual scenarios, the operator can flexibly select according to the on-site situation.

[0070] The current coordinate system of the candidate view point is converted to the robot coordinate system according to the conversion relationship, positions of the converted candidate view points that are unreachable or close to singular points are filtered, the camera optical axis direction is calculated after the filtering, and whether the camera angle is within the threshold of the view angle is verified according to the camera optical axis direction. If it is, it is added to the final view point set.

[0071] S3: The deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, controls the actual six-axis collaborative robot to move according to the planning result, and the 3D camera collects data at each viewpoint position to obtain 3D point cloud data of the workpiece under the viewpoint, and the multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data.

[0072] Preferably, in step S3, the deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, including:

[0073] The deep reinforcement learning mechanism determines the number of iterations for obtaining a stable initial viewpoint and an acceptable viewpoint reward value through Monte-Carlo initialization evaluation experiments, and obtains the initial viewpoint.

[0074] The deep reinforcement learning mechanism searches for the next viewpoint with the highest reward in the candidate viewpoint set within a given number of iterations according to an ε-greedy strategy and a PPO algorithm, judges whether to add the next viewpoint to the current viewpoint list according to a viewpoint judgment mechanism, and obtains the final viewpoint list according to a viewpoint list judgment mechanism.

[0075] The planning result is obtained according to the final viewpoint list.

[0076] The state space includes the pose information of the current robot, the obtained point cloud, the existing viewpoint information, the related geometric information of the workpiece model (such as surface normal vector, curvature, etc.), the visible surface points in the current state, the reward in the current state, and the obstacle information in the workspace, etc. These information is reasonably coded and integrated to form a state space, so that the deep reinforcement learning mechanism can perceive the current environment state.

[0077] The action space is mainly about selecting the next viewpoint. Since the position and pose of the viewpoint can be represented by continuous parameters (such as three-dimensional coordinates and rotation angles), and there are discrete decisions at certain discrete decision points (such as whether to select a certain candidate viewpoint, adjust the direction of the viewpoint, etc.), the action space has a hybrid characteristic. For the continuous part, the range of changes in the position and pose of the viewpoint in space is defined; for the discrete part, discrete actions such as selecting or not selecting a certain candidate viewpoint, adjusting the direction of the viewpoint, etc. are defined.

[0078] The reward function is designed based on multiple factors, such as the coverage degree of the new viewpoint to the workpiece model (positive reward is given if the newly obtained point cloud can fill in the blank area of the model), the redundancy between viewpoints (positive reward is given to avoid selecting too similar viewpoints and reduce data redundancy), and whether the field of view and occlusion constraints are met (see Figure 7 , Figure 7The dark points are unobstructed points, the light points are obstructed points, and a positive reward is given when the constraints are met and a negative reward is given when the constraints are violated, reachability of sensors (a positive reward is given when the viewpoint is within the effective working range of the sensor), and the like. These factors are combined to develop a reward function that can measure the pros and cons of viewpoint selection.

[0079] Preferably, the next viewpoint with the highest reward within a given number of iterations is searched from the candidate viewpoint set according to the epsilon-greedy strategy and the PPO algorithm, including:

[0080] The policy network uses the epsilon-greedy strategy to determine which direction to explore from the current viewpoint. With a probability of 1-epsilon, the direction with the highest estimated value is selected from all directions, and with a probability of epsilon, a direction is randomly selected from all directions. The next viewpoint is determined according to the exploration direction in the candidate viewpoint set;

[0081] When the next viewpoint is selected, the PPO algorithm fine-tunes the specific position and pose of the viewpoint. A series of actions are generated using the existing policy, and the policy is adjusted according to the rewards of these actions. The PPO algorithm limits the step size of policy update to prevent the policy from updating too much, thereby fine-tuning the specific position and pose of the viewpoint.

[0082] The reward calculation and action selection in the epsilon-greedy strategy and the PPO algorithm are all constrained by constraint conditions including field of view constraints, occlusion constraints, view angle constraints, data redundancy constraints, and sensor reachability constraints.

[0083] Preferably, the next viewpoint obtained is determined whether to be added to the current viewpoint list according to a viewpoint judgment mechanism, including:

[0084] The viewpoint judgment mechanism determines whether the next viewpoint obtained meets the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations. If the next viewpoint obtained meets the viewpoint judgment mechanism, it is added to the current viewpoint list.

[0085] If the viewpoint judgment mechanism is not met, the viewpoint is iteratively searched again by the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0086] Preferably, the current viewpoint list obtained is determined to be the final viewpoint list according to a viewpoint list judgment mechanism, further including:

[0087] The viewpoint list judgment mechanism determines whether the current viewpoint list meets the given three-dimensional reconstruction requirements and whether the number of viewpoints is the smallest within a given number of iterations. If the viewpoint list meets the viewpoint list judgment mechanism, it is determined that the final viewpoint list has been obtained.

[0088] If the constraint condition is not met, the current viewpoint column is recorded, and the viewpoints in the viewpoint column except the initial viewpoint are emptied. The depth reinforcement learning mechanism is iterated again to find the viewpoints before the iteration number reaches a given value.

[0089] Preferably, the planning result is obtained according to the final viewpoint column, including:

[0090] Each viewpoint in the final viewpoint column is taken as a target position. A six-axis collaborative robot is used to calculate a feasible path from the initial pose to the first viewpoint using a preset path planning. Specifically, in this embodiment, an A algorithm, a Dijkstra algorithm, an RRT (rapidly-exploring random tree) algorithm and its variants are used to find a feasible path.

[0091] When planning the path to the next viewpoint, the pose of the previous viewpoint is taken as a new starting pose, and the above path planning process is repeated until a complete path connecting all viewpoints is planned. Specifically, in this embodiment, the smoothness of the path is ensured during the planning process to avoid sharp changes in pose or sudden changes in speed during the movement of the robot, which may affect the stability and working accuracy of the robot.

[0092] Collision detection is performed on the planned path to check whether the robot will collide with obstacles or its own components in the workspace when moving along the path. If a collision is detected, the path is optimized and adjusted. Specifically, in this embodiment, methods such as re-planning the path, adjusting the position or pose of the path points, and the like are used to avoid collision areas until a collision-free feasible path is obtained.

[0093] Referring to Figure 3 The embodiment implements a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, and provides a three-dimensional reconstruction viewpoint planning device based on collaborative robot 3D vision, which includes an industrial computer, a six-axis collaborative robot, a 3D camera, and a digital twin system.

[0094] The digital twin system is loaded on an industrial computer, the industrial computer is loaded with all software programs required for running algorithms and processing data, is used for processing relevant data collected by sensors, executing algorithm programs, and is connected with a six-axis collaborative robot and a 3D camera, the 3D camera is fixed on the six-axis collaborative robot in a manner of "eyes on hands", the 3D camera is used for collecting point cloud information of a workpiece surface and is used for three-dimensional reconstruction, the six-axis collaborative robot drives the 3D camera to move in space with six degrees of freedom, so as to drive the viewpoint of the 3D camera to different poses, after visual calibration of the 3D camera fixed on the six-axis collaborative robot, a candidate viewpoint set is constructed on an existing or acquired standard model of the workpiece, then the next best viewpoint is iteratively searched through a deep reinforcement learning method, the digital twin system performs robot motion planning according to the result of viewpoint planning, and the viewpoint planning and robot motion planning are completed to perform multi-viewpoint three-dimensional reconstruction on the workpiece.

[0095] Second embodiment

[0096] In some embodiments of the present application, a computer device is also provided, comprising a memory and a processor, the memory has computer readable instructions stored therein, and the computer readable instructions are executed by the processor to enable the processor to perform the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision in an embodiment of the present application.

[0097] The present application also provides a storage medium having computer readable instructions stored therein, and the computer readable instructions are executed by one or more processors to enable the one or more processors to perform the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision in an embodiment of the present application.

[0098] It can be understood that, for the aforementioned three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, if all are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, the computer software product is stored in a storage medium, includes a plurality of instructions to enable a computer device (which can be a personal computer server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read only memory (Read Only memory, ROM), a random access memory (Random Access memory, RAM), a magnetic disk or an optical disk, and various program code storage media.

[0099] The computer readable storage medium can include a tangible medium such as those previously described. Additionally, or alternatively, the computer readable storage medium can represent a transmission medium such as those described herein.

[0100] The above merely illustrates the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical scheme falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principle of the present application shall be considered as the protection scope of the present application.

Claims

1. A method for 3D reconstruction view planning based on collaborative robot 3D vision, characterized in that, The method comprises the following steps: S1: visual calibration is performed on the 3D camera on the collaborative robot, the visual calibration comprises internal parameter calibration for determining the internal parameters of the camera according to a calibration board, hand-eye calibration and acquisition of an effective reconstruction visual angle threshold range, wherein the hand-eye calibration is to obtain a conversion relationship between a camera coordinate system and a six-axis collaborative robot coordinate system according to position information of an end effector of the six-axis collaborative robot and a pose relationship of a calibration board coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to acquire the effective visual angle threshold range; S2: it is judged whether there is a workpiece standard model, if the workpiece standard model exists, a candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, if the workpiece standard model does not exist, the workpiece standard model is established according to the camera internal parameters, the conversion relationship and the visual angle threshold range, and after the establishment, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model; S3: a deep reinforcement learning mechanism acquires a planning result according to the candidate viewpoint set, controls the actual six-axis collaborative robot to move according to the planning result, a 3D camera collects data at each viewpoint position to acquire 3D point cloud data of a workpiece under the visual angle, and the multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data.

2. The method of claim 1, wherein, In step S2, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, comprising: a model upper surface of the workpiece standard model is acquired through modeling software, uniform triangular mesh division is performed on the model upper surface by mesh division software to obtain divided meshes, and point cloud of the divided meshes is acquired; a normal direction of each vertex in the point cloud is calculated, and the normal direction is adjusted to ensure that the normal vector direction is perpendicular to the model upper surface and outward, Gaussian distribution sampling is performed along the vertex normal direction, and a candidate viewpoint is generated; the current coordinate system of the candidate viewpoint is converted to the robot coordinate system according to the conversion relationship, positions of the converted candidate viewpoints that are unreachable or close to singular points are filtered, the camera optical axis direction is calculated after the filtering, whether the camera angle is within the visual angle threshold range is verified according to the camera optical axis direction, and if yes, the final viewpoint set is added.

3. The method of claim 2, wherein, In step S2, the workpiece standard model is established according to the camera internal parameters, the conversion relationship and the visual angle threshold range, comprising: within the visual angle threshold range, the six-axis collaborative robot moves the camera according to a preset path to collect multi-angle point cloud data, the multi-angle point cloud data is preprocessed according to the camera internal parameters, and the multi-angle point cloud data is unified to the robot coordinate system to obtain converted point cloud through the conversion relationship; iterative closest point (ICP) or feature matching is adopted to adjust the position alignment point cloud between the converted point cloud to obtain spliced point cloud data; the spliced point cloud data is converted into a triangular mesh model, and the triangular mesh model is taken as the workpiece standard model.

4. The method of claim 3, wherein, In step S3, the deep reinforcement learning mechanism acquires a planning result according to the candidate viewpoint set, comprising: The deep reinforcement learning mechanism determines the number of iterations for obtaining a stable initial viewpoint and an acceptable viewpoint reward value through Monte-Carlo initialization evaluation experiments, and obtains the initial viewpoint; The deep reinforcement learning mechanism searches for a next viewpoint with the highest reward within a given number of iterations from the candidate viewpoint set according to an ε-greedy strategy and a PPO algorithm, judges whether to add the obtained next viewpoint to the current viewpoint list according to a viewpoint judgment mechanism, and obtains a final viewpoint list according to a viewpoint list judgment mechanism; The planning result is obtained according to the final viewpoint list.

5. The method of claim 4, wherein, The deep reinforcement learning mechanism searches for a next viewpoint with the highest reward within a given number of iterations from the candidate viewpoint set according to an ε-greedy strategy and a PPO algorithm, judges whether to add the obtained next viewpoint to the current viewpoint list according to a viewpoint judgment mechanism, and obtains a final viewpoint list according to a viewpoint list judgment mechanism; The policy network uses the ε-greedy strategy to decide which direction to explore from the current viewpoint, selects a direction with the highest current estimated value from all directions with a probability of 1-ε, and selects a direction randomly from all directions with a probability of ε, and determines a next viewpoint according to the exploration direction in the candidate viewpoint set; After the next viewpoint is selected, the PPO algorithm fine-tunes the specific position and pose of the viewpoint, generates a series of actions using the existing strategy, and adjusts the strategy according to the rewards of these actions; The reward calculation and action selection in the ε-greedy strategy and the PPO algorithm are both constrained by constraint conditions including field of view constraints, occlusion constraints, viewing angle constraints, data redundancy constraints, and sensor accessibility constraints.

6. The method of claim 5, wherein, The obtained next viewpoint is judged whether to be added to the current viewpoint list according to a viewpoint judgment mechanism, which includes: The viewpoint judgment mechanism judges whether the obtained next viewpoint meets the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations, and if the obtained next viewpoint meets the viewpoint judgment mechanism, it is added to the current viewpoint list; If the viewpoint judgment mechanism is not met, the deep reinforcement learning mechanism is iterated again to find the viewpoint before the number of iterations reaches a given value.

7. The method of claim 6, wherein, The obtained current viewpoint list obtains a final viewpoint list according to a viewpoint list judgment mechanism, which further includes: The viewpoint list judgment mechanism judges whether the current viewpoint list meets the given three-dimensional reconstruction requirements and whether the number of viewpoints is the smallest within a given number of iterations, and if the viewpoint list meets the viewpoint list judgment mechanism, it is determined that the final viewpoint list has been obtained; If the constraint conditions are not met, the current viewpoint list is recorded, and the viewpoint list is emptied except for the initial viewpoint, and the deep reinforcement learning mechanism is iterated again to find the viewpoint before the number of iterations reaches a given value.

8. The method of claim 7, wherein, The planning result is obtained according to the final viewpoint list, which includes: Each viewpoint in the final viewpoint list is taken as a target position, and a feasible path from the initial pose to the first viewpoint is calculated using a preset path planning for a six-axis collaborative robot; When planning the path to the next viewpoint, the pose of the previous viewpoint is taken as a new starting pose, and the above path planning process is repeated until a complete path connecting all viewpoints is planned. Collision detection is performed on the planned path to check whether the robot will collide with obstacles in the workspace or its own components when moving along the path, and the path is adjusted optimally if a collision is detected.

9. A computer device, comprising: The computer readable instructions, when executed by the one or more processors, cause the one or more processors to perform the steps of the method for planning a view point for 3D reconstruction based on collaborative robot 3D vision as claimed in any one of claims 1 to 8.

10. A storage medium storing computer readable instructions, wherein, The computer readable instructions, when executed by the one or more processors, cause the one or more processors to perform the steps of the method for planning a view point for 3D reconstruction based on collaborative robot 3D vision as claimed in any one of claims 1 to 8.

Images