Three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision

Abstract

The invention relates to the technical field of robot vision, in particular to a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, which comprises the following steps: S1, performing visual calibration on a 3D camera on a collaborative robot; s2, whether a workpiece standard model exists or not is judged, if yes, a candidate viewpoint set is constructed according to the workpiece standard model, and if not, the workpiece standard model is established according to preset fixed multiple viewpoints, and then the candidate viewpoint set is constructed; and S3, a deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, controls an actual six-axis collaborative robot to move according to the planning result, carries out data acquisition, obtains 3D point cloud data of the workpiece at the visual angle, carries out preprocessing including noise point removal and filtering, and generates a three-dimensional model of the workpiece based on the fused point cloud data. The three-dimensional reconstruction method is high in automation degree, almost does not need manual operation in the whole process, and can effectively perform efficient and rapid three-dimensional reconstruction on a large batch of different types of non-standard customized small-batch workpieces.

Description

Technical Field

[0001] The present invention relates to the field of robot vision technology, and in particular to a three-dimensional reconstruction viewpoint planning method, device and storage medium based on collaborative robot 3D vision. Background Art

[0002] In the wave of rapid development of modern industry, digital twin technology has become an indispensable part of manufacturing and intelligent manufacturing systems, and has been widely used in industrial manufacturing, urban management, aerospace, medical and other fields. Among the many key technologies involved in digital twin technology, 3D reconstruction technology undoubtedly occupies a pivotal position.

[0003] With the development of digital twin technology, there are more and more diverse and high-standard demands for 3D reconstruction technology. The increasing complexity of industrial manufacturing processes has led to the need for 3D reconstruction technology to develop in the direction of automation to improve efficiency and reduce human intervention. Traditional manual data collection methods can no longer meet the needs of efficiency and accuracy, so automated viewpoint planning has become a key technology to improve the quality of 3D reconstruction. The core of viewpoint planning is to automatically select the best shooting position and angle to ensure that the acquired point cloud data can cover the workpiece surface as much as possible. Reasonable viewpoint planning can significantly improve the reconstruction integrity and accuracy of complex workpieces and improve production efficiency, which is particularly important for the application of 3D reconstruction technology in digital twins. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and provide a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, comprising the following steps: S1: Perform visual calibration on the 3D camera on the collaborative robot, the visual calibration includes determining the intrinsic parameter calibration of the camera according to the calibration plate, hand-eye calibration and obtaining the effective reconstruction viewing angle threshold range, wherein the hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the end effector of the six-axis collaborative robot and the position relationship of the calibration plate coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to obtain the effective viewing angle threshold range; S2: Determine whether there is a workpiece standard model. If there is the workpiece standard model, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model. If there is no workpiece standard model, the workpiece standard model is established according to the camera intrinsic parameters, the conversion relationship and the viewing angle threshold range. After the establishment, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model. S3: The deep reinforcement learning mechanism obtains the planning results according to the candidate viewpoint set, controls the actual movement of the six-axis collaborative robot according to the planning results, and the 3D camera collects data at each viewpoint position to obtain the 3D point cloud data of the workpiece at the viewing angle. The multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data; Unless otherwise specified, all viewpoints mentioned are generalized viewpoints that include position and posture.

[0005] Preferably, in step S2, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, including: The upper surface of the model of the standard workpiece model is obtained by the modeling software, and the meshing software performs uniform triangular meshing on the upper surface of the model to obtain a divided mesh, and obtains a point cloud of the divided mesh; Calculating the normal direction of each vertex in the point cloud, adjusting the normal direction to ensure that the normal vector direction is perpendicular to the upper surface of the model and points outward, performing Gaussian distribution sampling along the vertex normal direction, and generating candidate viewpoints; According to the conversion relationship, the current coordinate system of the candidate viewpoint is converted to the robot coordinate system, and the positions of the converted candidate viewpoints that are unreachable or close to singular points are filtered out. After filtering, the direction of the camera optical axis is calculated, and the camera angle is verified according to the direction of the camera optical axis whether it is within the viewing angle threshold range. If so, it is added to the final viewpoint set.

[0006] Preferably, in step S2, establishing the workpiece standard model according to the camera intrinsic parameter, the conversion relationship and the viewing angle threshold range includes: Within the viewing angle threshold range, the six-axis collaborative robot moves the camera along a preset path to collect multi-view point cloud data, pre-processes the multi-view point cloud data according to the camera internal parameters, and unifies the multi-view point cloud data into the robot coordinate system through the conversion relationship to obtain a converted point cloud; Adopting iterative closest point ICP or feature matching to adjust the positions of the converted point clouds and align the point clouds to obtain spliced ​​point cloud data; The stitching point cloud data is converted into a triangular mesh model, and the triangular mesh model is used as the workpiece standard model.

[0007] Preferably, in step S3, the deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, including: The deep reinforcement learning mechanism determines the number of iterations to obtain a stable initial viewpoint and an acceptable viewpoint reward value through a Monte-Carlo initialization evaluation experiment, and simultaneously obtains the initial viewpoint; The deep reinforcement learning mechanism searches the candidate viewpoint set for the next viewpoint with the highest reward within a given number of iterations according to the ε-greedy strategy and the PPO algorithm, and determines whether to add the next viewpoint to the current viewpoint column according to the viewpoint judgment mechanism, and obtains the final viewpoint column according to the viewpoint column judgment mechanism; The planning result is obtained according to the final viewpoint sequence.

[0008] Preferably, searching the candidate viewpoint set for the next viewpoint with the highest reward within a given number of iterations according to the ε-greedy strategy and the PPO algorithm includes: The policy network uses the ε-greedy strategy to determine which direction of the current viewpoint to explore, selects the direction with the highest current estimated value from all directions with a probability of 1-ε, randomly selects a direction from all directions with a probability of ε, and determines the next viewpoint in the candidate viewpoint set according to the exploration direction; After selecting the next viewpoint, the PPO algorithm fine-tunes the specific position and posture of the viewpoint, generates a series of actions using the existing strategy, and adjusts the strategy based on the rewards of these actions; Among them, the reward calculation and action selection in the ε-greedy strategy and the PPO algorithm are subject to constraints including field of view constraints, occlusion constraints, viewing angle constraints, data redundancy constraints, and sensor reachability constraints.

[0009] Preferably, the next viewpoint obtained is judged whether to add it to the current viewpoint list according to the viewpoint judgment mechanism, including: The viewpoint judgment mechanism judges whether the next viewpoint obtained satisfies the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations. If the next viewpoint obtained satisfies the viewpoint judgment mechanism, it is added to the current viewpoint column; If the viewpoint judgment mechanism is not satisfied, the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0010] Preferably, the current viewpoint sequence obtained is used to obtain a final viewpoint sequence according to a viewpoint sequence determination mechanism, further comprising: The viewpoint sequence judgment mechanism judges whether the current viewpoint sequence meets the given 3D reconstruction requirements and whether the number of viewpoints is the minimum within a given number of iterations. If the viewpoint sequence meets the viewpoint sequence judgment mechanism, it is determined that the final viewpoint sequence has been obtained. If the constraint condition is not met, after recording the current viewpoint sequence, other viewpoints in the viewpoint sequence except the initial viewpoint are cleared, and the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0011] Preferably, obtaining the planning result according to the final viewpoint sequence includes: Taking each viewpoint in the final viewpoint list as the target position, calculating the feasible path from the current pose to the first viewpoint of the six-axis collaborative robot starting from the initial pose, and using the preset path planning; When planning the path to the next viewpoint, the posture of the previous viewpoint is used as the new starting posture, and the above path planning process is repeated until a complete path connecting all viewpoints is planned; Perform collision detection on the planned path to check whether the robot will collide with obstacles or its own parts in the workspace when moving along the path. If a collision is detected, the path is optimized and adjusted.

[0012] Based on the same concept, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision as described in any one of the embodiments.

[0013] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision as described in any one of the embodiments.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention determines whether there is a workpiece standard model. If there is a workpiece standard model, the candidate viewpoint set construction module constructs a viewpoint set according to the workpiece standard model. If there is no workpiece standard model, the workpiece standard model is established according to the camera internal parameters, the conversion relationship and the viewing angle threshold range. After the establishment, the candidate viewpoint set construction module constructs the viewpoint set according to the workpiece standard model. By considering the two situations of having and not having a workpiece standard model, the present invention has high flexibility and can cope with different working scenarios. The present invention adopts a deep reinforcement learning method mechanism to obtain planning results according to a candidate viewpoint set, wherein the strategy network module can be replaced and modified in a targeted manner for different types of tasks, and has strong scalability; The present invention obtains planning results according to a candidate viewpoint set through a deep reinforcement learning mechanism, controls the actual movement of the six-axis collaborative robot according to the planning results, collects data at each viewpoint position with a 3D camera, obtains 3D point cloud data of the workpiece at that viewpoint, performs preprocessing including noise point removal and filtering on the multi-viewpoint 3D point cloud data, and generates a three-dimensional model of the workpiece based on the fused point cloud data. The present invention has a high degree of automation and requires almost no manual operation throughout the process, and can effectively and efficiently perform three-dimensional reconstruction on a large number of different types of non-standard customized small batch workpieces. The present invention adjusts the goal of three-dimensional reconstruction by changing the reward function, selectively realizes global reconstruction or focused local reconstruction, and has high flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0016] Figure 1 This is a flow chart of a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision of the present invention; Figure 2 Another flow chart of the three-dimensional reconstruction viewpoint planning method based on the collaborative robot 3D vision of the present invention; Figure 3 This is a structural diagram of the six-axis collaborative robot of the present invention; Figure 4 A mesh map of the workpiece surface of the three-dimensional reconstruction viewpoint planning method based on the 3D vision of the collaborative robot according to the present invention; Figure 5 A schematic diagram of constructing a candidate viewpoint set by Gaussian random sampling of a three-dimensional reconstruction viewpoint planning method based on 3D vision of a collaborative robot according to the present invention; Figure 6 It is a schematic diagram of field visualization of the three-dimensional reconstruction viewpoint planning method based on the 3D vision of the collaborative robot of the present invention; Figure 7 It is a schematic diagram of occlusion discrimination of the three-dimensional reconstruction viewpoint planning method based on the 3D vision of the collaborative robot according to the present invention.

[0017] Reference numerals: 1: Six-axis collaborative robot, 2: 3D camera, 3: tool, 4: target workpiece. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0019] It is understood by those skilled in the art that, unless otherwise stated, when an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or indirectly connected to the other element. In addition, connection may be used for fixing or for circuit connection.

[0020] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0021] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0022] First embodiment See also Figure 1 and Figure 2 As shown, the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision provided in this embodiment includes the following steps: Unless otherwise specified, all viewpoints mentioned in this embodiment are generalized viewpoints that include position and posture.

[0023] S1: Perform visual calibration on the 3D camera on the collaborative robot. The visual calibration includes the intrinsic calibration of the camera based on the calibration plate, the hand-eye calibration and the acquisition of the effective reconstruction viewing angle threshold range. The hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system based on the position information of the six-axis collaborative robot's end effector and the posture relationship of the calibration plate coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to obtain the effective viewing angle threshold range.

[0024] The intrinsic parameter calibration of the camera based on the calibration plate is to select a suitable calibration plate, change multiple positions and angles in front of the 3D camera, collect a large number of images containing the calibration plate, use algorithms such as Zhang Zhengyou calibration method, detect the feature points of the calibration plate, and accurately calculate the camera's internal parameters including focal length, principal point coordinates, and distortion coefficient based on the correspondence between the image coordinates of the feature points and the actual world coordinates.

[0025] The hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the end effector of the six-axis collaborative robot and the position and posture relationship of the calibration plate coordinate system relative to the camera coordinate system, and further includes: firmly mounting the 3D camera on the manipulator, controlling the manipulator to move in different positions, and synchronously recording the manipulator position and posture information (such as joint angles) and the image taken by the 3D camera. Specifically, in this embodiment, please refer to Figure 6 As shown, the best working distance of the 3D camera provided is 500mm, and the field of view distance range is 250mm~750mm. Figure 6 The ray represents the current viewpoint position and line of sight direction, and the pyramid geometry constructed around the ray and composed of points and lines is the field of view of the 3D camera.

[0026] S2: Determine whether there is a workpiece standard model. If there is a workpiece standard model, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model. If there is no workpiece standard model, the workpiece standard model is established according to the camera intrinsic parameters, conversion relationship and viewing angle threshold range. After the establishment, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model.

[0027] Preferably, in step S2, a workpiece standard model is established according to the camera intrinsic parameters, the conversion relationship and the viewing angle threshold range, including: Within the viewing angle threshold range, the six-axis collaborative robot moves the camera along a preset path to collect multi-view point cloud data, pre-processes the multi-view point cloud data according to the camera's internal parameters, and unifies the multi-view point cloud data into the robot coordinate system through a transformation relationship to obtain a transformed point cloud; Adopt iterative closest point ICP or feature matching to adjust the position of the transformed point clouds and align the point clouds to obtain the spliced ​​point cloud data; The stitched point cloud data is converted into a triangular mesh model, and the triangular mesh model is used as the standard model of the workpiece.

[0028] Preferably, in step S2, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, including: See also Figure 4 As shown, the upper surface of the model of the workpiece standard model is obtained by using modeling software (SolidWorks or other software that can operate the standard model), and the meshing software (Abaqus or other software that can mesh the model) performs uniform triangular meshing on the upper surface of the model to obtain a mesh, and obtains a point cloud of the mesh; See also Figure 5 As shown, the normal direction of each vertex in the point cloud is calculated, and the normal direction is adjusted to ensure that the normal vector direction is perpendicular to the upper surface of the model and points outward. Gaussian distribution sampling is performed along the vertex normal direction to generate candidate viewpoints. Specifically, in this embodiment, for each point in the point cloud, the point in the local neighborhood is used to calculate its normal direction. During the calculation process, it may be necessary to adjust the size of the neighborhood and the calculation method according to the density and distribution of the point cloud to ensure the accuracy of the normal direction. The center of the point cloud is taken as the mean, and the variance is determined according to the range of the point cloud and the distribution of the normal. A series of candidate viewpoints are sampled in space, that is, Gaussian random sampling is performed according to the set parameters near the optimal working distance of the 3D camera and within the range that meets the field of view distance of the 3D camera to construct candidate viewpoints. This embodiment does not limit the specific sampling method, nor does it limit the relevant parameters of Gaussian random sampling or other sampling methods. In actual scenes, operators can flexibly choose according to on-site conditions; According to the conversion relationship, the current coordinate system of the candidate viewpoint is converted to the robot coordinate system, and the positions of the converted candidate viewpoints that are unreachable or close to singular points are filtered out. After filtering, the direction of the camera optical axis is calculated, and the camera angle is verified based on the direction of the camera optical axis to see if it is within the viewing angle threshold range. If so, it is added to the final viewpoint set.

[0029] S3: The deep reinforcement learning mechanism obtains the planning results based on the candidate viewpoint set, and controls the actual movement of the six-axis collaborative robot based on the planning results. The 3D camera collects data at each viewpoint position to obtain the 3D point cloud data of the workpiece at that perspective. The multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data.

[0030] Preferably, in step S3, the deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, including: The deep reinforcement learning mechanism uses Monte-Carlo initialization evaluation experiments to determine the number of iterations to obtain a stable initial viewpoint and an acceptable viewpoint reward value, and obtains the initial viewpoint at the same time; The deep reinforcement learning mechanism searches for the next viewpoint with the highest reward within a given number of iterations in the candidate viewpoint set according to the ε-greedy strategy and the PPO algorithm. The next viewpoint obtained is determined by the viewpoint judgment mechanism to determine whether to add it to the current viewpoint column. The current viewpoint column obtained is used to obtain the final viewpoint column according to the viewpoint column judgment mechanism. Get the planning result based on the final viewpoint column.

[0031] The state space includes the current robot's posture information, the acquired point cloud, the existing viewpoint information, the relevant geometric information of the workpiece model (such as surface normal vector, curvature, etc.), the visible surface points in the current state, the rewards in the current state, and the obstacle information in the workspace, etc. This information is reasonably encoded and integrated to form a state space, so that the deep reinforcement learning mechanism can perceive the current environmental state.

[0032] The action space is mainly about selecting the next viewpoint. Since the position and posture of the viewpoint can be represented by continuous parameters (such as three-dimensional coordinates and rotation angles), and there are discrete decisions at certain discrete decision points (such as whether to select a candidate viewpoint, adjust the direction of the viewpoint, etc.), the action space has a mixed characteristic. For the continuous part, the range of variation of the position and posture of the viewpoint in space is defined; for the discrete part, discrete actions such as selecting or not selecting a candidate viewpoint, adjusting the direction of the viewpoint, etc. are defined.

[0033] The reward function is designed based on multiple factors, such as the coverage of the workpiece model by the new viewpoint (a positive reward is given if the newly acquired point cloud can fill the blank area of ​​the model), the redundancy between viewpoints (avoid selecting viewpoints that are too similar, reduce data redundancy and give positive rewards), whether the field of view and occlusion constraints are met (see Figure 7 As shown, Figure 7 The darker points are unobstructed points, and the lighter points are obstructed points. Positive rewards are given if the constraints are met, and negative rewards are given if they are violated), sensor reachability (positive rewards are given if the viewpoint is within the effective working range of the sensor), etc. By combining these factors, a reward function that can measure the quality of viewpoint selection is formulated.

[0034] Preferably, searching for the next viewpoint with the highest reward within a given number of iterations in the candidate viewpoint set according to the ε-greedy strategy and the PPO algorithm includes: The policy network uses the ε-greedy strategy to decide which direction to explore from the current viewpoint. It selects the direction with the highest estimated value from all directions with a probability of 1-ε, randomly selects a direction from all directions with a probability of ε, and determines the next viewpoint in the candidate viewpoint set based on the explored direction. After selecting the next viewpoint, the PPO algorithm fine-tunes the specific position and posture of the viewpoint, generates a series of actions using the existing strategy, and adjusts the strategy based on the rewards of these actions. The PPO algorithm limits the step size of the strategy update to prevent the strategy update from being too large, thereby achieving fine-tuning of the specific position and posture of the viewpoint; Among them, the reward calculation and action selection in the ε-greedy strategy and PPO algorithm are subject to constraints including field of view constraints, occlusion constraints, viewing angle constraints, data redundancy constraints, and sensor reachability constraints.

[0035] Preferably, the next viewpoint obtained is judged whether to add it to the current viewpoint list according to the viewpoint judgment mechanism, including: The viewpoint judgment mechanism judges whether the next viewpoint obtained satisfies the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations. If the next viewpoint obtained satisfies the viewpoint judgment mechanism, it is added to the current viewpoint column; If the viewpoint judgment mechanism is not satisfied, the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0036] Preferably, obtaining the current viewpoint sequence according to the viewpoint sequence determination mechanism to obtain the final viewpoint sequence further includes: The viewpoint column judgment mechanism judges whether the current viewpoint column meets the given 3D reconstruction requirements and whether the number of viewpoints is the minimum within a given number of iterations. If the viewpoint column meets the viewpoint column judgment mechanism, it is determined that the final viewpoint column has been obtained. If the constraints are not met, after recording the current viewpoint column, all viewpoints except the initial viewpoint in the viewpoint column are cleared, and the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

[0037] Preferably, obtaining the planning result according to the final viewpoint sequence includes: Each viewpoint in the final viewpoint column is taken as the target position, and the six-axis collaborative robot starts from the initial posture, and uses the preset path planning to calculate a feasible path from the current posture to the first viewpoint. Specifically, in this embodiment, a feasible path is found according to the A algorithm, the Dijkstra algorithm, the RRT (fast random tree) algorithm and its variants; When planning the path to the next viewpoint, the posture of the previous viewpoint is used as the new starting posture, and the above path planning process is repeated until a complete path connecting all viewpoints is planned. Specifically, in this embodiment, during the planning process, the smoothness of the path is ensured to avoid drastic posture changes or sudden changes in speed during the movement of the robot, which may affect the stability and working accuracy of the robot; The planned path is subjected to collision detection to check whether the robot will collide with obstacles or its own parts in the workspace when moving along the path. If a collision is detected, the path is optimized and adjusted. Specifically, in this embodiment, methods such as replanning the path and adjusting the position or posture of the path points are adopted to avoid the collision area until a feasible path without collision is obtained.

[0038] See also Figure 3 As shown, this embodiment implements a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, and provides a three-dimensional reconstruction viewpoint planning device based on collaborative robot 3D vision including an industrial computer, a six-axis collaborative robot, a 3D camera, and a digital twin system; The digital twin system is loaded on an industrial computer, which is loaded with all the software programs required to run algorithms and process data. It is used to process relevant data collected by sensors and execute algorithm programs. The industrial computer is connected to the six-axis collaborative robot and a 3D camera. The 3D camera is fixed on the six-axis collaborative robot in an "eye-in-hand" manner. The 3D camera is used to collect point cloud information on the surface of the workpiece for three-dimensional reconstruction. The six-axis collaborative robot drives the 3D camera to move in space with six degrees of freedom to drive the viewpoint of the 3D camera to different postures. After visual calibration of the 3D camera fixed on the six-axis collaborative robot, a set of candidate viewpoints is constructed on the existing or acquired workpiece standard model, and then the next best viewpoint is continuously iterated through the deep reinforcement learning method. The digital twin system performs robot motion planning based on the results of viewpoint planning, completes viewpoint planning and robot motion planning, and performs multi-viewpoint three-dimensional reconstruction of the workpiece.

[0039] Second embodiment In some embodiments of the present application, a computer device is also provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision in an embodiment of the present invention.

[0040] The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of a three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision in an embodiment of the present invention.

[0041] It is understandable that, for the aforementioned three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, if it is implemented in the form of software function modules and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0042] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0043] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision, characterized in that: The following steps are involved: S1: Perform visual calibration on the 3D camera on the collaborative robot, the visual calibration includes determining the intrinsic parameter calibration of the camera according to the calibration plate, hand-eye calibration and obtaining the effective reconstruction viewing angle threshold range, wherein the hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the six-axis collaborative robot coordinate system according to the position information of the end effector of the six-axis collaborative robot and the position relationship of the calibration plate coordinate system relative to the camera coordinate system, and the camera performs multi-angle scanning to obtain the effective viewing angle threshold range; S2: Determine whether there is a workpiece standard model. If there is the workpiece standard model, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model. If there is no workpiece standard model, the workpiece standard model is established according to the camera intrinsic parameters, the conversion relationship and the viewing angle threshold range. After the establishment, the candidate viewpoint set construction module constructs the candidate viewpoint set according to the workpiece standard model. S3: The deep reinforcement learning mechanism obtains the planning results according to the candidate viewpoint set, controls the actual movement of the six-axis collaborative robot according to the planning results, and the 3D camera collects data at each viewpoint position to obtain the 3D point cloud data of the workpiece at this perspective. The multi-viewpoint 3D point cloud data is preprocessed including removing noise points and filtering, and a three-dimensional model of the workpiece is generated based on the fused point cloud data.

2. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 1 is characterized in that: In step S2, the candidate viewpoint set construction module constructs a candidate viewpoint set according to the workpiece standard model, including: The upper surface of the model of the standard workpiece model is obtained by the modeling software, and the meshing software performs uniform triangular meshing on the upper surface of the model to obtain a divided mesh, and obtains a point cloud of the divided mesh; Calculating the normal direction of each vertex in the point cloud, adjusting the normal direction to ensure that the normal vector direction is perpendicular to the upper surface of the model and points outward, performing Gaussian distribution sampling along the vertex normal direction, and generating candidate viewpoints; According to the conversion relationship, the current coordinate system of the candidate viewpoint is converted to the robot coordinate system, and the positions of the converted candidate viewpoints that are unreachable or close to singular points are filtered out. After filtering, the direction of the camera optical axis is calculated, and the camera angle is verified according to the direction of the camera optical axis whether it is within the viewing angle threshold range. If so, it is added to the final viewpoint set.

3. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 2 is characterized in that: In step S2, the workpiece standard model is established according to the camera intrinsic parameters, the conversion relationship and the viewing angle threshold range, including: Within the viewing angle threshold range, the six-axis collaborative robot moves the camera along a preset path to collect multi-view point cloud data, pre-processes the multi-view point cloud data according to the camera internal parameters, and unifies the multi-view point cloud data into the robot coordinate system through the conversion relationship to obtain a converted point cloud; Adopting iterative closest point ICP or feature matching to adjust the positions of the converted point clouds and align the point clouds to obtain spliced ​​point cloud data; The stitching point cloud data is converted into a triangular mesh model, and the triangular mesh model is used as the workpiece standard model.

4. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 3 is characterized in that: In step S3, the deep reinforcement learning mechanism obtains a planning result according to the candidate viewpoint set, including: The deep reinforcement learning mechanism determines the number of iterations to obtain a stable initial viewpoint and an acceptable viewpoint reward value through a Monte-Carlo initialization evaluation experiment, and simultaneously obtains the initial viewpoint; The deep reinforcement learning mechanism searches the candidate viewpoint set for the next viewpoint with the highest reward within a given number of iterations according to the ε-greedy strategy and the PPO algorithm, and determines whether to add the next viewpoint to the current viewpoint column according to the viewpoint judgment mechanism, and obtains the final viewpoint column according to the viewpoint column judgment mechanism; The planning result is obtained according to the final viewpoint sequence.

5. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 4 is characterized in that: Searching the next viewpoint with the highest reward within a given number of iterations in the candidate viewpoint set according to the ε-greedy strategy and the PPO algorithm includes: The policy network uses the ε-greedy strategy to determine which direction of the current viewpoint to explore, selects the direction with the highest current estimated value from all directions with a probability of 1-ε, randomly selects a direction from all directions with a probability of ε, and determines the next viewpoint in the candidate viewpoint set according to the exploration direction; After selecting the next viewpoint, the PPO algorithm fine-tunes the specific position and posture of the viewpoint, generates a series of actions using the existing strategy, and adjusts the strategy based on the rewards of these actions; Among them, the reward calculation and action selection in the ε-greedy strategy and the PPO algorithm are subject to constraints including field of view constraints, occlusion constraints, viewing angle constraints, data redundancy constraints, and sensor reachability constraints.

6. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 5 is characterized in that: The next viewpoint obtained is determined according to the viewpoint determination mechanism to determine whether to add it to the current viewpoint list, including: The viewpoint judgment mechanism judges whether the next viewpoint obtained satisfies the constraint conditions, whether the reward value reaches an acceptable value, and whether the reward value is the highest within a given number of iterations. If the next viewpoint obtained satisfies the viewpoint judgment mechanism, it is added to the current viewpoint column; If the viewpoint judgment mechanism is not satisfied, the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

7. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 6 is characterized in that: The obtained current viewpoint sequence obtains a final viewpoint sequence according to a viewpoint sequence determination mechanism, further comprising: The viewpoint sequence judgment mechanism judges whether the current viewpoint sequence meets the given 3D reconstruction requirements and whether the number of viewpoints is the minimum within a given number of iterations. If the viewpoint sequence meets the viewpoint sequence judgment mechanism, it is determined that the final viewpoint sequence has been obtained. If the constraint condition is not met, after recording the current viewpoint sequence, other viewpoints in the viewpoint sequence except the initial viewpoint are cleared, and the viewpoint is iteratively searched again through the deep reinforcement learning mechanism before the number of iterations reaches a given value.

8. The three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision according to claim 7, characterized in that: Acquiring the planning result according to the final viewpoint sequence includes: Taking each viewpoint in the final viewpoint list as the target position, calculating the feasible path from the current pose to the first viewpoint of the six-axis collaborative robot starting from the initial pose, and using the preset path planning; When planning the path to the next viewpoint, the posture of the previous viewpoint is used as the new starting posture, and the above path planning process is repeated until a complete path connecting all viewpoints is planned; Perform collision detection on the planned path to check whether the robot will collide with obstacles or its own parts in the workspace when moving along the path. If a collision is detected, the path is optimized and adjusted.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision as described in any one of claims 1 to 8.

10. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional reconstruction viewpoint planning method based on collaborative robot 3D vision as described in any one of claims 1 to 8.

Images