A self-calibration three-dimensional reconstruction device and method for intelligent spraying

Abstract

The application discloses a kind of self-calibration three-dimensional reconstruction device and method for intelligent spraying, device includes control module, motion amount acquisition module, holder and binocular vision system, holder and binocular vision system can be linear motion along three axes and rotate motion around Z linear and horizontal linear, two cameras can be mutual close or separate horizontal linear motion and rotary motion, the focal length and aperture of each camera can be adjusted;Motion amount acquisition module collects the motion amount of holder and camera, the adjustment amount of focal length and aperture, control module receives the data collected by motion amount acquisition module and camera;Method generates multi-view point cloud by collecting images of multiple viewing angles and automatically calibrating camera internal parameter, distortion parameter and external parameter, and is registered to the same viewing angle to carry out three-dimensional reconstruction.The application can realize high-precision three-dimensional reconstruction of arbitrary viewing angle and orientation according to the shape and size of different to-be-sprayed workpieces, with the advantages of large working space and high degree of freedom, high efficiency.

Description

Technical Field

[0001] This invention relates to the field of workpiece three-dimensional reconstruction, specifically a self-calibrating three-dimensional reconstruction device and method for intelligent spraying. Background Technology

[0002] With the development of computer vision technology, 3D reconstruction technology is gradually being applied in industrial production. Among them, point cloud data, which uses a binocular structured light system to collect geometric information of the workpiece surface, is a non-contact, active, fast, and stable 3D reconstruction method. Point cloud is a simple and intuitive data that can express the three-dimensional information of the surface of the workpiece to be coated. Intelligent coating robots need to acquire the surface information of the workpiece to be coated, and then combine the workpiece surface point cloud with the coating process to automatically generate the coating trajectory, ultimately completing the intelligent coating task.

[0003] Camera calibration is an essential step in 3D reconstruction, involving determining both internal camera parameters (such as principal point coordinates and focal length) and external parameters (such as camera rotation and translation). Currently, 3D reconstruction systems based on binocular cameras require recalibration of these parameters after changes to the camera's internal structural parameters (such as focal length and aperture) and the relative position of the binocular cameras. In practical applications, when dealing with workpieces of different sizes and types, and needing to perform 3D reconstruction of workpiece edges or areas requiring high precision, the camera's position, focal length, and aperture must be adjusted. However, ordinary binocular cameras cannot automatically adjust their relative position, focal length, and aperture during operation; these adjustments must be made manually, and the camera's internal and external parameters must be recalibrated afterward. This causes significant inconvenience for 3D reconstruction of multiple types of workpieces in small batches, reducing work efficiency. Since single-viewpoint point clouds often cannot reflect complete surface information of the workpiece, after reconstructing point clouds from multiple viewpoints using binocular structured light technology, these point clouds need to be registered to a reference viewpoint. The ICP algorithm is widely used for point cloud registration due to its simplicity and low computational complexity. However, it requires that the initial positions of the two point clouds be sufficiently close, otherwise it is prone to getting trapped in local maxima.

[0004] Patent application number "202010641003.9" proposes a three-dimensional measurement device and method based on binocular camera imaging. This method uses a zoom camera and a high-order polynomial fitting method to determine the mapping model between the camera parameters and the focal length. However, this method does not consider the influence of aperture on camera parameters, and high-order polynomials are prone to overfitting. Furthermore, although the camera position and angle can be manually adjusted, the adjustable space is small, and the degrees of freedom are low. Once adjusted, the camera cannot be moved again during the entire measurement process; otherwise, the external parameters need to be recalibrated.

[0005] Patent application number "202210045927.1" proposes a method for determining the external parameters of a stereo camera. This method assumes that the camera rotates around its own coordinate axis and that the rotation center line of the turntable coincides with the camera's coordinate axis, thus simplifying the relationship of the stereo camera's position change. However, it does not take into account the phenomenon that in practical applications, the rotation center line of the turntable often does not coincide with the camera's own coordinate axis.

[0006] Patent application number "201810310517.9" ​​proposes a three-dimensional measurement device and method. This method calibrates the rotation and translation matrices of the camera at eight different positions and registers the point clouds at each position to obtain a complete point cloud of the object being measured. However, this method acquires point clouds at eight fixed positions for any type of object, making it difficult to guarantee reconstruction accuracy for some special types and sizes of objects.

[0007] Therefore, when it is necessary to acquire three-dimensional information of the surface of multiple varieties of small batches of workpieces to be sprayed to complete intelligent spraying tasks, it is particularly important to develop a multi-view three-dimensional reconstruction device and method with a large working space, multiple reconstruction perspectives, high accuracy, and the ability to automatically calibrate the camera's intrinsic and extrinsic parameters after changes in camera posture and internal parameters. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a self-calibrated 3D reconstruction device and method for intelligent spraying, so as to solve the problems that the camera cannot be automatically calibrated when the workpiece to be sprayed is reconstructed in 3D based on point cloud, and the high requirements for the initial position of point cloud in traditional ICP algorithm in point cloud registration.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A self-calibrating 3D reconstruction device for intelligent spraying includes a gimbal, a binocular vision system for acquiring images of the surface of the target to be sprayed, a control module, and a motion acquisition module. The binocular vision system includes two cameras. One side of the gimbal serves as the mounting surface, and the two cameras are mounted on this surface, with the X-axis of each camera's coordinate system perpendicular to the mounting surface. The gimbal can move linearly along the X, Y, and Z directions, and rotate about the Z-axis as its rotation center line. This allows the binocular vision system to move linearly along the X, Y, and Z directions along with the gimbal as a whole. The gimbal can rotate around a Z-axis straight line as its rotation center line, and can also rotate around a horizontal straight line as its rotation center line to adjust the pitch angle, thereby making the pitch angle of the two cameras in the binocular vision system adjustable; the two cameras can move horizontally towards or away from each other on the gimbal platform, thereby making the distance between the two cameras adjustable, and thus the baseline distance between the two cameras adjustable; each camera can rotate around a straight line perpendicular to the gimbal mounting surface as its rotation center line, thereby making the angle between the optical axes of the two cameras adjustable, and the focal length and aperture of each camera adjustable;

[0011] The motion acquisition module collects the motion of the gimbal and each camera, as well as the adjustment of the focal length and aperture of each camera. The motion acquisition module is electrically connected to the control module, and the two cameras are electrically connected to the control module respectively. The control module receives the data collected by the motion acquisition module and the image data collected by the cameras, and processes the received data to perform three-dimensional reconstruction of the target to be sprayed.

[0012] Furthermore, the gimbal is mounted on the pitch mechanism, the pitch mechanism is mounted on the rotating platform mechanism, and the rotating platform mechanism is mounted on the spatial movement mechanism. The spatial movement mechanism drives the rotating platform mechanism, the pitch mechanism, and the gimbal as a whole to move in straight lines in the X, Y, and Z directions. The rotating platform mechanism drives the pitch adjustment mechanism and the gimbal as a whole to rotate around the Z-direction straight line as the rotation center line. The pitch adjustment mechanism drives the gimbal to rotate around the horizontal straight line as the rotation center line.

[0013] Furthermore, the spatial movement mechanism includes a ball screw three-axis motion platform mechanism, the moving part of which can move linearly in the X, Y, and Z directions.

[0014] Furthermore, the rotary platform mechanism includes a first worm gear mechanism, which is integrally mounted on the moving part of the ball screw three-axis motion platform mechanism.

[0015] Furthermore, the pitch mechanism includes a rotating shaft and a motor that drives the rotating shaft to rotate. The motor and the rotating shaft are connected to the first worm gear mechanism through the same connecting plate, and the gimbal is connected to the rotating shaft.

[0016] Furthermore, the mounting surface of the gimbal is equipped with a baseline adjustment mechanism. Each camera is mounted on the baseline adjustment mechanism via a convergence angle adjustment mechanism. The baseline adjustment mechanism drives the two convergence angle adjustment mechanisms and the corresponding camera to move horizontally in a straight line, either moving closer to or separating from each other. The convergence angle adjustment mechanism also drives the corresponding camera to rotate around a straight line perpendicular to the gimbal mounting surface as its rotation center line.

[0017] Furthermore, the baseline adjustment mechanism includes two sets of ball screw linear motion platform mechanisms, and the moving parts of the two sets of ball screw linear motion platform mechanisms can perform horizontal linear motion that moves closer to or separates from each other.

[0018] Furthermore, the convergence angle adjustment mechanism includes a second worm gear mechanism, which is integrally mounted on the moving part of the corresponding ball screw linear motion platform mechanism, and the camera is connected to the corresponding second worm gear mechanism.

[0019] Furthermore, each camera is equipped with a camera focal length and aperture adjustment mechanism, which includes a motor and a gear set. The motor is connected to the focal length adjustment ring and aperture adjustment ring of the corresponding camera through the gears. The motor drives the focal length adjustment ring and aperture adjustment ring of the corresponding camera to rotate, thereby adjusting the focal length and aperture of the corresponding camera.

[0020] A self-calibration 3D reconstruction method based on the above-mentioned self-calibration 3D reconstruction device includes the following steps:

[0021] Step 1: Using the viewpoints of the two cameras in the current binocular vision system as the initial viewpoints and the camera coordinate system as the initial camera coordinate system, acquire images of the surface of the target workpiece to be sprayed under the initial camera coordinate system. By acquiring the images and combining them with the current intrinsic parameters, distortion parameters and extrinsic parameters of the two cameras in the binocular vision system, obtain the point cloud data of the surface of the target workpiece to be sprayed under the initial viewpoint.

[0022] Step 2: Adjust the position of the two cameras in the binocular vision system in the X, Y, and Z directions by making the gimbal move linearly in the X, Y, and Z directions. Adjust the position of the two cameras by making the gimbal and the two cameras rotate around the Z-direction line as the rotation center line. Adjust the pitch angle of the two cameras by making the gimbal and the two cameras rotate around the horizontal line as the rotation center line. Adjust the distance between the two cameras by making them move closer or further apart in a linear motion. Adjust the angle between the optical axes of the two cameras by making each camera rotate around a line perpendicular to the gimbal mounting surface as the rotation center line. Adjust the focal length and aperture of each camera to complete the adjustment of the camera's viewing angle, focal length, and aperture. Use the camera with the adjusted viewing angle, focal length, and aperture to acquire an image of the surface of the target workpiece to be sprayed from the new viewing angle.

[0023] Step 3: The trained radial basis function neural network is used to process the camera focal length and aperture adjusted in Step 2. After training, the radial basis function neural network establishes the mapping relationship between the intrinsic parameters and distortion parameters of each camera and the camera focal length and aperture size. The camera focal length and aperture adjusted in Step 2 are input into the trained radial basis function neural network, and the intrinsic parameters and distortion parameters of each camera are automatically calibrated by the radial basis function neural network.

[0024] Based on the adjusted distance between the two cameras, the angle between the optical axes of the two cameras, and the positional relationship between the X-axis of each camera's coordinate system and the rotation center line of that camera, the extrinsic parameters of the two cameras in the binocular vision system are automatically calibrated according to the extrinsic parameter calibration formula, which is as follows:

[0025]

[0026] Where R and T are the camera extrinsic parameters, i.e., rotation and translation matrices, respectively; 2θ is the angle between the optical axes of the two cameras after adjustment; p is the initial distance between the rotation center lines of the two cameras, which is equal to the distance between the rotation center lines of the two cameras under the initial viewpoint; p0 is the change in the distance between the rotation center lines of the two cameras, i.e., the adjustment amount of the distance between the two cameras; m is the component of the distance between the X-axis of each camera coordinate system and the rotation center line of that camera in the Y-axis direction of that camera coordinate system; q is the component of the distance between the X-axis of each camera coordinate system and the rotation center line of that camera in the Z-axis direction of that camera coordinate system.

[0027] Step 4: Using the intrinsic parameters, distortion parameters, and extrinsic parameters of the two cameras in the binocular vision system obtained in Step 3, process the surface image of the target workpiece to be sprayed under the new perspective obtained by the two cameras in Step 2 to obtain the point cloud data of the surface of the target workpiece to be sprayed under the new perspective.

[0028] Step 5: Using the linear motion and rotational motion of each camera in Step 2, obtain the transformation matrix of the camera coordinate system relative to its own initial camera coordinate system at any position during the motion. Use this matrix as the transformation matrix of the new viewpoint surface point cloud data of the target workpiece to be sprayed relative to the initial viewpoint surface point cloud data. Perform matrix transformation on the new viewpoint surface point cloud data using the transformation matrix to obtain the matrix transformation result.

[0029] The ICP algorithm is used, and the matrix transformation result is used as the initial value of the ICP algorithm to obtain the point cloud fine registration matrix. Based on the point cloud fine registration matrix, the new view surface point cloud data is registered to the initial view.

[0030] The target workpiece to be sprayed is reconstructed in three dimensions using the initial viewpoint surface point cloud data in the initial camera coordinate system and the new viewpoint surface point cloud data registered to the initial camera coordinate system.

[0031] Step 6: Repeat steps 2 to 5 continuously to register all surface point cloud data from the new perspectives to the initial perspective camera coordinate system in order to perform 3D reconstruction of the target workpiece to be sprayed until the 3D reconstruction of the target workpiece to be sprayed meets the requirements.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] 1. The multi-view 3D reconstruction device based on self-calibration technology designed in this invention has a large working space and degree of freedom, and high flexibility. It can reconstruct workpieces from multiple positions and perspectives according to the shape and size of different types of workpieces. Furthermore, these positions and perspectives are not fixed for different workpieces to be sprayed, and can be selected according to different reconstruction requirements. Therefore, this invention can achieve high-precision 3D reconstruction work from any perspective and orientation.

[0034] 2. The camera of the present invention uses a focal length and aperture adjustment mechanism to adjust the camera's focal length and aperture, which can adapt to the three-dimensional reconstruction of the workpiece to be sprayed under different tasks. Furthermore, by establishing a radial basis function (RBF) neural network model and combining it with the motor feedback signal controlling the camera's focal length and aperture adjustment mechanism, the camera's intrinsic parameters and distortion parameters can be directly obtained according to the method proposed in the present invention. This avoids the need to recalibrate the intrinsic parameters after changing the camera's focal length and aperture size, and enables automatic calibration of the camera's intrinsic parameters when the camera acquires point cloud images, thereby improving the adaptability to the three-dimensional reconstruction of different workpieces to be sprayed.

[0035] 3. This invention takes into account the fact that the camera rotates around the center line of the turntable in actual operation, rather than around its own coordinate axis. Combined with the motor feedback signal, the rotation and translation relationship between the binocular cameras (i.e., camera extrinsic parameters) can be obtained automatically and in real time. This enables automatic calibration of the camera extrinsic parameters when the camera acquires point cloud images, improving the working efficiency and convenience of the binocular camera in the field.

[0036] 4. This invention models the motion of a single camera, obtaining the transformation matrix of its coordinate system relative to its initial coordinate system at any position during motion. This, in turn, yields the transformation matrix of the point cloud from any viewpoint relative to the initial viewpoint, completing coarse registration of multi-viewpoint point clouds. This value is used as the initial value for the ICP algorithm to further obtain the fine registration matrix of the point cloud. Finally, through global fine registration, all viewpoint point clouds are registered to the initial viewpoint, completing the 3D reconstruction of the workpiece to be coated. The coarse registration method proposed in this invention provides a better initial value for the ICP algorithm in fine point cloud registration, improving the accuracy of point cloud registration. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of Embodiment 1 of the present invention.

[0038] Figure 2 This is a schematic diagram of the rotating platform mechanism in Embodiment 1 of the present invention.

[0039] Figure 3 This is a schematic diagram of the pitch mechanism, baseline adjustment mechanism, and convergence angle adjustment mechanism in Embodiment 1 of the present invention.

[0040] Figure 4 This is a schematic diagram of the camera focal length and aperture adjustment mechanism in Embodiment 1 of the present invention.

[0041] Figure 5 This is a schematic diagram of the RBF neural network structure in Embodiment 2 of the present invention.

[0042] Figure 6 This is a flowchart of the camera intrinsic parameters and distortion parameter fitting method in Embodiment 2 of the present invention.

[0043] Figure 7 This is a diagram showing the determination of the external parameter relationship between the two cameras in Embodiment 2 of the present invention.

[0044] Figure 8 This is a schematic diagram of the relative positions of the camera coordinate system, the device coordinate system, and the three rotation axes from the initial viewpoint in Embodiment 2 of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present invention, the embodiments will be described in detail below with reference to the accompanying drawings and examples. This will allow for a full understanding of how the present invention uses technical means to solve technical problems and achieve corresponding technical effects, and to facilitate its implementation. The embodiments of the present invention and the various features within them can be combined with each other without conflict, and all resulting technical solutions are within the protection scope of the present invention.

[0046] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0048] Example 1

[0049] like Figure 1 , Figure 2 , Figure 3 As shown, this embodiment discloses a self-calibrating 3D reconstruction device for intelligent spraying, including a control module, a motion acquisition module, a spatial movement mechanism 1, a rotating platform mechanism 2, a pitch mechanism 3, a baseline adjustment mechanism 4, a gimbal 401, two sets of convergence angle adjustment mechanisms 5, two sets of camera focal length and aperture adjustment mechanisms 6, and a binocular vision system, wherein:

[0050] The spatial movement mechanism 1 includes a rigid support 101. Two sets of X-direction ball screw linear motion platform mechanisms 109 are fixed to each of the Y-direction sides of the rigid support 101. The moving part of each X-direction ball screw linear motion platform mechanism 109 moves along the X-direction. The two sets of X-direction ball screw linear motion platform mechanisms 109 on each side are vertically distributed. A Z-direction ball screw linear motion platform mechanism 1010 is fixedly connected between the moving parts of the two sets of X-direction ball screw linear motion platform mechanisms 109 on each side. The moving part of the Z-direction ball screw linear motion platform mechanism 1010 moves along the Z-direction. A first connecting plate 102 is fixedly connected between the moving parts of the Z-direction ball screw linear motion platform mechanisms 1010 on both sides. The long side of the first connecting plate 102 is parallel to the Y-direction. A Y-direction ball screw linear motion platform mechanism 1011 is fixed to the bottom of the first connecting plate 102. The moving part of the Y-direction ball screw linear motion platform mechanism 1011 moves along the Y-direction.

[0051] In this embodiment, the X-axis ball screw linear motion platform mechanism 109, the Z-axis ball screw linear motion platform mechanism 1010, and the Y-axis ball screw linear motion platform mechanism 1011 have the same structure, all including a first screw rotatably mounted on a rigid bracket 101, a first closed-loop servo motor driving the first screw to rotate, a first nut screwed onto the first screw, and a first slider fixedly connected to the first nut as a moving part. This embodiment uses the Y-axis ball screw linear motion platform mechanism 1011 as an example for explanation. Figure 2As shown, in the Y-axis ball screw linear motion platform mechanism 1011, the first lead screw 105 is rotatably mounted on the bottom of the first connecting plate 102, and the axial direction of the first lead screw 105 is along the Y-axis. A first closed-loop servo motor 103 is fixedly mounted on the first connecting plate 102 at one end corresponding to the first lead screw 105. The output shaft of the first closed-loop servo motor 103 is coaxially and fixedly connected to the corresponding end of the first lead screw 105 via a first coupling 104. A first nut 106 is screwed onto the first lead screw 105, and a first slider 107, serving as a moving part, is fixedly connected to the first nut 106. The first slider 107 has a through hole for the first lead screw 105 to pass through, and the first slider 107 is slidably mounted on the bottom of the first connecting plate 102. Thus, when the first closed-loop servo motor 103 rotates, it drives the first lead screw 105 to rotate, causing the first slider 107 to move linearly along the Y-axis at the bottom of the first connecting plate 102.

[0052] In this embodiment, the X-axis ball screw linear motion platform mechanism 109, the Z-axis ball screw linear motion platform mechanism 1010, and the Y-axis ball screw linear motion platform mechanism 1011 differ in that their screw axes are respectively along the X, Y, and Z directions, thereby causing their corresponding moving parts to move along the X, Y, and Z directions, respectively. The X-axis ball screw linear motion platform mechanism 109, the Z-axis ball screw linear motion platform mechanism 1010, and the Y-axis ball screw linear motion platform mechanism 1011 constitute a three-axis ball screw motion platform mechanism, with the moving part of the Y-axis ball screw linear motion platform mechanism 1011, namely the first slider 107, serving as the moving part of the three-axis ball screw motion platform mechanism. The bottom of the first slider 107 is fixed with a second connecting plate 108, which is used to install the rotating platform mechanism 2.

[0053] like Figure 2 As shown, the rotating platform mechanism 2 includes a mounting base and a first worm gear mechanism mounted on the mounting base. Specifically, the top of the mounting base is fixed to the bottom of the second connecting plate 108. The first worm gear mechanism includes a worm wheel rotatably mounted in the mounting base along the Z-axis, and a worm gear rotatably mounted in the mounting base and meshing with the worm wheel. The lower end of the central shaft of the worm wheel protrudes from the bottom of the mounting base. A motor is fixed to one side of the mounting base, and the motor output shaft is coaxially fixed to the worm gear. The lower end of the central shaft of the worm wheel protrudes from the bottom of the mounting base and is used to connect to the pitch mechanism 3. The worm gear is driven to rotate by the motor, thereby causing the worm wheel and its central shaft to rotate around the Z-axis as the rotation center line.

[0054] like Figure 2 , Figure 3As shown, the pitch mechanism 3 includes a third connecting plate 306 and a fourth connecting plate 307. The fourth connecting plate 307 is L-shaped. The horizontal part of the L-shaped fourth connecting plate 307 is fixedly connected to the lower end of the worm gear central shaft in the rotating platform mechanism 2. The vertical part of the L-shaped fourth connecting plate 307 is fixed with a first bearing seat 303 and a second bearing seat 305 that are horizontally distributed. A rotating shaft 304 is rotatably mounted between the first bearing seat 303 and the second bearing seat 305 along the horizontal axis. A second closed-loop servo motor 301 is fixed at one end of the vertical part of the L-shaped fourth connecting plate 307 corresponding to the axial position of the rotating shaft 304. The output shaft of the second closed-loop servo motor 301 is coaxially fixedly connected to the corresponding shaft end of the rotating shaft 304 through a second coupling 302. The rotating shaft 304 is fixedly connected to the back of the third connecting plate 306, and the back of the gimbal 401 is fixed to the front of the third connecting plate 306. The second closed-loop servo motor 301 drives the rotating shaft 304 to rotate around the horizontal straight line as the rotation center line, and then the third connecting plate 306 and the gimbal 401 rotate around the horizontal straight line as the rotation center line.

[0055] Therefore, the ball screw three-axis motion platform mechanism in the spatial movement mechanism 1 can drive the rotary platform mechanism 2, the pitch mechanism 3, and the gimbal 401 to move linearly along the X, Y, and Z directions. When the worm gear in the rotary platform mechanism 2 rotates, it can drive the fourth connecting plate 307 in the pitch mechanism 3 to rotate around the Z-direction line, thereby causing the gimbal 401 to rotate around the Z-direction line. When the rotating shaft 304 in the pitch mechanism 3 rotates, it can cause the third connecting plate 306 and the gimbal 401 to rotate around the horizontal line, thereby adjusting the pitch angle of the gimbal 401.

[0056] like Figure 3As shown, the front of the gimbal 401 serves as the mounting surface for mounting the baseline adjustment mechanism 4. The baseline adjustment mechanism 4 includes a central projector 408 fixed in the middle of the mounting surface of the gimbal 401, and ball screw linear motion platform mechanisms located on both sides of the projector 408 in the Y direction. Each ball screw linear motion platform mechanism includes a pair of horizontally distributed bearing seats 407, one of which is located on the corresponding side of the projector 408, and the other is located on the mounting surface of the gimbal 401. A second screw 404, which is axially horizontal, is rotatably mounted between the two bearing seats 407 of each ball screw linear motion platform mechanism, and the second screws 404 in the two sets of ball screw linear motion platform mechanisms are coaxial. Each ball screw linear motion platform mechanism also includes a third closed-loop servo motor 402 coaxially fixedly connected to one end of the second screw 404 via a third coupling 403. A second nut 405 is screwed onto each of the second screws 404 in each ball screw linear motion platform mechanism. A second slider 406 is fixedly connected to each second nut 405. The second slider 406 has a through hole through which the second screw 404 passes, and it is slidably mounted on the mounting surface of the gimbal 401. The second slider 406 serves as the moving part of the corresponding ball screw linear motion platform mechanism. When the third closed-loop servo motor 402 drives the second screw 404 to rotate, the second slider 406 moves horizontally in a straight line on the mounting surface of the gimbal 401. Each of the two sets of ball screw linear motion platform mechanisms has a convergence angle adjustment mechanism 5 mounted on its second slider 406, thus the two sets of convergence angle adjustment mechanisms 5 are horizontally and linearly distributed.

[0057] like Figure 3 , Figure 4 As shown, each convergence angle adjustment mechanism 5 includes a connecting seat and a second worm gear mechanism 501 disposed on the connecting seat. Specifically, one side of the connecting seat is fixed to the corresponding second slider 406. The second worm gear mechanism 501 includes a worm wheel that is axially perpendicular to the mounting surface of the gimbal 401 and rotatably mounted in the connecting seat, and a worm that is rotatably mounted in the connecting seat and meshes with the worm wheel. One end of the central shaft of the worm wheel passes through the connecting seat. A motor is fixed on one side of the connecting seat. The output shaft of the motor is coaxially fixedly connected to the worm. The motor drives the worm to rotate, causing the worm wheel and its central shaft to rotate around a straight line perpendicular to the mounting surface of the gimbal as the rotation center line.

[0058] In this embodiment, the binocular vision system includes two cameras 602, which are mounted one-to-one on the convergence angle adjustment mechanism 5 via a fifth connecting plate 601. Specifically, each camera 602 is fixed to the front of the fifth connecting plate 601, and the X-axis of the coordinate system of each camera 602 is perpendicular to the mounting surface of the gimbal 401. The back of the fifth connecting plate 601 is fixed to the protruding end of the central shaft of the worm gear in the corresponding convergence angle adjustment mechanism 5, thus the two cameras 602 are horizontally aligned with the convergence angle adjustment mechanism 5. The lenses 603 of the two cameras 602 face the same direction. When the mounting surface of the gimbal 401 is horizontal, the lenses 603 of the two cameras 602 face horizontally. When the mounting surface of the gimbal 401 is vertical, the lenses 603 of the two cameras 602 face downwards or upwards. The optical axes (i.e., the Z-axis of the camera coordinate system) of the two cameras 602 are always parallel to the mounting surface.

[0059] When the second lead screw 404 in the two sets of ball screw linear motion platform mechanisms in the baseline adjustment mechanism 4 rotates, the two sets of second sliders 406 can move horizontally towards or away from each other in a linear motion, thereby enabling the two sets of convergence angle adjustment mechanisms 5 to move horizontally towards or away from each other in a linear motion. When the two sets of convergence angle adjustment mechanisms 5 move horizontally towards or away from each other in a linear motion, the two cameras 602 also move horizontally towards or away from each other in a linear motion, thus making the baseline distance between the two cameras 602 adjustable. When the worm gear and its central axis in each set of convergence angle adjustment mechanisms 5 rotate, the corresponding camera 602 and the fifth connecting plate 601 as a whole rotate around a straight line perpendicular to the gimbal mounting surface (i.e., the central axis of the worm gear in the convergence angle adjustment mechanism 5) as the rotation center line, thus making the angle between the optical axes of the two cameras 602 adjustable when both cameras 602 rotate.

[0060] Therefore, in this embodiment, when the first slider 107 in the ball screw three-axis motion platform mechanism of the spatial movement mechanism 1 moves linearly along the X, Y, and Z directions, the binocular vision system, convergence angle adjustment mechanism 5, baseline adjustment mechanism 4, gimbal 401, pitch mechanism 3, and rotation platform mechanism 2 move linearly along the X, Y, and Z directions along with the first slider 107. When the worm gear in the rotation platform mechanism 2 rotates, the fourth connecting plate 307 in the pitch mechanism 3 rotates around the Z-direction line as the rotation center line, thereby causing the binocular vision system, convergence angle adjustment mechanism 5, baseline adjustment mechanism 4, and gimbal 401 to rotate around the Z-direction line as the rotation center line. When the third connecting plate 306 in the pitch mechanism 3 rotates with the rotation shaft 304, the binocular vision system, convergence angle adjustment mechanism 5, baseline adjustment mechanism 4, and gimbal 401 rotate around the horizontal line as the rotation center line, thereby adjusting the pitch angle of the two cameras 602. When the second slider 406 in the baseline adjustment mechanism 4 moves horizontally towards or away from each other in a straight line, the camera 602 and the convergence angle adjustment mechanism 5 move horizontally along with the second slider 406. Furthermore, the two cameras 602 and the convergence angle adjustment mechanism 5 can move horizontally towards or away from each other in a straight line, making the baseline distance between the two cameras 602 adjustable. When the worm gear in the convergence angle adjustment mechanism 5 rotates, the corresponding camera 602 rotates around a straight line perpendicular to the gimbal mounting surface as its rotation center line, thereby making the angle between the optical axes of the two cameras 602 adjustable.

[0061] like Figure 4 As shown, two sets of camera focal length and aperture adjustment mechanisms 6 are configured one-to-one with the cameras 602 in the binocular vision system. Specifically, each set of camera focal length and aperture adjustment mechanisms 6 includes a focal length adjustment motor 604 and an aperture adjustment motor 605 mounted on the front of the corresponding fifth connecting plate 601. The output shaft of the focal length adjustment motor 604 is coaxially fixedly connected to a focal length adjustment gear 606, and the output shaft of the aperture adjustment motor 605 is coaxially fixedly connected to an aperture adjustment gear 607. The focal length adjustment ring of the lens of each camera 602 is coaxially fitted and fixed with a focal length clamp gear 608, and the aperture adjustment ring of the lens of each camera 602 is coaxially fitted and fixed with an aperture clamp gear 609. The focal length adjustment gear 606 and the focal length clamp gear 608 are engaged in transmission, and the aperture adjustment gear 607 and the aperture clamp gear 609 are engaged in transmission. Therefore, the focal length adjustment motor 604 in each camera focal length and aperture adjustment mechanism 6 can drive the lens focal length adjustment ring in the corresponding camera 602 to rotate through the focal length adjustment gear 606 and the focal length clamp gear 608 to adjust the focal length; the aperture adjustment motor 605 in each camera focal length and aperture adjustment mechanism 6 can drive the lens aperture adjustment ring in the corresponding camera 602 to rotate through the aperture adjustment gear 607 and the aperture clamp gear 609 to adjust the aperture.

[0062] In summary, in this embodiment, the spatial movement mechanism 1 is used to adjust the distance between the camera and the surface to be reconstructed. When the target workpiece is small, the distance between the camera and the target workpiece needs to be reduced. When the target workpiece is large, the distance between the camera and the target workpiece needs to be increased. The rotating platform mechanism 2 allows the two cameras to rotate as a whole around the vertical axis. The pitch mechanism 3 is used to adjust the pitch angle of the two cameras. The baseline adjustment mechanism 4 is used to adjust the baseline distance between the two cameras. The baseline is crucial to binocular 3D reconstruction. When the target workpiece is small, a large baseline distance will create blind spots between the two cameras. When the target workpiece is large, a small baseline distance will reduce the accuracy of the 3D reconstruction. The convergence angle adjustment mechanism 5 is used to adjust the angle between the optical axes of the two cameras to ensure that the surface to be reconstructed is always centered on the camera's imaging plane. The camera focal length and aperture adjustment mechanism 6 is used to adjust the camera's focal length by changing the distance between the camera and the surface to be reconstructed, so that the camera can capture a clear image while controlling the amount of light passing through the lens to achieve the best image quality.

[0063] In this embodiment, the motion acquisition module is used to acquire the motion of the gimbal 401 and each camera 602, as well as the adjustment of the focal length and aperture of each camera 602. In this embodiment, the motion acquisition module uses encoders, which are respectively installed on the output shafts of the first closed-loop servo motors of the ball screw three-axis motion platform mechanism in the spatial movement mechanism 1, the output shafts of the worm gear drive motors in the rotary platform mechanism 2, the output shafts of the second closed-loop servo motors 301 in the pitch mechanism 3, the output shafts of the two third closed-loop servo motors 402 in the baseline adjustment mechanism 4, the output shafts of the worm gear drive motors in each convergence angle adjustment mechanism 5, and the output shafts of the focal length adjustment motor 604 and the aperture adjustment motor 605 in each camera focal length and aperture adjustment mechanism 6. The motion acquisition module collects the rotational amounts of the first closed-loop servo motor 103, the motor on the mounting side of the rotating platform mechanism 2, the second closed-loop servo motor 301, the third closed-loop servo motor 402, the motor on the connecting seat side of the convergence angle adjustment mechanism 5, the focal length adjustment motor 604, and the aperture adjustment motor 605. This allows us to obtain the linear motion and rotational motion of the gimbal 401, as well as the linear motion, rotational motion, focal length, and aperture adjustment of each camera 602.

[0064] The motion acquisition module and the control module are electrically connected for data transmission. Two cameras 602 are also electrically connected to the control module for data transmission. The control module receives motion data acquired by the motion acquisition module and image data of the target surface to be sprayed acquired by the cameras 602. Simultaneously, the control module is also electrically connected to the first closed-loop servo motor 103, the motor on the mounting side of the rotating platform mechanism 2, the second closed-loop servo motor 301, the third closed-loop servo motor 402, the motor on the connecting seat side of the convergence angle adjustment mechanism 5, the focus adjustment motor 604, and the aperture adjustment motor 605, respectively, and controls the operation of each motor. Based on motion data, the control module obtains the X, Y, and Z-axis motion positions of the gimbal 401 and camera 602, the Z-axis linear rotation angle adjustment of the gimbal 401 and camera 602, the pitch angle adjustment of the gimbal 401 and camera 602, the baseline distance between the two cameras 602, the optical axis angle between the two cameras 602, and the focal length and aperture adjustment of each camera 602. The control module processes the obtained image data to perform three-dimensional reconstruction of the target to be sprayed.

[0065] Example 2

[0066] This embodiment discloses a self-calibration three-dimensional reconstruction method based on the self-calibration three-dimensional reconstruction device described in Embodiment 1, including the following steps:

[0067] Step 1: Using the viewpoints of the two cameras in the current binocular vision system as the initial viewpoints and the camera coordinate system as the initial camera coordinate system, acquire images of the surface of the target workpiece to be sprayed under the initial camera coordinate system. By acquiring the images and combining them with the current intrinsic parameters, distortion parameters and extrinsic parameters of the two cameras in the binocular vision system, obtain the point cloud data of the surface of the target workpiece to be sprayed under the initial viewpoints.

[0068] Step 2: Adjust the position of the two cameras in the binocular vision system in the X, Y, and Z directions by making the gimbal move linearly in the X, Y, and Z directions. Adjust the position of the two cameras by making the gimbal and the two cameras rotate around the Z-direction line as the rotation center line. Adjust the pitch angle of the two cameras by making the gimbal and the two cameras rotate around the horizontal line as the rotation center line. Adjust the distance between the two cameras by making them move closer to or further apart in a linear motion. Adjust the angle between the optical axes of the two cameras by making each camera rotate around a line perpendicular to the gimbal mounting surface as the rotation center line. Adjust the focal length and aperture of each camera to complete the adjustment of the camera's viewing angle, focal length, and aperture. Use the camera with the adjusted viewing angle, focal length, and aperture to acquire an image of the surface of the target workpiece to be sprayed from the new viewing angle.

[0069] In this embodiment, by adjusting the position and viewing angle of the two cameras, the surface of the workpiece to be reconstructed is positioned at the center of the imaging plane of the two cameras, occupying three-quarters of the field of view. The image data quality is optimized by adjusting the focal length and aperture of the two cameras.

[0070] Step 3: Automatically calibrate the intrinsic, distortion, and extrinsic parameters of the two cameras. The process is as follows:

[0071] (3.1) Automatic calibration of camera intrinsic parameters and distortion parameters.

[0072] In this embodiment, a trained radial basis function neural network is used to process the camera focal length and aperture adjusted in step 2. After training, the radial basis function neural network establishes a mapping relationship between the camera intrinsic parameters and distortion parameters and the camera focal length and aperture size. The camera focal length and aperture adjusted in step 2 are input into the trained radial basis function neural network, and the camera intrinsic parameters and distortion parameters are obtained by the radial basis function neural network.

[0073] like Figure 5 , Figure 6 As shown, since the self-calibrated 3D reconstruction device described in Embodiment 1 needs to adjust the focal length f and aperture size F of each camera during operation, if the intrinsic parameters of each camera need to be recalibrated every time the focal length and aperture size are changed, it increases the complexity of use. Therefore, this embodiment proposes a method for fitting camera intrinsic parameters and distortion parameters based on radial basis function (RBF) neural networks. RBF neural networks are a three-layer feedforward neural network with a single hidden layer. Researchers have proven that RBF neural networks can approximate any nonlinear function with any accuracy. Since the camera's intrinsic parameters are only related to the camera's own structure and not to its position, and in this embodiment, the adjustment of each camera's own structure only includes adjusting the focal length and aperture size, the RBF neural network has two input layer nodes, and the intrinsic parameters of each camera are... The distortion parameters are k1, k2, k3, p1, and p2, a total of nine parameters. Therefore, the number of nodes in the output layer of the RBF neural network is 9, and the number of hidden layers is set to 6. Where f x f y Focal length is expressed in pixels along both the horizontal and vertical directions, c x c y These are the principal point coordinates, k1, k2, and k3 are the radial distortion parameters, and p1 and p2 are the tangential distortion parameters.

[0074] In this embodiment, the radial basis function (RBF) neural network has been trained beforehand, and the training process is as follows:

[0075] Assuming the average focal length commonly used in this embodiment is f0, the aperture is F0, and the sampling interval for focal length and aperture for each camera is set to 0.5 units, that is, the sampling range for focal length is [f0-0.5*n, f0+0.5*n], and the sampling range for aperture is [F0-0.5*n, F0+0.5*n], for a total of 4n. 2 There are 10 data points, where n is a user-defined variable used to control the size of the dataset.

[0076] Each camera is instructed to photograph a calibration board, and the zoom mechanism (focal length adjustment motor) and aperture mechanism (aperture adjustment motor) of each camera are adjusted to obtain the focal length f and aperture F of each camera, forming an input vector in_feature. The calibration board is moved to ensure that the image in each camera is clear, and then the position of the calibration board is adjusted to obtain images from different angles. Then, the intrinsic parameters and distortion parameters of the cameras are calculated using the Zhang Zhengyou calibration method, forming an output vector out_feature. The input and output vectors are used to train an RBF neural network to obtain a mapping model between the intrinsic parameters and distortion parameters of each camera and the camera's focal length and aperture size. After obtaining the mapping model, as long as the focal length and aperture size of each camera are known, the intrinsic parameters and distortion parameters of each camera can be directly obtained through the trained RBF neural network without the need for recalibrating the camera's intrinsic parameters.

[0077] In this embodiment, during the training of the RBF neural network, an autofocus algorithm is used when adjusting the zoom mechanism of each camera. The control module receives the image information sent by the camera, calculates and compares the sharpness of the current frame and the previous frame. If the sharpness of the current frame is greater than that of the previous frame, the focus adjustment motor keeps its rotation unchanged until the sharpness of the current frame is less than that of the previous frame. Then, the control module controls the focus adjustment motor to stop rotating, thus completing the focusing process.

[0078] In this embodiment, during the training of the RBF neural network, an automatic aperture algorithm is used to adjust the aperture adjustment mechanism of each camera. After multiple experiments, a desired average image brightness value is obtained, which allows for visual assessment of the optimal exposure for the image of the workpiece to be coated. This brightness is set as the desired brightness. When the camera captures a color image, the color image is converted into a brightness value and compared with the desired brightness value. If the absolute value of the difference between the current frame's brightness and the desired brightness is decreasing, the aperture adjustment motor is kept in the same direction until the absolute value of the difference between the current frame's brightness and the desired brightness is less than a given threshold. At this point, the control module stops the aperture adjustment motor, completing the aperture adjustment.

[0079] Therefore, by training the RBF neural network, a mapping relationship is established between the intrinsic parameters and distortion parameters of each camera and the camera's focal length and aperture. During the operation of the self-calibrating 3D reconstruction device described in Embodiment 1, when adjusting the camera's focal length and aperture, the trained RBF neural network can automatically obtain the corresponding camera's intrinsic parameters and distortion parameters based on the adjusted focal length and aperture of each camera, thus improving the efficiency and convenience of the self-calibrating 3D reconstruction device described in Embodiment 1.

[0080] (3.2) Automatic calibration of camera external parameters.

[0081] like Figure 7 As shown, the calibration of the extrinsic parameters of the two cameras in a binocular vision system is to determine the relative position and orientation relationship between the two cameras. The camera extrinsic parameters are only related to the relative motion of the two cameras. If the two cameras are moving as a whole, the camera extrinsic parameters will not change. Therefore, in this embodiment, the camera extrinsic parameters are only related to the adjustment amount of the distance between the two cameras and the adjustment amount of the optical axis angle.

[0082] The principle of calibrating the extrinsic parameters of the two cameras is as follows:

[0083] The direction of the camera's optical axis is taken as the Z-axis of the camera coordinate system (i.e., Z in the figure). C ), establish as Figure 7 The coordinate system shown. Figure 7 Axis 903 is the central axis of the worm gear in the convergence angle adjustment mechanism 5. The camera rotates around axis 903 as its rotation center line. Axis 903 is parallel to the X-axis of the camera coordinate system (i.e., the X-axis in the figure). C ).

[0084] like Figure 7 As shown, the initial distance between the rotation center axes of the two cameras is p, and the initial convergence angle of the two cameras is 0. This embodiment assumes a movement and rotation process to illustrate the establishment process of the camera extrinsic model. The movement and rotation process is as follows: Figure 7 The transformation relationship between the right camera and the left camera can be viewed as the left camera first moving to its own X-axis. C The axis is positioned so that it coincides with the rotation axis 903 corresponding to the left camera. Then, it rotates by θ around the rotation axis 903 corresponding to the left camera. Next, it moves a distance p along the line connecting the two rotation center axes. Then, it rotates by θ around the rotation axis 903 corresponding to the right camera. Finally, it moves to a position coinciding with the camera coordinate system of the right camera. Where:

[0085] The transformation matrix for moving along the line connecting the two rotation center axes is:

[0086] Move the left camera to the left camera X C The transformation matrix for the position where the axis coincides with the rotation axis 903 corresponding to the left camera is:

[0087] m is the component of the distance between the X-axis of each camera coordinate system and the rotation centerline of the camera along the Y-axis of that camera coordinate system. In this embodiment, during the assumed movement and rotation process, m represents the distance between the X-axis and the rotation centerline of the left camera. C When the axis moves to a position coinciding with the rotation axis 903 corresponding to the left camera, along the left camera's Y... C The distance of the axis translation;

[0088] q represents the component of the distance between the X-axis of each camera coordinate system and the rotation centerline of that camera along the Z-axis of that camera coordinate system. In this embodiment, during the assumed movement and rotation process, q represents the distance between the left camera and the X-axis. C When the axis moves to a position coinciding with the rotation axis 903 corresponding to the left camera, along the Z-axis of the left camera... C The distance of the axis translation.

[0089] p is the initial distance between the rotation center lines of the two cameras (i.e., between the two rotation axes 903) (equal to the distance between the rotation center lines of the two cameras under the initial viewpoint).

[0090] Left camera rotates around itself X C The transformation matrix for axis rotation is

[0091] Therefore, we obtain formula (1) as follows:

[0092]

[0093] Where A represents the left camera moving to itself at X. C The transformation matrix for the position where the axis coincides with the rotation axis 903 corresponding to the left camera, where B is the transformation matrix of the left camera around the left camera X. C The transformation matrix for the rotation angle θ of the axis, C is the transformation matrix for the left camera moving p along the line connecting the two rotation axes 90° and 3°, and D is the transformation matrix for the left camera rotating around the camera X. C The transformation matrix for axis rotation angle θ, E is the transformation matrix for the left camera to move from the rotation axis 903 corresponding to the right camera to a position coinciding with the right camera coordinate system, and we have:

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] The above formula (1) contains three unknown fixed parameters: m, q, and p. In order to find the values ​​of these three parameters, the extrinsic parameters of the binocular camera are obtained by using the calibration plate images acquired by the two cameras and the existing traditional binocular calibration method. In addition, the camera extrinsic parameters calculated by formula (1) are equal, so the values ​​of the three unknowns m, q, and p can be obtained. Later, when the relative position of the binocular camera changes, the camera extrinsic parameters can be directly calculated by using the extrinsic parameter calibration formula shown in formula (2), without having to use the traditional binocular calibration method to obtain the camera extrinsic parameters every time.

[0100] After obtaining m, q, and p, let the change in distance between the rotation center axes of the two cameras be p0. The sign of p0 is determined by the relative movement direction of the binocular cameras, with negative for closer and positive for farther. Therefore, the distance between the rotation center axes of the two cameras after translational motion is p + p0. When the angle between the optical axes of the two cameras is 2θ after rotational motion, the external parameter calibration formula is obtained as follows:

[0101]

[0102] In this embodiment, when the relative position between the two cameras and the angle between the optical axes are adjusted during the operation of the self-calibrated three-dimensional reconstruction device described in Embodiment 1, the corresponding camera extrinsic parameters (i.e., rotation matrix R and translation matrix T) can be automatically calculated using formula (2) based on the adjusted relative position between the two cameras and the angle between the optical axes, without the need for recalibration, thereby further improving the efficiency and convenience of the self-calibrated three-dimensional reconstruction device described in Embodiment 1.

[0103] Step 4: Using the intrinsic parameters, distortion parameters, and extrinsic parameters of the two cameras in the binocular vision system obtained in Step 3, process the surface image of the target workpiece to be sprayed under the new perspective obtained by the two cameras in Step 2 to obtain the point cloud data of the surface of the target workpiece to be sprayed under the new perspective.

[0104] Step 5: Using the linear motion and rotational motion of each camera in Step 2, obtain the transformation matrix of the camera coordinate system relative to its own initial camera coordinate system at any position during the motion. Use this matrix as the transformation matrix of the new viewpoint surface point cloud data of the target workpiece to be sprayed relative to the initial viewpoint surface point cloud data. Perform matrix transformation on the new viewpoint surface point cloud data using the transformation matrix to obtain the matrix transformation result.

[0105] The ICP algorithm is used, and the matrix transformation result is used as the initial value of the ICP algorithm to obtain the point cloud fine registration matrix. Based on the point cloud fine registration matrix, the new view surface point cloud data is registered to the initial view.

[0106] The target workpiece to be sprayed is reconstructed in three dimensions using the initial viewpoint surface point cloud data in the initial camera coordinate system and the new viewpoint surface point cloud data registered to the initial camera coordinate system.

[0107] After obtaining new single-view surface point cloud data, the self-calibrating 3D reconstruction device described in Example 1 needs to register multiple new single-view surface point cloud data of the target workpiece to be sprayed to the initial view camera coordinate system. The single-view surface point cloud data is constructed based on the camera coordinate system. If a relatively complete 3D model of the target workpiece to be sprayed is required, multiple scans from multiple new single views are needed, and the surface point clouds under multiple new single views are rotated and translated to a unified coordinate system (i.e., the initial view camera coordinate system).

[0108] This embodiment describes the process of registering new single-view surface point cloud data to the initial view camera coordinate system, which includes coarse registration and precise registration. The purpose of coarse point cloud registration is to find the mapping relationship between the rotation and translation of the surface point cloud data from different camera viewpoints and the initial viewpoint, roughly registering multiple new single-view target workpiece surface point cloud data together. Due to factors such as mechanical system errors, the coarsely registered surface point cloud data often cannot meet the needs of actual use, therefore, further correction is required, i.e., precise registration.

[0109] like Figure 8 As shown, the camera moves along the X, Y, and Z directions of the device, rotates around axes 901, 902, and 903, and moves along the line connecting the two rotation center axes. From the motion relationships of a single camera, the transformation matrix of a single camera at different positions relative to its initial position can be obtained, thus yielding the transformation matrix of the single camera coordinate system at different viewpoints relative to its initial coordinate system at the initial viewpoint. Here, axis 901 is the rotation axis of the rotating platform mechanism 2, i.e., the central axis of the worm gear in the rotating platform mechanism 2; axis 902 is the central axis of the rotation axis 304 of the pitch mechanism 3; and axis 903 is the central axis of the worm gear in the convergence angle adjustment mechanism 5.

[0110] Based on the established mathematical model, the point clouds of the target workpiece under multiple new single-viewpoints are transformed to the initial viewpoint camera coordinate system, completing the coarse registration of the point clouds of the target workpiece. Finally, the global accurate registration of the point clouds of the target workpiece is achieved based on the nearest point iteration method (ICP).

[0111] The coarse registration process of the point cloud in this embodiment is as follows:

[0112] The transformation matrices between the aforementioned new viewpoints and the initial viewpoint can be obtained from the following mathematical relationships: (X, camera coordinate system under the initial viewpoint of this invention) C Y C Z CThe X, Y, and Z directions are parallel to those of the self-calibrating 3D reconstruction device described in Embodiment 1, meaning the camera does not rotate. The transformation matrix of the camera relative to the initial viewpoint position 0 after moving a, b, c along the X, Y, and Z directions respectively, rotating by an angle α around axis 901, rotating by an angle β around axis 902, moving h along the line connecting the two rotation center axes, and rotating by an angle γ around axis 903, can be obtained from the translation and rotation relationship of the single camera coordinate system itself relative to its own coordinate system in the initial state during the motion, as shown in formula (3).

[0113]

[0114] Where: T1 is the transformation matrix for the camera's movement along the three axes; T2 is the transformation matrix for the camera's rotation around axis 901 by an angle α; T3 is the transformation matrix for the camera's rotation around axis 902 by an angle β; T4 is the transformation matrix for the camera's movement along the line connecting the two rotation center axes by an angle h; and T5 is the transformation matrix for the camera's rotation around axis 903 by an angle γ. Then we have:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] Therefore, we can obtain formula (4):

[0121]

[0122] in:

[0123] t 11 =cosαcosβ,t 12 =cosαsinβsinγ-sinαcosγ

[0124] t 13 =sinαsinγ+cosαcosγsinβ

[0125] t 14 =a+d+j*(cosγsinα-cosαsinβsinγ)

[0126] -k*(sinαsinγ+cosαcosγsinβ)+cosα*(fd)

[0127] t 15=sinα*(ehj)-fcosαcosβ+cosαsinβ*(kg)

[0128] t 21 =cosβsinα,t 22 =cosαcosγ+sinαsinβsinγ

[0129] t 23 =cosγsinαsinβ-cosαsinγ

[0130] t 24 =b+ej*(cosαcosγ+sinαsinβsinγ)+k*(cosαsinγ-cosγsinαsinβ)

[0131] t 25 =cosα*(h+je)+sinα*(fd)-fcosβsinα+sinαsinβ*(kg)

[0132] t 31 = -sinβ,t 32 =cosβsinγ,t 33 =cosβcosγ

[0133] t 34 =c+g+cosβ*(kg)+fsinβ-kcosβcosγ-jcosβsinγ

[0134] d, e, f, g, j, k are fixed parameters.

[0135] To solve for the parameters d, e, f, g, j, k, an auxiliary calibration robot is used. and Let be the transformation matrices of the camera at positions 0 and k relative to the coordinate system of the auxiliary calibration robot. These can be determined by the robot's hand-eye calibration, thus:

[0136]

[0137] From this, we can obtain The value will Substituting the values ​​into the left side of equation (4), we obtain the values ​​of d, e, f, g, j, and k.

[0138] Therefore, the transformation matrix of the camera after moving a, b, c, rotating α angle around axis 901, rotating β angle around axis 902, moving h along the line connecting the two rotation center axes, and rotating γ angle around axis 903 to reach position k, relative to position 0 under the initial viewpoint, can be determined by formulas (3) and (4).

[0139] The transformation matrix of position k relative to position k-1 can be determined by the following equation:

[0140]

[0141] Based on this transformation relationship, multiple new single-view surface point cloud data can be registered in pairs to complete the coarse registration of the workpiece point cloud.

[0142] The fine registration process of point clouds in this embodiment is as follows:

[0143] Global accurate registration of workpiece point clouds is achieved based on the nearest-point iterative method ICP. As the initial values ​​for point cloud registration between two adjacent frames, the transformation matrix after fine registration is obtained through the ICP algorithm. After registering the point clouds from N viewpoints pairwise, it is necessary to transform all point clouds to the initial viewpoint and obtain the global registration matrix. When k = 0 The point cloud in frame k can be registered to the initial viewpoint.

[0144] Step 6: Repeat steps 2 to 5 continuously to register all surface point cloud data from the new perspectives to the initial perspective for 3D reconstruction of the target workpiece to be sprayed, until the 3D reconstruction of the target workpiece to be sprayed meets the requirements.

[0145] This invention provides a multi-view 3D reconstruction device based on self-calibration technology. It offers a large working space and high degree of freedom, enabling high-precision 3D reconstruction from any angle and orientation based on the shape and size of different workpieces. Compared to traditional calibration methods, this invention can perform self-calibration after initial calibration, even when the camera position and angle change, without requiring manual recalibration. This improves the efficiency of 3D reconstruction of various types of small batches of workpieces to be painted.

[0146] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0147] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A self-calibrating three-dimensional reconstruction device for intelligent spraying, characterized in that, The device includes a gimbal, a binocular vision system for acquiring images of the surface of the target to be sprayed, a control module, and a motion acquisition module. The binocular vision system includes two cameras. One side of the gimbal serves as the mounting surface, and the two cameras are mounted on the mounting surface of the gimbal. The X-axis of the coordinate system of each camera is perpendicular to the mounting surface of the gimbal. The gimbal can move linearly along the X, Y, and Z directions, and rotate about the Z-direction as the rotation center line. This allows the binocular vision system to move linearly along the X, Y, and Z directions and rotate about the Z-direction as the rotation center line. The gimbal can also rotate about a horizontal line to adjust the pitch angle, thus making the pitch angle of the two cameras in the binocular vision system adjustable. The two cameras can move horizontally closer to each other or further apart on the gimbal, thus making the distance between the two cameras adjustable, and consequently, the baseline distance between the two cameras adjustable. Each camera can rotate about a line perpendicular to the gimbal mounting surface, thus making the angle between the optical axes of the two cameras adjustable, and the focal length and aperture of each camera adjustable. The motion acquisition module acquires the motion of the gimbal and each camera, as well as the adjustment of the focal length and aperture of each camera. The motion acquisition module is electrically connected to the control module, and the two cameras are electrically connected to the control module respectively. The control module receives the data acquired by the motion acquisition module and the image data acquired by the cameras, and processes the received data to perform three-dimensional reconstruction of the target to be sprayed. The motion acquisition module and the control module are electrically connected for data transmission. The two cameras are also electrically connected to the control module for data transmission. The control module receives motion data acquired by the motion acquisition module and image data of the target surface to be sprayed acquired by the cameras. Based on the motion data, the control module obtains the X, Y, and Z-axis motion positions of the gimbal and cameras, the adjustment amount of the gimbal and cameras' linear rotation angle around the Z-axis, the adjustment amount of the gimbal and cameras' pitch angle, the baseline distance between the two cameras, the optical axis angle between the two cameras, and the adjustment amount of the focal length and aperture of each camera. The control module processes the obtained image data to perform three-dimensional reconstruction of the target to be sprayed.

2. The self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 1, characterized in that, The gimbal is mounted on the pitch mechanism, the pitch mechanism is mounted on the rotating platform mechanism, and the rotating platform mechanism is mounted on the spatial movement mechanism. The spatial movement mechanism drives the rotating platform mechanism, the pitch mechanism, and the gimbal as a whole to move in straight lines in the X, Y, and Z directions. The rotating platform mechanism drives the pitch adjustment mechanism and the gimbal as a whole to rotate around the Z-direction straight line as the rotation center line. The pitch adjustment mechanism drives the gimbal to rotate around the horizontal straight line as the rotation center line.

3. The self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 2, characterized in that, The spatial movement mechanism includes a ball screw three-axis motion platform mechanism, the moving part of which can move linearly in the X, Y, and Z directions.

4. The self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 3, characterized in that, The rotary platform mechanism includes a first worm gear mechanism, which is integrally mounted on the moving part of the ball screw three-axis motion platform mechanism.

5. A self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 4, characterized in that, The pitch mechanism includes a rotating shaft and a motor that drives the rotating shaft to rotate. The motor and the rotating shaft are connected to the first worm gear mechanism through the same connecting plate. The gimbal is connected to the rotating shaft.

6. The self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 1, characterized in that, The gimbal mounting surface is equipped with a baseline adjustment mechanism. Each camera is mounted on the baseline adjustment mechanism via a convergence angle adjustment mechanism. The baseline adjustment mechanism drives the two convergence angle adjustment mechanisms and the corresponding camera to move horizontally in a straight line, either moving closer to or separating from each other. The convergence angle adjustment mechanism also drives the corresponding camera to rotate around a straight line perpendicular to the gimbal mounting surface as its rotation center line.

7. A self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 6, characterized in that, The baseline adjustment mechanism includes two sets of ball screw linear motion platform mechanisms, and the moving parts of the two sets of ball screw linear motion platform mechanisms can perform horizontal linear motion that moves closer to or separates from each other.

8. A self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 7, characterized in that, The convergence angle adjustment mechanism includes a second worm gear mechanism, which is integrally mounted on the moving part of the corresponding ball screw linear motion platform mechanism, and the camera is connected to the corresponding second worm gear mechanism.

9. A self-calibrating three-dimensional reconstruction device for intelligent spraying according to claim 1, characterized in that, Each camera is equipped with a camera focal length and aperture adjustment mechanism, which includes a motor and a gear set. The motor is connected to the focal length adjustment ring and aperture adjustment ring of the corresponding camera through the gears. The motor drives the focal length adjustment ring and aperture adjustment ring of the corresponding camera to rotate, thereby adjusting the focal length and aperture of the corresponding camera.

10. A self-calibration three-dimensional reconstruction method based on the self-calibration three-dimensional reconstruction device according to any one of claims 1-9, characterized in that, Includes the following steps: Step 1: Using the viewpoints of the two cameras in the current binocular vision system as the initial viewpoints and the camera coordinate system as the initial camera coordinate system, acquire images of the surface of the target workpiece to be sprayed under the initial camera coordinate system. By acquiring the images and combining them with the current intrinsic parameters, distortion parameters and extrinsic parameters of the two cameras in the binocular vision system, obtain the point cloud data of the surface of the target workpiece to be sprayed under the initial viewpoint. Step 2: Adjust the position of the two cameras in the binocular vision system in the X, Y, and Z directions by making the gimbal move linearly in the X, Y, and Z directions. Adjust the position of the two cameras by making the gimbal and the two cameras rotate around the Z-direction line as the rotation center line. Adjust the pitch angle of the two cameras by making the gimbal and the two cameras rotate around the horizontal line as the rotation center line. Adjust the distance between the two cameras by making them move closer or further apart in a linear motion. Adjust the angle between the optical axes of the two cameras by making each camera rotate around a line perpendicular to the gimbal mounting surface as the rotation center line. Adjust the focal length and aperture of each camera to complete the adjustment of the camera's viewing angle, focal length, and aperture. Use the camera with the adjusted viewing angle, focal length, and aperture to acquire an image of the surface of the target workpiece to be sprayed from the new viewing angle. Step 3: The trained radial basis function neural network is used to process the camera focal length and aperture adjusted in Step 2. After training, the radial basis function neural network establishes the mapping relationship between the intrinsic parameters and distortion parameters of each camera and the camera focal length and aperture size. The camera focal length and aperture adjusted in Step 2 are input into the trained radial basis function neural network, and the intrinsic parameters and distortion parameters of each camera are automatically calibrated by the radial basis function neural network. Based on the adjusted distance between the two cameras, the angle between the optical axes of the two cameras, and the positional relationship between the X-axis of each camera's coordinate system and the rotation center line of that camera, the extrinsic parameters of the two cameras in the binocular vision system are automatically calibrated according to the extrinsic parameter calibration formula, which is as follows: Where R and T are the camera extrinsic parameters, namely the rotation matrix and the translation matrix, respectively; To adjust the angle between the optical axes of the two rear cameras; p The initial distance between the rotation center lines of the two cameras is equal to the distance between the rotation center lines of the two cameras under the initial viewpoint; This refers to the change in the distance between the rotation center lines of the two cameras, i.e., the adjustment of the distance between the two cameras. m The component of the distance between the X-axis of each camera coordinate system and the rotation centerline of the camera in the Y-axis direction of the camera coordinate system; q The component of the distance between the X-axis of each camera coordinate system and the rotation centerline of the camera in the Z-axis direction of the camera coordinate system; Step 4: Using the intrinsic parameters, distortion parameters, and extrinsic parameters of the two cameras in the binocular vision system obtained in Step 3, process the surface image of the target workpiece to be sprayed under the new perspective obtained by the two cameras in Step 2 to obtain the point cloud data of the surface of the target workpiece to be sprayed under the new perspective. Step 5: Using the linear motion and rotational motion of each camera in Step 2, obtain the transformation matrix of the camera coordinate system relative to its own initial camera coordinate system at any position during the motion. Use this matrix as the transformation matrix of the new viewpoint surface point cloud data of the target workpiece to be sprayed relative to the initial viewpoint surface point cloud data. Perform matrix transformation on the new viewpoint surface point cloud data using the transformation matrix to obtain the matrix transformation result. The ICP algorithm is used, and the matrix transformation result is used as the initial value of the ICP algorithm to obtain the point cloud fine registration matrix. Based on the point cloud fine registration matrix, the new view surface point cloud data is registered to the initial view. The target workpiece to be sprayed is reconstructed in three dimensions using the initial viewpoint surface point cloud data in the initial camera coordinate system and the new viewpoint surface point cloud data registered to the initial camera coordinate system. Step 6: Repeat steps 2-5 continuously to register all surface point cloud data from the new perspectives to the initial perspective camera coordinate system in order to perform 3D reconstruction of the target workpiece to be sprayed until the 3D reconstruction of the target workpiece to be sprayed meets the requirements.

Images