Three-dimensional reconstruction system and method for operation in dexterous palm

Abstract

The invention discloses a three-dimensional reconstruction system and method for operation in a dexterous palm. The system comprises a shell, a multispectral monocular camera, a white light emitter, an infrared light emitter, an ultraviolet light emitter, a rotating mechanism, a control module and an intelligent terminal. Wherein the rotating mechanism can enable the shell to rotate by 360 degrees, the multispectral monocular camera is used for capturing object images under different spectrums, and the white light emitter, the infrared light emitter and the ultraviolet light emitter are installed at the tail ends of the three arms and emit white light, infrared light and ultraviolet light respectively. The control module and the intelligent terminal are located behind the shell and are connected through a local area network. According to the three-dimensional reconstruction system and method for operation in the dexterous palm provided by the invention, the surface topography information of the object can be quickly and accurately obtained, and the dexterous palm is helped to make a decision for the operation in the palm. The system is high in reconstruction precision and good in environmental adaptability, and a new thought is provided for environment information acquisition during operation in the dexterous palm.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a three-dimensional reconstruction system and method for dexterous palm operation. Background Art

[0002] The robotic arm that controls the dexterous hand often cannot accurately obtain the situation near the palm of the dexterous hand. The palm operation of the dexterous hand needs to rely on external or its own sensors to measure and judge the touched object. Photometric stereo is widely used in 3D reconstruction. It captures the image of the object under at least 3 different light angles, calculates the surface normal vector of the object and generates a 3D shape. Multispectral imaging technology can capture the image of the object under different wavelengths, provide more surface information, and help improve the accuracy of 3D reconstruction.

[0003] In the process of implementing the technical method of the embodiment of the present invention, the inventor of this patent found at least the following technical problems in the prior art: 1) Dexterous palm operations rely on external or palm sensors, but their accuracy and efficiency are difficult to meet the requirements.

[0004] 2) Traditional photometric stereo methods use visible light images, and their reconstruction results are less accurate when dealing with complex surfaces and materials.

[0005] 3) Most existing multispectral photometric stereo methods use fixed light source configurations and simple weighted averaging methods for data fusion, which results in insufficient accuracy and efficiency.

[0006] In summary, the existing three-dimensional reconstruction systems and methods for dexterous palm operations cannot meet the actual usage requirements. Summary of the invention

[0007] The embodiments of the present invention provide a three-dimensional reconstruction system and method for dexterous palm operations, which solve the problems of low accuracy and efficiency of existing dexterous palm operations, limited three-dimensional reconstruction accuracy of traditional photometric stereo methods, and insufficient fusion accuracy and efficiency of existing multi-spectral photometric stereo methods.

[0008] The embodiment of the present invention provides a three-dimensional reconstruction system for dexterous palm operation, including a housing, a multi-spectral monocular camera, a white light emitter, an infrared light emitter, an ultraviolet light emitter, a rotating mechanism, a control module and an intelligent terminal; The housing comprises three arms, is a centrally symmetrical structure, is made of metal or plastic, and is mounted on the palm of the dexterous hand through the rotating mechanism; The rotating mechanism is installed at the rear end of the geometric center of the housing and is used to drive the housing to rotate at a specific angle in a predetermined direction and speed; The multispectral monocular camera is installed at the front end of the geometric center of the shell and has three shooting modes: visible light shooting mode, infrared light shooting mode and ultraviolet light shooting mode; The white light emitter, the infrared light emitter and the ultraviolet light emitter are installed at the ends of the three arms; The white light emitter is used to emit white light; The infrared light emitter is used to emit infrared light; The ultraviolet light emitter is used to emit ultraviolet light; The control module is implanted in the arm of the dexterous hand and is connected to the intelligent terminal via a local area network.

[0009] Optionally, the rotating mechanism includes a small servo motor, an angle sensor module and a mechanism housing; The small servo motor can rotate a specific angle in a predetermined direction and speed; The housing of the mechanism is made of metal or plastic; Optionally, the angle sensor module includes an angle sensor and a data sending circuit, the angle sensor can measure the angle of the rotating mechanism, and the data sending circuit can send angle data to the control module.

[0010] Optionally, the control module includes a central processing unit, a servo drive circuit, a local area network interface, and a power supply circuit; The central processing unit is responsible for processing the data received from the sensors and the camera, and controlling the movement of the servo motor through the servo drive circuit; The servo drive circuit can control the small servo motor to rotate at a specific angle in a predetermined direction and speed; The LAN interface is used for communication between the control module and the intelligent terminal; The power supply circuit is used for supplying power.

[0011] The present invention also provides a three-dimensional reconstruction method for dexterous palm operation, which is applied to the above-mentioned three-dimensional reconstruction system for dexterous palm operation, and the method comprises the following steps: S1. Each time the shell rotates once, the white light emitter, infrared light emitter and ultraviolet light emitter respectively emit white light, infrared light and ultraviolet light to illuminate the object to be measured, and the multi-spectral monocular camera takes 3 multi-level spectral images in each posture. After the shell rotates 3 times, a total of 9 multi-level spectral images are obtained; S2, using photometric stereo method to calculate the surface normal vectors of the object under test at three levels respectively; S3, then use the deep learning algorithm to nonlinearly fuse the data at each level, and automatically adjust the fusion weight through the training model to obtain a high-precision three-dimensional image; S4. Finally, the surface shape of the object to be measured is obtained by analyzing the high-precision three-dimensional image for operation in the dexterous palm.

[0012] Optionally, the method also includes: each time the shell is in a new posture, the multispectral monocular camera first enters a debugging mode to monitor the intensity and spectral distribution of the reflected light in real time, the intelligent terminal analyzes the surface reflection characteristics of the object under test, and the control module dynamically adjusts the intensity and angle of the light source to optimize multispectral data acquisition.

[0013] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: (1) Compared with the traditional photometric stereo method that only obtains object surface information at the visible light level, the present invention obtains object information at three levels: visible light, infrared light, and ultraviolet light, and uses a 360° rotatable mechanism and a multi-spectral light source to work together, expanding the way to obtain information and improving the accuracy and efficiency of three-dimensional reconstruction; (2) The present invention provides a multi-level spectral nonlinear fusion method, which integrates the advantages of different wavelengths by training a neural network model to nonlinearly fuse normal vector data of different wavelengths, thereby improving the accuracy and efficiency of three-dimensional reconstruction and the efficiency of dexterous palm operation; (3) Compared with the fixed light source configuration of the traditional photometric stereo method, the present invention dynamically monitors and adjusts the intensity and angle of the light source, improves the flexibility of light source configuration, and optimizes lighting conditions; (4) The multi-spectral fusion 3D reconstruction system and method provided by the present invention brings new possibilities to the fields of dexterous palm operation, industrial inspection and cultural relics protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the overall structure of a three-dimensional reconstruction system for dexterous palm operation in one embodiment of the present invention; Figure 2 A system structure block diagram of a three-dimensional reconstruction system for dexterous palm operation according to an embodiment of the present invention; Figure 3 A flowchart of the working principle of 3D reconstruction of a 3D reconstruction method for dexterous palm operation in one embodiment of the present invention; Figure 4 A flowchart of the working principle of the palm operation of a three-dimensional reconstruction method for dexterous palm operation in one embodiment of the present invention; In the figure: 1- multi-spectral monocular camera, 2- white light emitter, 3- infrared light emitter, 4- ultraviolet light emitter, 5- rotating mechanism, 6- housing, 7- control module, 8- intelligent terminal. DETAILED DESCRIPTION

[0015] A three-dimensional reconstruction system and method for dexterous palm operation is described in detail, subject to implementation by technicians in this field.

[0016] like Figure 1-2 As shown, a three-dimensional reconstruction system for dexterous palm operation includes a housing 6, a multi-spectral monocular camera 1, a white light emitter 2, an infrared light emitter 3, an ultraviolet light emitter 4, a rotating mechanism 5, a control module 7 and an intelligent terminal 8.

[0017] The shell 6 includes three arms and is a centrally symmetrical structure made of metal or plastic. It is installed in the palm of the dexterous hand through a rotating mechanism 5. Therefore, the shell 6 can move flexibly with the movement of the dexterous hand, thereby ensuring that the system accurately obtains the surface information of the object being measured during various operations.

[0018] The multispectral monocular camera 1 is installed at the front end of the geometric center of the shell 6, and has three shooting modes: visible light shooting mode, infrared light shooting mode and ultraviolet light shooting mode; the control module 7 is used to control the multispectral monocular camera 1 to quickly switch shooting modes; the multispectral monocular camera 1 takes into account the close-range shooting needs of the dexterous palm, so its shooting angle is specially optimized to cover as large a field of view as possible.

[0019] The white light emitter 2, the infrared light emitter 3 and the ultraviolet light emitter 4 are installed at the ends of the three arms.

[0020] The white light emitter 2 is used to emit white light.

[0021] The infrared light emitter 3 is used to emit infrared light with a wavelength of 780nm to 900nm, and is suitable for infrared imaging with strong penetration.

[0022] The ultraviolet light emitter 4 is used to emit ultraviolet light with a wavelength of 320nm to 400nm, and the ultraviolet light is used to capture tiny details on the surface of the object being measured.

[0023] The rotating mechanism 5 is installed at the rear end of the geometric center of the housing 6 and is used to drive the housing 6 to rotate at a specific angle in a predetermined direction and speed.

[0024] The control module 7 and the intelligent terminal 8 are located behind the housing 6, and the control module 7 and the intelligent terminal 8 are connected via a local area network.

[0025] Furthermore, the rotating mechanism 5 includes a small servo motor, an angle sensor module and a mechanism housing.

[0026] Small servo motors can turn a specific angle in a predetermined direction and speed; The mechanism housing is made of metal or plastic to provide protection for the entire rotating mechanism 5; The mechanism shell is fixedly connected to the dexterous hand. During the operation in the palm of the dexterous hand, the entire shell 6 moves with the movement of the dexterous hand to help the mechanical arm obtain the situation near the palm of the dexterous hand.

[0027] Furthermore, the angle sensor module includes an angle sensor and a data sending circuit. The angle sensor can measure the angle of the rotating mechanism 5, and the data sending circuit can send angle data to the control module 7. The angle of the shell 6 can be measured through the angle sensor module, and real-time feedback information of the posture of the shell 6 can be obtained to help the system make more accurate decisions.

[0028] Furthermore, the control module 7 includes a central processing unit, a servo drive circuit, a local area network interface, and a power supply circuit.

[0029] The central processing unit is responsible for processing the data received from each sensor and camera, and controlling the movement of the servo motor through the servo drive circuit; the servo drive circuit can control the small servo motor to rotate a specific angle in a predetermined direction and speed; the local area network interface is used for communication between the control module 7 and the intelligent terminal 8; the power supply circuit is used for power supply.

[0030] Embodiment 1 Reference Figure 3 The three-dimensional reconstruction working principle flow chart of a three-dimensional reconstruction system applied to dexterous palm operation shown in the figure includes the following steps: Step 1: Connect the device to the power supply, fix it in the measuring position, and aim it at the object to be measured. The relative distance between the object to be measured and the device should remain unchanged. At this time, the housing 6 is at the initial angle. attitude.

[0031] Step 2: The device enters the debugging mode, and the white light emitter 2, infrared light emitter 3 and ultraviolet light emitter 4 emit white light and wavelength respectively. Infrared light and wavelength The multi-spectral monocular camera 1 takes pictures of the object under different spectra to obtain three multi-level spectral images. , and , and sent to the intelligent terminal 8.

[0032] Step 3: The intelligent terminal 8 analyzes the reflective characteristics of the object surface based on the received image data. The reflectivity is modeled using the following formula: ; in, Indicates at location At, wavelength Reflectivity under Indicates at location At, wavelength The intensity of reflected light under For the location At, wavelength According to the analysis results, the intelligent terminal 8 adjusts the intensity of the light source and angle to optimize image quality.

[0033] Step 4: The adjusted white light emitter 2, infrared light emitter 3 and ultraviolet light emitter 4 illuminate the object to be measured in turn, and the multispectral monocular camera 1 takes pictures under the three light sources respectively to obtain three optimized multi-level spectral images. , , .

[0034] Step 5: Rotate the mechanism 5 120° clockwise. At this time, the housing 6 is in Repeat steps 2 to 4 to obtain 3 new multi-level spectral images. , , ; Then the rotating mechanism 5 rotates 120° clockwise, and the housing 6 is in Repeat steps 2 to 4 to obtain the last three images. , , ; Thus, three rounds of shooting were completed and 9 multi-level spectral images were obtained.

[0035] Step 6: Use photometric stereo to calculate the surface normal vectors of objects at three levels , , ; Photometric stereo is based on the following formula: ; in, is in position The reflected light intensity at is in position The surface reflectivity at is a known quantity; is the unit vector of the light source direction; For the location The surface normal vector at . The surface normal vector can be solved using three independent equations .

[0036] The three levels of normal vector data are passed as input to the deep learning model for nonlinear fusion. The trained model is used to learn the normal vectors of each level. , , The complex relationship between them automatically adjusts the fusion weight , , , and generate the fused normal vector: ; in, It refers to the deep learning model. Finally, the fused normal vector is fused using the photometric stereo method. Processing is performed to generate high-precision three-dimensional images.

[0037] Step 7: The intelligent terminal 8 analyzes the high-precision three-dimensional image and generates an in-palm operation strategy according to the surface shape of the object being measured. This process can be achieved by solving the inverse kinematics equation To plan the motion path of the dexterous hand, is the Jacobian matrix, is the joint velocity vector, is the velocity vector of the end effector. The dexterous hand performs palm-mounted operations according to the planned path, enabling it to complete tasks more accurately.

[0038] Embodiment 2 Reference Figure 4 The flowchart of the working principle of the palm operation of a 3D reconstruction system applied to the dexterous palm operation shown in the figure includes the following steps: Step 1: According to the preset path planning, the robot arm controls the dexterous hand to move to the initial position close to the object to be grasped. Then the 3D reconstruction system in the dexterous hand scans one surface of the object to collect surface information, and then uses the deep learning algorithm to process the point cloud data and analyze the surface geometry and material information; Step 2: The intelligent terminal analyzes the collected surface information to determine the surface stability, friction and pressure resistance of the surface. If the surface of the grasping point is determined to be too smooth or unstable, the system will issue an alarm to prompt the dexterous hand to move to the next position. Step 3: The dexterous hand moves around the object to another position according to the optimal path planned by the algorithm and performs multispectral scanning again. Here, the scanning and analysis process in steps 1 and 2 is repeated to obtain the geometric data of the surface of the object and re-evaluate the safety of the grasping point. Step 4: If a safe grasping point is still not found at the current position, the dexterous hand will continue to move around the object, scanning different positions in sequence until a suitable grasping point is found. The data of each scan will be transmitted to the smart terminal, which will evaluate the best grasping position based on the results of the previous scans. Step 5: When it is determined that the grasping point meets the safety standards, the system sends a confirmation signal, and the dexterous hand performs the grasping action, adjusts the position of the fingers, accurately grasps the object and completes the operation.

[0039] The beneficial effects of the present invention are: (1) Compared with the traditional photometric stereo method that only obtains object surface information at the visible light level, the present invention obtains object information at three levels: visible light, infrared light, and ultraviolet light, and uses a 360° rotatable mechanism and a multi-spectral light source to work together, expanding the way to obtain information and improving the accuracy and efficiency of three-dimensional reconstruction; (2) The present invention provides a multi-level spectral nonlinear fusion method, which integrates the advantages of different wavelengths by training a neural network model to nonlinearly fuse normal vector data of different wavelengths, thereby improving the accuracy and efficiency of three-dimensional reconstruction and the efficiency of dexterous palm operation; (3) Compared with the fixed light source configuration of the traditional photometric stereo method, the present invention dynamically monitors and adjusts the intensity and angle of the light source, improves the flexibility of light source configuration, and optimizes lighting conditions; (4) The multi-spectral fusion 3D reconstruction system and method provided by the present invention brings new possibilities to the fields of dexterous palm operation, industrial inspection and cultural relics protection.

[0040] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A 3D reconstruction system for dexterous palm manipulation, characterized in that: The three-dimensional reconstruction system includes a housing, a multi-spectral monocular camera, a white light emitter, an infrared light emitter, an ultraviolet light emitter, a rotating mechanism, a control module and an intelligent terminal; The housing comprises three arms, is a centrally symmetrical structure, is made of metal or plastic, and is mounted on the palm of the dexterous hand through the rotating mechanism; The rotating mechanism is installed at the rear end of the geometric center of the housing and is used to drive the housing to rotate at a specific angle in a predetermined direction and speed; The multispectral monocular camera is installed at the front end of the geometric center of the shell and has three shooting modes: visible light shooting mode, infrared light shooting mode and ultraviolet light shooting mode; The white light emitter, the infrared light emitter and the ultraviolet light emitter are installed at the ends of the three arms; The white light emitter is used to emit white light; The infrared light emitter is used to emit infrared light; The ultraviolet light emitter is used to emit ultraviolet light; The control module is implanted in the arm of the dexterous hand and is connected to the intelligent terminal via a local area network.

2. A 3D reconstruction system for dexterous palm manipulation according to claim 1, characterized in that: The rotating mechanism includes a small servo motor, an angle sensor module and a mechanism housing; The small servo motor can rotate a specific angle in a predetermined direction and speed; The mechanism shell is made of metal or plastic.

3. The three-dimensional reconstruction system for dexterous palm manipulation according to claim 2, characterized in that: The angle sensor module includes an angle sensor and a data sending circuit. The angle sensor can measure the angle of the rotating mechanism, and the data sending circuit can send angle data to the control module.

4. The three-dimensional reconstruction system for dexterous palm manipulation according to claim 1, characterized in that: The control module includes a central processing unit, a servo drive circuit, a local area network interface, and a power supply circuit; The central processing unit is responsible for processing the data received from the sensors and the camera, and controlling the movement of the servo motor through the servo drive circuit; The servo drive circuit can control the small servo motor to rotate at a specific angle in a predetermined direction and speed; The LAN interface is used for communication between the control module and the intelligent terminal; The power supply circuit is used for supplying power.

5. A three-dimensional reconstruction method for dexterous palm operation, applied to a three-dimensional reconstruction system for dexterous palm operation as claimed in any one of claims 1 to 4, characterized in that: The following steps are involved: S1. Each time the shell rotates once, the white light emitter, infrared light emitter and ultraviolet light emitter respectively emit white light, infrared light and ultraviolet light to illuminate the object to be measured, and the multi-spectral monocular camera takes 3 multi-level spectral images in each posture. After the shell rotates 3 times, a total of 9 multi-level spectral images are obtained; S2, using photometric stereo method to calculate the surface normal vectors of the object under test at three levels respectively; S3, then use the deep learning algorithm to nonlinearly fuse the data at each level, and automatically adjust the fusion weight through the training model to obtain a high-precision three-dimensional image; S4. Finally, the surface shape of the object to be measured is obtained by analyzing the high-precision three-dimensional image for operation in the dexterous palm.

6. The three-dimensional reconstruction method for dexterous palm manipulation according to claim 5, characterized in that: The method further comprises: Each time the shell is in a new posture, the multispectral monocular camera first enters the debugging mode to monitor the intensity and spectral distribution of reflected light in real time; the intelligent terminal analyzes the surface reflection characteristics of the object under test, and the control module dynamically adjusts the intensity and angle of the light source to optimize multispectral data acquisition.

Images