Visual assist robot system for brain electrode assembly and assembly method thereof

Abstract

The application provides a visual auxiliary robot system for brain electrode assembly and an assembly method thereof, and relates to the technical field of brain electrode assembly, and comprises a camera, a support, a displacement table, a water tank, a robot, an upper light source, a lower light source and a light source controller; the camera is supported by the support and suspended directly above the water tank; the upper light source is installed below the camera lens and close to the end of the water tank, and the lower light source is placed below the water tank; the robot carries a guide needle and moves on the displacement platform located above the water tank; and the light source controller is used for controlling the illumination of the upper light source and the lower light source. The application can save time and labor in the automatic assembly process, has a lower requirement for the proficiency of operators, and does not have the risk of scratching the electrode.

Description

Technical Field

[0001] This invention relates to the field of brain electrode assembly technology, and more specifically, to a visual assistive robot system for brain electrode assembly and its assembly method. Background Technology

[0002] In brain-related research, methods similar to Figure 1 Medium-sized electrodes are implanted into the cerebral cortex to collect brain signals. These electrodes are typically 50 micrometers wide and 1-2 micrometers thick, with an opening of about 20 micrometers in diameter at their tip.

[0003] The extremely thin and light structure of the electrode prevents it from penetrating the cortex and entering the brain independently for implantation. Therefore, a rigid tungsten wire is often used as a guide needle to assist in the implantation process. The process of the tungsten wire's tip passing through the electrode hole and securing itself firmly is called the "assembly" process. After assembly, the tungsten wire pierces the cerebral cortex and carries the electrode in. Once the target depth is reached, the tungsten wire is withdrawn, and the electrode remains in the brain due to the pressure of the brain tissue, thus completing the implantation process.

[0004] Currently, the "assembly" task is mainly completed manually by operators. This manual assembly process is time-consuming, labor-intensive, requires a high level of operator skill, and carries the risk of scratching the electrodes. Therefore, an automated electrode assembly system is meaningful. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a visual assistive robot system for brain electrode assembly and its assembly method.

[0006] According to the present invention, a visual assistive robot system and its assembly method for assembling brain electrodes are provided, the solution of which is as follows:

[0007] In a first aspect, a visual-assisted robot system for assembling brain electrodes is provided, the system comprising: a camera, a support, a displacement stage, a water tank, a robot, an upper light source, a lower light source, and a light source controller;

[0008] The camera is suspended directly above the water tank by a bracket;

[0009] The upper light source is installed below the camera lens near the end of the water tank, and the lower light source is placed below the water tank.

[0010] The robot, carrying a guide needle, moves on a displacement platform located above the water tank;

[0011] The light source controller is used to control the upper and lower light sources for illumination.

[0012] Preferably, the transparent material of the water tank allows light to pass through the bottom of the tank for illumination.

[0013] Preferably, the upper light source has a strong illumination effect, while the lower light source has a relatively weak illumination effect.

[0014] Preferably, the weight of the light source should be increased when the environment is dark or the magnification is high.

[0015] Preferably, the displacement platform is horseshoe-shaped and the water tank is circular; in a top view, the water tank is located in the recess of the displacement platform.

[0016] Secondly, a method for assembling a vision-assisted robot for brain electrode assembly is provided, the assembly method comprising:

[0017] Step S1: Place the electrode in a fixed position in the water tank, turn on the camera and adjust the camera bracket so that the camera is focused on the electrode on the water surface;

[0018] Step S2: Control the robot to move on the displacement platform, so that it moves the guide pin into the camera's field of view, and adjust the guide pin up and down so that the tip of the pin approaches the electrode plane from above;

[0019] Step S3: Identify the electrode hole, the tip of the guide needle, and the direction of the centerline;

[0020] Step S4: After identification, the tungsten filament movement and electrode assembly begin automatically;

[0021] Step S5: After completion, remove the electrode and guide pin for later use.

[0022] Preferably, step S4 includes:

[0023] Step S4.1: Adjust the camera bracket so that the electrodes and guide pins are clearly visible in the field of view;

[0024] Step S4.2: The guide pin moves from far to near the electrode, and the speed of movement gradually decreases. Fine adjustments are made until the tip of the tungsten wire reaches directly above the electrode hole.

[0025] Step S4.3: Control the guide pin to descend and pass through the electrode hole to complete the assembly.

[0026] Preferably, in step S4.2, the distance between the guide pin and the electrode is greater than 200 pixels, and the movement speed is 250-1000 μm / s; when the distance is greater than 200 pixels, the movement speed is reduced to 10-50 μm / s.

[0027] Preferably, in step S4.3, since the mechanical arm controlling the guide pin descends not in a vertical motion, but in a circular motion around the rear end, the guide pin needs to move forward 100 pixels to allow for compensation.

[0028] Thirdly, a computer-readable storage medium storing a computer program is provided, which, when executed by a processor, implements the steps of the visual-assisted robot assembly method for brain electrode assembly.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. By using visual-assisted detection and visual servo control, the completely random manual assembly process is replaced, thereby improving assembly efficiency and reducing electrode damage during the assembly process.

[0031] 2. By establishing a robot system with nanometer-level displacement accuracy, the automated assembly of electrodes was completed, achieving the technical effect of freeing up manpower and shortening assembly time.

[0032] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0033] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0034] Figure 1 This is a schematic diagram of the electrodes;

[0035] Figure 2 This is a schematic diagram of the overall structure of the present invention;

[0036] Figure 3 This is a schematic diagram of the displacement platform and the water tank;

[0037] Figure 4 This is a schematic diagram of the assembly process;

[0038] Figure 5 This is a schematic diagram of the experimental results;

[0039] Figure 6 This is a schematic diagram of the assembly process for 60-micron and 90-micron mounting holes to detect artifacts.

[0040] Reference numerals: 1. Robot; 2. Water tank; 3. Displacement stage; 4. Tungsten filament; 5. Electrode; 6. Camera; 7. Camera bracket; 8. Upper light source; 9. Lower light source; 10. Water tank bracket. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] This invention provides a visual assistive robot system for fitting brain electrodes, referring to... Figure 2 As shown, the system specifically includes: camera 6, bracket 7, displacement stage 3, water tank 2, robot 1, upper light source 8, lower light source 9, and light source controller; wherein, camera 6 is supported by camera bracket 7 and suspended directly above water tank 2; upper light source 8 is installed below the lens of camera 6 near the end of water tank 2, and lower light source 9 is placed below water tank 2. The transparent material of water tank 2 allows light to pass through the bottom of water tank 2 for illumination. Upper light source 8 has a strong illumination effect, while lower light source 9 has a relatively weak illumination effect. When the environment is dark or the magnification is high, the weight of lower light source 9 should be increased.

[0043] Reference Figure 3 As shown, the displacement platform 3 is horseshoe-shaped, and the water tank 2 is circular; in a top view, the water tank 2 is located in the recess of the displacement platform 3. The robot 1, carrying a guide needle, moves on the displacement platform located above the water tank 2; the light source controller is used to control the upper light source 8 and the lower light source 9 for illumination. In this embodiment, the guide needle is the tungsten filament 4.

[0044] This invention also provides a method for assembling a vision-assisted robot for brain electrode assembly, the method specifically comprising:

[0045] Step S1: Place electrode 5 in a fixed position in water tank 2, turn on camera 6 and adjust camera bracket 7 so that camera 6 is focused on electrode 5 on the water surface;

[0046] Step S2: Control the robot 1 to move on the displacement platform, so that it moves the guide needle into the field of view of the camera 6, and adjust the guide needle up and down so that the needle tip approaches the plane of the electrode 5 from above;

[0047] Step S3: Identify the direction of electrode 5 holes, guide needle tip, and centerline;

[0048] Step S4: After identification, the guide pin movement and electrode 5 assembly will begin automatically;

[0049] Step S4 specifically includes:

[0050] Step S4.1: Adjust the camera bracket 7 so that the electrode 5 and the guide pin can be clearly seen in the field of view;

[0051] Step S4.2: The guide pin moves from far to near the electrode 5, and the movement speed gradually decreases. Fine adjustments are made until the tip of the tungsten wire 4 reaches directly above the hole of the electrode 5. In this step, the distance between the guide pin and the electrode 5 is >200 pixels, and the movement speed is 250-1000um / s. When the distance is >200 pixels, the movement speed is reduced to 10-50um / s.

[0052] Step S4.3: Control the guide pin to descend and pass through electrode hole 5 to complete the assembly. Since the robotic arm controlling the guide pin descends not in a vertical motion, but in a circular motion around the rear end, the guide pin needs to move forward 100 pixels to allow for compensation.

[0053] Step S5: After completion, remove electrode 5 and guide pin for later use.

[0054] The present invention will now be described in more detail.

[0055] Reference Figure 2 As shown, this invention provides a visual-assisted robot system for assembling brain electrodes. The system includes a high-resolution RGB camera 6, a camera mount 7 adjustable with three degrees of freedom, an upper light source 8 located above the operating table, a lower light source 9 located below the operating table, and a light source controller. The illumination of the visual system is achieved jointly by the upper and lower light sources 9. The upper light source 8 provides strong illumination, but it is prone to halos when applied to water surfaces, affecting observation clarity; the lower light source 9 provides softer but weaker illumination and does not produce halos. In the electrode assembly task, the typical illumination strategy is "upper light source 8 as the primary source, lower light source 9 as the secondary source," but when the environment is dark or the magnification is high, the weight of the lower light source 9 should be appropriately increased.

[0056] Reference Figure 3 As shown, electrode 5 is placed in a circular water tank 2 and floats on the water surface. The aquatic environment provides a more stable assembly environment than air. A four-DOF robot 1 with nanometer displacement precision carries a tungsten wire 4 and moves on a horseshoe-shaped displacement platform located above the water tank 2. The camera 6 in the vision system is supported by a camera bracket 7 and suspended directly above the water tank 2. The upper light source 8 is installed below the lens near the end of the water tank 2. The lower light source 9 is placed below the water tank 2, which is located above the water tank bracket 10. The transparent material of the water tank 2 allows light to pass through the bottom of the water tank 2 for illumination.

[0057] Reference Figure 4As shown, the assembly process is performed with the assistance of computer vision. In the field of view of the high-resolution camera 6, the electrode 5 hole and the needle tip can be clearly seen. We first identify the tungsten wire 4 using a deep neural network. We manually labeled some of the tungsten wire segmentation data using a thresholding method and expanded the dataset using data augmentation. We then used these pseudo-labels to train the designed convolutional neural network, employing a common multi-layer convolutional neural network as its backbone. The trained network can automatically segment the tungsten wire from the input test image, separating the tungsten wire 4 from other objects in the background. Further, based on the segmented image of the tungsten wire 4, we extracted the centerline of the segmented tungsten wire 4 skeleton using dilation and erosion operations in morphology. The tilt angle of the tungsten wire 4 was calculated using the angle between the centerline and a given direction (horizontal direction), and the coordinates of the tip of the tungsten wire 4 in the coordinate system of camera 6 were calculated using the position of the tip of the centerline. The coordinates of the tip of the tungsten wire 4 were used as the starting point of the motion. To train the deep neural network for segmenting the tungsten wire 4, we generated some data using manual annotation and expanded the data volume using data augmentation. We further determined the position of the electrode 5 hole through circular hole detection and used optical flow to track the detected motor hole, with the coordinates of its center considered as the endpoint of the motion. Since the robot 1 used could not perform velocity synthesis, the point-to-point motion needed to be decomposed into two steps: one along the centerline of the tungsten wire 4 and the other perpendicular to the centerline of the tungsten wire 4.

[0058] The movement consists of five steps: 1. Manually adjust the camera bracket 7 to ensure that the electrode 5 and tungsten wire 4 are clearly visible in the field of view; 2. Move at a higher speed (usually 250-1000um / s) when the distance is relatively far; 3. When the distance is relatively close (<200 pixels), reduce the movement speed (10-50um / s) and make fine adjustments until the tip of the tungsten wire 4 reaches directly above the hole of the electrode 5; 4. Since the descent of the robotic arm 1 is not a vertical movement, but a circular movement around a point at the rear end, the tungsten wire 4 needs to move forward 100 pixels to allow for compensation; 5. Control the tungsten wire 4 to descend and pass through the hole of the electrode 5 to complete the assembly.

[0059] In addition to electrode 5, the algorithm is also applicable to the assembly of all perforated sheet objects with needles.

[0060] When starting the electrode 5 assembly task:

[0061] 1. Place electrode 5 in a fixed position in water tank 2, turn on camera 6 and adjust camera bracket 7 so that camera 6 is focused on electrode 5 on the water surface.

[0062] 2. Control robot 1 to move on the plane, so that it moves the guide needle into the field of vision, and adjust the guide needle up and down so that the needle tip approaches the plane of electrode 5 from above.

[0063] 3. Begin identifying the direction of the electrode 5 holes, the tip of the guide needle, and the centerline.

[0064] 4. After identification, the tungsten wire 4 movement and electrode 5 assembly process are automatically carried out according to the control strategy shown in the figure above.

[0065] 5. After completion, remove electrode 5 and the guide pin for later use.

[0066] Experimental results are as follows Figure 5 As shown. (ad) Screenshot of key nodes in the assembly process of electrode 5 (left) and tungsten wire 4. (b) When the distance is relatively far, identification is completed and movement begins. (c) When the distance is close, the speed is switched to low for fine-tuning. (d) Compensation is reserved for the descent of the guide pin. To better demonstrate the versatility of the assembly system and software, a larger artifact with assembly holes of different diameters was customized. (eh) Assembly process of the 30-micron diameter electrode 5 artifact.

[0067] Appendix Figure 6 The image shown is a screenshot illustrating the assembly process of the 60-micron and 90-micron mounting holes for artifacts.

[0068] This invention provides a visual-assisted robot system and assembly method for assembling brain electrodes. The automatic assembly process is time-saving and labor-saving, requires less operator skill, and eliminates the risk of scratching the electrodes.

[0069] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0070] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A visual aid robotic system for brain electrode assembly, characterized by, Comprising: a camera, a camera holder, a displacement table, a water tank, a robot, an upper light source, a lower light source, and a light source controller; the camera is supported by the camera holder and suspended directly above the water tank; the upper light source is installed below the camera lens and close to the end of the water tank, and the lower light source is placed below the water tank; the robot carries a guide needle to move on the displacement table above the water tank; the light source controller is used to control the illumination of the upper light source and the lower light source.

2. The visual aid robotic system for brain electrode assembly of claim 1, wherein, The transparent material of the water tank allows light to pass through the bottom of the water tank for illumination.

3. The visual aid robotic system for brain electrode assembly of claim 1, wherein, The upper light source has strong illumination effect, and the lower light source has relatively weak illumination effect.

4. The visual aid robotic system for brain electrode assembly of claim 3, wherein, When the environment is relatively dark or the magnification is relatively high, the weight of the lower light source should be increased.

5. The visual aid robotic system for brain electrode assembly of claim 1, wherein, The displacement table is arranged in a horseshoe shape, and the water tank is arranged in a circular shape; in a top view, the water tank is located in the concave part of the displacement table.

6. A vision-assisted robotic assembly method for brain electrode assembly based on the vision-assisted robotic system for brain electrode assembly according to any one of claims 1-5, characterized in that, Comprising: Step S1: Place the electrode at a fixed position in the water tank, turn on the camera and adjust the camera holder to focus the camera on the electrode at the water surface; Step S2: Control the robot to move on the displacement table to drive the guide needle to move into the field of view of the camera, and adjust the guide needle up and down to make the needle tip approach the electrode plane from above; Step S3: Identify the electrode hole, guide needle tip, and centerline direction; Step S4: After identification, automatically perform tungsten wire movement and electrode assembly; Step S5: After completion, remove the electrode and guide needle for use.

7. The visual aid robotic assembly method for brain electrode assembly of claim 6, wherein, The step S4 comprises: Step S4.1: Adjust the camera holder so that the electrode and guide needle can be clearly seen in the field of view; Step S4.2: The distance between the guide needle and the electrode gradually decreases, and the movement speed gradually decreases for fine adjustment until the tungsten wire tip reaches directly above the electrode hole; Step S4.3: Control the guide needle to descend and pass through the electrode hole to complete the assembly.

8. The visual aid robotic assembly method for brain electrode assembly of claim 7, wherein, In step S4.2, the distance between the guide needle and the electrode is > 200 pixels, and the movement speed is 250-1000um / s; when the distance is < 200 pixels, the movement speed is reduced to 10-50um / s.

9. The visual aid robotic assembly method for brain electrode assembly of claim 7, wherein, In step S4.3, since the mechanical arm controlling the guide needle descends is not vertical movement, but ring movement around the rear end, the guide needle needs to move forward by 100 pixels to reserve compensation.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the visual auxiliary robot assembly method for brain electrode assembly according to any one of claims 6 to 9.

Images