An automatic pickup method and system for flexible electrodes

Abstract

This application provides an automatic pickup method and system for flexible electrodes. The method includes: acquiring a test image using a microscope camera; obtaining a target region based on the test image; and iteratively focusing the target region; measuring the pose of an implanted needle and a flexible electrode within the focused target region to obtain pose features of the implanted needle and flexible electrode; and automatically pickup of the flexible electrode based on these pose features. The pose features include position features and orientation features. To improve the accuracy of position and orientation measurements, this application proposes a two-step target detection and pose measurement algorithm. First, a target detection algorithm suitable for out-of-focus scenes is studied, achieving automatic focusing of out-of-focus targets. Then, the location and orientation measurement of key points of the target within the target region are achieved, effectively avoiding image compression and background interference, and significantly improving accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to an automatic pickup method and system for flexible electrodes. Background Technology

[0002] Flexible electrodes, due to their excellent flexibility and biocompatibility, can reduce damage to brain tissue while providing more stable and durable signal recording performance. However, because of their small size (on the order of tens of micrometers), manual implantation is quite difficult. To ensure the accuracy and efficiency of flexible electrode implantation, robotic-controlled implantation needles are needed to achieve precise pickup of the flexible electrodes. This requires high-precision pose measurement of the implantation needle and the flexible electrode, and the execution of a reliable alignment and insertion control procedure.

[0003] For locating the tip of the implanted needle and the center point of the flexible electrode ring, traditional image processing algorithms can be used, such as edge detection and line fitting, to obtain the coordinates of the implanted needle tip through the intersection of characteristic lines; edge detection and ellipse fitting, to calculate the center of the ellipse as the coordinates of the center point of the flexible electrode ring; or deep learning methods can be used, by collecting and annotating images, training the network, and directly obtaining the coordinates of key points from the input images.

[0004] In existing technologies, the pose measurement of implanted needles and flexible electrode rings can be achieved by edge detection, line fitting, ellipse fitting, template matching, etc., to obtain the pose vector. However, these methods are affected by image compression and background and lighting, resulting in poor accuracy and robustness in pose measurement. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic pickup method and system for flexible electrodes, aiming to solve the above-mentioned problems in the prior art.

[0006] This invention provides an automatic pickup method for flexible electrodes, comprising:

[0007] A test image is acquired using a microscope camera; the target region is obtained from the test image; and the target region is iteratively focused upon.

[0008] The implanted needle and flexible electrode are subjected to pose measurement in the target area after focusing to obtain the pose features of the implanted needle and flexible electrode, and the flexible electrode is automatically picked up based on the pose features; wherein, the pose features include position features and attitude features.

[0009] This invention provides an automatic pickup system for flexible electrodes, comprising:

[0010] A focusing module is used to acquire an image to be tested using a microscope camera, obtain the target region to be tested based on the image, and iteratively focus on the target region to be tested; and

[0011] The pickup module is used to perform pose measurement on the implanted needle and flexible electrode in the focused target area to obtain the pose features of the implanted needle and flexible electrode, and to automatically pick up the flexible electrode based on the pose features; wherein, the pose features include position features and attitude features.

[0012] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described automatic pickup method for flexible electrodes.

[0013] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described automatic pickup method for flexible electrodes.

[0014] The embodiments of this invention can include the following beneficial effects: To improve the accuracy of position and pose measurement, the embodiments of this invention propose a two-step target detection and pose measurement algorithm. Specifically, firstly, a target detection algorithm suitable for out-of-focus scenes is studied, realizing automatic focusing of out-of-focus targets. Then, the localization of key points of the target and pose measurement are realized within the target area, effectively avoiding image compression and background interference, and achieving a significant improvement in accuracy and robustness. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of an automatic pickup method for flexible electrodes according to an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of the pose measurement method according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the ODKL-Net network structure according to an embodiment of the present invention;

[0019] Figure 4 This is an automatic focusing flowchart of an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of the target template and feature primitives in an embodiment of the present invention;

[0021] Figure 6 This is a control flowchart of the implantation needle picking up the flexible electrode according to an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of an automatic pickup system for flexible electrodes according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0024] Method Implementation Examples

[0025] According to embodiments of the present invention, an automatic pickup method for flexible electrodes is provided. Figure 1 This is a flowchart of the automatic pickup method for flexible electrodes according to an embodiment of the present invention, as shown below. Figure 1 As shown, the automatic pickup method for flexible electrodes according to an embodiment of the present invention specifically includes:

[0026] Step S101 involves acquiring a test image using a microscope camera, obtaining the target region based on the test image, and iteratively focusing on the target region. Specifically, this includes:

[0027] The image to be tested is input into the target detection network ODKL-Net for inference. The feature extractor ResPNet extracts features from the image to be tested. The extracted features are pooled using RoIAlign to obtain the target region to be tested. The hill climbing method is used to iteratively focus the target region to be tested.

[0028] The iterative focusing of the target area using the hill-climbing method specifically includes:

[0029] S31: Move the focusing slide from the initial position according to the preset initial step size;

[0030] S32: Calculate the evaluation function after the movement. If the absolute value of the difference between the evaluation function before and after the movement is greater than a preset threshold, and the evaluation function after the movement is greater than the evaluation function before the movement, then continue to move the focusing slide with the current step size; if the evaluation function after the movement is less than the evaluation function before the movement, then reduce the step size and move the focusing slide in the opposite direction.

[0031] S33: Iterate through S32 until the absolute value of the difference between the evaluation function before and after the movement is less than or equal to the preset threshold, then stop moving the focusing slide and take the current position of the microscope camera as the optimal focusing position.

[0032] The evaluation function is a point sharpness function, and its expression is shown in Formula 1.

[0033]

[0034] Where P represents the evaluation function value, m and n are the length and width of the image, df is the gray level change magnitude, dx is the distance increment between pixels, i represents the calculation of the local gradient of the i-th pixel, and a represents the a-th neighboring pixel.

[0035] Step S102: The pose of the implanted needle and flexible electrode is measured in the focused target area to obtain their pose features. Based on these pose features, the flexible electrode is automatically picked up. The pose features include positional features and orientation features, specifically including:

[0036] After focusing, the output feature map of the last layer decoding module of ResPNet is cropped to obtain the feature map of the target region to be tested. The feature map is input into the key point localization branch of ODKL-Net. The feature map is convolved by 3×3 convolution kernel and 1×1 convolution kernel to obtain a single-channel heat map. The single-channel heat map is subjected to argmax operation to obtain the position coordinates of the implanted needle tip and the center point of the flexible electrode ring as shown in Formula 2.

[0037]

[0038] Where H(u,v) represents the response value of the output keypoint location heatmap at position (u,v), (u key ,v key () represents the image coordinates of key points;

[0039] A standard image is selected according to a preset standard. Edge detection is performed on the standard image and redundant features are removed to obtain a target template and feature primitives. The Canny algorithm is used to perform edge detection on the focused target area to generate a search map. Template matching is performed on the target template and the search map using a template matching algorithm based on rotation and feature pyramid. The feature primitives are rotated according to the rotation angle obtained by matching, and edge points are searched along the normal of the obtained feature line. The edge points are fitted with RANSAC to obtain a new feature line.

[0040] Calculate the midpoint of the new feature line, obtain the central axis of the implantation needle based on the midpoint, and use the central axis as the attitude vector of the implantation needle;

[0041] Points are uniformly selected on the new feature line in the left-view image. For each point obtained, the epipolar line in the right-view image is obtained through epipolar constraint. The intersection point of the epipolar line and the new feature line in the right-view image is calculated. The intersection point is used as the matching point of the corresponding point. RANSAC plane fitting is performed on the matching point. The obtained plane normal vector is used as the normal vector of the flexible electrode ring. The normal vector of the flexible electrode ring is reprojected into the image space to obtain the attitude vector of the flexible electrode ring.

[0042] Based on the pose characteristics of the implanted needle and the flexible electrode, the position error and angle error of the implanted needle are obtained using the image Jacobian matrix. The three-dimensional translational motion and three-dimensional rotational motion of the implanted needle are calculated using the position error and angle error. Based on the three-dimensional translational motion and three-dimensional rotational motion, the pose of the implanted needle is adjusted using a controller. After adjustment, the implanted needle is inserted into the flexible electrode ring, and the flexible electrode is automatically picked up.

[0043] The above technical solutions of the present invention will be described in detail below with reference to the specific details of the automatic pickup method for flexible electrodes in the embodiments of the present invention.

[0044] This invention proposes an automatic flexible electrode pickup method for minimally invasive brain-computer interface implantation applications, including a method for measuring the pose of the implantation needle and the flexible electrode, as well as an automatic alignment and insertion control process.

[0045] The procedure for measuring the pose of implanted needles and flexible electrodes is as follows: Figure 2 As shown, the target area to be measured is first obtained using a target detection network, then the target area to be measured is automatically focused, and finally the tip of the implanted needle and the center point of the flexible electrode ring are detected respectively, and the attitude vectors of the implanted needle and the flexible electrode ring are calculated.

[0046] To achieve object detection, this invention proposes a two-stage object detection network, ODKL-Net, with the network structure as follows: Figure 3 As shown, this network combines ResNet50 with a feature pyramid structure to construct the feature extractor ResPNet. To reduce potential accuracy loss during pooling, RoIAlign is introduced for pooling operations. During training, a zero-padding strategy is used instead of resizing the image to maintain image proportions and minimize compression, thereby improving localization accuracy.

[0047] After obtaining the target area, the hill-climbing method is used to achieve automatic focusing of the camera. The hill-climbing method is a commonly used local optimization algorithm for automatic focusing. This method continuously adjusts the focal length to find the position where the evaluation function reaches its maximum value. The point sharpness function is used as the evaluation function, which is based on the local gradient information of the image and can effectively reflect the image sharpness. Its formula is shown below:

[0048]

[0049] Where m and n are the length and width of the image, df is the grayscale change magnitude, dx is the distance increment between pixels, i represents the calculation of the local gradient of the i-th pixel, and a represents the a-th neighboring pixel. The autofocus process is as follows: Figure 4 As shown, specifically as follows:

[0050] a) Starting from the initial position, move the focusing slide with a relatively long initial step size;

[0051] b) Calculate the evaluation function value after the move. If the absolute value of the difference between the two evaluation functions is greater than a certain threshold, move the focusing slide. If the evaluation function value after the move is higher than the evaluation function value at the current position, continue moving the focusing slide with the current step size; otherwise, decrease the step size and move the focusing slide in the opposite direction.

[0052] c) Repeat step b until the difference between the two evaluation function values ​​is less than a certain threshold, then stop the search. At this point, the camera's position is the optimal focus position.

[0053] After focusing on the target region, the position and orientation of the implanted needle and flexible electrode are measured. To reduce GPU memory usage, the feature map of the target region is directly cropped from the output feature map of the last decoding module of the ResPNet target detection feature extractor. This cropped map is then input into the subsequent keypoint localization branch. After processing by a 3×3 convolution kernel and a 1×1 convolution kernel, a heatmap with the same dimensions as the input and one channel is obtained. An argmax operation is performed on the obtained single-channel heatmap to obtain the coordinates of the needle tip and the center point of the flexible electrode ring.

[0054]

[0055] Where H(u,v) represents the response value of the output keypoint location heatmap at position (u,v), (u key ,v key () represents the image coordinates of key points.

[0056] Specifically, during the focusing iteration process, the entire image is re-inputted for target detection and the point sharpness value is calculated at each step. After focusing is completed, the target region is cropped from the feature map of the last iteration and input into the key point detection branch to obtain the heat map.

[0057] For pose measurement, template matching is used to extract characteristic straight lines from the implanted needle and flexible electrode. This method, based on contour features to locate similar regions, consists of four steps: template creation, edge detection, template matching, and feature extraction. In the template creation stage, a standard image is first selected according to preset criteria. Then, edge detection is performed on the selected standard image, and unnecessary features are removed to obtain the target template and feature primitives. The target template and feature primitives are as follows: Figure 5 As shown. In the edge detection stage, the Canny algorithm is used to process the image and generate a search map. In the template matching stage, a template matching algorithm based on rotation and feature pyramid is proposed, which optimizes the matching result through four iterative steps: First, on the image downsampled to 1 / 8 size, coarse matching is performed with steps of 5°, 1°, and 0.1° within ±180°, ±5°, and ±1° of the current best result, respectively; then, on the image downsampled to 1 / 2 size, fine matching is performed with a step of 0.1° within ±0.2° of the current best result. In the feature extraction stage, the feature primitives are rotated according to the rotation angle obtained by matching, and edge points are searched for in the vicinity along the normal of the feature line. Then, a new feature line is obtained by fitting with RANSAC. For the implanted needle, the midpoint of the endpoints of the two feature lines is calculated to obtain the central axis of the implanted needle as the final implanted needle pose vector. For the flexible electrode, points are uniformly selected on the feature line of the left-view image. For each point, the epipolar line in the right-view image is obtained through epipolar constraint. The intersection of the epipolar line and the feature line of the right-view image is calculated as the matching point of that point. RANSAC plane fitting is performed on these matching points in three-dimensional space. The plane normal vector is the normal vector of the flexible electrode ring. The vector is reprojected into the image space to obtain the attitude vector of the flexible electrode ring.

[0058] Based on the pose measurement of the implantation needle and flexible electrode, the implantation needle can automatically pick up the flexible electrode, and the control process is as follows: Figure 6As shown, the pose alignment process includes position alignment and attitude alignment, both of which employ a visual servo control method based on the image Jacobian matrix. The image Jacobian matrix is ​​used to describe the mapping relationship between image features and the pose of the robot's end effector, thereby adjusting the robot's motion according to the error of the image features to ensure that its end effector reaches the expected pose. First, the implantation needle and flexible electrode are moved into the field of view of the microscope camera. Then, attitude alignment is performed. During attitude alignment, the attitude features of the flexible electrode and the implantation needle are extracted from the images of the two cameras. Then, the three-dimensional rotational motion of the implantation needle is calculated based on the angle error using the image Jacobian matrix. The attitude of the implantation needle is adjusted using a controller. The attitude alignment ends when the angle error is less than a given threshold. Next is the position alignment process. Similarly, the position features of the flexible electrode and the implantation needle are extracted from the two cameras. Then, the three-dimensional translational motion of the implantation needle is calculated based on the position error using the image Jacobian matrix. The position of the implantation needle is adjusted using a controller. The position alignment ends when the position error is less than a given threshold. Finally, in the insertion stage, the linear push rod is moved downward so that the implantation needle passes through the flexible electrode ring, completing the automatic pickup of the flexible electrode.

[0059] System Implementation Examples

[0060] According to embodiments of the present invention, an automatic pickup system for flexible electrodes is provided. Figure 7 This is a schematic diagram of an automatic pickup system for flexible electrodes according to an embodiment of the present invention, as shown below. Figure 7 As shown, the automatic pickup system for flexible electrodes according to an embodiment of the present invention specifically includes:

[0061] The focusing module 70 is used to acquire the image to be tested through a microscope camera, obtain the target area to be tested based on the image to be tested, and perform iterative focusing on the target area to be tested;

[0062] The pickup module 72 is used to perform pose measurement on the implanted needle and flexible electrode in the focused target area to obtain the pose features of the implanted needle and flexible electrode, and to complete the automatic pickup of the flexible electrode based on the pose features; wherein, the pose features include position features and attitude features.

[0063] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0064] In summary, this invention proposes a two-step target detection and pose measurement algorithm. First, a target detection algorithm suitable for out-of-focus scenes is studied. Then, automatic focusing is achieved for the target area to be measured. Finally, pose measurement of the implanted needle and flexible electrode is realized within the target area, effectively avoiding image compression and background interference, and improving measurement accuracy.

[0065] Device Example 1

[0066] This invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps described in the method embodiment.

[0067] Device Example 2

[0068] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, performs the steps described in the method embodiment.

[0069] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic pickup method for flexible electrodes, characterized in that... include: The process involves acquiring an image to be tested using a microscope camera, determining the target region based on the image, and iteratively focusing on the target region. Specifically, this includes: The image to be tested is input into the target detection network ODKL-Net for inference. The feature extractor ResPNet extracts features from the image to be tested. The extracted features are pooled using RoIAlign to obtain the target region to be tested. The hill climbing method is used to iteratively focus the target region to be tested. Specifically, iterative focusing of the target area using the hill-climbing method includes: S31: Move the focusing slide from the initial position according to the preset initial step size; S32: Calculate the evaluation function after the movement. If the absolute value of the difference between the evaluation function before and after the movement is greater than a preset threshold, and the evaluation function after the movement is greater than the evaluation function before the movement, then continue to move the focusing slide with the current step size; if the evaluation function after the movement is less than the evaluation function before the movement, then reduce the step size and move the focusing slide in the opposite direction. S33: Iterate through S32 until the absolute value of the difference between the evaluation function before and after the movement is less than or equal to the preset threshold, then stop moving the focusing slide and take the current position of the microscope camera as the optimal focusing position. The implanted needle and flexible electrode are subjected to pose measurement in the target area after focusing to obtain the pose features of the implanted needle and flexible electrode, and the flexible electrode is automatically picked up based on the pose features; wherein, the pose features include position features and attitude features.

2. The method according to claim 1, characterized in that, The evaluation function is a point sharpness function, and its expression is shown in Formula 1. Official 1; in, This represents the evaluation function value. , For the length and width of the image, The grayscale change amplitude, The distance increment between pixels, Indicates the calculation of the first Local gradient of each pixel, Indicates the first Each neighboring pixel.

3. The method according to claim 1, characterized in that, The pose measurement of the implanted needle and flexible electrode in the focused target area yields the pose features of the implanted needle and flexible electrode, specifically including: After focusing, the output feature map of the last layer decoding module of ResPNet is cropped to obtain the feature map of the target region to be tested. The feature map is input into the key point localization branch of ODKL-Net. The feature map is convolved by 3×3 convolution kernel and 1×1 convolution kernel to obtain a single-channel heat map. The single-channel heat map is subjected to argmax operation to obtain the position coordinates of the implanted needle tip and the center point of the flexible electrode ring as shown in Formula 2. Official 2; in, This indicates that the output key point location heatmap is located at the position. The response value on Represents the image coordinates of key points.

4. The method according to claim 3, characterized in that, The pose measurement of the implanted needle and flexible electrode in the focused target area yields the pose features of the implanted needle and flexible electrode, specifically including: A standard image is selected according to a preset standard. Edge detection is performed on the standard image and redundant features are removed to obtain a target template and feature primitives. The Canny algorithm is used to perform edge detection on the focused target area to generate a search map. Template matching is performed on the target template and the search map using a template matching algorithm based on rotation and feature pyramid. The feature primitives are rotated according to the rotation angle obtained by matching, and edge points are searched along the normal of the obtained feature line. The edge points are fitted with RANSAC to obtain a new feature line. Calculate the midpoint of the new feature line, obtain the central axis of the implantation needle based on the midpoint, and use the central axis as the attitude vector of the implantation needle; Points are uniformly selected on the new feature line in the left-view image. For each point obtained, an epipolar line in the right-view image is obtained through epipolar constraint. The intersection point of the epipolar line and the new feature line in the right-view image is calculated. The intersection point is used as the matching point of the corresponding point. RANSAC plane fitting is performed on the matching point. The obtained plane normal vector is used as the normal vector of the flexible electrode ring. The normal vector of the flexible electrode ring is reprojected into the image space to obtain the attitude vector of the flexible electrode ring.

5. The method according to claim 1, characterized in that, The automatic pickup of the flexible electrode based on the aforementioned pose features specifically includes: Based on the pose characteristics of the implanted needle and the flexible electrode, the position error and angle error of the implanted needle are obtained using the image Jacobian matrix. The three-dimensional translational motion and three-dimensional rotational motion of the implanted needle are calculated using the position error and angle error. Based on the three-dimensional translational motion and three-dimensional rotational motion, the pose of the implanted needle is adjusted using a controller. After adjustment, the implanted needle is inserted into the flexible electrode ring, and the flexible electrode is automatically picked up.

6. An automatic pickup system for flexible electrodes, characterized in that... include: The focusing module is used to acquire an image to be tested using a microscope camera, obtain the target region to be tested based on the image, and iteratively focus on the target region to be tested. Specifically, it is used for: The image to be tested is input into the target detection network ODKL-Net for inference. The feature extractor ResPNet extracts features from the image to be tested. The extracted features are pooled using RoIAlign to obtain the target region to be tested. The hill climbing method is used to iteratively focus the target region to be tested. Specifically, iterative focusing of the target area using the hill-climbing method includes: S31: Move the focusing slide from the initial position according to the preset initial step size; S32: Calculate the evaluation function after the movement. If the absolute value of the difference between the evaluation function before and after the movement is greater than a preset threshold, and the evaluation function after the movement is greater than the evaluation function before the movement, then continue to move the focusing slide with the current step size; if the evaluation function after the movement is less than the evaluation function before the movement, then reduce the step size and move the focusing slide in the opposite direction. S33: Iterate through S32 until the absolute value of the difference between the evaluation function before and after the movement is less than or equal to the preset threshold, then stop moving the focusing slide and take the current position of the microscope camera as the optimal focusing position. The pickup module is used to perform pose measurement on the implanted needle and flexible electrode in the focused target area to obtain the pose features of the implanted needle and flexible electrode, and to automatically pick up the flexible electrode based on the pose features; wherein, the pose features include position features and attitude features.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the automatic pickup method for the flexible electrode as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an information transmission implementation program, which, when executed by a processor, implements the steps of the automatic pickup method for the flexible electrode as described in any one of claims 1-5.

Images