Automatic picking method and system for flexible electrode

Abstract

The invention provides an automatic pickup method and system for a flexible electrode, and the method comprises the steps: obtaining a to-be-detected image through a microscopic camera, obtaining a to-be-detected target region according to the to-be-detected image, and carrying out the iteration focusing of the to-be-detected target region; and performing pose measurement on the implantation needle and the flexible electrode in the focused target area to be measured to obtain pose characteristics of the implantation needle and the flexible electrode, and completing automatic picking of the flexible electrode based on the pose characteristics. Wherein the pose features comprise position features and pose features. In order to improve the precision of position and attitude measurement, the invention provides a two-step strategy target detection and pose measurement algorithm, that is, firstly, a target detection algorithm suitable for a virtual focus scene is researched, automatic focusing of an out-of-focus target is realized, then positioning and attitude measurement of a target key point are realized in a target area, and the accuracy of position and attitude measurement is improved. Image compression and background interference are effectively avoided, and the accuracy and robustness are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interface, and in particular to an automatic picking method and system for a flexible electrode. Background Art

[0002] Flexible electrodes can reduce damage to brain tissue due to their good flexibility and biocompatibility, while providing more stable and lasting signal recording performance. However, due to the small size of flexible electrodes (on the order of tens of microns), manual implantation is difficult. In order to ensure the implantation accuracy and efficiency of flexible electrodes, a robot is required to control the implantation needle to achieve precise pickup of the flexible electrode. The prerequisite for achieving this is to perform high-precision posture measurement of the implantation needle and the flexible electrode, and to perform a reliable alignment and insertion control process.

[0003] For the positioning of the implant needle tip and the center point of the flexible electrode ring, traditional image processing algorithms can be used. For example, edge detection and straight line fitting methods can be used to obtain the coordinates of the implant needle tip through the intersection of feature straight lines; edge detection and ellipse fitting methods can be used to calculate the center of the ellipse as the coordinates of the center point of the flexible electrode ring; deep learning methods can also be used to collect pictures for annotation, train the network, and input pictures to directly obtain the coordinates of key points.

[0004] In the prior art, the posture measurement of implant needles and flexible electrode rings can obtain posture vectors through edge detection, straight line fitting, ellipse fitting, template matching, etc. However, these methods are affected by image compression and background and lighting, so the accuracy and robustness of posture measurement are poor. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for automatically picking up flexible electrodes, aiming to solve the above-mentioned problems in the prior art.

[0006] An embodiment of the present invention provides a method for automatically picking up a flexible electrode, comprising:

[0007] Acquire an image to be measured by a microscope camera, obtain a target area to be measured according to the image to be measured, and iteratively focus the target area to be measured; and

[0008] The implant needle and the flexible electrode are measured in the focused target area to obtain posture features of the implant needle and the flexible electrode, and the flexible electrode is automatically picked up based on the posture features; wherein the posture features include position features and posture features.

[0009] An embodiment of the present invention provides an automatic picking system for a flexible electrode, comprising:

[0010] A focusing module, used for acquiring an image to be measured through a microscope camera, obtaining a target area to be measured according to the image to be measured, and iteratively focusing the target area to be measured; and

[0011] The picking module is used to measure the posture of the implant needle and the flexible electrode in the target area to be measured after focusing, obtain the posture characteristics of the implant needle and the flexible electrode, and complete the automatic picking of the flexible electrode based on the posture characteristics; wherein the posture characteristics include position characteristics and posture characteristics.

[0012] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned method for automatically picking up a flexible electrode when executed by the processor.

[0013] An embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored, and when the program is executed by a processor, the steps of the above-mentioned method for automatically picking up a flexible electrode are implemented.

[0014] The use of the embodiment of the present invention may include the following beneficial effects: In order to improve the accuracy of position and attitude measurement, the embodiment of the present invention proposes a two-step strategy for target detection and position and attitude measurement algorithm. Specifically, the target detection algorithm suitable for out-of-focus scenes is first studied to achieve automatic focusing of out-of-focus targets, and then the positioning and attitude measurement of target key points are achieved in the target area, effectively avoiding image compression and background interference, and achieving a significant improvement in accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 is a flow chart of an automatic picking method of a flexible electrode according to an embodiment of the present invention;

[0017] Figure 2 is a flow chart of a posture measurement method according to an embodiment of the present invention;

[0018] Figure 3 Schematic diagram of the ODKL-Net network structure of an embodiment of the present invention;

[0019] Figure 4 is an automatic focusing flow chart of an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram of a target template and a feature primitive in an embodiment of the present invention;

[0021] Figure 6 is a control flow chart of the implant needle picking up the flexible electrode according to an embodiment of the present invention;

[0022] Figure 7 Schematic diagram of an automatic picking system for flexible electrodes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0024] Method Embodiment

[0025] According to an embodiment of the present invention, a method for automatically picking up a flexible electrode is provided. Figure 1 FIG. 1 is a flow chart of a method for automatically picking up a flexible electrode according to an embodiment of the present invention. Figure 1 As shown, the automatic picking method of the flexible electrode according to the embodiment of the present invention specifically includes:

[0026] Step S101, acquiring an image to be tested through a microscope camera, obtaining a target area to be tested according to the image to be tested, and iteratively focusing the target area to be tested, specifically includes:

[0027] The image to be tested is input into the target detection network ODKL-Net for inference, the feature extractor ResPNet is used to extract features of the image to be tested, the extracted features are pooled by RoIAlign to obtain the target area to be tested, and the hill climbing method is used to iteratively focus on the target area to be tested;

[0028] The iterative focusing of the target area to be measured by using the hill climbing method specifically includes:

[0029] S31: moving the focus slide from the initial position according to a preset initial step length;

[0030] S32: calculating the evaluation function after the movement, and when the absolute value of the difference between the evaluation function before and after the movement is greater than a preset threshold, if the evaluation function after the movement is greater than the evaluation function before the movement, continuing to move the focus slide at the current step length; if the evaluation function after the movement is less than the evaluation function before the movement, reducing the step length and moving the focus slide in the opposite direction;

[0031] S33: iteratively executing S32 until the absolute value of the difference between the evaluation function before and after the movement is less than or equal to a preset threshold, then stopping moving the focusing slide, and taking the current position of the microscope camera as the optimal focusing position;

[0032] The evaluation function is a point sharpness function, and the expression is shown in Formula 1;

[0033]

[0034] Where P represents the evaluation function value, m and n are the length and width of the image, df is the grayscale change amplitude, dx is the distance increment between pixels, i represents the calculation of the local gradient of the i-th pixel, and a represents the a-th neighborhood pixel;

[0035] Step S102, measuring the posture of the implant needle and the flexible electrode in the focused target area to be measured, obtaining posture features of the implant needle and the flexible electrode, and automatically picking up the flexible electrode based on the posture features; wherein the posture features include position features and posture features, specifically including:

[0036] After focusing, the output feature map of the last layer decoding module of ResPNet is cropped to obtain the feature map of the target area to be measured, and the feature map is input into the key point positioning branch of ODKL-Net. The feature map is convolved by 3×3 convolution kernel and 1×1 convolution kernel to obtain a single-channel thermal map, and the single-channel thermal map is subjected to argmax operation to obtain the position coordinates of the implant needle tip and the center point of the flexible electrode ring as shown in Formula 2;

[0037]

[0038] Among them, H(u,v) represents the response value of the output key point positioning heat map at the position (u,v), (u key ,v key ) represents the image coordinates of key points;

[0039] A standard image is selected according to a preset standard, edge detection is performed on the standard image and redundant features are removed to obtain a target template and a feature primitive, the Canny algorithm is used to perform edge detection on the focused target area to be measured, a search graph is generated, template matching is performed on the target template and the search graph using a template matching algorithm based on rotation and feature pyramid, the feature primitive is rotated according to the rotation angle obtained by matching, and edge points are searched along the normal direction of the obtained feature straight line, and a new feature straight line is obtained by fitting the edge points using RANSAC;

[0040] Calculating the midpoint of the new characteristic straight line, obtaining the central axis of the implant needle according to the midpoint, and using the central axis as the posture vector of the implant needle;

[0041] Uniformly select points on the new feature straight line in the left-view image, obtain the epipolar line in the right-view image through epipolar line constraints for each obtained point, calculate the intersection of the epipolar line and the new feature straight line in the right-view image, use the intersection as the matching point of the corresponding point, perform RANSAC plane fitting on the matching point, use the obtained plane normal vector as the normal vector of the flexible electrode ring, reproject the normal vector of the flexible electrode ring into the image space, and obtain the posture vector of the flexible electrode ring;

[0042] According to the posture characteristics of the implant needle and the flexible electrode, the position error and angle error of the implant needle are obtained using the image Jacobian matrix. The three-dimensional translational motion and three-dimensional rotational motion of the implant needle are calculated through the position error and angle error. Based on the three-dimensional translational motion and three-dimensional rotational motion, the posture of the implant needle is adjusted using a controller. After the adjustment, the implant needle is inserted into the flexible electrode ring, and the flexible electrode is automatically picked up.

[0043] The above technical solution of the embodiment of the present invention is described in detail below in combination with the specific situation of the automatic picking method of the flexible electrode of the embodiment of the present invention.

[0044] The embodiment of the present invention proposes a method for automatically picking up flexible electrodes for minimally invasive brain-computer interface implantation applications, including a method for measuring the posture of an implant needle and a flexible electrode and an automatic alignment and insertion control process.

[0045] The process of measuring the posture of the implant needle and the flexible electrode is as follows: Figure 2 As shown, the target detection network is first used to obtain the target area to be measured, and then the target area to be measured is automatically focused. Finally, the tip of the implant needle and the center point of the flexible electrode ring are detected respectively, and the posture vectors of the implant needle and the flexible electrode ring are calculated.

[0046] In order to realize target detection, the embodiment of the present invention proposes a two-stage target detection network ODKL-Net, the network structure is as follows: Figure 3 As shown in the figure. The network combines ResNet50 with the feature pyramid structure to construct the feature extractor ResPNet. In order to reduce the accuracy loss that may occur during the pooling process, RoIAlign is introduced for the pooling operation. During the training process, the 0-filling strategy is used instead of the image resizing method to maintain the image ratio and minimize compression, thereby improving positioning accuracy.

[0047] After obtaining the target area to be measured, the hill climbing method is used to realize the automatic focus of the camera. The hill climbing method is a commonly used local optimization algorithm for automatic focus. This method continuously adjusts the focal length to find the position where the evaluation function reaches the maximum value. The point sharpness function is used as the evaluation function. It is based on the local gradient information of the image and can effectively reflect the clarity of the image. Its formula is as follows:

[0048]

[0049] Where m and n are the length and width of the image, df is the grayscale change amplitude, dx is the distance increment between pixels, i represents the calculation of the local gradient of the i-th pixel, and a represents the a-th neighborhood pixel. The automatic focusing process is as follows Figure 4 As shown, specifically as follows:

[0050] a) Starting from the initial position, move the focus slide with a longer initial step length;

[0051] b) Calculate the evaluation function value after the move, and if the absolute value of the difference between the two evaluation function values ​​is greater than a certain threshold, move the focus slide. If the evaluation function value after the move is higher than the evaluation function value at the current position, continue to move the focus slide at the current step length; otherwise, reduce the step length and move the focus slide in the opposite direction;

[0052] c) Repeat step b until the difference between the two evaluation function values ​​is less than a certain threshold, then stop searching. At this time, the position of the camera is the best focus position.

[0053] After the focus of the target area to be measured is completed, the position and posture of the implant needle and the flexible electrode are measured respectively. In order to reduce the use of video memory, the feature map of the target area to be measured is directly cropped from the output feature map of the last layer decoding module of the target detection feature extractor ResPNet, and then input into the subsequent key point positioning branch. After being processed by a 3×3 convolution kernel and a 1×1 convolution kernel, a heat map with the same length and width as the input and a channel number of 1 is obtained. The argmax operation is performed on the obtained single-channel heat map to obtain the coordinates of the implant needle tip and the center point of the flexible electrode ring:

[0054]

[0055] Among them, H(u,v) represents the response value of the output key point positioning heat map at the position (u,v), (u key ,v key ) represents the image coordinates of key points.

[0056] Specifically, in the focusing iteration process, the entire image will be re-input for target detection and point sharpness value calculation at each step. After focusing is completed, the target area will be cropped out from the feature map of the last iteration and input into the key point detection branch to obtain a heat map.

[0057] For posture measurement, template matching is used to extract the characteristic straight lines of the implant needle and the flexible electrode. This method locates similar areas based on contour features and is divided into four steps: template creation, edge detection, template matching, and feature extraction. In the template creation stage, a standard image is first selected according to a preset standard, and then edge detection is performed on the selected standard image and unnecessary features are removed to obtain the target template and feature primitives. The target template and feature primitives are as follows: Figure 5 As shown. In the edge detection stage, the Canny algorithm is used to process the image and generate a search graph. In the template matching stage, a template matching algorithm based on rotation and feature pyramid is proposed, and the matching results are optimized through four steps of iteration: first, on the image downsampled to 1 / 8 size, coarse matching is performed in the range of ±180°, ±5°, ±1° of the current optimal result with a step size of 5°, 1°, and 0.1°; then, on the image downsampled to 1 / 2 size, fine matching is performed in the range of ±0.2° of the current optimal result with a step size of 0.1°. In the feature extraction stage, the feature primitives are rotated according to the rotation angle obtained by matching, and edge points are searched nearby along the normal of the feature line, and then RANSAC is used to fit the new feature line. For the implant needle, the midpoint of the two feature line endpoints is calculated to obtain the central axis of the implant needle as the final implant needle posture vector. For the flexible electrode, points are evenly selected on the characteristic straight line of the left-view image, and the epipolar line in the right-view image is obtained for each point through epipolar line constraint. The intersection of the epipolar line and the characteristic straight line of the right-view image is calculated as the matching point of the point, and RANSAC plane fitting is performed on these matching points in three-dimensional space. The plane normal vector is the normal vector of the flexible electrode ring. The vector is reprojected into the image space to obtain the posture vector of the flexible electrode ring.

[0058] Based on the position measurement of the implant needle and the flexible electrode, the implant needle can automatically pick up the flexible electrode. The control process is as follows: Figure 6As shown in the figure. The pose alignment process includes position alignment and attitude alignment, both of which use the visual servo control method based on the image Jacobian matrix. The image Jacobian matrix is ​​used to describe the mapping relationship between the image features and the pose of the robot end effector, so as to adjust the movement of the robot according to the error of the image features so that the end effector reaches the expected pose. First, all the implant needles and flexible electrodes are moved into the field of view of the microscope camera, and then posture alignment is performed. During the posture alignment process, the posture features of the flexible electrode and the implant needle are first extracted from the two-way camera images, and then the image Jacobian matrix is ​​used to calculate the three-dimensional spatial rotational motion of the implant needle according to the angle error, and the controller is used to adjust the posture of the implant needle. The posture alignment ends when the angle error is less than a given threshold; followed by a position alignment process. Similarly, the position features of the flexible electrode and the implant needle are extracted from the two cameras, and then the image Jacobian matrix is ​​used to calculate the three-dimensional spatial translational motion of the implant needle according to the position error, and the controller is used to adjust the position of the implant needle. The position alignment ends when the position error is less than a given threshold; finally, in the insertion stage, the linear push rod is moved downward so that the implant needle passes through the flexible electrode ring to complete the automatic picking of the flexible electrode.

[0059] System Example

[0060] According to an embodiment of the present invention, an automatic picking system for a flexible electrode is provided. Figure 7 Schematic diagram of an automatic picking system for a flexible electrode according to an embodiment of the present invention. Figure 7 As shown, the automatic picking system of the flexible electrode according to the embodiment of the present invention specifically includes:

[0061] A focusing module 70 is used to obtain an image to be measured through a microscope camera, obtain a target area to be measured according to the image to be measured, and iteratively focus the target area to be measured;

[0062] The picking module 72 is used to measure the posture of the implant needle and the flexible electrode in the target area to be measured after focusing, obtain the posture characteristics of the implant needle and the flexible electrode, and complete the automatic picking of the flexible electrode based on the posture characteristics; wherein the posture characteristics include position characteristics and posture characteristics.

[0063] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0064] In summary, the embodiment of the present invention proposes a two-step strategy for target detection and posture measurement algorithm. First, the target detection algorithm suitable for out-of-focus scenarios is studied, then automatic focusing is implemented for the target area to be measured, and finally the posture measurement of the implant needle and the flexible electrode is implemented in the target area, effectively avoiding image compression and background interference and improving measurement accuracy.

[0065] Device Example 1

[0066] An embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps described in the method embodiment when executed by the processor.

[0067] Device Example 2

[0068] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps described in the method embodiment are implemented.

[0069] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically picking up a flexible electrode, characterized in that include: Acquire an image to be measured by a microscope camera, obtain a target area to be measured according to the image to be measured, and iteratively focus the target area to be measured; as well as The implant needle and the flexible electrode are measured in the focused target area to obtain posture features of the implant needle and the flexible electrode, and the flexible electrode is automatically picked up based on the posture features; wherein the posture features include position features and posture features.

2. The method according to claim 1, characterized in that Obtaining a target area to be measured according to the image to be measured, and iteratively focusing the target area to be measured specifically includes: The image to be tested is input into the target detection network ODKL-Net for inference, the feature extractor ResPNet is used to extract features of the image to be tested, the extracted features are pooled by RoIAlign to obtain the target area to be tested, and the hill climbing method is used to iteratively focus on the target area to be tested.

3. The method according to claim 2, characterized in that The iterative focusing of the target area to be measured by using the hill climbing method specifically includes: S31: moving the focus slide from the initial position according to a preset initial step length; S32: calculating the evaluation function after the movement, and when the absolute value of the difference between the evaluation function before and after the movement is greater than a preset threshold, if the evaluation function after the movement is greater than the evaluation function before the movement, continuing to move the focus slide at the current step length; if the evaluation function after the movement is less than the evaluation function before the movement, reducing the step length and moving the focus slide in the opposite direction; S33: iteratively execute S32 until the absolute value of the difference between the evaluation function before and after the movement is less than or equal to a preset threshold, then stop moving the focusing slide, and take the current position of the microscope camera as the optimal focusing position.

4. The method according to claim 3, characterized in that The evaluation function is a point sharpness function, and the expression is shown in Formula 1; Among them, P represents the evaluation function value, m and n are the length and width of the image, df is the grayscale change amplitude, dx is the distance increment between pixels, i represents the calculation of the local gradient of the i-th pixel, and a represents the a-th neighborhood pixel.

5. The method according to claim 1, characterized in that The posture of the implant needle and the flexible electrode is measured in the focused target area to obtain posture features of the implant needle and the flexible electrode, which specifically include: After focusing, the output feature map of the last layer decoding module of ResPNet is cropped to obtain the feature map of the target area to be measured, and the feature map is input into the key point positioning branch of ODKL-Net. The feature map is convolved by 3×3 convolution kernel and 1×1 convolution kernel to obtain a single-channel thermal map, and the single-channel thermal map is subjected to argmax operation to obtain the position coordinates of the implant needle tip and the center point of the flexible electrode ring as shown in Formula 2; Among them, H(u,v) represents the response value of the output key point positioning heat map at the position (u,v), (u key ,v key ) represents the image coordinates of key points.

6. The method according to claim 5, characterized in that The posture of the implant needle and the flexible electrode is measured in the focused target area to obtain posture features of the implant needle and the flexible electrode, which specifically include: A standard image is selected according to a preset standard, edge detection is performed on the standard image and redundant features are removed to obtain a target template and a feature primitive, the Canny algorithm is used to perform edge detection on the focused target area to be measured, a search graph is generated, template matching is performed on the target template and the search graph using a template matching algorithm based on rotation and feature pyramid, the feature primitive is rotated according to the rotation angle obtained by matching, and edge points are searched along the normal direction of the obtained feature straight line, and a new feature straight line is obtained by fitting the edge points using RANSAC; Calculating the midpoint of the new characteristic straight line, obtaining the central axis of the implant needle according to the midpoint, and using the central axis as the posture vector of the implant needle; Points are uniformly selected on the new feature straight line in the left-view image, and the epipolar line in the right-view image is obtained by applying epipolar line constraints to each obtained point. The intersection of the epipolar line and the new feature straight line in the right-view image is calculated, and the intersection is used as the matching point of the corresponding point. RANSAC plane fitting is performed on the matching point, and the obtained plane normal vector is used as the normal vector of the flexible electrode ring. The flexible electrode ring normal vector is reprojected into the image space to obtain the posture vector of the flexible electrode ring.

7. The method according to claim 1, characterized in that The automatic picking of the flexible electrode based on the posture feature specifically includes: According to the posture characteristics of the implant needle and the flexible electrode, the position error and angle error of the implant needle are obtained using the image Jacobian matrix. The three-dimensional translational motion and three-dimensional rotational motion of the implant needle are calculated through the position error and angle error. Based on the three-dimensional translational motion and three-dimensional rotational motion, the posture of the implant needle is adjusted using a controller. After the adjustment, the implant needle is inserted into the flexible electrode ring, and the flexible electrode is automatically picked up.

8. An automatic picking system for flexible electrodes, characterized in that include: A focusing module, used for acquiring an image to be measured through a microscope camera, obtaining a target area to be measured according to the image to be measured, and iteratively focusing the target area to be measured; as well as The picking module is used to measure the posture of the implant needle and the flexible electrode in the target area to be measured after focusing, obtain the posture characteristics of the implant needle and the flexible electrode, and complete the automatic picking of the flexible electrode based on the posture characteristics; wherein the posture characteristics include position characteristics and posture characteristics.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for automatically picking up a flexible electrode as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the automatic picking method for a flexible electrode according to any one of claims 1 to 7 are implemented.

Images