Electrode implantation method and system based on machine vision

Abstract

The present disclosure relates to a machine vision-based electrode implantation method and system. The method includes: a first camera captures a first image of the brain surface, and a second camera captures a second image of the brain surface; the first image and the second image are processed by operation, wherein a brain surface vascular area mask is obtained based on a vascular segmentation algorithm to determine an implantable area in the brain surface image; at least one implantation position is selected in the implantable area, and the distance between the at least one implantation position and the electrode position is calculated according to the known electrode position, so as to determine the implantation order of the electrode; the imaging of the first camera and the second camera is matched to obtain a transformation matrix, and the first straight line where the implantation position is located in the imaging of the first camera is projected into the imaging of the second camera, and the intersection between the first straight line and the second straight line where the implantation position is located in the imaging of the second camera is determined as the predicted landing point of the implantation device; and the implantation device is controlled in real time according to the predicted landing point until the implantation point coincides with the predicted landing point.

Description

Technical Field

[0001] The present disclosure relates to the field of life science technology, and in particular to a method and system for electrode implantation based on machine vision. Background Art

[0002] In the field of neurosurgery robots, it involves the implantation of a flexible electrode into the brain surface. During the implantation process, the implantation device first passes through the electrode, and then implants the electrode into the brain surface. The surgical area on the brain surface is mostly a small window of millimeter level. The robot arm or external stepper motor is used to initially move to the top of the small window, analyze the implantable site that avoids the blood vessels, and control the implantation tool for implantation. In order to reduce or reduce bleeding when implanting electrodes, it is necessary to automatically identify the implantable area. Due to the complex distribution of blood vessels on the surface of animal brains, there are many capillaries; and the image quality is more demanding on imaging conditions, and changes in illumination will greatly affect the effect of blood vessel identification. Therefore, a stable optical system is required to ensure image quality, as well as a stable algorithm that can identify cerebral blood vessels and provide implantation areas.

[0003] Furthermore, in the implantation of electrodes, it is required to clearly define the correspondence between the implanted brain area and the electrode channel, and the implantation positions need to be numbered. In the process of electrode implantation, there are often multiple sites that need to be implanted. However, the electrode length is limited. It is necessary to consider that the electrode is not pulled and the arrangement relationship with other electrodes, and the order of electrode implantation positions is required. In addition, since the implantation tool is not necessarily completely vertical and there are ups and downs on the brain surface, the implantation requires precise and strict control of the implantation angle and position of the implantation tool and the surface, and accurate judgment of the spatial position of the implantation tool. Therefore, it is also necessary to design a corresponding stereoscopic microscopic imaging system to monitor the position of the implantation tool in real time and predict its landing point on the brain surface. Summary of the invention

[0004] The present application proposes a machine vision-based electrode implantation method and system.

[0005] According to a first aspect of an embodiment of the present disclosure, a machine vision-based electrode implantation method is provided, comprising: capturing a first image of a brain surface by a first camera, and capturing a second image of a brain surface by a second camera; performing computational processing on the first image and the second image, wherein a brain surface vascular area mask is obtained based on a vascular segmentation algorithm to determine an implantable area in the brain surface image; selecting at least one implantation position in the implantable area, and calculating a distance between the at least one implantation position and the electrode position based on a known electrode position, thereby determining an implantation order of the electrodes; matching the imaging of the first camera and the second camera to obtain a transformation matrix, projecting a first straight line where the implantation position is located in the imaging of the first camera into the imaging of the second camera, and determining an intersection point between the first straight line and a second straight line where the implantation position is located in the imaging of the second camera as a predicted landing point of the implantation device; and controlling the implantation device in real time according to the predicted landing point until the implantation point coincides with the predicted landing point.

[0006] According to a second aspect of an embodiment of the present disclosure, there is provided an electrode implantation system based on machine vision, comprising: a first camera configured to capture a first image of a brain surface; a second camera configured to capture a second image of a brain surface; a blood vessel segmentation operation unit configured to perform operation processing on the first image and the second image, wherein a brain surface blood vessel area mask is obtained based on a blood vessel segmentation algorithm to determine an implantable area in the brain surface image; an implantation sequence determination unit configured to select at least one implantation position in the implantable area, and calculate the distance between the at least one implantation position and the electrode position according to a known electrode position, so as to determine the implantation sequence of the electrodes; an implantation landing point prediction unit configured to match the imaging of the first camera and the second camera to obtain a transformation matrix, project a first straight line where the implantation position is located in the imaging of the first camera to the imaging of the second camera, and determine the intersection between the first straight line and the second straight line where the implantation position is located in the imaging of the second camera as the predicted landing point of the implantation device; and an implantation device control unit configured to control the implantation device in real time according to the predicted landing point until the implantation point coincides with the predicted landing point.

[0007] The advantages of the embodiments of the present disclosure are that they are applicable to a variety of different imaging modes, the algorithms used have relatively good universality, and can achieve better boundary segmentation to obtain stable imaging recognition results.

[0008] Another advantage of the embodiments of the present disclosure is that it can provide an automated blood vessel segmentation algorithm, which is easy to adjust algorithm parameters and can reduce the computational complexity of the segmentation algorithm.

[0009] It should be appreciated that the above advantages do not need to be all realized in one or some specific embodiments, but may be partially dispersed in different embodiments according to the present disclosure. The embodiments according to the present disclosure may have one or some of the above advantages, or may have other advantages alternatively or additionally.

[0010] Other features and advantages of the present invention will become more apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram showing an electrode implantation system based on machine vision according to an embodiment of the present disclosure.

[0012] Figure 2 is a configuration diagram showing a machine vision-based electrode implantation system according to an embodiment of the present disclosure.

[0013] Figure 3 is a flow chart illustrating a blood vessel segmentation algorithm according to an embodiment of the present disclosure.

[0014] Figure 4 FIG. 4 is a schematic diagram showing the steps of a blood vessel segmentation algorithm according to an embodiment of the present disclosure.

[0015] Figure 5 is a diagram showing the effect of a blood vessel segmentation algorithm according to an embodiment of the present disclosure.

[0016] Figure 6 is a schematic diagram showing implantable position selection and path planning of brain surface electrodes according to an embodiment of the present disclosure.

[0017] Figure 7 is a schematic diagram showing the prediction of the landing point of an implantation tool according to an embodiment of the present disclosure.

[0018] Figure 8 is a flow chart illustrating an implant tool control algorithm according to an embodiment of the present disclosure.

[0019] Fig. 9 is a schematic diagram showing the steps of an electrode implantation method based on machine vision according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0021] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0022] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0023] During the process of implanting electrodes on the brain surface, it is necessary to identify blood vessels and select implantation sites, plan implantation paths, and position implantation tools.

[0024] Generally, the recognition of blood vessels on the brain surface mainly includes regional segmentation of brain surface imaging, that is, distinguishing between vascular and non-vascular areas. The methods for implementing regional segmentation include at least threshold segmentation, edge detection, and segmentation methods based on mathematical morphology.

[0025] Specifically, threshold segmentation is the most common parallel direct detection area segmentation method, and it is also the simplest segmentation method. This method generally has certain assumptions about the image, assuming that the target and background of the image occupy different grayscale ranges, and the grayscale value difference between adjacent pixels inside the target and background is small, but the pixels on both sides of the target and background have a large difference in grayscale value. If an appropriate grayscale threshold T is selected, and then the grayscale value of each pixel in the image is compared with the threshold T, the pixels can be divided into two categories according to the comparison results: the grayscale value of the pixel is greater than the threshold, and they are assigned 1; the grayscale value of the pixel is less than the threshold, and they are assigned 0. In this way, a binary image is obtained, and the target is extracted from the background. The segmentation threshold is usually determined based on prior knowledge, and it can also be determined using grayscale histogram features and statistical judgment methods. The grayscale histogram of the image will show a double peak and a valley. The two peaks correspond to the central grayscale of the target and the central grayscale of the background respectively. The boundary point is located around the target, and its grayscale is between the target grayscale and the background grayscale. Therefore, the grayscale of the boundary corresponds to the valley point between the two peaks. In order to minimize the probability of pixel misclassification, the grayscale of the valley point is used as the segmentation threshold. Due to the heterogeneity of the histogram, the valley value of the histogram is difficult to determine, and a specific method needs to be designed for search. At present, there are many methods to determine the optimal threshold (bottom), such as the method of obtaining Gaussian model parameters and the method of fitting the histogram curve to find the extreme value.

[0026] The threshold-based segmentation method has the advantages of simple calculation and high operational efficiency. However, this type of method does not take into account spatial characteristics, is sensitive to noise and grayscale diversity, and it is difficult to obtain an accurate segmentation threshold for images where the grayscale difference between the target and background is not obvious. In practical applications, it is usually used in conjunction with other image segmentation methods to achieve satisfactory results. The prior art proposes a statistical model based on the physical model of blood flow when extracting blood vessels from time-flight magnetic resonance angiography images. In order to improve the segmentation ability of blood vessels, the speed and phase information of PCA are fused, and the adaptive local threshold method and the single global threshold method are used to segment the two different statistical models, so that aneurysms with very low signals near them can achieve better segmentation effects. The prior art also combines local thresholds and global thresholds to perform three-dimensional reconstruction of cerebral vascular images. The use of local thresholds can enhance the contrast of small blood vessels, and the use of global thresholds can extract target blood vessels from the background.

[0027] Edge detection is a parallel boundary segmentation technology based on grayscale discontinuity, and is the first step of all boundary segmentation methods. Because the edge is the dividing line between the target and the background, the target and the background can be distinguished only by extracting the edge. Edge detection is generally achieved by using the difference between the target and the background in certain characteristics, such as grayscale, color, texture and other features. Edge detection is generally completed by first-order or second-order derivatives, but in actual digital images, the derivative is approximated by differential operations instead of differential operations. The grayscale values ​​of the points on both sides of the edge in the image change suddenly, so these points will have larger differential values. When the direction of the differential is perpendicular to the boundary, the differential value is the largest. It can be seen that the differential is a directional operation, which is used to measure the change of grayscale in the differential direction.

[0028] The basic principle of the segmentation method based on mathematical morphology is to use structural elements with certain shapes to perform basic operations on the image in order to achieve the purpose of image analysis and recognition. The basic operations of mathematical morphology are dilation and erosion, as well as the opening and closing operations formed by their combination. The opening operation is to erode first and then dilate, and the closing operation is to dilate first and then erode. Each operation has its own characteristics in processing images. Dilation makes the image larger, while erosion makes the image smaller. Both opening and closing operations can make the image contour smooth, but the two operations have opposite effects. The opening operation can disconnect narrow discontinuities and eliminate thin protrusions; the closing operation can eliminate small holes in the image, fill the breaks in the contour line, and merge narrow gaps and elongated gaps. Combined with the specific features of the image, different mathematical morphology algorithms can be derived and combined according to these basic operations to process and analyze the image shape and structure, such as edge detection, image filtering, feature extraction, image enhancement, etc. Commonly used processing algorithms for medical images are top-hat transformation and watershed transformation.

[0029] In order to realize the above-mentioned various methods, neural networks are also applied to image region segmentation in practice. On the one hand, neural networks can learn; on the other hand, the nonlinearity of the network can be used for boundary segmentation during the training process. The disadvantage is that every time a new feature is added to the network system, it needs to be re-learned and trained, and the debugging process is also complicated. In order for the network system to use its learnability to classify boundaries in features, as many features of objects as possible should be selected. An algorithm widely used in the learning process is the back propagation algorithm. Since the training data set determines the learning, the amount of training data determines the learning process.

[0030] Furthermore, the positioning of implant tools mainly includes hand-eye calibration in robot vision applications. Its goal is to obtain the relationship between the robot coordinate system and the camera coordinate system, and finally transfer the visual recognition results to the robot coordinate system.

[0031] There are two forms of hand-eye calibration in the industry, depending on where the camera is fixed. If the camera and the end of the robot are fixed together, it is called "eye in hand"; if the camera is fixed to a base outside the robot, it is called "eye to hand".

[0032] In industry, common hand-eye calibration methods are mainly divided into two methods: nine-point calibration method and calibration plate calibration method. Nine-point calibration directly establishes the coordinate transformation relationship between the camera and the manipulator. Let the pointer at the end of the manipulator touch these nine points to obtain the coordinates in the robot coordinate system. At the same time, the camera is used to identify the nine points in the initial image to obtain the pixel coordinates. In this way, nine sets of corresponding coordinates are obtained. Then, the transformation matrix can be solved to obtain the conversion affine matrix of the image and the manipulator coordinates. The calibration plate calibration method uses a chessboard calibration plate or a circular grid calibration plate to obtain the internal and external parameters of the camera, and then the coordinate transformation relationship between the image and the manipulator can be obtained.

[0033] The calibration of a monocular system is only applicable to situations where an observed object is on a horizontal plane, and depth information cannot be obtained. The traditional hand-eye calibration method cannot accurately predict the landing point of the implanted tool on the brain surface. In contrast, binocular systems are divided into two categories based on the positional relationship of the optical axes, namely, parallel binocular systems with basically parallel optical axes and convergent binocular systems with intersecting optical axes. Parallel binocular systems, due to the parallel optical axes, have a smaller overlap of fields of view, and are rarely used in situations where the field of view is small. They are generally used in situations where the working distance is much larger than the distance between the lenses. If you need to observe millimeter-level objects, a convergent binocular system is more in line with your needs.

[0034] However, the existing technology still has many shortcomings. On the one hand, for the image region segmentation algorithm, some algorithms proposed for specific imaging modes are not universal, that is, they cannot be applied to other imaging modes. The boundary judgment of the blood vessel is based on the grayscale gradient field of the pixel, but in areas with low blood flow velocity and complex blood flow, the gradient value is often not high enough, which will lead to a decrease in the accuracy of border inspection judgment. The algorithm assumes that the grayscale distribution of each tissue is a Gaussian distribution, but the actual situation is not entirely the case, resulting in a deviation between the proposed model and clinical data. At the same time, multiple parameters involved in the image segmentation algorithm need to be adjusted, and the parameter estimation process is very difficult. For some interactive algorithms, it is necessary to manually select the seed point or end point in the blood vessel, which affects the degree of automation. In addition, the overall segmentation method has a large amount of calculation and is computationally expensive.

[0035] On the other hand, for the positioning of the implant tool, in order to accurately predict the landing point of the implant tool on the brain surface during hand-eye calibration, the required imaging system must use at least two cameras, that is, a binocular system. In addition, due to the imaging characteristics of different binocular systems, a converging binocular system is required. In order to obtain clear imaging, it is also necessary to use a lens with a magnification of ×1 or ×2. Under the same camera pixels, the field of view is small and the depth of field of the lens is very limited, generally around 1mm. This requires adjusting the camera position to obtain clear imaging, resulting in relative non-fixation between the two cameras. However, the calibration of the binocular camera is only valid when the camera position is relatively fixed, which also means that the traditional binocular calibration method cannot be directly applied to the brain surface electrode implant system disclosed in this application.

[0036] In order to solve the above technical problems, the inventors of the present application have proposed an improved electrode implantation method and system based on machine vision, and in particular, a method and system for brain surface electrode implantation based on vascular segmentation processing of machine imaging. In summary, the technical solution disclosed in the present invention mainly includes automatically detecting and avoiding blood vessels during the electrode implantation process, and automatically detecting multiple candidates according to the size, number and shape of the implantation tool and the implanted electrode. Using image registration and fusion technology, brain partitions are projected on the brain surface to facilitate the positioning of the brain area where the electrode is implanted during surgery. After the point is selected, the implantation system automatically moves to the selected position, controls the precise movement of the electrode implantation device in three-dimensional space based on the selected position and the angle of electrode implantation, and monitors the precise distance between the implantation device and the brain surface in real time. The depth of electrode implantation is estimated based on the distance from the brain surface. In this process, target tracking and other technologies are used to achieve precise movement and implantation of the electrode to the selected implantation site.

[0037] The following will describe the embodiments of the present disclosure in detail with reference to the accompanying drawings. Figure 1 and Figure 2 Schematic diagram and configuration diagram of an electrode implantation system based on machine vision according to an embodiment of the present disclosure are respectively shown. Figure 1As shown in , the hardware structure of the brain surface electrode implantation system of the present application includes an optical system and a motion control system. Among them, the optical system is mainly associated with two cameras 101 and 102 (hereinafter also referred to as "first camera" and "second camera"), such as a high-definition array CMOS industrial camera, which has telecentric lenses 103 and 104 for imaging and magnifying the surgical area 106. Cameras 101 and 102 have the same imaging plane, form a certain angle with each other on a projection plane, are fixed on a rigid base plate, and are installed with a coaxial light source (not shown) to shorten the exposure time and increase the frame rate. The light source can be an external point light source, which is mainly used to make the surgical area 106 receive light evenly, avoid blurred or over-exposed imaging of the implant device 105 and the surgical area 106, thereby facilitating subsequent image and data processing. The light source can be white light, or other light of a given wavelength. Preferably, due to the color influence of the brain surface and the blood vessels themselves, green light (such as light with a wavelength of 495nm to 570nm) can be used to achieve better imaging effects.

[0038] In addition, three-axis slides are respectively arranged behind the cameras 101 and 102, and the magnifying lenses 103 and 104 can be adjusted to reach a working distance to image the position and angle of the implant device 105 relative to the surgical area 106 in multiple orientations. Figure 1 A non-limiting embodiment of the system disclosed in the present application is shown, wherein camera 101 and camera 102 are at an angle of approximately 90° to each other in horizontal projection.

[0039] The motion control system is mainly composed of three stepper motors, which are used to control the implant device 105 in three directions of a certain spatial coordinate system (such as Figure 1 The cameras 101 and 102 are respectively coupled to the motion control system at Figure 1 In the non-limiting embodiment shown, the three motors include a stepper motor for controlling movement in the ±z directions and two micro stepper motors (not shown), such as with a stroke of 5 mm, for respectively controlling movement of the implant device 105 in the ±x and ±y directions, i.e., movement in an area substantially parallel to the surgical area 106.

[0040] Alternatively, the motion control system may also include a robotic arm with similar motion control functions. In this case, the camera 101 / 102 is arranged above the robotic arm, but since the motion accuracy of the robotic arm (such as ±30 μm) cannot meet the accuracy requirements (such as ±10 μm) required by the system of the present application, the robotic arm is used to roughly find the implantation position, and the fine adjustment of the electrode position is still completed by two micro-operating motors.

[0041] Additionally, the implantation device 105 is configured to implant the flexible electrode into a designated position of the surgical area 106, and includes an implantation needle, an implantation feeding mechanism, and an implantation actuator. Among them, the implantation needle mechanism is used to engage the free end of the electrode with the needle head portion so as to drive the electrode to move. The implantation feeding mechanism is used to move the implantation needle along the longitudinal direction of the implantation device. The implantation actuator mechanism is used to drive the implantation needle to insert the needle head portion of the implantation needle into the surgical area 106. In addition, the implantation device 105 can also be equipped with an implantation movement mechanism, which is used to enable the implantation device 105 to implant electrodes from different angles and in different directions.

[0042] Figure 2 A non-limiting embodiment of a brain surface electrode implantation system is shown. A binocular system is used in the brain surface electrode implantation system 20, which mainly includes a first camera 201, a second camera 202, a blood vessel segmentation operation unit 203, an implantation sequence determination unit 204, an implantation point prediction unit 205, and an implantation device control unit 206. The first camera 201 and the second camera 202 correspond to Figure 1 The cameras 101 and 102 are used to image the position and direction of the implant device relative to the surgical area at an angle to each other. Similar features are not repeated here.

[0043] Specifically, the brain surface electrode implantation system 20 uses a first camera 201 and a second camera to image the brain surface, and captures a first image 2010 and a second image 2020 respectively. The first image 2010 and the second image 2020 are images of the implantation device and the surgical area in different directions, such as Figure 1 As shown, in a non-limiting example, if a three-dimensional coordinate system is established according to the control direction of the stepper motor in the motion control system, then according to the first camera 201 and the second camera 202 at an angle to each other, the position coordinates of the implant device and the surgical area can be determined within the coordinate system based on the first image 2010 and the second image 2020.

[0044] The blood vessel segmentation operation unit 203 is configured to perform operation processing on the first image 2010 and the second image 2020. The main function performed by the blood vessel segmentation operation unit 203 is to obtain a brain surface blood vessel area mask based on a blood vessel segmentation algorithm to determine an implantable area 2030 in the brain surface image. Generally, the blood vessel segmentation algorithm can be implemented in many ways, including threshold division, edge extraction, and mathematical morphology processing, as described above. The algorithm used in this application combines the advantages of several processing methods to remove the jitter of the video itself, process multiple frames of images, and obtain a smooth blood vessel image mask. Figure 3 A non-limiting embodiment of the vessel segmentation algorithm is shown. Figure 4 A schematic diagram showing the results obtained at each step in the blood vessel segmentation algorithm.

[0045] Specifically, in Figure 3 In step S301, the first image 2010 and / or the second image 2020 are input into the blood vessel segmentation algorithm. Next, in step S302, a series of image processing steps are performed on the input image. First, the input image is converted into a grayscale image, and then an adaptive threshold segmentation process is performed. In the processed result, the contour of the blood vessel is found and the small contour noise is removed. Then, an opening operation is performed to eliminate the bubble-like noise pattern belonging to the blood vessel in the original image, and then an inversion operation is performed. In the processed result, the blood vessel part is expanded to obtain a safe distance (also called "erosion" a safe displacement), and finally the inversion operation is performed again. At this time, the image mask of the implantable area is obtained in step S303. Further, a judgment is set in step S304 so that the series of processing in S302 is repeated before the number of images obtained reaches a preset smoothing number n. After the above judgment, the implantable area in the n images obtained recently is intersected in step S305, and finally a relatively stable blood vessel image mask is output in S306.

[0046] Accordingly, Figure 4 It mainly shows the intermediate results obtained after each step of the series of processing in step S302. As shown in the figure, the interference of blood vessel color is eliminated in the result after conversion to grayscale in S402, and the blood vessel and non-blood vessel areas are roughly divided in the result after adaptive threshold segmentation in S403. Since the adaptive threshold algorithm is adopted, it is not necessary to calculate the blood vessel grayscale threshold in advance, and no prior data is required. Under suitable imaging conditions, stable results can be obtained. Next, after removing the contour noise in S404 and the bubble-shaped noise pattern in the blood vessel in S405, the extracted contour edge is minimized in the time domain, thereby improving the stability and safety of the contour extraction algorithm. Finally, through the processing from S406 to S408, the image area identified as a blood vessel has a reasonable safety distance to minimize the risk of identifying the blood vessel as an implantable area.

[0047] Figure 5 The effect diagram of the blood vessel segmentation algorithm according to the above embodiment is shown. As shown in the figure, the following image blood vessel analysis effect is obtained in the 3mm x 3mm macaque brain surgical area. After the camera original imaging result is input, the image recognition result is obtained through a series of algorithm processing, where the stripe shape mask is the segmented blood vessel area, and the blank part is the implantable area. It can be seen from the figure that the blood vessel segmentation algorithm disclosed in the present application can effectively and stably identify the blood vessel area in the brain surface imaging, and ensure that the implantable area only contains the non-vascular part with high accuracy.

[0048] Additionally, the blood vessel segmentation algorithm disclosed in the present application can flexibly adjust parameters. In general, the algorithm parameters of the brain surface electrode implantation system can be adjusted based on the site of the implanted electrode, such as when the requirements for the number of insertable points and accuracy change, it may affect the accuracy threshold of edge detection in the blood vessel segmentation algorithm or the safe distance of the expansion processing. For the optical system, the change in the object distance between its lens and the surgical area will cause the specific area in the imaging to be enlarged or reduced, thereby affecting the number of sites to be detected, the point spacing and the imaging resolution. Alternatively, the number of electrode sites and the point spacing required can be determined by the user, automatically selected by the system, or assisted by the system, and the algorithm parameters can be adjusted based on this.

[0049] return Figure 2 Continue to describe. The implantation sequence determination unit 204 is configured to select at least one implantation position in the implantable area 2030. In particular, for the situation where multiple electrode implantation positions need to be selected, the implantation sequence determination unit 204 calculates the distance between the implantation position and the electrode position based on the known electrode position, thereby performing path planning for the order of electrode implantation, and obtaining the electrode implantation sequence 2040. In the designated area determined based on the surgical area, with the electrode direction as a reference, the motion control system controls the movement direction of the implantation device via a stepper motor or a robotic arm, so that the electrodes are implanted in sequence to the determined positions. In order to prevent the electrodes from having undesirable mutual influences, that is, the electrodes being implanted will not exert force on the implanted electrodes, the implantation sequence determination unit 204 needs to perform path planning according to the following principles: the electrodes implanted later cannot interfere with the electrodes implanted previously; the electrodes implanted cannot be pulled during movement. That is, the ideal electrode implantation path should avoid crossing, lateral jumping, and the like as much as possible.

[0050] In a non-limiting embodiment, the implantation sequence determination unit 204 may adopt an order from near to far and from left to right relative to the brain surface of the implanted electrodes, such as Figure 6 As shown. Figure 5 Take the image processing results based on the blood vessel segmentation algorithm obtained in as an example. Figure 6 Determination of the implantation position and implantation sequence based on the image processing result is shown. Figure 6 (A) is a reference path planning sequence, in which the implantable positions and their distribution on the brain surface obtained according to the above series of processing are simplified to a 5×7 dot matrix in a two-dimensional coordinate system, and the implanted electrodes (not shown) are arranged from top to bottom relative to the dot matrix on the brain surface (i.e., the positive direction of the y axis in the figure), and the sequence shown by the arrows between the dot matrices in 6 (A) is obtained, i.e., along the positive direction of the x axis and along the positive direction of the y axis. It should be noted that Figure 6 The number of dots in (A) is for illustration only and is not limiting. Apply the example sequence of 6 (A) to Figure 5 In the image processing results, we get Figure 6 (B) Path planning shown. Figure 6 (B) shows 19 calculated positions, where the positions indicated by square marks indicate implantation sites that are not recommended, the positions indicated by circular marks indicate implantation sites recommended by the algorithm, and the positions indicated by triangular marks indicate sites where electrodes have been implanted. The size of the numbers indicates the implantation order. It can be seen from the figure that the implantation order formed by these implantation positions does not cross, and the electrode sites are indicated by different marks to assist in the observation and implantation process, so that the electrodes implanted later will not interfere with or pull the electrodes implanted earlier.

[0051] Next, we will continue to describe Figure 2 The implantation point prediction unit 205 in the implantation point prediction unit 205 is configured to match the imaging of the first camera 201 and the second camera 202 to obtain a transformation matrix. Since both cameras can clearly obtain the vascular imaging of the surgical area, and there are many features in the blood vessels, the two cameras can be calibrated based on feature matching of the data. In practical applications, features that can be used for matching include SURF features or SIFT features.

[0052] When SURF features are used, two cameras are matched based on SURF features to obtain the affine transformation matrix between the two cameras. Among them, Gaussian filters of continuous different scales are used to process the image, and the scale-invariant feature points in the image are detected through Gaussian difference. In addition, the Hessian matrix of spot detection is used to detect feature points. Its determinant value represents the amount of change around the pixel point, so the feature point needs to take the maximum and minimum values ​​of the determinant. In addition, in order to achieve scale invariance, SURF also uses the determinant value of scale σ for feature point detection. Given a point p = (x, y) in the figure, the Hessian matrix at scale σ is H(p,σ):

[0053]

[0054] The L in the matrix xx The functions (p,σ) and so on are grayscale images after second-order differentiation. The 9×9 square filter is used as the lowest scale of SURF, which is similar to a Gaussian filter with σ=1.2.

[0055] When SIFT features are used, the computational efficiency is high and image matching can be performed quickly. Among them, Gaussian filters of different scales are used to process the image, and the scale-invariant feature points in the image are detected through Gaussian difference. SURF uses a square filter to replace the Gaussian filter in SIFT to achieve the approximation of Gaussian blur. Its filter can be expressed as:

[0056]

[0057] In addition, the use of a square filter can greatly increase the calculation speed by utilizing an integral graph, and only the values ​​at the four corners of the filter square need to be calculated.

[0058] After feature matching, the first camera 201 and the second camera 202 are calibrated, that is, the correspondence and conversion relationship between any one or more sets of first images 2010 and second images 2020 is obtained. Based on this, Figure 7 The schematic diagram of the principle of implant tool landing point prediction is further shown.

[0059] In a non-limiting embodiment, when the implant device appears in the imaging of the binocular system, each of the two cameras can locate a straight line where the implant tool is located in its imaging, that is, the first camera 201 and the second camera 202 can respectively determine a plane where the implant tool is located in the three-dimensional space (and / or the established spatial coordinate system). Due to the different viewing angles of the two cameras, the images and position information obtained in their imaging are not exactly the same. Taking the first camera 201 capturing the first image 2010 and the second camera 202 capturing the second image 2020 as an example, Figure 7 (A) and Figure 7 (B) is a schematic diagram showing the imaging direction of projecting one of the first camera 201 and the second camera 202 onto the other. Figure 7 (C) shows a schematic diagram of the implantation position in the second image 2020. The implantation point prediction unit 205 projects the first straight line 7001 where the implantation position in the first image 2010 is located into the second image 2020, and the implantation position itself falls on the second straight line 7002 in the second image 2020, and the intersection between the first straight line 7001 and the second straight line 7002 is determined as the predicted landing point 2050 of the implantation device in the second image 2020. Similarly, Figure 7 (D) shows the predicted landing point of the implantation position in the first image 2010. Due to the calculation error of the variable relationship between the two cameras, the two obtained intersection points may not completely overlap in the actual image, but it can be determined that the actual predicted landing point is near the two intersection points.

[0060] Continue to return Figure 2 The implant device control unit 206 is described. Based on the implant sequence 2040 determined by the implant sequence determination unit 204 and the predicted landing point 2050 determined by the implant landing point prediction unit 205, the implant device control unit 206 controls the implant device in real time according to the implant sequence 2040 and the predicted landing point 2050 in sequence until the implant point coincides with the predicted landing point. The implant device control unit 206 can adopt full supervision or semi-supervised control, and can make different choices according to user selections to achieve personalized control effects.

[0061] Figure 8 A flowchart of an implant tool control algorithm in a non-limiting embodiment is shown. The control algorithm uses the camera intrinsic parameters in the optical system as input at step S801. First, the series of processes in step S802 are performed. The calibration plate is placed in the surgical area, and it is ensured that the surgical area is within the moving range of the tungsten wire fine-tuning electrode. At this time, the imaging conditions of the camera are adjusted to make the imaging of the brain surface clear. After obtaining the camera extrinsic parameters through the position of the calibration plate, the calibration plate is removed to keep the position of the surgical area unchanged. Next, in step S803, the falling position of the tungsten wire is monitored in real time, and it stops when it contacts the brain surface, and the position of the tungsten wire landing point in the current image is recorded. In step S804, blood vessel identification is performed, and the implantation point is manually selected based on the restrictions of the implantation sequence 2040, or it can be selected by the system automatically or by auxiliary calculation. Next, coordinate conversion is performed in step S805, and the two image coordinates are converted into world coordinates. Specifically, the conversion is performed according to "pixel coordinate system→image coordinate system→camera coordinate system→world coordinate system". In step S806, the movement vector of the electrode by the implant device is calculated, and the vector is decomposed into two directions of electrode movement, such as finely controlling the movement of two micro-stepping motors in the x and y directions in addition to the z-axis direction of the stepping motor. Then, in step S807, the motors in the above-mentioned different directions are controlled to move the tungsten wire to be implanted. Step S808 repeats the process of step S803.

[0062] Next, a judgment is performed at step S809. When there is no blood vessel at the tungsten wire landing position (step S809 is judged as "No"), another judgment is continued at step S810, that is, whether the difference between the tungsten wire landing point and the implantation point is within an acceptable error range. If the judgment at step S809 is "Yes", that is, there is a blood vessel at the tungsten wire landing position in the current image, then return to S804 to select a new implantation point that meets the requirements. Then at step S810, if the judgment result is "Yes", it indicates that the implantation tool control algorithm navigation is successful and the electrode implantation can be started. If the judgment result of step S810 is "No", it means that the error between the tungsten wire landing position in the current image and the selected implantation point is unacceptable, then return to S805 to re-perform coordinate transformation to obtain a more accurate landing position.

[0063] Fig. 91 is a schematic diagram showing the steps of the electrode implantation method based on machine vision according to an embodiment of the present disclosure. Based on the above, the electrode implantation method 9000 mainly includes the following steps: at step S901, the first camera captures the first image for the brain surface, and the second camera captures the second image for the brain surface. At step S902, the first image and the second image are processed by operation, wherein the brain surface vascular area mask is obtained based on the vascular segmentation algorithm to determine the implantable area in the brain surface image. Next, at step S903, at least one implantation position is selected in the implantable area, and the distance between the at least one implantation position and the electrode position is calculated according to the known electrode position, so as to determine the implantation order of the electrode. Subsequently, in step S904, the imaging of the first camera and the second camera is matched to obtain a transformation matrix, and the first straight line where the implantation position is located in the imaging of the first camera is projected into the imaging of the second camera, and the intersection between the first straight line and the second straight line where the implantation position is located in the imaging of the second camera is determined as the predicted landing point of the implantation device. Finally, at step S905, the implantation device is controlled in real time according to the predicted landing point until the implantation point coincides with the predicted landing point.

[0064] Alternatively, the machine vision-based electrode implantation method and system disclosed in the present application may also have other implementations.

[0065] In one example, the brain surface electrode implantation system disclosed in the present application does not necessarily need to use a binocular system with two cameras for calibration, and a monocular system can also perform position control on the plane of the implantation tool. There are two possible implementation methods, namely vertical observation and oblique observation. In the case of vertical observation, the camera shoots vertically above the brain surface to obtain a still image, in which blood vessels are segmented, and the position of the implantation device is controlled by transforming the pixel coordinates and the mechanical arm coordinates, and it is moved above the brain surface for implantation. In this implementation, an obliquely placed observation camera can be added to facilitate observation of the implantation. In the case of oblique observation, the camera is placed obliquely to the brain surface to obtain a picture with an inclination angle, and the position of the implantation device is obtained by coordinate transformation, and the electrode is controlled to move its position for implantation. In contrast, due to the problem of imaging angle, the control accuracy and blood vessel segmentation accuracy of the vertical observation method are better than those of the oblique observation method.

[0066] If a monocular system is used, the calibration plate method and the nine-point calibration method are mainly used in the calibration to achieve the target requirements of this system. The calibration plate method refers to placing a calibration plate close to the surgical area, and after obtaining the camera external parameters, the conversion relationship between the image and the actual coordinates is calculated. The nine-point calibration method refers to photographing the brain area at a certain position, and then controlling the tungsten wire to move to the specified nine points, and recording the position of the tungsten wire in the image and the position in the actual coordinate system at this time, so as to obtain the coordinate conversion relationship. Based on the conversion relationship, the displacement of the tungsten wire that should be moved in the actual position can be calculated by the distance between the tungsten wire landing point and the target point in the image. Compared with the binocular system, both methods of the monocular system require that the height position of the brain surface from the camera remains unchanged, which will affect its accuracy. In practical applications, since the brain surface itself is not a plane, the camera needs to be adjusted in real time, that is, to reach the same position above the brain surface with micron-level accuracy.

[0067] The words "front", "rear", "top", "bottom", "above", "below", etc., if present, in the specification and claims are used for descriptive purposes and are not necessarily used to describe invariant relative positions. It should be understood that the words so used are interchangeable under appropriate circumstances, such that the embodiments of the disclosure described herein, for example, are capable of operation in other orientations than those illustrated or otherwise described herein.

[0068] As used herein, the word "exemplary" means "serving as an example, instance, or illustration," rather than as a "model" to be exactly copied. Any implementation described as an example herein is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, the present disclosure is not limited by any expressed or implied theory given in the above technical field, background technology, summary of the invention, or detailed description.

[0069] As used herein, the term "substantially" is intended to include any minor variations due to design or manufacturing imperfections, device or component tolerances, environmental influences, and / or other factors. The term "substantially" also allows for deviations from a perfect or ideal situation due to parasitic effects, noise, and other practical considerations that may exist in actual implementations.

[0070] The terms "first", "second" and the like may be used herein for reference purposes only and are not intended to be limiting. For example, the terms "first", "second" and other such numerical terms referring to structures or elements do not imply a sequence or order unless the context clearly indicates otherwise.

[0071] It should also be understood that when the term “include / comprises” is used in this document, it indicates the presence of the specified features, integers, steps, operations, units and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, units and / or components and / or their combinations.

[0072] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0073] Those skilled in the art will appreciate that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed in additional operations, and operations may be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in other various embodiments. However, other modifications, variations, and replacements are equally possible. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.

[0074] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. The various embodiments disclosed herein may be combined in any manner without departing from the spirit and scope of the present disclosure. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for determining an electrode implantation position based on machine vision, comprising: capturing a first image of a brain surface by a first camera, and capturing a second image of the brain surface by a second camera; Performing computational processing on the first image and the second image, wherein a brain surface vascular region mask is obtained based on a vascular segmentation algorithm to determine an implantable region in the brain surface image; Selecting at least one implantation position in the implantable area, and calculating the distance between the at least one implantation position and the electrode position according to the known electrode position, thereby determining the implantation order of the electrodes; as well as Match the imaging of the first camera and the second camera to obtain a transformation matrix, project a first straight line where the implant position is located in the imaging of the first camera to the imaging of the second camera, and determine the intersection between the first straight line and the second straight line where the implant position is located in the imaging of the second camera as the predicted landing point of the implant device.

2. The method for determining the electrode implantation position according to claim 1, wherein: Fully supervised or semi-supervised control of the implanted device based on the predicted landing point.

3. The method for determining the electrode implantation position according to claim 1, wherein: The imaging plane of the first camera is the same as that of the second camera.

4. The method for determining the electrode implantation position according to claim 3, wherein: The first camera and the second camera form an angle of about 90° in horizontal projection.

5. The method for determining the electrode implantation position according to claim 1, wherein: Each of the first camera and the second camera includes an optical system, and the first camera and the second camera are coupled to a motion control system, respectively.

6. The method for determining the electrode implantation position according to claim 5, wherein: The optical system includes an external light source that allows the brain surface to be illuminated evenly.

7. The method for determining the electrode implantation position according to claim 6, wherein: The wavelength range of the external light source is 495 nm to 570 nm.

8. The method for determining the electrode implantation position according to claim 5, further comprising: The first camera and the second camera perform image processing respectively, and merge the image processed data from the first camera and the second camera; The coordinates of the implant device in the images of the first camera and the second camera are determined based on the merged data.

9. The method for determining the electrode implantation position according to claim 8, wherein: The motion control system controls the movement of the implant device according to the coordinates.

10. The method for determining the electrode implantation position according to claim 5, wherein: The motion control system includes three stepper motors for controlling the implant device to move in an area substantially parallel to the electrode implantation area.

11. The electrode implantation position determination method according to claim 1, wherein the blood vessel segmentation algorithm comprises the following steps: Convert brain surface images into grayscale images; Segment the grayscale image according to an adaptive threshold; Remove small contour noise from the segmented grayscale image; Perform an opening operation to remove the bubble-like noise pattern in the blood vessels; Perform inverse operation; Performing dilation processing to obtain a safe distance at the border of the vascular region; and Perform the inverse operation again.

12. The method for determining the electrode implantation position according to claim 11, wherein: The parameters in the blood vessel segmentation algorithm are adjusted based on the number of sites to be detected, the point spacing, and the imaging resolution.

13. The method for determining the electrode implantation position according to claim 1, wherein: The order of implantation of the electrodes is determined so that the electrodes being implanted do not exert forces on the electrodes already implanted.

14. The method for determining the electrode implantation position according to claim 13, wherein: The paths of the implanted electrodes are planned based on the implantation sequence of the electrodes, wherein the paths do not cross each other.

15. The method for determining the electrode implantation position according to claim 1, wherein: The first camera and the second camera are calibrated by matching the features of the data.

16. The method for determining the electrode implantation position according to claim 15, wherein: Use a square filter to implement a Gaussian blur image processing effect.

17. An electrode implantation system based on machine vision, comprising: a first camera configured to capture a first image of a brain surface; a second camera configured to capture a second image of the brain surface; A blood vessel segmentation operation unit is configured to perform operation processing on the first image and the second image, wherein a brain surface blood vessel area mask is obtained based on a blood vessel segmentation algorithm to determine an implantable area in the brain surface image; an implantation sequence determination unit, configured to select at least one implantation position in the implantable area, calculate the distance between the at least one implantation position and the electrode position according to the known electrode position, thereby determining the implantation sequence of the electrodes; an implantation point prediction unit, configured to match the imaging of the first camera and the second camera to obtain a transformation matrix, project a first straight line where the implantation position is located in the imaging of the first camera to the imaging of the second camera, and determine an intersection point between the first straight line and a second straight line where the implantation position is located in the imaging of the second camera as a predicted landing point of the implantation device; and The implant device control unit is configured to control the implant device in real time according to the predicted landing point until the implant point coincides with the predicted landing point.

18. The electrode implantation system according to claim 17, wherein: Fully supervised or semi-supervised control of the implanted device based on the predicted landing point.

19. The electrode implantation system according to claim 17, wherein: The imaging plane of the first camera is the same as that of the second camera.

20. The electrode implantation system of claim 17, wherein: Each of the first camera and the second camera includes an optical system, and the first camera and the second camera are coupled to a motion control system, respectively.

21. The electrode implantation system according to claim 20, wherein: The optical system includes an external light source that allows the brain surface to be illuminated evenly.

22. The electrode implant system of claim 17, wherein: The first camera and the second camera perform image processing respectively, and merge the image processed data from the first camera and the second camera; The coordinates of the implant device in the images of the first camera and the second camera are determined based on the merged data.

23. The electrode implant system of claim 22, wherein: The movement of the implant device is controlled by a motion control system according to the coordinates.

24. The electrode implantation system according to claim 17, wherein the blood vessel segmentation algorithm comprises the following steps: Convert brain surface images into grayscale images; Segment the grayscale image according to an adaptive threshold; Remove small contour noise from the segmented grayscale image; Perform an opening operation to remove the bubble-like noise pattern in the blood vessels; Perform inverse operation; Performing dilation processing to obtain a safe distance at the border of the vascular region; and Perform the inverse operation again.

25. The electrode implant system of claim 24, wherein: The implantation sequence determination unit is further configured to determine the implantation sequence of the electrodes so that the electrodes being implanted do not exert force on the electrodes already implanted.

26. The electrode implant system of claim 25, wherein: The paths of the implanted electrodes are planned based on the implantation sequence of the electrodes, wherein the paths do not cross each other.

27. The electrode implant system of claim 17, wherein: The first camera and the second camera are calibrated by matching the features of the data.

28. The electrode implant system of claim 17, wherein: The implantation device comprises an implantation needle, an implantation feeding mechanism and an implantation actuator. The implant needle is configured to engage the free end of the electrode with its needle tip portion so as to drive the electrode to move. The implantation feeding mechanism is configured to move the implantation needle along the longitudinal direction of the implantation device, and The implantation actuator is configured to drive the implantation needle to insert the needle tip portion of the implantation needle into the brain.

Images