Position tracking method and device for endoscope adapter in neurosurgery robot

Abstract

The embodiment of the present application provides a method, device, equipment and medium for tracking the posture of an endoscope adapter in a neurosurgery robot. The method includes: obtaining a first image and a second image of the endoscope adapter collected by a binocular visual imaging device of an optical positioning tracker; identifying the marker area of ​​the first image and the second image respectively to obtain a first marker candidate area of ​​the first image and a second marker candidate area of ​​the second image; screening the marker area of ​​the first marker candidate area and the second marker candidate area respectively to obtain the first marker actual area of ​​the first image and the second marker actual area of ​​the second image; tracking the posture of the endoscope adapter according to the coordinates of the marker point in the first marker actual area and the coordinates of the marker point corresponding to the marker point in the second marker actual area. This solution can accurately track the posture of the endoscope adapter in space in real time.

Description

Technical Field

[0001] The present application relates to the technical field of neurosurgical robots in medical devices, specifically, to the field of image processing technology of endoscopic adapters in neurosurgical robots, and especially to a posture tracking method, device, electronic device and computer-readable medium of an endoscopic adapter in a neurosurgical robot. Background Art

[0002] In neurosurgery, since the technology relies more on manual experience, the workload of medical staff is very large, so the industry has developed surgical robots to treat patients. During the treatment of patients by the neurosurgery robot, the six-axis robotic arm on the operating table and console cart is used to move. The end of the robotic arm will be installed with the end tool for surgery to assist the doctor in removing lesions from the patient. An endoscope is installed at the end of the neurosurgery robot to collect images and internal structures that cannot be observed by the human eye. However, it is currently operated by the user holding the endoscope. This not only causes the user to be overly tired and poses a safety hazard to the observed object, but the user relies on experience to send the endoscope into the observed object and cannot accurately position it. Therefore, the surgical robot is operated by a robotic arm holding an endoscope adapter.

[0003] The endoscope adapter is a switching system between the endoscope eyepiece and the imaging chip, and is mainly used in minimally invasive surgery in modern medicine. Ultra-high-definition endoscope adapters can present better medical images and provide higher medical diagnosis and treatment services. In neurosurgery assisted by an endoscope adapter, it is necessary to accurately track the position of the endoscope adapter in space in real time to determine the relative position of the endoscope adapter in space with the patient's lesion. This task is usually completed by an optical positioning tracker. However, the software method currently implanted in the optical positioning tracker cannot accurately track the position of the endoscope adapter in space in real time. It can be seen that how to accurately track the position of the endoscope adapter in the neurosurgery robot in real time in space has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of this application is to propose a method, device, electronic device and computer-readable medium for tracking the position and posture of an endoscopic adapter in a neurosurgical robot, so as to solve the technical problem existing in the prior art of how to accurately track the position and posture of an endoscopic adapter in a neurosurgical robot in space in real time.

[0005] According to a first aspect of an embodiment of the present application, a method for tracking the position and posture of an endoscope adapter in a neurosurgical robot is provided, which is applied to an optical positioning tracker, and the method comprises: acquiring a first image and a second image of the endoscope adapter acquired by a binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among the multiple marker points included in the marker are not the same; identifying a marker area of ​​the first image according to pixel values ​​and pixel coordinates of pixel points in the first image to obtain a first marker candidate area of ​​the first image, and identifying a marker area of ​​the second image according to pixel values ​​and pixel coordinates of pixel points in the second image to obtain a first marker candidate area of ​​the second image. a second marker candidate area; screening the first marker candidate area for a marker area according to the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area to obtain the first marker actual area of ​​the first image, and screening the second marker candidate area for a marker area according to the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area to obtain the second marker actual area of ​​the second image; performing posture tracking on the endoscope adapter according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points in the second marker actual area corresponding to the marker points to obtain the posture information of the endoscope adapter.

[0006] According to a second aspect of an embodiment of the present application, a posture tracking device for an endoscope adapter in a neurosurgical robot is provided, which is applied to an optical positioning tracker, and the device includes: an acquisition module, used to acquire a first image and a second image of the endoscope adapter collected by a binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among the multiple marker points included in the marker are not the same; an identification module, used to identify a marker area of ​​the first image according to pixel values ​​and pixel coordinates of pixel points in the first image to obtain a first marker candidate area of ​​the first image, and to identify a marker area of ​​the second image according to the pixel values ​​and pixel coordinates of pixel points in the second image to obtain a first marker candidate area of ​​the second image. Two marker candidate areas; a screening module, used to screen the first marker candidate area according to the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area, so as to obtain the first marker actual area of ​​the first image, and to screen the second marker candidate area according to the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area, so as to obtain the second marker actual area of ​​the second image; a posture tracking module, used to perform posture tracking on the endoscope adapter according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points in the second marker actual area corresponding to the marker points, so as to obtain the posture information of the endoscope adapter.

[0007] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: one or more processors; a computer-readable medium configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the posture tracking method of the endoscopic adapter in a neurosurgical robot as described in the first aspect of the embodiment of the present application.

[0008] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for tracking the posture of an endoscopic adapter in a neurosurgical robot as described in the first aspect of an embodiment of the present application is implemented.

[0009] Through the posture tracking scheme of the endoscope adapter in the neurosurgical robot provided in the embodiment of the present application, the first image and the second image of the endoscope adapter collected by the binocular visual imaging device of the optical positioning tracker are obtained, and the marker areas are identified for the first image and the second image respectively to obtain the first marker candidate area of ​​the first image and the second marker candidate area of ​​the second image, and then the first marker candidate area and the second marker candidate area are screened for the marker areas respectively to obtain the first marker actual area of ​​the first image and the second marker actual area of ​​the second image, and then the posture of the endoscope adapter is tracked according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points corresponding to the marker points in the second marker actual area. The posture of the endoscope adapter in the neurosurgical robot can be tracked in real time and accurately, so as to determine the relative posture of the endoscope adapter in the neurosurgical robot and the patient's lesion in space in real time and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0011] Figure 1 A schematic diagram of an endoscope adapter and an optical positioning tracker in a neurosurgery robot provided in an embodiment of the present application;

[0012] Figure 2 A schematic diagram of an endoscope adapter in a neurosurgery robot provided in an embodiment of the present application;

[0013] Figure 3A A flowchart of the steps of the method for tracking the position and posture of an endoscope adapter in a neurosurgery robot provided in the first embodiment;

[0014] Figure 3B A schematic diagram of a marker of an endoscope adapter in a neurosurgery robot provided according to the first embodiment;

[0015] Figure 3C A schematic diagram of calculating the image distance cross ratio of a marker according to the first embodiment;

[0016] Figure 3D is a schematic diagram of a local physical coordinate system of a marker provided according to the first embodiment;

[0017] Figure 4 A schematic diagram of the structure of a posture tracking device for an endoscope adapter in a neurosurgery robot provided in the second embodiment;

[0018] Figure 5 A schematic diagram of the structure of an electronic device provided in the third embodiment;

[0019] Figure 6 This is the hardware structure of the electronic device provided in the fourth embodiment. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0021] Before describing in detail the position and posture tracking method of an endoscope adapter in a neurosurgical robot provided by an embodiment of the present application, a specific application scenario of the position and posture tracking method of an endoscope adapter in a neurosurgical robot provided by the present embodiment is described in detail.

[0022] In neurosurgery assisted by an endoscopic adapter, it is necessary to accurately track the position of the endoscopic adapter in the neurosurgery robot in space in real time to determine the relative position of the endoscopic adapter in the neurosurgery robot and the patient's lesion in space. This task is usually completed by an optical positioning tracker. In principle, an optical positioning tracker is equipped with a binocular visual imaging device, which uses two cameras in the binocular visual imaging device to simultaneously capture images of target points in space to determine the position of the target point in space in the coordinate system of the binocular visual imaging device. Figure 1 As shown, the optical positioning tracker 1 is tracking the position of the endoscope adapter 2 in the neurosurgery robot in space in real time. The endoscope adapter in the neurosurgery robot is provided with a marker, which is used to help the optical positioning tracker to track the endoscope adapter in the neurosurgery robot in real time during the operation. Figure 2 As shown, the position indicated by box 3 is the installation position of the marker.

[0023] The endoscope adapter in the neurosurgery robot involved in the embodiment of the present application may include: a clamping component, a motion component, a motion control component, a motion base and an adapter frame; the clamping component is provided with a first through hole for clamping the endoscope, the motion component is provided with a second through hole for the endoscope to pass through; the clamping component is detachably connected to the motion component; the motion control component is provided with a third through hole for the motion component to pass through; the motion base is provided with a fourth through hole for accommodating the motion component and the motion control component, the motion base is fixedly connected to the adapter frame, the motion base is annular, the motion base is connected to the motion control component, so that the motion component is stuck between the motion control component and the motion base, and the motion component can rotate relative to the motion base. Wherein, the motion component includes a connecting part and a moving part; the connecting part is a column provided with a second through hole, and the moving part is a spherical body provided with a second through hole; the inner wall of the motion base is provided with a curved surface adapted to the outer wall of the spherical body. The clamping assembly includes a locking cover and a clamping sleeve; the clamping sleeve is connected to the locking cover, the clamping sleeve is sleeved inside the motion assembly, the locking cover is sleeved outside the motion assembly, and the locking cover is threadedly connected to the motion assembly. The locking sleeve is made of elastic material, and a strip-shaped notch is provided on the locking sleeve. The motion control assembly includes an adjustment cover and an inner sleeve, the adjustment cover and the inner sleeve are detachably connected or fixedly connected, and the motion control assembly is made of elastic material.

[0024] Reference Figure 3A , showing a step flow chart of the method for tracking the posture of the endoscope adapter in the neurosurgical robot provided in the first embodiment of the present invention.

[0025] This embodiment will describe in detail the posture tracking method of the endoscope adapter in the neurosurgery robot provided in this embodiment from the perspective of the optical positioning tracker. Specifically, the posture tracking method of the endoscope adapter provided in this embodiment includes the following steps:

[0026] In step S101, a first image and a second image of the endoscope adapter captured by a binocular visual imaging device of the optical positioning tracker are acquired.

[0027] In this embodiment, the binocular visual imaging device includes a left visual imaging device and a right visual imaging device. The first image can be acquired by the left visual imaging device, and the second image can be acquired by the right visual imaging device. Both the first image and the second image contain the marker of the endoscope adapter, and the distances between any two marker points in the multiple marker points included in the marker are not the same. Both the first image and the second image are single-channel grayscale images with a height of h and a width of w, and the grayscale value range of each pixel in the first image and the second image is 0-255, 255 is white, and 0 is black.

[0028] In a specific example, Figure 3B As shown, the marker is composed of a white frame on the outermost circle, a black background inside, and four white dots in the black background, and the center distances between the four white dots are different. The four white dots are the four marking points of the marker.

[0029] In step S102, the landmark area of ​​the first image is identified based on the pixel values ​​and pixel coordinates of the pixel points in the first image to obtain the first landmark candidate area of ​​the first image, and the landmark area of ​​the second image is identified based on the pixel values ​​and pixel coordinates of the pixel points in the second image to obtain the second landmark candidate area of ​​the second image.

[0030] In some optional embodiments, when identifying the marker area of ​​the first image based on the pixel values ​​and pixel coordinates of the pixels in the first image, the pixels in the first image are binarized, and the first image after binarization is searched for boundaries based on the pixel values ​​and pixel coordinates of the pixels in the first image after binarization to obtain all boundaries of the first image after binarization; all boundaries of the first image after binarization are screened to obtain a parent boundary containing four sub-boundaries, and the area within the parent boundary is determined to be the first marker candidate area. Thus, by binarizing the pixels in the first image, searching for boundaries of the first image after binarization, and then screening all boundaries of the first image after binarization, the first marker candidate area of ​​the first image can be accurately obtained.

[0031] In a specific example, when binarizing the pixels in the first image, the average grayscale value, the maximum grayscale value and the minimum grayscale value of the pixels within the preset range of the target pixels to be binarized in the first image are determined; in response to determining that the difference between the maximum grayscale value and the minimum grayscale value is less than the preset difference threshold, or the grayscale value of the target pixel is less than or equal to the average grayscale value, the pixel value of the target pixel is adjusted to zero, and the binarization result of the target pixel is determined to be the adjusted pixel value of the target pixel; in response to determining that the grayscale value of the target pixel is greater than the average grayscale value, the pixel value of the target pixel is adjusted to one, and the binarization result of the target pixel is determined to be the adjusted pixel value of the target pixel. Among them, the difference threshold can be set by those skilled in the art according to actual needs, and this embodiment does not make any limitation on this. Thereby, the binarization effect of the pixels in the first image can be effectively improved through the average grayscale value, the maximum grayscale value and the minimum grayscale value.

[0032] In a specific example, a binary image is a black-and-white image in which a grayscale image is converted into a pixel with only two values, 0 or 1. The pixel value of a pixel whose grayscale value is higher than a certain pixel grayscale threshold is 1, and the pixel value of a pixel whose grayscale value is lower than the pixel grayscale threshold is 0. In order to eliminate the influence of different light intensities in different parts of the image on the effect of the binary processing, the pixel grayscale threshold of each pixel is determined separately, and the threshold is the average grayscale value of all pixels in a square range centered on the pixel. The side length of the square can be adjusted according to the image size and field of view. This algorithm will fail when there are large black or white image areas, that is, when the grayscale values ​​of all pixels in the square range of the target pixel are not much different. Therefore, the maximum and minimum grayscale values ​​of all pixels in the square range can be calculated, and the above situation can be judged by the difference between the maximum and minimum values. The mathematical expression is as follows:

[0033] Use p to represent any pixel on the image, (u p ,v p ) is the pixel coordinate of the pixel point, and the target pixel point of the binary transformation is set to p t , whose coordinates on the image are (u t ,v t ), let the side length of the square range centered on the target pixel be d, and the set of all pixel points within the square range be S,

[0034] S={p||u p -u t | <d&|v p -v t | <d}

[0035] Suppose the gray value of the pixel is I, and the average gray value of all pixels within the square is I avg ,

[0036]

[0037] Assume the binarization result of the target pixel is b t , the maximum gray value of all pixels within the square is I max , the minimum value is I min :

[0038] if I max -I min <50||I t ≤I avg ,

[0039] b t =0

[0040] else if I t >I avg ,

[0041] b t =1

[0042] The above operations are performed on all pixels in the image to obtain a complete binary processed image.

[0043] In some optional embodiments, when searching for boundaries of the first image after binarization based on the pixel values ​​and pixel coordinates of the pixel points in the first image after binarization, a first boundary point adding step is: raster scanning and traversing the pixel points in the first image after binarization, and in response to determining that the traversed pixel points are boundary points based on the pixel values ​​and pixel coordinates of the traversed pixel points, adding the traversed pixel points to a boundary point set; a boundary point determining step is: determining whether the pixel points within the eight-domain of the pixel points most recently added to the boundary point set are boundary points based on the pixel values ​​and pixel coordinates of the pixel points within the eight-domain of the pixel points most recently added to the boundary point set; a second boundary point adding step is: in response to determining that the pixel points within the eight-domain of the pixel points most recently added to the boundary point set are boundary points, adding the pixel points The pixel points within the eight regions of the pixel points in the boundary point set are added to the boundary point set; iterative execution step: iteratively execute the boundary point determination step and the second boundary point addition step until there are no boundary points added to the boundary point set, so as to obtain the boundary point set of the same boundary of the first image after binarization; the next pixel point of the pixel point where the raster scanning traversal stopped in the previous iteration is used as the pixel point where the raster scanning traversal starts in this iteration, and the first boundary point addition step and the iterative execution step are iteratively executed until all the pixel points in the first image after binarization are traversed by the raster scanning, so as to obtain the boundary point set of all boundaries of the first image after binarization; according to the boundary point set of all boundaries of the first image after binarization, all boundaries of the first image after binarization are determined. In this way, all boundaries of the first image after binarization can be accurately determined.

[0044] In some optional embodiments, when determining whether a pixel point in the eight-domain of a pixel point most recently added to the boundary point set is a boundary point based on the pixel value and pixel coordinates of the pixel point in the eight-domain of the pixel point most recently added to the boundary point set, in response to determining that the pixel value of a target pixel point in the eight-domain of the pixel point most recently added to the boundary point set is one, and determining that there is a pixel point with a pixel value of zero in the eight-domain of the target pixel point based on the pixel coordinates of the target pixel point, the target pixel point is determined to be a boundary point. In this way, it is possible to accurately determine whether a pixel point in the eight-domain of a pixel point most recently added to the boundary point set is a boundary point.

[0045] In a specific example, we first define the boundary points of the binary image as pixels with a pixel value of 1 and a pixel value of 0 in its 8-neighborhood. Definition of 8-neighborhood: If a pixel point p(u p ,v p) at a pixel q(u q ,v q ) is within the 8-neighborhood of , then max(|u p -u q |,|v p -v q |)≤1. All connected boundary points form a set of boundaries. The method to find all boundaries in an image is as follows: establish a boundary point set, which is initially an empty set, perform raster scanning on the binary image (traverse the pixels from left to right and from top to bottom in sequence), add the first pixel that meets the boundary point definition to the boundary point set, find out whether there are any pixels that meet the boundary point definition in the 8-neighborhood of the pixel, and add them to the boundary point set, find out whether there are new pixels that meet the boundary point definition in the 8-neighborhood of the pixel newly added to the boundary point set, and add them to the set, until there are no pixels that can be added to the set, then all the pixels in the set form the same boundary. After that, establish a new empty boundary set, start scanning from where the last raster scan stopped, find the next pixel that meets the boundary point definition, and add it to the new boundary set, and repeat until all the pixels in the image have been traversed. All the boundary sets obtained are all the boundaries in the binary image.

[0046] In some optional embodiments, when screening all the boundaries of the first image after the binarization process, for any two boundaries among the all boundaries, in response to determining that the line connecting any boundary point on one boundary to the edge of the first image after the binarization process intersects with the boundary point on the other boundary, the one boundary is determined to be a subboundary of the other boundary, and the other boundary is a parent boundary of the one boundary; in response to determining that the parent boundary includes four subboundaries, a parent boundary including four subboundaries is obtained. In this way, the parent boundary including four subboundaries can be accurately obtained.

[0047] In a specific example, the parent boundary and the child boundary are defined. If the line connecting any point on boundary 1 to the edge of the image intersects with a point on boundary 2, then boundary 1 is the child boundary of boundary 2, and boundary 2 is the parent boundary of boundary 1. Among all the boundaries, find a boundary whose parent boundary contains four child boundaries, and the area in the parent boundary is the candidate landmark area.

[0048] In some optional embodiments, the method further includes: performing ellipse fitting on each of the four sub-boundaries according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting results of each of the sub-boundaries; in response to determining that the ellipse fitting results of each of the sub-boundaries are all ellipses, determining that the area within the parent boundary containing the four sub-boundaries is a new marker candidate area obtained by screening the first marker candidate area. Thereby, the new marker candidate area obtained by screening the first marker candidate area can be accurately obtained.

[0049] In a specific example, in response to determining that the ellipse fitting results of each sub-boundary are not all ellipses, it is determined that the area within the parent boundary containing the four sub-boundaries is not a new marker candidate area obtained by screening the first marker candidate area. In addition, the sub-pixel coordinates of the center of the ellipse fitted to each sub-boundary can be determined based on the ellipse fitting parameters of each sub-boundary, and the sub-pixel coordinates of the center of the ellipse fitted to each sub-boundary are determined as the sub-pixel coordinates of the center of each sub-boundary.

[0050] In a specific example, when performing ellipse fitting on each sub-boundary according to the pixel coordinates of the boundary points on each of the four sub-boundaries, the preset boundary fitting ellipse formula is solved according to the pixel coordinates of the boundary points on each sub-boundary to obtain the ellipse fitting parameters of each sub-boundary; the closest distance from the boundary points on each sub-boundary to the ellipse fitted to each sub-boundary is determined according to the ellipse fitting parameters of each sub-boundary and the pixel coordinates of the boundary points on each sub-boundary; in response to determining that the ratio of the number of boundary points on each sub-boundary whose closest distance is less than the preset distance threshold is equal to or greater than the preset ratio threshold, the ellipse fitting result of each sub-boundary is determined to be an ellipse. Wherein, the distance threshold and the ratio threshold can be set by those skilled in the art according to actual needs, and this embodiment does not impose any limitation on this. Thereby, the ellipse fitting result of each sub-boundary can be accurately obtained.

[0051] In a specific example, after determining that the ellipse fitting result of each sub-boundary is an ellipse, the method further includes: eliminating the boundary points on each sub-boundary whose nearest distance is equal to or greater than the distance threshold, so as to obtain all boundary points on each sub-boundary whose nearest distance is less than the distance threshold; solving the boundary fitting ellipse formula again according to the pixel coordinates of all boundary points on each sub-boundary whose nearest distance is less than the distance threshold, so as to obtain new ellipse fitting parameters of each sub-boundary; determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary according to the new ellipse fitting parameters of each sub-boundary, and determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary as the sub-pixel coordinates of the center of each sub-boundary. In this way, not only the new ellipse fitted to each sub-boundary can be obtained more accurately, but also the sub-pixel coordinates of the center of each sub-boundary can be obtained.

[0052] In a specific example, first determine whether the sub-boundary is an ellipse. Specifically, the following boundary fitting ellipse formula is solved by the least squares method based on the pixel coordinates of all boundary points on the sub-boundary:

[0053] And meet 4ac-b 2 =1

[0054] According to the ellipse fitting parameters (a, b, c, d, e and f) of the sub-boundary and the pixel coordinates of the boundary points on the sub-boundary, the closest distance from the boundary points on the sub-boundary to the ellipse fitted by the sub-boundary is determined. If the proportion of boundary points on the sub-boundary whose closest distance is less than the preset distance threshold is equal to or greater than the preset ratio threshold, the ellipse fitting result of the sub-boundary is determined to be an ellipse. Among them, the distance threshold is generally set to 1.5 pixels and can be adjusted. The ratio threshold can be 80%. If the sub-boundary meets the judgment of the ellipse, all boundary points on the sub-boundary that are not near the ellipse just fitted (boundary points on the sub-boundary whose closest distance is equal to or greater than the distance threshold) are removed, and the pixel coordinates of the remaining boundary points on the sub-boundary are used to re-fit the ellipse, and the sub-pixel coordinates of the center of the new ellipse fitted by the sub-boundary are set as the sub-pixel coordinates of the center of the sub-boundary. Finally, the parent boundary of the four sub-boundaries, which are all ellipses, is defined as a new candidate marker area, and the sub-pixel coordinates of the centers of the four sub-boundaries are recorded. Among them, the sub-pixel coordinates of the centers of the four sub-boundaries are the sub-pixel coordinates of the centers of the four white dots in the marker.

[0055] In some optional embodiments, when identifying the marker area of ​​the second image based on the pixel values ​​and pixel coordinates of the pixels in the second image, the pixels in the second image are binarized, and the second image after binarization is searched for boundaries based on the pixel values ​​and pixel coordinates of the pixels in the second image after binarization to obtain all boundaries of the second image after binarization; all boundaries of the second image after binarization are screened to obtain a parent boundary containing four sub-boundaries, and the area within the parent boundary is determined as the second marker candidate area. Thus, by binarizing the pixels in the second image, searching for boundaries of the second image after binarization, and then screening all boundaries of the second image after binarization, the second marker candidate area of ​​the second image can be accurately obtained.

[0056] In a specific example, when binarizing the pixels in the second image, the average grayscale value, the maximum grayscale value and the minimum grayscale value of the pixels within the preset range of the target pixels to be binarized in the second image are determined; in response to determining that the difference between the maximum grayscale value and the minimum grayscale value is less than the preset difference threshold, or the grayscale value of the target pixel is less than or equal to the average grayscale value, the pixel value of the target pixel is adjusted to zero, and the binarization result of the target pixel is determined to be the adjusted pixel value of the target pixel; in response to determining that the grayscale value of the target pixel is greater than the average grayscale value, the pixel value of the target pixel is adjusted to one, and the binarization result of the target pixel is determined to be the adjusted pixel value of the target pixel. Among them, the difference threshold can be set by those skilled in the art according to actual needs, and this embodiment does not make any limitation on this. Thereby, the binarization effect of the pixels in the second image can be effectively improved through the average grayscale value, the maximum grayscale value and the minimum grayscale value.

[0057] Since the specific implementation of binarizing the pixels in the second image is similar to the specific implementation of binarizing the pixels in the first image, it will not be described in detail here.

[0058] In some optional embodiments, when searching for boundaries of the second image after binarization based on the pixel values ​​and pixel coordinates of the pixel points in the second image after binarization, a first boundary point adding step is: raster scanning and traversing the pixel points in the second image after binarization, and in response to determining that the traversed pixel points are boundary points based on the pixel values ​​and pixel coordinates of the traversed pixel points, adding the traversed pixel points to a boundary point set; a boundary point determining step is: determining whether the pixel points within the eight-domain of the pixel points most recently added to the boundary point set are boundary points based on the pixel values ​​and pixel coordinates of the pixel points within the eight-domain of the pixel points most recently added to the boundary point set; a second boundary point adding step is: in response to determining that the pixel points within the eight-domain of the pixel points most recently added to the boundary point set are boundary points, adding the pixel points The pixel points within the eight regions of the pixel points in the boundary point set are added to the boundary point set; iterative execution step: iteratively execute the boundary point determination step and the second boundary point addition step until there are no boundary points added to the boundary point set, so as to obtain the boundary point set of the same boundary of the second image after binarization; the next pixel point of the pixel point where the raster scanning traversal stopped in the previous iteration is used as the pixel point where the raster scanning traversal starts in this iteration, and iteratively execute the first boundary point addition step and the iterative execution step until all the pixel points in the second image after binarization are traversed by the raster scanning, so as to obtain the boundary point set of all boundaries of the second image after binarization; according to the boundary point set of all boundaries of the second image after binarization, determine all boundaries of the second image after binarization. In this way, all boundaries of the second image after binarization can be accurately determined.

[0059] In some optional embodiments, when determining whether a pixel point in the eight-domain of a pixel point most recently added to the boundary point set is a boundary point based on the pixel value and pixel coordinates of the pixel point in the eight-domain of the pixel point most recently added to the boundary point set, in response to determining that the pixel value of a target pixel point in the eight-domain of the pixel point most recently added to the boundary point set is one, and determining that there is a pixel point with a pixel value of zero in the eight-domain of the target pixel point based on the pixel coordinates of the target pixel point, the target pixel point is determined to be a boundary point. In this way, it is possible to accurately determine whether a pixel point in the eight-domain of a pixel point most recently added to the boundary point set is a boundary point.

[0060] Since the specific implementation method of performing boundary search on the second image after binarization based on the pixel values ​​and pixel coordinates of the pixel points in the second image after binarization is similar to the specific implementation method of performing boundary search on the first image after binarization based on the pixel values ​​and pixel coordinates of the pixel points in the first image after binarization, it will not be repeated here.

[0061] In some optional embodiments, when screening all the boundaries of the second image after binarization, for any two boundaries among all the boundaries, in response to determining that the line connecting any boundary point on one boundary to the edge of the second image after binarization intersects with the boundary point on the other boundary, the one boundary is determined to be a subboundary of the other boundary, and the other boundary is a parent boundary of the one boundary; in response to determining that the parent boundary includes four subboundaries, a parent boundary including four subboundaries is obtained. In this way, a parent boundary including multiple subboundaries can be accurately obtained.

[0062] Since the specific implementation of screening all the boundaries of the second image after the binarization process is similar to the specific implementation of screening all the boundaries of the first image after the binarization process, it is not repeated here.

[0063] In some optional embodiments, the method further includes: performing ellipse fitting on each of the four sub-boundaries according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting results of each of the sub-boundaries; in response to determining that the ellipse fitting results of each of the sub-boundaries are all ellipses, determining that the area within the parent boundary containing the four sub-boundaries is a new marker candidate area obtained by screening the second marker candidate area. Thereby, the new marker candidate area obtained by screening the second marker candidate area can be accurately obtained.

[0064] In a specific example, in response to determining that the ellipse fitting results of each sub-boundary are not all ellipses, it is determined that the area within the parent boundary containing the four sub-boundaries is not a new marker candidate area obtained by screening the second marker candidate area. In addition, the sub-pixel coordinates of the center of the ellipse fitted to each sub-boundary can be determined based on the ellipse fitting parameters of each sub-boundary, and the sub-pixel coordinates of the center of the ellipse fitted to each sub-boundary are determined as the sub-pixel coordinates of the center of each sub-boundary.

[0065] In a specific example, when performing ellipse fitting on each sub-boundary according to the pixel coordinates of the boundary points on each of the four sub-boundaries, the preset boundary fitting ellipse formula is solved according to the pixel coordinates of the boundary points on each sub-boundary to obtain the ellipse fitting parameters of each sub-boundary; the closest distance from the boundary points on each sub-boundary to the ellipse fitted to each sub-boundary is determined according to the ellipse fitting parameters of each sub-boundary and the pixel coordinates of the boundary points on each sub-boundary; in response to determining that the ratio of the number of boundary points on each sub-boundary whose closest distance is less than the preset distance threshold is equal to or greater than the preset ratio threshold, the ellipse fitting result of each sub-boundary is determined to be an ellipse. Wherein, the distance threshold and the ratio threshold can be set by those skilled in the art according to actual needs, and this embodiment does not impose any limitation on this. Thereby, the ellipse fitting result of each sub-boundary can be accurately obtained.

[0066] In a specific example, after determining that the ellipse fitting result of each sub-boundary is an ellipse, the method further includes: eliminating the boundary points on each sub-boundary whose nearest distance is equal to or greater than the distance threshold, so as to obtain all boundary points on each sub-boundary whose nearest distance is less than the distance threshold; solving the boundary fitting ellipse formula again according to the pixel coordinates of all boundary points on each sub-boundary whose nearest distance is less than the distance threshold, so as to obtain new ellipse fitting parameters of each sub-boundary; determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary according to the new ellipse fitting parameters of each sub-boundary, and determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary as the sub-pixel coordinates of the center of each sub-boundary. In this way, not only the new ellipse fitted to each sub-boundary can be obtained more accurately, but also the sub-pixel coordinates of the center of each sub-boundary can be obtained.

[0067] In step S103, the first marker candidate area is screened as a marker area based on the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area to obtain the first marker actual area of ​​the first image, and the second marker candidate area is screened as a marker area based on the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area to obtain the second marker actual area of ​​the second image.

[0068] In some optional embodiments, when screening the marker region for the first marker candidate region according to the image distance between the four marker points in the first marker candidate region and the identification information of the four marker points in the first marker candidate region, the image distance cross ratio between the four marker points in the first marker candidate region is determined according to the image distance between the four marker points in the first marker candidate region and the identification information of the four marker points in the first marker candidate region; the actual distance cross ratio between the four marker points in the marker is determined according to the actual distance between the four marker points in the marker and the identification information of the four marker points in the marker, wherein the identification information of the four marker points in the marker is the same as the identification information of the four marker points in the first marker candidate region; in response to determining that the image distance cross ratio between the four marker points in the first marker candidate region is the same as the actual distance cross ratio between the four marker points in the marker, the first marker candidate region is determined to be the first marker actual region. The number of the first marker candidate regions is multiple, and the number of the first marker actual region is one. In this way, the first marker actual region of the first image can be accurately determined.

[0069] In some optional embodiments, the method also includes: in response to determining that the image distance cross-ratio between the four marker points in the first marker candidate area is different from the actual distance cross-ratio between the four marker points in the marker, adjusting the identification information of the four marker points in the first marker candidate area, and determining the adjusted image distance cross-ratio between the four marker points in the first marker candidate area based on the image distance between the four marker points in the first marker candidate area and the adjusted identification information of the four marker points in the first marker candidate area; determining the adjusted actual distance cross-ratio between the four marker points in the marker based on the actual distance between the four marker points in the marker and the adjusted identification information of the four marker points in the marker, wherein the adjusted identification information of the four marker points in the marker is the same as the adjusted identification information of the four marker points in the first marker candidate area; in response to determining that the adjusted image distance cross-ratio between the four marker points in the first marker candidate area is the same as the adjusted actual distance cross-ratio between the four marker points in the marker, determining that the first marker candidate area is the first marker actual area. Thereby, when it is determined that the image distance cross ratio between the four marker points in the first marker candidate area is different from the actual distance cross ratio between the four marker points in the marker, the actual area of ​​the first marker in the first image can be accurately determined.

[0070] In a specific example, Figure 3C As shown, the four dots in the marker are marked as 1, 2, 3 and 4. The distances between the four dots are a, b, c and d. The marker is imaged by the camera, which is equivalent to a projective transformation. The cross ratio (ratio of the ratio of the distances) of the projective transformation remains unchanged (for example, ), so the cross ratio of the distance between the dots on the image is the same as the cross ratio of the actual distance between the dots on the landmark. The four dots in the parent boundary are preliminarily sorted clockwise as 1, 2, 3, 4, and the cross ratio is calculated based on the sub-pixel coordinates of the centers of the four dots in the parent boundary. Compare the cross ratio with the actual distance between the dots on the marker (which can be calculated from the actual coordinates of the center of the dots on the marker). If they are the same, the label order is consistent with the actual label order, and the area within the parent boundary is the actual area of ​​the marker. If they are different, adjust the label one position clockwise, that is, 1 to 2, 2 to 3, 3 to 4, and 4 to 1, and then calculate the cross ratio, and compare the calculated cross ratio with the cross ratio of the actual distance between the dots on the marker. Repeat this four times. If all four labels are tried and there is no cross ratio that is the same as the cross ratio of the actual distance between the dots on the marker, then the parent boundary is not the actual area of ​​the marker.

[0071] In some optional embodiments, when screening the second marker candidate region as a marker region according to the image distance between the four marker points in the second marker candidate region and the identification information of the four marker points in the second marker candidate region, the image distance cross ratio between the four marker points in the second marker candidate region is determined according to the image distance between the four marker points in the second marker candidate region and the identification information of the four marker points in the second marker candidate region; the actual distance cross ratio between the four marker points in the marker is determined according to the actual distance between the four marker points in the marker and the identification information of the four marker points in the marker, wherein the identification information of the four marker points in the marker is the same as the identification information of the four marker points in the first marker candidate region; in response to determining that the image distance cross ratio between the four marker points in the second marker candidate region is the same as the actual distance cross ratio between the four marker points in the marker, the second marker candidate region is determined to be the second marker actual region. The number of the second marker candidate regions is multiple, and the number of the second marker actual region is one. In this way, the second marker actual region of the second image can be accurately determined.

[0072] In some optional embodiments, the method also includes: in response to determining that the image distance cross-ratio between the four marker points in the second marker candidate area is different from the actual distance cross-ratio between the four marker points in the marker, adjusting the identification information of the four marker points in the second marker candidate area, and determining the adjusted image distance cross-ratio between the four marker points in the second marker candidate area based on the image distance between the four marker points in the second marker candidate area and the adjusted identification information of the four marker points in the second marker candidate area; determining the adjusted actual distance cross-ratio between the four marker points in the marker based on the actual distance between the four marker points in the marker and the adjusted identification information of the four marker points in the marker, wherein the adjusted identification information of the four marker points in the marker is the same as the adjusted identification information of the four marker points in the second marker candidate area; in response to determining that the adjusted image distance cross-ratio between the four marker points in the second marker candidate area is the same as the adjusted actual distance cross-ratio between the four marker points in the marker, determining that the second marker candidate area is the second marker actual area. Thereby, when it is determined that the image distance cross ratio between the four marker points in the second marker candidate area is different from the actual distance cross ratio between the four marker points in the marker, the actual area of ​​the second marker in the second image can be accurately determined.

[0073] Since the specific implementation method of screening the marker area for the second marker candidate area based on the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area is similar to the specific implementation method of screening the marker area for the first marker candidate area based on the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area, it will not be repeated here.

[0074] In step S104, the posture of the endoscope adapter is tracked according to the coordinates of the marker point in the actual area of ​​the first marker and the coordinates of the marker point corresponding to the marker point in the actual area of ​​the second marker to obtain the posture information of the endoscope adapter.

[0075] In some optional embodiments, when the posture tracking of the endoscope adapter is performed according to the coordinates of the marker points in the actual area of ​​the first marker and the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points, the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device are determined according to the coordinates of the marker points in the actual area of ​​the first marker, the projection matrix of the left visual imaging device that collects the first image in the binocular vision imaging device, the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points, and the projection matrix of the right visual imaging device that collects the second image in the binocular vision imaging device; the posture information of the marker in the coordinate system of the binocular vision imaging device is determined according to the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device, and the posture information of the marker in the coordinate system of the binocular vision imaging device is determined as the posture information of the endoscope adapter in the coordinate system of the binocular vision imaging device. The projection matrix of the left visual imaging device in the binocular visual imaging device that collects the first image can be obtained from the internal parameters of the left visual imaging device, and the projection matrix of the right visual imaging device in the binocular visual imaging device that collects the second image can be obtained from the internal parameters of the right visual imaging device. In this way, the position information of the endoscope adapter in the coordinate system of the binocular visual imaging device can be accurately determined.

[0076] In some optional embodiments, when determining the posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point of the marker point in the marker in the coordinate system of the binocular vision imaging device are determined according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device; the coordinates of the second center of gravity point of the marker point in the marker in the local physical coordinate system of the marker are determined according to the coordinates of the marker point in the marker in the local physical coordinate system of the marker; the covariance matrix of the marker point in the marker in the coordinate system of the binocular vision imaging device and the local physical coordinate system of the marker is determined according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point, the coordinates of the marker point in the marker in the local physical coordinate system of the marker, and the coordinates of the second center of gravity point; the position information and posture information of the marker in the coordinate system of the binocular vision imaging device are determined according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the covariance matrix. Thereby, the position information and posture information of the marker in the coordinate system of the binocular vision imaging device can be accurately determined.

[0077] In some optional embodiments, when determining the covariance matrix of the marker point in the marker in the coordinate system of the binocular vision imaging device and the local physical coordinate system of the marker according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point, the coordinates of the marker point in the marker in the local physical coordinate system of the marker, and the coordinates of the second center of gravity point, determine the first vector from the coordinate point of the marker point in the marker in the coordinate system of the binocular vision imaging device to the first center of gravity point according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device and the coordinates of the first center of gravity point; determine the second vector from the coordinate point of the marker point in the marker in the local physical coordinate system of the marker to the second center of gravity point according to the coordinates of the marker point in the marker in the local physical coordinate system of the marker and the coordinates of the second center of gravity point; and determine the covariance matrix according to the first vector and the second vector. In this way, the covariance matrix can be accurately determined.

[0078] In some optional embodiments, when determining the position information and posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the covariance matrix, singular value decomposition is performed on the covariance matrix to obtain the left singular matrix and the right singular matrix of the covariance matrix; the posture information of the marker in the coordinate system of the binocular vision imaging device is determined according to the left singular matrix and the right singular matrix of the covariance matrix; the position information of the marker in the coordinate system of the binocular vision imaging device is determined according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the posture information of the marker in the coordinate system of the binocular vision imaging device. Thereby, the position information and posture information of the marker in the coordinate system of the binocular vision imaging device can be accurately determined.

[0079] In a specific example, the three-dimensional coordinates of the center of the circle in the coordinate system of the optical positioning tracker can be obtained by the sub-pixel coordinates of the center of the circle corresponding to the left and right images (the first image and the second image). Suppose the projection matrices of the left visual imaging device and the right visual imaging device are P and l (3x4 matrix) and P r (3x4 matrix), the three-dimensional homogeneous coordinates of the center of the circle in the coordinate system of the optical positioning tracker are V (4x1 vector), and the homogeneous coordinates of the center of the same circle in the left and right eye images are v l (3x1 vector) and v r (3x1 vector), then there is a mapping relationship vl =P l V, v r =P r V, we can deduce: v l ×P l V=0,v r ×P r V=0, the above two equations can be arranged into the form of AV=0, where:

[0080]

[0081] P iT is the ith row of P, and V is solved by the least square method, which is the three-dimensional coordinate of the center of the circle in the coordinate system of the optical positioning tracker. Through the above method, the three-dimensional coordinates of the center of the four circles of the marker in the coordinate system of the optical positioning tracker can be calculated.

[0082] Then, the position and posture information of the marker under the optical positioning tracker can be calculated through the three-dimensional coordinates of the centers of the four points of the marker in the coordinate system of the optical positioning tracker. The local physical coordinate system of the marker is as follows: Figure 3D As shown in the figure, the coordinates of the centers of the four dots in the local physical coordinate system are known. Let the coordinates of the centers of the four dots in the local physical coordinate system of the marker be m i The coordinates of the centers of the four dots in the coordinate system of the optical positioning tracker are e i , there are 4 centers in total. First find the coordinates of the center of gravity in each coordinate system (the coordinates of the first center of gravity and the coordinates of the second center of gravity):

[0083]

[0084] Find the vector (first vector and second vector) from each coordinate point to its corresponding centroid in each coordinate system:

[0085]

[0086] Find the covariance matrix:

[0087] S=FG T

[0088] F and G are respectively i and g i is a 3×4 matrix with columns. Find the singular value decomposition of S = UΣV T , then the posture information of the marker in the coordinate system of the optical positioning tracker is:

[0089]

[0090] Among them, U represents the left singular matrix, V represents the right singular matrix;

[0091] The position information of the marker in the coordinate system of the optical positioning tracker is:

[0092]

[0093] At this point, the tracking of the marker in the coordinate system of the optical positioning tracker is completed.

[0094] By using the posture tracking method of the endoscope adapter in the neurosurgical robot provided in the present embodiment, the first image and the second image of the endoscope adapter collected by the binocular visual imaging device of the optical positioning tracker are obtained, and the marker areas are identified for the first image and the second image respectively to obtain the first marker candidate area of ​​the first image and the second marker candidate area of ​​the second image, and then the marker areas are screened for the first marker candidate area and the second marker candidate area respectively to obtain the first marker actual area of ​​the first image and the second marker actual area of ​​the second image, and then the posture of the endoscope adapter is tracked according to the coordinates of the marker point in the first marker actual area and the coordinates of the marker point corresponding to the marker point in the second marker actual area. The posture of the endoscope adapter in the neurosurgical robot can be tracked in real time and accurately, so as to determine the relative posture of the endoscope adapter in the neurosurgical robot and the patient's lesion in space in real time and accurately.

[0095] The posture tracking method of the endoscope adapter in the neurosurgical robot provided in this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices, etc.

[0096] Reference Figure 4 , shows a schematic structural diagram of the position tracking device of the endoscope adapter in the neurosurgery robot provided in the second embodiment of the present invention.

[0097] The posture tracking device of the endoscope adapter in the neurosurgical robot provided in this embodiment is applied to an optical positioning tracker, and the device includes: an acquisition module 201, which is used to acquire a first image and a second image of the endoscope adapter collected by a binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among the multiple marker points included in the marker are not the same; an identification module 202, which is used to identify the marker area of ​​the first image according to the pixel values ​​and pixel coordinates of the pixel points in the first image to obtain a first marker candidate area of ​​the first image, and to identify the marker area of ​​the second image according to the pixel values ​​and pixel coordinates of the pixel points in the second image to obtain a second marker candidate area of ​​the second image. domain; a screening module 203, used to screen the first marker candidate area for marker areas according to the image distances between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area, so as to obtain the first marker actual area of ​​the first image, and to screen the second marker candidate area for marker areas according to the image distances between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area, so as to obtain the second marker actual area of ​​the second image; a posture tracking module 204, used to perform posture tracking on the endoscope adapter according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points in the second marker actual area corresponding to the marker points, so as to obtain the posture information of the endoscope adapter.

[0098] Optionally, the identification module 202 includes: a first processing submodule, used to binarize pixel points in the first image; a first search submodule, used to perform boundary search on the first image after binarization according to the pixel values ​​and pixel coordinates of the pixel points in the first image after binarization, so as to obtain all boundaries of the first image after binarization; a first screening submodule, used to screen all boundaries of the first image after binarization, so as to obtain a parent boundary containing multiple sub-boundaries, and determine that the area within the parent boundary is the first marker candidate area.

[0099] Optionally, the first processing submodule is specifically used to: determine the average grayscale value, maximum grayscale value and minimum grayscale value of pixel points within a preset range of the target pixel points to be binarized in the first image; in response to determining that the difference between the maximum grayscale value and the minimum grayscale value is less than a preset difference threshold, or the grayscale value of the target pixel point is less than or equal to the average grayscale value, the pixel value of the target pixel point is adjusted to zero, and the binarization processing result of the target pixel point is determined to be the adjusted pixel value of the target pixel point; in response to determining that the grayscale value of the target pixel point is greater than the average grayscale value, the pixel value of the target pixel point is adjusted to one, and the binarization processing result of the target pixel point is determined to be the adjusted pixel value of the target pixel point.

[0100] Optionally, the first search submodule includes: a first adding unit, which is used to execute a first boundary point adding step, that is, to perform a raster scanning traversal on the pixel points in the first image after the binarization processing, and in response to determining that the traversed pixel point is a boundary point according to the pixel value and pixel coordinate of the traversed pixel point, the traversed pixel point is added to the boundary point set; a first determining unit, which is used to execute a boundary point determining step, that is, to determine whether the pixel point within the eight domain of the pixel point most recently added to the boundary point set is a boundary point according to the pixel value and pixel coordinate of the pixel point within the eight domain of the pixel point most recently added to the boundary point set; a second adding unit, which is used to execute a second boundary point adding step, that is, in response to determining that the pixel point within the eight domain of the pixel point most recently added to the boundary point set is a boundary point, the pixel point within the eight domain of the pixel point most recently added to the boundary point set is a boundary point. The pixel point is added to the boundary point set; a first iterative unit is used to perform an iterative execution step, that is, iteratively execute the boundary point determination step and the second boundary point adding step until there are no boundary points added to the boundary point set, so as to obtain a boundary point set of the same boundary of the first image after binarization; a second iterative unit is used to use the next pixel point of the pixel point where the raster scanning traversal stopped in the previous iteration as the pixel point where the raster scanning traversal starts in this iteration, and iteratively execute the first boundary point adding step and the iterative execution step until all the pixel points in the first image after binarization are traversed by the raster scanning, so as to obtain a boundary point set of all boundaries of the first image after binarization; a second determining unit is used to determine all boundaries of the first image after binarization according to the boundary point set of all boundaries of the first image after binarization.

[0101] Optionally, the first determination unit is specifically used to: in response to determining that the pixel value of the target pixel point within the eight-domain of the pixel point most recently added to the boundary point set is one, and determining based on the pixel coordinates of the target pixel point that there is a pixel point with a pixel value of zero within the eight-domain of the target pixel point, then determine that the target pixel point is a boundary point.

[0102] Optionally, the first screening submodule is specifically used to: for any two boundaries among all the boundaries, in response to determining that the lines connecting any boundary point on one boundary to the edge of the first image after binarization processing intersect with the boundary points on the other boundary, determine that the one boundary is a sub-boundary of the other boundary, and the other boundary is a parent boundary of the one boundary; in response to determining that the parent boundary includes four sub-boundaries, obtain the parent boundary containing the four sub-boundaries.

[0103] Optionally, the identification module 202 also includes: a first fitting submodule, used to perform ellipse fitting on each sub-boundary according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting result of each sub-boundary; a first determination submodule, used to determine, in response to determining that the ellipse fitting results of each sub-boundary are ellipses, that the area within the parent boundary containing the four sub-boundaries is a new marker candidate area obtained by screening the first marker candidate area.

[0104] Optionally, the first fitting submodule includes: a first solving unit, used to solve a preset boundary fitting ellipse formula according to the pixel coordinates of the boundary points on each sub-boundary to obtain the ellipse fitting parameters of each sub-boundary; a first determining unit, used to determine the closest distance from the boundary points on each sub-boundary to the ellipse fitted to each sub-boundary according to the ellipse fitting parameters of each sub-boundary and the pixel coordinates of the boundary points on each sub-boundary; and a second determining unit, used to determine that the ellipse fitting result of each sub-boundary is an ellipse in response to determining that the proportion of the number of boundary points on each sub-boundary whose closest distance is less than a preset distance threshold is equal to or greater than a preset proportion threshold.

[0105] Optionally, after the second determination unit, the first fitting submodule also includes: a first elimination unit, used to eliminate boundary points on each sub-boundary whose nearest distance is equal to or greater than the distance threshold, so as to obtain all boundary points on each sub-boundary whose nearest distance is less than the distance threshold; a second solving unit, used to solve the boundary fitting ellipse formula again according to the pixel coordinates of all boundary points on each sub-boundary whose nearest distance is less than the distance threshold, so as to obtain new ellipse fitting parameters of each sub-boundary; a third determination unit, used to determine the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary according to the new ellipse fitting parameters of each sub-boundary, and determine the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary as the sub-pixel coordinates of the center of each sub-boundary.

[0106] Optionally, the identification module 202 includes: a second processing submodule, used to binarize pixel points in the second image; a second search submodule, used to perform boundary search on the second image after binarization according to the pixel values ​​and pixel coordinates of the pixel points in the second image after binarization to obtain all boundaries of the second image after binarization; a second screening submodule, used to screen all boundaries of the second image after binarization to obtain a parent boundary containing four sub-boundaries, and determine that the area within the parent boundary is the second marker candidate area.

[0107] Optionally, the second processing submodule is specifically used to: determine the average grayscale value, maximum grayscale value and minimum grayscale value of pixel points within a preset range of the target pixel points to be binarized in the second image; in response to determining that the difference between the maximum grayscale value and the minimum grayscale value is less than a preset difference threshold, or the grayscale value of the target pixel point is less than or equal to the average grayscale value, the pixel value of the target pixel point is adjusted to zero, and the binarization processing result of the target pixel point is determined to be the adjusted pixel value of the target pixel point; in response to determining that the grayscale value of the target pixel point is greater than the average grayscale value, the pixel value of the target pixel point is adjusted to one, and the binarization processing result of the target pixel point is determined to be the adjusted pixel value of the target pixel point.

[0108] Optionally, the second search submodule includes: a third adding unit, which is used to execute the first boundary point adding step, that is, to perform raster scanning traversal on the pixel points in the second image after binarization processing, and in response to determining that the traversed pixel points are boundary points based on the pixel values ​​and pixel coordinates of the traversed pixel points, the traversed pixel points are added to the boundary point set; a third determining unit, which is used to execute the boundary point determining step, that is, to determine whether the pixel points within the eight domains of the pixel points most recently added to the boundary point set are boundary points based on the pixel values ​​and pixel coordinates of the pixel points within the eight domains of the pixel points most recently added to the boundary point set; a fourth adding unit, which is used to execute the second boundary point adding step, that is, to determine that the pixel points within the eight domains of the pixel points most recently added to the boundary point set are boundary points in response to determining that the pixel points within the eight domains of the pixel points most recently added to the boundary point set are boundary points. The pixel point is added to the boundary point set; a third iterative unit is used to perform the iterative execution step, that is, iteratively execute the boundary point determination step and the second boundary point adding step until there are no boundary points added to the boundary point set, so as to obtain the boundary point set of the same boundary of the second image after binarization; a fourth iterative unit is used to use the next pixel point of the pixel point where the raster scanning traversal stopped in the previous iteration as the pixel point where the raster scanning traversal starts in this iteration, and iteratively execute the first boundary point adding step and the iterative execution step until all the pixel points in the second image after binarization are traversed by the raster scanning, so as to obtain the boundary point set of all boundaries of the second image after binarization; a fourth determining unit is used to determine all boundaries of the second image after binarization according to the boundary point set of all boundaries of the second image after binarization.

[0109] Optionally, the third determination unit is specifically used to: in response to determining that the pixel value of the target pixel point within the eight-domain of the pixel point most recently added to the boundary point set is one, and determining based on the pixel coordinates of the target pixel point that there is a pixel point with a pixel value of zero within the eight-domain of the target pixel point, then determine that the target pixel point is a boundary point.

[0110] Optionally, the second screening submodule is specifically used to: for any two boundaries among all the boundaries, in response to determining that the lines connecting any boundary point on one boundary to the edge of the second image after binarization processing intersect with the boundary points on the other boundary, determine that the one boundary is a sub-boundary of the other boundary, and the other boundary is a parent boundary of the one boundary; in response to determining that the parent boundary includes four sub-boundaries, obtain the parent boundary containing the four sub-boundaries.

[0111] Optionally, the identification module 202 also includes: a second fitting submodule, used to perform ellipse fitting on each sub-boundary according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting result of each sub-boundary; a second determination submodule, used to determine, in response to determining that the ellipse fitting results of each sub-boundary are ellipses, that the area within the parent boundary containing the four sub-boundaries is a new marker candidate area obtained by screening the second marker candidate area.

[0112] Optionally, the second fitting submodule includes: a third solving unit, used to solve a preset boundary fitting ellipse formula according to the pixel coordinates of the boundary points on each sub-boundary to obtain the ellipse fitting parameters of each sub-boundary; a fourth determining unit, used to determine the closest distance from the boundary points on each sub-boundary to the ellipse fitted to each sub-boundary according to the ellipse fitting parameters of each sub-boundary and the pixel coordinates of the boundary points on each sub-boundary; and a fifth determining unit, used to determine that the ellipse fitting result of each sub-boundary is an ellipse in response to determining that the proportion of the number of boundary points on each sub-boundary whose closest distance is less than a preset distance threshold is equal to or greater than a preset proportion threshold.

[0113] Optionally, after the fifth determination unit, the second fitting submodule also includes: a second elimination unit, used to eliminate boundary points on each sub-boundary whose nearest distance is equal to or greater than the distance threshold, so as to obtain all boundary points on each sub-boundary whose nearest distance is less than the distance threshold; a fourth solving unit, used to solve the boundary fitting ellipse formula again according to the pixel coordinates of all boundary points on each sub-boundary whose nearest distance is less than the distance threshold, so as to obtain new ellipse fitting parameters of each sub-boundary; a sixth determination unit, used to determine the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary according to the new ellipse fitting parameters of each sub-boundary, and determine the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary as the sub-pixel coordinates of the center of each sub-boundary.

[0114] Optionally, the screening module 203 includes: a third determination submodule, used to determine the image distance cross ratio between the four marker points in the first marker candidate area based on the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area; a fourth determination submodule, used to determine the actual distance cross ratio between the four marker points in the marker based on the actual distance between the four marker points in the marker and the identification information of the four marker points in the marker, wherein the identification information of the four marker points in the marker is the same as the identification information of the four marker points in the first marker candidate area; a fifth determination submodule, used to determine that the first marker candidate area is the first marker actual area in response to determining that the image distance cross ratio between the four marker points in the first marker candidate area is the same as the actual distance cross ratio between the four marker points in the marker.

[0115] Optionally, the screening module 203 also includes: a sixth determination submodule, for adjusting the identification information of the four marker points in the first marker candidate area in response to determining that the image distance cross-ratio between the four marker points in the first marker candidate area is different from the actual distance cross-ratio between the four marker points in the marker, and determining the adjusted image distance cross-ratio between the four marker points in the first marker candidate area according to the image distance between the four marker points in the first marker candidate area and the adjusted identification information of the four marker points in the first marker candidate area; a seventh determination submodule, for determining the adjusted actual distance cross-ratio between the four marker points in the marker according to the actual distance between the four marker points in the marker and the adjusted identification information of the four marker points in the marker, wherein the adjusted identification information of the four marker points in the marker is the same as the adjusted identification information of the four marker points in the first marker candidate area; an eighth determination submodule, for determining that the first marker candidate area is the first marker actual area in response to determining that the adjusted image distance cross-ratio between the four marker points in the first marker candidate area is the same as the adjusted actual distance cross-ratio between the four marker points in the marker.

[0116] Optionally, the screening module 203 includes: a ninth determination submodule, used to determine the image distance cross ratio between the four marker points in the second marker candidate area based on the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area; a tenth determination submodule, used to determine the actual distance cross ratio between the four marker points in the marker based on the actual distance between the four marker points in the marker and the identification information of the four marker points in the marker, wherein the identification information of the four marker points in the marker is the same as the identification information of the four marker points in the first marker candidate area; an eleventh determination submodule, used to determine that the second marker candidate area is the second marker actual area in response to determining that the image distance cross ratio between the four marker points in the second marker candidate area is the same as the actual distance cross ratio between the four marker points in the marker.

[0117] Optionally, the screening module 203 also includes: a twelfth determination submodule, which is used to adjust the identification information of the four marker points in the second marker candidate area in response to determining that the image distance cross ratio between the four marker points in the second marker candidate area is different from the actual distance cross ratio between the four marker points in the marker, and determine the adjusted image distance cross ratio between the four marker points in the second marker candidate area according to the image distance between the four marker points in the second marker candidate area and the adjusted identification information of the four marker points in the second marker candidate area; a thirteenth determination submodule, which is used to determine the adjusted actual distance cross ratio between the four marker points in the marker according to the actual distance between the four marker points in the marker and the adjusted identification information of the four marker points in the marker, wherein the adjusted identification information of the four marker points in the marker is the same as the adjusted identification information of the four marker points in the second marker candidate area; and a fourteenth determination submodule, which is used to determine that the second marker candidate area is the second marker actual area in response to determining that the adjusted image distance cross ratio between the four marker points in the second marker candidate area is the same as the adjusted actual distance cross ratio between the four marker points in the marker.

[0118] Optionally, the posture tracking module 204 includes: a fifteenth determination submodule, which is used to determine the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the actual area of ​​the first marker, the projection matrix of the left visual imaging device in the binocular vision imaging device that collects the first image, the coordinates of the marker point in the actual area of ​​the second marker corresponding to the marker point, and the projection matrix of the right visual imaging device in the binocular vision imaging device that collects the second image; a sixteenth determination submodule, which is used to determine the posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, and determine that the posture information of the marker in the coordinate system of the binocular vision imaging device is the posture information of the endoscope adapter in the coordinate system of the binocular vision imaging device.

[0119] Optionally, the sixteenth determination submodule includes: a seventh determination unit, used to determine the coordinates of a first center of gravity point of the marker point in the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device; an eighth determination unit, used to determine the coordinates of a second center of gravity point of the marker point in the marker in the local physical coordinate system of the marker according to the coordinates of the marker point in the marker in the local physical coordinate system of the marker; a ninth determination unit, used to determine the covariance matrix of the marker point in the marker in the coordinate system of the binocular vision imaging device and the local physical coordinate system of the marker according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point, the coordinates of the marker point in the marker in the local physical coordinate system of the marker, and the coordinates of the second center of gravity point; a tenth determination unit, used to determine the position information and posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the covariance matrix.

[0120] Optionally, the ninth determination unit is specifically used to: determine a first vector from the coordinate point of the marker point in the marker in the coordinate system of the binocular vision imaging device to the first center of gravity point according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device and the coordinates of the first center of gravity point; determine a second vector from the coordinate point of the marker point in the marker in the local physical coordinate system of the marker to the second center of gravity point according to the coordinates of the marker point in the marker in the local physical coordinate system of the marker and the coordinates of the second center of gravity point; determine the covariance matrix based on the first vector and the second vector.

[0121] Optionally, the tenth determination unit is specifically used to: perform singular value decomposition on the covariance matrix to obtain a left singular matrix and a right singular matrix of the covariance matrix; determine the posture information of the marker in the coordinate system of the binocular vision imaging device according to the left singular matrix and the right singular matrix of the covariance matrix; determine the position information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the posture information of the marker in the coordinate system of the binocular vision imaging device.

[0122] The posture tracking device of the endoscope adapter in the neurosurgical robot provided in this embodiment is used to implement the posture tracking method of the endoscope adapter in the neurosurgical robot corresponding to the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0123] Figure 5 This is a schematic diagram of the structure of an electronic device in the third embodiment; the electronic device may include:

[0124] One or more processors 301;

[0125] The computer readable medium 302 may be configured to store one or more programs,

[0126] When the one or more programs are executed by the one or more processors, the one or more processors implement the posture tracking method of the endoscopic adapter in the neurosurgical robot as described in the first embodiment.

[0127] Figure 6 is the hardware structure of the electronic device in the fourth embodiment; Figure 6 As shown, the hardware structure of the electronic device may include: a processor 401, a communication interface 402, a computer readable medium 403 and a communication bus 404;

[0128] The processor 401, the communication interface 402, and the computer readable medium 403 communicate with each other via a communication bus 404;

[0129] Optionally, the communication interface 402 may be an interface of a communication module, such as an interface of a GSM module;

[0130] The processor 401 can be specifically configured to: obtain a first image and a second image of the endoscope adapter captured by the binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among the multiple marker points included in the marker are not the same; identify the marker area of ​​the first image according to the pixel values ​​and pixel coordinates of the pixel points in the first image to obtain a first marker candidate area of ​​the first image, and identify the marker area of ​​the second image according to the pixel values ​​and pixel coordinates of the pixel points in the second image to obtain a second marker candidate area of ​​the second image; and identify the marker area of ​​the second image according to the first marker candidate area. According to the image distance between the four marker points in the domain and the identification information of the four marker points in the first marker candidate area, the first marker candidate area is screened as a marker area to obtain the first marker actual area of ​​the first image, and according to the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area, the second marker candidate area is screened as a marker area to obtain the second marker actual area of ​​the second image; according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points corresponding to the marker points in the second marker actual area, the endoscope adapter is tracked in position to obtain the position information of the endoscope adapter.

[0131] The processor 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0132] The computer readable medium 403 can be, but is not limited to, a random access memory medium (RAM), a read-only memory medium (ROM), a programmable read-only memory medium (PROM), an erasable programmable read-only memory medium (EPROM), an electrically erasable read-only memory medium (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0133] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in combination with an instruction execution system, an apparatus or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0134] Computer program code configured to perform the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. There are specific sequential relationships in the above-mentioned specific embodiments, but these sequential relationships are only exemplary. When the specific implementation is implemented, these steps may be less, more, or the execution order may be adjusted. That is, in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules involved in the embodiments described in the present application may be implemented by software or by hardware. The modules described may also be set in a processor, for example, may be described as: a processor includes an acquisition module, a recognition module, a screening module and a posture tracking module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances. For example, the acquisition module may also be described as "a module for acquiring the first image and the second image of the endoscopic adapter captured by the binocular visual imaging device of the optical positioning tracker".

[0137] As another aspect, the present application further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for tracking the position and posture of an endoscopic adapter in a neurosurgical robot as described in the first embodiment above.

[0138] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: obtains the first image and the second image of the endoscope adapter captured by the binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain the marker of the endoscope adapter, and the distances between any two marker points of the multiple marker points included in the marker are not the same; identifies the marker area of ​​the first image according to the pixel values ​​and pixel coordinates of the pixel points in the first image to obtain the first marker candidate area of ​​the first image, and identifies the marker area of ​​the second image according to the pixel values ​​and pixel coordinates of the pixel points in the second image to obtain the second marker candidate area of ​​the second image. domain; screening the first marker candidate area for a marker area according to the image distance between the four marker points in the first marker candidate area and the identification information of the four marker points in the first marker candidate area to obtain the first marker actual area of ​​the first image, and screening the second marker candidate area for a marker area according to the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area to obtain the second marker actual area of ​​the second image; tracking the posture of the endoscope adapter according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points in the second marker actual area corresponding to the marker points to obtain the posture information of the endoscope adapter.

[0139] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0140] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0141] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A method for tracking the position of an endoscope adapter in a neurosurgery robot, characterized in that: Applied to an optical positioning tracker, the method comprises: acquiring a first image and a second image of the endoscope adapter collected by a binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among the multiple marker points included in the marker are not the same; identifying a marker area of ​​the first image according to pixel values ​​and pixel coordinates of pixel points in the first image to obtain a first marker candidate area of ​​the first image, and identifying a marker area of ​​the second image according to pixel values ​​and pixel coordinates of pixel points in the second image to obtain a second marker candidate area of ​​the second image; and identifying a marker area of ​​the second image according to the first marker candidate area. According to the image distances between the four marker points in the domain and the identification information of the four marker points in the first marker candidate area, the first marker candidate area is screened as a marker area to obtain the first marker actual area of ​​the first image, and according to the image distances between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area, the second marker candidate area is screened as a marker area to obtain the second marker actual area of ​​the second image; according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points corresponding to the marker points in the second marker actual area, the posture tracking of the endoscope adapter is performed to obtain the posture information of the endoscope adapter; Among them, the posture tracking of the endoscope adapter is performed according to the coordinates of the marker points in the actual area of ​​the first marker and the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points to obtain the posture information of the endoscope adapter, including: determining the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker points in the actual area of ​​the first marker, the projection matrix of the left visual imaging device in the binocular vision imaging device that collects the first image, the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points, and the projection matrix of the right visual imaging device in the binocular vision imaging device that collects the second image; determining the posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device, and determining that the posture information of the marker in the coordinate system of the binocular vision imaging device is the posture information of the endoscope adapter in the coordinate system of the binocular vision imaging device.

2. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 1, characterized in that: The method of identifying a marker area of ​​the first image based on the pixel values ​​and pixel coordinates of the pixel points in the first image to obtain a first marker candidate area of ​​the first image includes: binarizing the pixel points in the first image, and performing boundary search on the binarized first image based on the pixel values ​​and pixel coordinates of the pixel points in the first image after the binarization to obtain all boundaries of the binarized first image; screening all boundaries of the first image after the binarization to obtain a parent boundary containing four sub-boundaries, and determining the area within the parent boundary as the first marker candidate area.

3. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 2, characterized in that: The binarization processing of the pixel points in the first image includes: determining the average grayscale value, maximum grayscale value and minimum grayscale value of the pixel points within a preset range of the target pixel points to be binarized in the first image; in response to determining that the difference between the maximum grayscale value and the minimum grayscale value is less than a preset difference threshold, or the grayscale value of the target pixel point is less than or equal to the average grayscale value, adjusting the pixel value of the target pixel point to zero, and determining that the binarization processing result of the target pixel point is the adjusted pixel value of the target pixel point; in response to determining that the grayscale value of the target pixel point is greater than the average grayscale value, adjusting the pixel value of the target pixel point to one, and determining that the binarization processing result of the target pixel point is the adjusted pixel value of the target pixel point.

4. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 2, characterized in that: The method comprises: performing boundary search on the first image after binarization according to the pixel values ​​and pixel coordinates of the pixel points in the first image after binarization to obtain all boundaries of the first image after binarization, including: a first boundary point adding step: performing raster scanning traversal on the pixel points in the first image after binarization, and in response to determining that the traversed pixel points are boundary points according to the pixel values ​​and pixel coordinates of the traversed pixel points, adding the traversed pixel points to the boundary point set; a boundary point determining step: determining whether the pixel points within the eight domains of the pixel points most recently added to the boundary point set are boundary points according to the pixel values ​​and pixel coordinates of the pixel points within the eight domains of the pixel points most recently added to the boundary point set; and a second boundary point adding step: in response to determining that the pixel points within the eight domains of the pixel points most recently added to the boundary point set are boundary points, Then, the pixel points within the eight regions of the pixel points most recently added to the boundary point set are added to the boundary point set; an iterative execution step: iteratively executing the boundary point determination step and the second boundary point adding step until there are no boundary points added to the boundary point set, so as to obtain a boundary point set of the same boundary of the first image after binarization processing; using the next pixel point of the pixel point where the raster scanning traversal stopped in the previous iteration as the pixel point where the raster scanning traversal starts in this iteration, and iteratively executing the first boundary point adding step and the iterative execution step until all the pixel points in the first image after binarization processing are traversed by the raster scanning, so as to obtain a boundary point set of all boundaries of the first image after binarization processing; determining all boundaries of the first image after binarization processing according to the boundary point set of all boundaries of the first image after binarization processing.

5. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 4, characterized in that: The method of determining whether a pixel point within the eight-domain of a pixel point most recently added to the boundary point set is a boundary point based on the pixel value and pixel coordinates of the pixel point within the eight-domain of the pixel point most recently added to the boundary point set comprises: in response to determining that the pixel value of a target pixel point within the eight-domain of the pixel point most recently added to the boundary point set is one, and determining that there is a pixel point with a pixel value of zero within the eight-domain of the target pixel point based on the pixel coordinates of the target pixel point, then determining that the target pixel point is a boundary point.

6. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 2, characterized in that: The screening of all boundaries of the first image after the binarization process to obtain a parent boundary containing four sub-boundaries includes: for any two boundaries among all the boundaries, in response to determining that a line connecting any boundary point on one boundary to an edge of the first image after the binarization process intersects with a boundary point on the other boundary, determining that the one boundary is a sub-boundary of the other boundary, and the other boundary is a parent boundary of the one boundary; in response to determining that the parent boundary includes the four sub-boundaries, obtaining a parent boundary containing the four sub-boundaries.

7. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 2 or 6, characterized in that: The method also includes: performing ellipse fitting on each of the four sub-boundaries according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting results of each sub-boundary; in response to determining that the ellipse fitting results of each sub-boundary are all ellipses, determining that the area within the parent boundary containing the four sub-boundaries is a new marker candidate area obtained by screening the first marker candidate area.

8. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 7, characterized in that: The method of performing ellipse fitting on each sub-boundary according to the pixel coordinates of the boundary points on each of the four sub-boundaries to obtain the ellipse fitting result of each sub-boundary includes: solving a preset boundary fitting ellipse formula according to the pixel coordinates of the boundary points on each sub-boundary to obtain the ellipse fitting parameters of each sub-boundary; determining the closest distance from the boundary points on each sub-boundary to the ellipse fitted to each sub-boundary according to the ellipse fitting parameters of each sub-boundary and the pixel coordinates of the boundary points on each sub-boundary; and in response to determining that the ratio of the number of boundary points on each sub-boundary whose closest distance is less than a preset distance threshold is equal to or greater than a preset ratio threshold, determining that the ellipse fitting result of each sub-boundary is an ellipse.

9. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 8, characterized in that: After determining that the ellipse fitting result of each sub-boundary is an ellipse, the method further includes: eliminating boundary points on each sub-boundary whose nearest distance is equal to or greater than the distance threshold to obtain all boundary points on each sub-boundary whose nearest distance is less than the distance threshold; solving the boundary fitting ellipse formula again according to the pixel coordinates of all boundary points on each sub-boundary whose nearest distance is less than the distance threshold to obtain new ellipse fitting parameters for each sub-boundary; determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary according to the new ellipse fitting parameters for each sub-boundary, and determining the sub-pixel coordinates of the center of the new ellipse fitted to each sub-boundary as the sub-pixel coordinates of the center of each sub-boundary.

10. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 1, characterized in that: The method of screening the marker area of ​​the first marker candidate area according to the image distance between the four marker points in the first marker candidate area and the identification information of the multiple marker points in the first marker candidate area to obtain the first marker actual area of ​​the first image includes: determining the image distance cross ratio between the four marker points in the first marker candidate area according to the image distance between the multiple marker points in the first marker candidate area and the identification information of the multiple marker points in the first marker candidate area; determining the actual distance cross ratio between the four marker points in the marker according to the actual distance between the four marker points in the marker and the identification information of the four marker points in the marker, wherein the identification information of the four marker points in the marker is the same as the identification information of the four marker points in the first marker candidate area; in response to determining that the image distance cross ratio between the four marker points in the first marker candidate area is the same as the actual distance cross ratio between the four marker points in the marker, determining the first marker candidate area to be the first marker actual area.

11. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 10, characterized in that: The method also includes: in response to determining that the image distance cross-ratio between the four marker points in the first marker candidate area is different from the actual distance cross-ratio between the four marker points in the marker, adjusting the identification information of the four marker points in the first marker candidate area, and determining the adjusted image distance cross-ratio between the four marker points in the first marker candidate area according to the image distance between the four marker points in the first marker candidate area and the adjusted identification information of the four marker points in the first marker candidate area; determining the adjusted actual distance cross-ratio between the four marker points in the marker according to the actual distance between the four marker points in the marker and the adjusted identification information of the four marker points in the marker, wherein the adjusted identification information of the four marker points in the marker is the same as the adjusted identification information of the four marker points in the first marker candidate area; in response to determining that the adjusted image distance cross-ratio between the four marker points in the first marker candidate area is the same as the adjusted actual distance cross-ratio between the four marker points in the marker, determining that the first marker candidate area is the first marker actual area.

12. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 1, characterized in that: The method of determining the position and posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device includes: determining the coordinates of the first center of gravity point of the marker point in the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device; determining the coordinates of the second center of gravity point of the marker point in the local physical coordinate system of the marker according to the coordinates of the marker point in the marker in the local physical coordinate system of the marker; determining the covariance matrix of the marker point in the marker in the coordinate system of the binocular vision imaging device and the local physical coordinate system of the marker according to the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point, the coordinates of the marker point in the marker in the local physical coordinate system of the marker, and the coordinates of the second center of gravity point; determining the position information and posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the covariance matrix.

13. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 12, characterized in that: The method of determining the covariance matrix of the marker point in the coordinate system of the binocular vision imaging device and the local physical coordinate system of the marker based on the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device, the coordinates of the first center of gravity point, the coordinates of the marker point in the marker in the local physical coordinate system of the marker, and the coordinates of the second center of gravity point includes: determining a first vector from the coordinate point of the marker point in the marker in the coordinate system of the binocular vision imaging device to the first center of gravity point based on the coordinates of the marker point in the marker in the coordinate system of the binocular vision imaging device and the coordinates of the first center of gravity point; determining a second vector from the coordinate point of the marker point in the marker in the local physical coordinate system of the marker to the second center of gravity point based on the coordinates of the marker point in the marker in the local physical coordinate system of the marker and the coordinates of the second center of gravity point; and determining the covariance matrix based on the first vector and the second vector.

14. The method for tracking the position and posture of an endoscope adapter in a neurosurgery robot according to claim 12, characterized in that: The method of determining the position information and posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the covariance matrix includes: performing singular value decomposition on the covariance matrix to obtain the left singular matrix and the right singular matrix of the covariance matrix; determining the posture information of the marker in the coordinate system of the binocular vision imaging device according to the left singular matrix and the right singular matrix of the covariance matrix; determining the position information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the first center of gravity point, the coordinates of the second center of gravity point, and the posture information of the marker in the coordinate system of the binocular vision imaging device.

15. An electronic device, characterized in that: The device includes: one or more processors; and a computer-readable medium configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the posture tracking method of the endoscopic adapter in the neurosurgical robot as described in any one of claims 1 to 14.

16. A computer readable medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method for tracking the position and posture of an endoscope adapter in a neurosurgical robot as claimed in any one of claims 1 to 14 is implemented.

17. A posture tracking device for an endoscope adapter in a neurosurgery robot, characterized in that: Applied to an optical positioning tracker, the device includes: an acquisition module, used to acquire a first image and a second image of the endoscope adapter collected by a binocular visual imaging device of the optical positioning tracker, wherein the first image and the second image both contain a marker of the endoscope adapter, and the distances between any two marker points among a plurality of marker points included in the marker are not the same; an identification module, used to identify a marker area of ​​the first image according to pixel values ​​and pixel coordinates of pixel points in the first image to obtain a first marker candidate area of ​​the first image, and to identify a marker area of ​​the second image according to pixel values ​​and pixel coordinates of pixel points in the second image to obtain a second marker candidate area of ​​the second image; a screening module, used to identify a marker area of ​​the second image according to the first The first marker candidate area is screened for the marker area according to the image distance between the four marker points in the marker candidate area and the identification information of the four marker points in the first marker candidate area to obtain the first marker actual area of ​​the first image, and the second marker candidate area is screened for the marker area according to the image distance between the four marker points in the second marker candidate area and the identification information of the four marker points in the second marker candidate area to obtain the second marker actual area of ​​the second image; a posture tracking module is used to perform posture tracking on the endoscope adapter according to the coordinates of the marker points in the first marker actual area and the coordinates of the marker points corresponding to the marker points in the second marker actual area to obtain the posture information of the endoscope adapter; Among them, the posture tracking of the endoscope adapter is performed according to the coordinates of the marker points in the actual area of ​​the first marker and the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points to obtain the posture information of the endoscope adapter, including: determining the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker points in the actual area of ​​the first marker, the projection matrix of the left visual imaging device in the binocular vision imaging device that collects the first image, the coordinates of the marker points in the actual area of ​​the second marker corresponding to the marker points, and the projection matrix of the right visual imaging device in the binocular vision imaging device that collects the second image; determining the posture information of the marker in the coordinate system of the binocular vision imaging device according to the coordinates of the marker points in the marker in the coordinate system of the binocular vision imaging device, and determining that the posture information of the marker in the coordinate system of the binocular vision imaging device is the posture information of the endoscope adapter in the coordinate system of the binocular vision imaging device.

Images