Positioning patch anti-interference identification method, AR medical glasses and storage medium

Abstract

The invention provides a positioning patch anti-interference identification method, AR medical glasses and a storage medium. Comprising the steps of positioning a patch array, obtaining an initial image through AR glasses, and providing a highlight area; the positioning patches are divided into abnormal patches and reference patches according to whether the grey-scale map of the positioning patches contains the highlight area, and the abnormal patches refer to the positioning patches with the grey-scale map containing the highlight area; stripping the highlight area from the grey-scale map; extracting an incomplete contour of the abnormal patch and a complete contour of the reference patch, fitting elliptic curves according to the extracted contours, and performing elliptic curve fitting on the abnormal patch by randomly sampling a plurality of points on the incomplete contour; performing approximation degree evaluation on the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch until the approximation degree reaches a set value; and finally, converting the coordinate of the central point into the coordinate of the positioning patch in a world coordinate system. And environmental interference such as operation shadowless lamp reflection and instrument reflection is effectively eliminated.

Description

Technical Field

[0001] The present application relates to the field of surgical navigation technology, and more specifically, to a positioning patch anti-interference recognition method, AR medical glasses, and a storage medium. Background Art

[0002] In the percutaneous puncture positioning technology guided by CT images, after the puncture needle reaches the lung lesion area, the barb structure deployed at the end of the inner needle is used to achieve mechanical anchoring with the target tissue, providing a stable positioning reference for subsequent thoracoscopic lobectomy.

[0003] The positioning system integrates real-time navigation of CT images and augmented reality (AR)-assisted technology, accurately locking the coordinates of the lesion through a visual interface. The spatial registration algorithm of CT images and metal markers can effectively improve the three-dimensional positioning accuracy, while the AR system uses optical projection to achieve real-time superposition of virtual anatomical structures and real surgical fields.

[0004] The common positioning patch array is a bracket formed by multiple metal balls. The metal balls are spherical together with the human body when scanned by CT. When the AR device is recognized by the camera, the image formed is circular no matter from which angle it is scanned. The use of sheet-like positioning patches can avoid affecting the doctor's operating field of view. Under strong ambient light interference (such as shadowless lamps), surgical instruments with smooth surfaces will reflect onto the positioning patch, forming a strong light spot during imaging, resulting in blurred optical features of the photosensitive layer, reducing edge recognition accuracy, and causing positioning data drift. The anti-interference methods of the existing technology mostly use filter membranes to accurately filter visible light, but the filter membrane will cause a reduction in light intensity under certain circumstances. Summary of the Invention

[0005] The purpose of this application is to provide a virtual overlay positioning patch position extraction method and AR medical glasses to solve the above-mentioned problems existing in the prior art.

[0006] In a first aspect, a positioning patch anti-interference identification method is provided, comprising a positioning patch array, wherein each positioning patch has the same shape, is circular, and includes a photosensitive layer, the method comprising:

[0007] S1. Obtain an initial image containing each positioning patch by scanning the photosensitive layer through AR glasses;

[0008] S2. Convert the initial image into a grayscale image, segment the grayscale image according to the set brightness threshold, and extract the highlight area;

[0009] S3. According to whether the grayscale image of the positioning patch contains a highlight area, the positioning patch is divided into an abnormal patch and a reference patch, wherein the abnormal patch refers to a positioning patch containing a highlight area in the grayscale image;

[0010] S4. Stripping the highlight area from the grayscale image and the background of the abnormal patch; stripping the background of the reference patch grayscale image;

[0011] S5. Extracting the incomplete outline of the abnormal patch and the complete outline of the reference patch, and fitting elliptic curves according to the extracted outlines, respectively, by randomly sampling several points on the incomplete outline of the abnormal patch for elliptic curve fitting;

[0012] S6. Evaluate the approximation between the elliptic curve fitted to the abnormal patch and the elliptic curve fitted to the reference patch; for abnormal patches whose evaluation value is less than the set value, randomly sample several points on the incomplete contour for elliptic curve fitting again until the approximation reaches the set value;

[0013] S7. According to the transformation relationship between the pixel map of the AR glasses and the world coordinate system, the coordinates of the center point of the fitted elliptic curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system.

[0014] Furthermore, in S6, the evaluation index includes at least two of the major semi-axis a, the minor semi-axis b, the rotation angle θ, and the area of the elliptic curve.

[0015] As a preferred embodiment of the above technical solution, the shape similarity S 形状 The approximation between the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch is evaluated, S 形状 =w1×S 比例 +w2×S 偏心率 , S 比例 Indicates the similarity of axis length ratio, S 偏心率 represents the eccentricity consistency, w1 and w2 represent weights, w1+w2=1;

[0016]

[0017] Where a1, a2, b1, and b2 represent the major and minor axes of the two ellipses respectively.

[0018] As a preferred embodiment of the above technical solution, performing approximation evaluation on the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch includes:

[0019] translating the center of the elliptic curve of the abnormal patch to the center of the elliptic curve of the reference patch;

[0020] rotating the elliptic curve of the anomaly patch so that its major axis is aligned with the major axis of the elliptic curve of the reference patch;

[0021] scaling the axis length of the elliptic curve of the elliptical anomaly patch so that its major semi-axis is consistent with the major semi-axis of the elliptic curve of the reference patch;

[0022] Compute the minor semi-axis b of the elliptic curve of the abnormal patch after residual comparison transformation 2′ The difference between the semi-minor axis b1 of the elliptic curve and the reference patch is:

[0023]

[0024] S 几何 Represents the geometric similarity between two elliptic curves.

[0025] Furthermore, the background of the reference patch grayscale image is stripped including:

[0026] Traverse each pixel point in the image of the positioning patch and compare its grayscale value with the set second threshold; if it is greater than or equal to the second threshold, assign it the maximum brightness value; otherwise, set it to the minimum brightness value; wherein, the second threshold is used to divide background pixels and non-background pixels.

[0027] Furthermore, the elliptic curve fitting of the reference patch includes:

[0028] Obtaining a discrete point set of the contour of the reference patch to be fitted;

[0029] The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion;

[0030] Obtain a discrete point set of the contour of the positioning patch to be fitted;

[0031] The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion;

[0032] The ellipse parameter solution process is optimized by introducing a dynamic weight coefficient, where the weight coefficient w i =1 / (1+αd i 2 );d i is the radial distance from the discrete point set to the current ellipse estimate; α is the robustness adjustment factor, and its value range is α∈[0.1,1.0];

[0033] Output the ellipse parametric equation that meets the preset fitting accuracy.

[0034] Furthermore, the accuracy of the ellipse parametric equation is verified by the error evaluation index. Let the discrete point set of the curve be {P i (x i ,y i ), i=1,2,...,n}, the ellipse parametric equation is a general quadratic curve form: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2 -4AC<0, A, B, C, D, E, and F are elliptic curve coefficients;

[0035] Error evaluation indicators include:

[0036] Average algebraic error: ε_alg=(Σw i (Ax i 2 +Bx i y i +Cy i 2 +Dx i +Ey i +F) 2 ) / n; where (x i ,y i ) is the coordinate of the i-th sampling point;

[0037] Maximum geometric error: ε_geo=max{|d i -(a+b) / 2|}; where a and b are the major and minor axes of the ellipse;

[0038] Goodness of fit: in, y i Coordinate mean.

[0039] In a second aspect, AR medical glasses are provided, which include a visual imaging module, a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store a computer program; and the processor is used to implement the above-mentioned positioning patch anti-interference recognition method when executing the computer program stored in the memory.

[0040] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned positioning patch anti-interference recognition method is implemented.

[0041] The present application has the following beneficial effects: through brightness threshold segmentation and highlight area stripping technology, it effectively eliminates environmental interference such as surgical shadowless lamp reflections and instrument reflections, and combines the background stripping algorithm to restore the true outline of the patch, so that the recognition accuracy of the positioning patch under strong light or surface pollution conditions is improved, and it is less dependent on the filter film. Using random sampling iterative fitting and shape-geometry dual similarity evaluation (S shape, S geometry), through multi-dimensional parameter verification such as axis length ratio, eccentricity, and short semi-axis residual, the ellipse fitting deviation caused by abnormal highlight light spots caused by reflections is solved, and the patch center positioning error is reduced. And based on elliptical curve fitting, no matter where the surgeon stands or at what angle the positioning patch is collected, the center position of the patch can be accurately calculated. The light spot of the positioning patch can be extracted quickly and accurately using binarization. It is helpful to fit the closest light spot elliptical curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A system architecture for an application scenario of a positioning patch anti-interference recognition method provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a positioning patch anti-interference recognition method provided in an embodiment of the present application;

[0045] Figure 3 A schematic structural diagram of a positioning patch array provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following, in conjunction with the accompanying drawings, provides a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the described embodiments represent only a portion of the embodiments of this application and do not constitute a complete set of embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are intended to fall within the scope of protection of this application. Unless otherwise defined, technical or scientific terms used in this application should have the same ordinary meanings as those understood by persons of ordinary skill in the art. The terms "first," "second," and similar expressions used in this application do not denote any order, quantity, or importance; they are merely used to distinguish between different components. Terms such as "include" or "comprising" mean that the element or object preceding the term includes the elements or objects listed after the term, and their equivalents, without excluding other elements or objects. Terms such as "connect," "couple," or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used solely to indicate relative positional relationships. When the absolute position of the described objects changes, the relative positional relationships may also change accordingly.

[0047] The positioning patch anti-interference recognition method provided in the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1 As shown, the system may include: medical imaging equipment, image processing server, PACS local area network, puncture device and AR medical glasses.

[0048] The implementation architecture of the image processing server is highly flexible. It can be built as an independent physical server entity or expanded into a server cluster or distributed computing architecture with multiple nodes working together. In the cloud solution, the server system can rely on cloud computing infrastructure to achieve service-oriented deployment, specifically covering cloud service resource pools, cloud database storage systems, elastic cloud computing computing units, and on-demand scheduled cloud function execution environments. Its technical support system integrates the cloud storage resource management system, network service communication protocol stack, cloud communication interaction interface and middleware service adaptation layer, as well as domain name resolution services, security protection systems and content distribution network (CDN) acceleration modules. At the advanced functional level, the system deeply integrates the big data processing platform and artificial intelligence algorithm framework to form a complete technical chain from basic resource scheduling to intelligent image analysis, providing full-stack cloud technology support for medical image processing.

[0049] The PACS (Picture Archiving and Communication System) local area network is a dedicated network architecture deployed in medical institutions. Its core function is to achieve centralized storage, intelligent retrieval, standardized management, and secure sharing of medical imaging data. By optimizing image transmission efficiency and data management processes, the system supports the efficient and collaborative application of massive medical imaging resources such as X-rays, CT scans, and magnetic resonance imaging (MRI) within hospitals. The image processing server performs three-dimensional reconstruction operations based on the original CT or MR image data, generates a three-dimensional digital model with spatial coordinate information, and then relies on the communication protocol of the PACS local area network to transmit the model data to the augmented reality (AR) medical glasses terminal, realizing the real-time fusion presentation of surgical navigation information and the actual surgical field.

[0050] The medical imaging technology ecosystem encompasses a variety of image acquisition devices: Computed tomography (CT) relies on X-ray beam tomography and computer reconstruction algorithms to generate high-resolution cross-sectional images of the human body, with tissue contrast significantly superior to traditional X-ray imaging. Magnetic resonance imaging (MRI) uses a strong magnetic field and radiofrequency pulse sequences to excite hydrogen nuclear resonance signals, constructing high-definition three-dimensional images of soft tissue structures. Ultrasound imaging equipment uses high-frequency sound wave reflection signals to dynamically display the movement of internal organs in real time. Positron emission tomography (PET) uses the metabolic distribution characteristics of radioactive tracers to generate functional three-dimensional molecular images. During image acquisition, the positioning patch array must be precisely aligned with the patient's lesion in advance, and the spatial coordinate data of the patch's metal layer is simultaneously acquired during the CT scan. The image processing server performs three-dimensional reconstruction operations on the raw CT or MR data, generating a digital model that includes the lesion's anatomical structure, vascular course, and adjacent relationships. The planned puncture path parameters are then transmitted via the medical network to augmented reality (AR) medical glasses. When the surgeon wears AR glasses, their built-in camera captures the optical markers of the surface positioning patch in real time and uses spatial geometry algorithms to analyze the 3D positional relationship between the glasses' optical coordinate system and the patch's reference coordinate system. Based on this spatial registration result, the system accurately overlays the pre-processed 3D virtual model (including anatomical structure projections, surgical path navigation lines, etc.) onto the actual surgical field, achieving enhanced visualization of key anatomical elements such as the course of subcutaneous blood vessels and the location of deep lesions, significantly improving the accuracy of puncture positioning and surgical operations.

[0051] The positioning patch array uses an elastic polymer substrate (Shore hardness 35A, thickness 0.8±0.1mm) to construct a flexible interconnected structure, and achieves mechanical coupling with human skin through acrylic pressure-sensitive adhesive. While maintaining the flatness of the patch group (error ≤ 0.15mm), it has the elastic deformation ability to adapt to the curvature of the body surface (≥50mm). Figure 2As shown, its structure consists of a base layer 1 forming the main frame. The base layer's surface is machined with an array of precision grooves for rigidly securing the positioning patch 12. A central, circular hollow area 13 provides an 8mm diameter operating channel for puncture instruments. The positioning patch 12 utilizes a multi-layer functional design. The bottom layer is a tungsten-nickel alloy metal layer (0.2mm thick), which leverages its high atomic number to produce a CT contrast ratio 1.5-2.0 times that of soft tissue in X-ray images. The surface photosensitive layer is composed of a titanium dioxide composite with a reflectivity of ≥85%, providing optical signature points with a high signal-to-noise ratio for the smart head-mounted display.

[0052] Five positioning patches are used to construct a double-row topology, forming a 2×2+1 asymmetric matrix along parallel axes with a 35mm spacing. This design utilizes the Laplace gradient symmetry of a circle to reduce the error in center coordinate extraction. A least-squares ellipse fitting algorithm is used to extract sub-pixel center coordinates and construct a spatial coordinate system transformation model, ultimately achieving high registration accuracy between the virtual model and the physical anatomical structure. During the 3D reconstruction process, the metal layer generates discrete marker points in the medical imaging data. A Delaunay triangulation algorithm is used to establish a mapping relationship with the optical markers of the photosensitive layer, completing the spatial synchronization of multimodal data.

[0053] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0054] Figure 3 A method for extracting a positioning patch position provided in an embodiment of the present application may include the following steps:

[0055] S1. Obtain an initial image containing each positioning patch by scanning the photosensitive layer using AR glasses. Specifically, the AR glasses' built-in high-resolution CMOS camera (resolution 1920×1080, frame rate 60fps) dynamically scans the patient's body surface positioning patch array to obtain a sequence of original images containing the photosensitive layers of the five positioning patches.

[0056] S2. Convert the initial image into a grayscale image, segment the grayscale image according to the set brightness threshold, and propose the highlight area. Specifically, first convert the RGB three-channel original image (resolution 1920×1080) collected by the AR glasses into a grayscale image, and use the Y channel weighted average method in the CIEXYZ color space (R: 0.299, G: 0.587, B: 0.114) for conversion. Grayscale processing can effectively reduce data dimensions and improve processing efficiency. After threshold segmentation, the 8-neighborhood connected domain labeling algorithm is applied to extract candidate highlight areas, and the highlight areas are screened by area threshold (retaining 300-800 pixel areas).

[0057] S3. Based on whether the grayscale image of the positioning patch contains a highlight area, the positioning patch is divided into abnormal patches and reference patches. The abnormal patch refers to the positioning patch containing a highlight area in the grayscale image. Specifically, based on the grayscale image features, the patches are divided into two categories: abnormal patches and reference patches: abnormal patches are defined as patches with highlight areas in the grayscale image (such as: pixel brightness value ≥ 220, area ≥ 500 pixels), and their spot shape usually shows edge overflow phenomenon; reference patches refer to patches whose grayscale value distribution meets the preset standard (such as: brightness range 180-210, spot area 300-500 pixels).

[0058] S4. Peel off the highlight area from the grayscale image, and peel off the background of the abnormal patch; peel off the background in the grayscale image of the reference patch. Traverse each pixel in the image of the positioning patch and compare its grayscale value with the set second threshold; if it is greater than or equal to the second threshold, assign the maximum brightness value; otherwise, set it to the minimum brightness value; wherein, the second threshold is used to divide background pixels and non-background pixels. In addition, the precise outline of the highlight area can be locked through edge detection, and then the core highlight area can be separated from the surrounding gradient halo according to the pixel brightness changes within the outline. The natural segmentation boundary can be found through gradient analysis to ensure that only the effective highlight part is removed. The peeled results can be verified and repaired. The entire process can also refer to the changes in multiple frames of images in real time, eliminate temporary interference through the continuity of the previous and next frames, and ensure the stability of the peeling results.

[0059] Background stripping uses a layered approach: first, Otsu global threshold segmentation (threshold T = 185) is applied to obtain a coarse binary image. A distance transform (Euclidean distance) is then used to generate a spatial weight map, which is then combined with GrabCut iterative optimization (three iterations) to precisely separate the patch outlines. Morphological inpainting is performed on the stripped foreground region: holes are filled using a 3×3 cross-shaped structuring element, followed by a thinning algorithm (Zhang-Suen parallel thinning) to extract single-pixel edges.

[0060] S5. Extract the incomplete outline of the abnormal patch and the complete outline of the reference patch, and fit an elliptic curve based on the extracted outlines. The abnormal patch is fitted with an elliptic curve by randomly sampling several points on the incomplete outline. The elliptic curve fitting method is as follows:

[0061] Obtain a discrete pixel point set of the positioning patch contour to be fitted;

[0062] The parametric equation of an ellipse is a general quadratic curve: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2-4AC<0, A, B, C, D, E, and F are elliptic curve coefficients.

[0063] The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion;

[0064]

[0065] Among them, (x i, y i ) represents the coordinates of the i-th discrete pixel in the pixel map, N is the total number of sampled pixels, and the standard least squares method is used to fit the ellipse and solve the parameters A, B, C, D, E, and F.

[0066] In order to further improve the fitting accuracy, the weight w can be assigned according to the distance between the remaining sampling pixels and the preliminary elliptic curve. i , fit again, weight coefficient w i =1 / (1+αd i 2 ), α is the robustness adjustment factor, and its value range is α∈[0.1,1.0];

[0067] Then, the objective function of the second fitting is:

[0068]

[0069] B 2 If -4AC<0, use the standard least squares method to fit the ellipse and solve the parameters A, B, C, D, E, and F again. Calculate the center position of the elliptic curve in the pixel image based on the values of A, B, C, D, E, and F.

[0070] The accuracy of the ellipse parametric equation is verified by error evaluation indicators, including:

[0071] Average algebraic error ε_alg=(Σw_i(Ax i 2 +Bx i y i +Cy i 2 +Dx i +Ey i +F) 2 ) / n;

[0072] Maximum geometric error ε_geo=max{|d i -(a+b) / 2|}, where a and b are the major and minor axes of the ellipse;

[0073] Goodness-of-fit indices y i i-coordinate mean.

[0074] S6. Evaluate the approximation between the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch; for abnormal patches whose evaluation value is less than the set value, randomly sample several points on the incomplete contour and perform elliptic curve fitting again until the approximation reaches the set value.

[0075] The approximation evaluation can be performed through multiple dimensions, and the evaluation indicators include at least two of the major semi-axis a, the minor semi-axis b, the rotation angle θ, and the area of the elliptic curve.

[0076] In the first embodiment of the present application, the shape similarity S 形状 The approximation between the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch is evaluated, S 形状 =w1×S 比例 +w2×S 偏心率 , S 比例 Indicates the similarity of axis length ratio, S 偏心率 represents the eccentricity consistency, w1 and w2 represent weights, w1+w2=1;

[0077]

[0078] Where a1, a2, b1, and b2 represent the major and minor axes of the two ellipses respectively.

[0079] If the reference patch's axial length ratio b1 / a1 = 1.05 (close to a circle), and the abnormal patch's axial length ratio b2 / a2 = 1.20, then:

[0080]

[0081] If the set value is 90%, the ellipse is not approximate and further fitting is required.

[0082] In a second embodiment of the present application, the similarity evaluation method includes:

[0083] The center of the abnormal patch ellipse (x2, y2) is translated to the center of the reference patch ellipse (x1, y1) to eliminate the interference of position offset on shape comparison. This operation ensures that the two ellipses are at the same coordinate system origin, and only the shape difference is retained. The mathematical expression is:

[0084]

[0085] Calculate the main axis direction angles θ1 (reference patch) and θ2 (abnormal patch) of the two ellipses, and rotate the abnormal patch ellipse around the center by Δθ = θ1-θ2 so that its main axis direction is consistent with the reference patch, eliminating the direction difference. The rotation matrix is:

[0086]

[0087] Scale the axis length of the elliptical curve of the elliptical anomaly patch so that its major semi-axis is consistent with the major semi-axis of the elliptical curve of the reference patch. This step eliminates the scale difference and makes the length of the major semi-axis of the two ellipses consistent;

[0088] Compute the residuals to compare the difference between the minor semi-axis b2′ of the elliptic curve of the abnormal patch after transformation and the minor semi-axis b1 of the elliptic curve of the reference patch:

[0089]

[0090] S 几何 It represents the geometric similarity between two elliptic curves. When b2′=b1, S 几何 =100%, indicating that the shapes are completely consistent; if b2′ deviates from b1, then S 几何 Linear decrease. Absolute value operation ensures symmetry and avoids the influence of positive and negative differences on the results.

[0091] When S 几何 ≥95%, the shape difference can be ignored and the two are judged to be similar.

[0092] Assume that the reference patch b1 = 10px, b2 = 9px, and after shrinking b2′ = 1210×9 = 7.5px, then:

[0093]

[0094] If it falls into the invalid range, the exception handling process needs to be started.

[0095] S7. According to the transformation relationship between the pixel map of AR glasses and the world coordinate system, the coordinates of the center point of the fitted elliptical curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system. In the augmented reality surgical navigation system, it is necessary to establish a multi-layer spatial mapping relationship to convert the center point of the elliptical positioning patch captured by the AR glasses camera from pixel coordinates to the world coordinate system. First, the image of the surgical area captured by the camera is presented in a two-dimensional pixel map, where the pixel coordinates (u, v) of the center point of the positioning patch need to be converted into a three-dimensional camera coordinate system through the intrinsic parameter matrix. The intrinsic parameter matrix contains the focal length (f_x, f_y) and optical center (c_x, c_y) parameters, which can eliminate the errors caused by lens distortion and restore the pixel points to the standard imaging plane.

[0096] Once the depth information Z_c of the positioning patch within the camera is known (obtained through binocular vision or a pre-set marker plane), the corresponding 3D camera coordinates (X_c, Y_c, Z_c) can be calculated through back projection. A coordinate system transformation is then required using an extrinsic matrix: the extrinsic matrix, consisting of a rotation matrix R and a translation vector T, reflects the spatial position of the camera in the world coordinate system. Through matrix operations, the center point of the ellipse can be mapped to a world coordinate system referenced to the operating table. This transformation requires the combined positioning of multiple positioning patches. For example, in thoracoscopic surgery, five elliptical patches are arranged in a regular pattern of two columns and three rows. When the coordinates of the center pixel of a patch are detected, the system verifies the coordinates by referencing the geometric relationship of adjacent patches. If the converted coordinates deviate from the preset topology, the algorithm automatically initiates multi-frame smoothing, using sliding window averaging to eliminate instantaneous jitter errors. The final output world coordinate accuracy reaches ±0.3mm, meeting the millimeter-level navigation requirements of surgical instruments.

[0097] Beneficial effects: Through brightness threshold segmentation and highlight area stripping technology, environmental interference such as surgical shadowless lamp reflections and instrument reflections can be effectively eliminated. Combined with the background stripping algorithm, the true outline of the patch is restored, so that the recognition accuracy of the positioning patch under strong light or surface pollution conditions is improved, and it is less dependent on the filter film. Using random sampling iterative fitting and shape-geometry dual similarity evaluation (S shape, S geometry), through multi-dimensional parameter verification such as axis length ratio, eccentricity, and short semi-axis residual, the ellipse fitting deviation caused by abnormal highlight spots caused by reflections is solved, and the patch center positioning error is reduced. It is adapted to multimodal data such as AR glasses, CT / MRI, and is compatible with heterogeneous positioning systems such as UWB and optical tracking through the standardized output of ellipse parameters (major semi-axis, short semi-axis, rotation angle), which significantly reduces the risk of positioning failure caused by abnormal light spots caused by reflections in thoracoscopic surgical navigation. And based on elliptical curve fitting, no matter where the surgeon stands or at what angle the positioning patch is collected, the center position of the patch can be accurately calculated. Binarization can be used to quickly and accurately extract the light spot of the positioning patch. It helps to fit the closest spot elliptical curve.

[0098] Corresponding to the above method, an embodiment of the present application also provides AR medical glasses, which include a visual imaging module, a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus, and the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store computer programs; and when the processor is used to execute the program stored in the memory, the above method is implemented.

[0099] The functions of each functional unit of the AR medical glasses provided in the above embodiments of the present application can be achieved through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the AR medical glasses provided in the embodiments of the present application will not be repeated here.

[0100] The communication bus mentioned above can be a Peripheral Component Interconnect bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0101] The communication interface is used for communication between the above electronic device and other devices.

[0102] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0103] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0104] In another embodiment provided by the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the methods described in the above embodiments.

[0105] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the above embodiments.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, AR medical glasses, or computer program products. Therefore, the embodiments of the present application may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, AR medical glasses, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1A computer program for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The computer program may be executed entirely on the target object computing device, partially on the target object device, as a standalone software package, partially on the target object computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the target object computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0111] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0112] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0113] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0114] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A positioning patch anti-interference identification method, characterized in that: The method comprises an array of positioning patches, each of which has the same shape, is circular, and includes a photosensitive layer. S1. Obtain an initial image containing each positioning patch by scanning the photosensitive layer through AR glasses; S2. Convert the initial image into a grayscale image, segment the grayscale image according to the set brightness threshold, and extract the highlight area; S3. According to whether the grayscale image of the positioning patch contains a highlight area, the positioning patch is divided into an abnormal patch and a reference patch, wherein the abnormal patch refers to a positioning patch containing a highlight area in the grayscale image; S4. Stripping the highlight area from the grayscale image and the background of the abnormal patch; stripping the background of the reference patch grayscale image; S5. Extract the incomplete outline of the abnormal patch and the complete outline of the reference patch, and fit the elliptic curve according to the extracted outlines. The abnormal patch is fitted with an elliptic curve by randomly sampling several points on the incomplete outline. S6. Evaluate the approximation between the elliptic curve fitted to the abnormal patch and the elliptic curve fitted to the reference patch; for abnormal patches whose evaluation value is less than the set value, randomly sample several points on the incomplete contour for elliptic curve fitting again until the approximation reaches the set value; S7. According to the transformation relationship between the pixel map of the AR glasses and the world coordinate system, the coordinates of the center point of the fitted elliptic curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system.

2. The positioning patch anti-interference identification method according to claim 1, characterized in that: In S6, the evaluation index includes at least two of the major semi-axis a, the minor semi-axis b, the rotation angle θ, and the area of the elliptic curve.

3. The positioning patch anti-interference identification method according to claim 2, characterized in that: By shape similarity S 形状 The approximation between the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch is evaluated, S 形状 =w1×S 比例 +w2×S 偏心率 , S 比例 Indicates the similarity of axis length ratio, S 偏心率 represents the eccentricity consistency, w1 and w2 represent weights, w1+w2=1; Where a1, a2, b1, and b2 represent the major and minor axes of the two ellipses respectively.

4. The positioning patch anti-interference identification method according to claim 2, characterized in that: The approximation evaluation of the elliptic curve fitted by the abnormal patch and the elliptic curve fitted by the reference patch includes: translating the center of the elliptic curve of the abnormal patch to the center of the elliptic curve of the reference patch; rotating the elliptic curve of the anomaly patch so that its major axis is aligned with the major axis of the elliptic curve of the reference patch; scaling the axis length of the elliptic curve of the elliptical anomaly patch so that its major semi-axis is consistent with the major semi-axis of the elliptic curve of the reference patch; Compute the minor semi-axis b of the elliptic curve of the abnormal patch after residual comparison transformation 2′ The difference between the semi-minor axis b1 of the elliptic curve and the reference patch is: S 几何 Represents the geometric similarity between two elliptic curves.

5. The positioning patch anti-interference identification method according to claim 1, characterized in that: The background in the stripped reference patch grayscale image includes: Traverse each pixel point in the image of the positioning patch and compare its grayscale value with the set second threshold; if it is greater than or equal to the second threshold, assign it the maximum brightness value; otherwise, set it to the minimum brightness value; wherein, the second threshold is used to divide background pixels and non-background pixels.

6. The positioning patch anti-interference identification method according to claim 2, wherein The elliptic curve fitting of the reference patch includes: Obtaining a discrete point set of the contour of the reference patch to be fitted; The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion; Obtain a discrete point set of the contour of the positioning patch to be fitted; The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion; The ellipse parameter solution process is optimized by introducing a dynamic weight coefficient, where the weight coefficient w i =1 / (1+αd i 2 );d i is the radial distance from the discrete point set to the current ellipse estimate; α is the robustness adjustment factor, and its value range is α∈[0.1,1.0]; Output the ellipse parametric equation that meets the preset fitting accuracy.

7. The method for extracting the position of a virtual superposition positioning patch according to claim 6, wherein: The accuracy of the ellipse parametric equation is verified by error evaluation index. Let the discrete point set of the curve be {P i (x i ,y i ), i=1,2,...,n}, the ellipse parametric equation is a general quadratic curve form: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2 -4AC<0, A, B, C, D, E, and F are elliptic curve coefficients; Error evaluation indicators include: Average algebraic error: ε_alg=(Σw i (Ax i 2 +Bx i y i +Cy i 2 +Dx i +Ey i +F) 2 ) / n; where (x i ,y i ) is the coordinate of the i-th sampling point; Maximum geometric error: ε_geo=max{|d i -(a+b) / 2|}; where a and b are the major and minor axes of the ellipse; Goodness of fit: in, y i Coordinate mean.

8. An AR medical glasses, characterized in that: The AR medical glasses include a visual imaging module, a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store computer programs; when the processor is used to execute the computer program stored on the memory, it implements the positioning patch anti-interference identification method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the positioning patch anti-interference identification method according to any one of claims 1 to 7 is implemented.

Images