Virtual superposition positioning patch position extraction method and AR medical glasses

Abstract

The invention provides a virtual superposition positioning patch position extraction method and AR medical glasses, and the method comprises the steps: obtaining an image of a positioning patch obtained by scanning a photosensitive layer through the AR medical glasses, carrying out the stripping of the image after noise reduction, and stripping the image of the positioning patch from a background; smoothing the zigzag edge of the positioning patch image to form a positioning patch contour; carrying out multiple times of fitting according to the outline of the positioning patch, and fitting an elliptic curve of the outline of the positioning patch; and taking the center point coordinate of the fitted elliptic curve as the position coordinate of the positioning patch. On the basis of elliptic curve fitting, the central position of the patch can be accurately calculated no matter where a surgeon stands and acquires and positions the patch at any angle. And the light spots of the positioning patch can be rapidly and accurately extracted by using binarization. The sawtooth-shaped edge of the light spot is smoothed, so that fitting of a closest light spot elliptic curve is facilitated. And the positioning patch is small in size and does not influence the surgical field of a doctor.

Description

Technical Field

[0001] The present application relates to the field of surgical navigation technology, and more specifically, to a method for extracting the position of a virtual overlay positioning patch and AR medical glasses. Background Art

[0002] The lung puncture positioning needle currently used in clinical practice is a percutaneous puncture positioning needle under CT guidance. When the puncture needle reaches the lesion, the needle tip releases the inner needle with a barb and fixes it to the lesion inside the lung, guiding the doctor to complete the lobectomy under the visual field of the thoracoscope.

[0003] A pulmonary nodule puncture positioning needle usually consists of the following main parts:

[0004] Needle body: The main body of the puncture needle, usually made of a hard material such as stainless steel. The length and diameter vary depending on the patient's body shape and the depth of the nodule.

[0005] Positioning catheter: This part ensures that the needle reaches the target location accurately. The catheter may have scales or markings to help the doctor determine the depth of the puncture.

[0006] Needle: This is the tip at the end of a needle, typically a sharp point, used to penetrate tissue and collect samples. Needle designs may vary depending on the intended use; common designs include cutting needles and vacuum needles.

[0007] Scale and depth indicator: The positioning needle is usually equipped with a scale or depth indicator to help the doctor judge the depth of the puncture and avoid puncturing too deep or too shallow.

[0008] Positioning systems: These include CT-guided, ultrasound-guided, or AR-assisted systems to help doctors accurately locate lung nodules. The systems may include positioning markers or guide devices to ensure needle accuracy.

[0009] In existing AR-assisted lung puncture surgeries, the model building module constructs a virtual model of lung tissue and the planned puncture path based on CT or ultrasound images. The AR device uses the recognition and positioning module to superimpose the virtual puncture path on the human body. A common positioning module is a bracket formed by multiple metal balls. The metal balls are scanned together with the human body by CT. The metal balls are spherical, such as Figure 4 When AR devices use cameras for recognition, the resulting image is circular regardless of the scanning angle, which can easily affect the doctor's operating field of view and has poor positioning accuracy. Summary of the Invention

[0010] The purpose of this application is to provide a virtual overlay positioning patch position extraction method and AR medical glasses to solve the above-mentioned problems existing in the prior art.

[0011] In a first aspect, the present application provides a method for extracting the position of a virtual superimposed positioning patch, where each positioning patch includes a circular photosensitive layer, and the method includes:

[0012] Obtain a pixel map containing multiple positioning patches obtained by scanning the photosensitive layer through AR glasses;

[0013] Perform preliminary noise reduction on the pixel image to remove the noise in the pixel image;

[0014] The denoised pixel image is stripped to separate the positioning patch image from the background;

[0015] Smoothing the jagged edges of the positioning patch image to form the positioning patch outline;

[0016] Perform multiple fittings based on the positioning patch contour to fit an elliptical curve of the positioning patch contour;

[0017] According to the transformation relationship between the pixel map of the AR glasses and the world coordinate system, the coordinates of the center point of the fitted elliptic curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system.

[0018] Furthermore, multiple fittings are performed based on the positioning patch contour, and the elliptical curve of the positioning patch contour is fitted, including:

[0019] Sampling the positioning patch contour and fitting a preliminary elliptical curve of the positioning patch contour;

[0020] Calculate the shortest distance d from each sampling point to the preliminary elliptic curve i , determine the shortest distance d i Is it greater than the set first threshold, remove the distance d i The sampling points that are greater than the first threshold are selected, and a second fitting is performed based on the remaining sampling points to fit an elliptic curve of the positioning patch outline.

[0021] Furthermore, stripping the positioning patch image from the background includes:

[0022] Traverse each pixel point in the pixel map of the positioning patch and compare its grayscale value with the set second threshold; if it is greater than or equal to the second threshold, assign it the maximum brightness value; otherwise, set it to the minimum brightness value; wherein, the second threshold is used to divide background pixels and non-background pixels.

[0023] Furthermore, the step of smoothing the jagged edges of the positioning patch image includes:

[0024] Obtaining the edge gradient distribution map of the patch image;

[0025] The standard deviation σ of the Gaussian kernel function is dynamically calculated based on the local curvature radius;

[0026] Construct a spatially adaptive anisotropic Gaussian filter, where the main axis direction is consistent with the edge tangent direction;

[0027] Implement multi-scale Gaussian convolution operations along the normal direction of the curve and perform weighted fusion processing across color channels simultaneously;

[0028] Output the positioning patch contour image with smooth edges.

[0029] Furthermore, the construction of a spatially adaptive anisotropic Gaussian filter includes:

[0030] Decompose the two-dimensional Gaussian kernel into the tangential component G_t and the normal component G_n; where the standard deviation of the normal component σ_n = 2σ_t, σ_t = σ basic standard deviation;

[0031] The kernel function is expressed as: G(x,y) = G_t(x')·G_n(y'); where x' and y' are the projections of the point (x, y) in the image along the tangent-normal coordinate system;

[0032] Based on the tangential component G_t, the normal component G_n and the kernel function, an anisotropic Gaussian filter is determined.

[0033] Furthermore, a first fitting is performed based on the entire contour of the positioning patch, and a preliminary elliptical curve of the positioning patch contour is fitted, including:

[0034] Obtain a discrete pixel point set of the positioning patch contour to be fitted;

[0035] The parametric equation of an ellipse is a general quadratic curve: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2 -4AC<0, A, B, C, D, E, and F are elliptic curve coefficients,

[0036] The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion;

[0037]

[0038] Among them, (x i, y i ) represents the coordinates of the i-th discrete pixel in the pixel map, N is the total number of sampled pixels, and the standard least squares method is used to fit the ellipse and solve the parameters A, B, C, D, E, and F.

[0039] Furthermore, the distance d from the sampling point to the elliptic curve i The judgment threshold is 1 / 10 to 1 / 20 of the positioning patch radius.

[0040] Furthermore, the distance d from the sampling point (xi,yi) to the elliptic curve i The calculation method is as follows:

[0041]

[0042] x e =a cosθ、y e =b sinθ

[0043] The objective function is:

[0044] F(θ)=(x i -a cosθ) 2 +(y i -b sinθ) 2

[0045] The optimal parameter θ is solved by minimizing the objective function F(θ), where a and b are the semi-major axis and semi-minor axis lengths of the preliminary elliptic curve in the pixel image, respectively. e ,y e ) represents the point on the preliminary elliptic curve that is closest to the sampling pixel point (xi,yi).

[0046] Furthermore, in the second fitting process of the elliptic curve, the weight w is assigned according to the distance between the remaining sampling pixels and the preliminary elliptic curve. i , weight coefficient w i =1 / (1+αd i 2 ), α is the robustness adjustment factor, and its value range is α∈[0.1,1.0];

[0047] Assume that the objective function of the second fitting is:

[0048]

[0049] B 2 If -4AC<0, use the standard least squares method to fit the ellipse and solve the parameters A, B, C, D, E, and F again. Calculate the center position of the elliptic curve in the pixel image based on the values of A, B, C, D, E, and F.

[0050] In the second aspect, the present application provides an AR medical glasses, which includes a visual imaging module, a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store computer programs; and the processor is used to implement the above-mentioned virtual overlay positioning patch position extraction method when executing the program stored in the memory.

[0051] This application has the following beneficial effects:

[0052] This application uses an algorithm to filter out interference from indoor lighting and reflections from surgical instruments, thereby accurately extracting the center position of the positioning patch. Based on elliptical curve fitting, the center position of the patch can be accurately calculated regardless of where the surgeon stands or at what angle the positioning patch is collected. Binarization can be used to quickly and accurately extract the light spot of the positioning patch. In addition, the jagged edges of the light spot are smoothed, which helps to fit the closest elliptical curve of the light spot. The positioning patch is small in size and does not affect the doctor's surgical field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A system architecture for an application scenario of a positioning patch position extraction method provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a method for extracting a positioning patch position provided in an embodiment of the present application;

[0056] Figure 3 A schematic structural diagram of a positioning patch provided in an embodiment of the present application;

[0057] Figure 4 This is a structural diagram of a positioning module provided in the background technology. DETAILED DESCRIPTION

[0058] The following, in conjunction with the accompanying drawings, provides a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the described embodiments represent only a portion of the embodiments of this application and do not constitute a complete set of embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are intended to fall within the scope of protection of this application. Unless otherwise defined, technical or scientific terms used in this application should have the same ordinary meanings as those understood by persons of ordinary skill in the art. The terms "first," "second," and similar expressions used in this application do not denote any order, quantity, or importance; they are merely used to distinguish between different components. Terms such as "include" or "comprising" mean that the element or object preceding the term includes the elements or objects listed after the term, and their equivalents, without excluding other elements or objects. Terms such as "connect," "couple," or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used solely to indicate relative positional relationships. When the absolute position of the described objects changes, the relative positional relationships may also change accordingly.

[0059] The method for extracting the position of a puncture surgery positioning patch provided in the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1As shown, the system may include: medical imaging equipment, an image processing server, a PACS local area network, a puncture device, and AR medical glasses. The image processing server can be a physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. A PACS (Picture Archiving and Communication Systems) local area network refers to a dedicated network used in a medical environment to store, retrieve, manage, and share medical images. The PACS local area network enables hospitals or medical institutions to efficiently process large amounts of medical imaging data, such as X-rays, CT scans, and MRIs. The image processing server converts CT or MR images into three-dimensional models and sends them to the AR medical glasses via the PACS local area network. The medical imaging equipment may be a computed tomography (CT) scanner, which uses X-rays and complex computer processing to generate cross-sectional images of the body. CT scans can provide more detailed images than standard X-rays. Magnetic resonance imaging (MRI): Uses powerful magnetic fields and radio waves to create detailed images of internal organs and structures. Ultrasound imaging equipment: Uses high-frequency sound waves to produce real-time images of internal structures. Positron emission tomography (PET): A nuclear medicine imaging technique that generates three-dimensional images by detecting the distribution of radioactive substances in the body. Before a medical imaging device performs a scan, a positioning patch is placed on the patient's skin surface where the lesion is located. The CT scan also includes the positioning patch. The CT or MRI medical image is processed by an image processing server and converted into a three-dimensional model of the organ, lesion, or other area. AR glasses receive the 3D model and the planned puncture path from the server. The doctor, wearing AR glasses, uses the onboard camera to scan and identify the positioning patch. Based on the spatial position of the positioning patch, the doctor calculates the actual position of the AR glasses relative to these markers. With accurate spatial information, the system can precisely overlay pre-designed virtual images (such as anatomical structures and surgical paths) onto the corresponding locations in the real scene. For example, during surgery, it can help doctors see the specific location of blood vessels or organs hidden under the skin, improving the accuracy of the surgery.

[0060] Positioning patch module, used for identification of optical and medical imaging equipment, uses flexible silicone pads to connect five positioning patches together, and the silicone pad is connected to the patient's body. The bottom of the silicone pad is connected to the patient's body with medical grade tape. The specific structure is as follows Figure 2As shown, the positioning patch module includes a base layer 1 with a groove for positioning patch 12. The base layer is annular with a hollow area 13 in the middle for the puncture needle to pass through. From bottom to top, the positioning patch 12 consists of a metal layer, a photosensitive layer, and a filter membrane. The photosensitive layer is used for identification by the smart head-mounted display device and is generally made of a material with good light reflectivity. The metal layer is used for identification by the CT device. The CT scan captures the metal layer and locates a specific location on the patient's body. When 3D modeling is performed based on the CT data, these positioning marks are also included on the 3D model. The smart head-mounted display device can detect the location of the positioning patch during scanning and then overlay the 3D model on the patient's body for matching. The filter membrane can filter visible light. During surgery, the positioning patch is exposed to the shadowless lamp. The filter membrane filters the filtered light to prevent reflection from the shadowless lamp light, thereby preventing it from affecting the doctor's vision and operation, and also preventing interference with the smart head-mounted display camera. In this embodiment of the application, there are five positioning patches, distributed on two parallel lines. The positioning patches are circular in shape, which facilitates the identification of the center point.

[0061] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0062] Figure 3 A method for extracting a positioning patch position provided in an embodiment of the present application may include the following steps:

[0063] Step S1: Perform preliminary noise reduction on the image captured by the AR glasses to remove image noise. Denoising can be performed using methods such as mean filtering, median filtering, bilateral filtering, wavelet transform, adaptive filtering, and deep learning-based methods. Mean filtering is a simple linear filtering method that replaces the value of a pixel by taking the average of the values of its neighborhood. This method effectively removes Gaussian noise. Median filtering is very effective for randomly appearing black and white dots in an image. It replaces the central pixel value by calculating the median of all pixel values in its neighborhood, effectively removing noise while preserving edge information. Bilateral filtering combines spatial proximity and pixel value differences to perform a weighted average, preserving edges while reducing noise. It is suitable for situations where edge preservation is critical. Wavelet transform decomposes the image into detail and approximate components at different scales, then thresholds the detail components to remove noise, and finally reconstructs the image. Adaptive filtering adjusts filter parameters based on local image characteristics. For example, adaptive Wiener filtering dynamically adjusts the filter strength based on the information surrounding each pixel to achieve better denoising while preserving image features. Convolutional neural network (CNN)-based methods have also been applied to image denoising. These methods are able to learn complex noise patterns and effectively restore clear images from noisy images.

[0064] Step S2: Separate the image of the positioning patch from the background. Because there is a photosensitive layer in the patch, it will show a more obvious light spot shape under the influence of the operating room light. Therefore, binary separation is preferred. The specific separation method is as follows:

[0065] Grayscale conversion: The positioning patch image needs to be converted into a grayscale image. This is because color information is not important for detecting bright spots, and grayscale images can simplify subsequent processing.

[0066] Contrast adjustment: If the original image has low contrast, you can enhance the contrast through methods such as histogram equalization or adaptive histogram equalization to make the light spot more obvious.

[0067] Choosing an appropriate threshold: This is a key step in the binarization process. To choose an appropriate threshold T, the pixels are divided into two groups: one group is pixels with a brightness greater than T (considered to be part of the light spot), and the other group is pixels with a brightness less than or equal to T (considered to be the background). You can choose a global threshold or a local (adaptive) threshold. The global threshold is suitable for cases where the background is uniform. You can set a threshold manually or use an algorithm to automatically calculate it, such as the Otsu method. The local threshold / adaptive threshold is suitable for cases where the background is uneven. It dynamically adjusts the threshold based on the local information around each pixel.

[0068] Binarization: Apply a selected threshold to the image and convert all pixel values to 0 or 1 (or 0 and 255 in the case of an 8-bit grayscale image), forming a binary image. In this image, the foreground (i.e., the light spot) is usually white (1 or 255) and the background is black (0).

[0069] Morphological operations: To improve the quality of binary images, you can perform several morphological operations, such as erosion to remove small noise points, dilation to fill small holes within foreground objects or to connect adjacent objects, opening (erosion followed by dilation) to smooth object boundaries and break narrow connections, and closing (dilation followed by erosion) to help fill small holes within objects.

[0070] Connected domain analysis: Connected domain analysis (also known as component labeling) can identify and label independent regions in binary images. Then, the true spot area can be screened based on characteristics such as area and shape, and small areas that do not meet the requirements or misjudged areas can be eliminated.

[0071] Extracting light spots: The final step is to extract the light spot region of interest from the processed image. This may involve further image processing steps such as contour detection and bounding box drawing to facilitate subsequent analysis or display.

[0072] Step S3: Smoothing the jagged edges of the positioning patch image by Gaussian filtering to form the positioning patch outline. The specific steps are as follows:

[0073] S31: Obtaining an edge gradient distribution map of the positioning patch image;

[0074] S32: Dynamically calculate the standard deviation σ of the Gaussian kernel function based on the local curvature radius. The calculation formula of the standard deviation σ is:

[0075]

[0076] Where κ is the local curvature value, k∈[0.5,1.5] is the adjustment coefficient, and ε=0.01 is the anti-zero division constant.

[0077] S33: Construct a spatially adaptive anisotropic Gaussian filter whose main axis direction is consistent with the edge tangent direction; the construction method of the anisotropic Gaussian filter is as follows: decompose the two-dimensional Gaussian kernel into a tangential component G_t and a normal component G_n, where the standard deviation of the normal component σ_n = 2σ_t, σ_t = σ is the basic standard deviation, and the kernel function is expressed as: G(x, y) = G_t(x')·G_n(y'), x' and y' are the projections of the point (x, y) in the image along the tangent-normal coordinate system.

[0078] S34: Implement multi-scale Gaussian convolution operations along the normal direction of the curve, and synchronously perform weighted fusion processing across color channels. The multi-scale Gaussian convolution operation includes parallel implementation of filtering processing at three scales of σ = 0.8, 1.2, and 1.6. The final result is synthesized through the Laplacian pyramid fusion algorithm, where the fusion weight is positively correlated with the edge sharpness response value in the scale space.

[0079] After completing the above steps, the output curve can be verified. The edge oscillation index (EI) of the smoothed image can be calculated. When EI exceeds the preset threshold, the σ value is automatically increased and the filtering is re-executed until EI ≤ 0.1 × initial value and the structural similarity (SSIM) ≥ 0.98.

[0080] Finally, the positioning patch contour image with smooth edges is output.

[0081] Step S4: Fitting an elliptical curve. Although the positioning patch is circular, due to angle issues, the shape of the positioning patch captured by the AR glasses is actually closer to an ellipse. Therefore, using an elliptical curve to simulate the positioning patch shape is more accurate than a circular curve. Therefore, in this application, an elliptical curve is fitted. The following is a specific method for fitting an elliptical curve.

[0082] S41: Perform the first fitting based on the positioning patch contour to preliminarily fit the elliptical curve of the positioning patch contour. The specific fitting process is as follows:

[0083] Obtain a discrete pixel point set of the positioning patch contour to be fitted;

[0084] The parametric equation of an ellipse is a general quadratic curve: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2 -4AC<0, A, B, C, D, E, and F are elliptic curve coefficients

[0085] The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion;

[0086]

[0087] Among them, (x i, y i ) represents the coordinates of the i-th discrete pixel in the pixel map, N is the total number of sampled pixels, and the standard least squares method is used to fit the ellipse and solve the parameters A, B, C, D, E, and F.

[0088] S42: During the image acquisition process of the positioning patch, the influence of indoor lighting or reflections from surgical instruments will affect the imaging of the positioning patch's light spot, and may form bumps or defects at the edge of the light spot. The bumps or defects have a significant interference with the fitting of the elliptical curve, thereby affecting the center point of the elliptical curve and ultimately affecting the positioning accuracy of the positioning patch. Therefore, these bumps or defects need to be removed before performing a second fitting of the elliptical curve. The method for removing bumps or defects is as follows:

[0089] Sample the patch contour and calculate the shortest distance d from each sampling point to the elliptic curve i , determine the shortest distance d i Is it greater than the set threshold? The threshold can be set to 1 / 10 to 1 / 20 of the positioning patch radius, and its size is set according to the accuracy requirement. The calculation of the shortest distance requires finding the projection point (x e ,y e ), which reflects the real spatial relationship between the point and the ellipse. The shortest distance d from the sampling point to the elliptic curve i The calculation method is as follows:

[0090]

[0091] Parameterize the ellipse as:

[0092] x e =a cosθ、y e =b sinθ

[0093] Among them, a and b are the semi-major axis and semi-minor axis lengths of the preliminary elliptic curve in the pixel image, respectively, and the optimal parameter θ is solved by minimizing the objective function F(θ).

[0094] F(θ)=(x i -a cosθ) 2 +(y i -b sinθ) 2

[0095] The optimal parameter θ is solved by minimizing the objective function F(θ).

[0096] S43: Remove distance d i For sampling points greater than the threshold, a second fitting is performed based on the remaining sampling points to fit the elliptical curve of the positioning patch contour.

[0097] In the second fitting process of the elliptic curve, the weight w is assigned according to the distance between the remaining sampling pixels and the preliminary elliptic curve. i , weight coefficient w i =1 / (1+αd i 2), α is the robustness adjustment factor, and its value range is α∈[0.1,1.0];

[0098] Then, the objective function of the second fitting is:

[0099]

[0100] B 2 If -4AC<0, use the standard least squares method to fit the ellipse and solve the parameters A, B, C, D, E, and F again. Calculate the center position of the elliptic curve in the pixel image based on the values of A, B, C, D, E, and F.

[0101] The accuracy of the ellipse parametric equation is verified by error evaluation indicators, including:

[0102] Average algebraic error ε_alg=(Σw_i(Ax i 2 +Bx i y i +Cy i 2 +Dx i +Ey i +F) 2 ) / n;

[0103] Maximum geometric error ε_geo=max{|d i -(a+b) / 2|}, where a and b are the major and minor axes of the ellipse;

[0104] Goodness-of-fit indices y i i-coordinate mean.

[0105] Finally, based on the transformation relationship between the AR glasses' pixel map and the world coordinate system, the coordinates of the center point of the fitted elliptical curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system. The world coordinate system here describes the position of an object in real three-dimensional space, usually with a fixed reference point (such as the base of the robotic arm, the origin of the calibration plate, the location of the AR glasses) as the origin, and the coordinate axis direction is set according to the application scenario.

[0106] The present method for extracting the positioning patch position uses an algorithm to filter out interference from indoor lighting and reflections from surgical instruments, thereby accurately extracting the center of the positioning patch. Furthermore, based on elliptical curve fitting, the center position of the patch can be accurately calculated regardless of the surgeon's position or the angle at which the positioning patch is captured. Binarization allows for rapid and accurate extraction of the positioning patch's light spot. Furthermore, the jagged edges of the light spot are smoothed, facilitating the fitting of the closest elliptical curve for the light spot.

[0107] Corresponding to the above method, an embodiment of the present application also provides AR medical glasses, which include a visual imaging module, a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus, and the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store computer programs; and when the processor is used to execute the program stored in the memory, the above positioning patch position extraction method is implemented.

[0108] The functions of each functional unit of the AR medical glasses provided in the above embodiments of the present application can be achieved through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the AR medical glasses provided in the embodiments of the present application will not be repeated here.

[0109] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0110] The communication interface is used for communication between the above electronic device and other devices.

[0111] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0112] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0113] In another embodiment provided by the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the methods described in the above embodiments.

[0114] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the above embodiments.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, AR medical glasses, or computer program products. Therefore, the embodiments of the present application may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, AR medical glasses, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1A computer program for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The computer program may be executed entirely on the target object computing device, partially on the target object device, as a standalone software package, partially on the target object computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the target object computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0120] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0122] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0123] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A virtual superposition positioning patch position extraction method, characterized in that: Each positioning patch includes a circular photosensitive layer, and the method includes: Obtain a pixel map containing multiple positioning patches obtained by scanning the photosensitive layer through AR glasses; Perform preliminary noise reduction on the pixel image to remove the noise in the pixel image; The denoised pixel image is stripped to separate the positioning patch image from the background; Smoothing the jagged edges of the positioning patch image to form the positioning patch outline; Perform multiple fittings based on the positioning patch contour to fit an elliptical curve of the positioning patch contour; According to the transformation relationship between the pixel map of the AR glasses and the world coordinate system, the coordinates of the center point of the fitted elliptic curve in the pixel map are converted into the coordinates of the positioning patch in the world coordinate system.

2. The method for extracting the position of a virtual superposition positioning patch according to claim 1, characterized in that: Multiple fittings are performed based on the positioning patch contour to obtain the elliptical curves that fit the positioning patch contour, including: Sampling the positioning patch contour and fitting a preliminary elliptical curve of the positioning patch contour; Calculate the shortest distance d from each sampling point to the preliminary elliptic curve i , determine the shortest distance d i Is it greater than the set first threshold, remove the distance d i The sampling points that are greater than the first threshold are selected, and a second fitting is performed based on the remaining sampling points to fit an elliptic curve of the positioning patch outline.

3. The method for extracting the position of a virtual superposition positioning patch according to claim 1, wherein: Separating the positioning patch image from the background includes: Traverse each pixel point in the pixel map of the positioning patch and compare its grayscale value with the set second threshold; if it is greater than or equal to the second threshold, assign it the maximum brightness value; otherwise, set it to the minimum brightness value; wherein, the second threshold is used to divide background pixels and non-background pixels.

4. The method for extracting the position of a virtual superposition positioning patch according to claim 3, wherein: The step of smoothing the jagged edges of the positioning patch image comprises: Obtaining the edge gradient distribution map of the patch image; The standard deviation σ of the Gaussian kernel function is dynamically calculated based on the local curvature radius; Construct a spatially adaptive anisotropic Gaussian filter, where the main axis direction is consistent with the edge tangent direction; Implement multi-scale Gaussian convolution operations along the normal direction of the curve and perform weighted fusion processing across color channels simultaneously; Output the positioning patch contour image with smooth edges.

5. The method for extracting the position of a virtual superposition positioning patch according to claim 4, wherein: The method of constructing a spatially adaptive anisotropic Gaussian filter comprises: Decompose the two-dimensional Gaussian kernel into the tangential component G_t and the normal component G_n; where the standard deviation of the normal component σ_n = 2σ_t, σ_t = σ basic standard deviation; The kernel function is expressed as: G(x,y) = G_t(x')·G_n(y'); where x' and y' are the projections of the point (x, y) in the image along the tangent-normal coordinate system; Based on the tangential component G_t, the normal component G_n and the kernel function, an anisotropic Gaussian filter is determined.

6. The method for extracting the position of a virtual superposition positioning patch according to claim 2, wherein: The first fitting is performed based on the entire contour of the positioning patch. The preliminary elliptical curve of the positioning patch contour is fitted, including: Obtain a discrete pixel point set of the positioning patch contour to be fitted; The parametric equation of an ellipse is a general quadratic curve: Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0, where B 2 -4AC<0, A, B, C, D, E, and F are elliptic curve coefficients, The ellipse fitting objective function is constructed based on the algebraic distance minimization criterion; Among them, (x i, y i ) represents the coordinates of the i-th discrete pixel in the pixel map, N is the total number of sampled pixels, and the standard least squares method is used to fit the ellipse and solve the parameters A, B, C, D, E, and F.

7. The method for extracting the position of a virtual superposition positioning patch according to claim 6, wherein: The distance d from the sampling point to the elliptic curve i The judgment threshold is 1 / 10 to 1 / 20 of the positioning patch radius.

8. The method for extracting the position of a virtual superposition positioning patch according to claim 7, wherein: The distance d from the sampling point (xi,yi) to the elliptic curve i The calculation method is as follows: x e =acoaθ、y e =bsinθ The objective function is: F(θ)=(x i -a cosθ) 2 +(y i -b sinθ) 2 The optimal parameter θ is solved by minimizing the objective function F(θ), where a and b are the semi-major axis and semi-minor axis lengths of the preliminary elliptic curve in the pixel image, respectively. e ,y e ) represents the point on the preliminary elliptic curve that is closest to the sampling pixel point (xi,yi).

9. The method for extracting the position of a virtual superposition positioning patch according to claim 8, wherein: In the second fitting process of the elliptic curve, the weight w is assigned according to the distance between the remaining sampling pixels and the preliminary elliptic curve. i , weight coefficient w i =1 / (1+αd i 2 ), α is the robustness adjustment factor, and its value range is α∈[0.1,1.0]; Assume that the objective function of the second fitting is: B 2 If -4AC<0, use the standard least squares method to fit the ellipse and solve the parameters A, B, C, D, E, and F again. Calculate the center position of the elliptic curve in the pixel image based on the values of A, B, C, D, E, and F.

10. AR medical glasses, characterized in that: The AR medical glasses include a visual imaging module, a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the visual imaging module is used to capture the positioning patch image and send it to the processor; the memory is used to store computer programs; when the processor is used to execute the computer program stored on the memory, it implements the virtual superimposed positioning patch position extraction method as described in any one of claims 1-9.

Images