Tumor margin design method based on local image of skin

Abstract

The application relates to the technical field of image processors, and discloses a tumor incisal edge design method based on a skin local image. The method comprises the following steps: establishing a global space coordinate system associated with the skin of a patient; collecting a local image and a six-degree-of-freedom pose by using a handheld device integrated with a pose tracking function; projecting and fusing the image to a digital surface model in real time to generate a panoramic image; segmenting a tumor boundary by using a deep learning model, and generating a safe incisal edge path by outward expansion based on a geometric algorithm; and finally projecting the incisal edge to the skin surface in the form of structured light by using a built-in projection unit of the device. The application realizes high-precision and low-delay dynamic planning and visualized guidance of the tumor incisal edge, and improves the precision and efficiency of the operation.

Description

Technical Field

[0001] This invention belongs to the field of image processor technology, and specifically relates to a method for designing tumor resection margins based on local skin images. Background Technology

[0002] With the increasing demands for precise resection in the surgical treatment of skin tumors, real-time acquisition of complete and high-precision panoramic images of the tumor area during surgery has become a key prerequisite for the design of surgical margins.

[0003] In clinical practice, for large or irregularly shaped skin tumors, doctors usually need to take multiple high-resolution images of the local area in succession and rely on image stitching technology to synthesize a panoramic image covering the entire lesion area to help determine the tumor boundary and plan the resection range.

[0004] This process places extremely high demands on the accuracy and real-time performance of image stitching, especially during surgery, where any delay or misalignment may affect the reliability of the cutting edge decision.

[0005] Tumor margin design methods based on local skin images focus on how to efficiently fuse multi-view, high-resolution local images to construct a distortion-free, seamless panoramic view. Its core objective is to reduce the time delay from image acquisition to panoramic output while ensuring geometric consistency and texture continuity, thereby providing surgeons with an immediate and complete macroscopic view to support dynamic and precise margin delineation.

[0006] Existing technologies typically employ a sequential processing flow, first acquiring all local images and then performing feature extraction, matching, registration, and fusion in sequence, resulting in a lengthy overall time consumption. This is especially true at high resolutions, where computational complexity increases dramatically, making it highly susceptible to feature mismatches or geometric correction deviations, leading to stitching misalignments, ghosting, or structural breaks. These defects are particularly critical in surgical settings, not only weakening the anatomical reliability of panoramic images but also potentially misleading the assessment of surgical margins, increasing the risk of residual or excessive resection.

[0007] Therefore, there is an urgent need for an image stitching architecture that can balance real-time performance and high precision to solve the technical problems of slow response and inaccurate stitching in existing methods in dynamic surgical environments. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a tumor resection margin design method based on local skin images, which aims to overcome the shortcomings of the prior art in constructing a panoramic tumor image by image stitching, such as long processing time, easy stitching misalignment at high resolution, and inability to meet the real-time requirements of surgery.

[0009] To address the aforementioned technical issues, this invention provides a tumor resection margin design method based on local skin images. This method no longer relies on traditional image feature matching-based stitching techniques. Instead, it constructs a global spatial coordinate system associated with the patient's skin surface and tracks the six-degree-of-freedom pose information of the imaging device in this coordinate system in real time. Each frame of the acquired local image is directly projected onto the dynamically updated global digital surface model based on its real-time spatial pose, thereby instantly generating a seamless and accurate panoramic digital image of the tumor region.

[0010] Subsequently, a deep learning model was used to perform pixel-level segmentation of the tumor boundary in the global digital image, and a surgical safety margin was generated by expanding outward based on the segmentation results using a geometric algorithm.

[0011] The generated digital cutting path is then projected back onto the patient's skin surface in real time and with precision using a projection system integrated into the imaging device, in the form of structured light, providing intuitive and dynamic visual guidance for the surgical procedure.

[0012] This invention provides a method for designing tumor resection margins based on local skin images, which includes the following steps:

[0013] Establish a global spatial coordinate system for the target surgical area, wherein the global spatial coordinate system forms a stable and fixed mapping relationship with the patient's skin surface;

[0014] By using a handheld imaging device that integrates a high-resolution image acquisition unit and a spatial pose tracking unit, the target surgical area is scanned in the global spatial coordinate system, and a series of high-resolution local skin images and the real-time six-degree-of-freedom pose data of the handheld imaging device when acquiring each frame of the local skin image are simultaneously acquired.

[0015] Based on the real-time six-degree-of-freedom pose data, the series of high-resolution local skin images are projected in real time and fused onto a preset digital surface model to dynamically construct a global panoramic digital image representing the target surgical area;

[0016] The global panoramic digital image is processed, and the tumor region boundary contour in the global panoramic digital image is identified and extracted through a pre-trained tumor boundary segmentation model.

[0017] Based on the tumor region boundary contour, a safe surgical margin path is calculated and generated around the tumor region boundary contour using a preset geometric expansion algorithm;

[0018] The coordinate data of the surgical safety cutting edge path, combined with the real-time six-degree-of-freedom pose data of the handheld imaging device, is precisely projected onto the corresponding physical location on the patient's skin surface in the form of visible structured light through the micro-projection unit integrated on the handheld imaging device.

[0019] Furthermore, establishing the global spatial coordinate system of the target surgical area specifically includes:

[0020] A physical reference mark with unique geometric features is placed on a stable skin surface adjacent to the target surgical area;

[0021] The handheld imaging device is activated, and the physical reference mark is identified through the spatial pose tracking unit. A three-dimensional Cartesian global spatial coordinate system is established with the geometric center of the physical reference mark as the origin and its preset plane and direction as the reference axis.

[0022] In one embodiment of the present invention, the spatial pose tracking unit includes an inertial measurement unit and a wide-angle vision sensor; the acquisition of real-time six-degree-of-freedom pose data specifically includes:

[0023] The inertial measurement unit acquires the angular velocity and linear acceleration data of the handheld imaging device at a sampling frequency greater than 1000 Hz.

[0024] The wide-angle vision sensor captures visual feature information of the surrounding environment at a frame rate greater than 30 Hz.

[0025] In the data fusion processor, an algorithm based on extended Kalman filtering is run to fuse the angular velocity and linear acceleration data with the visual feature information in real time, and calculate the three-dimensional translation vector and three-dimensional rotation matrix of the handheld imaging device relative to the origin of the global spatial coordinate system, which together constitute the real-time six-degree-of-freedom pose data.

[0026] Furthermore, the step of projecting and fusing the series of high-resolution local skin images onto a preset digital surface model in real time specifically includes:

[0027] In the global spatial coordinate system, a two-dimensional plane or a three-dimensional curved surface is predefined as the digital surface model, which geometrically approximates the skin surface morphology of the target surgical area;

[0028] For each high-resolution local skin image acquired, distortion correction is first performed using a pre-calibrated camera intrinsic parameter matrix;

[0029] Then, the homogeneous transformation matrix represented by the real-time six-degree-of-freedom pose data of the frame image is applied to each pixel of the distortion-corrected image to calculate its three-dimensional spatial ray in the global spatial coordinate system.

[0030] Calculate the coordinates of the intersection point between the three-dimensional spatial ray and the digital surface model;

[0031] The color value of the pixel is assigned to the intersection coordinates at the corresponding texture map position on the digital surface model;

[0032] When a new projection area overlaps with an existing projection area, a weighted average fusion algorithm is performed on the pixel color values ​​of the overlapping area. The weight coefficient is inversely proportional to the distance from the pixel's position in its original image to the image center, thereby achieving smooth and seamless image fusion.

[0033] In one embodiment of the present invention, the tumor boundary segmentation model is a convolutional neural network based on an encoder-decoder architecture; the specific steps of identifying and extracting the tumor region boundary contour in the global panoramic digital image include:

[0034] The dynamically updated global panoramic digital image is used as input and fed into the convolutional neural network.

[0035] The convolutional neural network performs pixel-by-pixel semantic segmentation on the input image and outputs a binary mask image with the same size as the global panoramic digital image, wherein pixels of a first preset value represent tumor regions and pixels of a second preset value represent non-tumor regions.

[0036] A contour extraction algorithm is performed on the binarized mask image to obtain a set of ordered pixel coordinates that characterize the boundary of the tumor region, i.e., the contour of the tumor region boundary.

[0037] Furthermore, the step of calculating and generating a safe surgical margin path based on the tumor region boundary contour specifically includes:

[0038] For the coordinates of each pixel on the boundary contour of the tumor region, calculate its local tangent vector on that contour;

[0039] The unit normal vector pointing outward from the tumor region is calculated based on the local tangent vector.

[0040] The coordinates of each pixel on the boundary contour of the tumor region are translated along its corresponding unit normal vector direction by a preset surgical safety distance value to obtain new coordinate points.

[0041] Connect all the newly generated coordinate points to form a closed curve that surrounds the boundary of the tumor region; this curve is the surgical safety margin path. The unit of the surgical safety distance value is millimeters, and its value is set according to clinical medical guidelines.

[0042] In one embodiment of the present invention, the handheld imaging device, the high-resolution image acquisition unit, the spatial pose tracking unit, the micro-projection unit, and the embedded processing unit for running the algorithms of each step of the method are integrated into a unified, ergonomically designed handheld casing.

[0043] The relative spatial relationships between the optical axis of the high-resolution image acquisition unit, the measurement center of the spatial pose tracking unit, and the projection optical axis of the micro-projection unit are precisely calibrated in advance and stored in the embedded processing unit in the form of a fixed transformation matrix.

[0044] The present invention also provides a tumor resection margin design system based on local skin images to implement the above method, the system comprising:

[0045] The global spatial coordinate system establishment module is used to establish a global spatial coordinate system associated with the patient's skin surface in the target surgical area;

[0046] The handheld imaging device integrates a high-resolution image acquisition module, a spatial pose tracking module, a micro-projection module, and a central data processing module.

[0047] The high-resolution image acquisition module is used to continuously acquire local skin images of the target surgical area during the scanning process;

[0048] The spatial pose tracking module is used to synchronously acquire the real-time six-degree-of-freedom pose data of the handheld imaging device in the global spatial coordinate system when acquiring each frame of local skin image;

[0049] The central data processing module is configured to perform the following operations:

[0050] Receive the local skin image and the real-time six-degree-of-freedom pose data;

[0051] A real-time projection mapping algorithm is run to project and fuse the local skin image onto the digital surface model based on its pose data, thereby dynamically generating a global panoramic digital image.

[0052] The tumor boundary intelligent segmentation algorithm is run to identify and extract the boundary contour of the tumor region from the global panoramic digital image;

[0053] Run the safe surgical margin generation algorithm to calculate and generate a safe surgical margin path by extending outward by a preset distance based on the boundary contour of the tumor region;

[0054] The micro-projection module is used to receive the surgical safety cutting edge path data generated by the central data processing module, and combine it with the real-time pose provided by the spatial pose tracking module to project the surgical safety cutting edge path in the form of visible light onto the corresponding physical position on the patient's skin in real time and accurately, forming a closed-loop visual guidance.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] 1. This invention completely eliminates the traditional image stitching process based on feature matching by tracking the spatial pose of the imaging device in real time and directly projecting and mapping. It fundamentally solves the time-consuming problem caused by the large amount of computation in the stitching algorithm, as well as the image misalignment and deformation caused by mismatched feature points at high resolution. It realizes the instantaneous and high-precision construction of panoramic images of large-area irregular tumor regions.

[0057] 2. This invention combines intelligent segmentation of tumor boundaries, automatic generation of safe surgical margins, and real-time augmented reality projection to construct a complete closed-loop workflow from data acquisition and analysis to surgical guidance. Surgeons can immediately see the precise surgical path planned by the system on the patient's body while scanning the skin. This intuitive and dynamic visual feedback greatly improves the accuracy and efficiency of surgical resection and reduces reliance on the surgeon's personal experience.

[0058] 3. This invention integrates all functional units into a handheld device, making it easy to operate without the need for complex external tracking equipment. It can be seamlessly integrated into existing surgical procedures, has strong clinical applicability and promotional value, and provides a brand-new high-precision and intelligent solution for skin tumor surgery. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall technical solution architecture of the tumor resection margin design method based on local skin images proposed in this invention;

[0060] Figure 2 This is a schematic diagram of the core principle framework of real-time projection mapping based on spatial pose tracking and digital surface model in this invention.

[0061] Figure 3 This is a flowchart illustrating the logical flow of establishing the global spatial coordinate system and tracking the pose of the handheld imaging device in this invention.

[0062] Figure 4 This is a logical flowchart of the intelligent segmentation of tumor boundaries and generation of safe cutting edge paths in this invention;

[0063] Figure 5 This is a schematic diagram of the multi-level data flow of local skin image fusion and dynamic construction of global panoramic digital image in this invention;

[0064] Figure 6 This is a schematic diagram of the multi-level interaction and data flow between real-time projection of surgical safety cutting path and augmented reality visual guidance in this invention. Detailed Implementation

[0065] Please refer to Figures 1 to 6 This invention provides a tumor resection margin design method based on local skin images. Its core lies in abandoning the traditional stitching process that relies on image feature matching. Instead, it constructs a global spatial coordinate system that is stably associated with the patient's skin surface and combines it with the real-time six-degree-of-freedom pose information of the handheld imaging device in this coordinate system. The continuously acquired high-resolution local skin images are directly projected and fused onto a preset digital surface model, thereby dynamically generating a seamless and accurate panoramic digital image of the tumor area.

[0066] Based on this, a deep learning model is used to segment the tumor boundary at the pixel level in the panoramic image. Based on the segmentation results, a safe surgical resection path is generated through a geometric extension algorithm. Finally, the path is projected back to the patient's skin surface in real time in the form of visible structured light through a micro projection unit integrated into the imaging device, providing surgeons with intuitive and dynamic visual guidance.

[0067] The method includes the following steps: establishing a global spatial coordinate system for the target surgical area; synchronously acquiring local skin images and their real-time six-degree-of-freedom pose data using a handheld imaging device; projecting and fusing the local images onto a digital surface model based on the pose data to dynamically construct a global panoramic digital image; segmenting the tumor boundary of the panoramic image; generating a safe surgical margin path based on the boundary contour; and projecting the margin path onto the skin surface in real time through a projection unit.

[0068] Establish a global spatial coordinate system for the target surgical area. This coordinate system must form a stable and fixed mapping relationship with the patient's skin surface to ensure spatial consistency between all subsequent image projections and path projections.

[0069] In practice, a physical reference marker with unique geometric features is placed on a stable skin surface adjacent to the target surgical area.

[0070] The physical reference marker is made of a high-contrast material and its geometry includes asymmetrical structures, such as a triangle composed of three non-collinear points or a circle with a notch at a specific angle, to ensure that it can be uniquely identified from any viewing angle.

[0071] After the handheld imaging device is activated, its built-in spatial pose tracking unit first captures an image of the physical reference marker and identifies its position and orientation through a pre-stored template matching algorithm.

[0072] Using the geometric center of the physical reference mark as the origin, its preset plane normal vector as the Z-axis direction, and a specific side or axis of symmetry as the X-axis direction, a right-handed three-dimensional Cartesian global spatial coordinate system is constructed.

[0073] Once established, this coordinate system remains unchanged throughout the entire surgical procedure. Even if the patient experiences slight changes in position, the coordinate system remains valid as long as the physical reference markers do not shift.

[0074] By using a handheld imaging device that integrates a high-resolution image acquisition unit and a spatial pose tracking unit, the target surgical area is scanned in the global spatial coordinate system, and a series of high-resolution local skin images and the device's real-time six-degree-of-freedom pose data are acquired simultaneously when each frame of the image is acquired.

[0075] The handheld imaging device features an ergonomic, integrated casing design, housing a high-resolution image acquisition module, a spatial pose tracking module, a micro-projection module, and an embedded processing unit. The high-resolution image acquisition unit utilizes a global shutter CMOS sensor with over eight megapixels, coupled with a fixed-focus lens, to ensure clear, motion-blur-free skin texture images even at close range.

[0076] The spatial pose tracking unit is composed of an inertial measurement unit and a wide-angle vision sensor.

[0077] The inertial measurement unit continuously outputs triaxial angular velocity and triaxial acceleration data at a sampling frequency greater than 1000 Hz; the wide-angle vision sensor captures grayscale images of the surrounding environment at a frame rate greater than 30 Hz, with a field of view of not less than 120 degrees, and is used to capture the macroscopic structural features of physical reference markers and the surrounding skin surface.

[0078] The embedded processing unit runs a data fusion algorithm based on extended Kalman filtering, which fuses the high-frequency dynamic data from the inertial measurement unit with the low-frequency but high-precision position observation data from the wide-angle vision sensor in real time.

[0079] The state vector of the algorithm includes the device's position, attitude, velocity, and sensor bias. The observation model is constructed based on the known position of visual feature points in the global coordinate system and their projection relationship on the image plane.

[0080] Through iterative prediction and updating, the three-dimensional translation vector and three-dimensional rotation matrix of the device relative to the origin of the global spatial coordinate system are calculated, and the two together constitute complete six-degree-of-freedom pose data.

[0081] The pose data is strictly time-synchronized with the corresponding frame's local skin image, with the timestamp error controlled within 0.5 milliseconds.

[0082] Based on the real-time six-degree-of-freedom pose data, the series of high-resolution local skin images are projected in real time and fused onto a preset digital surface model to dynamically construct a global panoramic digital image representing the target surgical area.

[0083] The digital surface model is predefined during the system initialization phase, and its geometry is selected as a two-dimensional plane or a three-dimensional curved surface based on the anatomical features of the target surgical area.

[0084] For relatively flat areas such as the torso and limbs, a two-dimensional planar model is used; for areas with significant curvature such as the face and joints, a quadratic surface or piecewise smooth surface model fitted by sparse point clouds is used.

[0085] The model is fixed in the global spatial coordinate system, and its surface parameters are stored in the memory of the embedded processing unit.

[0086] For each newly acquired local skin image, radial and tangential distortion corrections are first performed using a pre-calibrated camera intrinsic parameter matrix to restore the geometric authenticity of the image.

[0087] Subsequently, each pixel of the corrected image is considered as a ray originating from the optical center of the camera.

[0088] The direction of the ray is obtained by inverse transformation of the pixel coordinates using the camera intrinsic parameters. Its starting point and direction in the global coordinate system are determined by the homogeneous transformation matrix corresponding to the six-degree-of-freedom pose data of the current frame.

[0089] Calculate the intersection point between the ray and the digital surface model. If a unique intersection point exists, write the color value of the pixel into the texture map buffer at the corresponding position of the digital surface model.

[0090] When the projection regions of multiple image frames overlap on the model surface, a weighted average fusion algorithm is executed.

[0091] The weighting coefficient is inversely proportional to the normalized radial distance of a pixel in its original image; that is, the closer a pixel is to the image center, the higher its contribution weight. Specifically, let the coordinates of a pixel in the original image be... The image center is If the length of the image diagonal is D, then its weight is:

[0092] ;

[0093] This strategy suppresses the impact of image edge quality degradation caused by lens distortion or illumination attenuation on the fusion result, ensuring smooth transitions in overlapping areas without obvious stitching marks.

[0094] As the scan progresses, the texture maps on the digital surface model are continuously updated, forming a dynamically growing global panoramic digital image. Its resolution adaptively adjusts as the coverage area expands, always maintaining clear local details.

[0095] The global panoramic digital image is processed, and the tumor region boundary contour is identified and extracted using a pre-trained tumor boundary segmentation model.

[0096] The tumor boundary segmentation model adopts a convolutional neural network based on an encoder-decoder architecture. The encoder part is composed of multiple residual blocks stacked together, which extracts multi-scale semantic features of the image layer by layer. The decoder part gradually restores the spatial resolution through transposed convolution and skip connections, and finally outputs a pixel-level classification result with the same size as the input image.

[0097] The model is trained end-to-end on a large dataset of skin images with tumor boundaries labeled. The loss function uses a combination of weighted cross-entropy and Dice coefficients to address the class imbalance problem, where the tumor region usually accounts for a small proportion.

[0098] In actual operation, the embedded processing unit continuously monitors the update status of the global panoramic digital image.

[0099] Whenever the coverage area of ​​a newly added projection region exceeds a preset threshold (e.g., 5%), the current complete panoramic image is fed into the segmentation model for inference.

[0100] The model outputs a binary mask image, where regions with a pixel value of 255 represent tumor tissue and regions with a pixel value of 0 represent normal skin.

[0101] A chain code-based contour extraction algorithm is performed on the mask image to traverse all connected domain boundaries and obtain an ordered sequence of two-dimensional pixel coordinates, which is the contour of the tumor region boundary.

[0102] To improve robustness, the system performs post-processing on the extracted contours, including removing isolated noise areas with an area of ​​less than 10 square millimeters, and performing a closing operation on the contours to fill in minor breaks.

[0103] Based on the tumor region's boundary contour, a pre-defined geometric expansion algorithm is used to calculate and generate a safe surgical resection path around the tumor region. This algorithm first parameterizes the boundary contour, treating it as a closed planar curve. For each vertex coordinate on the contour... Calculate its local tangent vector:

[0104] (Using periodic boundary conditions);

[0105] Calculate the unit normal vector perpendicular to the tangent vector and pointing outwards from the tumor:

[0106] ;

[0107] The criterion for determining "external" here is: when traversing the contour clockwise, the normal vector points to the right.

[0108] Each vertex Along its corresponding unit normal vector Pre-set surgical safety distance for directional translation A new vertex is obtained:

[0109] ;

[0110] The surgical safety distance The unit is millimeters, and its value is set according to clinical medical guidelines, typically ranging from 3 to 10 millimeters. The specific value is input by the doctor into the system via the human-computer interface before surgery. Connect all new vertices. The resulting closed curve is the safe surgical margin path.

[0111] To ensure the smoothness of the path, the system performs cubic spline interpolation on the path to generate a continuously differentiable tangent trajectory, and discretizes it into a point sequence suitable for the projection system.

[0112] The coordinate data of the surgical safety cutting edge path is combined with the real-time six-degree-of-freedom pose data of the handheld imaging device, and the cutting edge path is precisely projected onto the corresponding physical position on the patient's skin surface in the form of visible structured light through the micro projection unit integrated on the device.

[0113] The micro-projection unit uses a combination of a laser diode and a MEMS micromirror to project green or red visible light lines. Its optical axis is parallel to the optical axis of the high-resolution image acquisition unit, and the relative pose relationship between the two has been precisely calibrated and is based on a fixed homogeneous transformation matrix. It is stored in the embedded processing unit in the form of [data / format].

[0114] At the projection moment, the system first transforms the global coordinate point sequence of the tangent path to the current device coordinate system: for any point on the path Its coordinates in the device coordinate system:

[0115] ;

[0116] in Let be the homogeneous pose matrix of the six degrees of freedom in the current frame.

[0117] Will Further transformation to the projected element coordinate system:

[0118] ;

[0119] Based on the intrinsic parameter model of the projection unit, Qᵖʳᵒʲ is projected onto the driving coordinates of the MEMS micromirror to generate the corresponding scanning trajectory command.

[0120] The projection system continuously updates the projection path at a refresh rate greater than 60 Hz, ensuring that even if the device moves, the projected edge lines remain stably attached to the actual position on the skin surface, forming augmented reality visual guidance.

[0121] During the surgery, doctors can directly observe the safe cutting edge outlined by light on the skin and make precise excisions accordingly, without having to repeatedly compare the screen image with the actual lesion.

[0122] All functional modules of the handheld imaging device are integrated into a single housing. The relative positions of the optical and sensing units inside are precisely assembled and calibrated before leaving the factory to ensure the overall spatial consistency of the system.

[0123] The embedded processing unit adopts a heterogeneous computing architecture, which includes a multi-core ARM processor and a dedicated neural network accelerator, respectively responsible for pose calculation, image fusion and deep learning inference tasks, and achieves efficient data exchange through shared memory.

[0124] The entire system's workflow is fully automated, with an end-to-end delay of no more than 200 milliseconds from the start of scanning to the projection of the cutting edge, meeting the real-time requirements of surgery.

[0125] Throughout the entire method execution process, the system has a comprehensive exception handling mechanism.

[0126] If the spatial pose tracking unit loses its pose due to occlusion or sudden changes in lighting, the system immediately pauses image fusion and projection, and issues an audible and visual alarm to prompt the operator to realign with the physical reference marker.

[0127] If the tumor region output by the segmentation model is abnormally enlarged or morphologically mutated, the system will trigger a manual review request, requiring the doctor to confirm the segmentation results before continuing to generate the resection margin.

[0128] In addition, the projection unit is equipped with power limiting and eye safety protection circuits to ensure that the projected light intensity is always less than the international safety standard limit.

[0129] In summary, this embodiment constructs an efficient, accurate, and intuitive tumor resection design and guidance system through four core technical components: spatial pose-driven real-time projection mapping, deep learning-assisted intelligent segmentation, geometrically constrained resection edge generation, and closed-loop augmented reality projection. This system solves the real-time and accuracy bottlenecks faced by traditional image stitching methods in surgical scenarios, and provides reliable technical support for skin tumor surgery.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0131] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for designing tumor resection margins based on local skin images, characterized in that, include: Establish a global spatial coordinate system for the target surgical area, wherein the global spatial coordinate system forms a stable and fixed mapping relationship with the patient's skin surface; By using a handheld imaging device that integrates a high-resolution image acquisition unit and a spatial pose tracking unit, the target surgical area is scanned in the global spatial coordinate system, and a series of high-resolution local skin images and the real-time six-degree-of-freedom pose data of the handheld imaging device when acquiring each frame of the local skin image are simultaneously acquired. Based on the real-time six-degree-of-freedom pose data, the series of high-resolution local skin images are projected in real time and fused onto a preset digital surface model to dynamically construct a global panoramic digital image representing the target surgical area; The global panoramic digital image is processed, and the tumor region boundary contour in the global panoramic digital image is identified and extracted through a pre-trained tumor boundary segmentation model. Based on the tumor region boundary contour, a safe surgical margin path is calculated and generated around the tumor region boundary contour using a preset geometric expansion algorithm; The coordinate data of the surgical safety cutting edge path, combined with the real-time six-degree-of-freedom pose data of the handheld imaging device, is precisely projected onto the corresponding physical location on the patient's skin surface in the form of visible structured light through the micro-projection unit integrated on the handheld imaging device.

2. The tumor resection margin design method based on local skin images according to claim 1, characterized in that, The establishment of the global spatial coordinate system for the target surgical area specifically includes: A physical reference marker with unique geometric features is placed on a stable skin surface adjacent to the target surgical area; The handheld imaging device is activated, and the physical reference mark is identified through the spatial pose tracking unit. A three-dimensional Cartesian global spatial coordinate system is established with the geometric center of the physical reference mark as the origin and its preset plane and direction as the reference axis.

3. The tumor resection margin design method based on local skin images according to claim 1, characterized in that, The spatial pose tracking unit includes an inertial measurement unit and a wide-angle vision sensor; Acquiring real-time six-DOF pose data specifically includes: The inertial measurement unit acquires the angular velocity and linear acceleration data of the handheld imaging device at a sampling frequency greater than 1000 Hz. The wide-angle vision sensor captures visual feature information of the surrounding environment at a frame rate greater than 30 Hz. In the data fusion processor, an algorithm based on extended Kalman filtering is run to fuse the angular velocity and linear acceleration data with the visual feature information in real time, and calculate the three-dimensional translation vector and three-dimensional rotation matrix of the handheld imaging device relative to the origin of the global spatial coordinate system, which together constitute the real-time six-degree-of-freedom pose data.

4. The tumor resection margin design method based on local skin images according to claim 1, characterized in that, The step of projecting and fusing the series of high-resolution local skin images onto a preset digital surface model in real time specifically includes: In the global spatial coordinate system, a two-dimensional plane or a three-dimensional curved surface is predefined as the digital surface model, which geometrically approximates the skin surface morphology of the target surgical area; For each high-resolution local skin image acquired, distortion correction is first performed using a pre-calibrated camera intrinsic parameter matrix; Then, the homogeneous transformation matrix represented by the real-time six-degree-of-freedom pose data of the frame image is applied to each pixel of the distortion-corrected image to calculate its three-dimensional spatial ray in the global spatial coordinate system. Calculate the coordinates of the intersection point between the three-dimensional spatial ray and the digital surface model; The color value of the pixel is assigned to the intersection coordinates at the corresponding texture map position on the digital surface model; When a new projection area overlaps with an existing projection area, a weighted average fusion algorithm is performed on the pixel color values ​​of the overlapping area. The weight coefficient is inversely proportional to the distance from the pixel's position in its original image to the image center, thereby achieving smooth and seamless image fusion.

5. The tumor resection margin design method based on local skin images according to claim 1, characterized in that, The tumor boundary segmentation model is a convolutional neural network based on an encoder-decoder architecture; the specific steps of identifying and extracting the tumor region boundary contour in the global panoramic digital image include: The dynamically updated global panoramic digital image is used as input and fed into the convolutional neural network. The convolutional neural network performs pixel-by-pixel semantic segmentation on the input image and outputs a binary mask image with the same size as the global panoramic digital image, wherein pixels of a first preset value represent tumor regions and pixels of a second preset value represent non-tumor regions. A contour extraction algorithm is performed on the binarized mask image to obtain a set of ordered pixel coordinates that characterize the boundary of the tumor region, i.e., the contour of the tumor region boundary.

6. The tumor resection margin design method based on local skin images according to claim 5, characterized in that, The calculation and generation of a safe surgical margin path based on the tumor region boundary contour specifically includes: For the coordinates of each pixel on the boundary contour of the tumor region, calculate its local tangent vector on that contour; The unit normal vector pointing outward from the tumor region is calculated based on the local tangent vector. The coordinates of each pixel on the boundary contour of the tumor region are translated along its corresponding unit normal vector direction by a preset surgical safety distance value to obtain new coordinate points. Connect all the newly generated coordinate points to form a closed curve that surrounds the boundary of the tumor region. This curve is the safe surgical margin path.

7. The tumor resection margin design method based on local skin images according to claim 6, characterized in that, The unit of the surgical safety distance value is millimeters, and its value is set according to clinical medical standards.

8. The tumor resection margin design method based on local skin images according to claim 1, characterized in that, The high-resolution image acquisition unit, the spatial pose tracking unit, the micro-projection unit, and the embedded processing unit for running the algorithms of each step of the method are integrated into a unified, ergonomically designed handheld shell. The relative spatial relationships between the optical axis of the high-resolution image acquisition unit, the measurement center of the spatial pose tracking unit, and the projection optical axis of the micro-projection unit are precisely calibrated in advance and stored in the embedded processing unit in the form of a fixed transformation matrix.

9. The tumor resection margin design method based on local skin images according to claim 4, characterized in that, The digital surface model is a two-dimensional plane, a quadratic surface fitted by sparse point clouds, or a piecewise smooth surface, and its selection is determined based on the skin surface curvature characteristics of the target surgical area.

10. The tumor resection margin design method based on local skin images according to claim 5, characterized in that, The loss function of the convolutional neural network uses a combination of weighted cross-entropy and Dice coefficients to address the class imbalance problem where the tumor region accounts for a small proportion.

Images