Multimodal image fusion thyroid breast surgical biopsy, surgical planning method and system

Abstract

This invention discloses a method and system for thyroid and breast surgical biopsy and surgical planning using multimodal image fusion, belonging to the fields of medical image processing and surgical computer-aided planning technology. The method includes the following steps: acquiring multimodal medical image data and dynamic body posture information of the thyroid and breast regions, and organizing them into a continuous time series according to time sequence as the input basis for subsequent analysis and calculation; comparing magnetic field interference signs and brightness extension trajectories in the continuous time series frame by frame, extracting regions with abnormal signal distributions, generating a candidate list of artifacts, and identifying the suspected artifact range. This invention achieves synchronous correction of artifact regions and restoration of image geometric accuracy through time series reconstruction and spatial offset correction, ensuring the spatial consistency of the fusion model. By generating a list of coordinate anomalies and high-risk locations, it enables dynamic path review and risk identification, ensuring that the biopsy and surgical paths correspond to the real tissue, improving sampling accuracy and surgical safety.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and surgical computer-aided planning technology, specifically to a method and system for thyroid and breast surgical biopsy and surgical planning using multimodal image fusion. Background Technology

[0002] Multimodal image fusion-based thyroid and breast surgical biopsy and surgical planning refers to the spatial registration, object identification, and information fusion of multi-source medical images such as ultrasound, CT, and MRI during the diagnosis and treatment of thyroid and breast tumors. During the registration stage, key objects such as lesions, nerves, blood vessels, and glandular ducts are identified. By integrating the differences in resolution, contrast, and real-time performance among various images, a three-dimensional visualization model with both anatomical accuracy and tissue layer clarity is generated to comprehensively present the lesion's morphology, extent, and spatial relationship with surrounding key tissues. Based on this, computer-aided design and artificial intelligence analysis are used to conduct object-level path simulation, risk assessment, and scheme optimization for different access routes, forming a precise biopsy path and surgical plan that conforms to individual anatomical characteristics. Furthermore, dynamic corrections can be made during surgery through real-time image comparison and object identification results, achieving seamless integration between preoperative planning and intraoperative execution, supporting higher-precision sampling positioning and safer operation.

[0003] The existing technology has the following shortcomings:

[0004] In existing technologies, the presence of trace metallic foreign bodies in the patient's body during MRI imaging, such as surgical clips, implants, or needle remnants, can easily interfere with the uniform distribution of the magnetic field, causing local signal distortion and image stretching artifacts. These artifacts can cause abnormal extensions or voids in tissue boundaries within the image, leading fusion algorithms to misidentify artifacts as real anatomical regions when superimposing multimodal images. Due to the misalignment of signal coordinates in the artifact region, the spatial coordinates of that location are prone to abrupt changes after fusion, thereby compromising the geometric consistency of the overall image. When the system generates biopsy or surgical pathways based on the distorted fusion model, the pathway planning may deviate from the actual tissue location, causing needle path deviation, misjudgment of critical structures, and in severe cases, failure of vascular puncture, nerve damage, or lesion sampling.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion, so as to solve the problems in the background art mentioned above.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion, comprising the following steps:

[0008] Multimodal medical imaging data and postural dynamic information of the thyroid and breast regions were collected and organized into a continuous time series in chronological order as the input basis for subsequent analysis and calculation.

[0009] Frame-by-frame comparison of magnetic field interference signs and brightness extension trajectories in continuous time series is performed to extract regions with abnormal signal distribution, generate a candidate list of artifacts and identify the suspected range of artifacts.

[0010] Based on the list of artifact candidates, identify the boundary points between the real tissue boundary and the abnormal signal area in each suspected area, determine the spatial offset range caused by metal interference, and generate a coordinate anomaly draft containing offset coordinate information.

[0011] Based on the coordinate anomaly report, the established biopsy path and surgical path are reviewed and analyzed to determine the spatial offset area traversed by the needle path, form a list of high-risk locations, and identify the path segments that need to be corrected.

[0012] For the high-risk location list, dynamic adjustments are performed on the continuous image sequence. A reverse synchronization operation is introduced into the time series, short still images are inserted between continuous images, and the depth direction is reversed based on the positional difference between the preceding and following images. The time order of the fused images is rearranged so that the spatial position of the lesion area corresponds to the surrounding tissue again, thereby restoring the overall geometric accuracy of the image fusion model.

[0013] The preferred method for arranging continuous sequences is as follows:

[0014] During the image acquisition preparation stage, based on the anatomical features, tissue density differences, and image resolution requirements of the thyroid and breast regions, the acquisition sequence and time interval of multimodal medical images are determined, and an acquisition plan is established to unify imaging parameters, scan slice thickness, time stamp format, and coordinate reference system.

[0015] During the multimodal image synchronous acquisition phase, different modal image acquisition tasks are executed sequentially under the constraints of the time baseline, and the time node of each image generation is recorded using the time synchronization signal to form a time-stamped image set.

[0016] During the body dynamic information recording stage, body surface posture parameters and respiratory displacement parameters are collected by external sensing devices and stored synchronously in the posture information recording set according to time tags;

[0017] In the time series processing and construction stage, the image frames are sorted temporally and aligned spatially based on the time label and attitude information record set, and then processed into a continuous time series for multimodal fusion analysis.

[0018] Preferably, the steps for generating a candidate list of artifacts and identifying the suspected range of artifacts are as follows:

[0019] In the frame-by-frame analysis stage, the continuous sequence with complete time order is used as input data. The magnetic field signal distribution and brightness level of each frame image are scanned and features are extracted. The signal intensity change trend is recorded to form a time-series database.

[0020] In the interference feature comparison stage, the magnetic field signal features and brightness extension trajectory are compared frame by frame using a time-series database, and a list of interference signs is formed based on the brightness diffusion pattern and spatial extension direction.

[0021] In the abnormal region extraction stage, based on the list of interference signs, the abnormal signal regions at the same position in consecutive frames are temporally merged and spatially aggregated to obtain a set of abnormal signal regions.

[0022] During the artifact suspected range identification stage, the abnormal regions are classified and spatially marked based on the abnormal signal region set, forming an artifact candidate list and identifying the suspected artifact range.

[0023] Preferably, in the interference feature comparison stage, by continuously comparing adjacent frames on the time axis, the spatial coherence and temporal continuity of the brightness extension trajectory are jointly judged. When the brightness distribution has no corresponding tissue structure support in the spatial location and continues to extend in time, the area is identified as an artifact interference point, and its corresponding time range and spatial coordinate range are recorded in the interference sign list.

[0024] Preferably, the steps for generating a coordinate anomaly draft are as follows:

[0025] In the boundary feature identification stage, based on the candidate list of artifacts, the edge signal features of suspected artifact areas are analyzed. By tracking the brightness evolution trend through time correlation, the true tissue boundary and the abnormal signal area are distinguished, and a boundary feature database is formed.

[0026] In the boundary point extraction stage, the boundary feature database is used to detect the differences in brightness and shape within the suspected area, identify the intersection of the real tissue boundary and the abnormal area, and generate a set of boundary points.

[0027] In the spatial offset determination stage, based on the set of boundary points, the spatial offset range caused by metal interference is determined by comparing the positional changes of the boundary points in continuous time frames, and a spatial offset dataset is formed.

[0028] During the coordinate anomaly generation stage, the coordinates of the artifact regions are revised and anomaly annotations are performed based on the spatial offset dataset to generate a coordinate anomaly draft containing offset coordinate information.

[0029] Preferably, in the spatial offset determination stage, the spatial coordinates of the boundary points in the continuous time frame are compared in multiple directions. The spatial offset range caused by metal interference is determined based on the displacement distribution in the body surface direction, depth direction and lateral direction. In the coordinate anomaly generation stage, the offset range is marked in the image frame in the form of three-dimensional spatial projection to realize the spatial position correspondence between the artifact area and the real tissue boundary.

[0030] The preferred process for forming the list of high-risk locations is as follows:

[0031] During the path data loading and mapping stage, the spatial correspondence between the coordinate anomaly draft and the biopsy path and surgical path is established, and the path nodes and offset coordinates are aligned with a unified spatial coordinate system to form a composite path dataset.

[0032] In the spatial overlay analysis stage of the needle path and the offset area, the needle path trajectory is projected segment by segment into the offset area in three-dimensional space using the composite path dataset to analyze the interaction between the path points and the offset area and identify potential high-risk path segments.

[0033] In the risk path segment identification and classification stage, the risk level of the path segments is divided and classified according to the spatial overlay results, and a list of risk path segments including path number, start and end coordinates, offset direction and offset magnitude is generated.

[0034] In the stage of generating a list of high-risk locations and determining the path correction segments, the list of high-risk path segments and the offset coordinate data are combined to aggregate high-risk path segments and generate a list of high-risk locations, and determine the path segments that need to be corrected.

[0035] Preferably, in the stage of generating the high-risk location list and determining the path correction segment, the spatial proximity relationship of high-risk path segments is aggregated, adjacent path segments are merged into high-risk location units, and the high-risk location units are three-dimensionally located and identified according to the offset direction and spatial depth information, generating a high-risk location list containing location number, offset coordinates, path number and offset range, which is used to determine the path segments that need to be corrected.

[0036] Preferably, for the high-risk location list, the continuous image sequence is dynamically adjusted by introducing a reverse synchronization operation into the time series, inserting short still images between continuous images, and performing back-calculation correction based on the positional difference between consecutive images. The steps for rearranging the time order of the fused images are as follows:

[0037] During the image time synchronization stage, using the path segments and spatial coordinate information in the high-risk location list as a reference, the images of the time periods affected by metal interference are extracted to form a set of high-risk time segments, and the time labels are corrected to establish the correspondence between time and spatial offset coordinates.

[0038] During the reverse synchronization frame interpolation stage, short still frames are inserted between consecutive frames according to the time sequence. The still frames are used to balance the brightness and displacement differences between the preceding and following frames to form a time extension area.

[0039] In the depth-to-depth correction stage, based on the time sequence of the inserted still images, the positional difference between the preceding and following images is back-calculated in the depth direction, and the correction reference layer is determined to complete the spatial alignment.

[0040] In the time sequence rearrangement stage, the corrected time series are reordered and fused according to the time labels to restore the overall geometric accuracy of the image fusion model.

[0041] A multimodal image fusion-based system for thyroid and breast surgical biopsy and surgical planning, including an image acquisition sequence module, an artifact detection module, an offset identification and calibration module, a path risk analysis module, and an image correction and adjustment module:

[0042] The image acquisition sequence module collects multimodal medical image data and postural dynamic information of the thyroid and breast regions, and organizes them into a continuous time series in chronological order as the input basis for subsequent analysis and calculation.

[0043] The artifact detection module compares the magnetic field interference signs and brightness extension trajectories in a continuous time series frame by frame, extracts the regions with abnormal signal distributions, generates a candidate list of artifacts, and identifies the suspected range of artifacts.

[0044] The offset identification and calibration module identifies the boundary points between the real tissue boundary and the abnormal signal area in each suspected area around the artifact candidate list, determines the spatial offset range caused by metal interference, and generates a coordinate anomaly draft containing offset coordinate information.

[0045] The path risk analysis module reviews and analyzes the established biopsy path and surgical path based on the coordinate anomaly draft, determines the spatial deviation area traversed by the needle path, generates a list of high-risk locations, and identifies the path segments that need to be corrected.

[0046] The image correction and adjustment module performs dynamic adjustments on the continuous image sequence for the high-risk location list. It introduces reverse synchronization operation into the time series, inserts short static images between continuous images, and performs reverse correction in the depth direction based on the positional difference between the preceding and following images. It rearranges the time order of the fused images so that the spatial position of the lesion area corresponds to the surrounding tissue again, thereby restoring the overall geometric accuracy of the image fusion model.

[0047] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0048] This invention introduces temporal reconstruction and spatial offset correction mechanisms during multimodal image fusion, enabling synchronous correction of image regions affected by metallic interference in both temporal and spatial dimensions. By comparing magnetic field interference signs and brightness extension trajectories frame by frame, combined with reverse synchronization and the insertion of still images, temporal balance and spatial location restoration of artifact regions are achieved. This allows the fused image to re-establish a true spatial correspondence between the lesion area and surrounding tissues, effectively restoring the geometric accuracy and structural continuity of the image model and providing a stable imaging basis for biopsy and surgical planning.

[0049] This invention enables dynamic spatial review and risk identification of biopsy and surgical pathways by generating coordinate anomaly maps and a list of high-risk locations. This process allows for the timely detection and adjustment of disrupted pathway segments before image shifts spread to the overall model, ensuring the planned pathway remains consistent with the actual tissue structure. This approach not only reduces sampling errors caused by needle path deviation but also lowers the surgical risks of vascular puncture and nerve injury, thus improving the safety and reliability of image-guided navigation and pathway planning. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0051] Figure 1 This is a flowchart of the multimodal image fusion method for thyroid and breast surgical biopsy and surgical planning according to the present invention.

[0052] Figure 2 This is a schematic diagram of the modules of the multimodal image fusion thyroid and breast surgical biopsy and surgical planning system of the present invention. Detailed Implementation

[0053] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0054] This invention provides, for example Figure 1 The multimodal image fusion-based surgical biopsy and surgical planning method for thyroid and breast cancer includes the following steps:

[0055] Multimodal medical imaging data and postural dynamic information of the thyroid and breast regions were collected and organized into a continuous time series in chronological order as the input basis for subsequent analysis and calculation.

[0056] To ensure that multimodal medical images and body dynamics information can form a continuous and comparable data sequence over time, thus providing unified basic data support for subsequent image fusion and spatial analysis, a multi-stage collaborative acquisition and sequence construction process is employed to achieve high-precision temporal correlation and spatial consistency mapping of image information. The specific implementation steps are as follows:

[0057] During the image acquisition preparation phase, considering the anatomical features, tissue density differences, and image resolution requirements of the thyroid and breast regions, the acquisition sequence and time intervals for different imaging modalities were determined to ensure spatial alignment between modalities during data acquisition. This phase involved establishing an acquisition plan table to uniformly define imaging parameters, slice thickness, time stamp formats, and coordinate reference systems for ultrasound, CT, and MRI equipment, ensuring consistency in temporal annotation and spatial reference across different modalities. To ensure consistency in body posture during acquisition, the subject's position was fixed and marked at this stage, recording posture information through surface markers to achieve spatial correspondence of the same position across different modalities. The data output at this stage includes an acquisition task information set and a time synchronization baseline, providing a temporal framework for subsequent synchronous image acquisition.

[0058] In the multimodal image synchronous acquisition phase, different modalities of image acquisition are executed sequentially under the constraint of a time baseline. Each image modality is acquired using a spatial reference system consistent with the previous step, ensuring the mappability of image coordinates across different modalities. During acquisition, the time node of each image frame is recorded using a time synchronization signal, and the acquisition results of each modality are temporarily stored according to time tags to form a time-stamped image set. In this way, MRI image data can maintain a one-to-one mapping relationship with CT and ultrasound images on the timeline, ensuring that subsequent data processing stages can perform sequence matching and spatial fusion with time as the main axis. Furthermore, by recording the start and end timestamps of each scan during acquisition, each set of image data possesses continuity and traceability in the time dimension. The output of this phase is a multimodal image raw dataset and a time-stamped table, which together constitute the basic information of the time-correlated images.

[0059] In the posture dynamics information recording stage, the subject's posture changes throughout the entire acquisition process are monitored and recorded, combining the image data acquisition and time-annotation results from the previous two steps. This stage uses continuous time nodes as references, acquiring the subject's surface posture parameters and respiratory displacement parameters through external sensors, and storing this information synchronously in the posture information recording set according to time labels. Due to slight postural shifts during image acquisition, this stage precisely matches posture change data with image acquisition time nodes to ensure spatial consistency between images, ensuring each frame corresponds to complete posture information. The acquisition of posture dynamics information includes multi-dimensional parameters such as head and neck posture angles, chest cavity range of motion, and shoulder position coordinates. This data will serve as an important basis for timeline correction and posture alignment in the subsequent time series processing stage. The posture information recording set output in this step reflects the dynamic trajectory of the subject throughout the entire image acquisition cycle, providing necessary spatiotemporal registration information for subsequent image sequence processing.

[0060] In the time series processing and construction phase, the multimodal image data, time stamp tables, and attitude information record sets obtained in the preceding steps are comprehensively utilized to perform temporal sorting and spatial alignment processing on all image data. First, image frames from different modalities are arranged sequentially according to the order of their time stamps, ensuring that image frames at the same time point can be aligned with the attitude parameters at the same moment. Next, based on the body position parameters in the attitude information record set, corrections are made for changes in body surface position caused by breathing or minor movements, ensuring that the same tissue structure maintains positional continuity and orientation consistency in images at consecutive time points. In this way, all image data is processed into a continuous time series, with each image frame containing its corresponding time point, body position parameters, and modal attribute identifier, thus forming the input data framework for multimodal fusion analysis. After the sequence construction is completed, the entire time series, with time as the main axis and attitude as the reference, achieves temporal synchronization and spatial coordination between different image modalities, providing a unified reference standard for subsequent image fusion and spatial migration analysis.

[0061] Frame-by-frame comparison of magnetic field interference signs and brightness extension trajectories in continuous time series is performed to extract regions with abnormal signal distribution, generate a candidate list of artifacts and identify the suspected range of artifacts.

[0062] This system accurately identifies anomalous signal regions caused by magnetic field interference in continuous sequences and compares interference signs with brightness extension trajectories in both time and space, thereby generating a candidate list of artifacts and identifying suspected artifact areas. A series of continuous, multi-stage processing steps are used to complete data screening, analysis, extraction, and regional labeling. The specific implementation steps are as follows:

[0063] In the frame-by-frame analysis stage, the continuous sequence with complete time order obtained in the previous step is used as the input data basis. A detailed scan and feature extraction are performed on the magnetic field signal distribution and brightness levels of each frame. Since different image modalities exhibit differences in signal performance, the frame-by-frame analysis observes the stability of the signal distribution over time by comparing the continuity of the time series, thereby determining whether there are local brightness fluctuations caused by non-tissue structures. Each frame is divided into multiple small analysis regions, and the signal intensity change trend of each region is recorded. When the brightness change in a certain region within consecutive frames exceeds the range of normal tissue signal changes, and this change does not follow the movement pattern of anatomical structures, it can be preliminarily identified as a sign of magnetic field interference. Through this frame-by-frame recording stage, a time-series database containing the signal distribution status of each frame is formed, providing basic information for subsequent interference feature comparison.

[0064] In the interference feature comparison stage, based on the time-series database generated in the previous step, the magnetic field signal characteristics and brightness extension trajectory of each image region are continuously compared frame by frame. This stage observes the spatial extension direction of the interference signal and the continuity of brightness changes by comparing adjacent frames on the time axis. If an area continuously exhibits unnatural brightness diffusion between adjacent frames, and the brightness distribution shows a stretched or non-linear extension pattern, it can be determined that the area has magnetic field interference characteristics. To avoid the brightness gradient of normal tissue being mistaken for interference signals, the interference feature comparison also combines temporal continuity for comprehensive judgment. When an area continuously exhibits brightness extension in time but lacks corresponding tissue structure support in spatial location, the area is further confirmed as a potential artifact interference point. During this process, the comparison results of each frame sequence are summarized to form an interference sign list, which includes the temporal range, spatial coordinate range, and brightness change characteristics description of each suspicious area, thus providing a quantitative basis for the next stage of abnormal area extraction.

[0065] In the anomaly region extraction stage, using the list of interference signs as the core input, the spatial range of each image region exhibiting interference characteristics is defined and integrated across consecutive frames. Since artifacts often extend across multiple frames, it is necessary to temporally merge and spatially aggregate anomaly regions at the same location in consecutive frames to form continuous anomaly signal distribution regions. This stage first identifies consecutively occurring interference regions at the same spatial location on the time axis, treating them as the same artifact evolution sequence; then, in the spatial dimension, neighborhood fusion is performed on these regions, integrating overlapping or intersecting anomaly signal ranges in adjacent frames into a unified region. In this way, scattered anomaly point groups can be integrated into anomaly signal distribution regions with complete boundaries. The output of this stage is a series of anomaly signal region sets containing temporal range, spatial range, and brightness characteristics. Each set represents a potential artifact distribution location, laying the foundation for generating the artifact candidate list.

[0066] In the artifact suspicion area identification stage, based on the abnormal signal region set generated in the previous stage, each abnormal region is classified and spatially labeled to form a complete artifact candidate list. This stage first filters areas potentially related to metallic interference based on the location characteristics and brightness distribution patterns of abnormal regions in the image, marking areas with obvious magnetic field interference as artifact suspicion areas. Next, based on time series information, these artifact suspicion areas are numbered in chronological order of appearance to correspond to specific time nodes in subsequent spatial migration identification. For each artifact suspicion area, its projection range in three-dimensional space is further identified. By combining time series and spatial coordinates, the artifact suspicion area can be located and corresponded between multimodal images. The final artifact candidate list includes the spatial coordinate range, time period information, and brightness extension feature description of each artifact suspicion area, providing a complete input basis for the identification of the true boundary and the boundary point of the abnormal region in subsequent steps.

[0067] Based on the list of artifact candidates, identify the boundary points between the real tissue boundary and the abnormal signal area in each suspected area, determine the spatial offset range caused by metal interference, and generate a coordinate anomaly draft containing offset coordinate information.

[0068] Based on the candidate list of artifacts, the boundary points between the real tissue boundary and the abnormal signal region in the suspected area are identified, and the spatial offset range caused by metallic interference is further determined, thereby generating a coordinate anomaly image containing offset coordinate information. Through continuous multi-stage data analysis and spatial correlation processing, the boundary separation of artifacts and real structures in the image and the determination of coordinate offset are achieved. The specific implementation steps are as follows:

[0069] In the boundary feature identification stage, based on the artifact candidate list formed in the previous process, the edge signal features of each suspected artifact region are comprehensively analyzed. Because the signal distribution of artifact regions differs from that of real tissue, in the same image frame, real tissue boundaries typically exhibit a continuous and smooth brightness gradient, while the signal distribution of artifact regions shows abrupt changes or irregular brightness extensions. To distinguish between the two types of regions, this stage uses temporal correlation analysis across consecutive frames to track the brightness evolution trend of suspected regions over time. When a region maintains a stable morphology and its brightness changes conform to the regular changes in tissue structure over time, the region is identified as a real tissue boundary; conversely, when a region exhibits discontinuous changes in brightness direction relative to its morphological boundary over time, and does not align with the tissue extension direction in adjacent frames, the region is identified as an anomalous signal region. Through this temporal-spatial dual-dimensional comparative analysis, the potential distribution range of real tissue boundaries and artifact interference boundaries can be initially delineated in the artifact candidate list, forming a boundary feature database containing region identifiers and temporal correspondences, providing a basis for the next stage of boundary point extraction.

[0070] In the boundary point extraction stage, data from the boundary feature database is used as input to identify edge intersection points based on brightness and morphological differences within each suspected region. Since there are often abrupt changes in brightness and spatial orientation at the signal transition points between real tissue boundaries and artifact regions, this stage analyzes the intersection direction of the two types of boundaries frame by frame to determine the location of the boundary line. Each suspected artifact region is divided into multiple small segments, and intensity abrupt changes are progressively detected along the brightness gradient direction within each segment. When an abrupt change point exhibits a fixed extended trajectory at the same location in adjacent frames, that location is marked as a boundary point. The boundary point extraction process involves not only signal changes in a single frame image but also the continuity of boundary points in the time series to ensure that the spatial trajectory of the same artifact remains continuous and consistent in the time dimension. After extraction, the coordinate information and time series numbers of all boundary points are uniformly stored in the boundary point set, providing accurate spatial positioning basis for the subsequent spatial offset determination stage.

[0071] In the spatial offset determination stage, based on the set of boundary points formed in the previous stage, the relative positional difference between the real tissue boundary and the anomalous signal region in space is quantified. Since magnetic field distortion caused by metallic interference can lead to misalignment of the artifact region and the real structure in spatial coordinates, this stage calculates the spatial offset range of the interference region by analyzing the positional change patterns of the boundary points in the time series. Specifically, the spatial coordinates of the same boundary point in consecutive time frames are compared. If there is a continuous displacement in the time series that cannot be explained by tissue physiological movement, then this displacement interval is determined as the offset range caused by metallic interference. Furthermore, the offset range is projected in three-dimensional space to determine its offset distribution boundaries along the body surface direction, depth direction, and lateral direction, thereby obtaining complete offset coordinate information in the spatial dimension. In this way, the spatial misalignment of the artifact region in consecutive frames can be accurately determined, and the offset can be correlated with specific time nodes to form a spatial offset dataset. This dataset not only contains the offset direction and magnitude of each artifact region but also includes the corresponding time period information and reference boundary number, ensuring that the subsequent generation of coordinate anomaly drafts maintains dual consistency in both space and time.

[0072] In the coordinate anomaly generation stage, by combining all the offset information in the spatial offset dataset, the coordinates of artifact regions in the image sequence are revised and anomaly annotations are performed to form a coordinate anomaly document containing offset coordinate information. This stage first arranges the offset data of each artifact region in chronological order, using time as the main axis, so that the offset information forms a continuous record in the time dimension. Then, in the spatial dimension, based on the three-dimensional coordinate position of each boundary point, the offset range is annotated in the corresponding image frame in the form of spatial projection. Each anomaly region is assigned an independent number, and its corresponding time period, spatial location, and offset direction are identified in the coordinate anomaly document, thus forming a bidirectionally indexable anomaly coordinate record in time and space. To ensure the overall consistency of the image sequence, the time label and attitude parameters of the original continuous sequence are also embedded during the coordinate anomaly document generation process. This ensures that the document not only reflects the specific range of spatial offset but also directly corresponds to the original image sequence in the subsequent path analysis stage, achieving cross-matching between the offset region and the surgical path. Through this process, the spatial relationship between the artifact region and the real tissue boundary is clearly defined, the offset range caused by metal interference is quantitatively expressed, and the resulting coordinate anomaly draft provides a precise spatial reference for subsequent path review and risk analysis.

[0073] Based on the coordinate anomaly report, the established biopsy path and surgical path are reviewed and analyzed to determine the spatial offset area traversed by the needle path, form a list of high-risk locations, and identify the path segments that need to be corrected.

[0074] Based on the coordinate anomaly report, a retrospective analysis of the established biopsy path and surgical path is performed to determine the spatial deviation areas traversed by the needle path and create a list of high-risk locations. Simultaneously, path segments requiring correction are identified. Through continuous, multi-stage data comparison and spatial mapping methods, integrated spatial correlation and risk localization processing is applied to the path and anomaly areas. The specific implementation steps are as follows:

[0075] In the path data loading and mapping phase, a spatial correspondence is established between the coordinate anomaly draft generated in the previous phase and the biopsy path and surgical path defined in the surgical planning phase. To ensure accurate correspondence between the path information and the offset coordinates in the coordinate anomaly draft, a unified spatial coordinate system is used as a reference during loading to align the coordinates of each path node and path segment in the biopsy path with the offset coordinate set in the coordinate anomaly draft. In this way, each coordinate point of the biopsy path can correspond to a specific location containing offset information in space, thereby achieving spatial mapping between path coordinates and anomaly draft coordinates. After mapping, a composite path dataset containing path nodes, path segments, time labels, and corresponding offset coordinates is formed in the system. This dataset fully reflects the spatial position of the needle path in the time series and its overlap with the metal interference offset area, providing structured data support for subsequent analysis.

[0076] In the spatial overlay analysis phase of the needle path and offset region, using the composite path dataset as input, the needle path trajectory is projected segment by segment in three-dimensional space onto the offset region identified in the coordinate anomaly report. Since the coordinate anomaly report includes the spatial offset range, offset direction, and temporal correspondence caused by metal interference, this phase analyzes the interaction between each spatial point traversed by the needle path and the offset region by unfolding it segment by segment in the time dimension. If any point in the needle path trajectory spatially intersects with or is located within the offset boundary of the offset region defined in the coordinate anomaly report, that path segment is marked as a region potentially affected by the offset. During the spatial overlay process, the relative relationship between the needle path direction and the offset direction is also considered. When the needle path direction and the offset direction are opposite or intersecting, the region is identified as a potentially high-risk puncture path. Through this continuous spatial overlay and directional relationship analysis, the interaction distribution of the needle path and offset region can be completely reconstructed in three-dimensional space, thus providing basic data for the risk identification phase.

[0077] In the risk path segment identification and classification stage, based on the spatial overlay results obtained in the previous step, all path segments marked as affected by offset are classified and categorized according to their risk levels. This stage first performs a preliminary classification of path segments based on the overlap ratio between the offset area and the path segment. When a path segment has a wide spatial overlap with the offset area and a significant directional difference, it is classified as a high-risk category; when a path segment only briefly intersects at the offset boundary, it is classified as a medium-risk category. Subsequently, combined with time-series data, a continuous analysis of the offset impact of the same needle path at different times is performed. If a path segment spans multiple offset areas in time and the offset direction changes complexly, it is identified as a key risk area. After completing the risk classification, all high-risk path segments are summarized in a spatial coordinate system to form a list of risk path segments containing path number, start and end coordinates, offset direction, and offset magnitude. This list reflects the spatial distribution pattern of needle paths affected by metal interference, laying the foundation for generating a high-risk location list in the next stage.

[0078] In the high-risk location list generation and path correction segment determination stage, the offset coordinate data from the risk path segment list and the coordinate anomaly report are comprehensively utilized to aggregate and mark the risk of all offset areas traversed by the biopsy path. This stage first aggregates all high-risk path segments according to spatial proximity, grouping path segments located in the same or adjacent areas into a single high-risk location unit to avoid duplicate recording. Then, based on the offset direction and spatial depth information, each high-risk location unit is 3D positioned and labeled, generating a high-risk location list containing location number, offset coordinates, path number, and offset range. This list not only records the interaction information between the biopsy path and the offset area but also clarifies the spatial offset and influence direction of each path segment. Finally, combining the chronological order of the biopsy path, the path segments in the high-risk location list are reordered to identify path intervals with consecutive offsets in the time series; these intervals are determined as path segments requiring correction. After determination, all path segments requiring correction are centrally marked, providing basic data input for subsequent dynamic adjustments.

[0079] For the high-risk location list, dynamic adjustment is performed on the continuous image sequence. A reverse synchronization operation is introduced into the time series, short static images are inserted between continuous images, and the depth direction is reversed based on the position difference between the preceding and following images. The time order of the fused images is rearranged so that the spatial position of the lesion area corresponds to the surrounding tissue again, thereby restoring the overall geometric accuracy of the image fusion model.

[0080] Based on a list of high-risk locations, continuous image sequences are dynamically adjusted to re-correspond the spatial positions of lesions and surrounding tissues in fused images and restore overall geometric accuracy. The specific implementation steps are as follows:

[0081] During the image temporal synchronization phase, using the path segments and spatial coordinates identified in the high-risk location list as references, images of time periods affected by metal interference are extracted from the complete continuous image sequence to form a set of high-risk time segments. To ensure temporal continuity, transitional image frames before and after the high-risk segments are retained during extraction, preserving the transitional relationship between adjacent time segments. Subsequently, the time tags of each time segment are readjusted to maintain a linear distribution on the time axis, and these segments are stitched back into the overall sequence according to their original chronological order. In this way, a continuous and traceable temporal structure is formed between the interfered area and the normal area on the timeline. At this stage, a one-to-one correspondence between time tags and spatial offset coordinates is established, ensuring that each image frame can find a node on the time axis that matches its spatial offset range, thus providing a basic temporal framework for subsequent reverse synchronization operations.

[0082] In the reverse synchronization frame interpolation stage, based on the time series formed in the previous step, short still images are inserted between consecutive frames for the time segments corresponding to the high-risk location list to extend the transition process of spatial changes in the time dimension. Each short still image represents the transitional state of image changes in the interference area, used to balance the brightness and displacement differences between consecutive frames. When inserting still images, the time interval and spatial offset direction of consecutive frames are referenced to ensure that the still image is positioned in the middle of the two frames, thus making the image changes on the time axis more balanced and continuous. After insertion, a time extension zone is formed in the high-risk area segment of the entire time series. This extension zone can absorb the image abrupt changes caused by metallic interference, making the image sequence present a smooth transition both visually and spatially. In this process, the still images not only play a delaying role on the time axis, but also maintain the intermediate coordinate relationship with consecutive frames in the spatial dimension, so that the offset areas that originally had abrupt changes can achieve temporal continuity.

[0083] In the depth-to-depth correction stage, based on the time series of inserted still images, the spatial position difference between consecutive images is back-calculated layer by layer in the depth direction. Since spatial offsets caused by metallic interference often exhibit stretching or compression effects along the depth direction, this stage derives the actual depth-to-depth offset range of each affected area by analyzing the spatial differences between consecutive frames in the time series. Specifically, the spatial position of the lesion area in the previous frame is compared with the corresponding area in the next frame to determine its displacement trend in the depth direction, and a correction reference layer is determined at the midpoint of this trend. This reference layer represents the spatial midpoint during the time transition. By adjusting the position of the still image to this reference layer, the lesion and surrounding tissue in the image can be realigned in the depth dimension. After the back-calculation correction is completed, the spatial positions of all affected areas in the time series form a new continuous distribution in the depth direction, thereby eliminating the hierarchical misalignment caused by metallic interference. This stage restores the matching relationship between spatial coordinates and the time series, providing a spatially consistent basis for the final temporal reordering.

[0084] In the chronological rearrangement stage, the entire timeline is reordered and fused by comprehensively utilizing reverse synchronous frame interpolation and image sequences corrected for depth and shallowness. First, the position of each image frame on the timeline is reconfirmed based on time labels, and the time frames are fine-tuned according to the spatial correspondence between the lesion area and surrounding tissues to maintain a stable spatial difference between consecutive frames. Then, image frames from high-risk areas are re-stitched with those from normal areas on the timeline to ensure a natural transition without any discontinuities. During the rearrangement process, the positional difference between preceding and following images is used as the sorting criterion to ensure that the rearranged time sequence maintains a consistent depth distribution and tissue relationship in space. After rearrangement, the spatial correspondence between the lesion area and surrounding tissues is re-established, arranging the tissue layers in the image fusion model according to their actual anatomical locations. The final rearranged sequence not only restores temporal continuity but also eliminates spatial misalignments caused by offsets, restoring the overall geometric accuracy of the fusion model.

[0085] This invention introduces temporal reconstruction and spatial offset correction mechanisms during multimodal image fusion, enabling synchronous correction of image regions affected by metallic interference in both temporal and spatial dimensions. By comparing magnetic field interference signs and brightness extension trajectories frame by frame, combined with reverse synchronization and the insertion of still images, temporal balance and spatial location restoration of artifact regions are achieved. This allows the fused image to re-establish a true spatial correspondence between the lesion area and surrounding tissues, effectively restoring the geometric accuracy and structural continuity of the image model and providing a stable imaging basis for biopsy and surgical planning.

[0086] This invention enables dynamic spatial review and risk identification of biopsy and surgical pathways by generating coordinate anomaly maps and a list of high-risk locations. This process allows for the timely detection and adjustment of disrupted pathway segments before image shifts spread to the overall model, ensuring the planned pathway remains consistent with the actual tissue structure. This approach not only reduces sampling errors caused by needle path deviation but also lowers the surgical risks of vascular puncture and nerve injury, thus improving the safety and reliability of image-guided navigation and pathway planning.

[0087] This invention provides, for example Figure 2 The multimodal image fusion-based thyroid and breast surgery biopsy and surgical planning system shown includes an image acquisition sequence module, an artifact detection module, an offset identification and calibration module, a path risk analysis module, and an image correction and adjustment module.

[0088] The image acquisition sequence module collects multimodal medical image data and postural dynamic information of the thyroid and breast regions, and organizes them into a continuous time series in chronological order as the input basis for subsequent analysis and calculation.

[0089] The artifact detection module compares the magnetic field interference signs and brightness extension trajectories in a continuous time series frame by frame, extracts the regions with abnormal signal distributions, generates a candidate list of artifacts, and identifies the suspected range of artifacts.

[0090] The offset identification and calibration module identifies the boundary points between the real tissue boundary and the abnormal signal area in each suspected area around the artifact candidate list, determines the spatial offset range caused by metal interference, and generates a coordinate anomaly draft containing offset coordinate information.

[0091] The path risk analysis module reviews and analyzes the established biopsy path and surgical path based on the coordinate anomaly draft, determines the spatial deviation area traversed by the needle path, generates a list of high-risk locations, and identifies the path segments that need to be corrected.

[0092] The image correction and adjustment module performs dynamic adjustments on the continuous image sequence for the high-risk location list. It introduces reverse synchronization operation into the time series, inserts short static images between continuous images, and performs reverse correction in the depth direction based on the positional difference between the preceding and following images. It rearranges the time order of the fused images so that the spatial position of the lesion area corresponds to the surrounding tissue again, thereby restoring the overall geometric accuracy of the image fusion model.

[0093] The multimodal image fusion method for thyroid and breast surgical biopsy and surgical planning provided in this embodiment of the invention is implemented through the aforementioned multimodal image fusion system for thyroid and breast surgical biopsy and surgical planning. For details on the specific methods and procedures of the multimodal image fusion system for thyroid and breast surgical biopsy and surgical planning, please refer to the embodiments of the above-mentioned multimodal image fusion method for thyroid and breast surgical biopsy and surgical planning, which will not be repeated here.

[0094] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for thyroid and breast surgical biopsy and surgical planning using multimodal image fusion, characterized in that, Includes the following steps: Multimodal medical imaging data and postural dynamic information of the thyroid and breast regions were collected and organized into a continuous time series in chronological order as the input basis for subsequent analysis and calculation. Frame-by-frame comparison of magnetic field interference signs and brightness extension trajectories in continuous time series is performed to extract regions with abnormal signal distribution, generate a candidate list of artifacts and identify the suspected range of artifacts. Based on the list of artifact candidates, identify the boundary points between the real tissue boundary and the abnormal signal area in each suspected area, determine the spatial offset range caused by metal interference, and generate a coordinate anomaly draft containing offset coordinate information. Based on the coordinate anomaly report, the established biopsy path and surgical path are reviewed and analyzed to determine the spatial offset area traversed by the needle path, form a list of high-risk locations, and identify the path segments that need to be corrected. For the high-risk location list, dynamic adjustments are performed on the continuous image sequence. A reverse synchronization operation is introduced into the time series, short still images are inserted between continuous images, and the depth direction is reversed based on the positional difference between the preceding and following images. The time order of the fused images is rearranged so that the spatial position of the lesion area corresponds to the surrounding tissue again.

2. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 1, characterized in that, The process of arranging continuous sequences is as follows: During the image acquisition preparation stage, based on the anatomical features, tissue density differences, and image resolution requirements of the thyroid and breast regions, the acquisition sequence and time interval of multimodal medical images are determined, and an acquisition plan is established to unify imaging parameters, scan slice thickness, time stamp format, and coordinate reference system. During the multimodal image synchronous acquisition phase, different modal image acquisition tasks are executed sequentially under the constraints of the time baseline, and the time node of each image generation is recorded using the time synchronization signal to form a time-stamped image set. During the body dynamic information recording stage, body surface posture parameters and respiratory displacement parameters are collected by external sensing devices and stored synchronously in the posture information recording set according to time tags; In the time series processing and construction stage, the image frames are sorted temporally and aligned spatially based on the time label and attitude information record set, and then processed into a continuous time series for multimodal fusion analysis.

3. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 2, characterized in that, The steps for generating a candidate list of artifacts and identifying the suspected artifact range are as follows: In the frame-by-frame analysis stage, the continuous sequence with complete time order is used as input data. The magnetic field signal distribution and brightness level of each frame image are scanned and features are extracted. The signal intensity change trend is recorded to form a time-series database. In the interference feature comparison stage, the magnetic field signal features and brightness extension trajectory are compared frame by frame using a time-series database, and a list of interference signs is formed based on the brightness diffusion pattern and spatial extension direction. In the abnormal region extraction stage, based on the list of interference signs, the abnormal signal regions at the same position in consecutive frames are temporally merged and spatially aggregated to obtain a set of abnormal signal regions. During the artifact suspected range identification stage, the abnormal regions are classified and spatially marked based on the abnormal signal region set, forming an artifact candidate list and identifying the suspected artifact range.

4. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 3, characterized in that, In the interference feature comparison stage, the spatial coherence and temporal continuity of the brightness extension trajectory are jointly judged by continuously comparing adjacent frames on the time axis. When the brightness distribution has no corresponding tissue structure support in the spatial location and continues to extend in time, the area is identified as an artifact interference point, and its corresponding time range and spatial coordinate range are recorded in the interference sign list.

5. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 3, characterized in that, The steps for generating a coordinate anomaly draft are as follows: In the boundary feature identification stage, based on the candidate list of artifacts, the edge signal features of suspected artifact areas are analyzed. By tracking the brightness evolution trend through time correlation, the true tissue boundary and the abnormal signal area are distinguished, and a boundary feature database is formed. In the boundary point extraction stage, the boundary feature database is used to detect the differences in brightness and shape within the suspected area, identify the intersection of the real tissue boundary and the abnormal area, and generate a set of boundary points. In the spatial offset determination stage, based on the set of boundary points, the spatial offset range caused by metal interference is determined by comparing the positional changes of the boundary points in continuous time frames, and a spatial offset dataset is formed. During the coordinate anomaly generation stage, the coordinates of the artifact regions are revised and anomaly annotations are performed based on the spatial offset dataset to generate a coordinate anomaly draft containing offset coordinate information.

6. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 5, characterized in that, In the spatial offset determination stage, the spatial coordinates of the boundary points in the continuous time frame are compared in multiple directions. The spatial offset range caused by metal interference is determined based on the displacement distribution in the body surface direction, depth direction and lateral direction. In the coordinate anomaly generation stage, the offset range is marked in the image frame in the form of three-dimensional spatial projection.

7. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 5, characterized in that, The process for creating the list of high-risk locations is as follows: During the path data loading and mapping stage, the spatial correspondence between the coordinate anomaly draft and the biopsy path and surgical path is established, and the path nodes and offset coordinates are aligned with a unified spatial coordinate system to form a composite path dataset. In the spatial overlay analysis stage of the needle path and the offset area, the needle path trajectory is projected segment by segment into the offset area in three-dimensional space using the composite path dataset to analyze the interaction between the path points and the offset area and identify potential high-risk path segments. In the risk path segment identification and classification stage, the risk level of the path segments is divided and classified according to the spatial overlay results, and a list of risk path segments is generated. In the stage of generating a list of high-risk locations and determining the path correction segments, the list of high-risk path segments and the offset coordinate data are combined to aggregate high-risk path segments and generate a list of high-risk locations, and determine the path segments that need to be corrected.

8. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 7, characterized in that, In the stage of generating a high-risk location list and determining the path correction segment, the spatial proximity relationship of high-risk path segments is aggregated, adjacent path segments are merged into high-risk location units, and the high-risk location units are three-dimensionally located and identified according to the offset direction and spatial depth information, generating a high-risk location list containing location number, offset coordinates, path number and offset range, which is used to determine the path segments that need to be corrected.

9. The method for thyroid and breast surgical biopsy and surgical planning based on multimodal image fusion according to claim 7, characterized in that, For the high-risk location list, the continuous image sequence is dynamically adjusted. A reverse synchronization operation is introduced into the time series, short still images are inserted between continuous images, and the depth direction is reversed based on the positional difference between consecutive images. The steps to rearrange the time order of the fused images are as follows: During the image time synchronization stage, using the path segments and spatial coordinate information in the high-risk location list as a reference, the images of the time periods affected by metal interference are extracted to form a set of high-risk time segments, and the time labels are corrected to establish the correspondence between time and spatial offset coordinates. During the reverse synchronization frame interpolation stage, short still frames are inserted between consecutive frames according to the time sequence. The still frames are used to balance the brightness and displacement differences between the preceding and following frames to form a time extension area. In the depth-to-depth correction stage, based on the time sequence of the inserted still images, the positional difference between the preceding and following images is back-calculated in the depth direction, and the correction reference layer is determined to complete the spatial alignment. In the time sequence rearrangement stage, the corrected time series are reordered and fused according to the time labels to restore the overall geometric accuracy of the image fusion model.

10. A multimodal image fusion system for thyroid and breast surgical biopsy and surgical planning, used to implement the multimodal image fusion method for thyroid and breast surgical biopsy and surgical planning as described in any one of claims 1-9, characterized in that, It includes an image acquisition sequence module, an artifact detection module, an offset recognition and calibration module, a path risk analysis module, and an image correction and adjustment module. The image acquisition sequence module collects multimodal medical image data and postural dynamic information of the thyroid and breast regions, and organizes them into a continuous time series in chronological order as the input basis for subsequent analysis and calculation. The artifact detection module compares the magnetic field interference signs and brightness extension trajectories in a continuous time series frame by frame, extracts the regions with abnormal signal distributions, generates a candidate list of artifacts, and identifies the suspected range of artifacts. The offset identification and calibration module identifies the boundary points between the real tissue boundary and the abnormal signal area in each suspected area around the artifact candidate list, determines the spatial offset range caused by metal interference, and generates a coordinate anomaly draft containing offset coordinate information. The path risk analysis module reviews and analyzes the established biopsy path and surgical path based on the coordinate anomaly draft, determines the spatial deviation area traversed by the needle path, generates a list of high-risk locations, and identifies the path segments that need to be corrected. The image correction and adjustment module performs dynamic adjustments on the continuous image sequence for the high-risk location list. It introduces reverse synchronization operation into the time series, inserts short static images between continuous images, and performs reverse correction in the depth direction based on the positional difference between the preceding and following images. It rearranges the time order of the fused images so that the spatial position of the lesion area corresponds to the surrounding tissue again.

Images