Lung cancer intraoperative navigation positioning method based on three-dimensional reconstruction and deep neural network

Abstract

The invention relates to the technical field of medical image processing and surgical navigation, in particular to a lung cancer intraoperative navigation and positioning method based on three-dimensional reconstruction and a deep neural network, which comprises the following steps of: based on multi-source optical and acoustic fusion, performing fractal wavelet multilayer decomposition on preliminarily purified data acquired in an operation, and performing multi-source optical and acoustic fusion; multi-modal characteristics of optical and acoustic channels and fractal measurement are obtained; using quantum heuristic multi-agent search to find a potential focus or a key area with serious deformation, and performing interpolation completion on a defect or noise part in a nerve radiation field; performing elastic deformation registration on the interpolation result and the lung three-dimensional model, and calculating a nonlinear deformation field and intra-operative tumor prediction coordinate data; and superposing the deformed lung model to an augmented reality environment through external tracking information to display the focus position. And when lung collapse or the traction amplitude exceeds the limit is monitored, searching and registration are repeated for real-time updating, and accurate navigation positioning under dynamic deformation of the lung is realized.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and surgical navigation technology, and in particular to a navigation and positioning method for lung cancer surgery based on three-dimensional reconstruction and deep neural networks. Background Art

[0002] With the increasing popularity of early lung cancer screening and the widespread application of minimally invasive surgical methods, accurate positioning of small lung nodules has become a key link in thoracic surgery. However, existing technologies mostly rely on preoperative images and static markers for positioning. Once the lungs collapse significantly due to breathing or mechanical traction during surgery, the original annotation information often deviates significantly, resulting in a decrease in nodule recognition accuracy and resection efficiency. The main reason is that existing technologies mostly use single image registration or finite element static prediction methods, which are unable to timely integrate multimodal data to iteratively correct dynamic deformations, and do not fully utilize the advantages of deep neural networks in high-dimensional feature extraction and adaptive learning. Therefore, it is difficult to quickly and accurately locate nodules after complex lung collapse in real time. Summary of the Invention

[0003] In response to the many problems existing in the above-mentioned prior art, the present invention provides a method for intraoperative navigation and positioning of lung cancer based on three-dimensional reconstruction and deep neural networks. The present invention first obtains intraoperative optical and acoustic information and performs fractal wavelet multi-layer decomposition to obtain multimodal features; then uses quantum-inspired multi-agent search and neural radiation field interpolation to fill in missing or noisy areas; then combines the three-dimensional lung model data to perform elastic deformation registration, generate a nonlinear deformation field and infer the coordinates of the lesion. If the traction or collapse exceeds the threshold, the search and registration are repeated to ensure that the positioning matches the actual intraoperative anatomy. The present invention not only ensures real-time self-adaptation, but also improves the detection rate of small lesions.

[0004] A method for intraoperative navigation and positioning of lung cancer based on three-dimensional reconstruction and deep neural network, comprising the following steps:

[0005] S1. Obtain original lung tomographic image data, perform denoising and segmentation, and generate lung 3D model data;

[0006] S2. During the operation, multi-source optical and acoustic information is collected and preliminarily purified. The obtained data is subjected to fractal wavelet multi-layer decomposition to obtain fractal wavelet multimodal data with optical and acoustic channel characteristics.

[0007] S3. locating key regions matching the three-dimensional lung model data in the fractal wavelet multimodal data using a quantum-inspired multi-agent search, interpolating and completing defective or noisy regions using a neural radiation field, and then performing elastic deformation registration with the three-dimensional lung model data to obtain nonlinear deformation field data and intraoperative tumor prediction coordinate data;

[0008] S4. Apply elastic transformation to the three-dimensional lung model data according to the nonlinear deformation field data, output deformed lung model data consistent with the intraoperative state, fuse the deformed lung model data with external tracking information to obtain an augmented reality overlay image and display the intraoperative tumor prediction coordinate data, and repeat steps S2 and S3 for dynamic updating when the lung collapse or tissue traction amplitude exceeds a threshold.

[0009] Preferably, step S1 includes: normalizing the grayscale range and filtering out noise on the original lung tomographic image data, and converting the segmented lung tissue voxels into lung three-dimensional model data using a grid reconstruction algorithm.

[0010] Preferably, step S2 includes: acquiring optical depth information and acoustic echo information respectively and then performing threshold filtering to obtain preliminary multimodal purified data, calculating fractal measure through fractal wavelet multi-layer decomposition, and fusing the optical channel amplitude and the acoustic channel amplitude into the fractal measure according to corresponding layered mapping to form fractal wavelet multimodal data.

[0011] Preferably, when performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, the low-frequency sub-band and the high-frequency sub-band are used to distinguish the smooth area and the texture mutation area respectively, the fractal measure is calculated for each sub-band and the optical channel amplitude and the acoustic channel amplitude are sequentially associated with the fractal dimension of the corresponding sub-band to obtain fractal wavelet multimodal data with multi-scale characteristics.

[0012] Preferably, step S2 further includes: before performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, using spatial filtering to remove isolated reflection points and eliminate abnormal amplitude areas to obtain multimodal preliminary purified data that meets the fractal measurement calculation requirements.

[0013] Preferably, step S3 includes: setting a quantum-inspired multi-agent search unit in the fractal wavelet multimodal data, each search unit performing multiple iterative searches on different spatial blocks based on a probability amplitude superposition pattern, and judging key areas with reference to anatomical features in the three-dimensional lung model data.

[0014] Preferably, after the key area is retrieved, the spatial part of the defect or noise is interpolated and completed through the neural radiation field, the interpolated and completed data is fused with the surrounding area and elastic deformation alignment is performed to output nonlinear deformation field data and intraoperative tumor prediction coordinate data.

[0015] Preferably, the elastic deformation registration includes: first rigidly aligning the fused data with the three-dimensional lung model data to correct the global coordinate error, then determining the local grid transformation parameters based on the difference between the optical channel amplitude, the acoustic channel amplitude and the fractal measurement, generating nonlinear deformation field data and obtaining intraoperative tumor prediction coordinate data.

[0016] Preferably, spatial transformation is performed on the three-dimensional lung model data based on the nonlinear deformation field data to obtain deformed lung model data, the deformed lung model data is aligned with external tracking information and an augmented reality overlay image is generated to display intraoperative tumor prediction coordinate data.

[0017] Preferably, after monitoring lung collapse or tissue stretching exceeding a threshold, steps S2 and S3 are repeated to update the nonlinear deformation field data and the deformed lung model data, so that the intraoperative tumor prediction coordinate data in the augmented reality overlay image corresponds to the real-time lung status.

[0018] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0019] The present invention combines fractal wavelet multimodal data, quantum-inspired multi-agent search and neural radiation field interpolation to achieve precise tracking of the lesion location after lung collapse; through elastic deformation alignment and augmented reality overlay, adaptive correction of lung morphological changes in the navigation image is achieved; and through dynamic threshold triggering repeated search and alignment, continuous updating of traction and collapse is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the process of the present invention;

[0021] Figure 2 Schematic diagram of fractal wavelet multi-layer decomposition in the present invention;

[0022] Figure 3 Schematic diagram of quantum multi-agent search and neural radiation field interpolation in the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] like Figure 1 As shown, a navigation and positioning method for lung cancer surgery based on three-dimensional reconstruction and deep neural network includes the following steps:

[0027] S1. Obtain original lung tomographic image data, perform denoising and segmentation, and generate lung 3D model data;

[0028] The acquired raw lung tomographic image data is typically presented as a series of continuous cross-sectional slices, often subject to noise and uneven intensity. For subsequent 3D reconstruction and navigation positioning, this image data requires grayscale normalization and noise filtering to ensure that pixels (or voxels) within the same grayscale range are comparable and minimize the impact of artifacts on subsequent segmentation and modeling.

[0029] Segmentation aims to distinguish lung tissue from surrounding tissue in 3D tomographic images, thereby obtaining a set of voxels containing only the lungs. If a deep neural network-based segmentation approach is used, a loss function can be defined during training to ensure that the segmentation network outputs lung tissue probabilities on a voxel-by-voxel basis. If a threshold or morphological approach is used, lung tissue can be removed based on the grayscale differences between it and bone, blood vessels, and other components in the tomographic images.

[0030] In the present invention, in order to construct high-precision 3D lung model data, the accuracy of the segmentation method can be evaluated. The commonly used evaluation index is the Dice coefficient. Its expression is as follows:

[0031]

[0032] Among them, Ω seg represents the set of lung tissue voxels obtained by segmentation, Ω ref represents the reference voxel set obtained by annotation or other calibration means, and |·| represents the number of elements in the set. When it is close to 0, it means that the segmentation result has a high degree of overlap with the reference area.

[0033] After segmentation, the voxel collection of lung tissue needs to be converted into 3D model data for use in subsequent algorithms. This can be achieved through mesh-based reconstruction, where voxel points that conform to the lung boundary are extracted and output as a polygonal mesh. The 3D reconstruction yields the anatomical contours and internal voxel information of the lung in 3D space, providing a geometric foundation for subsequent algorithms such as quantum-inspired multi-agent search, neural radiation field interpolation, and augmented reality overlay.

[0034] In practical applications, convolution filtering or non-local mean filtering can be performed on the tomographic image to remove random noise and small artifacts by weighting the similarity of the local neighborhood of each pixel. If the denoised voxel intensity is defined as I d (x,y,z), which can be obtained by:

[0035] I d (x,y,z)=I(x,y,z)-φ(I(x,y,z),Ω local )

[0036] represents the filtering operation, where I(x,y,z) represents the original voxel intensity and φ(·) represents the relationship between the voxel intensity and the neighborhood Ω local Related filter function; in the present invention, the filter function achieves denoising through similarity measurement to ensure that the lung contour and fine structure information are preserved.

[0037] If a deep learning network is used for segmentation, the above denoising result I d (x, y, z) is used as the network input, and the network outputs a lung tissue probability map, which is then converted into a binary voxel label through a threshold. If the traditional method is used, other tissue areas can be removed based on the lung grayscale threshold and connectivity determination method. Regardless of the method, the final segmentation result is a set of lung tissue voxels, denoted as Ω seg , and meet the preset Dice coefficient requirements.

[0038] Ω seg Surface extraction is performed on the corresponding voxel points to generate a polygonal mesh representing the lung surface. For the trachea, interlobar fissures, or other anatomical structures, sub-meshes or attribute labels can be created within the same mesh as needed. The final output 3D lung model data can be stored in voxel or mesh format to support subsequent algorithms.

[0039] In this invention, the output of step S1—the 3D lung model data—is directly used for the coordinate alignment and benchmarking required in step S2. It also provides a reference shape for the quantum-inspired multi-agent search in step S3. Because the 3D lung model data preserves the precise boundaries of lung tissue, subsequent fractal wavelet decomposition, neural radiation field interpolation, and elastic deformation registration all rely on this base model for comparison with the actual collapsed lung state.

[0040] If noise filtering and segmentation accuracy of the 3D lung model data are insufficient, subsequent deformation registration and augmented reality overlay will be biased, easily leading to large errors in the predicted coordinates of the tumor during surgery. By strictly controlling the denoising and segmentation quality in S1, the 3D lung model data can be more closely aligned with the actual anatomy, providing a reliable benchmark for quantum-inspired search and neural radiation field completion, ultimately achieving more precise results in the navigation and positioning of the present invention.

[0041] In specific implementation, the pixel spacing of the tomographic image can be set to less than 1 mm, and the Dice coefficient can be set to The value is controlled below 0.2 (the closer to 0, the higher the overlap). Under this condition, the generated 3D lung model data accurately depicts the contours around the alveoli. Next, step S2 will compare the collected optical and acoustic multimodal information with the model, thereby fully supporting quantum multi-agent search and neural radiation field interpolation in subsequent stages.

[0042] Preferably, step S1 includes: normalizing the grayscale range and filtering out noise on the original lung tomographic image data, and converting the segmented lung tissue voxels into lung three-dimensional model data using a grid reconstruction algorithm.

[0043] Raw lung tomographic image data often contains pixel (voxel) grayscale variations caused by varying scanning conditions and imaging noise. To ensure consistency between subsequent segmentation and 3D reconstruction, grayscale interval mapping is required. A lower bound α and an upper bound β can be set to linearly map the raw intensity I(x,y,z) to the interval [0,1]. The normalized expression can be defined as:

[0044]

[0045] Among them I norm (x, y, z) represents the normalized voxel intensity, and α and β represent the minimum and maximum effective grayscale values of the original image in the target lung region, respectively. This operation can achieve more stable contrast in subsequent segmentation networks or thresholding strategies.

[0046] Lung cross-sectional images contain scattered noise and instrument noise. To ensure the accuracy of lung tissue boundaries and details, filtering is required on the grayscale-normalized images. Convolution filtering or adaptive filtering can be used to remove isolated speckle without destroying the edges of small lesions.

[0047] Based on the segmented lung tissue voxels, mesh generation algorithms (commonly used methods include MarchingCubes or isosurface extraction) are used to convert the voxel spatial coordinates into a three-dimensional polygonal mesh. Each mesh surface is interpolated from the isosurfaces between adjacent voxels, ultimately generating three-dimensional model data that matches the lung's contours. This three-dimensional model is then used in subsequent quantum-inspired multi-agent search, neural radiation field interpolation, and intraoperative augmented reality overlay.

[0048] In practical applications, the grayscale distribution range of the original tomographic image in the lung area is first counted to determine α and β. If the original minimum grayscale is weaker than α, it is truncated; if it is greater than β, it is set to β. After this operation, normalization mapping is performed to obtain I norm (x,y,z), ensuring that subsequent analyses are based on the same interval.

[0049] Statistical filtering is often used to cluster adjacent voxels and remove abnormal outlier intensities. For example, the voxel neighborhood is defined as Γ(x, y, z), and the distance weight is calculated as follows:

[0050]

[0051] Among them I f (x, y, z) represents the filtered result, and ω(u, v, w) is the distance to the central voxel or the grayscale similarity weight (the sum after normalization is 1). This allows for cleaner candidate lung tissue regions to be obtained during the segmentation phase.

[0052] The segmented lung tissue voxels form a discrete set, which can be recorded as Ω lung , represents the set of coordinate points that meet the lung tissue determination. The present invention selects the isosurface extraction method to extract Ω lung Perform a mesh rebuild:

[0053] When scanning 3D voxel space, each cubic voxel unit is determined to cross the tissue-nontissue boundary. If a boundary exists within a voxel unit, the edges are interpolated and connected to form triangles. All units are then accumulated to form a complete polygonal mesh. This mesh is reused in subsequent registration and augmented reality stages.

[0054] Because the present invention subsequently requires quantum-inspired multi-agent search and neural radiation fields to accurately locate and complete the lungs, if the 3D lung model data lacks sufficient accuracy, quantum search will be limited by noise and incomplete segmentation, and the neural radiation field may also suffer from interpolation distortion in edge regions. By properly setting α and β in step S1 and employing filtering and mesh reconstruction, high-quality input can be obtained for subsequent fractal wavelet decomposition and interpolation registration.

[0055] The 3D lung model generated in step S1 serves as the preoperative anatomical benchmark. During the intraoperative multi-source optical and acoustic information acquisition phase, all deformable registrations are compared to this 3D model. Normalization and filtering enhance the accuracy of alveolar detail and vascular boundaries, enabling smoother shape matching using quantum-inspired multi-agent algorithms and volumetric interpolation of neural radiation fields.

[0056] In one embodiment, 128 slices can be selected for synthesis under the same sampling interval, and the grayscale distribution range of the lung region is calculated and its upper and lower boundaries are set to α = -1024 and β = 400 respectively. Then, a 3×3×3 neighborhood statistical filter is applied to the normalized image to remove local noise. After the segmentation is completed, the obtained Ω lung MarchingCubes generates a polygonal mesh of over 100,000 faces, preserving the basic shapes of the alveoli and pleural surfaces. This processing effectively supports the fractal wavelet analysis in step S2 and the quantum heuristic search in step S3, ultimately providing a reliable foundational geometric model for augmented reality navigation.

[0057] S2. During the operation, multi-source optical and acoustic information is collected and preliminarily purified. The obtained data is subjected to fractal wavelet multi-layer decomposition to obtain fractal wavelet multimodal data with optical and acoustic channel characteristics.

[0058] During surgery, optical depth projection and acoustic echo imaging can be used to determine the distance and signal strength to the lung surface or superficial layers. The optical method uses structured light or laser texture projection in conjunction with an image acquisition device to obtain optical channel amplitudes. The acoustic method uses echo algorithms to obtain acoustic channel amplitudes based on the reflection of acoustic pulses. These amplitudes describe the differences in optical and acoustic perspectives within the same lung region, laying the data foundation for subsequent multimodal fusion.

[0059] Mixed optical and acoustic data often contain outliers introduced by factors such as outlier reflection points and local shadows. To ensure feature accuracy during the subsequent fractal wavelet multi-layer decomposition process, the collected data must be pre-screened. This involves using a threshold to remove oversaturated or extremely weak signals, removing isolated points, and unifying the coordinate scale to align the optical and acoustic information within the same grid or point cloud structure.

[0060] After preprocessing is completed, it is necessary to perform wavelet transform on both the optical channel and the acoustic channel and calculate the fractal measure. Fractal wavelet multi-layer decomposition usually includes the following points:

[0061] The input multimodal preliminary cleansed data is subjected to multi-scale decomposition to separate the optical part from the acoustic part in each frequency band or sub-band.

[0062] Fractal measure is introduced in the sub-band to describe the irregularity of the lung surface or shallow layer, and the fractal dimension is calculated by statistically analyzing the distribution of local sub-band coefficients.

[0063] The optical amplitude A of each sub-band opt and acoustic amplitude A ac Mapping to fractal dimension D sub , forming a set of multi-channel features.

[0064] In practical applications, in order to ensure that the optical channel amplitude and the acoustic channel amplitude correspond to the same lung surface position, it is necessary to unify the coordinate scale or scanning step used when collecting the two types of data. If the optical or acoustic echo exceeds the given amplitude range, it is truncated and recorded as the boundary value to avoid extreme noise interference. opt (x,y) and acoustic amplitude A ac (x, y) are paired by position (x, y) to obtain multimodal preliminary purified data Ω mm At this time Ω mm Each position contains two-channel information: optical amplitude and acoustic amplitude.

[0065] In the present invention, the multi-layer wavelet decomposition level is set to L, and the mm Apply the decomposition process:

[0066] Multi-layer wavelet transform: Perform wavelet transform on the optical amplitude and acoustic amplitude in parallel to generate several frequency bands or sub-bands. and They represent the subband coefficient sets corresponding to optics and acoustics in the lth layer (l≤L).

[0067] Fractal measure calculation: In each sub-band coefficient set, the fractal dimension is calculated using the spatial neighborhood characteristics. If C sub (x,y) represents the subband coefficients, and a fractal measure F(x,y) is defined to measure the texture or fluctuation of the region. In this process, local concavity and convexity can be estimated using box dimension or correlation dimension methods. A fractal measure F(l) is output through cumulative summation or fitting, associated with the corresponding l-layer subband.

[0068] Multimodal merging: combining optical amplitude and acoustic amplitude The same fractal measure F(l) is spliced in channel form, ultimately generating multi-channel features at layer l. After accumulating all layers, fractal wavelet multimodal data is obtained, storing optical-acoustic fusion and fractal structural features at multiple levels.

[0069] In this paper, quantum-inspired multi-agent search requires identifying the local areas on the lung surface or shallow layers that best reflect lesion characteristics to support neural radiation field interpolation and elastic deformation registration. The optical and acoustic channels and fractal measures contained in fractal wavelet multimodal data help the search process locate tiny surface bumps or local mutations at multiple scales, improving the perception of lesion location and deformation signals.

[0070] Compared to single-channel or simple filtering, wavelet decomposition can capture large-scale surfaces in low-frequency subbands and detail in high-frequency subbands. Fractal measurements provide a quantitative basis for determining irregular deformations. By using fractal wavelet multimodal data, quantum agents can more quickly focus on areas of lung change where optical and acoustic differences are most pronounced. Subsequent neural radiation fields can then complete high-quality interpolation of these areas.

[0071] Preferably, step S2 includes: acquiring optical depth information and acoustic echo information respectively and then performing threshold filtering to obtain preliminary multimodal purified data, calculating fractal measure through fractal wavelet multi-layer decomposition, and fusing the optical channel amplitude and the acoustic channel amplitude into the fractal measure according to corresponding layered mapping to form fractal wavelet multimodal data.

[0072] Optical depth information is obtained by projecting a structured light pattern onto the lung surface and recording the reflected image. Acoustic echo information uses the amplitude of the reflection of the sound wave pulse in the lung tissue layer to obtain another dimension of surface or shallow information. The two are complementary, helping to maintain usable data even when the line of sight is partially obstructed or there is heavy interference.

[0073] Because illumination saturation or extremely weak echo intensity may lead to outliers, it is necessary to impose upper and lower amplitude thresholds on the amplitude of each channel. If the optical depth information is recorded as I opt (x,y), the acoustic echo information is recorded as I ac (x,y), then according to the pre-set threshold interval [α opt ,β opt ] and [α ac ,β ac ]Truncate those that exceed the range to the boundary value and obtain the multimodal preliminary purified data Ω clean Ω clean The same coordinate (x, y) contains a pair of amplitudes [I opt (x,y),I ac (x,y)], ensuring that subsequent analysis is based on consistent reference coordinates.

[0074] Ω cleanA multi-layer wavelet transform is performed to decompose components of different scales or frequencies. The coefficient distribution of the optical and acoustic amplitudes is extracted within each subband, and a fractal measure is calculated to characterize the texture or morphological irregularity within that subband. This fractal measure can be used to perform local clustering statistics on the subband coefficients based on methods such as the box dimension or correlation dimension.

[0075] After obtaining the fractal measure for each subband layer, the optical and acoustic channel amplitudes are mapped to the fractal measure maps of the corresponding layers, and the results are superimposed in the channel dimension to form fractal wavelet multimodal data. This data not only preserves local irregularities at different scales but also contains optical and acoustic multi-channel features, facilitating subsequent deformation search and interpolation completion for complex situations such as local collapse, lesion edges, and blood occlusion.

[0076] In practical applications, after optical and acoustic acquisition is complete, it is necessary to ensure that they are aligned on the same coordinate grid. This can be achieved by combining external tracking information or the inherent registration of structured light and acoustic probes to ensure that (x,y) corresponds to the same area. If the sampling resolutions of the two are inconsistent, the grid size can be unified during interpolation.

[0077] Need to set α in advance opt ,β opt With α ac ,β ac If the optical or acoustic amplitude is detected to be outside this range, the amplitude at that location is limited to the boundary value (α or β can be selected). This can filter outliers caused by strong reflections or very weak noise, ensuring that subsequent wavelet and fractal analysis is more accurate.

[0078] In each sub-band, the local neighborhood size is set (e.g., r×r), the self-similarity of the sub-band coefficients in the neighborhood is counted, and the fractal value F is output by calculating the box dimension or correlation dimension. l (x, y), where l represents the lth layer subband. If l is larger, it means a deeper layer or a higher frequency subband.

[0079] The amplitude of the optical channel and the acoustic channel I opt ,I ac According to the layer l corresponding to F l On the above, multi-dimensional features are formed. It can be written as Finally, all sub-bands are aggregated to form fractal wavelet multimodal data, which contains multi-scale information and encompasses the complementary characteristics of optics and acoustics.

[0080] When localized stretching or tissue collapse occurs on the lung surface, significant differences in optical or acoustic amplitudes may occur within a sub-band. Fractal measurements can quantitatively reveal the edge irregularities caused by these differences, allowing subsequent steps to focus on these areas and improve the accuracy of identifying lesion locations or deformed regions.

[0081] If the optical channel amplitude drops sharply at certain points due to occlusion by blood or instruments, the acoustic channel amplitude can still retain some information; and vice versa. By mapping both to fractal metrics, it helps to effectively compensate for single-channel loss in multimodal fusion and avoid misjudgments in subsequent steps.

[0082] In one embodiment, during the operation, the optical depth and acoustic echo are collected in steps of 5 mm. opt =50,β opt =200,α ac =20,β ac =100 as the threshold interval. Optical amplitudes below 50 are considered invalid reflections, and acoustic amplitudes above 100 are considered as oversaturation points and limited to 100. 3-layer wavelet transform is performed to obtain subband coefficients, and the box dimension estimation method is used to calculate the local fractal value F of each subband. l (x,y). Splice the optical amplitude and acoustic amplitude at the [x,y] position corresponding to the l layer to F l (x,y). If F is found in the high frequency sub-band l If (x,y) increases significantly, it indicates that there may be a local edge mutation, which can indicate a suspicious lesion area for subsequent quantum agent search.

[0083] Preferably, Figure 2 As shown in the figure, when performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, the low-frequency sub-band and the high-frequency sub-band are used to distinguish the smooth area and the texture mutation area, respectively. The fractal measure is calculated for each sub-band, and the optical channel amplitude and the acoustic channel amplitude are sequentially associated with the fractal dimension of the corresponding sub-band to obtain fractal wavelet multimodal data with multi-scale characteristics.

[0084] Fractal wavelet multi-layer decomposition consists of two core elements: a wavelet transform, which decomposes input data into different frequencies or scales to distinguish between smooth (low-frequency) and texture abruptness (high-frequency); and a fractal measure, which quantitatively describes the local self-similarity or irregularity within different subbands. Combining these two elements allows for simultaneous acquisition of multi-scale texture information and fractal features across the lung surface or shallow layers.

[0085] Low-frequency subbands preserve the overall trend of the image or point cloud, primarily covering macroscopic fluctuations or slow changes in the lung surface. If the low-frequency component is significantly smooth in this method, it indicates that the area is closer to the main alveoli or has no obvious lesion protrusions.

[0086] High-frequency sub-bands: Highlight details such as edges, lesion fissures, or vascular attachment points. Intensity changes in high-frequency sub-bands map to local sharp structures or surface wrinkles caused by traction, which is particularly critical for locating small nodules during surgery.

[0087] In each subband, a fractal measurement operation is performed on the amplitudes of the optical channel and the acoustic channel to characterize the degree of self-similarity in the same area: if the fractal measurement is higher, it means that the irregularity is greater, which may be a tumor protrusion or a locally collapsed edge; if the optical channel amplitude is weak but the acoustic channel amplitude is strong, it means that the optical obstruction is severe but the acoustics are penetrable. The fractal measurement can combine the differences between the two channels to indicate the characteristics of potential lesions.

[0088] Wavelet multi-layer decomposition, set the decomposition layer number to L. In the decomposition of the lth layer (l≤L), a low-frequency subband LF can be obtained l And multiple high frequency sub-bands HF l,i , where i represents different high frequency components in horizontal, vertical or diagonal directions. Area mapping, the low frequency subband LF l Marked as smooth area, corresponding to the overall outline of the lungs; the high frequency sub-band HF l,i The marked areas are texture mutation areas, corresponding to local edges or lesions.

[0089] Local neighborhood, define the neighborhood radius r for each subband, in HF l,i or LF l In the present invention, the wavelet coefficients in the neighborhood are statistically analyzed; the fractal dimension is approximated. In this invention, the box dimension (or correlation dimension) can be used to perform self-similarity estimation on the coefficient distribution in the neighborhood to obtain the fractal dimension D l (x,y). If D l The high (x,y) value indicates that the irregularity of the area is enhanced.

[0090] After each sub-band corresponds to the fractal dimension, the optical channel amplitude A opt Harmonic channel amplitude A ac Map to D according to the layered correspondence l (x,y). If the fractal dimension is D l (x,y), a channel array is available By stacking l layers (l = 1 to l) of data, we obtain multi-scale fractal wavelet multimodal data, which records the differences between the optical and acoustic channels at each scale and the corresponding fractal measures.

[0091] For example, set l = 3 layers of wavelet decomposition to process the optical and acoustic data separately. The first layer outputs a set of high-frequency subbands and a low-frequency subband; the low-frequency subbands are further decomposed in the second and third layers. In the first layer's high-frequency subband, if the fractal dimension D1(x,y) is detected to be greater than the threshold γ, and the optical channel amplitude is significantly reduced but the acoustic amplitude remains within the normal range, this indicates that the area may be a nodule obscured by blood and requires special attention in subsequent search and interpolation. After three layers of processing, the optical and acoustic amplitudes and fractal measures are combined to ultimately generate fractal wavelet multimodal data with multi-scale characteristics, providing rich spatial information for the next step of quantum-inspired multi-agent search.

[0092] Low-frequency subbands can easily identify areas of gentle breathing or normal alveolar regions, allowing for subsequent ignoring of unnecessary searches. High-frequency subbands can also reveal lesion boundaries, traction wrinkles, or vascular adhesions, providing valuable insights into quantum-inspired multi-agent search.

[0093] If the optical channel amplitude is weakened due to blood stains or occlusion, but the acoustic channel amplitude remains intact, this difference is highlighted in the high-frequency subband of the fractal dimension, prompting the subsequent search algorithm to focus on this abnormal area. The reverse is also true. This effectively improves the capture rate of subtle lung lesions in complex surgical environments.

[0094] Fractal measurements can provide a quantitative reference for local irregularities in neural radiation fields. When interpolating the radiation field to fill in defective or noisy areas, points with significant fractal properties in the high-frequency subband can be used as key interpolation nodes to enhance the accuracy of reconstruction of lesions or collapsed edges.

[0095] Preferably, step S2 further includes: before performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, using spatial filtering to remove isolated reflection points and eliminate abnormal amplitude areas to obtain multimodal preliminary purified data that meets the fractal measurement calculation requirements.

[0096] Optical depth and acoustic echo information often contain isolated local reflection points or large amplitude noise. These interference points can cause anomalies in subband coefficients during wavelet multi-layer decomposition. Spatial filtering analyzes the amplitude distribution of pixels (or point cloud coordinates) within a local neighborhood, identifying and removing isolated values or areas with large amplitude jumps, making the overall data more suitable for fractal measurement calculations.

[0097] Both the optical and acoustic channels have upper and lower amplitude limits. Points exceeding the preset amplitude thresholds can easily cause distortion in the fractal dimension estimation. For example, strong light reflections or very weak acoustic attenuation points can create noticeable artifacts in subsequent subband analysis. Eliminating these areas of amplitude anomalies improves the local reliability of subsequent wavelet and fractal measurements.

[0098] In practical applications, the filter neighborhood size is set to p×p; the amplitude distribution of each pixel (or point cloud coordinate) (x, y) of the multimodal preliminary cleansed data is statistically analyzed within the neighborhood. opt (x,y) and I ac If (x,y) deviates too much from the neighborhood mean, it will be marked as an isolated reflection point. Amplitude clipping or removal operation is performed on the marked area, and the filtered data Ω is output. filtered ; The data maintains a smooth transition between the amplitudes of adjacent regions and does not contain outlier fluctuations.

[0099] Determine the optical channel threshold interval [α opt ,β opt ], acoustic channel threshold interval [α ac ,β ac ]. If I opt (x,y)<α opt or I opt (x,y)>β opt , then the pixel or point is marked as abnormal in amplitude and removed; the same applies to the acoustic channel. After the merging is completed, the extreme abnormal amplitude area is removed based on the filtering result to ensure the final output Ω filtered Satisfies the amplitude stability required for fractal measurement calculation.

[0100] After spatial filtering and amplitude removal, Ω filtered Most isolated reflection points are removed, and regions with extreme amplitudes are also eliminated. Subsequent fractal wavelet multi-layer decomposition, when calculating the fractal dimension within the subbands, is no longer affected by outliers, allowing for more accurate distinction between smooth and abrupt structures on the lung surface. This provides a reliable data foundation for quantum-inspired search and neural radiation field interpolation.

[0101] If isolated reflection points or abnormal amplitude data are not removed, falsely high fractal dimensions will appear during fractal measurement estimation, leading to subsequent steps to misidentify certain areas as having lesions or collapse. This spatial filtering and amplitude removal process eliminates the influence of interfering points, allowing the high- and low-frequency subbands after multi-layer decomposition to more accurately reflect the true deformation and tissue structure of the lungs.

[0102] Wavelet transforms are sensitive to spike noise. If there is a localized excessive amplitude mutation, it will propagate across all subbands, leading to erroneous judgments by quantum agents or neural radiation fields. Spatial filtering can eliminate the impact of these spikes at the source, reducing ineffective interference in high-frequency subbands and ensuring a stable estimation of the fractal dimension.

[0103] In one embodiment, the filter neighborhood is set to p = 5 corresponding to a 5×5 neighborhood. If the optical or acoustic amplitude of a point (x, y) differs from the neighborhood mean by more than δ, it is considered an isolated reflection point and interpolated or removed. Threshold allocation, set α opt =40,βopt =200,α ac =15,β ac =90. If I opt If (x,y) is greater than 200, it is set to 200; if it is less than 40, it is set to 40. The acoustic channel amplitude is processed accordingly. This process corrects most localized scintillation points and extremely weak wave points, and the multimodal preliminary cleansing data maintains overall continuity, enabling subsequent wavelet decomposition to accurately extract texture features and calculate fractal measures across multiple channels.

[0104] S3. locating key regions matching the three-dimensional lung model data in the fractal wavelet multimodal data using a quantum-inspired multi-agent search, interpolating and completing defective or noisy regions using a neural radiation field, and then performing elastic deformation registration with the three-dimensional lung model data to obtain nonlinear deformation field data and intraoperative tumor prediction coordinate data;

[0105] This paper applies a quantum-inspired multi-agent search within the space of fractal wavelet multimodal data to rapidly locate key regions consistent with 3D lung model data within multi-scale features. The core concept of quantum heuristics lies in the fact that each agent possesses probabilistic superposition properties similar to quantum states, enabling multiple parallel searches in high-dimensional space, thus escaping the local optimum that traditional searches often fall into.

[0106] It can cross complex or noisy areas and give priority to detecting high-fractal, high-difference or irregular areas where lesions may exist. When compared with the 3D lung model data, it can quickly focus on the areas that are most likely to match the lung anatomy protrusions or collapses, facilitating subsequent interpolation and registration.

[0107] Neural Radiance Field can map optical and acoustic information as well as fractal measures into an implicit space, and achieve continuous approximation of locally missing data by performing interpolation or reconstruction on defective or noisy areas.

[0108] If the optical channel has a large number of omissions due to blood or instrument shadows, and the acoustic channel also has echo blind spots, it will directly affect the accuracy of subsequent elastic deformation registration; the neural radiation field can learn the radiation distribution in the neighborhood and interpolate at these noises or defects to restore the possible morphology of the real lung surface or shallow tissue, generating more complete fusion data for registration.

[0109] After the key areas are interpolated and completed, the data needs to be elastically aligned with the three-dimensional lung model data to obtain a deformation field that characterizes the intraoperative collapse or traction state, and from this, infer the three-dimensional coordinates of the tumor under the current lung morphology.

[0110] Correcting large localized lung deformation caused by breathing and mechanical traction aligns the fractal wavelet multimodal data with the 3D lung model data in the same coordinate system. Outputting nonlinear deformation field data allows for real-time lesion localization in augmented reality or other navigation scenarios. Intraoperative tumor prediction coordinate data is obtained to guide surgeons in determining the precise extent of resection.

[0111] In practical applications, a number of search agents are randomly or based on heuristic rules arranged in the fractal wavelet multimodal data space (including optical channels, acoustic channels and fractal measures). Each agent contains a quantum state vector and has the characteristics of probabilistic transition and parallel exploration. The agent will read the local multimodal features [A opt (x,y),A ac (x,y),F(x,y)] and compares the similarity with the anatomical or segmentation information of the region in the 3D lung model data. If the similarity exceeds the threshold γ, it is considered a possible key region. Through quantum heuristic probability superposition and collapse, the agent jumps through different regions to eliminate parts that differ greatly from the model data, and finally locates a set of key regions Ω. key .

[0112] If in the critical area Ω key If a large number of optical and acoustic channels are missing or oversaturated, these points can be marked as "interpolation pending" areas. The neural radiation field learns the radiation distribution function in the neighborhood by multimodal data (optical, acoustic, fractal measurement), denoted as R θ (x, y) (where θ is a network training parameter) can be used to interpolate the defect point and correct the amplitude. After interpolation, the local data is merged with the surrounding complete data to generate a more continuous multimodal feature, which reduces the uncertainty caused by noise or holes in the subsequent registration.

[0113] The interpolated key area fusion data and the lung 3D model data are first subjected to global rigid body transformation to eliminate rotation and translation, and then the deformation is estimated based on local features (such as optical-acoustic channel amplitude or fractal measure distribution). key If significant deformation occurs within the lung, elastic mesh optimization can be used to solve the nonlinear deformation field. The deformation vectors Δ(x,y) corresponding to each coordinate (x,y) in the initial model are accumulated to form a global field Φ(Ω). Φ is used to represent the collapsed or stretched state of the lung in real intraoperative coordinates. The "preoperative tumor reference coordinates" are mapped to the current collapsed 3D coordinates using Φ to obtain the "intraoperative tumor predicted coordinate data," providing precise lesion location for the next step of augmented reality navigation.

[0114] Preferably, Figure 3As shown, step S3 includes: setting a quantum-inspired multi-agent search unit in the fractal wavelet multimodal data, each search unit performing multiple iterative searches on different spatial blocks based on the probability amplitude superposition pattern, and judging the key areas with reference to the anatomical features in the three-dimensional lung model data.

[0115] Quantum-inspired multi-agent search draws on the concepts of quantum superposition and quantum collapse, enabling each search unit to explore high-dimensional feature spaces in parallel and dynamically adjust search directions during iterations. Unlike traditional genetic algorithms or particle swarm algorithms, quantum-inspired multi-agent search helps agents escape local optima while maintaining rapid convergence to the global model.

[0116] Fractal wavelet multimodal data preserves features such as optical and acoustic amplitudes and multiscale fractal measurements, enabling the quantum agent to determine whether each spatial block matches potential lesion characteristics, such as tumor protrusions, collapsed edges, and vascular adhesions. This multidimensional information enables the quantum agent to accurately locate suspicious lesion areas even in the presence of noise or partial occlusion.

[0117] During the search process, if the search unit finds that a certain area is highly mutated from the fractal wavelet multimodal data, it is necessary to simultaneously compare the anatomical structure in the 3D lung model data (such as the lobe fissure, bronchial direction, etc.) to determine whether the mutation is a normal anatomical variation or a possible lesion / collapse, thereby completing the screening of key areas.

[0118] In practical applications, space partitioning is to divide the space covered by the fractal wavelet multimodal data into grids or adaptive partitions, and each block contains the optical amplitude A. opt , acoustic amplitude A ac And fractal measure F and other characteristics. Agent initialization: put a number of quantum agents in each block or a group of blocks, each agent has a quantum state ψ j , using a probability amplitude superposition model to represent its exploration tendency within each block. The superposition and collapse mechanism: During iteration, each agent jumps between spatial blocks based on a probability distribution. When encountering an area with similar characteristics to the 3D lung model data, its probability amplitude increases; if the difference is significant, the probability amplitude decreases. After several rounds of iteration, the agent eventually collapses into a few key areas.

[0119] In the fractal wavelet multimodal data block, let the channel vector [A opt ,A ac ,F]. Compare it with the anatomical prior of the corresponding area of the 3D lung model data (such as the average curvature within the local radius or the lung segment identification mark), and record it as the similarity δ. If δ>γ (preset threshold), it indicates that it matches the model data.

[0120] If the agent has a high similarity in the current block, the probability of exploring its neighboring blocks will be increased; if the similarity is low, the agent has a certain probability of "quantum tunneling" to the remote block to avoid falling into the wrong local extreme value. After iteration, the agent gradually converges to the high similarity area in the global scope, forming a candidate key area set Ω key .

[0121] Ω key For each candidate block, the anatomical morphology (e.g., apex, lobar fissure, bronchial branching) in the 3D lung model data is examined. If the multimodal fractal characteristics of the region are highly consistent with anatomical expectations, it is further marked as a possible lesion or collapse area. If there is a clear conflict between the tracheal direction or vascular location in the model data, it can be excluded as a false or noise point.

[0122] The points that meet the anatomical prior and have abnormal fractal characteristics are marked as the final key area Ω′ key , providing key focus for neural radiation field interpolation and subsequent deformation registration.

[0123] Compared with traditional single morphology or threshold search, quantum agent's multiple iterative concurrent exploration can complete a global scan of the multimodal feature space in a short time, and determine potential abnormal areas based on anatomical features to avoid omissions or misjudgments.

[0124] The fractal wavelet multimodal data has been filtered and thresholded in the early stage. The quantum agent combines this with the three-dimensional lung model data for double screening, greatly reducing the risk of noise points being mistaken for real lesions.

[0125] In one embodiment, 50 quantum agents are set, and the blocks are divided into 16×16 grids. The optical amplitude A is recorded for each block. opt (x,y), acoustic amplitude A ac (x, y) and the fractal measure F(x, y). The quantum agent initially has a uniform probability distribution, and at each iteration, the probability amplitude is adjusted based on the similarity score δ. If δ exceeds 0.8, the agent remains in the neighborhood and continues to search deeper; if δ is below 0.4, it randomly jumps to distant neighboring blocks for exploration. After 20 rounds, the agent focused on three suspicious areas, comparing them with the lobar fissures and bronchial directions in the 3D lung model data. Ultimately, two of these areas were identified as highly suspicious lesions and output to the next stage for interpolation completion and elastic registration.

[0126] Preferably, after the key area is retrieved, the spatial part of the defect or noise is interpolated and completed through the neural radiation field, the interpolated and completed data is fused with the surrounding area and elastic deformation alignment is performed to output nonlinear deformation field data and intraoperative tumor prediction coordinate data.

[0127] Neural Radiance Field interpolates missing or noise-contaminated areas by learning implicit mappings in multi-channel space (including optical amplitude, acoustic amplitude, fractal measures, etc.). The previous step has already identified the location and general outline of key areas, but if there are large areas of numerical holes or outliers, these areas need to be filled in.

[0128] The features of the adjacent normal area are sampled, and the radiation value distribution function R is used θ (x, y, z) (where θ represents the network parameters) is used to infer the eigenvalues of the missing points, so that the overall multimodal data maintains continuity and smooth transition in space, thereby avoiding excessive errors in subsequent deformation registration.

[0129] To ensure that the interpolated data remains connected to the surrounding normal area in both optical and acoustic channels, elastic deformation registration of the overall data, including the key areas, is performed with the preoperative 3D lung model. This process can be performed using a grid-based or control point-based approach, first performing rigid body correction of global coordinates, then performing nonlinear matching based on local feature differences (optical, acoustic, and fractal measurements), and outputting nonlinear deformation field data.

[0130] Correct the structural deformation of the lung caused by intraoperative traction and collapse; use the nonlinear deformation field data to map the preoperative tumor reference coordinates to the three-dimensional position of the current collapsed state to obtain the intraoperative tumor prediction coordinate data.

[0131] In practical applications, after the key area retrieval is completed, it is determined whether the amplitude loss or noise severity of the optical and acoustic channels exceeds the threshold γ. d If it exceeds, it is recorded as a "to be completed area" and the characteristic values of the boundary and the adjacent normal area are recorded. Learning mapping of discrete sample points in the range:

[0132] R θ (x,y,z)→(A opt ,A ac ,F)

[0133] Where x, y, z represent coordinates or indices (possibly extended to spatial dimensions), A opt and A ac represent the optical and acoustic amplitudes respectively, and F represents the fractal measure.

[0134] For each point (x0, y0, z0) in the completion area, the characteristic values of several known points in its neighborhood are input into the radiation distribution function R θ , update θ based on backpropagation and gradient descent, so that the interpolated value output by the network is as close as possible to the actual channel amplitude and fractal information measured in the neighborhood. Finally, the completed local data is output.

[0135] The interpolated local data Ω interp Combined with the surrounding normal area to form the overall fusion data Ω meerged Linear transition or local smoothing can be performed at the seams to ensure no obvious mutation. merged The lung 3D model data is first rigidly aligned (translated, rotated, scaled) to correct large-scale errors. The nonlinear deformation field φ is introduced to adjust the optical amplitude A in the key area. opt , acoustic amplitude A ac The fractal measure F is iteratively aligned with the corresponding features on the 3D lung model mesh. By minimizing the difference function (such as ) method to determine the elastic transformation matrix or B-spline grid coefficients and output nonlinear deformation field data. By mapping the "preoperative tumor reference coordinates" to the coordinate system of the current fused data using Φ, the predicted intraoperative tumor coordinate data can be obtained, providing precise lesion location for navigation equipment or subsequent augmented reality output.

[0136] Through neural radiation field interpolation, smoother intensity or fractal information can be restored in areas with both optical and acoustic distortion, minimizing the impact on subsequent registration. The completed data can be closely aligned with the 3D lung model locally, generating precise nonlinear deformation field data. This allows for real-time reflection of lung tissue deformation during surgery, providing a reliable basis for the final determination of lung cancer lesion locations.

[0137] In one embodiment, a threshold is set: if the missing rate of both the optical and acoustic channels in the key area is greater than 40%, it is determined to be an area to be completed. Train the neural radiation field: use the surrounding known data as training samples, set the network depth to 5 layers (fully connected), converge after 50 rounds of local training, and then perform interpolation estimation on the points to be completed. Deformation registration: For the interpolated fusion data and the three-dimensional lung model, first align the rigid bodies, and then perform elastic optimization based on the local B-spline grid. The accuracy of the final nonlinear deformation field data can maintain the lesion coordinate deviation within 2 mm, meeting the requirements of accurate resection during surgery.

[0138] Preferably, the elastic deformation registration includes: first rigidly aligning the fused data with the three-dimensional lung model data to correct the global coordinate error, then determining the local grid transformation parameters based on the difference between the optical channel amplitude, the acoustic channel amplitude and the fractal measurement, generating nonlinear deformation field data and obtaining intraoperative tumor prediction coordinate data.

[0139] In the present invention, the fused data (multimodal features that have been interpolated and completed) and the 3D lung model data may initially have a large offset due to perspective, position or scale. The rigid body alignment step is performed by estimating the translation (t x ,t y ,t z), rotations (usually expressed in Euler angles or quaternions), and an optional uniform scaling parameter are added to roughly superimpose the two into the same coordinate system. This step does not fine-tune the local deformation; it only corrects macroscopic errors and makes subsequent nonlinear deformations easier to converge.

[0140] After rigid body alignment, to accommodate irregular deformations caused by localized stretching or collapse of the lung, transformation parameters (such as local translation vectors and stretch scaling) must be estimated within each small grid cell based on the differences between the optical and acoustic amplitudes and the fractal measure. If there are large protrusions or wrinkles, the optical and acoustic amplitudes at corresponding points in the grid and the 3D lung model will differ significantly, and the fractal measure will also increase significantly in that area.

[0141] The nonlinear deformation field can be viewed as Φ(x), mapping the coordinate x of the fused data to the coordinate x′ of the 3D lung model data. Once registration is complete, the known "preoperative tumor reference coordinates" are mapped to the current 3D position in the collapsed state using Φ, resulting in the "intraoperative tumor predicted coordinate data," which provides real-time information on the location of the lesion during surgical navigation.

[0142] In practical applications, identifiable global features (such as anatomical reference points or vascular intersections) are selected from the fused data and the 3D lung model data, and the transformation matrix T is initialized using the least squares method or ICP (Iterative Closest Point) rigid ,in:

[0143] x′=T rigid (x)

[0144] Optional unified scaling: When the overall size ratio of the two is inconsistent (for example, scaling error caused by resolution difference), you can rigid Add a global scale factor to the .

[0145] The coordinate domain of the fused data is divided into m×n small grid cells, each of which aggregates several points x i (including optical amplitude A opt (x i ), acoustic amplitude A ac (x i ), fractal measure F(x i )).

[0146] Let the corresponding coordinates in the lung 3D model data The above features are and The difference function Δ(x i ) to compare the fused data with the model data, for example:

[0147]

[0148] Where λ is the coefficient for adjusting the fractal measurement weight.

[0149] In each grid cell, the local transformation parameters α (including translation, scaling and local surface distortion parameters) are set to minimize ∑ i∈cell Δ(x i ) to solve α and finally obtain the deformation map Φ(x).

[0150] The local transformation parameters α of each grid cell are spliced and fused at the grid boundary through weighted smoothing (such as B-spline or radial basis function interpolation) to form a global nonlinear deformation field The “preoperative tumor reference coordinates” Mapping to That is, the "intraoperative tumor prediction coordinate data" is obtained to reflect the tumor position under the current lung collapse or traction state.

[0151] After the global coordinate error is resolved by the prior rigid body alignment, the local elastic registration can fine-tune and approximate the actual lung tissue deformation according to the distribution of optical amplitude, acoustic amplitude and fractal measurement, thereby reducing the resection error caused by breathing or mechanical traction.

[0152] If there are large differences between optics and acoustics in certain areas, and the fractal measure is significantly increased, it means that the area may have a lesion protrusion or collapsed edge. The present invention will give a larger weight to perform local deformation correction to ensure that the positioning result is more consistent with the actual status during the operation.

[0153] In one embodiment, the fusion data domain is divided into 16×16 grids; the difference function Δ(x i The local transformation parameter α is iteratively updated using gradient descent or the Levenberg–Marquardt (LM) algorithm. The final mesh deformation is obtained when the difference function converges to a preset threshold. Verification using known lung landmarks (or external tracking) revealed that the tumor position error is less than 3 mm, enabling accurate surgical resection.

[0154] S4. Apply elastic transformation to the three-dimensional lung model data according to the nonlinear deformation field data, output deformed lung model data consistent with the intraoperative state, fuse the deformed lung model data with external tracking information to obtain an augmented reality overlay image and display the intraoperative tumor prediction coordinate data, and repeat steps S2 and S3 for dynamic updating when the lung collapse or tissue traction amplitude exceeds a threshold.

[0155] In step S3, after quantum-inspired search, neural radiation field interpolation, and elastic registration, the nonlinear deformation field Φ is calculated. This deformation field describes the displacement relationship of the lung surface or shallow structure caused by factors such as lung collapse and traction. By mapping each coordinate point in the three-dimensional lung model data (usually in grid or voxel form) to x′=Φ(x), the deformed lung model data that matches the current intraoperative state can be obtained.

[0156] The 3D geometry of the lungs, initially generated based on preoperative or conventional segmentation, is updated to a model that matches the actual anatomical morphology in the current surgical environment, making subsequent surgical planning or assisted positioning more accurate.

[0157] External tracking devices (such as optical or electromagnetic positioning systems) are typically present in the operating room to monitor the spatial posture of the thoracoscope, robotic arm, or head-mounted display in real time. By combining the deformed lung model data with this tracking information in the same coordinate system or with a calibrated transformation matrix, the data can be superimposed on the surgeon's field of view or on the surgical screen to form an augmented reality (AR) image.

[0158] Using the posture information T provided by the tracking system, the coordinates of the deformed lung model are mapped to the reference coordinates of the operating room; in the graphics rendering pipeline, the three-dimensional model is overlapped with the camera viewpoint and the real image to generate a picture with perspective projection or superimposed marks; the "intraoperative tumor prediction coordinate data" is drawn at the corresponding position with a visual mark (such as a translucent cursor or outline) to help doctors identify lesions in the actual view.

[0159] If the monitored lung deformation exceeds a given threshold γ during surgery (e.g., a stretch displacement exceeding 5 mm), the previous deformation field no longer adequately represents the current anatomical state. At this point, the system should return to S2 (re-acquisition and processing of multi-source optical and acoustic information) and S3 (quantum multi-agent search and interpolation registration) to derive a new nonlinear deformation field and update the deformed model. This ensures that after significant collapse or stretch, the navigation system can promptly correct the lesion location and maintain continuous adaptation to the real-time lung state.

[0160] In practical applications, for each vertex v in a 3D lung model, the elastic deformation field Φ(v) is set to give the new coordinates v′. If mesh storage is used, the vertices are updated one by one while maintaining the connections between the facets to generate the deformed mesh. For relatively gentle changes such as breathing and subtle stretching, Φ can be periodically applied to the mesh and a local smoothing transition can be achieved. However, for significant displacement, a complete reconstruction is performed.

[0161] If the external tracking system has a coordinate transformation matrix M track , the midpoint v′ of the deformed model data is converted to track Rendering is performed after converting to the display device coordinate system.

[0162] Real-time video frames are captured by an operating room camera or head-mounted display, and a deformed model and tumor coordinate markers are projected and rendered for each frame based on the viewing angle. Doctors can visually see the virtual lung morphology overlaid on the actual image, accurately determining the location of nodules.

[0163] When multi-source data indicates that the displacement Δd of the lung surface or key points is greater than the threshold γ or a drastic change in the optical / acoustic amplitude is detected, it is determined that the current deformation exceeds the tolerable range.

[0164] After the update is triggered, a new round of multimodal information is obtained through S2 and fractal wavelet multimodal data is generated; then, in S3, a new nonlinear deformation field Φ is generated using processes such as quantum heuristic search. new ; Then this step will Φ new Applied to the 3D model of the lungs, refreshing augmented reality images.

[0165] Preferably, spatial transformation is performed on the three-dimensional lung model data based on the nonlinear deformation field data to obtain deformed lung model data, the deformed lung model data is aligned with external tracking information and an augmented reality overlay image is generated to display intraoperative tumor prediction coordinate data.

[0166] In the previous steps of this approach (typically involving quantum-inspired search, neural radiation field interpolation, and elastic registration), a nonlinear deformation field Φ is calculated to describe the complex deformation of lung tissue due to factors such as collapse and stretching. By mapping each point p in the original 3D lung model (which can be in grid or voxel format) to the new coordinates p′ = Φ(p), a deformed lung model data that matches the current intraoperative state is generated. At this point, lung segment boundaries, alveolar regions, or perilesional structures are updated accordingly to align with the true intraoperative anatomical state.

[0167] If an external tracking device (such as an optical or electromagnetic positioning system) is used, the posture of the thoracoscope, the end of the robotic arm, or the doctor's field of view in the operating room coordinate system can be obtained in real time. After the deformed lung model data is integrated into this coordinate system, it can be made to overlap with the actual video image during the rendering process to generate an augmented reality (AR) overlay image. In this fusion process, it is necessary to match the camera perspective, projection relationship, model scale, etc., so that the virtual lung morphology and the real surgical scene can be displayed synchronously on the screen or head-mounted display.

[0168] Similarly, the previous step maps the "preoperative tumor reference coordinates" to the current lung position using the deformation field Φ, generating the "intraoperative tumor predicted coordinate data." In the augmented reality overlay image, these coordinates are indicated with prominent markers (such as bright spots or translucent models), providing visual guidance for surgeons when planning lesion resection or surgery.

[0169] In practical applications, if the 3D model is stored as a grid, the vertices v are read one by one. i , apply v′ i =Φ(v i After updating all vertices, the original topology remains unchanged, while the surface shape adapts to the deformation field. If a voxel format is used, voxel points can be mapped to new coordinates using interpolation, and the voxels or mesh can be reconstructed. In areas of the lung where deformation is significant, mesh smoothing or boundary convergence strategies can be used to avoid distortion or interpolation between vertices and maintain geometric coherence of the lung surface.

[0170] The operating room coordinates and the 3D model coordinates have been calibrated before / during the operation so that M track Represents the transformation matrix between the tracking device and the model coordinates. For any point v′ in the deformed lung model i Application M track , which can be projected into a real-world perspective or device view. During the image rendering phase, the video frame captured by the surgical endoscope (or display) serves as the background, and the deformed model is perspective-projected, rendered, and superimposed on the real-time image. On the screen or head-mounted display, the doctor can see the virtual lungs (already deformed) overlaid on the actual chest cavity, ensuring that the lung surface morphology matches the actual anatomy.

[0171] The nonlinear deformation field Φ has been obtained in the elastic registration stage. Let the original "preoperative tumor reference coordinates" be x tumor The intraoperative position is x′ tumor =Φ(x tumor ). After aligning it with the external tracking coordinate system, the corresponding 2D projection coordinates in the AR screen can be determined. Use a clear cursor, color, or virtual model to indicate x' tumor The position in the AR image assists the surgeon in determining the direction of the lesion and the operation plan. More prompts can be provided for the center point or boundary of the lesion.

[0172] The deformed lung model fits the actual lung collapse or tissue changes, and is superimposed on the augmented reality screen, which can significantly improve the detection rate and positioning stability of small lesions, making surgical operations more intuitive.

[0173] If significant traction occurs during surgery, causing further collapse of the lungs, a new nonlinear deformation field Φ can be output by repeating steps such as multi-source data acquisition, quantum heuristic search, and elastic registration. new , and update the tumor position in the AR image to correct the deviation in time.

[0174] In one example, assuming a mesh containing 50,000-100,000 vertices, each vertex undergoes Φ processing, and its position is altered. GPU parallel processing allows reconstruction within seconds. Within the thoracoscopy field of view, the image is refreshed at 20 frames per second, and the deformed model is projected and rendered. The error between the cursor-indicated lesion position and the actual position is kept within 3 mm. Doctors can observe the dynamic movement of the lesion in real time, enabling precise resection.

[0175] Preferably, after monitoring lung collapse or tissue stretching exceeding a threshold, steps S2 and S3 are repeated to update the nonlinear deformation field data and the deformed lung model data, so that the intraoperative tumor prediction coordinate data in the augmented reality overlay image corresponds to the real-time lung status.

[0176] In this approach, the intraoperative monitoring system detects deformation caused by lung collapse or tissue stretching. When this value exceeds a pre-set threshold γ (e.g., a local displacement of 5 mm), the original nonlinear deformation field data is no longer applicable to the current anatomical state. Therefore, steps S2 and S3 must be repeated to update the multimodal features and elastic registration results, ensuring that navigation information remains closely aligned with the real-time lung morphology.

[0177] When a threshold is triggered, the system recollects optical depth and acoustic echo information during surgery, performs preliminary cleansing, and performs fractal wavelet multi-layer decomposition to generate new fractal wavelet multimodal data. This step can capture new surface wrinkles after lung collapse or possible blood interference.

[0178] After completing a new round of multimodal feature extraction, a quantum-inspired search mechanism is used to locate possible key areas. Neural radiation field interpolation is performed to fill in areas with severe noise or defects. Elastic registration is then performed with the 3D lung model data to generate new nonlinear deformation field data. This new deformation field is updated with the current actual lung condition, accurately calculating the new intraoperative tumor coordinate data.

[0179] In practical applications, several tracking markers are placed on the operating table (or an optical structured light system is used) to detect key points on the lung surface. Once a displacement of a key point Δd ≥ γ is detected, an update is triggered. In addition to geometric displacement, changes in optical and acoustic amplitudes can also be monitored; if there is a significant difference in the fractal measure between the two, this can also be considered as a condition to induce an update.

[0180] Every few seconds or manually triggered by a threshold determination, a new frame (or multiple frames) of optical depth and acoustic echoes are acquired. After filtering out excessive outliers, a multi-layer wavelet transform and fractal measurement calculation are performed to output brand-new fractal wavelet multimodal data, providing sufficient real-world basis for subsequent searches and interpolations.

[0181] Based on the new fractal wavelet multimodal data, the quantum agent is used to explore and locate potential lesion edges or collapse mutations in parallel in high-dimensional space. The defect or noise area is filled with radiation field and elastic deformation registration is performed again with the three-dimensional lung model to form a new version of the nonlinear deformation field Φ new The “preoperative tumor reference coordinates” are calculated by Φ new Mapping

[0182] New nonlinear deformation field Φ new This information, along with the deformed lung model, is then passed to the augmented reality rendering module, which updates the "predicted intraoperative tumor coordinate data" displayed on screen to match the current lung state. Surgeons can see the "tumor marker" on the screen or head-mounted display update its position in real time as the lung is stretched, ensuring accurate resection strategies and surgical paths.

[0183] Whether local deformation is caused by changes in ventilator strategy, traction, or blood stasis, as long as the amplitude exceeds the threshold γ, a new round of S2 and S3 is triggered to reconstruct new multimodal features and register the latest deformation field. This repeated update process maintains the tracking accuracy of lesion coordinates. When the cumulative displacement is large, there is no need for manual estimation or reliance on static data. The system automatically corrects the deformation field and refreshes the lesion marker in the augmented reality image.

[0184] In one embodiment, the threshold is γ = 5 mm. An update is initiated if a lung segment edge displacement greater than 5 mm or a 30% change in optical or acoustic amplitude is detected. This update can be triggered based on the progress of the surgery, perhaps every few minutes if the procedure is smooth, or more frequently if the procedure is frequently tense. Compared to a single initial positioning, experimental statistics show that this repeated update mechanism can reduce tumor positioning errors by 40%, a significant improvement during prolonged surgeries.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware.

[0186] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A navigation and positioning method for lung cancer surgery based on three-dimensional reconstruction and deep neural network, characterized in that: The following steps are involved: S1. Obtain original lung tomographic image data, perform denoising and segmentation, and generate lung 3D model data; S2. During the operation, multi-source optical and acoustic information is collected and preliminarily purified. The obtained data is subjected to fractal wavelet multi-layer decomposition to obtain fractal wavelet multimodal data with optical and acoustic channel characteristics. S3. locating key regions matching the three-dimensional lung model data in the fractal wavelet multimodal data using a quantum-inspired multi-agent search, interpolating and completing defective or noisy regions using a neural radiation field, and then performing elastic deformation registration with the three-dimensional lung model data to obtain nonlinear deformation field data and intraoperative tumor prediction coordinate data; S4. Apply elastic transformation to the three-dimensional lung model data according to the nonlinear deformation field data, output deformed lung model data consistent with the intraoperative state, fuse the deformed lung model data with external tracking information to obtain an augmented reality overlay image and display the intraoperative tumor prediction coordinate data, and repeat steps S2 and S3 for dynamic updating when the lung collapse or tissue traction amplitude exceeds a threshold.

2. The method according to claim 1, characterized in that Step S1 includes: normalizing the grayscale range and filtering out noise on the original lung tomographic image data, and converting the segmented lung tissue voxels into lung three-dimensional model data using a grid reconstruction algorithm.

3. The method according to claim 1, characterized in that Step S2 includes: acquiring optical depth information and acoustic echo information respectively, performing threshold filtering to obtain preliminary multimodal purified data, calculating fractal measure through fractal wavelet multi-layer decomposition, and fusing the optical channel amplitude and the acoustic channel amplitude into the fractal measure according to corresponding layered mapping to form fractal wavelet multimodal data.

4. The method according to claim 3, characterized in that When performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, the low-frequency sub-band and the high-frequency sub-band are used to distinguish the smooth area and the texture mutation area, respectively. The fractal measure is calculated for each sub-band, and the optical channel amplitude and the acoustic channel amplitude are sequentially associated with the fractal dimension of the corresponding sub-band to obtain fractal wavelet multimodal data with multi-scale characteristics.

5. The method according to claim 3, characterized in that Step S2 also includes: before performing fractal wavelet multi-layer decomposition on the multimodal preliminary purified data, using spatial filtering to remove isolated reflection points and eliminate abnormal amplitude areas to obtain multimodal preliminary purified data that meets the fractal measurement calculation requirements.

6. The method according to claim 1, characterized in that Step S3 includes: setting a quantum-inspired multi-agent search unit in the fractal wavelet multimodal data, each search unit performing multiple iterative searches on different spatial blocks based on the probability amplitude superposition pattern, and judging the key areas with reference to the anatomical features in the three-dimensional lung model data.

7. The method according to claim 1, characterized in that After retrieving the key area, the spatial part of the defect or noise is interpolated and completed through the neural radiation field, the interpolated data is fused with the surrounding area and elastic deformation registration is performed to output nonlinear deformation field data and intraoperative tumor prediction coordinate data.

8. The method according to claim 7, characterized in that The elastic deformation registration includes: first rigidly aligning the fused data with the three-dimensional lung model data to correct the global coordinate error, then determining the local grid transformation parameters based on the differences between the optical channel amplitude, the acoustic channel amplitude and the fractal measurement, generating nonlinear deformation field data and obtaining intraoperative tumor prediction coordinate data.

9. The method according to claim 1, characterized in that Step S4 includes: performing spatial transformation on the three-dimensional lung model data based on the nonlinear deformation field data to obtain deformed lung model data, aligning the deformed lung model data with external tracking information and generating an augmented reality overlay image to display the intraoperative tumor prediction coordinate data.

10. The method according to claim 9, characterized in that After monitoring lung collapse or tissue stretching exceeding a threshold, steps S2 and S3 are repeated to update the nonlinear deformation field data and the deformed lung model data, so that the intraoperative tumor prediction coordinate data in the augmented reality overlay image corresponds to the real-time lung status.

Images