Artificial intelligence based augmented reality three-dimensional visualization radiotherapy positioning device

Abstract

The application discloses an augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence and belongs to the technical field of precise radiotherapy. The device comprises five modules, namely, AR glasses, medical image processing, space registration, AI core driving and AR visualization guidance. The medical image processing fuses three-dimensional scanning data, CT images and automatic body surface sketching algorithms to construct a patient digital twin. The space registration adopts a multi-modal adaptive constraint registration algorithm to realize sub-millimeter alignment. The AI core driving extracts body position features based on CNN and UDA and generates an optimal positioning adjustment sequence through an SPA-DRL algorithm. The AR visualization guidance outputs dynamic positioning guidance in real time through double-layer differential virtual-real rendering and dislocation contour labeling and displays three-level six-dimensional error visualization information of "part, whole and target area" in layers. The device can realize positioning error of less than 2mm and single-time consumption of less than 3 minutes, and can meet the precision and efficiency requirements of precise radiotherapy in clinical practice.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an augmented reality 3D visualization radiotherapy positioning device based on artificial intelligence. Background Technology

[0002] Radiotherapy is a core method in clinical cancer treatment. The accuracy of radiotherapy positioning directly determines the local control effect on the tumor and the degree of radiation damage to normal tissues, making it a prerequisite and key to achieving precise radiotherapy. Currently, mainstream clinical radiotherapy positioning techniques include manual experience-based positioning, image-guided radiotherapy (IGRT), and optical surface tracking radiotherapy (SGRT). These techniques primarily rely on manual judgment, surface marking, laser positioning, or image assistance to complete the positioning. However, they have unavoidable technical drawbacks in clinical application, as detailed below: Manual positional placement relies entirely on the radiotherapist's skill, clinical experience, and subjective judgment. It lacks standardized and intelligent guidance mechanisms, and placement accuracy is easily affected by multiple factors, including differences in patient anatomy, patient compliance, and environmental interference. Clinical data shows that routine manual positional placement can have an error of 3-5 mm, and a single placement session can take over 2 minutes. It suffers from significant problems such as poor consistency, cumbersome procedures, low efficiency, and uncontrollable human error, making it difficult to meet the precision requirements of accurate radiotherapy.

[0003] Image-guided positioning: This technique relies on imaging such as CT and CBCT for positioning guidance. It requires a series of independent procedures, including image scanning, image reconstruction, anatomical registration, error calculation, and position adjustment, with the entire operation taking several minutes. There is a significant time lag between image-guided positioning and treatment; during this time, patient limb movement introduces new positioning errors, significantly reducing positioning accuracy. Clinically, multiple scans are often used to compensate for these errors, which further increases the patient's additional radiation exposure.

[0004] Optical surface tracking positioning: This technology indirectly calculates the spatial coordinates of the tumor inside the body by collecting information such as the patient's body surface contour and feature points. Its core drawback is that the surface tissue and the tumor inside the body are not rigidly connected, and their displacement is difficult to keep synchronized, resulting in the problem of "accurate alignment of the body surface, but actual displacement of the tumor".

[0005] The lack of intuitive guidance for patient positioning and the absence of a closed-loop control system are two common problems in existing radiotherapy positioning techniques. First, positioning errors are presented only in abstract forms such as numerical values ​​and two-dimensional coordinates, without providing visual guidance based on the patient's actual position. Second, error detection, position correction, and effect verification are independent of each other, failing to establish a complete control chain of "real-time error monitoring—visual dynamic correction—instant result verification."

[0006] The current status and shortcomings of augmented reality (AR) technology in radiotherapy positioning: AR technology has been gradually applied to radiotherapy positioning scenarios, but existing AR positioning guidance systems still have many key technical defects. (1) Single level of virtual-real fusion visualization: Existing systems only support the superposition of the overall three-dimensional solid model onto the patient's real scene, or simply display error values. They cannot simultaneously and clearly display the fine comparison between the reference body surface contour and the real-time patient body surface, and lack spatial misalignment markers at the contour level. (2) Lack of intelligent planning in positioning adjustment guidance: Existing systems present six-dimensional errors simultaneously, and the radiotherapist judges the adjustment order based on experience. They do not combine artificial intelligence (AI) to achieve optimal adjustment strategy planning, and there is no AR dynamic path guidance function. (3) Lack of error display levels: Existing systems only display a single overall six-dimensional error, and have not established a three-level six-dimensional error system of "location level - overall level - target area level". They cannot trace the source of error and lack target area-specific error assessment.

[0007] In summary, existing radiotherapy positioning technologies and AR-assisted positioning systems generally suffer from problems such as fragmented processes, delayed feedback, poor intuitiveness of deviation guidance, insufficient accuracy of virtual-real fusion comparison, lack of error display hierarchy, and inability to monitor tumor location in real time during treatment. These issues make it difficult to meet the core requirements of precision radiotherapy in terms of real-time response, visual guidance, precise positioning, and closed-loop management of the entire process, and have become key technical bottlenecks restricting the improvement of positioning quality and clinical efficacy.

[0008] The core problems of the prior art specifically solved by this invention are as follows: 1. Solving the problem of single-layer AR virtual-real fusion display: This invention innovatively proposes a dual-layer differentiated virtual-real fusion display architecture. In the AR field of view, it simultaneously uses differentiated visual encoding of semi-transparent wireframe outline (positioning reference) and solid filling (real-time body surface) to clearly display the body surface outline of the patient during positioning and the fine comparison with the real patient. It also automatically marks the misaligned area through spatial misaligned contour lines to achieve precise comparison at the outline level.

[0009] 2. Solving the problem of lack of intelligent planning in positioning adjustment guidance: This invention is the first to use a part priority ranking algorithm (SPA-DRL) based on deep reinforcement learning (DRL), which comprehensively considers error priority, dimensional coupling coefficient and ergonomic factors to intelligently plan the optimal adjustment order; combined with AR spatial dynamic path guidance (dynamic gradient arrows + Bezier trajectory prediction curves) and multimodal progressive feedback (visual + voice + tactile), it constructs a new closed-loop guidance mode of "intelligent sorting - dynamic guidance - step-by-step confirmation".

[0010] 3. Solving the problem of missing display levels for positioning errors: This invention pioneers a three-level six-dimensional error calculation and AR layered visualization system of "location level - overall level - target area level". It simultaneously displays the independent six-dimensional error of each anatomical location of the patient (head and neck, chest and abdomen, pelvis, etc.), the overall positioning six-dimensional error with isocenter as reference, and the target area-specific six-dimensional error with PTV geometric center as reference, and is equipped with error source tracing and intelligent attribution analysis functions.

[0011] 4. Solving the problem of the lack of 3D modeling methods for accelerator rooms: This invention proposes three methods: handheld 3D scanner modeling, drawing-based modeling, and scanning-drawing hybrid modeling. It achieves sub-millimeter registration accuracy through a three-level automatic alignment algorithm and constructs a 3D spatial reference with both real geometric accuracy and complete semantic information.

[0012] 5. Solving the problem of imprecise digital twin modeling methods: This invention proposes three methods to generate an enhanced digital twin that combines anatomical accuracy and visual realism: patient modeling based on handheld 3D scanners, modeling based on CT scans and automatic skin delineation (HU-AMSF-Net+APC-SRFrame), and adaptive fusion of handheld scans and CT models.

[0013] 6. Solving the problem of insufficient accuracy in cross-scene coordinate system transformation: This invention establishes four strictly defined core coordinate systems and a complete hierarchical transformation chain, proposes a multimodal adaptive constraint registration algorithm (MACR), achieves sub-millimeter level registration accuracy through a five-dimensional adaptive weight system and anatomical marker constraints, and pioneers a quantitative evaluation system for registration confidence. Summary of the Invention

[0014] The present invention proposes an augmented reality 3D visualization radiotherapy positioning device based on artificial intelligence, which can at least solve one of the technical problems in the background art.

[0015] To achieve the above objectives, the present invention adopts the following technical solution: An artificial intelligence-based augmented reality 3D visualization radiotherapy positioning device performs the following steps via computer equipment: AR glasses integrate a binocular camera, an RGB-D camera, an inertial measurement unit, and a display module. The AR glasses are used to collect point cloud data of the patient's body surface in real time and calculate their own spatial pose. At the same time, they serve as an output terminal for virtual-real fusion visualization images. The medical image processing module connects to the CT simulation positioning machine and the radiotherapy planning system respectively. It is used to receive the patient's positioning CT images, and sequentially completes image preprocessing, automatic skin segmentation and three-dimensional model reconstruction to generate a positioning reference model of the body surface. The spatial registration module is connected to the AR glasses and the medical image processing module. It is used to receive the reference body surface reference model and the real-time collected body surface point cloud data. It includes a coarse registration unit and a fine registration unit. It uses a multimodal adaptive constraint registration algorithm to complete the sub-millimeter level spatial alignment between the reference coordinate system and the real-time coordinate system and outputs the registered spatial transformation relationship. The AI ​​core driving module is connected to the spatial registration module. It is used to extract the six-degree-of-freedom positioning error based on the spatial transformation relationship, construct a state space containing the six-degree-of-freedom positioning error and the inter-dimensional coupling coefficient, run a part priority ranking algorithm based on deep reinforcement learning, and output the optimal adjustment order and corresponding adjustment parameters with the joint optimization objectives of minimizing the positioning error and the shortest operation time. The AR visualization guidance module is connected to both the AI ​​core driving module and the AR glasses. It receives the optimal adjustment order and adjustment parameters, and performs dual-layer differentiated virtual-real fusion rendering. The reference body surface contour layer is rendered in a semi-transparent wireframe form based on the reference body surface model and fixed in the positioning space. The real-time patient body surface layer is rendered in a solid filling form based on the data collected by the RGB-D camera. Based on the real-time distance field calculation results, the module automatically marks the spatial misalignment contour lines between the two layers. At the same time, according to the optimal adjustment order, the module renders dynamic gradient arrows and Bézier trajectory prediction curves in the AR field of view to provide spatial path guidance, and simultaneously outputs layered visualization results of three levels of six-dimensional error at the part level, overall level, and target area level. The AR glasses provide multimodal progressive positioning correction guidance to radiation therapists based on the virtual-real fusion image, spatial path guidance, and three-level six-dimensional error output by the AR visualization guidance module.

[0016] Furthermore, the medical image processing module employs the HU-adaptive multi-scale fusion skin delineation network for automatic skin segmentation, the network comprising: The HU statistical feature extraction module is used to extract the statistical features of the HU value distribution in CT data. The dual-branch decoder reconstructs high-resolution segmentation feature maps from the spatial domain and the frequency domain respectively, and performs feature fusion through cross-domain fusion weights adaptively generated based on the statistical features of HU. The HU-constrained segmentation head fuses the segmentation logits output by the convolutional network with a prior Gaussian function constructed based on typical HU values ​​of skin, outputting a skin tissue probability map.

[0017] Furthermore, the medical image processing module also includes a differentiable surface reconstruction framework with anatomical prior constraints, used to convert the skin tissue probability map into a signed distance function, and to perform end-to-end joint optimization through bony landmark constraints, body symmetry plane constraints and radiotherapy surface landmark constraints to generate a patient digital twin that combines anatomical accuracy and visual realism.

[0018] Furthermore, the multimodal adaptive constraint registration algorithm in the spatial registration module constructs a five-dimensional adaptive weight system. The five-dimensional adaptive weight is the product of geometric distance weight factor, texture feature similarity weight factor, semantic consistency weight factor, uncertainty estimation weight factor, and dynamic change detection weight factor. A joint optimization objective function is constructed based on anatomical feature point constraints, smoothness constraints, and regularization constraints.

[0019] Furthermore, the state space of the site priority ranking algorithm in the AI ​​core driving module consists of a six-degree-of-freedom positioning error and ten inter-dimensional coupling coefficients, which are determined jointly through statistical analysis of historical positioning data and kinematic calibration experiments of the treatment bed.

[0020] Furthermore, the reward function of the part priority ranking algorithm in the AI ​​core driving module is:

[0021] in, The residual error penalty weight, Weighted by operation time penalty. Weights are used to penalize coupling side effects. For dimension Historical average operation time For action The coupling coefficient for dimension j.

[0022] Furthermore, the dual-layer differentiated virtual-real fusion rendering in the AR visualization guidance module adopts Alpha blending technology, and the final pixel value is:

[0023] in, For reference, the transparency of the body surface contour layer, To measure the real-time transparency of the patient's skin surface. For contour color modulation function, The contour line intensity function, These are the pixel values ​​of the body surface image acquired in real time. These are the pixel values ​​of the real-world background.

[0024] Furthermore, the automatic annotation of spatially misaligned contour lines in the AR visualization guidance module is achieved through real-time distance field calculation. The distance field is the signed shortest distance from a real-time body surface point to a reference body surface model, and is mapped to a four-level visual code according to the distance threshold: transparent if the distance is no more than 1mm, yellow dashed line if the distance is greater than 1mm but no more than 2mm, orange solid line if the distance is greater than 2mm but no more than 3mm, and red flashing if the distance is greater than 3mm.

[0025] Furthermore, the dynamic gradient arrow in the AR visualization guidance module determines its length based on the current real-time error value and performs color mapping based on the normalized error ratio. The Bezier trajectory prediction curve is generated based on the current body position key feature points, the target body position key feature points, and the control point offset vector determined by the body part priority sorting algorithm.

[0026] Furthermore, the three-level six-dimensional error includes: Six-dimensional error at the site level is calculated based on independent local registration after anatomical region segmentation. The overall six-dimensional error is calculated based on a weighted fusion of the area of ​​each part and the registration confidence. The target area-level six-dimensional error is calculated by propagating the overall error to the target area center and superimposing local deformation compensation calculations based on the adjoint mapping.

[0027] As can be seen from the above technical solution, the AI-based augmented reality 3D visualization radiotherapy positioning device of the present invention addresses various shortcomings of existing radiotherapy positioning technologies and AR-assisted positioning systems. This invention proposes an AI-based augmented reality 3D visualization radiotherapy positioning device. This device uses AI algorithms as its core to complete 3D modeling optimization, multi-coordinate system registration, and intelligent calculation of positioning errors; combined with AR technology to build a digital twin model, it achieves virtual-real fusion visualization guidance, completing high-precision positioning across scenarios, while also possessing intelligent body position recognition, dynamic error correction, and full-process visualization control capabilities.

[0028] Compared to existing technologies such as human experience-based positioning, image-guided positioning (IGRT), and optical surface positioning (SGRT), this invention establishes an integrated technical architecture that combines AI real-time intelligent decision-making, AR virtual-real space overlay, high-precision digital twin registration, unified cross-scene coordinate calibration, intelligent positioning guidance, and three-level error visualization. Its specific advantages are as follows: AR virtual-real fusion display fully optimized: This invention abandons the traditional display mode of simply superimposing complete physical models, and innovatively proposes a dual-layer differentiated virtual-real fusion display architecture (semi-transparent wireframe outline + solid filling surface + misaligned contour line annotation), which realizes fine comparison at the outline level, enabling radiation therapists to intuitively identify millimeter-level spatial misalignment.

[0029] Intelligent upgrade of placement adjustment guidance: This invention pioneers the SPA-DRL intelligent sorting algorithm, which automatically plans the optimal adjustment order; it also pioneers the AR spatial dynamic path guidance system, which provides visual path guidance with dynamic gradient arrows and Bézier trajectory curves; and it pioneers the multimodal progressive step-by-step confirmation mechanism, which decomposes the complex six-degree-of-freedom correction into ordered sub-tasks.

[0030] The invention features a hierarchical innovation in error display: It pioneers a three-level, six-dimensional error system of "location level - overall level - target area level", which realizes accurate calculation of the entire chain of errors from the body surface to the tumor target area and AR hierarchical visualization, and is equipped with error source tracing and intelligent attribution analysis functions.

[0031] The three-dimensional modeling method system is improved: This invention proposes three accelerator room modeling methods: handheld scanning, drawing modeling, and hybrid modeling, as well as three patient modeling methods: handheld scanning, CT delineation, and adaptive fusion, to build a complete modeling technology system.

[0032] Significantly improved spatial registration accuracy: This invention proposes a MCR multimodal adaptive constraint registration algorithm, which, through a five-dimensional adaptive weight system and anatomical marker constraints, stably controls the registration error within 1.0 mm.

[0033] Closed-loop management throughout the entire process: This invention constructs a complete closed-loop management system of "data collection - intelligent decision-making - visual guidance - error correction - data storage" to achieve intelligent, visual and traceable placement throughout the entire process. Attached Figure Description

[0034] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the pre-radiotherapy planning and real-time positioning guidance stages of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0036] like Figure 1As shown in this embodiment, the AI-based augmented reality 3D visualization radiotherapy positioning device adopts an integrated core architecture of AI real-time decision-making and AR virtual-real fusion. This invention breaks through the passive mode of traditional radiotherapy positioning—"blind operation, post-verification"—and innovatively constructs an active intelligent positioning mode that combines dynamic visualization guidance with real-time precise correction, forming a highly collaborative integrated solution of hardware and software. The device utilizes three core technologies: patient digital twin construction, multi-source information spatial registration, and AR virtual-real fusion dynamic visualization, covering the entire radiotherapy positioning process and achieving full-link intelligent visualization closed-loop management. This technology is adaptable to various complex clinical radiotherapy scenarios and has extremely high clinical promotion value and broad industrial application prospects.

[0037] The augmented reality 3D visualization radiotherapy positioning device based on artificial intelligence described in this embodiment includes AR glasses, which integrate a binocular camera, an RGB-D camera, an inertial measurement unit and a display module. The AR glasses are used to collect point cloud data of the patient's body surface in real time and calculate its own spatial pose, and at the same time serve as the final output terminal of the virtual and real fusion image. The medical image processing module interfaces with the CT simulator and radiotherapy planning system to receive patient positioning CT images and perform preprocessing, automatic skin segmentation, and 3D model reconstruction to generate a reference surface model for positioning. The spatial registration module is connected to the AR glasses and the medical image processing module. It is used to receive the reference body surface model and the real-time collected body surface point cloud data. It includes a coarse registration unit and a fine registration unit. It uses a multimodal adaptive constraint registration algorithm to complete the sub-millimeter level spatial alignment between the reference coordinate system and the real-time coordinate system and outputs the registered spatial transformation relationship. The AI ​​core driving module is connected to the spatial registration module. It is used to extract the six-degree-of-freedom positioning error based on the spatial transformation relationship, construct a state space containing the six-degree-of-freedom positioning error and the inter-dimensional coupling coefficient, and run a part priority ranking algorithm based on deep reinforcement learning. With the minimum residual error, the shortest operation time and the minimum coupling side effects as the joint optimization objectives, it generates the optimal adjustment order and corresponding adjustment amount. The AR visualization guidance module is connected to the AI ​​core driving module and the AR glasses, respectively. It is used to receive the optimal adjustment order and adjustment amount, and perform dual-layer differentiated virtual-real fusion rendering. The reference body surface contour layer is rendered in the form of a semi-transparent wireframe based on the reference body surface model and fixed in the positioning space. The real-time patient body surface layer is rendered in the form of solid filling based on the data collected by the RGB-D camera. Based on the real-time distance field calculation results, the module automatically marks the spatial misalignment contour lines between the two layers. At the same time, according to the optimal adjustment order, the module renders dynamic gradient arrows and Bézier trajectory prediction curves in the AR field of view to provide spatial path guidance, and simultaneously outputs the layered visualization results of three levels of six-dimensional error at the part level, overall level, and target area level. The AR glasses provide multimodal progressive positioning correction guidance to radiation therapists based on the virtual-real fusion image, spatial path guidance, and three-level six-dimensional error output by the AR visualization guidance module.

[0038] The specific implementation process consists of eight steps, as detailed in Figure 2, with each step explained below: Step 1: Design a cross-scene spatial consistency algorithm to solve the problem of global coordinate unification. To achieve a globally unified coordinate system across different scenarios, this invention is equipped with AR glasses integrating a binocular camera, an RGB-D camera, an inertial measurement unit, and a display module. These AR glasses not only collect point clouds of the patient's body surface in real time and calculate the patient's real-time spatial pose, but also serve as an output terminal for a virtual-real fusion visualization.

[0039] This invention introduces an improved Factor Graph Optimization (FGO) calibration method, establishing four rigorously defined core coordinate systems: (1) DICOM image coordinate system The origin is set to the DICOM standard image voxel index (0,0,0), following the LPS convention.

[0040] (2) TPS planning coordinate system The origin is the isocenter defined by the radiotherapy planning system.

[0041] (3) Computer room physical / LINAC coordinate system The origin is taken as the center of the accelerator rack, and the standard follows IEC61217.

[0042] (4) AR display coordinate system The origin is the initial position at the moment the AR head-mounted display device is launched.

[0043] This invention uses homogeneous coordinates to represent rigid body transformations:

[0044] In the formula, for The homogeneous coordinate transformation matrix is ​​used to describe the composite transformation of rotation and translation between rigid body coordinate systems in three-dimensional space. In this invention... Used to accurately transform the patient's anatomical structure from the medical imaging coordinate system to the treatment device coordinate system; for Rotational torque; It is a three-dimensional translation vector, and its components are... , , These represent translational displacements along the corresponding coordinate axes; It is a three-dimensional zero row vector.

[0045] The complete transformation chain is as follows:

[0046] In the formula, This is a DICOM image coordinate system, with the origin at the DICOM image voxel index (0,0,0); This is the coordinate system for the TPS plan, with the origin being the isocenter of the radiotherapy plan. For the physical coordinate system of the computer room, the origin is the center of the accelerator rack; The coordinate system for the AR display is defined, with the origin being the initial spatial position of the AR display device. Convert DICOM voxel coordinates to TPS plan isocentric coordinates; Transmit the center point and beam parameters, such as TPS, to the treatment device; The FGO algorithm of this invention enables real-time calculation, ensuring precise alignment between the AR-overlaid virtual anatomical model and the actual patient's position. The physical meaning of the above transformation chain is: the DICOM image data acquired during the patient's CT scan is transformed step by step to be accurately mapped onto the display space of the AR glasses, achieving global spatial consistency across scenes.

[0047] The FGO error function is defined in Lie algebra form:

[0048] In the formula, For the first The six-dimensional Lie algebra error vector of each observation marker point; For optimization Homogeneous transformation matrix; (The vee operator) is a vectorized operator; The superscript "meas" indicates the measured value, and the subscript... Indicates from the first camera coordinate system To the coordinate system The transformation matrix; To display coordinate system from AR To the camera coordinate system The transformation matrix; From LINAC coordinate system To the A reference point coordinate system The transformation matrix; The inverse of the above matrix represents the coordinate system from the reference point. To the LINAC coordinate system The inverse transform of the function. The significance of this function lies in: by combining sensor observation data with the parameters to be optimized... By comparing the theoretical transformation chain, the spatial registration error under the current estimate is quantified, providing a differentiable residual measure for factor graph optimization.

[0049] The objective function for joint optimization of weighted Lie algebra error and prior regularization is:

[0050] In the formula, The transformation matrix to be optimized is represented as Minimize the independent variable; This represents the total number of observed marker pairs. For the index of observation marker numbers; For Cauchy bar kernel function; It is the Euclidean norm; For the first Information weight matrix corresponding to each observation The square root matrix; The six-dimensional Lie algebra error vector as defined above; , which is the prior regularization weighting coefficient, used to balance the contribution ratio between the observed data fitting term and the prior constraint term; The prior transformation estimation matrix is ​​obtained through coarse calibration or other auxiliary sensors. Initial approximation; For the current estimate with prior estimates The Lie algebra deviation vector between them; The first term is a prior regularization term, which prevents the optimization result from deviating from a reasonable physical prior. The first term of the function is a weighted robust data fitting term, which minimizes the weighted geometric residuals of all observation markers to ensure that the optimized transformation parameters are highly consistent with the actual sensor observation data. The second term is a prior regularization term, which constrains the optimization result to be close to the physically reasonable prior estimate. The two terms work together to achieve the optimal balance between data fitting and prior constraints.

[0051] Step 2: Acquisition and Preprocessing of Patient Radiotherapy CT Localization Data This step utilizes a CT simulator to perform three-dimensional localization scans on the patient for radiotherapy, acquiring tomographic CT image data including body surface contours, bony structures, tumor target areas, and surrounding organs at risk. Based on this, this invention integrates convolutional neural networks (CNNs) and unsupervised adversarial domain adaptation (UDA) algorithms for positional feature extraction, combines Transformer networks and graph convolutional networks (GCNs) to reconstruct 3D pose from 2D projection, and further introduces a multimodal adaptive constrained registration (MACR) algorithm to complete the matching of anatomical feature points, ultimately achieving sub-millimeter level high-precision image registration.

[0052] 1. Integrating CNN networks and UDA algorithms for body position feature extraction This invention employs an improved residual network as the backbone network for feature extraction, and conducts multi-label classification training for radiotherapy sites such as the head and neck, chest, abdomen, and pelvis. To adapt to the complex operating conditions of accelerator rooms and eliminate the interference of environmental background on feature extraction, an unsupervised adversarial domain adaptation (UDA) mechanism is introduced. By adding a domain discriminator constraint and optimizing the learning logic of the convolutional neural network, the model automatically eliminates redundant and interfering information.

[0053] 2. Achieving 2D projection to 3D pose reconstruction by combining Transformer and GCN networks. The device first extracts key points from the human body surface in a 2D image using a detection network, and then uses a Transformer network to convert the 2D coordinates to 3D spatial coordinates. In radiotherapy, fixation devices such as thermoplastic masks and vacuum bags often obscure the patient's body surface, causing traditional key point detection methods to fail. This invention designs an anatomical constraint model based on a graph convolutional network (GCN), using rigid features of the human skeleton, such as constant limb length and limited joint range of motion, as prior constraints. The device can calculate the geometric center of the obscured area based on the pose of the visible region.

[0054] 3. Introduce the MACC algorithm to complete the matching and sub-millimeter level registration of anatomical feature points. The algorithm in this invention employs an improved version of Iterative Closest Point (ICP)—the Multimodal Adaptive Constraint Registration (MACR) algorithm. This algorithm constructs a five-dimensional adaptive weight system:

[0055] In the formula, For the first The comprehensive adaptive weight coefficient of the corresponding point pair is determined by the product of five independent weight factors, comprehensively reflecting the value of the first point. The reliability of each point pair in the current registration iteration; Geometric distance weighting factor; This is a weighting factor for texture feature similarity; This is a semantic consistency weighting factor; Weighting factors for uncertainty estimation; To detect dynamically changing weighting factors, five weighting factors are multiplied to jointly reduce the weight of low-reliability point pairs, thus improving the overall weight. The range of values ​​is .

[0056] Constrained optimization objective function:

[0057] In the formula, Represented by rotation matrix Translation vector Minimize the independent variable; The objective function is denoted as , and a smaller value indicates higher registration accuracy. This is a data item, also known as the Point-to-Plane registration error item, which quantifies the point-to-plane distance residual from the source point cloud to the target point cloud; For anatomical feature point constraints, the clinical significance is to provide strong constraints using prior anatomical knowledge, accelerate convergence and avoid registration getting trapped in local optima; These are the constraint weight coefficients for anatomical feature points; This is a smoothness constraint term to suppress unreasonable local distortions generated during the registration process; These are the smoothness constraint weight coefficients, used to control the strength of local smoothness constraints; This is a regularization term to prevent numerical singularities from occurring during the optimization process; represents the weight coefficients of the regularization term. The joint optimization of the four weighted terms ensures that the registration result simultaneously satisfies four constraints: geometric data fitting, anatomical correspondence, local smoothness, and numerical stability.

[0058] Among them, the adaptive weighted Point-to-Plane data item is:

[0059] In the formula, For adaptive weighted Point-to-Plane data items; For all in the source cloud Summation operation is performed on each point. The number of source points participating in the registration; The first value calculated for the aforementioned five-dimensional adaptive weight system The comprehensive weighting coefficient of each point; for Rotation matrix; The first point in the source cloud A column vector of three-dimensional coordinates of points; As the source Three-dimensional coordinates after rigid body transformation; For the target point in the cloud and the source point The corresponding number A column vector of three-dimensional coordinates of points; The three-dimensional vector difference between the transformed source point and its corresponding target point; For the target point cloud at point The unit normal vector at that location; This is the scalar projection of the vector difference onto the normal direction; The vertical distance is squared. The advantage of the Point-to-Plane metric over the traditional Point-to-Point Euclidean distance metric is that it utilizes the local geometric information of the target point cloud surface, allowing the source point to slide along the tangent plane direction with almost no penalty, requiring only precise alignment in the normal direction. This results in a larger convergence range and faster convergence speed, which is particularly crucial for sub-millimeter-level high-precision registration scenarios in radiotherapy positioning.

[0060] This invention proposes a hybrid 6DoF solver, which first obtains the initial registration estimate through SVD closed-form solution, and only requires 3 to 5 Gauss-Newton iterations to complete the refinement. Compared with the 20 to 50 iterations required by the traditional ICP algorithm, the computational efficiency is improved by about an order of magnitude.

[0061] Step 3: AR Spatial Dynamic Guidance and Multimodal Progressive Feedback Based on SPA-DRL Intelligent Decision Making This invention innovatively proposes a three-in-one intelligent positioning and correction scheme: "SPA-DRL-based part priority ranking algorithm + AR spatial dynamic guidance + multimodal progressive feedback," constructing a novel closed-loop guidance mode of "optimal adjustment sequence planning—dynamic spatial path guidance—step-by-step progressive confirmation." Figure 2 As shown.

[0062] 1. Part Priority Ranking Algorithm Based on DRL (SPA-DRL) The state space of the SPA-DRL algorithm:

[0063] In the formula, The SPA-DRL algorithm at the decision time The state vector; physically, it represents the DRL agent's global perception of the current placement scenario, serving as the policy network. The input is used to generate the optimal adjustment action.

[0064] The first 6 dimensions of the state vector—six-degree-of-freedom positioning error (translation + rotation): X-axis translation error; Y-axis translation error; Z-axis translation error; : Rotation error around the X-axis; : Rotation error around the Y-axis; : Rotational error around the Z-axis; The last 10 dimensions of the state vector—the vector of coupling coefficients between dimensions: : The pairwise coupling coefficients between translation dimensions; : Coupling coefficient between X-axis translation adjustment and the three rotational dimensions; : Coupling coefficient between Y-axis translation adjustment and rotation dimension; : Coupling coefficient between Z-axis translation adjustment and rotation dimension; The introduction of coupling coefficients enables the SPA-DRL agent to possess the prior knowledge that "adjusting one dimension will affect other dimensions," thereby planning the globally optimal adjustment sequence that takes into account coupling effects and avoiding the inefficient operation mode of "repeated correction - over-adjustment - re-correction" often encountered by radiation therapists.

[0065] SPA-DRL reward function:

[0066] In the formula, : SPA-DRL agent in state Execute action The instantaneous scalar reward value obtained afterward; : No. The system state vector for each decision step, as described above. Dimensional state At any moment The possible values ​​of ; : No. The selected adjustment action; : Execute action The resulting six-dimensional residual error vector; The L2 norm of the residual error vector. ; Dimension Historical average operation time; :action Dimension The coupling coefficient; Dimension The current error value; : The absolute value of the coupling perturbation term; Residual error penalty weight; Operation time penalty weight; : Coupling side effect penalty weights.

[0067] Decision-making logic and three trade-offs: The three penalties in the reward function correspond to the three core optimization objectives of radiotherapy positioning operations—precision ( Item), efficiency ( Item) and stability ( (Item). The relative magnitude of the three weights. This reflects the clinical priority order: accuracy is always the primary goal, followed by stability, and then efficiency. During DRL training, the policy network learns to adaptively select the optimal adjustment order in different placement scenarios by maximizing this reward function—for example, when the error in one dimension is much greater than that in other dimensions, the dimension with the large error is processed first; when the errors in multiple dimensions are similar, the dimension that is easy to operate and has fewer coupling side effects is selected first.

[0068] 2. AR spatial dynamic path guidance 3D dynamic guide arrows:

[0069] In the formula, :time The three-dimensional dynamic guide arrow vector rendered in the AR field of view; its physical meaning is a spatial visualization representation of the adjustment operation that needs to be performed at the moment—the arrow direction indicates the direction of adjustment, the arrow length encodes the magnitude of the adjustment, and the arrow color encodes the degree of proximity to the target; its clinical meaning is to transform the abstract six-dimensional error value into an intuitive three-dimensional spatial visual guide, so that the radiation therapist only needs to "follow the arrow" to complete the adjustment, which greatly reduces the cognitive load. Time variable; Current step (time) In dimension The real-time error value; physically, it represents the remaining error in the dimension that needs correction; clinically, it is the core variable that determines the arrow length—the larger the error, the longer the arrow; as adjustments are made, the error decreases and the arrow shortens accordingly, providing continuous visual feedback. Dimension The unit direction vector; : Color mapping function, which maps the normalization error ratio to RGB color values; That is, from green ( (Error completely eliminated) to red ( Linear interpolation with initial full error, intermediate transition via yellow ( The physical meaning is that the adjustment progress is intuitively conveyed through color gradients—the closer to green, the closer to the target position; the clinical meaning is that therapists can judge whether the current adjustment is in place by color, without having to frequently check the numerical readings. Normalized error ratio; physically, it represents the remaining proportion of the current error relative to the initial error; the ratio is... This indicates that everything is in place, and the ratio is [value missing]. This indicates that adjustment has not yet begun; this ratio serves as the input to the color mapping function, achieving a gradual visual feedback of "error reduction → color change from red to green". Dimension The initial error value; its physical meaning is the reference standard for normalized color mapping.

[0070] Dynamically corrected trajectory prediction (Bezier curve):

[0071] In the formula, The dynamically corrected trajectory curve rendered in the AR field of view is the recommended motion path that guides the radiation therapist to adjust the patient from the current position to the target position. : The normalized parameter of the Bézier curve; its physical meaning is the normalized path length parameter along the trajectory curve; Corresponding trajectory starting point, Corresponding trajectory endpoint; : The three-dimensional spatial coordinates of key feature points in the current body position; : The three-dimensional spatial coordinates of key feature points of the target body position; : The displacement vector from the current position to the target position; : Offset vector of the second control point of the Bézier curve; : Quadratic Bessel basis functions in parameters The value at that location.

[0072] The significance of the trajectory curve: This invention uses a quadratic Bézier curve instead of a straight line as the guiding trajectory because in actual clinical positioning operations, the movement of the patient or treatment bed is often limited by the range of motion of the mechanical axis and operational space constraints, making a purely geometric shortest path not necessarily feasible. This is achieved by introducing a control point offset vector. The system can generate feasible paths that fit the actual scenario. For example, when the laser positioning light might collide with the lifting path of the treatment bed, It can be configured to laterally offset the trajectory on the horizontal plane to avoid obstacles. The path is rendered as a semi-transparent dashed line in the AR view, and... The frequency display shows the animation of bright particles moving along the path, indicating adjustments to direction and progress.

[0073] 3. Multimodal progressive feedback Visual progressive feedback: The length of the dynamic arrow shortens as the error decreases, and the color gradually changes from red to green. When it reaches the correct position, the green pulse flashes 3 times. Auditory voice broadcast: broadcasts the current step, adjustment direction, remaining adjustment amount, and confirmation of completion step in turn; Haptic vibration feedback (optional): The intensity of the handle vibration is proportional to the amount of residual error; Step-by-step automatic confirmation and progressive guidance: The next step is automatically unlocked after each dimension is completed, and a placement qualification signal is output after all six dimensions are completed.

[0074] Step 4: Construct a digital twin of the patient's radiotherapy position and label it with key information. This invention proposes three digital twin modeling methods, which can be flexibly selected or combined according to the needs of clinical scenarios.

[0075] 1. Method 1: Digital Twin Modeling Based on Handheld 3D Scanner This invention proposes a patient human body modeling method based on a handheld 3D scanner using multi-sensor fusion. The handheld 3D scanner integrates a LiDAR sensor, an RGB-D camera, and an IMU (Inertial Measurement Unit) to achieve synchronous acquisition and fusion processing of multimodal data.

[0076] This invention adaptively adjusts the scanning distance, scanning angle, and sampling density according to the different radiotherapy sites. The system receives the category of the area to be scanned from the operator through a preset radiotherapy site selection interface. And automatically configure scanning parameters accordingly:

[0077] In the formula, Configure functions for scan parameters; The working distance between the scanner and the patient's body surface; Configure the field of view for the LiDAR sensor; Point cloud sampling density; This is a suggested scan rate. This formula embodies the present invention's concept of adaptively optimizing scan parameters to address the differences in anatomical features at different radiotherapy sites.

[0078] During handheld scanning, IMU data and LiDAR's SLAM algorithm are used to track the scanner's pose changes in real time. The local point cloud data collected at various times is uniformly transformed to the global coordinate system:

[0079] In the formula, For a globally merged point cloud collection; This represents the total number of time steps in the scanning process. For the first Rigid body transformation matrix of the time-lapse scanner; The coordinates of a single 3D point in a local point cloud; For the first This formula represents the set of local point clouds acquired by the LiDAR sensor at any given time. It embodies the principle of the SLAM algorithm, which achieves global fusion of multiple frames of local point clouds through temporal pose tracking. Its significance lies in overcoming the coordinate system inconsistency caused by device movement during handheld scanning.

[0080] This invention performs multi-stage preprocessing on the fused global point cloud, and the normal vector is calculated through local covariance analysis:

[0081] In the formula, For point The unit normal vector at that location; The constraint condition is the unit normal vector; For the neighborhood The first 1 neighboring point; For the neighborhood The local centroid; For point of Nearest neighbor set; is the outer product matrix of points in the neighborhood with respect to the centroid; for Local covariance matrix of the neighborhood The eigenvector corresponding to its smallest eigenvalue is the normal vector. This formula obtains the normal vectors at each point by solving the optimization problem of the minimum eigendirection of the covariance matrix. Its clinical significance is to provide oriented geometric information for subsequent Poisson surface reconstruction.

[0082] This invention proposes an improved Poisson surface reconstruction method that incorporates patient body shape-adaptive octree depth:

[0083] In the formula, Let be the scalar indicator function to be solved; The gradient field of the indicator function; This is a vector field obtained by interpolating the point cloud normal vectors using radial basis functions; To define in the computational domain On Norm. This formula is a variational form of Poisson surface reconstruction, and its meaning is to minimize the gradient of the indicator function and the sampling vector field. The deviation transforms disordered point clouds into a continuous, smooth triangular mesh surface, which clinically means generating a high-precision 3D model of the patient's body surface that can be used for AR display and positioning comparison.

[0084] Adaptive octree depth:

[0085] In the formula, The adaptive octree depth is used to control the grid resolution of the Poisson reconstruction; This is the floor function (floor function). Based on depth, preferred This is used to ensure that the base resolution is not lower than An octree unit; These are adaptive coefficients; The bounding box volume of the point cloud on the patient's body surface; The reference volume constant is used; the clinical significance of this formula is that for larger patients, the octree depth is automatically increased to improve reconstruction resolution; for smaller patients, the depth is appropriately reduced to save computational resources, thereby achieving a clinical fit between accuracy and efficiency.

[0086] 2. Method Two: Digital Twin Modeling Based on CT Images and Automatic Skin Delineation This invention proposes a HU-Adaptive Multiscale Fusion Skin Delineation Network (HU-AMSF-Net) based on CT images and an Anatomically Prior Constrained Differentiable Surface Reconstruction Framework (APC-SRFrame) to achieve end-to-end construction from localized CT to a high-precision three-dimensional model of the patient's body surface.

[0087] 2.1 HU Statistical Feature Extraction and Adaptive Normalization This invention innovatively proposes a HU statistical feature extraction module to automatically analyze the HU value distribution characteristics of input CT data:

[0088] In the formula, Let HU be the statistical feature vector, with dimension 1. ; The global Hounsfield Unit (HU) mean for CT data; The global HU standard deviation of CT data; Typical HU value for air quality areas; This represents the typical HU value for adipose tissue; Typical HU value for soft tissue regions; This represents the typical HU value for bony structures. The clinical significance of this feature vector is that by extracting the patient-specific HU statistical distribution, the network can adapt to differences in HU values ​​under different CT equipment, scanning protocols, and patient body types, thereby enhancing the model's generalization ability.

[0089] Based on extracted HU statistical features Construct a patient-adaptive normalization transformation:

[0090] In the formula, The CT voxel values ​​are adaptively normalized. The original CT voxel HU value; The adaptive scaling function is derived from a two-layer fully connected network (FCN) based on the HU statistical characteristics. Dynamically generated; The adaptive offset function is determined by two fully connected layers based on... Dynamically generated; The global HU mean for CT data; The global HU standard deviation of CT data; This is the numerical stability constant. The clinical significance of this formula is that, unlike standard normalization with fixed parameters, this invention dynamically adjusts the normalization parameters based on the actual HU distribution of each patient's CT data, making the network robust to differences in HU scales across different CT devices, thereby improving the accuracy of skin segmentation.

[0091] 2.2 Dual-branch decoder and HU adaptive cross-domain fusion This invention innovatively proposes a dual-branch decoder structure, which simultaneously reconstructs high-resolution segmentation feature maps from both the spatial and frequency domains:

[0092] In the formula, This is a cross-domain fusion weight vector; Use the Sigmoid activation function; It is a multilayer perceptron, consisting of two fully connected layers, and its input is the HU statistical features. The output is the fusion weights; The aforementioned HU statistical feature vector The clinical significance of this formula is that different CT devices have different noise characteristics, and the structural frequency distributions of different anatomical sites are different. By dynamically adjusting the fusion weights based on the patient-specific HU statistical characteristics through MLP, the network can obtain the optimal feature representation under different device and site conditions.

[0093]

[0094] In the formula, This is the fused feature map; Output feature maps for the spatial domain branch; Output feature map for frequency domain branch; For the aforementioned cross-domain integration weights; This represents the contribution ratio of frequency domain features. This formula embodies the network design concept of "spatial-frequency dual-domain complementarity," and its clinical significance is that skin delineation tasks require both precise edge localization and structural perception of the overall body shape. By adaptively fusing the two, sub-millimeter-level skin segmentation accuracy can be achieved.

[0095] 2.3HU Constrained Segmentation Head This invention innovatively proposes a HU-constrained segmentation head method:

[0096] In the formula, voxels The probability value that it belongs to skin tissue; Use the Sigmoid activation function; The aforementioned space-frequency fusion feature map; HU is the prior constraint weight coefficient; The prior Gaussian function of HU; voxels The original HU value. The core innovation of this formula is to organically combine the powerful feature learning capabilities of deep learning with CT physical knowledge, and inject physical constraints into the segmentation decision process through additive fusion, thereby improving segmentation accuracy without increasing network depth.

[0097] Where HU is the prior function:

[0098] In the formula, The prior Gaussian function value for HU; This represents the typical mean HU value for skin tissue; The standard deviation of HU values ​​in skin tissue; The input is the square of the difference between the HU value and the typical HU value for the skin; The normalized denominator is denominator. This function is a simplified form of a one-dimensional Gaussian probability density function. Its clinical significance is that it utilizes the physical characteristic that the skin HU value in CT images exhibits an approximately Gaussian distribution to provide physical prior constraints for the network. Especially when the patient's body shape is unique, leading to uncertainty in deep learning features, the HU prior can effectively guide the network to make correct skin region judgments.

[0099] 2.4 Composite Loss Function This invention innovatively proposes a composite loss function:

[0100] In the formula, The total loss function is used to provide a comprehensive gradient signal for multi-objective optimization during network training, guiding the network to simultaneously optimize four dimensions: region overlap, pixel classification, boundary accuracy, and HU consistency. This is the Dice loss term; These are the weighting coefficients for the binary cross-entropy loss; For binary cross-entropy loss; These are the edge loss weighting coefficients; This is the marginal loss term; The HU consistency loss weighting coefficient; This is the HU consistency loss term. The clinical significance of this composite loss function is that radiotherapy surface delineation requires sub-millimeter precision. A single loss function cannot simultaneously optimize region coverage and boundary accuracy. By weighting and combining four loss terms, the network is guided to learn from multiple dimensions, achieving the clinically required precise surface extraction.

[0101] in:

[0102] In the formula, The Dice loss is used when the prediction is completely consistent with the label. When there is no overlap ; voxels Predicted skin probability; voxels Real skin label; The sum of the element-wise products of the prediction and the annotation; This is the sum of squares of the predicted probabilities; This represents the total number of real skin voxels. Its clinical significance lies in its ability to directly optimize the spatial overlap between the predicted and real skin regions, and to exhibit better gradient properties than cross-entropy for small targets in medical image segmentation.

[0103]

[0104] In the formula, Edge loss is used to specifically optimize the accuracy of skin boundary localization; Boundary area Total number of voxels; It is a narrow band of skin borders; Boundary area Medium voxel index; Predict edge probability maps for the network; The edge map is labeled realistically. The loss function adopts the form of mean squared error (MSE), which means that the skin boundary is a key feature of AR registration in radiotherapy positioning, and the sub-pixel accuracy of the boundary directly affects the accuracy of positioning error assessment.

[0105]

[0106] In the formula, The significance of HU consistency loss lies in constraining the distribution of HU values ​​in regions predicted as skin to conform to the physical characteristics of skin tissue. To predict the total number of voxels in a skin region; To predict voxel indexes within skin regions; voxels The original CT values; This loss function is used to predict the average HU value within the skin region. The clinical significance of this loss function is that skin exhibits a characteristic range of HU values ​​on CT scans. By constraining the minimization of the HU value variance in the predicted region, the network can prevent misclassification of areas with excessively thick or thin fat layers as non-skin, thus enhancing the plausibility of the segmentation results at the HU physical level.

[0107] 2.5 Differentiable transformation from segmentation probability volume to SDF This invention innovatively proposes a differentiable transformation from segmentation probability volume to SDF:

[0108] In the formula, voxels The signified meaning of the signified distance function (SDF) is as follows: Voxel representation Located on the outer surface of the skin, This indicates that it is located inside the skin surface. This indicates that it is exactly on the skin surface, which is significant because it provides continuous and smooth distance information, facilitating subsequent extraction of differentiable surfaces and AR rendering. For temperature parameters; It is the natural logarithm function; voxels Skin segmentation probability; This is the odds ratio for skin classification. The significance of this formula is that it transforms the discrete probability output into a continuously differentiable distance field, allowing the surface reconstruction process to be optimized through end-to-end gradient backpropagation.

[0109] 2.6 Optimization of Anatomical Prior Constraints Bony landmark constraints:

[0110] In the formula, The clinical significance of the loss of bony landmark constraint lies in the fact that the surface mesh reconstructed by constraint maintains anatomical consistency with the bone structure segmented by CT at the bony landmark points. This represents the total number of bony landmarks. To reconstruct the first on the body surface mesh The three-dimensional coordinates of the bony landmarks; For automatic detection of the first bone structure segmentation from CT scans Three-dimensional coordinates of bony landmarks; This is the square of the Euclidean distance. The clinical significance of this loss function is that bony structures have high contrast on CT scans, with localization accuracy down to the sub-millimeter level. By constraining the spatial relationship between the body surface grid and bony landmarks, the overall geometric accuracy of body surface reconstruction can be indirectly improved, making it particularly suitable for head and neck radiotherapy with well-defined bony landmarks.

[0111] Body symmetry plane constraints:

[0112] In the formula, The clinical significance of loss due to body symmetry constraints lies in using the anatomical prior of left-right symmetry of the human body to correct asymmetry bias in the segmentation results. The set of vertices of the left body surface mesh The vertices in; This refers to the set of vertices of the left mesh, determined by the initial symmetry plane partitioning. Regarding the plane of symmetry Mirror reflection transformation; for The paired right vertex. Clinical significance of this loss function: Human anatomical structures have good left-right symmetry, especially in the head, neck and pelvic regions. Using symmetry plane constraints can effectively correct unilateral segmentation deviations caused by CT artifacts, slight head tilt of the patient, etc., and improve the anatomical rationality of the reconstruction model.

[0113] Radiotherapy surface landmark constraints:

[0114] In the formula, The significance of constraining surface landmarks for radiotherapy lies in the consistency between the surface landmarks identified by automatic segmentation and the prior anatomical locations. The total number of surface landmarks; For the first Individual table marker weighting coefficients; To extract the first from the automatic segmentation results Three-dimensional coordinates of individual marker points; For the first Anatomical prior coordinates of individual surface landmarks; The value is the squared Euclidean distance. The significance of this loss function is that surface landmarks for radiotherapy are important reference points for positioning verification. By constraining their positions to conform to anatomical priors, we ensure that the reconstructed model can be used for subsequent quantitative assessment of positioning deviations.

[0115] End-to-end joint loss:

[0116] In the formula, The significance of this end-to-end joint optimization of total loss lies in simultaneously optimizing segmentation quality, distance field properties, anatomical consistency, and rendering effects, thereby achieving end-to-end differentiable training from CT to 3D models. For segmentation loss; Eikonal regularization weights; The loss is the Eikonal regularization loss; Weights for bony constraints; This refers to the aforementioned loss of bony landmark constraint; For symmetric constraint weights; This refers to the aforementioned loss due to constraints on the body's symmetry planes; Weights are constrained for surface markers; This refers to the aforementioned loss of surface landmark constraint during radiotherapy; For rendering loss weights; This represents the differentiable rendering loss. The clinical significance of this total loss is that through multi-objective collaborative optimization of the six losses, the final reconstructed 3D body surface model simultaneously meets the clinical requirements of "accurate segmentation, standardized distance field, reasonable anatomy, and visual realism," and can be directly used for AR positioning guidance.

[0117] 3. Method 3: Digital twin modeling that integrates handheld 3D scanners, CT images, and automatic skin delineation. This invention proposes an adaptive fusion method for handheld 3D scanning models and CT surface models. Coarse registration employs weighted least squares.

[0118] In the formula, For coarse registration transformation matrix Its significance lies in providing the initial spatial alignment between the handheld scanning model and the CT surface model; To find the parameter operators that minimize the objective function; Let be the rigid body transformation matrix to be optimized; The number of anatomical landmarks used for coarse registration; For the first Registration weights for each marker point; The first in the handheld scanning model The three-dimensional coordinates of the anatomical landmarks; The corresponding number in the CT surface model Three-dimensional coordinates of anatomical landmarks; The square of the Euclidean distance. This formula is a weighted least squares problem with a closed-form analytical solution (through SVD decomposition). Its significance lies in: using a small number (4-6) of anatomical landmarks to quickly calculate the approximate transformation between two models, resulting in low computational complexity. ), suitable for real-time clinical applications.

[0119] Fine registration employs a scan-CT surface model registration based on improved ICP:

[0120] In the formula, For fine registration transformation matrix Its clinical significance lies in achieving sub-millimeter-level precise alignment between two body surface models based on coarse registration; To find the minimum value in the rigid body transformation space; For the set of corresponding points The point pairs in, where From the scanning model, The nearest neighbor in the CT model; This is the set of corresponding points established through kd-tree nearest neighbor search; For Huber robust kernel function; Huber kernel threshold; The distance between the transformed scan point and the corresponding CT point is the Euclidean distance. This is an adaptive weighting function based on curvature. The clinical significance of this improved ICP is that the Huber kernel effectively handles outlier correspondences caused by scanning blind spots or patient positioning differences, and curvature weighting makes registration focus more on the flat trunk region, which plays a crucial role in clinical positioning.

[0121] Adaptive weight fusion:

[0122] In the formula, The fused signed distance function value has clinical significance in combining the advantages of CT and scanning models to form the optimal digital representation of the patient's body surface. The coordinates of the query point in three-dimensional space; For CT model at point The fusion weight at the location; The signed distance function value for the CT surface model; For scanning model at points The fusion weight at the location; This represents the signed distance function value of the handheld scanning model. The formula is a weighted linear fusion in the distance domain, meaning that the CT model provides accurate internal anatomical benchmarks, while the scanning model provides rich surface geometric details; adaptive weighting achieves complementary fusion of the two modalities.

[0123]

[0124] In the formula, For CT model at point Adaptive fusion weights at the location; Clinical preference parameter: when When the value is close to 1, priority is given to ensuring the geometric accuracy around the target area. When the value is close to 0, priority is given to ensuring surface details in the visible area of ​​the scan. Weighting of the target area; To scan visibility weights, This indicates that the point is within the scanning field of view. This indicates the area within the scanning blind zone. The clinical significance of this formula is that the core objective of radiotherapy positioning is to ensure the accuracy of the target area location; therefore, the high precision of the CT model is prioritized near the target area. However, in the visible surface areas, the high-resolution surface details of the scanned model are fused to enhance the visual realism of the AR display.

[0125] Step 5: Integrate TPS data to establish a spatial relationship between the radiotherapy plan and the digital twin. This step, based on the DICOMRT standard interface, enables precise data exchange between the digital twin and the radiotherapy planning system (TPS). It imports core treatment plan data in batches, such as target area prescription dose, organ-at-risk tolerance dose, treatment isocenter coordinates, gantry angle, and field direction, thus completing the spatial association between the radiotherapy plan and the digital twin.

[0126] Step Six: 3D Modeling and Multi-Coordinate System of the Accelerator Room This invention proposes three methods for 3D modeling of accelerator rooms, which can be flexibly selected or combined according to clinical scenario conditions.

[0127] 1. Method 1: 3D Modeling of Accelerator Room Based on Handheld 3D Scanner This invention employs a portable handheld 3D scanning device integrating a LiDAR depth sensor and an RGB-D camera. Through a progressive scanning strategy, multi-resolution fusion mechanism, closed-loop constraint data stitching, and least-squares isocenter automatic calibration, high-precision 3D digital reconstruction of the radiotherapy accelerator room is achieved. The progressive scanning strategy is executed in the following order: (1) control area scanning; (2) labyrinth scanning; (3) main treatment room scanning; (4) detailed equipment scanning; (5) top structure scanning. The progressive scanning strategy ensures that each subsequent region overlaps with the previous region by at least 30%.

[0128] Objective function for pose graph optimization of closed-loop constraint scan:

[0129] In the formula, Optimizing the total energy for the pose graph is significant because it optimizes the pose estimation of each frame during the scanning process by minimizing the total error under closed-loop constraints, thereby improving the global consistency of the 3D reconstruction of the computer room. For edge index pairs in the set of consecutive frame constraints; For the odometer edge set; For robust kernel functions; Odometer constraints The residual vector; The square of the Mahalanobis distance; These are edge index pairs in the set of closed-loop constraints; It is a set of closed-loop edges; For closed-loop constraints The residual vector. The clinical significance of this pose graph optimization: Radiotherapy rooms typically have an area of ​​50-100 square meters. The height is 3-4m, and handheld scanning takes several minutes to complete. SLAM drift is unavoidable. However, through closed-loop constraints and robust optimization, the cumulative error can be controlled at the mm level, ensuring the accuracy of subsequent isocenter calibration.

[0130] Multi-resolution fusion scanning mechanism: coarse scan mode with a pixel pitch of 5cm + fine scan mode with a pixel pitch of 1mm.

[0131] Automatic isocenter calibration – Least square optimal intersection point:

[0132] In the formula, The three-dimensional coordinates of the treatment isocenter are clinically significant as the spatial reference origin for radiotherapy implementation, and all coordinates in the treatment plan are referenced to this point; To find the spatial point that minimizes the objective function ; The coordinates of the isocenter candidate points to be optimized; These are the indices of three spatial lines; For the first The coordinates of a known reference point on a straight line in space; For the first The unit direction vector of a straight line in space; For vectors In direction The projected length on; For vectors In direction The projection vector on; For point To the The perpendicular distance vector of a straight line in space; The formula represents the squared Euclidean distance. This formula addresses the optimization problem of finding the nearest common point of three spatial lines using least squares. Its clinical significance lies in the fact that, physically, three axes cannot strictly intersect at a single point. By using least squares to find the optimal approximate intersection point, the sum of the squared distances from that point to the three axes is minimized, thus obtaining isocentric coordinates for clinical use with sub-millimeter accuracy.

[0133] Analytical solution:

[0134] In the formula, The solution is the analytical solution of the three-dimensional coordinates of the isocenter of the treatment, which represents the closed-form optimal solution of the least squares problem. It is the identity matrix; It is an auxiliary matrix formed by the direction vectors; For the first A straight-line auxiliary vector; For the first A unit direction vector for a straight line; For the first A known point on a straight line; This refers to the cross product operation of three-dimensional vectors. For matrix Transpose of; for The inverse of a matrix; Let be the vector on the right. This formula is an analytical closed-form solution to the least-squares intersection problem of three spatial lines, meaning that the coordinates of the isocenter can be calculated directly without iterative optimization, and the computational complexity only involves... Matrix inversion ( It is suitable for rapid calibration in clinical settings.

[0135] 2. Method Two: 3D Modeling of the Accelerator Room Based on Infrastructure Drawings This invention extracts building structure information, equipment layout parameters, and shielding design data from accelerator room CAD / BIM design drawings, and combines this with a parametric modeling rule base to achieve automatic conversion from two-dimensional drawings to three-dimensional models.

[0136] 3. Method 3: 3D Modeling of Accelerator Rooms Integrating 3D Scanners and Infrastructure Drawings This invention proposes a three-level automatic alignment algorithm: Level 1 – Dominant Plane Alignment: Solving rotation and translation using orthogonal constraints in the computer room:

[0137] In the formula, The significance of the rotation alignment energy function lies in solving the optimal rotation matrix between the two models by minimizing the deviation between the normal vector of the dominant plane of the scanning model and the corresponding normal vector of the drawing model. Let be the rotation matrix to be solved; The indices of the three orthogonal dominant planes correspond to the floor of the computer room. ), main wall ( ) and side walls ( ); For the first Alignment weights of the dominant planes; For the scanning model, the first The unit normal vector of each dominant plane; The normal vector of the drawing model after rotation The transformed vector; For the first in the drawing model The unit normal vector of each dominant plane; The square of the Euclidean distance is given. This formula addresses the rotation estimation problem based on orthogonal planes. Its significance lies in the fact that the architectural structure of the radiotherapy room possesses strict orthogonality, and the rotation matrix can be uniquely determined using the correspondence between the normal vectors of the three orthogonal planes. This method is more accurate than point-based registration.

[0138] Level 2 – Structural Wireframe Alignment: Extract wall intersections and doorway outlines for image matching.

[0139] Level 3 – Semantic Weighted ICP Fine Alignment:

[0140] In the formula, The significance of the semantically weighted ICP energy function lies in the fact that by introducing semantic category weights, the fine alignment process pays more attention to structural elements that are crucial to radiotherapy safety and operation. Let be the rotation matrix to be optimized; Let be the translation vector to be optimized; For the index of the scanned model points; This represents the total number of sampling points involved in registration in the scanning model; For the first Semantic weight of each point; For the scanning model, the first The coordinates of a three-dimensional point; The coordinates of the scan point after rigid body transformation; For the scanning points in the drawing model after transformation The most recent The coordinates of the points; For scan points The nearest neighbor index in the drawing model. This formula is a semantically weighted extension of the standard ICP algorithm, and its clinical significance is that different devices in a radiotherapy room have different clinical importance. Semantic weighting makes the registration results more in line with clinical safety requirements.

[0141] Scan-drawing registration energy function:

[0142] In the formula, The energy function for scan-drawing integrated registration is a weighted sum of planar normal constraints and planar distance constraints. Its significance lies in simultaneously optimizing the rotation matrix. Translation vector To achieve optimal spatial alignment between the scanned model and the drawing model; Dominant plane index; For the first Normal constraint weights of each plane; For the scanning model Normal vectors of the dominant plane; The vector obtained by rotating the normal vector of the drawing model; This refers to the alignment error of the normal vector. These are the distance constraint weighting coefficients; For the scanning model Signed distances from each dominant plane to the origin; The signed distance between the drawing model and the corresponding plane; This is the transformed expression for the planar distance on the drawing. This represents the planar distance alignment error. The clinical significance of this energy function is that the normal constraint ensures the consistent orientation of the wall / ground, while the distance constraint ensures accurate spatial positioning. The combination of both enables precise registration between the scanned model and the drawing model.

[0143] Adaptive fusion of multi-source data:

[0144] In the formula, The adaptive fusion weight for the drawing model (design model) is significant because it dynamically allocates the fusion ratio based on the confidence levels of the drawing model and the scanned model. Score the confidence level of the drawing model; Score the confidence level of the scanning model; This is the adaptive fusion weight for the scanning model. The significance of this formula is: when the drawing confidence is high, more drawing data is fused to maintain geometric accuracy; when the scan confidence is high, more scan data is fused to reflect the actual state, achieving the optimal combination of the two data sources.

[0145] Automatic establishment of a unified coordinate system:

[0146] In the formula, It is a transformation matrix from the scanning coordinate system to the unified coordinate system of the machine room. Its clinical significance lies in establishing a spatial mapping relationship between handheld scanning data and the radiotherapy planning coordinate system, which is the spatial reference for all AR displays and positioning guidance. The direction vector of the X-axis in the unified coordinate system of the computer room under the scanning coordinate system; The direction vector of the Y-axis in the unified coordinate system of the computer room under the scanning coordinate system; The direction vector of the Z-axis in the unified coordinate system of the computer room under the scanning coordinate system; The three-dimensional coordinates of the treatment isocenter in the scanning coordinate system; This is the bottom row vector of the homogeneous coordinates, ensuring the affine property of the transformation matrix. The clinical significance of this transformation matrix: In radiotherapy TPS, all coordinates are based on the unified coordinate system of the treatment room, through... By unifying the scanned room model and patient model into the TPS coordinate system, precise spatial alignment of the treatment plan with the AR display can be achieved, ensuring the clinical usability of the positioning guidance.

[0147] Step 7: Accurate comparison and display of the body contour based on dual-layer differentiated virtual-real fusion and spatially misaligned contour line annotation. This invention proposes a method for precise comparison and display of body surface contours based on a dual-layer differentiated virtual-real fusion architecture. Within the AR field of view, it simultaneously displays a clear and detailed comparison between the "reference body surface contour during positioning" and the "real-time patient body surface" using differentiated visual encoding, and automatically identifies spatial misalignment areas. Unlike existing technologies that simply overlay a complete 3D solid model onto the actual patient scene, this invention achieves, for the first time, a precise virtual-real comparison at the contour level.

[0148] 1. Dual-layer differentiated virtual-real fusion display architecture This invention constructs two independent and visually distinguishable information layers within the AR field of view: Reference surface contour layer: Isosurface contour lines are extracted from the surface model reconstructed by localization CT and rendered as a semi-transparent wireframe. Preferably, this contour layer uses a cool cyan color (RGB:[0,192,255]) and a transparency of... Set to 0.6 to fix the display at the spatial position during positioning.

[0149] Real-time patient surface layer: Point clouds of the patient's body surface are acquired in real time using an RGB-D camera integrated into the AR glasses and rendered as solid fills. Preferably, this surface layer uses realistic skin texture rendering with high transparency. Set it to 0.85.

[0150] Final pixel values ​​of dual-layer blended rendering:

[0151] In the formula, AR display at pixel coordinates The final fused pixel value represents the visual output presented to the radiation therapist after the dual-layer virtual-real fusion. Its clinical significance is the digital representation of the light intensity output by the AR glasses optical combiner at the human retina. It directly determines the quality of the radiation therapist's visual perception of the fused image. The value range is 0–255, where 0 represents pure black and 255 represents pure white. : Reference the transparency coefficient of the surface contour layer; when At this time, the outline layer is presented in the form of a semi-transparent wireframe, which ensures that it is clearly distinguishable without obscuring the real scene below; Real-time transparency coefficient of the patient's body surface, controlling the visibility of the real body surface image captured by the RGB-D camera; Outline color modulation function; : Contour line intensity function; : Pixel values ​​of patient body surface images acquired in real time by an RGB-D camera; : Real-world background pixel values; Pixel plane coordinates, with the top left corner of the AR display as the origin; , ,in This refers to the monitor resolution.

[0152] Significance: This formula uses Alpha blending technology to achieve two-layer differentiated virtual-real fusion, and the coefficients of the sum of the three terms satisfy... This ensures the conservation of pixel energy. By... (0.6) and (0.85) is set to an unequal value, so that the two layers of information have different transparency. Combined with the geometric difference of "wireframe to solid" and the color difference of "cool color to warm color", the three-layer differential coding ensures that the human visual system of the radiation therapist can process the two layers of information independently at the same time without perceptual confusion, which is in line with the principle of multi-channel parallel processing of visual attention.

[0153] Differentiated visual coding strategy: This invention adopts a differentiated visual coding strategy of "wireframe to solid, cool color to warm color", so that the two layers of information occupy different visual channels, and the human eye of the radiation therapist can clearly distinguish the two layers of information at the same time without visual confusion.

[0154] 2. Automatic detection of spatial misalignment and contour line labeling Real-time distance field calculation:

[0155] In the formula, Real-time body surface points To reference body surface model The shortest distance, i.e., the spatial misalignment distance at that point; its physical meaning is to quantify the geometric deviation of the patient's real-time position from the reference position during positioning in space; its clinical meaning is to directly reflect the degree of surface deformation caused by positioning errors, and is the input quantity of contour line color mapping. : Real-time acquisition of three-dimensional spatial points on the patient's body surface using an RGB-D camera; : A set of reference surface model points reconstructed from localized CT scans; Euclidean distance norm For all vertices of the reference model The search is performed by iterating through the points to find the minimum value.

[0156] Clinical significance: distance field This is the core intermediate variable of the spatial misalignment contour line annotation method of this invention. By calculating the shortest distance from each point in the real-time acquired body surface point cloud to the reference model, a distance field distribution covering the entire acquired body surface can be obtained. This distance field is not only used for visual warnings in subsequent contour line color mapping, but can also be integrated to calculate the total deviation of the body surface. This serves as a quantitative evaluation indicator of overall placement quality. In clinical applications, the distance field calculation frame rate must be no less than 10Hz to ensure the real-time performance of the AR display.

[0157] Contour line color mapping function:

[0158] In the formula, : Spatial misalignment contour line color mapping function, based on distance field value Output the corresponding visual coding style; the physical meaning is to discretize the continuous distance value into four clinically significant error levels and present them as differential visual attribute codes; the clinical meaning is to enable radiotherapists to identify the misaligned areas that need attention and their severity at a glance. Input the distance field value, i.e., the above. The threshold setting of this piecewise function directly corresponds to the accuracy level of clinical radiotherapy positioning. First interval (Transparent): Corresponds to the clinically acceptable range of positioning error; in radiotherapy positioning practice, Surface deviations are generally considered to be within the systematic error tolerance and require no special warning; visual coding is completely transparent (not displayed) to avoid unnecessary visual interference, allowing the radiation therapist to focus their attention on the areas that truly need adjustment. Second zone (Yellow dashed line) ): Corresponds to the warning zone for minor placement deviation; to While the deviation is within the acceptable range for some treatment techniques, it still warrants attention; preferred RGB color values ​​are preferred. The line pattern is dashed (dash pattern: 5px on / 3px off); clinically, this alerts the radiologist to a slight deviation in this area, which should be noted in subsequent fine-tuning steps. Third Interval (Orange solid line) ): Corresponds to the warning zone for moderate placement deviation; to The deviation has exceeded the tolerance of conventional radiotherapy positioning; optimal RGB color values ​​are preferred. The line is a continuous solid line; clinically, it clearly indicates that the positioning in this area is incorrect and requires significant adjustment. Fourth Interval (Red flashing) ): Corresponds to the emergency warning zone for severe placement deviation; greater than Deviations that fall under the category of serious positioning errors will directly affect treatment accuracy and patient safety; optimal RGB color values ​​are preferred. and with Frequency flickering; clinically, this is the highest level of visual alarm, forcibly drawing the radiation therapist's attention to a severely misaligned area, requiring immediate correction. Threshold parameter , , Three key clinical thresholds are set based on international radiotherapy positioning accuracy standards and the accuracy indicators of this device system. These three thresholds discretize the continuous distance field into four levels of clinical decision intervals, corresponding to the four levels of clinical response strategies: "no treatment required - attention required - adjustment required - emergency treatment required".

[0159] The reason this invention uses distance field contour lines instead of full surface coloring is that contour lines, as low-spatial-frequency visual elements, do not obscure the high-frequency texture information of the underlying real-world scene, which conforms to the ergonomic principle in AR displays that "supplementary information layers should not interfere with the main working vision." The four-level color coding follows the internationally accepted "green-yellow-orange-red" warning color spectrum order, which is consistent with the human visual system's high sensitivity to the red band and its evolved instinct for warning of danger.

[0160] 3. Adaptive contour linewidth based on perspective correction

[0161] In the formula, The width of the outline rendered on the AR display after perspective correction; its physical meaning is to ensure that the outline presents a constant visual angle size on the human retina; its clinical meaning is that the outline is always presented with a constant visual thickness regardless of the position of the radiologist's head, ensuring visual recognition consistency. : Base line width of the outline; this value balances the recognizability of the outline with the cleanliness of the image—too thin ( This can easily cause display flickering, and excessively thick text ( ) This would obscure too many details of the actual scene. The equivalent focal length of the AR glasses optical system; : Observer's eye to the rendering point on the body surface Depth distance along the line of sight; : View direction and surface normal vector at the rendering point of the body The angle between them; The factor is used to compensate for the visual expansion effect of linewidth at grazing angles; Perspective scaling factor, following the inverse perspective rules of the pinhole camera model.

[0162] This formula, based on the principles of perspective projection in computer graphics, achieves adaptive linewidth rendering of contour lines. In actual AR rendering pipelines, The value comes from a pixel-by-pixel depth test of the depth buffer (Z-buffer). The value is calculated using the dot product of the real-time body surface normal vector. This invention employs an occlusion detection mechanism based on Z-buffer. When the contour line is occluded by the actual body (Z-test fails), the fragment is not rendered, thus avoiding the erroneous visual effect of the contour line "penetrating" the body and appearing on an invisible surface. Perspective correction ensures that the radiation therapist always perceives a contour line of constant thickness during head movement, eliminating visual size fluctuations caused by changes in distance and angle, and improving visual comfort during long-term positioning operations.

[0163] Step 8: Based on the three-level six-dimensional error calculation and AR hierarchical visualization system of "partial level - overall level - target area level" This invention proposes a three-level, six-dimensional error calculation and AR hierarchical visualization system: "partial level - overall level - target area level".

[0164] 1. Six-dimensional error calculation at the part level The patient's body surface was divided according to anatomical regions. Each part performs local ICP registration independently:

[0165] In the formula, : No. The local six-dimensional registration transformation of the anatomical site; the physical meaning is to transform the ... Real-time point cloud of body surface of each part Best aligned to its reference model The required rigid transformation is the estimation of the local positioning error at that location. In special Euclidean groups The optimal transformation that minimizes the objective function is sought. For three-dimensional rigid body transformations, Lie groups are used. : The rigid body transformation matrix to be optimized; : No. A subset of real-time body surface point clouds for an anatomical location; : No. A set of reference surface model points for each anatomical region; and The difference is that the former is a dynamic point cloud acquired in real time, while the latter is a static model reconstructed by CT; the registration of the two reveals the spatial offset of the real-time body position relative to the positioning body position. A single 3D point in a real-time point cloud; :point In the reference model The nearest neighbor in; The squared Euclidean distance between the transformed point and its nearest neighbor reference point; :point The weighting coefficients; the calculation method takes into account the following factors: (a) confidence weights (a) Weighted by distance; (b) Weighted by distance. The weight of the depth measurement is directly proportional to the confidence level of the RGB-D camera depth measurement. The more reliable the depth value, the higher the weight. It is inversely proportional to the distance from the point to its nearest neighbor reference point. (c) Suppress the influence of outliers; (d) Anatomical importance weighting Higher weights are assigned to surface points near the target area, reflecting the greater impact of these regions on final dose accuracy; overall weighting. Normalization ; : Regarding the first The weighted sum of the real-time point cloud points of each part; its physical meaning is a comprehensive measure of the overall registration quality of that part.

[0166] Site segmentation strategy: Divide the patient's body surface into anatomical regions. The segmentation is based on the iconic anatomical structures of the human body and the relative positions of isocenters. The advantages of independent ICP registration for each segment are: (1) local registration can reduce the degree of violation of the rigid registration assumption by large-scale deformation; (2) it can identify the differential error patterns between different segments; and (3) it provides spatial positioning information for error tracing.

[0167] 2. Overall six-dimensional error calculation

[0168] In the formula, Overall six-dimensional placement error, encompassing all The local registration results of each anatomical site; the physical meaning is the comprehensive six-degree-of-freedom deviation of the patient's overall position relative to the reference position; the clinical meaning is the single global error index used in traditional positioning, which is directly used to judge whether the positioning is qualified. Total number of anatomical sites; : No. Reference body surface area for each part; : No. Confidence coefficients for local ICP registration at each location; : No. Normalized weights of each part in the overall error calculation; : No. Local six-dimensional registration transformation of each part; : For all The weighted sum of local errors for each anatomical location.

[0169] 3. Target-area level six-dimensional error calculation

[0170] In the formula, The target-level six-dimensional error refers to the equivalent six-dimensional deviation at the center of the tumor target volume (PTV) due to the propagation error from the positioning error. Physically, it represents the total spatial deviation actually borne by the center of the target volume after comprehensively considering global rigid body positioning errors and local non-rigid deformation between the body surface and the target area. Clinically, it directly relates to the dose delivery accuracy in the treatment plan—radiotherapy plans are designed with the PTV position in the positioning CT coordinate system as the reference for the radiation field design, and the target-level error... This refers to the aiming deviation of the shot field relative to the PTV during actual treatment, which is the most crucial indicator for assessing the dosimetric effect of positioning. Accompanying mapping from the therapeutic center to the target volume center (PTV); Rigid body transformation from ISO coordinate system to PTV center coordinate system; Overall six-dimensional setup error; global error propagation term The global setup error at ISO is transmitted to the PTV center via adjoint mapping. The number of surface landmarks surrounding the target area; : No. Deformation influence coefficient of each marker point; : No. Local deformation vectors at each marker point; local deformation compensation term The sum of local deformations at each marker point, weighted by Gaussian weights.

[0171] Clinical precision significance: target area level error This is the most clinically valuable indicator in the three-level error system of this invention. Traditional positional QA only focuses on the overall level error. However, research indicates that there is a surface-target region. – Non-rigid deformation can lead to the clinical risk of "the target volume actually shifting even when the body surface is properly positioned". This invention transmits the global error to the target volume through adjoint mapping and superimposes local deformation compensation, allowing radiation therapists to directly observe "how much the PTV will actually shift" in the AR interface rather than "how much the body surface deviates", thus achieving a cognitive upgrade from body surface positioning accuracy to target volume dose accuracy.

[0172] 4. AR spatial layering visualization presentation Location-level error display: Each anatomical location is displayed next to a floating 3D label, using a miniature six-dimensional radar chart; Overall error display: The main navigation panel is displayed in a fixed position in the AR field of view, including a large six-dimensional radar map and precise numerical values; Target-level error display: Centered on the actual spatial location of the PTV, a dedicated error panel for the target area is displayed around it, including a safety threshold indicator bar.

[0173] Six-dimensional radar chart parameter equations:

[0174] In the formula, The sixth dimension radar diagram The vertex position vector corresponding to each dimension; its physical meaning is the vertex position vector on the two-dimensional radar image display plane. The polar-to-rectangular coordinate transformation result of the error in each dimension is used to form a hexagonal radar image by connecting the six vertices in sequence. Dimension index; : No. The angular position of each dimension on the circumference of the radar chart; : No. Normalized radial length of each dimension on the radar chart; : No. The absolute value of the error magnitude in each dimension; : No. The maximum permissible error threshold for each dimension; Radar image shows the radius.

[0175] Visual encoding of radar charts: six vertices to Connecting them sequentially forms a closed hexagon. The center point of the hexagon represents the ideal state of zero error. The area of ​​the hexagon is proportional to the overall error level—the larger the area, the more severe the overall placement deviation. The asymmetry of the hexagon's shape visually reflects the differential distribution of errors in each dimension. In AR rendering, the interior of the hexagon is filled with a semi-transparent color, and the boundary lines are... Width drawing. Target area radar images are additionally displayed outside the hexagon. Compared to The incremental arrow indicates the deformation risk when "actual deviation of the target area > deviation of the body surface".

[0176] 5. Error source tracing and intelligent attribution analysis Error contribution:

[0177] In the formula, : No. The normalized contribution of each anatomical region to the overall translation error; its physical meaning is to quantify the contribution of the first anatomical region to the overall translation error. The relative proportion of local errors in each part of the overall error; clinically, this means that the system can automatically identify the main anatomical areas that cause overall positioning deviations, highlight the areas that contribute the most in the AR field of view and prompt "prioritize adjusting this part", thus achieving accurate attribution and targeted correction of error problems. : No. The overall effective weight of each part; : No. Local registration transformation of each part The L2 norm of the translation component; :all The sum of weighted translation errors for each part.

[0178] Intelligent Attribution and Clinical Decision Support: System by Contribution All parts are sorted in descending order, and intelligent attribution guidance is provided in the AR view in the following ways: (1) Highlighting the part with the largest contribution: A red pulse outline is rendered around the anatomical part with the highest contribution, with the text "Priority Adjustment" and the six-dimensional error value of the part; (2) Contribution ranking list: The contribution bar chart of each part is displayed in the sidebar of the AR interface for intuitive comparison; (3) Adjustment effect prediction: Based on the SPA-DRL model, the expected improvement of the overall error after prioritizing the adjustment of the part with the largest contribution is predicted and superimposed on the current radar chart with a green dashed radar chart to provide a visual comparison of "before adjustment - after adjustment". This error tracing and attribution analysis mechanism upgrades the traditional "blindly adjusting item by item" placement mode to an intelligent placement mode of "data-driven targeted adjustment", which significantly reduces the number of iterations and the total placement time.

[0179] System hardware and functional modules composition and connection relationships: The hardware and functional modules of this invention are designed around the core architecture of "AI real-time decision-making + AR virtual-real fusion". Each unit has a clear division of labor and works in concert to support the clinical application of the "dynamic visualization guidance and immediate correction" active positioning mode. The functions and connections of each hardware component and module are as follows: 1. AR Glasses: As the core interactive terminal of the system, it integrates a binocular camera, an inertial measurement unit, and a display module. It achieves bidirectional data interaction with the AI ​​core driving module and the AR visualization guidance module, primarily undertaking the following tasks: (a) collecting patient positional features, calculating spatial pose, outputting a fusion of virtual and real images, and providing multimodal interactive feedback; (b) acquiring real-time point clouds of the patient's body surface through an RGB-D camera to achieve dual-layer differentiated fusion of virtual and real displays; (c) rendering reference body surface contour lines, real-time patient body surface layers, and misaligned contour line annotations; (d) displaying SPA-DRL dynamic guidance arrows and Bezier trajectory prediction curves; and (e) outputting a three-level six-dimensional error radar map and numerical panel for body part, overall, and target area. This device serves as the interactive hub connecting the physical therapy space and the virtual digital model, and is the final visualization carrier for AR positioning guidance.

[0180] 2. Medical Image Processing Module: Seamlessly integrates with CT simulators and radiotherapy planning systems (TPS) to sequentially complete CT image preprocessing, AI automatic segmentation (HU-AMSF-Net), 3D model reconstruction (APC-SRFrame), digital twin construction, and marker point coordinate extraction.

[0181] 3. Spatial Registration Module: The module consists of two main units: coarse registration and fine registration. It integrates the MCR multimodal adaptive constraint registration algorithm to achieve sub-millimeter level spatial alignment.

[0182] 4. AI Core Driving Module: The Device's Decision-Making Center. Internally, it relies on CNN and UDA to extract body position features, runs the SPA-DRL intelligent sorting algorithm to plan the optimal adjustment sequence, and combines the MCR algorithm to accurately calculate the three-level six-dimensional positioning error, outputting error data and DRL correction parameters.

[0183] 5. AR Visualization Guidance Module: This module is the core execution module for achieving AR virtual-real fusion guidance, which completes two-layer differentiated fusion rendering, dynamic path guidance generation, three-level error visualization panel rendering, and multimodal feedback output.

[0184] 6. Data Recording and Quality Control Analysis Module: Automatically archives data from the entire placement process, generates standardized quality control reports, and provides strong support for the system's full-process digital management.

[0185] 7. External linkage equipment: including CT simulator, radiotherapy planning system (TPS), CBCT, radiotherapy host and treatment bed, which establish bidirectional communication with the device through the DICOMRT standard interface.

[0186] Summary of technical effects: This invention uses "AI real-time decision-making + AR virtual-real fusion" as its core technical architecture, breaking through the passive mode of traditional radiotherapy positioning of "blind operation and post-verification" and innovatively constructing an active positioning mode of "dynamic visualization guidance and instant correction". The overall technical advantages and application effects are as follows: Precision advantages: Device registration error ≤1mm, positioning error ≤±2mm, HU-AMSF-Net skin delineation Dice coefficient ≥0.95, and MACR registration accuracy reaches sub-millimeter level. Utilizing AI high-precision computation and AR virtual-real fusion guidance, it can meet the clinical application requirements of precision radiotherapy.

[0187] Efficiency advantages: Patient placement time per case is ≤3 minutes. The SPA-DRL intelligent sorting algorithm reduces the number of adjustment steps by approximately 40%, and the hybrid 6DoF solver improves registration calculation efficiency by approximately one order of magnitude. Leveraging AI-generated calibration schemes and AR visualization guidance, placement efficiency is significantly improved.

[0188] Real-time advantages: The overall device latency is ≤15ms, the body position monitoring frequency is ≥15Hz, the frame rate of dual-layer differentiated fusion rendering is not less than 30Hz, the three-level error calculation and visualization update frequency is not less than 10Hz, and the dynamic visualization guidance and real-time correction functions are guaranteed to respond quickly by relying on the end-edge-cloud collaborative architecture and AI high-speed computing capabilities. Safety advantages: The device dynamically monitors the patient's position throughout the entire process, and immediately triggers a safety interlock if the deviation exceeds the limit. Real-time assessment of target-level errors in the three-level error system ensures the safety of the tumor target area.

[0189] Visualization advantages: Dual-layer differentiated virtual-real fusion display enables fine contour-level comparison; SPA-DRL dynamic path guidance provides intelligent operation navigation; three-level six-dimensional error layered visualization realizes the presentation of the entire chain of errors from the part to the target area.

[0190] Intelligent advantages: Effectively reduces reliance on personnel experience in setup operations. SPA-DRL automatically plans the optimal adjustment sequence, and intelligent error tracing and attribution automatically locate problem areas, significantly improving the standardization of setup.

[0191] The innovative aspects that this invention aims to protect: 1. AR Virtual-Real Fusion Display Innovation (New): A pioneering method for precise comparison and display of body surface contours based on dual-layer differentiated virtual-real fusion and spatially misaligned contour line annotation. Through a triple display mechanism of "semi-transparent wireframe contour layer (positioning reference) + solid-filled body surface layer (real-time patient) + automatic spatially misaligned contour line annotation," it achieves precise virtual-real comparison at the contour level in the AR field of view, differing from the simple superposition of complete solid models in existing technologies. It includes dual-layer fusion rendering formulas, distance field contour line mapping functions, perspective correction adaptive line width formulas, and depth buffer occlusion detection mechanisms.

[0192] 2. Intelligent Positioning Guidance Innovation (New): A pioneering three-pronged intelligent positioning and correction solution integrating the "SPA-DRL part priority ranking algorithm + AR spatial dynamic path guidance + multimodal progressive feedback." The SPA-DRL algorithm comprehensively considers error priority, inter-dimensional coupling coefficients, and ergonomic factors to automatically plan the optimal adjustment sequence; AR spatial dynamic path guidance provides visual path guidance using dynamic gradient arrows and Bézier trajectory prediction curves; multimodal progressive feedback breaks down the complex six-degree-of-freedom correction into ordered sub-tasks. This differs from the passive mode of existing technologies that "simultaneously display six-dimensional errors and manually adjust them one by one."

[0193] 3. Innovative Three-Level Error Visualization (New): A pioneering three-level, six-dimensional error calculation and AR layered visualization system—"location-level—overall-level—target-area-level"—is implemented. Location-level errors are calculated based on independent ICP registration of the anatomical region; overall-level errors are based on area-confidence weighted fusion; and target-area-level errors are based on the superposition of global error propagation through adjoint mapping and local deformation compensation. These three levels of errors are presented layered in the AR view as a floating radar chart, main navigation panel, and target area surround panel, and are equipped with error source tracing and intelligent attribution analysis functions. This differs from existing technologies that only display a single overall error.

[0194] 4. Innovation in 3D modeling of accelerator rooms: The first three-level automatic alignment algorithm (dominant plane alignment → structural wireframe alignment → semantically weighted ICP fine alignment) for handheld scanning and paper-based hybrid modeling is proposed. A semantic ontology and adaptive fusion weight mechanism for radiotherapy are proposed to achieve sub-millimeter level hybrid modeling accuracy.

[0195] 5. Innovation in digital twin modeling: The first-ever integration of handheld 3D scanning equipment, HU adaptive multi-scale fusion skin segmentation network based on CT images (HU-AMSF-Net), and anatomically prior-constrained differentiable surface reconstruction framework (APC-SRFrame) to construct a radiotherapy digital twin; the model registration is divided into two levels, with coarse registration using weighted least squares method and fine registration relying on improved ICP scan-CT surface model matching.

[0196] 6. Spatial Registration Innovation: The first multimodal adaptive constrained registration algorithm (MACR) was created, and a five-dimensional adaptive weight system (geometric consistency + texture consistency + semantic consistency + uncertainty + dynamic consistency) was constructed. For the first time, the semantic information of anatomical landmarks was introduced into the ICP weight design. The first hybrid 6DoF solver (SVD closed solution + a small number of GN iterations) was created, which improved the computational efficiency by about an order of magnitude. The first four-level quantitative evaluation system for registration confidence was created.

[0197] 7. Global Unification Innovation of Cross-Scene Coordinate Systems: The first improved Factor Graph Optimization (FGO) calibration method was developed to establish four strictly defined core coordinate systems; AR glasses integrating binocular cameras, RGB-D cameras, IMU and display units can collect surface point clouds in real time, calculate spatial pose, and output virtual-real fusion visualization images.

[0198] Simulation Example: Application Verification of an Artificial Intelligence-Based Augmented Reality 3D Visualization Radiotherapy Positioning Device 1. Example Background and Scene Setting Simulation subjects: Adult patients diagnosed with head and neck tumors who require intensity-modulated radiotherapy (IMRT); Treatment site: Target area for head and neck tumors; Simulation environment: Linear accelerator treatment room; Simulation duration: 5 minutes (300 seconds) for a single positioning and treatment monitoring session; Implementation process and effect verification Step 1: 3D Modeling of the Accelerator Room A handheld 3D scanner was used in conjunction with a hybrid scanning-drawing modeling method. The radiation therapist used the handheld scanner to scan the control area → labyrinth → main treatment room → equipment details → ceiling in that order. The system automatically registered the model with the CAD drawings using a three-level alignment algorithm. The final hybrid model achieved an overall geometric accuracy of 0.8 mm and an isocenter calibration accuracy of 0.3 mm.

[0199] Step Two: Construction of the Patient's Digital Twin Patient-localized CT data was acquired, and the HU-AMSF-Net network was used to automatically delineate the skin surface (Dice coefficient 0.96). Differentiable surface reconstruction was then performed using the APC-SRFrame framework. Simultaneously, a handheld scanner was used to acquire images of the patient's body surface, which were adaptively fused with the CT model. The fused digital twin possesses both anatomical accuracy and visual realism.

[0200] Step 3: Cross-scene coordinate system calibration The four core coordinate systems were calibrated using an improved FGO method. The MACR algorithm was used to register the patient model with the computer room space, achieving a registration confidence level. (High confidence level).

[0201] Step 4: AR Positioning Guidance and Three-Level Error Visualization The radiation therapist wears AR glasses, which simultaneously display the following in the AR field of view: Reference body surface contour layer: a cyan semi-transparent wireframe outline, fixed in the positioning space; Real-time patient body surface layer: realistic skin color and texture solid rendering; Misaligned contour line annotation: a slightly yellow dashed contour line on the left side of the head and neck (misaligned by 1.5mm). The SPA-DRL algorithm outputs the optimal adjustment sequence: "Z-axis translation → Y-axis translation → X-axis rotation → fine-tuning the rest." Dynamic arrows guide the adjustment in the AR field of view, and Bézier curves display the predicted trajectory. After each step is completed, automatic voice confirmation is given before proceeding to the next step.

[0202] Level 3 error panel display: Head and neck area error: —Highlight the overall positioning error in orange for priority adjustment: Target area error: —The safety indicator bar is green (within the threshold). Step 5: Placement Completion and Quality Control After adjustments, all dimensional errors eventually converged to the translation error. 1.5mm, rotation 0.5 The AR interface displays a global green confirmation icon, and the system generates a standardized quality control report, taking a total of 2 minutes and 15 seconds.

[0203] Table 1 shows a comparison of the effects of the embodiments of the present invention; Table 1

[0204] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0205] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0206] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the artificial intelligence-based augmented reality 3D visualization radiotherapy positioning devices described in the above embodiments.

[0207] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0208] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0209] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0210] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0211] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence-based augmented reality 3D visualization radiotherapy positioning device, characterized in that, include: AR glasses that integrate an RGB-D camera, an inertial measurement unit, and a display module; The medical image processing module is used to generate a baseline body surface reference model based on the localized CT images; The spatial registration module, connected to the AR glasses and the medical image processing module, is used to register the reference body surface model with the real-time body surface point cloud and output the spatial transformation relationship. The AI ​​core driving module is connected to the spatial registration module and is used to extract the placement error based on the spatial transformation relationship and run the part priority sorting algorithm to output the optimal adjustment order and adjustment parameters. The AR visualization guidance module is connected to the AI ​​core driving module and AR glasses. It is used to perform dual-layer differentiated virtual-real fusion rendering based on the adjustment order and parameters, including a semi-transparent wireframe reference body surface contour layer, a real-time patient body surface layer filled with solids, and spatial misalignment contour line annotation based on the real-time distance field. The system renders dynamic gradient arrows and Bezier trajectory prediction curves in the AR field of view, and outputs hierarchical visualization results of six-dimensional errors at three levels: part level, overall level, and target area level. The AR glasses are used to output the virtual-real fusion image, spatial path guidance, and three-level six-dimensional error to realize radiotherapy positioning correction guidance; The reward function of the part priority ranking algorithm in the AI ​​core driving module is: in, For SPA-DRL agents in state Execute action The instantaneous scalar reward value obtained afterward; For the first The system state vector for each decision step, i.e., the state At any moment The value of ; For the first The selected adjustment action; To perform the action The resulting six-dimensional residual error vector; Let L2 norm be the residual error vector. ; For dimension Historical average operation time; For action Dimension The coupling coefficient; For dimension The current error value; This represents the absolute value of the coupling perturbation term; The residual error penalty weight; Weighting for operation time penalty; Weights are used to penalize coupling side effects.

2. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 1, characterized in that: The medical image processing module uses the HU-adaptive multi-scale fusion skin delineation network for automatic skin segmentation. The network includes: The HU statistical feature extraction module is used to extract the statistical features of the HU value distribution in CT data. The dual-branch decoder reconstructs high-resolution segmentation feature maps from the spatial domain and the frequency domain respectively, and performs feature fusion through cross-domain fusion weights adaptively generated based on the statistical features of HU. The HU-constrained segmentation head fuses the segmentation logits output by the convolutional network with a prior Gaussian function constructed based on typical HU values ​​of skin, outputting a skin tissue probability map.

3. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 2, characterized in that: The medical image processing module also includes a differentiable surface reconstruction framework with anatomical prior constraints, which is used to convert the skin tissue probability map into a signed distance function, and perform end-to-end joint optimization through bony landmark constraints, body symmetry constraints and radiotherapy surface landmark constraints to generate a patient digital twin that combines anatomical accuracy and visual realism.

4. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 1, characterized in that: The multimodal adaptive constraint registration algorithm in the spatial registration module constructs a five-dimensional adaptive weight system. The five-dimensional adaptive weight is the product of geometric distance weight factor, texture feature similarity weight factor, semantic consistency weight factor, uncertainty estimation weight factor, and dynamic change detection weight factor. A joint optimization objective function is constructed based on anatomical feature point constraints, smoothness constraints, and regularization constraints.

5. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 1, characterized in that: The state space of the site priority ranking algorithm in the AI ​​core driving module consists of six degrees of freedom positioning error and ten coupling coefficients between dimensions. The coupling coefficients are determined by statistical analysis of historical positioning data and kinematic calibration experiments of the treatment bed.

6. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 5, characterized in that: The dual-layer differentiated virtual-real fusion rendering in the AR visualization guidance module adopts Alpha blending technology, and the final pixel value is: in, For reference, the transparency of the body surface contour layer, To measure the real-time transparency of the patient's skin surface. For contour color modulation function, The contour line intensity function, These are the pixel values ​​of the body surface image acquired in real time. These are the pixel values ​​of the real-world background.

7. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 6, characterized in that: The automatic annotation of spatially misaligned contour lines in the AR visualization guidance module is achieved through real-time distance calculation. The distance is the signed shortest distance from the real-time body surface point to the reference body surface model, and is mapped to a four-level visual code according to the distance threshold: transparent if the distance is no more than 1mm, yellow dashed line if the distance is greater than 1mm but no more than 2mm, orange solid line if the distance is greater than 2mm but no more than 3mm, and red flashing if the distance is greater than 3mm.

8. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 7, characterized in that: The dynamic gradient arrow in the AR visualization guidance module determines its length based on the current real-time error value and performs color mapping based on the normalized error ratio. The Bezier trajectory prediction curve is generated based on the current body position key feature points, the target body position key feature points, and the control point offset vector determined by the body part priority sorting algorithm.

9. The augmented reality three-dimensional visualization radiotherapy positioning device based on artificial intelligence according to claim 1, characterized in that: The three-level six-dimensional error includes: Six-dimensional error at the site level is calculated based on independent local registration after anatomical region segmentation. The overall six-dimensional error is calculated based on a weighted fusion of the area of ​​each part and the registration confidence. The target area-level six-dimensional error is calculated by propagating the overall error to the target area center and superimposing local deformation compensation calculations based on the adjoint mapping.

Images