A virtual skeleton perspective method, device and storage medium

Abstract

This invention discloses a virtual skeleton perspective method, device, and storage medium, belonging to the field of medical image-assisted and computer vision-based 3D human body reconstruction technology. Addressing the shortcomings of existing medical imaging methods, such as the lack of visualization of the internal skeleton during positioning, poor real-time performance of iterative SKEL parameter solving, insufficient ability to measure real-world scales, and low reconstruction accuracy in occluded scenes, this invention replaces iterative optimization with a learned SKEL parameter estimation model, rapidly obtaining SKEL parameters through a single forward computation. The preferred scheme uses depth as a geometric prior to modulate attention, and adds bone length prediction to achieve metric scale recovery without internal parameters. Simultaneously, it realizes 3D skeleton virtual perspective overlay, positioning guidance, and real-world spatial distance measurement. This invention significantly improves the accuracy and efficiency of image positioning, reduces the radiation burden on patients, and balances reconstruction robustness and real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of medical imaging-assisted technology and computer vision three-dimensional human body reconstruction technology, and more particularly to the field of perspective calibration technology for DR / CT / MR medical images and visualization images, specifically to a virtual skeleton perspective method, device and storage medium. Background Technology

[0002] Digital radiography (DR), computed tomography (CT), and magnetic resonance imaging (MR) are currently the core medical imaging methods for clinical disease diagnosis, injury assessment, and rehabilitation follow-up. The image quality and diagnostic reliability of these imaging examinations are highly sensitive to imaging positioning and patient positioning. Positioning parameters such as the patient's limb rotation angle, joint flexion and extension, alignment of the X-ray centerline, and angle of incidence directly determine whether the target anatomical structure can be fully displayed, whether there is tissue overlap or obstruction, and the accuracy and reliability of subsequent image measurements. Currently, routine clinical image positioning mainly relies on the radiographer's personal experience, identification of surface landmarks, and simple geometric alignment tools, which has significant technical limitations.

[0003] To improve positioning accuracy, existing technologies are gradually introducing parametric human models to achieve 3D reconstruction of the human skeleton. Among them, the SMPL model is the mainstream parametric model of human skin mesh, and the SKEL model is a biomechanical skeletal parametric model that complements SMPL, which can accurately depict the shape and posture of the human skeleton. Existing skeletal reconstruction schemes based on these models mostly solve the SKEL skeletal parameters by constructing an energy function and using a case-by-case iterative optimization fitting method. Although skeletal reconstruction can achieve a certain degree of accuracy, in clinical positioning scenarios, there are still insurmountable shortcomings such as unstable accuracy of medical imaging positioning, high rework costs of imaging examinations, high radiation burden on patients, and the inability of skeletal parameter estimation to meet the real-time requirements of positioning. Therefore, there is an urgent need for a method that can visually determine that the target area is completely within the expected fluoroscopy range before X-ray irradiation is started, avoiding the problems of low accuracy caused by repeated fluoroscopy and the possibility of excessive radiation dose during re-irradiation. Summary of the Invention

[0004] To address the technical problems described in the background art, this application provides a virtual skeleton perspective method, apparatus, and storage medium. It is used to solve at least one of the following technical problems:

[0005] Firstly, the posture of the skeleton inside the body is not visible, and the positioning accuracy is uncontrollable. The core goal of positioning is to align with bony landmarks inside the body, such as joint spaces, bone axes, and pelvic tilt / rotation angles. However, current technology cannot visually present the true posture of the skeleton inside the body and the overlapping relationship of the projection before exposure / scanning. It can only be inferred indirectly by relying on surface landmarks. The positioning accuracy depends entirely on the technician's experience, and positioning deviations are very easy to occur.

[0006] Secondly, rework costs are high, and patients bear a heavy radiation burden. Positioning errors can directly lead to unqualified images, causing repeated exposures in DR, repeated localization scans in CT / MR, or partial rescans. This not only significantly increases examination time and technician workload, but also exposes patients to additional ionizing radiation, which does not comply with the basic principles of clinical radiation protection.

[0007] Third, the efficiency of solving skeletal parameters is low and cannot meet the real-time requirements of positioning. Existing iterative optimization fitting methods require multiple rounds of forward computation and reprojection operations for a single sample, and the inference time is usually hundreds of milliseconds or even several seconds, which cannot meet the core requirements of near real-time updates and dynamic adjustments of skeletal posture in clinical positioning.

[0008] Fourth, cross-modal applications are fragmented and have poor adaptability. DR is projection imaging, while CT / MR is volumetric data / multi-slice imaging. Existing methods are mostly designed for a single modality or a single task, lacking a unified parameterized representation and visualization output scheme for all-modal positioning assistance scenarios. They cannot be compatible with the positioning needs of multiple types of imaging equipment, resulting in high barriers to clinical deployment.

[0009] Fifth, the ability to measure true scale is insufficient and robustness is poor. Existing human body reconstruction methods based on monocular RGB mostly output results in a normalized relative coordinate system, which cannot directly output medical measurements such as metric bone length and human body dimensions. Existing reconstruction schemes that incorporate depth information mostly treat depth as an independent encoding branch for feature fusion, which not only significantly increases the computational cost and number of parameters of the model, but also has extremely poor robustness to depth loss and depth noise, and cannot stably output accurate measurement data that meets medical requirements.

[0010] Sixth, reconstruction accuracy is low and anatomical consistency is poor under occlusion and complex poses. Existing Transformer-based human reconstruction methods are prone to token mis-aggregation across foreground / background and across large depth transitions in clinical scenarios with limb occlusion and complex human poses. This results in significant deviations between the reconstructed skeletal model and the real human anatomical structure, failing to meet the accuracy requirements of clinical medical applications.

[0011] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0012] This invention provides a virtual skeleton perspective method, including acquiring a target image to be processed, the target image including DR / CT / MR medical images or RGB / RGB-D positioning camera images, and further including the following steps:

[0013] S1. Input the target image into a pre-trained SKEL parameter estimation model, and obtain the SKEL skeleton parameters through one forward computation. The SKEL skeleton parameters include at least shape parameters used to characterize the scale and proportion of the individual skeleton. Posture parameters used to characterize skeleton posture ;

[0014] S2. Input the SKEL skeletal parameters into the SKEL biomechanical skeletal model, and generate a three-dimensional skeletal model and skin mesh model through forward kinematics calculation;

[0015] S3. Based on the imaging geometry model corresponding to the target image, project the three-dimensional skeleton model and skin mesh model onto the two-dimensional plane of the target image, and align and superimpose them with the target image to achieve virtual skeleton perspective.

[0016] Preferably, in step S1, the SKEL parameter estimation model is an SMPL-to-SKEL parameter mapping model, and step S1 specifically includes:

[0017] S11. Process the target image to obtain the corresponding SMPL parameters, wherein the SMPL parameters include at least shape parameters. Attitude parameters ;

[0018] S12. Input the SMPL parameters into the pre-trained SMPL and convert it into the SKEL parameter mapping model. After one forward calculation, output the SKEL skeletal parameters.

[0019] The SMPL to SKEL parameter mapping model is constructed using a Transformer encoder and trained using SMPL and SKEL parameters as paired sample sets. During training, multiple loss constraints are applied to the pose parameters, shape parameters, and 3D joints generated by the forward feed of SKEL.

[0020] Preferably, in step S1, the SKEL parameter estimation model is an end-to-end image transformation to SKEL parameter regression model, and step S1 specifically includes:

[0021] S1a. Preprocess the target image, including cropping / scaling to a fixed resolution, normalizing pixel values, and generating an image patch that meets the network input requirements.

[0022] S1b: Use a visual Transformer to perform patch bedding on the preprocessed image to obtain a token feature sequence;

[0023] S1c: Use a multi-layer TransformerDecoder to perform cross-attention decoding on the token feature sequence and output a global human token;

[0024] S1d: Obtain SKEL skeleton parameters from the global human token through parameter regression head, and simultaneously obtain camera parameters that match the target image.

[0025] Preferably, the end-to-end image conversion to the SKEL parameter regression model is a DHSMR depth-enhanced regression model, and the target image further includes a depth map aligned with the RGB image; between steps S1b and S1c, a depth geometry prior modulation step is also included.

[0026] The depth map is used as a geometric prior, rather than an independent coding branch, to construct a geometric prior matrix. In the last 8 layers of the Transformer encoder, the logit of self-attention is modulated by geometric self-attention (GSA) to suppress token error aggregation across foreground / background and across large depth transitions.

[0027] The specific implementation of the geometric self-attention (GSA) is as follows: in the standard self-attention scoring item... Additional geometric offsets are added. , to make the final attention weight Wherein, the geometric offset The construction rule is: token pairs that are closer in 3D / have smaller depth differences within the foreground mask. Larger value range; token pairs that span foreground / background and large depth transitions. The value may be smaller or negative.

[0028] The geometric bias The corresponding geometric prior matrix is ​​constructed by calculating the depth-distance matrix on the patch mesh. Spatial distance matrix After robust normalization, the geometric prior matrix is ​​obtained by fusion. The fusion weight α and decay rate β are learnable parameters, σ(·) is the Sigmoid function, and the geometric prior is degenerated into a spatial prior only through the validity mask at the depth missing, so as to keep the RGB-only scene available.

[0029] Preferably, in step S1d, a bone length prediction head added to the Transformer decoding end is used to output the true metric length of each bone segment under the predefined bone segment topology; the method further includes a metric scale recovery step:

[0030] The scale factor is estimated by using the ratio of the predicted true bone length of the head output to the normalized bone length of the skeleton output by the SKEL model, and the 3D skeleton model and skin mesh model are scaled to the metric coordinate system. In scenes with camera intrinsics, the metric 3D coordinates are obtained by back-projecting the 2D pixels in combination with the depth map.

[0031] It also includes a real spatial distance measurement step: calculating the Euclidean distance on the metric 3D skeleton model, skin mesh model or projected 3D points after scale restoration, realizing the measurement output of the real spatial distance between any two points on the human body surface, such as bone length, shoulder width, height, and so on, and synchronously superimposing the measurement results on the target image.

[0032] Preferably, the pre-training step of the SKEL parameter estimation model specifically includes:

[0033] T1. Paired Sample Construction: For the acquired sample images, the ground truth value of SKEL parameters is obtained through an iterative optimization algorithm. The ground truth value of SKEL parameters is paired with the corresponding SMPL parameters or sample images and imaging geometric parameters to construct a training sample set. For RGB-D samples, the depth map is aligned simultaneously, a depth validity mask is generated, and the bone length label is calculated from the three-dimensional ground truth key points.

[0034] T2, Multi-stage coarse-to-fine training: A three-stage training strategy is adopted. The first stage freezes the backbone network and trains only the regression head to stabilize the global pose and camera parameters. The second stage introduces depth / geometric consistency constraints and enables geometric self-attention (GSA) fusion. The third stage adds bone length supervision and local joint constraints to perform hard example weighting and refinement.

[0035] T3. Human-machine collaborative iterative optimization: The SKEL parameters output by the model are manually corrected to meet the requirements of human biological anatomy. The corrected samples are added to the training sample set for retraining, and the high-quality labeled dataset is expanded iteratively.

[0036] Preferably, it also includes a medical image positioning assistance step:

[0037] Based on the generated 3D skeletal model and the radiographic protocol for target image examination, a quantitative index of positioning deviation is calculated. The quantitative index includes at least the angle between the bone axis and the detector / bed surface, the joint flexion and extension angle, the internal and external rotation angle, the centerline offset, and the left-right symmetry deviation. According to the quantitative index of positioning deviation, corresponding positioning adjustment suggestions are output.

[0038] Furthermore, the present invention also provides a virtual skeleton perspective device for realizing virtual skeleton perspective, specifically including:

[0039] The data input module is used to acquire the target image to be processed, which includes DR / CT / MR medical images or RGB / RGB-D positioned camera images; the SKEL parameter estimation module stores a pre-trained SKEL parameter estimation model, which is used to input the target image into the model and obtain the SKEL skeleton parameters through a forward calculation, wherein the SKEL skeleton parameters include at least shape parameters. With attitude parameters ;

[0040] The 3D model generation module is used to input the SKEL skeletal parameters into the SKEL biomechanical skeletal model and generate a 3D skeletal model and a skin mesh model through forward kinematic calculations.

[0041] The virtual perspective overlay module is used to project the three-dimensional skeleton model and skin mesh model onto the two-dimensional plane of the target image according to the imaging geometry model corresponding to the target image, and overlay them with the target image for display; and the positioning assistance module is used to calculate the positioning deviation quantification index based on the generated three-dimensional skeleton model and the photographic protocol of the target image examination, and output the corresponding positioning adjustment suggestions.

[0042] Preferably, the SKEL parameter estimation module includes an SMPL parameter acquisition submodule and an SMPL to SKEL parameter mapping submodule;

[0043] The SMPL parameter acquisition submodule is used to process the target image and obtain the corresponding SMPL parameters; the SMPL to SKEL parameter mapping submodule has a built-in pre-trained Transformer encoder mapping model, which is used to output SKEL skeleton parameters after a single forward calculation of the SMPL parameters; or

[0044] The SKEL parameter estimation module is a DHSMR depth-enhanced regression module, which includes an image preprocessing submodule, a visual coding submodule, a geometric self-attention modulation submodule, a Transformer decoding submodule, a parameter regression submodule, a bone length prediction submodule, and a scale restoration submodule.

[0045] The geometric self-attention modulation submodule is used to construct a geometric prior matrix and modulate the self-attention weights of the Transformer encoder using the depth map as the geometric prior; the bone length prediction submodule is used to output the true metric length of each bone segment; and the scale restoration submodule is used to scale the three-dimensional bone model and skin mesh model to the metric coordinate system.

[0046] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the virtual skeleton perspective method described above.

[0047] Beneficial effects:

[0048] 1. This invention achieves millisecond-level rapid estimation of skeletal parameters, completely solving the core problems of time-consuming iterative optimization and inability to meet the real-time positioning requirements of existing technologies. It replaces the iterative optimization fitting method of existing technologies with a pre-trained learning model, requiring only one forward computation to obtain SKEL skeletal parameters. Inference time can be compressed to the millisecond level, adapting to the near real-time updating and dynamic adjustment requirements of skeletal posture in medical imaging positioning, providing a core technological foundation for real-time virtual fluoroscopy and positioning assistance.

[0049] 2. This invention enables intuitive visualization of the internal skeletal posture before exposure, significantly improving the accuracy of medical image positioning and reducing rework costs and patient radiation burden. By precisely projecting and superimposing a three-dimensional skeletal model and skin model onto the target image, a "virtual skeletal perspective" effect is created. This allows radiologists to intuitively view the true posture and projection overlap of the internal skeleton before exposure / scanning, eliminating excessive reliance on surface landmarks and personal experience. Combined with quantitative calculation and adjustment suggestions for positioning deviation, one-time accurate positioning can be achieved, significantly reducing the probability of repeated exposure / scanning, lowering examination time costs and technician workload, while also reducing additional radiation exposure for patients, meeting clinical radiation protection requirements.

[0050] 3. This invention constructs a unified cross-modal technical solution, solving the problem of fragmented cross-modal applications in existing technologies. Specifically, it uses the SKEL biomechanical skeletal model as a unified parameterized output carrier, which is compatible with multimodal inputs such as DR projection imaging, CT / MR volumetric data imaging, and RGB / RGB-D camera images. Through unified imaging geometry adaptation, it achieves cross-modal virtual perspective overlay and positioning assistance, eliminating the need to develop independent solutions for different modal devices, and significantly reducing clinical deployment costs and adaptation difficulties.

[0051] 4. In the preferred DHSMR scheme of this invention, the depth map is innovatively used as a geometric prior rather than an independent encoding branch. The attention weight distribution of the Transformer is modulated by geometric self-attention (GSA) to suppress the incorrect aggregation of tokens across foreground / background and across large depth transitions with explicit three-dimensional geometric relationships. This significantly reduces the reconstruction error in limb occlusion and complex human pose scenes, ensures the consistency between the output skeletal model and the real human anatomical structure, and meets the accuracy requirements of clinical medical applications.

[0052] 5. This invention, through the addition of a bone length prediction head and a metric scale recovery mechanism, can achieve true scale restoration of 3D models in scenarios without camera intrinsic parameters. It can directly output quantitative measurement data such as bone length, shoulder width, and height that meet medical requirements. At the same time, it supports the measurement of the true spatial distance between any two points on the human body surface. It can not only be used to ensure the consistency of positioning in multiple examinations, but also directly serve extended scenarios such as clinical diagnosis and rehabilitation assessment.

[0053] 6. The GSA mechanism of this invention does not require additional depth coding branches, and will not significantly increase the computational overhead and number of parameters of the model. It can be adapted to clinical terminal devices with low computing power. At the same time, in scenarios with missing depth and noise, it can automatically degenerate to spatial prior only, maintain the availability of RGB-only scenarios, and have stronger compatibility with input data. It can be adapted to imaging equipment and positioning cameras with different clinical configurations.

[0054] 7. This invention adopts a multi-stage training strategy from coarse to fine, which ensures the stability and convergence accuracy of model training. At the same time, through the iterative backflow of samples after manual dissection correction, a high-quality clinical annotation dataset can be rapidly expanded, and the model's performance in difficult clinical scenarios such as special body types and complex postures can be continuously optimized to meet diverse clinical application needs. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This invention provides a non-standard lateral view full-body fluoroscopic imaging of the human body.

[0057] Figure 2 This invention provides a non-standard positional full-body fluoroscopic imaging image of the human body from the back.

[0058] Figure 3 This invention provides a non-standard frontal full-body perspective imaging of the human body.

[0059] Figure 4 This invention provides a standard lateral view full-body perspective imaging of the human body. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. Components of the embodiments of this application usually described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of this application claimed, but merely represents selected embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application.

[0062] Embodiment 1:

[0063] This embodiment discloses a virtual bone perspective method based on the parameter mapping from SMPL to SKEL. The core is to quickly convert the SMPL parameters obtained by the existing method into SKEL bone parameters through a pre-trained parameter mapping model, realizing virtual bone perspective overlay and DR positioning assistance.

[0064] The application scenario of this embodiment is the positioning assistance for DR knee anteroposterior radiography. The target image to be processed is the RGB image of the patient's body position collected by the positioning camera supporting the DR device. The specific implementation steps are as follows:

[0065] Step 1: Acquisition and preprocessing of the target image. An RGB image of the patient's knee positioning body position is collected by the positioning camera supporting the DR device. The collected RGB image is cropped and scaled to a fixed resolution of 256×256, and pixel value normalization is completed. At the same time, a human foreground mask is generated through a human semantic segmentation algorithm to suppress the interference of the background area on subsequent processing.

[0066] Step 2: Acquisition of SMPL parameters. The preprocessed RGB image is processed using an open-source human parameter estimation algorithm to obtain the corresponding SMPL parameter set, including 10-dimensional shape parameters , 46-dimensional pose parameters . At the same time, the three-dimensional coordinates of 24 joints generated by SMPL forward are obtained and the weak perspective camera parameters c matching the positioning camera; the camera parameters c include the scale s, translation amounts tx and ty.

[0067] Step 3: Fast mapping of SKEL parameters. The shape parameters , pose parameters , and three-dimensional joint coordinates The weak perspective camera parameters c are concatenated into an input feature vector, which is then input into the pre-trained SMPL and transformed into the SKEL parameter mapping model. After one forward computation, the output is a 10-dimensional SKEL shape parameter. 46-dimensional SKEL attitude parameters ;in,

[0068] The SMPL to SKEL parameter mapping model is constructed using a 6-layer Transformer encoder with a token dimension of 256 and 8 attention heads. The model is trained using paired (SMPL parameters, SKEL parameters) sample sets as training data. The total loss function during training is a weighted sum of all components, specifically:

[0069]

[0070] In the formula, For global orientation / root joint rotation MSE loss, weights ; For the MSE loss due to body joint rotation, weights , SKEL shape parameters MSE loss, weight ; The 3D joints after pelvic alignment are subjected to a confidence-weighted L1 loss, with the weights... ; For 2D keypoint reprojection with confidence-weighted L1 loss, the weights are... The true values ​​of the SKEL parameters in the training sample set are obtained by iteratively optimizing the energy function, which is:

[0071]

[0072] In the formula, For depth, the calculation formula is: ; The reprojection term for two-dimensional key points is calculated using the following formula: ;in, and ,default =100; For shape priors, the calculation formula is: ; B represents the joint angle prior, penalizing joint angle combinations that do not conform to the range of motion in human anatomy; B is the batch size. , , , Weights for each item;

[0073] Step 4: Generation of 3D skeleton and skin model. The SKEL shape parameters obtained in Step 3 are then used... Attitude parameters The SKEL biomechanical skeleton model, through forward kinematics calculations, generates a three-dimensional skeleton model containing 25 joints, as well as a corresponding human skin mesh model.

[0074] Step 5: Virtual Skeleton Perspective Overlay Visualization. Based on the camera parameters c obtained in Step 2, construct an imaging geometric model that matches the positioned camera. Project the 3D skeleton model and skin mesh model onto the 2D plane of the target RGB image to complete pixel-level alignment and overlay. The 3D skeleton is rendered using a semi-transparent red tubular shape, and the skin mesh is displayed using a white outline. Simultaneously, the projection center lines and angle scales of the bony structures corresponding to the knee joint space and tibial force line are overlaid and displayed.

[0075] Step 6: Positioning Assistance and Guidance. Based on the 3D skeleton model generated in Step 4, and in accordance with the standard radiographic protocol for DR knee joint anteroposterior radiographs, quantitative indicators of positioning deviation are calculated, including the deviation of the tibial force line from the detector's perpendicular angle, the knee flexion-extension angle, the lower limb internal and external rotation angles, and the offset of the X-ray centerline. Based on the calculated quantitative deviation indicators, corresponding adjustment suggestions are output to assist the technician in achieving precise positioning. Since the actual range of X-ray reception by the detector is limited, this embodiment only illustrates the method of fluoroscopy for a local area of ​​the human knee joint. However, this invention is not limited to local fluoroscopy and can perform fluoroscopic calculations for a large area of ​​the whole body. For details, please refer to the appendix of the instruction manual. Figures 1-4 As shown, where, Figures 1-3 These are illustrative images of the same person in different positions and postures. Figure 4 This is a perspective image of another person in front of the detector.

[0076] Example 2:

[0077] This embodiment provides an end-to-end virtual skeleton fluoroscopy and real-scale measurement method based on a DHSMR deep enhancement network, applied to lumbar spine MR scan positioning and orthopedic rehabilitation assessment scenarios. The specific implementation steps are as follows:

[0078] Step 1: Multimodal Image Preprocessing. RGB images and aligned depth maps of the patient's lumbar spine in positioning are acquired in real-time using an RGB-D positioning camera at a frame rate of 30fps. The following preprocessing operations are performed on the acquired multimodal images:

[0079] 1.1 The RGB image is cropped and scaled to a fixed resolution of 256×192, pixel value normalization is performed, and the image is divided into 16×16 image patches;

[0080] 1.2 Preprocessing of the depth map: The depth values ​​are uniformly converted to metric units and normalized to the [0,1] interval according to the maximum range of 5m, while retaining the original metric depth map. For pixels with missing or invalid depth, set them to zero and simultaneously generate a depth validity mask.

[0081] Step 2: End-to-end SKEL parameter estimation. The preprocessed RGB image and depth map are input into the pre-trained DHSMR depth-enhanced regression model. SKEL parameter estimation is completed through a single forward computation. The specific process is as follows:

[0082] 2.1 RGB backbone encoding: A visual Transformer is used to perform patch bedding on the preprocessed RGB image patch to obtain a token feature sequence with a dimension of 1280; the visual Transformer has a total of 32 layers, 16 attention heads, a patch size of 16, and a feature dimension of 1280.

[0083] 2.2 Depth geometry prior modulated attention: Geometric self-attention (GSA) is enabled starting from layer 24 of the Transformer encoder, covering layers 24-31 (8 layers in total). Standard self-attention is maintained for the first 0-23 layers. The depth map is not used as an additional token or independent encoder input; it is only used to construct the geometry prior matrix to modulate the self-attention logit. The specific implementation is as follows:

[0084] Calculate the depth-distance matrix on the patch mesh Spatial distance matrix After robust normalization, the geometric prior matrix is ​​obtained by fusion. Where α is the learnable fusion weight and σ(·) is the Sigmoid function; in the standard self-attention scoring term Additional geometric offsets are added. β is the learnable decay rate, and the final attention weights are: The geometric offset The construction rule is: token pairs that are closer in 3D / have smaller depth differences within the foreground mask. Larger value range; token pairs that span foreground / background and large depth transitions. The value is smaller or negative; at depth missing locations, the geometric prior is degenerated into a spatial prior only through a validity mask.

[0085] 2.3 Human Parameter Decoding. The encoded feature sequence is input into a 6-layer TransformerDecoder for cross-attention decoding, and a global human token is output. The TransformerDecoder has 8 attention heads, a single head dimension of 64, and a context dimension of 1280.

[0086] 2.4 Parameter and Bone Length Regression. The SKEL shape parameters were obtained by regressing from the global human tokens using a parameter regression head. Attitude parameters The parameters of the weak perspective camera are also included. At the same time, the bone length prediction head added at the decoding end outputs the true metric length of 24 bone segments under the predefined 25-joint skeleton topology, and the output is subject to non-negative constraints using the Softplus function.

[0087] Step 3: 3D Model Generation and Metric Scale Restoration. The SKEL shape parameters obtained from regression... Attitude parameters Input the SKEL biomechanical skeleton model, and generate a 3D skeleton joint and skin mesh model through forward kinematics calculation; root center the joints output by the SKEL model, select reference bone segments of the lumbar vertebrae and femur, calculate the ratio of the actual bone length output by the skeleton length prediction head to the normalized bone segment length, take the median of multiple ratios to obtain the scale factor s, and scale the 3D skeleton and skin mesh model to the metric coordinate system through the scale factor s to obtain a 3D model of true scale.

[0088] Step 4: Real-time virtual skeleton perspective overlay visualization. Based on the camera parameters obtained from regression, the metric 3D skeleton and skin mesh model are projected onto the real-time acquired RGB image to complete pixel-level alignment and overlay. The 3D skeleton is displayed using a semi-transparent blue tubular rendering, and the skin mesh is displayed using a semi-transparent gray mesh. Simultaneously, the bone length values ​​of each segment of the lumbar vertebra and the lumbar curvature angle scale are displayed overlaid.

[0089] Step 5: Human body real-space distance measurement, supporting the following three measurement modes:

[0090] Mode 1: Automatic bone length measurement. Based on a metric 3D skeleton model, it automatically calculates the actual length of each segment of the lumbar vertebrae, the bilateral iliac bones, and the difference in lower limb bone length between the affected and healthy sides, outputs the measurement values, and marks the corresponding bone segments in the overlay image;

[0091] Mode 2: Joint point distance measurement. Receives user-selected L1-L5 lumbar joint points and bilateral hip joint points, automatically calculating the total lumbar spine length and bilateral hip joint distance.

[0092] Mode 3: Distance measurement between any two points on the human body surface. The system receives two pixels selected by the user on a 2D RGB image, combines this with the camera's in-camera metric depth map to perform 3D projection of the pixels, obtains the corresponding metric 3D coordinates, calculates the Euclidean distance between the two points, and outputs the result.

[0093] Step 6: Real-time Positioning Assistance and Guidance. Based on a life-scale 3D skeleton model and combined with the standard positioning protocol for lumbar spine MR scanning, the positioning deviation quantification index is calculated in real time, including the angle between the lumbar spine midline and the scanning table surface, the pelvic tilt angle, the left and right rotation angle of the body, and the offset of the scanning centerline. Based on the calculated deviation quantification index, adjustment suggestions are output in real time, and the adjustment direction is marked in the overlay image.

[0094] The total training loss function of the DHSMR deep augmented regression model is:

[0095]

[0096] In the formula, For two-dimensional keypoints, use confidence-weighted L1 loss, with weights... ; Apply confidence-weighted L1 loss to the 3D keypoints after pelvic alignment, with weights... ; For attitude prior loss, weights ; For global orientation / root joint rotation MSE loss, weights ; For the MSE loss due to body joint rotation, weights ; For the MSE loss of shape parameters, weights ; For bone length supervision loss, weight ; For bone length consistency loss, weight ; For depth geometric constraint loss, weights .

[0097] Example 3:

[0098] This embodiment provides a pre-training method for a SKEL parameter estimation model used in virtual skeleton perspective. The specific implementation steps are as follows:

[0099] Step 1: Multimodal Sample Data Acquisition. Acquire clinical multimodal sample data, including DR images, CT / MR positioning images, and RGB / RGB-D positioning camera images, and simultaneously acquire the corresponding imaging geometric parameters, including fluoroscopic camera intrinsic / extrinsic parameters and DR SID / SDD projection parameters; for RGB-D samples, simultaneously acquire depth maps pixel-level aligned with the RGB images.

[0100] Step 2: Constructing the paired sample set, specifically including:

[0101] 2.1 Initial SKEL Parameter Truth Value Acquisition. For each sample data set, the corresponding SMPL parameters, 2D human keypoints, and human foreground mask are first obtained using existing algorithms. Then, based on the SKEL model, an energy function is constructed and iteratively optimized to obtain the initial SKEL parameter truth values. The energy function is:

[0102]

[0103] The definitions and calculation methods of each term in the formula are the same as in Example 1;

[0104] 2.2 Paired Sample Generation. The obtained ground truth values ​​of SKEL parameters are paired one by one with the corresponding SMPL parameters / sample images and imaging geometric parameters to construct SMPL and SKEL paired samples and images, respectively. For RGB-D samples, depth map alignment and dimension unification are completed simultaneously to generate a depth validity mask. The metric length labels of 24 bone segments are calculated from the 3D ground truth key points and added to the corresponding samples.

[0105] 2.3 Sample Preprocessing and Splitting. Paired samples were augmented and normalized, and then divided into training, validation, and test sets in an 8:1:1 ratio to complete the construction of the training sample set.

[0106] Step 3: Multi-stage training from coarse to fine, specifically including:

[0107] 3.1 Coarse training phase: Freeze the first 24 layers of the Transformer backbone network and train only the decoder, parametric regression head and bone length prediction head. The loss functions used are 2D keypoint loss, 3D keypoint loss and pose prior loss. The training rounds are 20.

[0108] Training phase 3.2: Unfreeze layers 24-31 of the Transformer encoder, enable Geometric Self Attention (GSA), add a new depth geometric constraint loss, and train for 30 epochs;

[0109] 3.3 Fine-tuning phase: Unfreeze all parameters of the entire network, add bone length supervision loss and bone length consistency loss, weight hard examples, and train for 50 rounds.

[0110] Step 4: Human-Machine Collaborative Iterative Optimization. Deploy the trained initial model in a clinical testing environment, input real-world patient positioning images, and obtain the model's output SKEL parameters and 3D skeletal model. Radiologists and clinical anatomy experts manually correct the model output, adjusting pelvic tilt / rotation, femoral neck axis, and knee flexion / extension and rotation parameters to conform to human biological anatomy. Add the manually corrected samples to the training sample set, and repeat the training process of Step 3, iterating 3-5 times to complete the final model training.

[0111] Example 4:

[0112] This embodiment provides a virtual skeleton perspective device, including a data input module, a pre-trained model storage module, a SKEL parameter estimation module, a 3D model generation module, a virtual perspective overlay module, a processor, and a memory; the data input module, the pre-trained model storage module, the SKEL parameter estimation module, the 3D model generation module, and the virtual perspective overlay module are all electrically connected to the processor and the memory.

[0113] The data input module is used to acquire the target image to be processed, including DR / CT / MR medical images and RGB / RGB-D positioning camera images, and is also used to receive the corresponding imaging geometric parameters and depth map data.

[0114] The pre-trained model storage module is used to store the SMPL-to-SKEL parameter mapping model and the DHSMR deep augmented regression model obtained by pre-training using the method described in Example 3.

[0115] The SKEL parameter estimation module is used to input the target image into the pre-trained model and obtain the SKEL skeleton parameters through a single forward computation. The SKEL skeleton parameters include at least shape parameters. With attitude parameters The 3D model generation module is used to input SKEL skeletal parameters into the SKEL biomechanical skeletal model, and generate a 3D skeletal model and a skin mesh model through forward kinematic calculations.

[0116] The virtual perspective overlay module is used to project the three-dimensional skeletal model and skin mesh model onto the two-dimensional plane of the target image according to the imaging geometry model corresponding to the target image, so as to complete the aligned overlay display.

[0117] In this embodiment, the device further includes a depth geometry prior modulation module, which is electrically connected to the SKEL parameter estimation module. This module is used to construct a geometric prior matrix using the depth map as the geometric prior, and to modulate the self-attention weights of the Transformer encoder. The device also includes a bone length prediction and scale restoration module, which is electrically connected to the 3D model generation module. This module outputs the true metric length of each bone segment and scales the 3D bone model and skin mesh model to a metric coordinate system. The device further includes a distance measurement module, which is electrically connected to the bone length prediction and scale restoration module and the virtual perspective overlay module. This module measures and annotates bone length and the true spatial distance between any two points on the human body surface. Finally, the device includes a positioning assistance module, which is electrically connected to the 3D model generation module. This module calculates a positioning deviation quantification index based on the generated 3D bone model and the photographic protocol for target image examination, and outputs corresponding adjustment suggestions.

[0118] Example 5:

[0119] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in any one of embodiments 1-3.

[0120] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A virtual skeleton perspective method, comprising acquiring a target image to be processed, said target image including DR / CT / MR medical images or RGB / RGB-D positioning camera images, characterized in that, It also includes the following steps: S1. Input the target image into a pre-trained SKEL parameter estimation model, and obtain the SKEL skeleton parameters through one forward computation. The SKEL skeleton parameters include at least shape parameters used to characterize the scale and proportion of the individual skeleton. Posture parameters used to characterize skeleton posture ; S2. Input the SKEL skeletal parameters into the SKEL biomechanical skeletal model, and generate a three-dimensional skeletal model and skin mesh model through forward kinematics calculation; S3. Based on the imaging geometry model corresponding to the target image, project the three-dimensional skeleton model and skin mesh model onto the two-dimensional plane of the target image, and align and superimpose them with the target image to achieve virtual skeleton perspective.

2. The virtual skeleton perspective method according to claim 1, characterized in that, In step S1, the SKEL parameter estimation model is an SMPL-to-SKEL parameter mapping model. Step S1 specifically includes: S11. Process the target image to obtain the corresponding SMPL parameters, wherein the SMPL parameters include at least shape parameters. Attitude parameters ; S12. Input the SMPL parameters into the pre-trained SMPL and convert it into the SKEL parameter mapping model. After one forward calculation, output the SKEL skeletal parameters. The SMPL to SKEL parameter mapping model is constructed using a Transformer encoder and trained using SMPL and SKEL parameters as paired sample sets. During training, multiple loss constraints are applied to the pose parameters, shape parameters, and 3D joints generated by the forward feed of SKEL.

3. The virtual skeleton perspective method according to claim 1, characterized in that, In step S1, the SKEL parameter estimation model is an end-to-end image transformation to SKEL parameter regression model. Step S1 specifically includes: S1a. Preprocess the target image, including cropping / scaling to a fixed resolution, normalizing pixel values, and generating an image patch that meets the network input requirements. S1b: Use a visual Transformer to perform patch bedding on the preprocessed image to obtain a token feature sequence; S1c: Use a multi-layer TransformerDecoder to perform cross-attention decoding on the token feature sequence and output a global human token; S1d: Obtain SKEL skeleton parameters from the global human token through parameter regression head, and simultaneously obtain camera parameters that match the target image.

4. The virtual skeleton perspective method according to claim 3, characterized in that, The end-to-end image conversion to the SKEL parameter regression model is a DHSMR depth-enhanced regression model, and the target image also includes a depth map aligned with the RGB image; between steps S1b and S1c, a depth geometry prior modulation step is also included. The depth map is used as a geometric prior, rather than an independent coding branch, to construct a geometric prior matrix. In the last 8 layers of the Transformer encoder, the logit of self-attention is modulated by geometric self-attention (GSA) to suppress token error aggregation across foreground / background and across large depth transitions. The specific implementation of the geometric self-attention (GSA) is as follows: in the standard self-attention scoring item... Additional geometric offsets are added. , to make the final attention weight Wherein, the geometric offset The construction rule is: token pairs that are closer in 3D / have smaller depth differences within the foreground mask. Larger value range; token pairs that span foreground / background and large depth transitions. The value may be smaller or negative. The geometric bias The corresponding geometric prior matrix is ​​constructed by calculating the depth-distance matrix on the patch mesh. Spatial distance matrix After robust normalization, the geometric prior matrix is ​​obtained by fusion. The fusion weight α and decay rate β are learnable parameters, σ(·) is the Sigmoid function, and the geometric prior is degenerated into a spatial prior only through the validity mask at the depth missing, so as to keep the RGB-only scene available.

5. The virtual skeleton perspective method according to claim 4, characterized in that, In step S1d, the true metric length of each bone segment under the predefined bone segment topology is output through a bone length prediction head added at the Transformer decoding end; the method also includes a metric scale recovery step: The scale factor is estimated by using the ratio of the predicted true bone length of the head output to the normalized bone length of the skeleton output by the SKEL model, and the 3D skeleton model and skin mesh model are scaled to the metric coordinate system. In scenes with camera intrinsics, the metric 3D coordinates are obtained by back-projecting the 2D pixels in combination with the depth map. It also includes a real spatial distance measurement step: calculating the Euclidean distance on the metric 3D skeleton model, skin mesh model or projected 3D points after scale restoration, realizing the measurement output of the real spatial distance between any two points on the human body surface, such as bone length, shoulder width, height, and so on, and synchronously superimposing the measurement results on the target image.

6. The virtual skeleton perspective method according to claim 1, characterized in that, The pre-training steps of the SKEL parameter estimation model specifically include: T1. Paired Sample Construction: For the acquired sample images, the ground truth value of SKEL parameters is obtained through an iterative optimization algorithm. The ground truth value of SKEL parameters is paired with the corresponding SMPL parameters or sample images and imaging geometric parameters to construct a training sample set. For RGB-D samples, the depth map is aligned simultaneously, a depth validity mask is generated, and the bone length label is calculated from the three-dimensional ground truth key points. T2, Multi-stage coarse-to-fine training: A three-stage training strategy is adopted. The first stage freezes the backbone network and trains only the regression head to stabilize the global pose and camera parameters. The second stage introduces depth / geometric consistency constraints and enables geometric self-attention (GSA) fusion. The third stage adds bone length supervision and local joint constraints to perform hard example weighting and refinement. T3. Human-machine collaborative iterative optimization: The SKEL parameters output by the model are manually corrected to meet the requirements of human biological anatomy. The corrected samples are added to the training sample set for retraining, and the high-quality labeled dataset is expanded iteratively.

7. The virtual skeleton perspective method according to claim 1, characterized in that, It also includes steps to assist in the positioning of medical images: Based on the generated 3D skeletal model and the radiographic protocol for target image examination, a quantitative index of positioning deviation is calculated. The quantitative index includes at least the angle between the bone axis and the detector / bed surface, the joint flexion and extension angle, the internal and external rotation angle, the centerline offset, and the left-right symmetry deviation. According to the quantitative index of positioning deviation, corresponding positioning adjustment suggestions are output.

8. A virtual skeleton perspective device, characterized in that, include: The data input module is used to acquire the target image to be processed, which includes DR / CT / MR medical images or RGB / RGB-D positioning camera images; The SKEL parameter estimation module stores a pre-trained SKEL parameter estimation model. The target image is input into the model, and SKEL skeleton parameters are obtained through a single forward computation. These SKEL skeleton parameters include at least shape parameters. With attitude parameters ; The 3D model generation module is used to input the SKEL skeletal parameters into the SKEL biomechanical skeletal model and generate a 3D skeletal model and a skin mesh model through forward kinematic calculations. The virtual perspective overlay module is used to project the three-dimensional skeleton model and skin mesh model onto the two-dimensional plane of the target image according to the imaging geometry model corresponding to the target image, and overlay them with the target image for display. And a positioning assistance module, which is used to calculate the positioning deviation quantification index based on the generated three-dimensional skeleton model and the photographic protocol of the target image inspection, and output the corresponding positioning adjustment suggestions.

9. The virtual skeleton perspective device according to claim 8, characterized in that, The SKEL parameter estimation module includes an SMPL parameter acquisition submodule and an SMPL to SKEL parameter mapping submodule; The SMPL parameter acquisition submodule is used to process the target image and obtain the corresponding SMPL parameters; the SMPL to SKEL parameter mapping submodule has a built-in pre-trained Transformer encoder mapping model, which is used to output SKEL skeleton parameters after a single forward calculation of the SMPL parameters; or The SKEL parameter estimation module is a DHSMR depth-enhanced regression module, which includes an image preprocessing submodule, a visual coding submodule, a geometric self-attention modulation submodule, a Transformer decoding submodule, a parameter regression submodule, a bone length prediction submodule, and a scale restoration submodule. The geometric self-attention modulation submodule is used to construct a geometric prior matrix and modulate the self-attention weights of the Transformer encoder using the depth map as the geometric prior; the bone length prediction submodule is used to output the true metric length of each bone segment; and the scale restoration submodule is used to scale the three-dimensional bone model and skin mesh model to the metric coordinate system.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the virtual skeleton perspective method according to any one of claims 1-7.

Images