A method and system for overlaying and presenting bone images based on augmented reality technology

Abstract

The present application provides a method and system for overlaying and presenting bone images based on augmented reality technology, which relates to the technical field of intelligent medical systems. The method includes: obtaining fracture image data for the fracture area of ​​the patient, the fracture image data including CT images, X-ray images and light field images; parsing the light model corresponding to the light field image; the light model provides the propagation information of light in the patient's fracture area space; determining the multimodal fusion features corresponding to the fracture image data based on the light model and the feature fusion network; the feature fusion network adopts a convolutional neural network; reconstructing a three-dimensional model of the fracture part based on the multimodal fusion features; aligning and calibrating the reconstructed three-dimensional model of the fracture part with the patient's fracture area in the actual surgical scene. Thus, the augmented reality technology, the light model and the feature fusion network are comprehensively applied to realize the real-time and accurate overlay presentation of the three-dimensional model of the fracture part in the real surgical scene space, which can significantly improve the accuracy and efficiency of the operation.

Description

Technical Field

[0001] The present application relates to the technical field of smart medical systems, and in particular to a method and system for overlaying and presenting bone images based on augmented reality technology. Background Art

[0002] Medical imaging technology plays a vital role in the diagnosis and treatment of fractures. Commonly used imaging techniques include X-rays, computed tomography (CT), and magnetic resonance imaging (MRI). Although these technologies have been widely used in clinical applications, they still have significant limitations in the presentation of fracture images.

[0003] Traditional two-dimensional imaging technologies, such as X-ray images, can provide a planar view of the fracture site, but lack a three-dimensional sense, forcing doctors to rely on their own spatial imagination to judge the three-dimensional structure of the fracture during diagnosis and treatment. This method not only increases the risk of misdiagnosis, but may also lead to inaccurate treatment plans and prolong the patient's recovery time. Although CT and MRI can provide more detailed tomographic images, they are still two-dimensional. Doctors need to piece together the three-dimensional structure through multiple tomographic images, which is time-consuming and error-prone.

[0004] In addition, during fracture surgery, doctors usually rely on preoperative imaging data to develop surgical plans, and rely on the corresponding imaging surgical plans during the operation. However, existing imaging technology cannot provide real-time visual guidance. Doctors need to rely on their own experience and imaging memory to judge the details of the operation, which not only increases the complexity and risk of the operation, but may also prolong the operation time, increase the patient's pain and medical costs.

[0005] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0006] The embodiments of the present application provide a method and system for overlaying and presenting bone images based on augmented reality technology, which is used to at least solve the problem that the two-dimensional imaging technology in the current related technology cannot provide real-time visual guidance for doctors.

[0007] In a first aspect, an embodiment of the present application provides a method for overlaying and presenting bone images based on augmented reality technology, which is applied to AR glasses. The method includes: obtaining fracture image data for a patient's fracture area, wherein the fracture image data includes a CT image, an X-ray image, and a light field image; parsing a light model corresponding to the light field image; the light model provides light propagation information in the patient's fracture area space; determining a multimodal fusion feature corresponding to the fracture image data based on the light model and a feature fusion network; the feature fusion network uses a convolutional neural network; wherein, in the feature fusion process, the light model is used as guiding information, and light consistency constraints are introduced into the feature fusion network to ensure the consistency of the fusion feature on the light propagation path, and the loss function L of the feature fusion network is:

[0008] L=λ ray L ray +λ feat L feat

[0009]

[0010]

[0011] Where, L feat represents the feature matching loss term, which ensures the consistency of the fusion feature and the input feature in the feature space; L ray represents the light consistency constraint loss term, which is used to ensure the consistency of the fusion features on the light propagation path; ray and λ feat are the weight coefficients of the corresponding loss terms; p represents a point on the light, P represents the set of all light points, and F fusion (p) represents the fusion feature at point p, R(p) represents the feature value of the light model at point p; f represents a point on the feature map, F represents the set of all points on the feature map, and F fusion (f) represents the fusion feature at point f, F input (f) represents the input feature at point f;

[0012] The weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of the features extracted by convolution:

[0013]

[0014]

[0015]

[0016]

[0017] In the formula, F conv represents the feature map after convolution, and b (k) Respectively represent the weight and bias term of the kth convolution kernel, represents the kth channel of the input feature map, * represents the convolution operation, and σ represents the ReLU activation function; represents the basic weight of the kth convolution kernel; α represents the adjustment coefficient, which is used to control the influence of the light model on the convolution kernel weight; represents the adjustment weight of the kth convolution kernel guided by the light model; w i represents the weight of the i-th ray, P i represents the set of all points on the i-th ray, represents the eigenvalue of the i-th ray in the k-th convolution kernel, N represents the number of rays; R i (p) represents the eigenvalue of the i-th ray at point p, θ represents the hyperparameter controlling the weight distribution, and Z represents the normalization factor;

[0018] Reconstructing a three-dimensional model of the fracture part based on the multimodal fusion features;

[0019] The reconstructed three-dimensional model of the fracture is aligned and calibrated with the patient's fracture area in the actual surgical scene.

[0020] In a second aspect, an embodiment of the present application provides a method for overlaying and presenting bone images based on augmented reality technology, wherein the system comprises: a data acquisition unit, for acquiring fracture image data for a patient's fracture area, wherein the fracture image data comprises a CT image, an X-ray image, and a light field image; a light model parsing unit, for parsing the light model corresponding to the light field image; the light model provides light propagation information in the patient's fracture area space; a fusion feature determination unit, for determining the multimodal fusion features corresponding to the fracture image data based on the light model and a feature fusion network; the feature fusion network adopts a convolutional neural network; wherein, in the feature fusion process, the light model is used as guiding information, and light consistency constraints are introduced into the feature fusion network to ensure the consistency of the fusion features on the light propagation path, and the loss function L of the feature fusion network is:

[0021] L=λ ray L ray +λ feat L feat

[0022]

[0023]

[0024] Where, L featrepresents the feature matching loss term, which ensures the consistency of the fusion feature and the input feature in the feature space; L ray represents the light consistency constraint loss term, which is used to ensure the consistency of the fusion features on the light propagation path; ray and λ feat are the weight coefficients of the corresponding loss terms; p represents a point on the light, P represents the set of all light points, and F fusion (p) represents the fusion feature at point p, R(p) represents the feature value of the light model at point p; f represents a point on the feature map, F represents the set of all points on the feature map, and F fusion (f) represents the fusion feature at point f, F input (f) represents the input feature at point f;

[0025] The weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of the features extracted by convolution:

[0026]

[0027]

[0028]

[0029]

[0030] In the formula, F conv represents the feature map after convolution, and b (k) Respectively represent the weight and bias term of the kth convolution kernel, represents the kth channel of the input feature map, * represents the convolution operation, and σ represents the ReLU activation function; represents the basic weight of the kth convolution kernel; α represents the adjustment coefficient, which is used to control the influence of the light model on the convolution kernel weight; represents the adjustment weight of the kth convolution kernel guided by the light model; w i represents the weight of the i-th ray, P i represents the set of all points on the i-th ray, represents the eigenvalue of the i-th ray in the k-th convolution kernel, N represents the number of rays; R i (p) represents the eigenvalue of the i-th ray at point p, θ represents the hyperparameter controlling the weight distribution, and Z represents the normalization factor;

[0031] A three-dimensional model construction unit, used for reconstructing a three-dimensional model of the fracture part based on the multimodal fusion feature;

[0032] The alignment and calibration unit is used to align and calibrate the reconstructed three-dimensional model of the fracture part with the patient's fracture area in the actual surgical scene.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method.

[0034] In a fourth aspect, an embodiment of the present application provides a storage medium, in which one or more program codes including execution instructions are stored, and the program code can be read and executed by an electronic device (including but not limited to a computer, a processor, a server, or a network device, etc.) to execute the steps of the above-mentioned method of the present application.

[0035] In a fifth aspect, an embodiment of the present application further provides a computer program product, wherein the computer program product comprises a computer program stored on a storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the above method.

[0036] The present application provides a method, system, electronic device and non-transitory computer-readable storage medium for overlaying bone images based on augmented reality technology, which realizes comprehensive visualization and real-time guidance of fracture sites through multimodal data fusion, innovative three-dimensional reconstruction technology and real-time AR (augmented reality) presentation, and can produce at least the following technical effects:

[0037] (1) By fusing CT images, X-ray images, and light field images, the light model and feature fusion network are used to extract multimodal features, and light consistency constraints are introduced to ensure the consistency of the fused features on the light propagation path. This not only improves the accuracy of the fused features, but also ensures the consistency of different image data during the reconstruction process, greatly improving the reconstruction accuracy of the three-dimensional structure of the fracture site, allowing doctors to intuitively observe the three-dimensional details of the fracture site, and overcoming the limitation of traditional two-dimensional images that lack stereoscopic perception.

[0038] (2) By utilizing multimodal fusion features, we can combine the advantages of different imaging data and make up for the shortcomings of single-modality imaging. CT images provide high-resolution tomographic information, X-ray images provide fast planar views, and light field images provide light propagation information. By fusing this information, we can obtain a more comprehensive and accurate three-dimensional structure of the fracture site.

[0039] (3) AR glasses are used to align and calibrate the reconstructed 3D fracture model with the patient's fracture area in the actual surgical scenario, allowing the doctor to see the 3D image of the patient's fracture site in real time during the operation, achieving real-time visual guidance, greatly reducing the doctor's need to rely on his or her own experience and image memory, reducing the complexity and risk of the operation, and reducing the patient's pain and medical costs.

[0040] (4) In this technical solution, the weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of convolution feature extraction and ensure that the influence of light propagation is considered in the feature extraction process. The convolution kernel weight adjustment method improves the spatial resolution capability of image data, ensures the accuracy of feature extraction, and further improves the reconstruction accuracy of the three-dimensional model.

[0041] Through this technical solution, the comprehensive application of augmented reality technology, light model and feature fusion network can realize the real-time and accurate superposition presentation of the three-dimensional model of the fracture in the real surgical scene space, which significantly improves the accuracy and efficiency of diagnosis and surgery, and has important clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A flowchart showing an example of a method for overlaying and presenting a bone image based on augmented reality technology according to an embodiment of the present application is shown;

[0044] Figure 2 Shown according to Figure 1 An exemplary operation flow chart of step S110 in FIG.

[0045] Figure 3 Shown according to Figure 1 An exemplary operation flow chart of S120;

[0046] Figure 4 Shown according to Figure 1 An exemplary operation flow chart of step S150 in FIG.

[0047] Figure 5 A structural block diagram of an example of a method for overlaying and presenting a bone image based on augmented reality technology according to an embodiment of the present application is shown;

[0048] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which this application belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantitative restrictions, but indicate the existence of at least one. "Include" or "comprises" and other similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and other similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0051] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in this application are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0052] Figure 1 A flowchart of an example of a method for overlaying and presenting a skeleton image based on augmented reality technology according to an embodiment of the present application is shown.

[0053] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities, which can be integrated or partially integrated in AR glasses, so as to achieve high-precision reconstruction of the three-dimensional structure of the fracture site by introducing light field images and feature fusion networks, combined with light consistency constraints and convolution kernel weight adjustments, and provide real-time visual guidance through AR technology, thereby significantly improving the accuracy and efficiency of fracture diagnosis and treatment.

[0054] In some examples, it may be integrated into the configuration server or client through software, hardware, or a combination of software and hardware, which should not be limited here.

[0055] The following will take the fracture virtual overlay platform as an exemplary implementation subject to introduce the details of the technical solution involved in this application. However, it should be understood that one or more steps involved in the following process can be implemented by one or more controllers or software installed and deployed in the client or server.

[0056] like Figure 1 As shown, in step S110, fracture image data for the fracture area of ​​the patient is acquired, and the fracture image data includes a CT image, an X-ray image, and a light field image.

[0057] In some embodiments, the patient uses various medical detection instruments to examine the patient's fracture area, and the detection instrument uploads the corresponding image data to the hospital service platform, and the AR glasses retrieve the corresponding image data from the hospital service platform through the patient ID. Here, the CT image can be obtained by scanning the patient's fracture area using a CT scanner to obtain a high-resolution tomographic image. The X-ray image can be an image of the fracture area taken by an X-ray device, providing two-dimensional plane information of the fracture site. In addition, a light field camera can be used to obtain a light field image of the fracture area to capture the direction and intensity information of the light. Thus, the fracture site is scanned at multiple angles and levels by computed tomography (CT), X-rays (X-ray) and light field cameras to obtain high-resolution two-dimensional image data and light field images.

[0058] According to the details of the shooting process of the light field camera, the light field camera can be placed in an appropriate position to ensure that the light propagation information of the fracture area can be fully captured, and then the parameters of the light field camera (such as focal length, exposure time, etc.) can be adjusted to shoot the light field image and obtain image data containing the light propagation path information.

[0059] In step S120 , a light model corresponding to the light field image is analyzed.

[0060] In some embodiments, a light model can be established based on the light data in the light field image using a ray tracing algorithm to analyze the direction, intensity, and propagation path of each light ray, which provides information on the propagation of light in the patient's fracture area space, helping to understand the three-dimensional structure and material properties of the fracture site.

[0061] In step S130, the multimodal fusion features corresponding to the fracture image data are determined based on the light model and the feature fusion network, and the feature fusion network adopts a convolutional neural network.

[0062] In some embodiments, a convolutional neural network (CNN) is used to extract the core features of the fracture from CT images and X-ray images, and the light propagation characteristics are extracted from the light model as additional features, and then the features extracted from the different modal image data are fused. Thus, the multimodal fusion features integrate the information of CT, X-ray and light field images, improve the understanding and characterization of the fracture area, and provide rich input data for the reconstruction of the three-dimensional model.

[0063] In some examples of the embodiments of the present application, during the feature fusion process, the light model is used as guiding information, and light consistency constraints are introduced into the feature fusion network to ensure the consistency of the fused features on the light propagation path. More specifically, the loss function L of the feature fusion network is:

[0064] L=λ ray L ray +λ feat L feat , Formula (1)

[0065]

[0066]

[0067] Where, L feat represents the feature matching loss term, which ensures the consistency of the fusion feature and the input feature in the feature space; L ray represents the light consistency constraint loss term, which is used to ensure the consistency of the fusion features on the light propagation path; ray and λ feat are the weight coefficients of the corresponding loss terms; p represents a point on the light, P represents the set of all light points, and F fusion (p) represents the fusion feature at point p, R(p) represents the feature value of the light model at point p; f represents a point on the feature map, F represents the set of all points on the feature map, and F fusion (f) represents the fusion feature at point f, F input (f) represents the input feature at point f.

[0068] Here, F fusion It can be a unified fusion feature of CT, X-ray and light field images. Exemplarily, a deep convolutional neural network (CNN) is used to extract features from CT, X-ray and light field images respectively. For each image, a multi-scale feature map is extracted. The extracted features are expressed as follows:

[0069] F CT =CNN CT (I CT ), Formula (4)

[0070] FX-ray =CNN X-ray (I X-ray ), Formula (5)

[0071] F LF =CNN LF (I LF ), Formula (6)

[0072] In the formula, I CT ,I X-ray and I LF Represent CT images, X-ray images and light field images respectively; F CT 、F X-ray and F LF represents the corresponding feature map.

[0073] Image registration technology is used to align the feature maps of CT, X-ray and light field images to ensure the consistency of features of different modalities. The features after registration are expressed as follows:

[0074] F CT′ =Align(F CT ,F X-ray ), Formula (7)

[0075] F LF′ =Align(F LF ,F X-ray ), Formula (8)

[0076] Then, the aligned multimodal features are connected to obtain the fused features:

[0077] F fusion =Fusion(F CT′ ,F X-ray ,F LF′ ), Formula (9)

[0078] Furthermore, the weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of the features extracted by convolution:

[0079]

[0080]

[0081]

[0082]

[0083] In the formula, F conv represents the feature map after convolution, and b (k) Respectively represent the weight and bias term of the kth convolution kernel, represents the kth channel of the input feature map, * represents the convolution operation, and σ represents the ReLU activation function; represents the basic weight of the kth convolution kernel; α represents the adjustment coefficient, which is used to control the influence of the light model on the convolution kernel weight; represents the adjustment weight of the kth convolution kernel guided by the light model; w i represents the weight of the i-th ray, P i represents the set of all points on the i-th ray, represents the eigenvalue of the i-th ray in the k-th convolution kernel, N represents the number of rays; R i (p) represents the eigenvalue of the i-th ray at point p, θ represents the hyperparameter controlling the weight distribution, and Z represents the normalization factor.

[0084] Here, the weight w i It reflects the influence of each ray on the fusion feature, which comprehensively considers the importance of the ray propagation path and the matching degree with other modal features. In the convolution operation, the weight of the convolution kernel is adjusted in combination with the ray model so that the reconstructed features have higher ray consistency.

[0085] Here, multimodal data fusion using CT, X-ray and light field images can effectively combine the advantages of different imaging modalities and help achieve high-precision 3D reconstruction. In particular, ray tracing technology can provide rich light propagation information and enhance the stereoscopic and realistic feeling of 3D reconstruction.

[0086] In this embodiment, by introducing the light model for feature fusion, the consistency and coherence of multimodal features on the light propagation path are ensured. Specifically, the light consistency constraint loss function is used to ensure the consistency of the fused features on the light propagation path. In addition, the light model is used to guide the weight adjustment of the convolution kernel, so that different convolution kernels can capture different feature information, thereby improving the diversity and accuracy of feature extraction.

[0087] In step S140, a three-dimensional model of the fractured part is reconstructed based on the multimodal fusion features.

[0088] In some embodiments, based on the fusion features, a voxel generation network is used to generate a three-dimensional voxel model. Specifically, the voxel generation network may include a convolution layer and a transposed convolution layer, and a higher-level feature is extracted through multiple convolution layers to extract higher-level features, including spatial information and depth information. Through the transposed convolution layer, the high-level features are converted into a three-dimensional voxel model, and the features are mapped into a three-dimensional space by expanding the spatial size of the feature map. Then, a triangular mesh generation algorithm (e.g., Marching Cubes) can be used to extract a three-dimensional surface model of the fracture site from the voxel model.

[0089] In some examples of the present application, the multimodal fusion feature is input into a three-dimensional convolutional neural network (3DCNN) to generate an initial fracture 3D model. Then, the initial fracture 3D model is input into a generative adversarial network to update the initial fracture 3D model to obtain a fracture 3D model. More specifically, the generative adversarial network includes a generator and a discriminator.

[0090] The generator receives the initial fracture 3D model as input and generates a detail-enhanced 3D model through multi-layer convolution and deconvolution operations. The channel attention and spatial attention modules are introduced into the convolution layer of the generator to calculate the importance of each position and adjust the weight of the feature map. The specific formula is as follows:

[0091] F CA =Sigmoid(FC2(ReLU(FC1(GAP(F))))) , Equation (14)

[0092] F SA =Sigmoid(Conv(Concat[AvgPool(F);MaxPool(F)])) , Formula (15)

[0093] In the formula, F CA represents the channel attention map, FC1 and FC2 are fully connected layers, and GAP is the global average pooling; F SA Represents the spatial attention map; AvgPool and MaxPool represent the average pooling operation and the maximum pooling operation respectively, and Concat represents the feature concatenation operation.

[0094] The discriminator receives the real 3D model and the generated detail-enhanced 3D model, and judges the authenticity of the input model through multi-layer convolution operations.

[0095] The loss function L of the generator G for:

[0096] L G =L gen +λ pixel L pixel , Formula (16)

[0097] L gen = -log(D attn (G attn (V initial ))) , Formula (17)

[0098] L pixel =‖V real -G attn (V initial )‖1, Formula (18)

[0099] Where, L gen is the generator loss, D attn represents the discriminator network, G attn represents the generator network, V initial represents the initial three-dimensional model of the fracture; L pixel represents pixel-level reconstruction loss; ‖·‖1 represents L1 norm; V real Represents the three-dimensional model of the real fracture part of the sample; pixel Represents the weight coefficient of pixel-level reconstruction loss.

[0100] Through pixel-level reconstruction loss, the generated 3D model is ensured to be consistent with the real model in details.

[0101] The loss function L of the discriminator D for:

[0102] L D = -log(D attn (V real ))-log(1-D attn (G attn (V initial ))) , Formula (19)

[0103] In this embodiment, a preliminary three-dimensional model of the fracture part is generated from the multimodal fusion features using 3D CNN, and a generator and a discriminator including an attention mechanism are constructed, and a detail-enhanced three-dimensional model is generated through multi-layer convolution and deconvolution operations. Here, channel attention and spatial attention modules are introduced into the convolution layer of the generator, the importance of each position is calculated, and the weight of the feature map is adjusted, and then the discriminator receives the real three-dimensional model and the generated detail-enhanced three-dimensional model, and judges the authenticity of the input model through multi-layer convolution operations.

[0104] Therefore, using 3D CNN to generate the initial three-dimensional model can effectively capture the spatial structural information of the fracture area and improve the basic reconstruction accuracy of the model. Introducing channel attention and spatial attention modules in the convolutional layer of the generator can calculate the importance of each position and adjust the weight of the feature map, so that the generator pays more attention to the details of important areas. The discriminator continuously improves the authenticity of the generated model through adversarial training, making the generated three-dimensional model more realistic in details. In the loss function design, pixel-level reconstruction loss is introduced on the basis of adversarial loss to guide the generator to generate a more realistic three-dimensional model and ensure that the generated three-dimensional model is consistent with the real model in details.

[0105] In step S150, the reconstructed three-dimensional model of the fractured part is aligned and calibrated with the fractured part of the patient in the actual surgical scene.

[0106] In some embodiments, the AR glasses can align and calibrate the information collected by the visual sensor module with the three-dimensional model of the fracture, for example, by combining the data of the camera and the inertial sensor, and calculating the position and posture of the AR glasses in real time to ensure the accurate alignment of the virtual image with the real scene. Exemplarily, through the feature point detection and tracking algorithm, the feature points in the surgical scene are identified and tracked in real time to ensure that the virtual image can dynamically adapt to changes in the surgical scene. Thus, through high-precision three-dimensional reconstruction models and real-time feature fusion, the augmented reality (AR) system can more accurately superimpose the virtual three-dimensional fracture model on the real surgical scene.

[0107] Through the embodiments of the present application, high-precision 3D reconstruction and precise augmented reality overlay are provided, so that doctors can view the 3D structure of the fracture site more comprehensively and intuitively. Through multimodal feature fusion and light model guidance, a high-precision 3D reconstruction model is provided to help doctors more accurately judge the fracture situation. In addition, through AR technology, doctors can intuitively show patients the fracture situation and treatment plan, enhancing patients' understanding and trust.

[0108] Figure 2 Shown according to Figure 1 An example operation flow chart of step S110 in FIG.

[0109] like Figure 2 As shown, in step S210, an original CT image, an original X-ray image, and an original light field image are received from a CT scanner, an X-ray device, and a light field camera, respectively.

[0110] In step S220, contrast enhancement processing is performed on the original CT image, the original X-ray image and the original light field image respectively to obtain corresponding enhanced CT image, enhanced X-ray image and enhanced light field image.

[0111] More specifically, contrast-limited adaptive histogram equalization (CLAHE) is used to process the original CT image. The image is divided into multiple small blocks, and histogram equalization is performed on each small block, but the contrast of each gray level is limited (clipping limit), and then the processed small blocks are recombined and bilinear interpolation is performed to eliminate the boundary effect. The specific formula is as follows:

[0112]

[0113] Where I1(x,y) represents the pixel value of the original CT image at position (x,y), I CLAHE (x, y) is the pixel value of the corresponding position of the enhanced CT image; L is the grayscale level; M and N are the width and height of the image respectively, h clip (k) is the cumulative distribution function of the clipped histogram.

[0114] Here, CLAHE can significantly improve the contrast of the fracture edge, effectively enhancing the contrast while avoiding over-enhancement, making subtle fracture lines more clearly visible, and helping to improve the clarity of the structural contours in the reconstructed three-dimensional model.

[0115] Adaptive Contrast Enhancement is used to process the original X-ray image. Specifically, the contrast is adaptively adjusted according to the brightness characteristics of the local area. Then, the local contrast enhancement factor of each pixel is calculated. The specific formula is as follows:

[0116]

[0117] Where I2(x,y) represents the pixel value of the original X-ray image at position (x,y), I adaptive (x, y) is the pixel value of the corresponding position of the enhanced X-ray image; μ local (x,y) is the average value of the local neighborhood of the image at position (x,y), σ local (x,y) is the standard deviation of the image in the local neighborhood at position (x,y), and ò represents a preset constant.

[0118] Here, in the fracture images, adaptive contrast enhancement processing can enhance the contrast in a local area, highlight the details of the fracture area, and help the reconstructed model to more prominently display the structure and position of the key fracture.

[0119] The multi-scale Retinex algorithm is used to process the original light field image. Specifically, the contrast of the light field image is enhanced by simulating the adaptability of the human visual system to changes in illumination, and then the weighted sum of different scales is calculated. The specific formula is as follows:

[0120]

[0121] Where I3(x,y) represents the pixel value of the original light field image at position (x,y), I adaptive (x, y) is the pixel value of the corresponding position of the enhanced light field image; w(s) represents the weight corresponding to scale s, G s represents a Gaussian filter of scale s.

[0122] Here, the multi-scale Retinex algorithm can effectively improve the contrast of light field images, simulate the adaptability of the human visual system to lighting changes, enhance the contrast of light field images, make the details of the image under different lighting conditions more obvious, and help improve the accuracy of diagnosis.

[0123] In step S230, feature points in the CT, X-ray and light field images are detected respectively based on a feature point detection algorithm, and the enhanced CT image, enhanced X-ray image and enhanced light field image are registered by feature point matching.

[0124] In some implementations, feature point detection algorithms such as SIFT, SURF, or ORB may be used to detect feature points in each enhanced modality image, and image alignment may be achieved through feature point matching.

[0125] In step S240, fracture image data for the fracture area of ​​the patient is determined based on the registered enhanced CT image, enhanced X-ray image and enhanced light field image.

[0126] In this embodiment, the contrast enhancement process is used to make the fracture details clearer in the image, providing more detailed information for 3D reconstruction and improving the accuracy and realism of the reconstruction results. In addition, the feature point registration method is used to ensure the spatial alignment of images of different modalities, so that the advantages of multimodal data can be fully utilized in the 3D reconstruction process to improve the accuracy of the reconstructed model.

[0127] Figure 3 Shown according to Figure 1 An example operation flow chart of S120 in FIG.

[0128] In step S310, a ray tracing algorithm is used to calculate the propagation path of light at different angles to construct an initial light model.

[0129] Specifically, a high-resolution light field image is obtained from a light field camera, denoised and contrast enhanced, and a ray tracing algorithm is used to calculate the propagation path of light at different angles to obtain an initial light model. The specific formula is as follows:

[0130]

[0131] In the formula, R initial (x, y, θ, φ) represents the initial light model at the position (x, y) and angle (θ, φ), N represents the number of rays; w e represents the weight of the e-th ray, L(x e ,y e ,θ,φ) indicates that the e-th ray is at position (x e ,y e ) at the light characteristics.

[0132] Here, the ray tracing algorithm is used to calculate the propagation path of light at different angles, accurately calculate the direction and intensity information of the light, build the initial light model, and ensure the basic accuracy of the light model:

[0133] In step S320, a convolutional neural network is used to extract multi-scale features from the light field image.

[0134] Here, a convolutional neural network (CNN) is used to extract multi-scale features from light field images through multi-layer convolution and pooling operations, ensuring that rich feature information can be obtained, providing strong support for the optimization of the light model.

[0135] F LF =CNN LF (I LF ), Formula (24)

[0136] In the formula, I LF represents the input light field image, F LF Represents the extracted multi-scale features.

[0137] In step S330, the initial light model and the multi-scale features are input into the deep learning model to determine the optimization weight of the light model. And according to the optimization weight Reconstruct the light model.

[0138] Here, in order to improve the accuracy and consistency of the initial light model, a deep learning model is used to optimize the weights in the initial light model by learning the relationship between the features of the light field image and the light model. Exemplarily, the deep learning model includes multiple convolutional layers and fully connected layers. The input layer receives the initial light model and the extracted feature map. The hidden layer extracts high-level features through convolution operations and performs weight optimization through the fully connected layer. The specific formula is as follows:

[0139]

[0140]

[0141] In the formula, DNN(·) represents the deep neural network function, R target (x,y,θ,φ) represents the reconstructed light model, Represents the weights optimized by the deep learning model.

[0142] In this embodiment, the initial light model and multi-scale features are input into the deep learning model, the relationship between the features of the light field image and the light model is learned to determine the optimization weight of the light model, and the light model is reconstructed according to the optimized weight, which significantly improves the accuracy and consistency of the model. Thus, the accuracy of the light propagation path is ensured through the optimized light model, so that the reconstructed three-dimensional model can better reflect the detailed information of the fracture area, and improve the accuracy and consistency of the three-dimensional reconstruction.

[0143] Figure 4 Shown according to Figure 1 An example operation flow chart of step S150 in FIG.

[0144] like Figure 4 As shown, in step S410, based on the optical tracking module, the coordinates of the marking points of multiple marking points are collected, and each marking point is preset in the fracture area of ​​the patient according to a preset marking position relationship.

[0145] In some embodiments, marker points are placed on the fracture area of ​​the patient and the surgical instrument, and the three-dimensional coordinates of these marker points are collected in real time through an optical tracking module provided in the VR glasses. The optical tracking module can accurately collect the coordinates of multiple marker points preset in the fracture area of ​​the patient, ensuring high accuracy and reliability of the data.

[0146] In step S420, the least square method is used to calculate the rotation matrix R and the translation vector t according to the position relationship of the markers. The specific formula is as follows:

[0147]

[0148] The optimal R and t are solved by singular value decomposition; where H g and Q g are the coordinates of the g-th marker point in the three-dimensional model of the fracture and the actual surgical scene, respectively.

[0149] In step S430, based on the solved optimal rotation matrix R and translation vector t, a rigid transformation is used to preliminarily align the three-dimensional model of the fracture with the actual surgical scene. The specific formula is as follows:

[0150] T rigid (x) = R x + t , Formula (28)

[0151] Where x represents the initial coordinates of any voxel point in the three-dimensional model of the fracture, T rigid (x) represents the coordinate after rigid transformation.

[0152] Here, based on the marker position relationship, the least square method is used to calculate the rotation matrix and translation vector, and the optimal solution is solved through singular value decomposition (SVD) to achieve the preliminary alignment of the 3D model.

[0153] In step S440, the Laplacian matrix is ​​used to calculate each control point p u The weight w u , and based on the weight w u A non-rigid transformation is performed to adapt to the deformation and displacement of the patient's fracture site. The specific formula is as follows:

[0154]

[0155] Among them, T non-rigid (x) represents the coordinates after non-rigid transformation, and φ(r) represents the Gaussian function.

[0156] Here, the Laplace matrix is ​​used to calculate the weight of each control point, and a non-rigid transformation is performed based on the weight to adapt to the deformation and displacement of the patient's fracture site. Thus, a preliminary alignment is performed through a rigid transformation, and then a fine adjustment is performed through a non-rigid transformation to improve the alignment accuracy.

[0157] In step S450, the rigidly transformed coordinate T is adjusted by an adaptive adjustment algorithm. rigid (x) and the non-rigid transformed coordinates T non-rigid (x) is fused to dynamically adjust the alignment state of the 3D model. The specific formula is as follows:

[0158] T adaptive (x) = T rigid (x)+β·(T non-rigid (x)-T rigid (x)), Formula (30)

[0159] Where, T adaptive (x) represents the coordinate after adaptive adjustment, and β represents the adaptive adjustment coefficient.

[0160] Thus, the coordinates after the rigid transformation and the coordinates after the non-rigid transformation are fused through an adaptive adjustment algorithm to dynamically adjust the alignment state of the three-dimensional model. In this embodiment, the position data of the patient and the surgical instrument are collected in real time through an optical tracking module, and the alignment state of the three-dimensional model is dynamically updated to ensure that the model is highly consistent with the actual scene during the operation, so that the doctor can directly refer to the virtual surgical plan (for example, the surgical navigation path) in the three-dimensional model during the operation, making the surgical process more efficient and accurate.

[0161] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0162] Figure 5 A structural block diagram of an example of a method for overlaying and presenting a skeleton image based on augmented reality technology according to an embodiment of the present application is shown.

[0163] like Figure 5 As shown, the skeleton image overlay presentation method 500 based on augmented reality technology includes a data acquisition unit 510, a light model parsing unit 520, a fusion feature determination unit 530, a three-dimensional model construction unit 540 and an alignment calibration unit 550.

[0164] The data acquisition unit 510 is used to acquire fracture image data of the fracture area of ​​the patient, wherein the fracture image data includes CT images, X-ray images and light field images.

[0165] The light model parsing unit 520 is used to parse the light model corresponding to the light field image; the light model provides the propagation information of the light in the patient's fracture area space.

[0166] The fusion feature determination unit 530 is used to determine the multimodal fusion features corresponding to the fracture image data based on the light model and the feature fusion network; the feature fusion network adopts a convolutional neural network.

[0167] In the feature fusion process, the light model is used as the guiding information, and the light consistency constraint is introduced into the feature fusion network to ensure the consistency of the fusion features on the light propagation path. The loss function L of the feature fusion network is:

[0168] L=λ ray L ray +λ feat L feat

[0169]

[0170]

[0171] Where, L feat represents the feature matching loss term, which ensures the consistency of the fusion feature and the input feature in the feature space; L ray represents the light consistency constraint loss term, which is used to ensure the consistency of the fusion features on the light propagation path; ray and λ feat are the weight coefficients of the corresponding loss terms; p represents a point on the light, P represents the set of all light points, and F fusion (p) represents the fusion feature at point p, R(p) represents the feature value of the light model at point p; f represents a point on the feature map, F represents the set of all points on the feature map, and F fusion (f) represents the fusion feature at point f, F input (f) represents the input feature at point f;

[0172] The weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of the features extracted by convolution:

[0173]

[0174]

[0175]

[0176]

[0177] In the formula, F conv represents the feature map after convolution, and b (k) Respectively represent the weight and bias term of the kth convolution kernel, represents the kth channel of the input feature map, * represents the convolution operation, and σ represents the ReLU activation function; represents the basic weight of the kth convolution kernel; α represents the adjustment coefficient, which is used to control the influence of the light model on the convolution kernel weight; represents the adjustment weight of the kth convolution kernel guided by the light model; w i represents the weight of the i-th ray, P i represents the set of all points on the i-th ray, represents the eigenvalue of the i-th ray in the k-th convolution kernel, N represents the number of rays; R i (p) represents the eigenvalue of the i-th ray at point p, θ represents the hyperparameter controlling the weight distribution, and Z represents the normalization factor.

[0178] The three-dimensional model construction unit 540 is used to reconstruct the three-dimensional model of the fracture part based on the multimodal fusion feature.

[0179] The alignment and calibration unit 550 is used to align and calibrate the reconstructed three-dimensional model of the fractured part with the fractured area of ​​the patient in the actual surgical scene.

[0180] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the above-mentioned bone image overlay presentation method based on augmented reality technology in the present application.

[0181] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned bone image overlay presentation method based on augmented reality technology.

[0182] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a bone image overlay presentation method based on augmented reality technology.

[0183] Figure 6 is a schematic diagram of the hardware structure of an electronic device for executing a method for overlaying and presenting a skeleton image based on augmented reality technology provided by another embodiment of the present application, such as Figure 6 As shown, the device includes:

[0184] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0185] The device for executing the skeleton image superposition presentation method based on augmented reality technology may further include: an input device 630 and an output device 640 .

[0186] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0187] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for overlaying and presenting a skeletal image based on augmented reality technology in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the method for overlaying and presenting a skeletal image based on augmented reality technology in the above method embodiment is implemented.

[0188] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 640 can include a display device such as a display screen.

[0190] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the bone image overlay presentation method based on augmented reality technology in any of the above method embodiments is executed.

[0191] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0192] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0193] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0194] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0195] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0196] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0197] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for overlaying and presenting a skeleton image based on augmented reality technology, applied to AR glasses, characterized in that: The method comprises: Acquire fracture image data for the fracture area of ​​the patient, wherein the fracture image data includes a CT image, an X-ray image, and a light field image; Analyzing the light model corresponding to the light field image; the light model provides light propagation information in the patient's fracture area space; Determining the multimodal fusion features corresponding to the fracture image data based on the light model and the feature fusion network; the feature fusion network adopts a convolutional neural network; Reconstructing a three-dimensional model of the fracture part based on the multimodal fusion features; Aligning and calibrating the reconstructed three-dimensional model of the fractured part with the fractured area of ​​the patient in the actual surgical scene; Wherein, the light model corresponding to the analysis of the light field image includes: Use the ray tracing algorithm to calculate the propagation path of light at different angles to build an initial light model: In the formula, R initial (x, y, θ, φ) represents the initial light model at the position (x, y) and angle (θ, φ), N represents the number of rays; w e represents the weight of the e-th ray, L(x e ,y e ,θ,φ) indicates that the e-th ray is at position (x e ,y e ) at the light characteristics; Extracting multi-scale features from light field images using convolutional neural networks: F LF =CNN LF (I LF ) In the formula, I LF represents the input light field image, F LF Represents the extracted multi-scale features; Input the initial light model and the multi-scale features into a deep learning model to determine the optimization weights of the light model And optimize the weights according to the Reconstruct the light model: In the formula, DNN(·) represents the deep neural network function, R target (x,y,θ,φ) represents the reconstructed light model, Represents the weights optimized by the deep learning model; In the feature fusion process, the light model is used as the guiding information, and the light consistency constraint is introduced into the feature fusion network to ensure the consistency of the fusion features on the light propagation path. The loss function of the feature fusion network is for: In the formula, Represents the feature matching loss term, ensuring the consistency of the fusion features and the input features in the feature space; represents the light consistency constraint loss term, which is used to ensure the consistency of the fusion features on the light propagation path; ray and λ feat are the weight coefficients of the corresponding loss terms; p represents a point on the ray, represents the set of all light points, F fusion (p) represents the fusion feature at point p, R(p) represents the feature value of the light model at point p; f represents a point on the feature map, Represents the set of all points on the feature map, F fusion (f) represents the fusion feature at point f, F input (f) represents the input feature at point f.

2. The method according to claim 1, characterized in that: The weights of each convolution kernel in the feature fusion network are adjusted based on the light model to improve the light consistency of the features extracted by convolution: In the formula, F conv represents the feature map after convolution, and b (k) Respectively represent the weight and bias term of the kth convolution kernel, represents the kth channel of the input feature map, * represents the convolution operation, and σ represents the ReLU activation function; represents the basic weight of the kth convolution kernel; α represents the adjustment coefficient, which is used to control the influence of the light model on the convolution kernel weight; represents the adjustment weight of the kth convolution kernel guided by the light model; w i represents the weight of the i-th ray, P i represents the set of all points on the i-th ray, represents the eigenvalue of the i-th ray in the k-th convolution kernel, N represents the number of rays; R i (p) represents the eigenvalue of the i-th ray at point p, θ represents the hyperparameter controlling the weight distribution, and Z represents the normalization factor.

3. The method according to claim 1, characterized in that The step of obtaining fracture imaging data for the fracture area of ​​the patient includes: receiving a raw CT image, a raw X-ray image, and a raw light field image from a CT scanner, an X-ray device, and a light field camera, respectively; The original CT image, the original X-ray image and the original light field image are respectively subjected to contrast enhancement processing to obtain corresponding enhanced CT image, enhanced X-ray image and enhanced light field image: The original CT image is processed using contrast limited adaptive histogram equalization: Where I1(x,y) represents the pixel value of the original CT image at position (x,y), I CLAHE (x, y) is the pixel value of the corresponding position of the enhanced CT image; L is the grayscale level; M and N are the width and height of the image respectively, h clip (k) is the cumulative distribution function of the clipped histogram; The original X-ray image is processed using adaptive contrast enhancement: Where I2(x,y) represents the pixel value of the original X-ray image at position (x,y), I adaptive (x, y) is the pixel value of the corresponding position of the enhanced X-ray image; μ local (x,y) is the average value of the local neighborhood of the image at position (x,y), σ local (x,y) is the standard deviation of the image in the local neighborhood of position (x,y), and ∈ represents a preset constant; The original light field image is processed using a multi-scale Retinex algorithm: Where I3(x,y) represents the pixel value of the original light field image at position (x,y), I adaptive (x, y) is the pixel value of the corresponding position of the enhanced light field image; w(s) represents the weight corresponding to scale s, G s represents a Gaussian filter of scale s; Based on a feature point detection algorithm, feature points in the CT, X-ray and light field images are detected respectively, and the enhanced CT image, the enhanced X-ray image and the enhanced light field image are registered by matching the feature points; The fracture imaging data for the fracture area of ​​the patient is determined according to the registered enhanced CT image, enhanced X-ray image and enhanced light field image.

4. The method according to claim 3, characterized in that The method of reconstructing a three-dimensional model of a fractured part based on the multimodal fusion feature comprises: Inputting the multimodal fusion features into a three-dimensional convolutional neural network to generate an initial three-dimensional model of the fracture; Inputting the initial three-dimensional model of the fracture part into a generative adversarial network to update the initial three-dimensional model of the fracture part to obtain a three-dimensional model of the fracture part; the generative adversarial network includes a generator and a discriminator; Introduce channel attention and spatial attention modules in the convolutional layer of the generator, calculate the importance of each position, and adjust the weight of the feature map: F CA =Sigmoid(FC2(ReLU(FC1(GAP(F))))) F SA =Sigmoid(Conv(Concat[AvgPool(F);MaxPool(F)])) In the formula, F CA represents the channel attention map, FC1 and FC2 are fully connected layers, and GAP is the global average pooling; F SA Represents the spatial attention map; AvgPool and MaxPool represent the average pooling operation and the maximum pooling operation respectively, and Concat represents the feature concatenation operation; The loss function of the generator for: In the formula, is the generator loss, D attn represents the discriminator network, G attn represents the generator network, V initial represents the three-dimensional model of the initial fracture; represents pixel-level reconstruction loss; ‖·‖1 represents L1 norm; V real Represents the three-dimensional model of the real fracture part of the sample; pixel Represents the weight coefficient of pixel-level reconstruction loss; Loss function of the discriminator for:

5. The method according to claim 4, characterized in that AR glasses are equipped with an optical tracking module. The step of aligning and calibrating the reconstructed three-dimensional model of the fractured part with the fractured part of the patient in the actual surgical scene comprises: Based on the optical tracking module, the coordinates of the marking points of the plurality of marking points are collected; each of the marking points is preset in the fracture area of ​​the patient according to a preset marking position relationship; According to the marker position relationship, the least squares method is used to calculate the rotation matrix R and the translation vector t: The optimal R and t are solved by singular value decomposition; where H g and Q g are the coordinates of the g-th marker point in the three-dimensional model of the fracture and the actual surgical scene, respectively; Based on the optimal R and t solved, the 3D model of the fracture is preliminarily aligned with the actual surgical scene using rigid transformation: T rigid (x)=R·x+t Where x represents the initial coordinates of any voxel point in the three-dimensional model of the fracture, T rigid (x) represents the coordinate after rigid transformation; Use the Laplacian matrix to calculate each control point p u The weight w u , and based on the weight w u Perform a non-rigid transformation to accommodate deformation and displacement of the patient's fracture site: Among them, T non-rigid (x) represents the coordinate after non-rigid transformation, φ(r) represents the Gaussian function; The coordinates T after rigid transformation are adjusted by adaptive adjustment algorithm rigid (x) and the non-rigid transformed coordinates T non-rigid (x) Fusion is performed to dynamically adjust the alignment state of the 3D model: T adaptive (x)=T rigid (x)+β·(T non-rigid (x)-T rigid (x)) Where, T adaptive (x) represents the coordinate after adaptive adjustment, and β represents the adaptive adjustment coefficient.

6. A skeleton image superposition presentation system based on augmented reality technology, used to implement the method as described in any one of claims 1 to 5, characterized in that: The system comprises: A data acquisition unit, used to acquire fracture image data for a fracture area of ​​a patient, wherein the fracture image data includes a CT image, an X-ray image, and a light field image; A light model parsing unit, used to parse the light model corresponding to the light field image; the light model provides the propagation information of the light in the patient's fracture area space; A fusion feature determination unit, used for determining the multimodal fusion features corresponding to the fracture image data based on the light model and the feature fusion network; the feature fusion network adopts a convolutional neural network; A three-dimensional model construction unit, used for reconstructing a three-dimensional model of the fracture part based on the multimodal fusion feature; The alignment and calibration unit is used to align and calibrate the reconstructed three-dimensional model of the fracture part with the patient's fracture area in the actual surgical scene.

7. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A storage medium, characterized in that: The storage medium stores program code, which can be called by a processor to execute the method as claimed in any one of claims 1 to 5.

Images