Fracture reduction path auxiliary planning method and system based on augmented reality technology

Abstract

The invention provides a fracture reduction path auxiliary planning method and system based on an augmented reality technology, and relates to the technical field of intelligent medical systems. The method comprises the following steps: acquiring fracture image data for a fracture area of a patient, wherein the fracture image data comprises CT data and X-ray data; reconstructing a fracture part three-dimensional model based on the fracture part image data; matching and calibrating the three-dimensional model of the fracture part and the fracture area of the patient in the actual scene; the calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture operation requirements are input into a fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path, and the fracture reduction path planning model comprises a reinforcement learning module and a multi-physics field simulation optimization module which are cascaded; and rendering each fracture reduction path, and determining a target fracture reduction path according to the detected interaction instruction. Therefore, the augmented reality technology is applied to the fracture reduction scene, and the success rate of the operation and the postoperative rehabilitation effect of the patient can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of smart medical systems, and in particular to a fracture reduction path auxiliary planning method and system based on augmented reality technology. Background Art

[0002] With the development of medical imaging technology, fracture reduction surgery has gradually developed towards minimally invasive and precise directions. Traditional fracture reduction surgery mainly relies on the doctor's experience and visual feedback. Especially for complex fractures, doctors need to complete the reduction through repeated attempts and adjustments, which not only prolongs the operation time and increases the surgical risk, but also makes it difficult to ensure the accuracy of fracture reduction, which can easily lead to poor alignment of the fracture site, thus affecting fracture healing and functional recovery.

[0003] In addition, in the current imaging-assisted fracture reduction technology, doctors mainly operate through two-dimensional images (such as X-rays) or direct vision with the naked eye. Doctors cannot intuitively feel the three-dimensional spatial structure of the fracture site, especially when the field of vision is limited or the fracture type is complex. It is difficult to accurately judge the relative position of the bone fragments and the optimal reduction direction, which also relatively increases the difficulty and risk of the operation.

[0004] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0005] The embodiments of the present application provide a fracture reduction path auxiliary planning method and system based on augmented reality technology, which is used to at least solve the problem that the current related technology relying on two-dimensional fracture reduction image-assisted technology cannot provide a better fracture reduction reference.

[0006] In a first aspect, an embodiment of the present application provides a fracture reduction path auxiliary planning method based on augmented reality technology, which is applied to AR glasses, and the method comprises: obtaining fracture image data for a patient's fracture area, wherein the fracture image data comprises CT data and X-ray data; reconstructing a three-dimensional model of the fracture part based on the fracture image data; the three-dimensional model of the fracture part comprises a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; matching and calibrating the reconstructed three-dimensional model of the fracture part with the patient's fracture area in an actual surgical scene; inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgical requirements into a fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; rendering each of the optimized fracture reduction paths, and determining a target fracture reduction path according to the detected interactive instructions; the fracture reduction path planning model comprises a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0007] The reinforcement learning module is used to determine at least one initial reduction path according to the fracture type, the matched and calibrated three-dimensional model of the fracture part and the fracture surgery requirements; the state of the reinforcement learning module is defined according to the geometric features of the matched and calibrated three-dimensional model of the fracture part, the relative orientation information of the bone blocks and the fracture type; the action space of the reinforcement learning module is defined according to the movement, rotation and force of the bone blocks; the reward function of the reinforcement learning module is defined according to the fracture surgery requirements, and the fracture surgery requirements include reduction accuracy requirements, path smoothness requirements and physiological constraint requirements;

[0008] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path;

[0009] The reward function R of the reinforcement learning module is expressed as follows:

[0010] R=r accuracy +r smoothness +r physiology

[0011] r accuracy = -α·||M target -M current ||

[0012]

[0013] In the formula, r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness Represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissue; M target is the target position coordinate, indicating the position that the bone block should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the accuracy adjustment parameter, which is used to adjust the impact of reset accuracy on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0014] In a second aspect, an embodiment of the present application provides a fracture reduction path auxiliary planning system based on augmented reality technology, the system comprising: a data acquisition unit, used to acquire fracture image data for a patient's fracture area, the fracture image data comprising CT data and X-ray data; a model reconstruction unit, used to reconstruct a three-dimensional model of the fracture part based on the fracture image data; the three-dimensional model of the fracture part comprises a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; a model calibration unit, used to match and calibrate the reconstructed three-dimensional model of the fracture part with the patient's fracture area in an actual surgical scene; a path planning unit, used to input the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgical requirements into a fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; a path confirmation unit, used to render each of the optimized fracture reduction paths, and determine a target fracture reduction path according to the detected interaction instructions; the fracture reduction path planning model comprises a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0015] The reinforcement learning module is used to determine at least one initial reduction path according to the fracture type, the matched and calibrated three-dimensional model of the fracture part and the fracture surgery requirements; the state of the reinforcement learning module is defined according to the geometric features of the matched and calibrated three-dimensional model of the fracture part, the relative orientation information of the bone blocks and the fracture type; the action space of the reinforcement learning module is defined according to the movement, rotation and force of the bone blocks; the reward function of the reinforcement learning module is defined according to the fracture surgery requirements, and the fracture surgery requirements include reduction accuracy requirements, path smoothness requirements and physiological constraint requirements;

[0016] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path;

[0017] The reward function R of the reinforcement learning module is expressed as follows:

[0018] R=r accuracy +r smoothness +r physiology

[0019] r accuracy = -α·||M target -M current ||

[0020]

[0021] In the formula, r accuracyRepresents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness Represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissue; M target is the target position coordinate, indicating the position that the bone block should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the accuracy adjustment parameter, which is used to adjust the impact of reset accuracy on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method.

[0023] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of the above-mentioned method of the present application.

[0024] In a fifth aspect, an embodiment of the present application further provides a computer program product, wherein the computer program product comprises a computer program stored on a storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the above method.

[0025] The method, system, electronic device and non-transitory computer-readable storage medium for auxiliary planning of fracture reduction path based on augmented reality technology provided by the present application can produce at least the following technical effects:

[0026] (1) The fracture reduction path auxiliary planning method based on augmented reality technology can provide an intuitive three-dimensional view by obtaining the patient's fracture imaging data and reconstructing a three-dimensional model, which can significantly improve the accuracy of reduction and reduce blind adjustments during the reduction process. Through the combination of reinforcement learning module and multi-physics field simulation optimization module, the system can generate a reduction path with excellent simulation performance according to the fracture type and surgical requirements, further improve the accuracy of reduction, ensure the precise alignment of bone fragments, and promote fracture healing and functional recovery.

[0027] (2) Use the reinforcement learning algorithm to perform preliminary path planning for the fracture area, and input the preliminary path into the multi-physics field simulation model for mechanical simulation and optimization. The adaptability and flexibility of the reinforcement learning algorithm can optimize the path in real time, but it may lack mechanical feasibility and safety in actual operation. The multi-physics field simulation model can effectively combine mechanical simulation and actual operating conditions to provide a more accurate and safe path, but it requires high computational costs and complex simulation models. In this technical solution, the fracture reduction path planning model effectively integrates the advantages of both, realizes the adaptability of path planning and the accuracy and safety of mechanical simulation optimization, and can not only provide preliminary path planning, but also further optimize the path through mechanical simulation to ensure the feasibility and safety of actual operation, and has higher practical value.

[0028] (3) In this technical solution, finite element analysis is performed through a multi-physics field simulation optimization module to simulate the stress distribution on the reduction path, ensuring that the path meets the physiological and mechanical requirements and avoiding damage to surrounding tissues. This is especially suitable for situations where the field of vision is limited or the fracture site is complex. It can accurately determine the relative position of the bone fragments and the reduction direction, reducing the uncertainty and potential risks in the reduction process and improving the safety of the operation.

[0029] (4) In this technical solution, the reward function of the reinforcement learning module takes into account the reduction accuracy, path smoothness and physiological constraints, and can be adaptively adjusted according to the actual surgical situation. Through the flexible setting of the accuracy adjustment parameters, smoothness adjustment parameters and physiological constraint adjustment parameters, the system can dynamically adjust the reduction path to adapt to different types of fractures and surgical requirements, with strong adaptability and flexibility to meet the needs of different surgical scenarios.

[0030] (5) Traditional fracture reduction surgery relies on the doctor's experience and tactile feedback, often requiring repeated attempts and adjustments, which is time-consuming. Through augmented reality technology and an automated path planning model, this solution can quickly generate and display the fracture reduction path, reducing the doctor's operation time and trial and error process, greatly improving surgical efficiency, significantly shortening the operation time, reducing the doctor's workload, and reducing the patient's surgical risk.

[0031] Through this technical solution, the reconstructed 3D model and reduction path are superimposed on the actual surgical scene through AR glasses, and the doctor can view the matching of the reduction path and the patient's fracture area in real time, providing intuitive and real-time guidance. The application of augmented reality technology in the fracture reduction scene improves the doctor's operating experience, makes the surgical process more controllable and intuitive, and helps to improve the success rate of the operation and the patient's rehabilitation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart showing an example of a fracture reduction path auxiliary planning method based on augmented reality technology according to an embodiment of the present application is shown;

[0034] Figure 2 An operation flow chart of an example of reconstructing a three-dimensional model of a fractured part based on fractured part image data according to an embodiment of the present application is shown;

[0035] Figure 3 An operation flow chart of an example of generating a three-dimensional surface model based on an enhanced surface reconstruction algorithm according to an embodiment of the present application is shown;

[0036] Figure 4 An operation flow chart of an example of matching and calibrating the reconstructed three-dimensional model of the fractured part with the fractured part of the patient in the actual surgical scene according to an embodiment of the present application is shown;

[0037] Figure 5 A structural block diagram of an example of a fracture reduction path planning model according to an embodiment of the present application is shown;

[0038] Figure 6 A structural block diagram of an example of a fracture reduction path auxiliary planning system based on augmented reality technology according to an embodiment of the present application is shown;

[0039] Figure 7 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which this application belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantitative restrictions, but indicate the existence of at least one. "Include" or "comprises" and other similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and other similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0042] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in this application are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0043] Figure 1 A flowchart of an example of a fracture reduction path auxiliary planning method based on augmented reality technology according to an embodiment of the present application is shown.

[0044] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities, so as to significantly improve the accuracy and efficiency of fracture reduction surgery, reduce the difficulty and risk of surgery, improve postoperative recovery effects, and promote the standardization and repeatability of surgical operations by utilizing augmented reality technology, reinforcement learning, and multi-physics field simulation optimization.

[0045] In some examples, it may be integrated into the configuration server through software, hardware, or a combination of software and hardware, which should not be limited here.

[0046] The following will take the fracture reduction planning platform as an exemplary implementation subject to introduce the details of the technical solution involved in this application. However, it should be understood that one or more steps involved in the following process can be implemented by one or more controllers or software installed and deployed in the client or server.

[0047] like Figure 1 As shown, in step S110, fracture image data for the fracture area of ​​the patient is obtained, and the fracture image data includes CT (computer tomography) data and X-ray data.

[0048] In some embodiments, CT and X-ray data of the patient's fracture area are collected to ensure high resolution and integrity of the image data. In addition, the collected image data can be preprocessed, including denoising, standardization and calibration, to improve data quality and accuracy. Then, the processed image data is input into the system to generate a complete fracture area image data set. Thus, the accuracy and reliability of the basic data for subsequent three-dimensional model reconstruction and path planning are ensured, and the accuracy of the entire fracture reduction path planning is improved.

[0049] More specifically, image data of the fracture area of ​​the patient are obtained from CT and X-ray scanners, and these image data should include multiple slices, covering all levels of the fracture area. In addition, CT and X-ray data are usually stored in DICOM format, which needs to be standardized for subsequent processing.

[0050] In step S120, a three-dimensional model of the fractured part is reconstructed based on the fractured part image data. The three-dimensional model of the fractured part includes a plurality of bone block modules, and each bone block module is respectively marked with a corresponding position, boundary and color.

[0051] In some embodiments, an image segmentation algorithm is used to segment the bone fragments in the fracture image data. Then, based on the segmentation result, a three-dimensional reconstruction algorithm (such as a surface reconstruction algorithm or a volume reconstruction algorithm) is used to generate a three-dimensional model of the fracture site. Then, the corresponding position, boundary and color are marked for each bone block module, which can be more conducive to the personalized display of different bone block modules, and is also convenient for subsequent matching and path planning. Thus, through accurate image segmentation and three-dimensional reconstruction, a detailed three-dimensional model of the fracture site is generated, so that the doctor can clearly understand the structure of the fracture area, and accurate spatial information is provided for path planning.

[0052] Figure 2 FIG. 2 shows an operation flow chart of an example of reconstructing a three-dimensional model of a fractured part based on fractured part image data according to an embodiment of the present application. Figure 2As shown, in step S210, data preprocessing and fusion are performed for the CT data and X-ray data in the fracture image data to obtain a superimposed image of the fracture portion corresponding to the same coordinate system. More specifically, a filter (such as Gaussian filtering) can be used to denoise the CT and X-ray image data, remove noise and artifacts in the image, and enhance the clarity of the bone structure. In addition, image enhancement techniques (such as histogram equalization) are used to improve the contrast of the image data and highlight the difference between bones and soft tissues. Afterwards, the slices of the CT and X-ray image data are matched to ensure that the images of different levels are accurately superimposed in the same coordinate system. In step S220, the bone block module and other tissues in the superimposed image of the fracture portion are identified, and the edge feature points of the bone block module are extracted. In one example, the image data can be segmented using a pre-trained deep learning model to automatically identify and extract the bone structure. In another example, according to the gray value difference between the bone and soft tissue, an appropriate threshold is set for segmentation to extract the bone area, thereby achieving the segmentation of the bone structure in the image data from other tissues. Then, use edge detection algorithm (for example, Canny edge detection) to extract the edge of bone block, generate the outline of bone block, realize extracting the outline and the border of bone block from the segmentation result.In step S230, process each edge feature point based on surface reconstruction algorithm, to generate three-dimensional surface model.Here, surface reconstruction algorithm can adopt general algorithm or other enhanced innovative algorithms, for example, use marching cubes (Marching Cubes) algorithm to connect the bone edge points in the slice, generate three-dimensional surface model.In step S240, record the spatial position, border and shape of each bone block module in the three-dimensional surface model, and mark corresponding color respectively for each bone block module, to obtain fracture part three-dimensional model.Like this, based on segmentation result and edge detection, bone block is divided into multiple modules, and each module is marked, realize the modular processing of the bone block in the three-dimensional model.

[0053] Thus, through data acquisition and preprocessing, image segmentation, bone structure extraction, three-dimensional reconstruction and bone block modularization, the three-dimensional model of the fracture part can be reconstructed based on the image data of the fracture part. Through data acquisition and preprocessing technology, CT and X-ray image data can be quickly processed, noise can be removed, and image quality can be enhanced. In this way, after the doctor obtains the image data, the system can quickly generate processed images, saving data processing time and speeding up the surgical preparation process. Through image segmentation and bone structure extraction technology, bones and other tissues can be accurately separated to ensure that the bone blocks in the three-dimensional model are consistent with the actual fracture area. In this way, in fracture reduction surgery, doctors can formulate more accurate surgical plans based on accurate three-dimensional models, reduce surgical errors, and improve the accuracy of reduction.

[0054] In some examples of the embodiments of the present application, the surface reconstruction algorithm is enhanced and optimized in the process of practicing the present application, and an adaptive Marching Cubes algorithm based on deep learning is proposed. By introducing a deep learning model to predict the geometric characteristics and isosurface positions in the local grid, the grid division is adaptively adjusted to further improve the reconstruction accuracy and efficiency.

[0055] Figure 3 An operational flow chart of an example of generating a three-dimensional surface model based on an enhanced surface reconstruction algorithm according to an embodiment of the present application is shown.

[0056] like Figure 3 As shown, in step S310, multi-scale features are extracted from the superimposed image of the fracture. In step S320, the multi-scale features are input into the deep learning model to predict the local mesh geometric characteristics, which include mesh vertex scalar values ​​and mesh isosurface positions. In step S330, the density and shape of the mesh division are adaptively adjusted according to the prediction results of the deep learning model to ensure higher resolution in complex structural areas:

[0057]

[0058] Where V adaptive represents the vertex index value in the adaptive mesh; the local geometric properties predicted by the deep learning model are P = {p1, p2, ..., p n}, where p i represents the geometric characteristics of the local area i; f(p i ) is based on the local geometric characteristics p i Predicted vertex scalar value function. In step S340, in the adaptive mesh, the enhanced marching cubes algorithm is applied to extract isosurface vertices according to the geometric characteristics predicted by the deep learning model.

[0059] Specifically, the intersection points of the isosurfaces on the edges of the cube are determined by interpolation methods as the vertices of the isosurfaces:

[0060]

[0061] Where V interpolated is the isosurface vertex obtained by interpolation, V0 is the scalar value of the cube vertex, V target In step S350, these isosurface vertices are connected into triangles to generate a three-dimensional surface model.

[0062] Through this embodiment, in the adaptive grid, the improved Marching Cubes algorithm is applied to extract isosurfaces according to the geometric characteristics predicted by the deep learning model. Adaptive meshing can provide higher resolution in complex structural areas, ensuring that the details of the three-dimensional model are clearer and more accurate. The local geometric characteristics are predicted by the deep learning model, and the mesh density is adaptively adjusted to improve the reconstruction accuracy, reduce unnecessary meshing and calculations, and improve the overall efficiency of surface reconstruction. Therefore, combining the advantages of deep learning and traditional algorithms, the adaptive Marching Cubes algorithm can better adapt to different types of fracture structures and generate more stable and reliable three-dimensional models.

[0063] In step S130, the reconstructed three-dimensional model of the fractured part is matched and calibrated with the fractured area of ​​the patient in the actual surgical scene.

[0064] In some embodiments, a registration algorithm (e.g., a point cloud registration algorithm) can be used to match the reconstructed 3D model with the patient's fracture area in the actual surgical scene, and fine adjustments can be made by comparing key points and feature areas to ensure that the model is highly consistent with the actual fracture area. In addition, the matching calibration process can be displayed in real time in the AR glasses, allowing the doctor to make fine adjustments and confirmations.

[0065] Therefore, through precise matching and calibration, the high consistency between the three-dimensional model and the actual surgical scene is ensured, allowing doctors to accurately refer to the model during the operation, thereby improving the accuracy and safety of reduction.

[0066] It should be noted that various non-restrictive matching and calibration methods can be used, such as point cloud depth image matching technology, to achieve matching and calibration between the three-dimensional model of the fracture and the actual surgical scene.

[0067] Figure 4 FIG. 1 is a flowchart showing an example of matching and calibrating the reconstructed three-dimensional model of the fractured part with the fractured part of the patient in the actual surgical scene according to an embodiment of the present application. Figure 4As shown, in step S410, at least one key feature area in the reconstructed three-dimensional model of the fracture is identified, and a corresponding first identifier is assigned to each key feature area. More specifically, a detailed feature analysis is performed on the reconstructed three-dimensional model of the fracture, for example, geometric features such as feature points, feature lines and feature surfaces in the three-dimensional model of the fracture are extracted using image processing and computer vision technology to identify the key feature area. In combination with the business scenario, the first identifier can be a fracture line, for example, the starting point and end point of the fracture line are identified in the three-dimensional model as the first identifier. In addition, the first identifier can also be a key feature area such as an articular surface and a condyle. In step S420, based on the sensor group in the AR glasses, environmental sensing data in the surgical scene is collected, and the sensor group includes an optical sensor, a depth sensor and an IMU sensor. The real-time image of the surgical scene is captured by the optical sensor, the depth information of the surgical scene is obtained by the depth sensor, and the device posture data is captured by the IMU sensor. Further, after obtaining multiple types of sensor sampling data, the data of the optical sensor, the depth sensor and the IMU sensor need to be synchronized in time and matched in space to generate a comprehensive data stream of the surgical scene. In step S430, when at least one second identifier is detected in the environmental sensing data, the three-dimensional model of the fracture part is matched and calibrated according to the calibrated position and calibrated size corresponding to each second identifier, and with reference to each first identifier. The second identifier is pre-set at a specific part of the patient's fracture area in the surgical scene, and the specific part is associated with a key feature area. More specifically, the doctor can pre-place a second identifier, such as a reflective ball, a QR code, or a surgical nail, etc., at the key feature area (e.g., a fracture line, an articular surface, and a bone protrusion).

[0068] In this embodiment, by performing feature analysis on the reconstructed three-dimensional model of the fracture, a stable and easily identifiable feature area (such as fracture line, articular surface, bone protrusion, etc.) is selected as the first identifier to ensure the accuracy of the identifier. The data of the optical sensor, depth sensor and IMU sensor are used to provide comprehensive information of the surgical scene and achieve high-precision environmental perception. The second identifier in the surgical scene is quickly detected by the image processing algorithm, and the correlation between the first identifier and the second identifier is used to automate the matching calibration process and reduce manual operation time. Through this embodiment, the real-time data of the IMU sensor and the depth sensor are used to dynamically adjust the matching of the three-dimensional model with the surgical scene to ensure the continued effectiveness of the three-dimensional model matching. In combination with the business scenario, during the operation, when the posture of the AR glasses changes or the patient moves slightly, the doctor can respond to changes in the operation process in a timely manner by dynamically adjusting and calibrating the three-dimensional model in real time to ensure the safety and stability of the operation.

[0069] In step S140, the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements are input into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path.

[0070] In some embodiments, the matched and calibrated three-dimensional model of the fracture part is input into the fracture reduction path planning model, so that the platform can fully capture the spatial orientation information between the different bone block modules in the fracture area. Here, the fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module. The state, action space and reward function of the reinforcement learning module are defined by the input fracture type and surgical requirements, the reinforcement learning module is used to generate the initial reduction path, and the multi-physics field simulation optimization module is used to simulate and optimize the stress distribution of the initial path. Thus, the reinforcement learning and multi-physics field simulation optimization modules are used to generate an optimized fracture reduction path, ensure the accuracy and smoothness of the path, meet the physiological and mechanical requirements, reduce damage to surrounding tissues, and improve the reduction effect.

[0071] In step S150, each optimized fracture reduction path is rendered, and a target fracture reduction path is determined according to the detected interaction instruction.

[0072] In some embodiments, each resetting path is rendered in AR glasses, providing an intuitive three-dimensional view and path information. Then, through gesture recognition or voice commands, the doctor is allowed to select the path and interact. Then, according to the doctor's interactive instructions, the final target resetting path is determined, and real-time guidance is provided during the operation. Thus, through augmented reality technology, intuitive resetting path visualization is provided, allowing doctors to quickly select the optimal path, reduce operation time and difficulty, and improve the success rate and safety of the operation.

[0073] The fracture reduction path planning method provided in the embodiment of the present application can provide a unified operation reference for different doctors, reducing the dependence of surgical results on doctor experience. In addition, through standardized reduction path planning, the repeatability and standardization of surgical operations can be improved, which is helpful to promote minimally invasive and precise fracture reduction surgery and improve the overall medical level.

[0074] Figure 5 A structural block diagram of an example of a fracture reduction path planning model according to an embodiment of the present application is shown.

[0075] like Figure 5 As shown, the fracture reduction path planning model 500 includes a cascaded reinforcement learning module 510 and a multi-physics field simulation optimization module 520.

[0076] The reinforcement learning module 510 is used to determine at least one initial reduction path according to the fracture type, the matched and calibrated three-dimensional model of the fracture part and the fracture surgery requirements. More specifically, the state of the reinforcement learning module 510 is defined according to the geometric features of the matched and calibrated three-dimensional model of the fracture part, the relative orientation information of the bone fragments and the fracture type, and the action space of the reinforcement learning module is defined according to the movement of the bone fragments, the rotation of the bone fragments and the force applied by the bone fragments. The reward function of the reinforcement learning module 510 is defined according to the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements and physiological constraint requirements.

[0077] More specifically, a RL environment based on a three-dimensional model is established, and a large number of simulation trainings are carried out through a reinforcement learning algorithm to generate multiple preliminary reset paths that meet the reward constraints.

[0078] Each initial reduction path is input into the multi-physics field simulation optimization module 520 to perform stress distribution simulation on each initial reduction path through finite element analysis, and the corresponding initial path is optimized and adjusted according to the simulation results to determine at least one corresponding optimized fracture reduction path.

[0079] More specifically, finite element analysis is used to simulate the stress distribution of the preliminary path to evaluate the mechanical feasibility of the path. In addition, dynamic response analysis is performed on the path to ensure the stability of the path in actual operation. Then, the preliminary path is optimized and adjusted according to the simulation results to generate one or more optimized fracture reduction paths that meet the conditions.

[0080] Here, the reinforcement learning (RL) algorithm is used to generate an optimized fracture reduction path, which can optimize the reduction path through continuous simulation and feedback learning to achieve the best effect. In addition, multi-physics field simulation technology is used to perform mechanical simulation and adjustment optimization on the generated reduction path.

[0081] In this embodiment, a reinforcement learning algorithm is used to perform preliminary path planning on the fracture area, and the preliminary path is input into a multi-physics field simulation model for mechanical simulation and optimization. The adaptability and flexibility of the reinforcement learning algorithm can optimize the path in real time, but it may lack mechanical feasibility and safety in actual operation. The multi-physics field simulation model can effectively combine mechanical simulation and actual operating conditions to provide a more accurate and safe path, but it requires high computational costs and complex simulation models. Through this embodiment, the advantages of both are effectively integrated in the fracture reduction path planning model, achieving the adaptability of path planning and the accuracy and safety of mechanical simulation optimization, which can not only provide preliminary path planning, but also further optimize the path through mechanical simulation, ensuring the feasibility and safety of actual operation, and having higher practical value.

[0082] The reward function R of the reinforcement learning module 510 is expressed as follows:

[0083] R=r accuracy +r smoothness +r physiology , Formula (3)

[0084] r accuracy = -α·||M target -M current ||, Formula (4)

[0085]

[0086] In the formula, r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness Represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissue; M target is the target position coordinate, indicating the position that the bone block should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the accuracy adjustment parameter, which is used to adjust the impact of reset accuracy on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0087] The design explanation and optimization description of the above reward function are as follows:

[0088] The position error of the bone after reduction is reflected by the reduction accuracy reward. The smaller the position error, the higher the reward. In this way, the reinforcement learning module can effectively optimize the fracture reduction path, making the position of the bone after reduction closer to the target position, significantly improving the success rate of the operation and reducing the risk of poor bone alignment after surgery.

[0089] The smoothness of the path is reflected by the path smoothness reward. The smoother the path, the higher the reward. In this way, the reinforcement learning module can generate a smoother resetting path, avoiding overly complex and tortuous paths, making the resetting operation smoother, improving the smoothness of the operation, reducing unnecessary adjustments during the operation, and avoiding operational errors caused by complex paths.

[0090] Physiological constraint rewards ensure that the path meets physiological and mechanical requirements and avoids damaging surrounding tissues during the operation of bone movement. In this way, the reinforcement learning module can ensure that the generated reduction path is mechanically safe and feasible, ensure that no additional damage is caused to surrounding tissues during the operation, ensure the physiological safety of the operation, and reduce the occurrence of postoperative complications.

[0091] Therefore, by adopting the above-mentioned reward function, the reinforcement learning module can achieve higher reduction accuracy, smoother surgical path and higher physiological safety in fracture reduction path planning, which not only improves the success rate and safety of the operation, but also significantly reduces postoperative complications and promotes the rapid recovery of patients, providing strong support for the development of smart medical technology.

[0092] In some examples of the embodiments of the present application, the multi-physics field simulation optimization module is used to perform the following operations:

[0093] A finite element model is established based on the geometric model, fracture type and material properties of the matched and calibrated three-dimensional model of the fracture part.

[0094] Here, the material properties include the material properties of the bone block and other surrounding tissues, which can be input through the system or inferred based on the patient's basic information in the fracture surgery requirements, such as gender and age, and should not be limited here.

[0095] More specifically, the geometric model of the bone block is established using 3D reconstruction data, and the material properties are defined according to the physical characteristics of the bone and soft tissue. Mechanical boundary conditions are then applied to each path node on the initial path, and the contact relationship and constraints between the bone blocks are defined to simulate the external forces and moments applied during surgery.

[0096] Mechanical boundary conditions are applied to each node on the initial reset path to calculate the stress distribution on the path through finite element analysis:

[0097] σ=D(C)·ò, Formula (7)

[0098]

[0099] In the formula, σ is the stress vector, which describes the stress state inside the bone; D(C) is the elastic matrix of the material, which depends on the material property matrix C. The elastic matrix represents the elastic properties of the material in all directions; C is the material property matrix, which contains the anisotropic properties of the bone material; ò is the strain vector, which describes the deformation state of the bone under the action of external force; is a symmetric gradient operator, which is used to calculate the symmetric gradient of the displacement vector u; u is a displacement vector, which represents the displacement of the material in all directions; u x ,u y ,u zRepresent the displacement components in the x, y, and z directions respectively.

[0100] Here, the linear elastic mechanics relationship is used to describe the relationship between stress and strain, while taking into account the anisotropic material properties. In addition, in order to improve the accuracy of the model, D is expanded to a matrix containing nonlinear terms, considering the elastic modulus of the bone in different directions:

[0101]

[0102] In the formula, C ij are the terms in the material property matrix, depending on the anisotropic properties of the bone material.

[0103] In complex fracture reduction surgery, doctors need to understand the stress distribution of the bone under different reduction paths. Through this embodiment, the anisotropic material properties and nonlinear contact stress of the bone are taken into account during the stress calculation process, and the stress distribution of the bone and surrounding tissues can be calculated more accurately, avoiding the selection of a path that will cause stress concentration at the fracture site or surrounding tissues and may cause secondary damage.

[0104] Dynamic response analysis of the initial reset path:

[0105]

[0106] Where M is the mass matrix, which describes the mass distribution of the bone and surrounding tissues; is the acceleration vector, which indicates the acceleration of the bone in all directions; λM indicates the mass damping term, λ is the mass damping coefficient; μK indicates the stiffness damping term, μ is the stiffness damping coefficient; K(u) is the nonlinear stiffness matrix, which depends on the displacement u; is the velocity vector, indicating the velocity of the bone in all directions; N contact (u) is the contact force term, describing the contact force between bone fragments; is the friction term, describing the friction between the bones; F(t) is the time-dependent external force vector, representing the external force applied during the operation.

[0107] In some embodiments, the mass matrix is ​​calculated according to the material properties and geometric shape of the bone block, the damping matrix is ​​calculated according to the damping characteristics of the bone block, and the stiffness matrix is ​​calculated according to the stiffness characteristics and geometric shape of the bone block, and the dynamic equation is used for simulation to calculate the dynamic response of the bone block during the reduction process. Thus, the dynamic response analysis of the path is performed to ensure the stability and robustness of the path in actual operation.

[0108] During surgery, the bone fragment may be subjected to various dynamic loads (such as the force of surgical tools, slight movements of the patient, etc.). Here, the advanced dynamic equation provided by equation (10) combines nonlinear force terms and damping effects to more accurately simulate the dynamic loads and mechanical responses of the bone fragment during surgery, allowing doctors to foresee and avoid possible mechanical instability and ensure a smooth surgical procedure.

[0109] The initial reset path is optimized and adjusted according to the dynamic response analysis result to ensure the mechanical safety of the reset path in actual operation.

[0110] In some embodiments, a variety of non-restrictive target optimization algorithms, such as genetic algorithms and particle swarm optimization algorithms, can also be used to solve the problem. According to the stress distribution results, the positions and directions of the path nodes are adjusted, and the stress distribution of the adjusted path is recalculated to verify the safety of the path, and the optimization steps are repeated until the path meets the requirements of the optimization objective function.

[0111] In this embodiment, based on the simulation results of stress distribution and dynamic response, the initial reset path can be optimized for multiple iterations to ensure that the final path is optimal in terms of accuracy, smoothness and mechanical safety. In combination with the business scenario, before surgery, through multiple simulations and path optimization, a reset path with excellent simulation performance can be determined. This path can not only accurately reset the bone block, but also reduce damage to surrounding tissues, improve the success rate of surgery and the patient's recovery speed. In addition, by considering the nonlinear characteristics of the material and the nonlinear contact force between the bone blocks, the system can better adapt to different types of fractures and complex surgical scenarios. Therefore, when dealing with a variety of complex fracture types (such as multiple fractures, comminuted fractures, etc.), the system can provide targeted stress analysis and path optimization to ensure the best operation path for each surgical scenario. In addition, during the operation, the bone block may produce tiny vibrations due to tool operation or patient movement, and by accurately simulating the damping effect, the system can better predict and offset the vibration and mechanical instability that may occur during the operation. Based on the accurate damping simulation results, the path can be adjusted in time to reduce the impact of vibration on the operation and improve the stability of the operation.

[0112] In some examples of the embodiments of the present application, in order to better combine the results of the dynamic response analysis, a more comprehensive optimization objective function formula may be adopted.

[0113] More specifically, the solution formula for optimizing the objective function value minF(p,q) is:

[0114] min F(p,q)=w1·E accuracy +w2·E smoothness +w3·E stress +w4·Edynamic , Formula (11)

[0115]

[0116] In the formula, E accuracy Represents the reset accuracy error term, E smoothness represents the path smoothness error term, E stress represents the stress safety error term, E dynamic represents the dynamic response error term; w1, w2, w3, w4 represent weight parameters, which are used to measure the importance of the corresponding error term in the overall optimization; p target,i represents the target position coordinates of the i-th path node, p current,i represents the current path position coordinates of the i-th path node; p current,i+1 and p current,i-1 represents the position coordinates of the adjacent path nodes for the i-th path node; represents the acceleration of the ith path node, represents the speed of the ith path node, u i represents the displacement of the i-th point on the path, N contact (u i ) represents the contact force at the i-th path node, represents the friction force at the i-th path node.

[0119] The specific implementation details are described in context as follows: First, the initial reset path is generated using the reinforcement learning module. Then, the dynamic response analysis is performed using advanced dynamic equations to calculate the displacement, velocity, acceleration, contact force, and friction of each path point. Furthermore, based on the results of the dynamic response analysis, the optimization objective function min F(p,q) is calculated. Finally, the position and direction of the path node are adjusted according to the objective function value.

[0120] By optimizing the stress safety error term E in the objective function stress and the dynamic response error term E dynamic , which can effectively avoid stress concentration and dynamic instability caused by improper path selection, and ensure the mechanical safety of the bone and surrounding tissues during the reduction process. Therefore, in complex fracture reduction surgery, accurate stress analysis and dynamic response evaluation can be used to select the most mechanically safe reduction path to avoid secondary injuries during and after the operation.

[0121] By optimizing the reset accuracy error term E in the objective function accuracy and the path smoothness error term E smoothness, can generate a smoother and more accurate reduction path, improve the smoothness and success rate of surgical operations. Therefore, before surgery, through multiple simulations and path optimization, one or more reduction paths with excellent simulation performance can be determined, which can not only accurately reduce the bone fragments, but also reduce damage to surrounding tissues, improve the success rate of surgery and the patient's recovery speed.

[0122] By optimizing the dynamic response error term E in the objective function dynamic , which can ensure the stability of the path mechanical response during the operation and avoid instability during the operation. At the same time, through real-time monitoring and dynamic adjustment, the reduction path can be adjusted in time according to unexpected situations during the operation to ensure the smooth progress of the operation. Therefore, during the operation, the platform system can monitor the dynamic response of the bone block in real time and provide real-time adjustment suggestions to help doctors adjust the operation path in time to ensure the stability and safety of the operation.

[0123] Therefore, by optimizing the error terms in the objective function, the system can adapt to different types of fractures and complex surgical scenarios, provide accurate path optimization and mechanical analysis, and improve the success rate of surgery and the patient's recovery effect. In this way, when dealing with a variety of complex fracture types (such as multiple fractures, comminuted fractures, etc.), the platform system can provide targeted stress analysis and path optimization to ensure that the operation path output by the platform system in each surgical scenario can have better performance.

[0124] It should be noted that in the fracture reduction path planning model, the reduction path is optimized and constrained by the reinforcement learning module and the simulation analysis module respectively, which can comprehensively utilize the advantages of the two technical means. For example, both contain constraints on reduction accuracy and path smoothness. In this way, by constraining the same goal through the reward function of the reinforcement learning module and the objective function of the simulation analysis module, the advantages of both can be fully utilized to improve the overall path planning effect.

[0125] Through the dual constraints of the reinforcement learning module and the simulation analysis module, the robustness of path planning can be improved, and the reliability of the path in different environments and conditions can be ensured. The reinforcement learning module and the simulation analysis module optimize different goals of the path (such as accuracy, smoothness, and mechanical safety) respectively, and the comprehensive use of the advantages of both can achieve the global optimization of multiple goals. In addition, the path generated by the reinforcement learning module is calibrated and optimized through the simulation analysis module to ensure the accuracy and safety of the path in actual operation and avoid the infeasibility of the path generated by the algorithm in physical implementation. Therefore, through the dual constraints of the reinforcement learning module and the simulation analysis module, the robustness of path planning can be improved, and the reliability of the path in different environments and conditions can be ensured.

[0126] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0127] Figure 6 A structural block diagram of an example of a fracture reduction path auxiliary planning system based on augmented reality technology according to an embodiment of the present application is shown.

[0128] like Figure 6 As shown, the fracture reduction path auxiliary planning system 600 based on augmented reality technology includes a data acquisition unit 610, a model reconstruction unit 620, a model calibration unit 630, a path planning unit 640 and a path confirmation unit 650.

[0129] The data acquisition unit 610 is used to acquire fracture image data for the fracture area of ​​the patient, and the fracture image data includes CT data and X-ray data.

[0130] The model reconstruction unit 620 is used to reconstruct a three-dimensional model of the fracture part based on the fracture part image data; the three-dimensional model of the fracture part includes a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color.

[0131] The model calibration unit 630 is used to match and calibrate the reconstructed three-dimensional model of the fractured part with the fractured area of ​​the patient in the actual surgical scene.

[0132] The path planning unit 640 is used to input the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path.

[0133] The path confirmation unit 650 is used to render various optimized fracture reduction paths and determine the target fracture reduction path according to the detected interaction instructions.

[0134] The fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0135] The reinforcement learning module is used to determine at least one initial reduction path according to the fracture type, the matched and calibrated three-dimensional model of the fracture part and the fracture surgery requirements; the state of the reinforcement learning module is defined according to the geometric features of the matched and calibrated three-dimensional model of the fracture part, the relative orientation information of the bone blocks and the fracture type; the action space of the reinforcement learning module is defined according to the movement, rotation and force of the bone blocks; the reward function of the reinforcement learning module is defined according to the fracture surgery requirements, and the fracture surgery requirements include reduction accuracy requirements, path smoothness requirements and physiological constraint requirements;

[0136] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding fracture reduction path;

[0137] The reward function R of the reinforcement learning module is expressed as follows:

[0138] R=r accuracy +r smoothness +r physiology

[0139] r accuracy = -α·||M target -M current ||

[0140]

[0141] In the formula, r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness Represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissue; M target is the target position coordinate, indicating the position that the bone block should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the accuracy adjustment parameter, which is used to adjust the impact of reset accuracy on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0142] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the above-mentioned fracture reduction path auxiliary planning method based on augmented reality technology in the present application.

[0143] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned fracture reduction path auxiliary planning method based on augmented reality technology.

[0144] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a fracture reduction path auxiliary planning method based on augmented reality technology.

[0145] Figure 7 is a schematic diagram of the hardware structure of an electronic device for executing a fracture reduction path auxiliary planning method based on augmented reality technology provided by another embodiment of the present application, such as Figure 7 As shown, the device includes:

[0146] One or more processors 710 and memory 720, Figure 7 A processor 710 is taken as an example.

[0147] The device for executing the fracture reduction path auxiliary planning method based on augmented reality technology may further include: an input device 730 and an output device 740 .

[0148] The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0149] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the fracture reduction path auxiliary planning method based on augmented reality technology in the embodiment of the present application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the fracture reduction path auxiliary planning method based on augmented reality technology in the above method embodiment is implemented.

[0150] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The input device 730 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0152] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, the fracture reduction path auxiliary planning method based on augmented reality technology in any of the above method embodiments is executed.

[0153] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0154] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0155] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0156] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0157] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0158] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0159] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fracture reduction path auxiliary planning method based on augmented reality technology, applied to AR glasses, characterized in that: The method comprises: Acquire fracture image data for the fracture area of ​​the patient, wherein the fracture image data includes CT data and X-ray data; Reconstructing a three-dimensional model of the fracture part based on the fracture part image data; the three-dimensional model of the fracture part includes a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; Matching and calibrating the reconstructed three-dimensional model of the fractured part with the fractured area of ​​the patient in the actual surgical scene; Inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; Render each of the optimized fracture reduction paths, and determine a target fracture reduction path according to the detected interaction instructions.

2. The method according to claim 1, characterized in that The fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module; The reinforcement learning module is used to determine at least one initial reduction path according to the fracture type, the matched and calibrated three-dimensional model of the fracture part and the fracture surgery requirements; The state of the reinforcement learning module is defined according to the geometric features of the matched and calibrated three-dimensional model of the fracture, the spatial relative position information of the bone blocks and the fracture type; the action space of the reinforcement learning module is defined according to the movement, rotation and force of the bone blocks; the reward function of the reinforcement learning module is defined according to the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements and physiological constraint requirements; Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path; The reward function R of the reinforcement learning module is expressed as follows: R＝r accuracy +r smoothness +r physiology r accuracy =-α·||M target -M current || In the formula, r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness Represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the biomechanical constraint reward, which is used to ensure that the path meets the biomechanical requirements to avoid damaging surrounding tissues; M target is the target position coordinate, indicating the position that the bone block should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the accuracy adjustment parameter, which is used to adjust the impact of reset accuracy on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

3. The method according to claim 2, characterized in that The multi-physics simulation optimization module is used to perform the following operations: Establishing a finite element model based on the geometric model, fracture type and material properties of the matched and calibrated three-dimensional model of the fracture part; Mechanical boundary conditions are applied to each node on the initial reset path to calculate the stress distribution on the path through finite element analysis: σ=D(C)·ò In the formula, σ is the stress vector, which describes the stress state inside the bone; D(C) is the elastic matrix of the material, which depends on the material property matrix C. The elastic matrix represents the elastic properties of the material in all directions; C is the material property matrix, which contains the anisotropic properties of the bone material; ò is the strain vector, which describes the deformation state of the bone under the action of external force; is a symmetric gradient operator, which is used to calculate the symmetric gradient of the displacement vector u; u is a displacement vector, which represents the displacement of the material in all directions; u x ,u y ,u z Represent the displacement components in the x, y, and z directions respectively; Dynamic response analysis of the initial reset path: Where M is the mass matrix, which describes the mass distribution of the bone and surrounding tissues; is the acceleration vector, which indicates the acceleration of the bone in all directions; λM indicates the mass damping term, λ is the mass damping coefficient; μK indicates the stiffness damping term, μ is the stiffness damping coefficient; K(u) is the nonlinear stiffness matrix, which depends on the displacement u; is the velocity vector, indicating the velocity of the bone in all directions; N contact (u) is the contact force term, describing the contact force between bone fragments; is the friction term, describing the friction between the bone fragments; F(t) is the time-dependent external force vector, representing the external force applied during the surgery; The initial reset path is optimized and adjusted according to the dynamic response analysis result to ensure the mechanical safety of the reset path in actual operation.

4. The method according to claim 3, wherein: The optimizing and adjusting the initial reset path according to the dynamic response analysis result includes: According to the results of dynamic response analysis, calculate the optimization objective function value F(p,q); According to the optimization objective function value F(p,q), the position and direction of the path node are adjusted: In the formula, p current Indicates the position coordinates of the current path node, q current Represents the direction vector of the current path node, Represents the adjusted path position coordinates, represents the adjusted path direction vector; η represents the learning rate, which is used to control the step size of each adjustment; Represents the gradient of the objective function with respect to the path position; Represents the gradient of the objective function with respect to the path direction; The dynamic response analysis and path adjustment steps are repeated until the objective function converges to the minimum value.

5. The method according to claim 4, wherein: The solution formula for optimizing the objective function value minF(p,q) is: minF(p,q)=w1·E accuracy +w2·E smoothness +w3·E stress +w4·E dynamic In the formula, E accuracy Represents the reset accuracy error term, E smoothness represents the path smoothness error term, E stress represents the stress safety error term, E dynamic represents the dynamic response error term; w1, w2, w3, w4 represent weight parameters, which are used to measure the importance of the corresponding error term in the overall optimization; p target,i represents the target position coordinates of the i-th path node, p current,i represents the current path position coordinates of the i-th path node; p current,i+1 and p current,i-1 represents the position coordinates of the adjacent path nodes for the i-th path node; represents the acceleration of the ith path node, represents the speed of the ith path node, u i represents the displacement of the i-th point on the path, N contact (u i ) represents the contact force at the i-th path node, represents the friction force at the i-th path node.

6. The method according to claim 1, wherein: The reconstructing the three-dimensional model of the fracture part based on the fracture part image data comprises: Preprocessing and fusing the CT data and X-ray data in the fracture image data to obtain a superimposed image of the fracture part corresponding to the same coordinate system; Identifying bone modules and other tissues in the superimposed image of the fracture part, and extracting edge feature points of the bone modules; Processing each of the edge feature points based on a surface reconstruction algorithm to generate a three-dimensional surface model; The spatial position, boundary and shape of each bone block module in the three-dimensional surface model are recorded, and each bone block module is marked with a corresponding color to obtain a three-dimensional model of the fracture part.

7. The method according to claim 6, wherein: The processing of each of the edge feature points based on a surface reconstruction algorithm to generate a three-dimensional surface model includes: Extract multi-scale features from superimposed images of fractured parts; Inputting the multi-scale features into a deep learning model to predict local mesh geometric properties; the local mesh geometric properties include mesh vertex scalar values ​​and mesh isosurface locations; Based on the predictions of the deep learning model, the density and shape of the mesh are adaptively adjusted to ensure higher resolution in complex structure areas: Where V adaptive represents the vertex index value in the adaptive mesh; the local geometric properties predicted by the deep learning model are P = {p1, p2, ..., p n }, where p i represents the geometric characteristics of the local area i; f(p i ) is based on the local geometric characteristics p i predicted vertex scalar-valued function; In the adaptive mesh, the enhanced marching cubes algorithm is applied to extract isosurface vertices based on the geometric characteristics predicted by the deep learning model, including: The intersection points of the isosurfaces on the edges of the cube are determined by interpolation methods as the vertices of the isosurfaces: Where V interpolated is the isosurface vertex obtained by interpolation, V0 is the scalar value of the cube vertex, V target is the target equivalent face value; These isosurface vertices are connected into triangles to generate a three-dimensional surface model.

8. The method according to claim 6 or 7, wherein: The step of matching and calibrating the reconstructed three-dimensional model of the fractured part with the fractured part of the patient in the actual surgical scene comprises: Identifying at least one key feature region in the reconstructed three-dimensional model of the fracture, and assigning a corresponding first identifier to each key feature region; Based on the sensor group in the AR glasses, environmental sensor data in the surgical scene is collected; the sensor group includes an optical sensor, a depth sensor, and an IMU sensor; When at least one second marker is detected in the environmental sensor data, the three-dimensional model of the fracture is matched and calibrated according to the calibration positions and calibration sizes corresponding to each second marker and with reference to each first marker; the second marker is pre-set at a specific location in the patient's fracture area in the surgical scene, and the specific location is associated with the key feature area.

9. A fracture reduction path auxiliary planning system based on augmented reality technology, used to implement the method according to any one of claims 1 to 8, characterized in that: The system comprises: A data acquisition unit, used to acquire fracture image data for a fracture area of ​​a patient, wherein the fracture image data includes CT data and X-ray data; A model reconstruction unit, used for reconstructing a three-dimensional model of the fracture part based on the fracture part image data; the three-dimensional model of the fracture part comprises a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; A model calibration unit, used to match and calibrate the reconstructed three-dimensional model of the fractured part with the fractured area of ​​the patient in the actual surgical scene; A path planning unit, used for inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; The path confirmation unit is used to render each of the optimized fracture reduction paths and determine a target fracture reduction path according to the detected interaction instructions.

Images