A fracture reduction path assisted planning method and system based on augmented reality technology

Abstract

The present application provides a fracture reduction path auxiliary planning method and system based on augmented reality technology, which relates to the field of smart medical system technology. The method includes: obtaining fracture imaging data for the patient's fracture area, the fracture imaging data including CT data and X-ray data; reconstructing a three-dimensional model of the fracture part based on the fracture imaging data; matching and calibrating the three-dimensional model of the fracture part with the patient's fracture area in the actual scene; inputting the calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path, the fracture reduction path planning model including a cascaded reinforcement learning module and a multi-physics field simulation optimization module; rendering each fracture reduction path, and determining the target fracture reduction path based on the detected interactive instructions. Therefore, the application of augmented reality technology in the fracture reduction scene helps to improve the success rate of the operation and the postoperative rehabilitation effect of the patient.

Description

Technical Field

[0001] The present application relates to the technical field of smart medical systems, and in particular to a fracture reduction path auxiliary planning method and system based on augmented reality technology. Background Art

[0002] With the development of medical imaging technology, fracture reduction surgery has gradually developed towards minimally invasive and precise directions. Traditional fracture reduction surgery mainly relies on the doctor's experience and visual feedback. Especially for complex fractures, doctors need to complete the reduction through repeated attempts and adjustments. This not only prolongs the operation time and increases the surgical risk, but also makes it difficult to ensure the accuracy of fracture reduction, which can easily lead to poor alignment of the fracture site, thereby affecting fracture healing and functional recovery.

[0003] In addition, in the current imaging-assisted fracture reduction technology, doctors mainly operate through two-dimensional images (such as X-rays) or direct vision with the naked eye. Doctors cannot intuitively feel the three-dimensional spatial structure of the fracture site. Especially when the field of view is limited or the fracture type is complex, it is difficult to accurately judge the relative position of the bone fragments and the optimal reduction direction, which also relatively increases the difficulty and risk of the operation.

[0004] To address the above issues, the industry has not yet proposed a better technical solution. Summary of the Invention

[0005] The embodiments of the present application provide a fracture reduction path auxiliary planning method and system based on augmented reality technology, which is used to at least solve the problem that the current related technology relying on two-dimensional fracture reduction image auxiliary technology cannot provide a better fracture reduction reference.

[0006] In a first aspect, an embodiment of the present application provides a fracture reduction path auxiliary planning method based on augmented reality technology, which is applied to AR glasses. The method includes: obtaining fracture imaging data for a patient's fracture area, the fracture imaging data including CT data and X-ray data; reconstructing a three-dimensional model of the fracture part based on the fracture imaging data; the three-dimensional model of the fracture part including multiple bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; matching and calibrating the reconstructed three-dimensional model of the fracture part with the patient's fracture area in an actual surgical scene; inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgical requirements into a fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; rendering each of the optimized fracture reduction paths, and determining a target fracture reduction path based on the detected interaction instructions; the fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0007] The reinforcement learning module is used to determine at least one initial reduction path based on the fracture type, the matched and calibrated three-dimensional model of the fracture site, and the fracture surgery requirements; the state of the reinforcement learning module is defined based on the geometric features of the matched and calibrated three-dimensional model of the fracture site, the relative orientation information of the bone fragments, and the fracture type; the action space of the reinforcement learning module is defined based on the movement, rotation, and force of the bone fragments; the reward function of the reinforcement learning module is defined based on the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements, and physiological constraint requirements;

[0008] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path;

[0009] The reward function R of the reinforcement learning module is expressed as follows:

[0010] R=r accuracy +r smoothness +r physiology

[0011] r accuracy =-α·||M target -M current ||

[0012]

[0013] Where r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissues; M target is the target position coordinate, indicating the position the bone should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the precision adjustment parameter, which is used to adjust the impact of reset precision on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0014] In a second aspect, an embodiment of the present application provides a fracture reduction path auxiliary planning system based on augmented reality technology, the system comprising: a data acquisition unit for acquiring fracture imaging data for a patient's fracture area, the fracture imaging data comprising CT data and X-ray data; a model reconstruction unit for reconstructing a three-dimensional model of the fracture part based on the fracture imaging data; the three-dimensional model of the fracture part comprises a plurality of bone block modules, and each of the bone block modules is respectively marked with a corresponding position, boundary and color; a model calibration unit for matching and calibrating the reconstructed three-dimensional model of the fracture part with the patient's fracture area in an actual surgical scene; a path planning unit for inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgical requirements into a fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; a path confirmation unit for rendering each of the optimized fracture reduction paths and determining a target fracture reduction path according to the detected interaction instructions; the fracture reduction path planning model comprises a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0015] The reinforcement learning module is used to determine at least one initial reduction path based on the fracture type, the matched and calibrated three-dimensional model of the fracture site, and the fracture surgery requirements; the state of the reinforcement learning module is defined based on the geometric features of the matched and calibrated three-dimensional model of the fracture site, the relative orientation information of the bone fragments, and the fracture type; the action space of the reinforcement learning module is defined based on the movement, rotation, and force of the bone fragments; the reward function of the reinforcement learning module is defined based on the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements, and physiological constraint requirements;

[0016] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path;

[0017] The reward function R of the reinforcement learning module is expressed as follows:

[0018] R=r accuracy +r smoothness +r physiology

[0019] r accuracy =-α·||M target -M current ||

[0020]

[0021] Where r accuracyRepresents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissues; M target is the target position coordinate, indicating the position the bone should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the precision adjustment parameter, which is used to adjust the impact of reset precision on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0022] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method.

[0023] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by electronic devices (including but not limited to computers, servers, or network devices, etc.) to execute the steps of the above-mentioned method of the present application.

[0024] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored on a storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the above method.

[0025] The augmented reality-based fracture reduction path auxiliary planning method, system, electronic device, and non-transitory computer-readable storage medium provided in this application can produce at least the following technical effects:

[0026] (1) The fracture reduction path auxiliary planning method based on augmented reality technology can provide an intuitive three-dimensional view by obtaining the patient's fracture imaging data and reconstructing a three-dimensional model. It can significantly improve the accuracy of reduction and reduce blind adjustments during the reduction process. Through the combination of reinforcement learning modules and multi-physics field simulation optimization modules, the system can generate a reduction path with excellent simulation performance according to the fracture type and surgical requirements, further improving the accuracy of reduction, ensuring the precise alignment of bone fragments, and promoting fracture healing and functional recovery.

[0027] (2) A reinforcement learning algorithm is used to perform preliminary path planning for the fracture area, and the preliminary path is input into a multi-physics simulation model for mechanical simulation and optimization. The adaptability and flexibility of the reinforcement learning algorithm can optimize the path in real time, but it may lack mechanical feasibility and safety in actual operation. The multi-physics simulation model can effectively combine mechanical simulation and actual operating conditions to provide a more accurate and safe path, but it requires high computational costs and complex simulation models. In this technical solution, the fracture reduction path planning model effectively integrates the advantages of the two, achieving the adaptability of path planning and the accuracy and safety of mechanical simulation optimization. It can not only provide preliminary path planning, but also further optimize the path through mechanical simulation to ensure the feasibility and safety of actual operation, and has higher practical value.

[0028] (3) In this technical solution, finite element analysis is performed through a multi-physics field simulation optimization module to simulate the stress distribution on the reduction path, ensuring that the path meets the physiological and mechanical requirements and avoids damage to surrounding tissues. It is especially suitable for situations where the field of view is limited or the fracture site is complex. It can accurately determine the relative position and reduction direction of the bone fragments, reduce the uncertainty and potential risks in the reduction process, and improve the safety of the operation.

[0029] (4) In this technical solution, the reward function of the reinforcement learning module takes into account the reduction accuracy, path smoothness, and physiological constraints, and can be adaptively adjusted according to the actual surgical situation. Through the flexible setting of the accuracy adjustment parameters, smoothness adjustment parameters, and physiological constraint adjustment parameters, the system can dynamically adjust the reduction path to adapt to different types of fractures and surgical requirements. It has strong adaptability and flexibility to meet the needs of different surgical scenarios.

[0030] (5) Traditional fracture reduction surgery relies on the doctor's experience and tactile feedback, often requiring repeated attempts and adjustments, which is time-consuming. Through augmented reality technology and an automated path planning model, this solution can quickly generate and display the fracture reduction path, reducing the doctor's operation time and trial and error process, greatly improving surgical efficiency, significantly shortening the operation time, reducing the doctor's workload, and reducing the patient's surgical risk.

[0031] This technical solution overlays the reconstructed 3D model and reduction path onto the actual surgical scene using AR glasses. Doctors can view the alignment of the reduction path with the patient's fracture area in real time, providing intuitive, real-time guidance. Applying augmented reality to fracture reduction scenarios improves the doctor's operating experience, making the surgical process more controllable and intuitive, and helping to increase surgical success rates and patient recovery outcomes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A flowchart of an example of a fracture reduction path auxiliary planning method based on augmented reality technology according to an embodiment of the present application is shown;

[0034] Figure 2 An operational flowchart of an example of reconstructing a three-dimensional model of a fractured part based on fractured part image data according to an embodiment of the present application is shown;

[0035] Figure 3 An operational flowchart of an example of generating a three-dimensional surface model based on an enhanced surface reconstruction algorithm according to an embodiment of the present application is shown;

[0036] Figure 4 An operational flowchart of an example of matching and calibrating a reconstructed three-dimensional model of a fracture with a patient's fracture area in an actual surgical scenario according to an embodiment of the present application is shown;

[0037] Figure 5 A structural block diagram of an example of a fracture reduction path planning model according to an embodiment of the present application is shown;

[0038] Figure 6 A structural block diagram of an example of a fracture reduction path auxiliary planning system based on augmented reality technology according to an embodiment of the present application is shown;

[0039] Figure 7 This is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Unless otherwise defined, the technical or scientific terms used in this application should have the usual meaning understood by people with ordinary skills in the field to which this application belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, "one", "an" or "the" and other similar words do not indicate a quantity limitation, but rather indicate the existence of at least one. "Include" or "comprising" and other similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and other similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0042] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in this application are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0043] Figure 1 A flowchart of an example of a fracture reduction path auxiliary planning method based on augmented reality technology according to an embodiment of the present application is shown.

[0044] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities, so as to significantly improve the accuracy and efficiency of fracture reduction surgery, reduce the difficulty and risk of surgery, improve postoperative recovery effects, and promote the standardization and repeatability of surgical operations by utilizing augmented reality technology, reinforcement learning and multi-physics field simulation optimization.

[0045] In some examples, it may be integrated into the configuration server through software, hardware, or a combination of software and hardware, which should not be limited here.

[0046] The following will use the fracture reduction planning platform as an exemplary implementation subject to introduce the details of the technical solution involved in this application. However, it should be understood that one or more steps involved in the following process can be implemented by one or more controllers or software installed and deployed in the client or server.

[0047] like Figure 1 As shown, in step S110, fracture image data for the fracture area of the patient is acquired, and the fracture image data includes CT (computer tomography) data and X-ray data.

[0048] In some embodiments, CT and X-ray data of the patient's fracture area are collected to ensure high resolution and completeness of the image data. Furthermore, the collected image data can be preprocessed, including denoising, standardization, and calibration, to improve data quality and accuracy. The processed image data is then input into the system to generate a complete image dataset of the fracture area. This ensures the accuracy and reliability of the basic data for subsequent 3D model reconstruction and path planning, improving the precision of the entire fracture reduction path planning process.

[0049] More specifically, imaging data of the patient's fracture area is acquired from CT and X-ray scanners. These imaging data should include multiple slices covering all levels of the fracture area. Furthermore, CT and X-ray data are typically stored in DICOM format and need to be standardized for subsequent processing.

[0050] In step S120, a three-dimensional model of the fractured part is reconstructed based on the fractured part image data. The three-dimensional model of the fractured part includes a plurality of bone block modules, and each bone block module is marked with a corresponding position, boundary and color.

[0051] In some embodiments, an image segmentation algorithm is used to segment the bone fragments in the fracture image data. Then, based on the segmentation results, a three-dimensional reconstruction algorithm (such as a surface reconstruction algorithm or a volume reconstruction algorithm) is used to generate a three-dimensional model of the fracture site. Then, the corresponding position, boundary, and color are marked for each bone block module, which can be more conducive to the personalized display of different bone block modules and facilitate subsequent matching and path planning. Thus, through precise image segmentation and three-dimensional reconstruction, a detailed three-dimensional model of the fracture site is generated, allowing doctors to clearly understand the structure of the fracture area and provide accurate spatial information for path planning.

[0052] Figure 2 FIG1 shows an operation flow chart of an example of reconstructing a three-dimensional model of a fracture part based on fracture part image data according to an embodiment of the present application. Figure 2As shown, in step S210, data preprocessing and fusion are performed on the CT data and X-ray data in the fracture image data to obtain a superimposed image of the fracture portion corresponding to the same coordinate system. More specifically, a filter (such as Gaussian filtering) can be used to denoise the CT and X-ray image data to remove noise and artifacts in the image and enhance the clarity of the bone structure. In addition, image enhancement techniques (such as histogram equalization) are used to improve the contrast of the image data and highlight the differences between bones and soft tissues. Afterwards, the slices of the CT and X-ray image data are matched to ensure that images at different levels are accurately superimposed in the same coordinate system. In step S220, the bone module and other tissues in the superimposed image of the fracture portion are identified, and the edge feature points of the bone module are extracted. In one example, a pre-trained deep learning model can be used to segment the image data to automatically identify and extract the bone structure. In another example, according to the grayscale value difference between the bone and soft tissue, an appropriate threshold is set for segmentation to extract the bone area, thereby achieving the segmentation of the bone structure in the image data from other tissues. Then, use edge detection algorithm (for example, Canny edge detection) to extract the edge of bone block, generate the outline of bone block, realize extracting outline and the boundary of bone block from segmentation result.In step S230, process each edge feature point based on surface reconstruction algorithm, to generate three-dimensional surface model.Here, surface reconstruction algorithm can adopt general algorithm or other enhanced innovative algorithms, for example, uses marching cubes (Marching Cubes) algorithm that the bone edge point in section is connected, generates three-dimensional surface model.In step S240, record spatial position, boundary and the shape of each bone block module in the three-dimensional surface model, and mark corresponding color respectively for each bone block module, to obtain fracture part three-dimensional model.Like this, based on segmentation result and edge detection, bone block is divided into multiple modules, and each module is marked, realize the modularization processing of the bone block in the three-dimensional model.

[0053] Thus, through data acquisition and preprocessing, image segmentation, bone structure extraction, 3D reconstruction, and bone fragment modularization, a 3D model of the fracture site can be reconstructed based on the fracture image data. Data acquisition and preprocessing technologies enable rapid processing of CT and X-ray image data, removing noise and enhancing image quality. This allows the doctor to quickly generate processed images after acquiring the imaging data, saving data processing time and accelerating surgical preparation. Image segmentation and bone structure extraction technologies allow for precise separation of bones from other tissues, ensuring that the bone fragments in the 3D model correspond to the actual fracture area. This allows the doctor to develop a more precise surgical plan based on the precise 3D model, reducing surgical errors and improving the accuracy of the reduction.

[0054] In some examples of the embodiments of the present application, the surface reconstruction algorithm was enhanced and optimized in the process of practicing the present application, and an adaptive Marching Cubes algorithm based on deep learning was proposed. By introducing a deep learning model to predict the geometric characteristics and isosurface positions in the local grid, the grid division is adaptively adjusted to further improve the reconstruction accuracy and efficiency.

[0055] Figure 3 An operational flowchart of an example of generating a three-dimensional surface model based on an enhanced surface reconstruction algorithm according to an embodiment of the present application is shown.

[0056] like Figure 3 As shown, in step S310, multi-scale features are extracted from the superimposed images of the fracture site. In step S320, the multi-scale features are input into a deep learning model to predict local mesh geometric properties, which include mesh vertex scalar values and mesh isosurface positions. In step S330, based on the prediction results of the deep learning model, the density and shape of the mesh division are adaptively adjusted to ensure higher resolution in complex structural areas:

[0057]

[0058] Where V adaptive represents the vertex index value in the adaptive mesh; the local geometric characteristics predicted by the deep learning model are P = {p1,p2,...,p n}, where p i represents the geometric characteristics of the local area i; f(p i ) is based on the local geometric characteristics p i Predicted vertex scalar value function. In step S340, in the adaptive mesh, the enhanced marching cubes algorithm is applied to extract isosurface vertices based on the geometric properties predicted by the deep learning model.

[0059] Specifically, the intersection points of the isosurfaces on the edges of the cube are determined by interpolation as the vertices of the isosurfaces:

[0060]

[0061] Where V interpolated is the isosurface vertex obtained by interpolation, V0 is the scalar value of the cube vertex, V target In step S350, these isosurface vertices are connected into triangles to generate a three-dimensional surface model.

[0062] Through this embodiment, an improved Marching Cubes algorithm is applied to extract isosurfaces in an adaptive mesh based on the geometric characteristics predicted by the deep learning model. Adaptive meshing can provide higher resolution in complex structural areas, ensuring that the details of the three-dimensional model are clearer and more accurate. By predicting local geometric characteristics through the deep learning model and adaptively adjusting the mesh density, reconstruction accuracy is improved, unnecessary meshing and calculations can be reduced, and the overall efficiency of surface reconstruction can be improved. Therefore, combining the advantages of deep learning and traditional algorithms, the adaptive Marching Cubes algorithm can better adapt to different types of fracture structures and generate more stable and reliable three-dimensional models.

[0063] In step S130 , the reconstructed three-dimensional model of the fracture is matched and calibrated with the patient's fracture area in the actual surgical scene.

[0064] In some implementations, a registration algorithm (e.g., a point cloud registration algorithm) can be used to align the reconstructed 3D model with the patient's fracture area in the actual surgical scenario. By comparing key points and feature areas, fine-tuning can be performed to ensure that the model is highly consistent with the actual fracture area. Furthermore, the matching and calibration process can be displayed in real time on AR glasses, allowing the doctor to make fine adjustments and confirmation.

[0065] Therefore, through precise matching and calibration, the high consistency between the three-dimensional model and the actual surgical scene is ensured, allowing doctors to accurately refer to the model during the operation, thereby improving the accuracy and safety of reduction.

[0066] It should be noted that various non-restrictive matching and calibration methods can be used, such as point cloud depth image matching technology, to achieve matching and calibration between the three-dimensional model of the fracture site and the actual surgical scene.

[0067] Figure 4 The following is an operational flow chart showing an example of matching and calibrating the reconstructed three-dimensional model of the fracture part with the patient's fracture area in the actual surgical scene according to an embodiment of the present application. Figure 4As shown, in step S410, at least one key feature region in the reconstructed three-dimensional model of the fracture is identified, and a corresponding first identifier is assigned to each key feature region. More specifically, a detailed feature analysis is performed on the reconstructed three-dimensional model of the fracture, for example, using image processing and computer vision techniques to extract geometric features such as feature points, feature lines, and feature surfaces in the three-dimensional model of the fracture to identify key feature regions. In a specific application scenario, the first identifier can be a fracture line, for example, by identifying the starting and ending points of the fracture line in the three-dimensional model as the first identifier. Alternatively, the first identifier can be a key feature region such as an articular surface or bony protrusion. In step S420, environmental sensor data from the surgical scene is collected using the sensor group in the AR glasses. The sensor group includes an optical sensor, a depth sensor, and an IMU sensor. The optical sensor captures real-time images of the surgical scene, the depth sensor obtains depth information of the surgical scene, and the IMU sensor captures device posture data. Furthermore, after obtaining multiple types of sensor sampling data, the data from the optical sensor, depth sensor, and IMU sensor must be synchronized in time and spatially aligned to generate a comprehensive data stream of the surgical scene. In step S430, when at least one second identifier is detected in the environmental sensor data, the three-dimensional model of the fracture part is matched and calibrated according to the calibration position and calibration size corresponding to each second identifier and with reference to each first identifier. The second identifier is pre-set at a specific location in the patient's fracture area in the surgical scene, and the specific location is associated with a key feature area. More specifically, the doctor can pre-place a second identifier at the key feature area (e.g., fracture line, articular surface, and bone protrusion), such as a reflective ball, QR code, or surgical pin, etc.

[0068] In this embodiment, by performing feature analysis on the reconstructed three-dimensional model of the fracture, stable and easily identifiable feature areas (such as fracture lines, articular surfaces, bone protrusions, etc.) are selected as the first identifier to ensure the accuracy of the identifier. The data of the optical sensor, depth sensor and IMU sensor are used to provide comprehensive information of the surgical scene and achieve high-precision environmental perception. The second identifier in the surgical scene is quickly detected by the image processing algorithm, and the correlation between the first identifier and the second identifier is used to automate the matching calibration process and reduce manual operation time. Through this embodiment, the real-time data of the IMU sensor and the depth sensor are used to dynamically adjust the matching of the three-dimensional model with the surgical scene to ensure the continued effectiveness of the three-dimensional model matching. In combination with the business scenario, during the operation, when the posture of the AR glasses changes or the patient moves slightly, the doctor can respond to changes in the surgical process in a timely manner by dynamically adjusting and calibrating the three-dimensional model in real time to ensure the safety and stability of the operation.

[0069] In step S140 , the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements are input into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path.

[0070] In some embodiments, the matched and calibrated three-dimensional model of the fracture is input into the fracture reduction path planning model so that the platform can fully capture the spatial orientation information between the different bone modules in the fracture area. Here, the fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module. The state, action space and reward function of the reinforcement learning module are defined by the input fracture type and surgical requirements, the reinforcement learning module is used to generate the initial reduction path, and the multi-physics field simulation optimization module is used to simulate and optimize the stress distribution of the initial path. Thus, the reinforcement learning and multi-physics field simulation optimization modules are used to generate an optimized fracture reduction path, ensuring the accuracy and smoothness of the path while meeting the physiological and mechanical requirements, reducing damage to surrounding tissues, and improving the reduction effect.

[0071] In step S150, each optimized fracture reduction path is rendered, and a target fracture reduction path is determined according to the detected interaction instruction.

[0072] In some implementations, each repositioning path is rendered within AR glasses, providing an intuitive 3D view and path information. Then, through gesture recognition or voice commands, the surgeon can select and interact with the path. Based on the surgeon's interactive instructions, the final target repositioning path is determined, and real-time guidance is provided during the procedure. Thus, augmented reality technology provides intuitive visualization of the repositioning path, enabling the surgeon to quickly select the optimal path, reducing surgical time and difficulty, and improving surgical success and safety.

[0073] The fracture reduction path planning method provided in the embodiments of this application can provide a unified operational reference for different doctors, reducing the dependence of surgical results on the doctor's experience. In addition, standardized reduction path planning can improve the repeatability and standardization of surgical operations, help promote minimally invasive and precise fracture reduction surgery, and improve overall medical standards.

[0074] Figure 5 A structural block diagram of an example of a fracture reduction path planning model according to an embodiment of the present application is shown.

[0075] like Figure 5 As shown, the fracture reduction path planning model 500 includes a cascaded reinforcement learning module 510 and a multi-physics field simulation optimization module 520.

[0076] The reinforcement learning module 510 is used to determine at least one initial reduction path based on the fracture type, the matched and calibrated three-dimensional model of the fracture site, and the fracture surgery requirements. More specifically, the state of the reinforcement learning module 510 is defined based on the geometric features of the matched and calibrated three-dimensional model of the fracture site, the relative orientation information of the bone fragments, and the fracture type. The action space of the reinforcement learning module is defined based on the movement, rotation, and force applied by the bone fragments. The reward function of the reinforcement learning module 510 is defined based on the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements, and physiological constraint requirements.

[0077] More specifically, a RL environment based on a three-dimensional model is established, and a large number of simulation trainings are performed through a reinforcement learning algorithm to generate multiple preliminary reset paths that satisfy reward constraints.

[0078] Each initial reduction path is input into the multi-physics field simulation optimization module 520 to perform stress distribution simulation on each initial reduction path through finite element analysis, and the corresponding initial path is optimized and adjusted according to the simulation results to determine at least one corresponding optimized fracture reduction path.

[0079] More specifically, finite element analysis is used to simulate the stress distribution of the preliminary path to assess its mechanical feasibility. Furthermore, dynamic response analysis is performed to ensure its stability during actual operation. Furthermore, based on the simulation results, the preliminary path is optimized and adjusted to generate one or more optimal fracture reduction paths that meet the requirements.

[0080] Here, a reinforcement learning (RL) algorithm is used to generate an optimized fracture reduction path. Through continuous simulation and feedback learning, the reduction path is optimized to achieve the best results. Furthermore, multi-physics simulation technology is used to perform mechanical simulation and adjustment optimization on the generated reduction path.

[0081] In this embodiment, a reinforcement learning algorithm is used to perform preliminary path planning on the fracture area, and the preliminary path is input into a multi-physics field simulation model for mechanical simulation and optimization. The adaptability and flexibility of the reinforcement learning algorithm can optimize the path in real time, but it may be lacking in mechanical feasibility and safety in actual operation. The multi-physics field simulation model can effectively combine mechanical simulation and actual operating conditions to provide a more accurate and safe path, but it requires high computing costs and complex simulation models. Through this embodiment, the advantages of both are effectively integrated in the fracture reduction path planning model, achieving the adaptability of path planning and the accuracy and safety of mechanical simulation optimization. It can not only provide preliminary path planning, but also further optimize the path through mechanical simulation to ensure the feasibility and safety of actual operation, and has higher practical value.

[0082] The reward function R of the reinforcement learning module 510 is expressed as follows:

[0083] R=r accuracy +r smoothness +r physiology , Formula (3)

[0084] r accuracy =-α·||M target -M current ||, Formula (4)

[0085]

[0086] Where r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissues; M target is the target position coordinate, indicating the position that the bone should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the precision adjustment parameter, which is used to adjust the impact of reset precision on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0087] The design explanation and optimization instructions for the above reward function are as follows:

[0088] The reduction accuracy reward reflects the positional error of the bone fragment after reduction. The smaller the position error, the higher the reward. In this way, the reinforcement learning module can effectively optimize the fracture reduction path, bringing the position of the bone fragment closer to the target position after reduction, significantly improving the success rate of the surgery and reducing the risk of postoperative bone misalignment.

[0089] The smoothness of the path is reflected through a path smoothness reward; the smoother the path, the higher the reward. This allows the reinforcement learning module to generate a smoother repositioning path, avoiding overly complex and tortuous paths. This makes the repositioning operation smoother, improves the smoothness of the surgical operation, reduces unnecessary adjustments during the procedure, and avoids operational errors caused by complex paths.

[0090] Physiologically constrained rewards ensure that the path meets physiological and mechanical requirements, avoiding damage to surrounding tissue during the manipulation of the bone fragment. In this way, the reinforcement learning module can ensure that the generated reduction path is mechanically safe and feasible, ensuring that no additional damage to surrounding tissue occurs during the operation, ensuring the physiological safety of the operation and reducing the occurrence of postoperative complications.

[0091] Therefore, by adopting the above-mentioned reward function, the reinforcement learning module can achieve higher reduction accuracy, smoother surgical path and higher physiological safety in fracture reduction path planning, which not only improves the success rate and safety of the operation, but also significantly reduces postoperative complications, promotes the rapid recovery of patients, and provides strong support for the development of smart medical technology.

[0092] In some examples of the embodiments of the present application, the multi-physics simulation optimization module is used to perform the following operations:

[0093] A finite element model is established based on the geometric model, fracture type and material properties of the matched and calibrated three-dimensional model of the fracture part.

[0094] Here, the material properties include the material properties of the bone and other surrounding tissues, which can be input through the system or inferred based on the patient's basic information in the fracture surgery requirements, such as gender and age, and should not be restricted here.

[0095] More specifically, a geometric model of the bone block is established using 3D reconstruction data, and the material properties are defined based on the physical characteristics of the bone and soft tissue. Mechanical boundary conditions are then applied to each path node on the initial path, and the contact relationship and constraint conditions between the bone blocks are defined to simulate the external forces and moments applied during surgery.

[0096] Mechanical boundary conditions are applied to each node on the initial reset path to calculate the stress distribution along the path using finite element analysis:

[0097] σ=D(C)·ò , Formula (7)

[0098]

[0099] Where σ is the stress vector, describing the stress state inside the bone; D(C) is the elastic matrix of the material, which depends on the material property matrix C. The elastic matrix represents the elastic properties of the material in all directions; C is the material property matrix, which includes the anisotropic properties of the bone material; ò is the strain vector, describing the deformation state of the bone under the action of external force; is a symmetric gradient operator, which is used to calculate the symmetric gradient of the displacement vector u; u is a displacement vector, which represents the displacement of the material in all directions; u x ,u y ,uz Represent the displacement components in the x, y, and z directions respectively.

[0100] Here, the linear elastic mechanics relationship is used to describe the relationship between stress and strain, while taking into account the anisotropic material properties. In addition, to improve the accuracy of the model, D is expanded to a matrix containing nonlinear terms, considering the elastic modulus of the bone in different directions:

[0101]

[0102] Where C ij are the terms in the material property matrix, which depend on the anisotropic properties of the bone material.

[0103] During complex fracture reduction surgery, surgeons need to understand the stress distribution of the bone fragment under different reduction paths. This embodiment considers the anisotropic material properties and nonlinear contact stress of the bone fragment during stress calculation, enabling more accurate calculation of the stress distribution of the bone fragment and surrounding tissue. This avoids selecting a path that would lead to stress concentration at the fracture site or surrounding tissue, potentially causing secondary damage.

[0104] Perform dynamic response analysis on the initial reset path:

[0105]

[0106] Where M is the mass matrix, which describes the mass distribution of the bone and surrounding tissues; is the acceleration vector, which represents the acceleration of the bone in all directions; λM represents the mass damping term, λ is the mass damping coefficient; μK represents the stiffness damping term, μ is the stiffness damping coefficient; K(u) is the nonlinear stiffness matrix, which depends on the displacement u; is the velocity vector, indicating the velocity of the bone in all directions; N contact (u) is the contact force term, which describes the contact force between bone fragments; is the friction term, which describes the friction between the bones; F(t) is the time-dependent external force vector, which represents the external force applied during the operation.

[0107] In some embodiments, a mass matrix is calculated based on the material properties and geometry of the bone fragment, a damping matrix is calculated based on the damping characteristics of the bone fragment, and a stiffness matrix is calculated based on the stiffness characteristics and geometry of the bone fragment. Dynamic equations are then applied to perform simulations to calculate the dynamic response of the bone fragment during reduction. This allows for dynamic response analysis of the path to ensure its stability and robustness during actual operation.

[0108] During surgery, the bone fragment may be subjected to various dynamic loads (such as the force of surgical tools, slight movements of the patient, etc.). Here, the advanced dynamic equation provided by Equation (10) combines nonlinear force terms and damping effects to more accurately simulate the dynamic loads and mechanical responses of the bone fragment during surgery, allowing doctors to foresee and avoid possible mechanical instability and ensure a smooth surgical procedure.

[0109] The initial reset path is optimized and adjusted according to the dynamic response analysis results to ensure the mechanical safety of the reset path in actual operation.

[0110] In some implementations, a variety of non-restrictive objective optimization algorithms, such as genetic algorithms and particle swarm optimization algorithms, can also be used to solve the problem. Based on the stress distribution results, the positions and directions of the path nodes are adjusted, and the stress distribution is recalculated for the adjusted path to verify the safety of the path. The optimization steps are repeated until the path meets the requirements of the optimization objective function.

[0111] In this embodiment, based on the simulation results of stress distribution and dynamic response, the initial reduction path can be iteratively optimized multiple times to ensure that the final path is optimal in terms of accuracy, smoothness, and mechanical safety. In terms of business scenarios, before surgery, through multiple simulations and path optimization, a reduction path with excellent simulation performance can be determined. This path not only accurately reduces the bone fragments, but also reduces damage to surrounding tissues, improving the success rate of surgery and the patient's recovery speed. In addition, by considering the nonlinear properties of the material and the nonlinear contact forces between the bone fragments, the system can better adapt to different types of fractures and complex surgical scenarios. Therefore, when treating various complex fracture types (such as multiple fractures, comminuted fractures, etc.), the system can provide targeted stress analysis and path optimization to ensure the optimal operation path for each surgical scenario. In addition, during surgery, the bone fragments may produce small vibrations due to tool operation or patient movement. By accurately simulating the damping effect, the system can better predict and offset the vibrations and mechanical instabilities that may occur during surgery. Based on the accurate damping simulation results, the path can be adjusted in time to reduce the impact of vibration on the surgery and improve the stability of the surgery.

[0112] In some examples of the embodiments of the present application, in order to better combine the results of the dynamic response analysis, a more comprehensive optimization objective function formula can be adopted.

[0113] More specifically, the solution formula for optimizing the objective function value minF(p,q) is:

[0114] min F(p,q)=w1·E accuracy +w2·E smoothness +w3·E stress +w4·Edynamic , Formula (11)

[0115]

[0116] Where, E accuracy Represents the reset accuracy error term, E smoothness represents the path smoothness error term, E stress Represents the stress safety error term, E dynamic represents the dynamic response error term; w1, w2, w3, w4 represent weight parameters, which are used to measure the importance of the corresponding error term in the overall optimization; p target,i represents the target position coordinates of the i-th path node, p current,i Indicates the current path position coordinates of the i-th path node; p current,i+1 and p current,i-1 Indicates the location of the adjacent path nodes for the i-th path node

[0117] mark; u i represents the acceleration of the i-th path node, u i represents the speed of the i-th path node, u i represents the displacement of point i on the path, N contact (u i ) represents the contact force at the i-th path node, represents the friction force at the i-th path node.

[0118] To put this into context, the implementation details are as follows: First, an initial reset path is generated using the reinforcement learning module. Next, dynamic response analysis is performed using advanced dynamic equations to calculate the displacement, velocity, acceleration, contact force, and friction at each path point. Furthermore, based on the results of the dynamic response analysis, the optimization objective function min F(p,q) is calculated. Finally, the position and orientation of the path nodes are adjusted based on the objective function values.

[0119] By optimizing the stress safety error term E in the objective function stress and the dynamic response error term E dynamic , which can effectively avoid stress concentration and dynamic instability caused by improper path selection, ensuring the mechanical safety of the bone fragment and surrounding tissue during reduction. Therefore, in complex fracture reduction surgery, accurate stress analysis and dynamic response assessment can be used to select the most mechanically safe reduction path, avoiding secondary injury during and after surgery.

[0120] By optimizing the reset accuracy error term E in the objective function accuracy and the path smoothness error term E smoothness, can generate a smoother and more precise reduction path, improving the smoothness and success rate of surgical operations. Therefore, before surgery, through multiple simulations and path optimization, one or more reduction paths with excellent simulation performance can be determined. This not only accurately reduces bone fragments, but also reduces damage to surrounding tissues, improving the success rate of surgery and the patient's recovery speed.

[0121] By optimizing the dynamic response error term E in the objective function dynamic , which can ensure the stable mechanical response of the path during surgery and avoid instability during the operation. At the same time, through real-time monitoring and dynamic adjustment, the reduction path can be adjusted in time according to unexpected situations during the operation to ensure the smooth progress of the operation. Therefore, during the operation, the platform system can monitor the dynamic response of the bone fragment in real time and provide real-time adjustment suggestions to help the doctor adjust the operation path in time to ensure the stability and safety of the operation.

[0122] Therefore, by optimizing the various error terms in the objective function, the system can adapt to different types of fractures and complex surgical scenarios, providing precise path optimization and mechanical analysis, thereby improving surgical success rates and patient recovery outcomes. This allows the platform system to provide targeted stress analysis and path optimization when treating a variety of complex fracture types (such as multiple fractures and comminuted fractures), ensuring optimal performance of the operational paths output by the platform system in each surgical scenario.

[0123] It should be noted that in the fracture reduction path planning model, optimizing and constraining the reduction path through the reinforcement learning module and the simulation analysis module, respectively, allows for the combined advantages of both techniques. For example, both modules include constraints on reduction accuracy and path smoothness. Thus, by constraining the same objective through both the reinforcement learning module's reward function and the simulation analysis module's objective function, the advantages of both can be fully utilized, improving the overall path planning effect.

[0124] Through the dual constraints of the reinforcement learning module and the simulation analysis module, the robustness of path planning can be improved, ensuring the reliability of the path in different environments and conditions. The reinforcement learning module and the simulation analysis module respectively optimize different objectives of the path (such as accuracy, smoothness, and mechanical safety). By comprehensively utilizing the advantages of both, the global optimization of multiple objectives can be achieved. In addition, the path generated by the reinforcement learning module is calibrated and optimized through the simulation analysis module to ensure the accuracy and safety of the path in actual operation and avoid the infeasibility of the algorithm-generated path in physical implementation. Therefore, through the dual constraints of the reinforcement learning module and the simulation analysis module, the robustness of path planning can be improved, ensuring the reliability of the path in different environments and conditions.

[0125] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0126] Figure 6 A structural block diagram of an example of a fracture reduction path auxiliary planning system based on augmented reality technology according to an embodiment of the present application is shown.

[0127] like Figure 6 As shown, the fracture reduction path auxiliary planning system 600 based on augmented reality technology includes a data acquisition unit 610, a model reconstruction unit 620, a model calibration unit 630, a path planning unit 640 and a path confirmation unit 650.

[0128] The data acquisition unit 610 is used to acquire fracture image data of a patient's fracture area, where the fracture image data includes CT data and X-ray data.

[0129] The model reconstruction unit 620 is used to reconstruct a three-dimensional model of the fracture part based on the fracture part image data; the three-dimensional model of the fracture part includes multiple bone block modules, and each of the bone block modules is marked with a corresponding position, boundary and color.

[0130] The model calibration unit 630 is used to match and calibrate the reconstructed three-dimensional model of the fracture part with the patient's fracture area in the actual surgical scene.

[0131] The path planning unit 640 is used to input the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path.

[0132] The path confirmation unit 650 is used to render various optimized fracture reduction paths and determine a target fracture reduction path according to the detected interaction instructions.

[0133] The fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module;

[0134] The reinforcement learning module is used to determine at least one initial reduction path based on the fracture type, the matched and calibrated three-dimensional model of the fracture site, and the fracture surgery requirements; the state of the reinforcement learning module is defined based on the geometric features of the matched and calibrated three-dimensional model of the fracture site, the relative orientation information of the bone fragments, and the fracture type; the action space of the reinforcement learning module is defined based on the movement, rotation, and force of the bone fragments; the reward function of the reinforcement learning module is defined based on the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements, and physiological constraint requirements;

[0135] Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding fracture reduction path;

[0136] The reward function R of the reinforcement learning module is expressed as follows:

[0137] R=r accuracy +r smoothness +r physiology

[0138] r accuracy =-α·||M target -M current ||

[0139]

[0140] Where r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the physiological constraint reward, which is used to ensure that the path meets the physiological mechanical requirements to avoid damaging the surrounding tissues; M target is the target position coordinate, indicating the position the bone should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the precision adjustment parameter, which is used to adjust the impact of reset precision on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

[0141] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by electronic devices (including but not limited to computers, servers, or network devices, etc.) to execute the above-mentioned fracture reduction path auxiliary planning method based on augmented reality technology in this application.

[0142] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned fracture reduction path auxiliary planning method based on augmented reality technology.

[0143] In some embodiments, an embodiment of the present application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a fracture reduction path auxiliary planning method based on augmented reality technology.

[0144] Figure 7 is a hardware structure diagram of an electronic device for executing a fracture reduction path auxiliary planning method based on augmented reality technology provided by another embodiment of the present application, such as Figure 7 As shown, the device includes:

[0145] One or more processors 710 and memory 720, Figure 7 A processor 710 is taken as an example.

[0146] The device for executing the fracture reduction path auxiliary planning method based on augmented reality technology may further include: an input device 730 and an output device 740 .

[0147] The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0148] Memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the augmented reality-based fracture reduction path assisted planning method in the embodiments of the present application. Processor 710 executes the non-volatile software programs, instructions, and modules stored in memory 720 to execute various server functional applications and data processing, thereby implementing the augmented reality-based fracture reduction path assisted planning method in the above-mentioned method embodiment.

[0149] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 720 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely located relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The input device 730 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0151] The one or more modules are stored in the memory 720 , and when executed by the one or more processors 710 , perform the fracture reduction path auxiliary planning method based on augmented reality technology in any of the above method embodiments.

[0152] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0153] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0154] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0155] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs.

[0156] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0157] (4) Other onboard electronic devices with data interaction functions, such as onboard computer devices installed in vehicles.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fracture reduction path auxiliary planning method based on augmented reality technology, applied to AR glasses, characterized in that: The method comprises: Acquiring fracture imaging data for a fracture area of a patient, wherein the fracture imaging data includes CT data and X-ray data; Reconstructing a three-dimensional model of the fracture part based on the fracture image data; the three-dimensional model of the fracture part includes a plurality of bone block modules, and each of the bone block modules is marked with a corresponding position, boundary and color; Match and calibrate the reconstructed three-dimensional model of the fracture with the patient's fracture area in the actual surgical scene; Inputting the matched and calibrated three-dimensional model of the fracture part and the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; Rendering each of the optimized fracture reduction paths, and determining a target fracture reduction path according to the detected interaction instructions; The fracture reduction path planning model includes a cascaded reinforcement learning module and a multi-physics field simulation optimization module; The reinforcement learning module is used to determine at least one initial reduction path based on the fracture type, the matched and calibrated three-dimensional model of the fracture site, and the fracture surgery requirements; the state of the reinforcement learning module is defined based on the geometric features of the matched and calibrated three-dimensional model of the fracture site, the spatial relative orientation information of the bone fragments, and the fracture type; the action space of the reinforcement learning module is defined based on the movement, rotation, and force of the bone fragments; the reward function of the reinforcement learning module is defined based on the fracture surgery requirements, which include reduction accuracy requirements, path smoothness requirements, and physiological constraint requirements; Inputting each of the initial reduction paths into a multi-physics field simulation optimization module to perform stress distribution simulation on each of the initial reduction paths through finite element analysis, and optimizing and adjusting the corresponding initial paths according to the simulation results to determine at least one corresponding optimized fracture reduction path; The reward function R of the reinforcement learning module is expressed as follows: R＝r accuracy +r smoothness +r physiology r accuracy =-α·‖M target -M current ‖ Where r accuracy Represents the reduction accuracy reward item, which is used to reflect the position error of the bone after reduction; r smoothness represents the path smoothness reward item, which is used to reflect the smoothness of the path; r physiology represents the biomechanical constraint reward, which is used to ensure that the path meets the biomechanical requirements to avoid damaging surrounding tissues; M target is the target position coordinate, indicating the position the bone should reach after reduction; M current is the current bone block position coordinate, indicating the current position of the bone block; α is the precision adjustment parameter, which is used to adjust the impact of reset precision on the total reward; κ i is the path curvature, which represents the curvature value of the i-th path node on the path; N is the total number of path nodes; β is the smoothness adjustment parameter, which is used to adjust the impact of path smoothness on the total reward; σ i is the stress value of the i-th path node on the path, σ limit is the physiological stress limit, which means that when the stress on the path exceeds this limit, it will cause damage to the tissue; γ is the physiological constraint adjustment parameter, which is used to adjust the impact of physiological constraints on the total reward.

2. The method according to claim 1, characterized in that The multi-physics simulation optimization module is used to perform the following operations: Establishing a finite element model based on the geometric model, fracture type and material properties of the matched and calibrated three-dimensional model of the fracture site; Mechanical boundary conditions are applied to each node on the initial reset path to calculate the stress distribution along the path using finite element analysis: σ=D(C)·∈ Where σ is the stress vector, describing the stress state inside the bone; D(C) is the elastic matrix of the material, which depends on the material property matrix C. The elastic matrix represents the elastic properties of the material in all directions; C is the material property matrix, which includes the anisotropic properties of the bone material; ∈ is the strain vector, describing the deformation state of the bone under the action of external force; is a symmetric gradient operator, which is used to calculate the symmetric gradient of the displacement vector u; u is a displacement vector, which represents the displacement of the material in all directions; u x ,u y ,u z Represent the displacement components in the x, y, and z directions respectively; Perform dynamic response analysis on the initial reset path: Where M is the mass matrix, which describes the mass distribution of the bone and surrounding tissues; is the acceleration vector, which represents the acceleration of the bone in all directions; λM represents the mass damping term, λ is the mass damping coefficient; μK represents the stiffness damping term, μ is the stiffness damping coefficient; K(u) is the nonlinear stiffness matrix, which depends on the displacement u; is the velocity vector, indicating the velocity of the bone in all directions; N contact (u) is the contact force term, which describes the contact force between bone fragments; is the friction term, describing the friction between the bone fragments; F(t) is the time-dependent external force vector, representing the external force applied during the surgery; The initial reset path is optimized and adjusted according to the dynamic response analysis results to ensure the mechanical safety of the reset path in actual operation.

3. The method according to claim 2, wherein: The optimizing and adjusting the initial reset path according to the dynamic response analysis result includes: According to the results of dynamic response analysis, calculate the optimization objective function value F(p,q); According to the optimization objective function value F(p,q), the position and direction of the path node are adjusted: Where p current Indicates the position coordinates of the current path node, q current Represents the direction vector of the current path node, Indicates the adjusted path position coordinates, Represents the adjusted path direction vector; η represents the learning rate, which is used to control the step size of each adjustment; represents the gradient of the objective function with respect to the path position; Represents the gradient of the objective function with respect to the path direction; The dynamic response analysis and path adjustment steps are repeated until the objective function converges to a minimum value.

4. The method according to claim 3, wherein: The solution formula for optimizing the objective function value minF(p,q) is: minF(p,q)=w1·E accuracy +w2·E smoothness +w3·E stress +w4·E dynamic Where, E accuracy Represents the reset accuracy error term, E smoothness represents the path smoothness error term, E stress Represents the stress safety error term, E dynamic represents the dynamic response error term; w1, w2, w3, w4 represent weight parameters, which are used to measure the importance of the corresponding error term in the overall optimization; p target,i represents the target position coordinates of the i-th path node, p current,i Indicates the current path position coordinates of the i-th path node; p current,i+1 and p current,i-1 Represents the position coordinates of the adjacent path nodes for the i-th path node; represents the acceleration of the i-th path node, represents the speed of the i-th path node, u i represents the displacement of point i on the path, N contact (u i ) represents the contact force at the i-th path node, represents the friction force at the i-th path node.

5. The method according to claim 1, wherein The reconstructing of a three-dimensional model of the fracture part based on the fracture image data comprises: Preprocessing and fusing the CT data and X-ray data in the fracture imaging data to obtain a superimposed image of the fracture portion corresponding to the same coordinate system; Identifying bone modules and other tissues in the superimposed image of the fracture part, and extracting edge feature points of the bone modules; Processing each of the edge feature points based on a surface reconstruction algorithm to generate a three-dimensional surface model; The spatial position, boundary and shape of each bone block module in the three-dimensional surface model are recorded, and each bone block module is marked with a corresponding color to obtain a three-dimensional model of the fracture part.

6. The method according to claim 5, wherein: The processing of each of the edge feature points based on a surface reconstruction algorithm to generate a three-dimensional surface model includes: Extract multi-scale features from superimposed images of fracture sites; Inputting the multi-scale features into a deep learning model to predict local mesh geometric properties; the local mesh geometric properties include mesh vertex scalar values and mesh isosurface locations; Based on the predictions of the deep learning model, the density and shape of the mesh are adaptively adjusted to ensure higher resolution in complex structural areas: Where V adaptive represents the vertex index value in the adaptive mesh; the local geometric characteristics predicted by the deep learning model are P = {p1,p2,...,p n }, where p i represents the geometric characteristics of the local area i; f(p i ) is based on the local geometric characteristics p i predicted vertex scalar-valued function; In the adaptive mesh, an enhanced marching cubes algorithm is applied to extract isosurface vertices based on the geometric characteristics predicted by the deep learning model. Specifically, the algorithm includes: The intersection points of the isosurfaces on the cube edges are determined by interpolation as the vertices of the isosurfaces: Where V interpolated is the isosurface vertex obtained by interpolation, V0 is the scalar value of the cube vertex, V target is the target equivalent face value; These isosurface vertices are connected into triangles to generate a three-dimensional surface model.

7. The method according to claim 5 or 6, wherein: The matching and calibration of the reconstructed three-dimensional fracture model with the patient's fracture area in the actual surgical scene includes: Identifying at least one key feature region in the reconstructed three-dimensional model of the fracture, and assigning a corresponding first identifier to each key feature region; Collect environmental sensor data in the surgical scene based on the sensor group in the AR glasses; the sensor group includes optical sensors, depth sensors, and IMU sensors; When at least one second identifier is detected in the environmental sensing data, the three-dimensional model of the fracture is matched and calibrated according to the calibration position and calibration size corresponding to each second identifier, and with reference to each first identifier; the second identifier is pre-set at a specific position of the patient's fracture area in the surgical scene, and the specific position is associated with the key feature area.

8. A fracture reduction path auxiliary planning system based on augmented reality technology, used to implement the method according to any one of claims 1 to 7, characterized in that: The system comprises: a data acquisition unit, configured to acquire fracture imaging data of a patient's fractured area, wherein the fracture imaging data includes CT data and X-ray data; A model reconstruction unit, configured to reconstruct a three-dimensional model of the fractured part based on the fracture image data; the three-dimensional model of the fractured part comprises a plurality of bone block modules, and each of the bone block modules is marked with a corresponding position, boundary and color; A model calibration unit, used to match and calibrate the reconstructed three-dimensional model of the fracture with the patient's fracture area in the actual surgical scene; A path planning unit, configured to input the matched and calibrated three-dimensional model of the fracture part, the corresponding fracture type and fracture surgery requirements into the fracture reduction path planning model to determine at least one corresponding optimized fracture reduction path; The path confirmation unit is used to render each of the optimized fracture reduction paths and determine a target fracture reduction path according to the detected interaction instruction.

Images