Metaverse surgery planning three-dimensional scene establishment system, method, device and medium

Abstract

The application discloses a metaverse surgery planning three-dimensional scene establishment system, method, device and medium. The metaverse surgery planning three-dimensional scene establishment system comprises a scene establishment module, a data acquisition module and a surgery planning module. The scene establishment module is used for establishing a metaverse surgery planning three-dimensional scene, establishing a target planning sub-scene and a virtual character linked with user actions in the metaverse surgery planning three-dimensional scene. The data acquisition module is used for acquiring medical image data of a target object and constructing an object three-dimensional model corresponding to the medical image data in the target planning sub-scene. The surgery planning module is used for realizing surgery planning of the target object according to a virtual action signal of the virtual character manipulating the object three-dimensional model. The application realizes surgery planning by means of the metaverse concept, provides accurate and favorable technical support and guarantee for surgery planning, supports multiple people to collaboratively complete surgery planning, enhances user experience, and makes the surgery planning process more convenient and efficient.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a system, method, equipment, and medium for establishing a three-dimensional scene for metaverse surgical planning. Background Technology

[0002] Currently, for some surgeries, such as joint replacement surgery, medical staff conduct surgical planning before the procedure. Joint replacement surgery is currently an effective treatment for joint diseases, especially the only treatment for end-stage osteoarthritis. With the increasing aging of China's population, the high prevalence of arthritis, and the improvement of surgical techniques, imaging equipment, surgical devices, and medical informatization in my country, the number of joint replacement surgeries is rapidly increasing. However, joint replacement surgery is complex and requires a high level of skill from surgeons. Generally, surgeons will measure specific physiological parameters or predict the location of joint lesions based on the patient's imaging data (DR films, CT scans, etc.). Some surgeons will also use tools such as mimics and CAD to manually segment CT data, then mark feature points on the segmented bone tissue and plan the prosthesis to obtain a more detailed surgical plan. However, these current planning methods are cumbersome, inefficient, and lack high precision in parameter calculation. They also require surgeons to visualize the patient's three-dimensional environment, which is not very engaging and makes it difficult for multiple surgeons to consult and discuss the procedure simultaneously. Summary of the Invention

[0003] The technical problem to be solved by this invention is to overcome the shortcomings of existing surgical planning methods, which are cumbersome, inefficient and lack user experience, and to provide a metaverse surgical planning 3D scene creation system, method, equipment and medium.

[0004] The present invention solves the above-mentioned technical problems through the following technical solution:

[0005] Firstly, a metaverse surgical planning 3D scene creation system is provided, including:

[0006] The scene creation module is used to create a metaverse surgical planning 3D scene, and to create a target planning sub-scene and a virtual character that interacts with the user's actions in the metaverse surgical planning 3D scene; the surgical planning process of the metaverse surgical planning 3D scene includes multiple planning process nodes, and each planning process node corresponds to a planning sub-scene.

[0007] The data acquisition module is used to acquire medical image data of the target object and construct a 3D model of the object corresponding to the medical image data in the target planning sub-scene;

[0008] The surgical planning module is used to plan the surgery for the target object based on the virtual action signals of the virtual character manipulating the three-dimensional model of the object.

[0009] Optionally, the scene creation module includes:

[0010] The virtual character creation unit is used to create a virtual character that interacts with the user's actions in the metaverse surgical planning three-dimensional scene in response to a surgical planning request.

[0011] And / or, the virtual character creation unit is used to create a virtual character in the three-dimensional scene of the metaverse surgical planning that is linked to the user actions of the second user, based on the invitation credential provided by the second user; the invitation credential is generated based on the invitation request of the first user.

[0012] Optionally, it also includes:

[0013] The permission setting module is used to respond to the permission authorization request of the first user and set user permissions for the second user corresponding to the permission authorization request.

[0014] Optionally, the scene creation module includes:

[0015] A switching unit is used to respond to a scene switching command of the virtual character and cause the virtual character to enter another target planning sub-scene corresponding to the scene switching command;

[0016] Alternatively, the switching unit is used to sequentially determine the target planning sub-scene from multiple planning sub-scenes according to the order of planning process nodes, and when the target event is triggered, to cause the virtual character to enter the target planning sub-scene.

[0017] Optionally, the planning process data of the previously displayed target planning sub-scenario can be used as the input data for the currently displayed target planning sub-scenario.

[0018] Alternatively, planning process data can be shared across different planning sub-scenarios.

[0019] Optionally, the scene creation module further includes:

[0020] The sub-scene creation unit is used to determine user permissions and cause the virtual character to enter the target planning sub-scene that matches the user permissions.

[0021] Optionally, it also includes:

[0022] The permission modification module is used to respond to the permission modification request of the first user and modify the user permissions of the second user according to the permission modification request.

[0023] Optionally, the planning sub-scenario includes at least one of the following:

[0024] The sub-scenes include: raw data warehouse, 3D model building, surgical planning, prosthesis warehouse, and simulation testing.

[0025] Optionally, the metaverse surgical planning 3D scene also includes a surgical planning hall;

[0026] The metaverse surgical planning 3D scene creation system also includes:

[0027] The login module is used to complete user login in the surgical planning hall in response to login requests;

[0028] And / or, a registration module, used to complete user registration in the surgical planning hall in response to a registration request.

[0029] Secondly, a method for establishing a 3D scene for metaverse surgical planning is provided, including:

[0030] A metaverse surgical planning 3D scene is established, and a target planning sub-scene and a virtual character that interacts with the user's actions are established in the metaverse surgical planning 3D scene; the surgical planning process includes multiple planning process nodes, and each planning process node corresponds to a planning sub-scene.

[0031] Acquire medical image data of the target object, and construct a 3D model of the object corresponding to the medical image data in the target planning sub-scene;

[0032] Based on the virtual action signals of the virtual character manipulating the three-dimensional model of the object, surgical planning for the target object is realized.

[0033] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for establishing a three-dimensional scene for metaverse surgical planning.

[0034] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the above-described method for establishing a three-dimensional scene for metaverse surgical planning.

[0035] The positive and progressive effects of this invention are as follows: This invention utilizes the concept of metaverse to realize surgical planning, providing accurate and advantageous technical support and guarantee for surgical planning, supporting multiple people to complete surgical planning collaboratively, enhancing the user experience, and making the surgical planning process more convenient and efficient. Attached Figure Description

[0036] Figure 1a A schematic diagram of the architecture of a metaverse surgical planning 3D scene creation system is provided as an exemplary embodiment;

[0037] Figure 1b A schematic diagram of a module for establishing a 3D scene for surgical planning in a metaverse is provided as an exemplary embodiment.

[0038] Figure 1c A schematic diagram of a module for establishing a 3D scene for metaverse surgical planning provided as an exemplary embodiment;

[0039] Figure 2 A flowchart of a method for establishing a 3D scene for metaverse surgical planning provided as an exemplary embodiment of the present invention;

[0040] Figure 3a A schematic diagram of a 3D scene for metaverse surgical planning constructed by a method for establishing a 3D scene for metaverse surgical planning, provided as an exemplary embodiment of the present invention;

[0041] Figure 3b A schematic diagram of a metaverse surgical planning 3D scene constructed by another metaverse surgical planning 3D scene establishment method provided for an exemplary embodiment of the present invention;

[0042] Figure 4 A flowchart illustrating another method for establishing a 3D scene for metaverse surgical planning, provided as an exemplary embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0045] Figure 1a This is a schematic diagram of the architecture of a metaverse surgical planning 3D scene creation system, provided as an exemplary embodiment. For example... Figure 1a As shown, the system may include a server 11 and VR (Virtual Reality) devices, such as VR glasses 13, VR headsets 14, etc. The network 12 for interaction between the VR devices and the server 11 may include various types of wired or wireless networks. The server 11 may be a physical server containing an independent host, or the server 11 may be a virtual server hosted by a host cluster.

[0046] During operation, server 11 can run the server-side program of the Metaverse Surgical Planning 3D Scene Creation System to achieve related processing functions. During operation, the VR device can display the Metaverse Surgical Planning 3D Scene created by server 11, sense the user's actions and send them to server 11 to achieve human-computer interaction functions. Thus, through the cooperation between server 11 and VR device, the creation of the Metaverse Surgical Planning 3D Scene is realized.

[0047] Figure 1b This is a schematic diagram of a metaverse surgical planning 3D scene establishment system provided as an exemplary embodiment of the present invention. It is applicable to preoperative and intraoperative planning of various surgeries. The metaverse surgical planning 3D scene establishment system includes: a scene establishment module 101, a data acquisition module 102, and a surgical planning module 103.

[0048] The scene creation module 101 is used to create a metaverse surgical planning 3D scene, and to create a target planning sub-scene and a virtual character that interacts with the user's actions in the metaverse surgical planning 3D scene.

[0049] The surgical planning process includes multiple planning process nodes, each planning process node corresponds to a planning sub-scenario, and the target planning sub-scenario is the scene currently displayed to the user or the scene in which the user is currently located.

[0050] The Metaverse Surgical System supports the creation of multiple virtual characters in the 3D scene of Metaverse surgical planning, which can be linked with the actions of multiple users. It supports multi-person collaborative surgical planning, multi-angle observation of the surgical planning process, and provides users with an immersive surgical planning environment. Even if users participate remotely, they will feel as if they are there, enhancing the user experience.

[0051] The data acquisition module 102 is used to acquire medical image data of the target object and construct a 3D model of the object corresponding to the medical image data in the target planning sub-scene.

[0052] In one embodiment, the Metaverse Surgical Planning 3D scene displays medical image data of multiple objects. Users can select the medical image data of a target object from the multiple objects according to actual needs, so as to construct a 3D model of the object corresponding to the medical image data in the Metaverse Surgical Planning 3D scene.

[0053] Regarding the selection method for the medical image data of the target object, in one implementation, identifiers corresponding to the medical image data of each object are displayed in the metaverse surgical planning 3D scene. The identifier selected by the virtual character is determined as the medical image data of the target object, where the identifier can be, but is not limited to, icons and / or text identifiers. In another implementation, a virtual administrator is created in the metaverse surgical planning 3D scene. The virtual administrator can interact with the virtual character, and the interaction method can include, but is not limited to, voice interaction. The virtual character informs the virtual administrator of relevant information about the target object, such as at least one of name, number, age, and gender, and the virtual administrator retrieves the corresponding medical image data from the database.

[0054] The surgical planning module 103 is used to plan the surgery for the target object based on the virtual motion signals of the three-dimensional model of the object manipulated by the virtual character.

[0055] The virtual character interacts with the user's actions. The user can generate virtual action signals for the virtual character to manipulate the 3D model of the object through gestures, thereby realizing surgical planning for the target object. The operation is convenient and can improve planning efficiency.

[0056] In one embodiment, see Figure 1c The scene creation module 101 includes a virtual character creation unit 1011. In response to the surgical planning request, the virtual character creation unit 1011 creates a virtual character in the metaverse surgical planning three-dimensional scene that is linked with the user's actions.

[0057] The first user performs corresponding operations by wearing a device capable of connecting to the internet and accessing a browser (such as VR glasses or a VR headset). This includes using voice commands, entering a relevant URL in the browser, or launching a surgical planning app. The scene creation module 101 displays the metaverse's scene homepage, which provides user login and registration functions. Surgical planning requests can be generated based on the first user's login request or upon successful verification of the first user's account and / or password. Once the user completes login, the metaverse surgical planning 3D scene is created, and a virtual character that interacts with the user's actions is established within that scene.

[0058] In one embodiment, the virtual character creation unit 1011 is used to create a virtual character in the metaverse surgical planning 3D scene that is linked to the user actions of the second user based on the invitation credential provided by the second user, the invitation credential being generated based on the invitation request of the first user.

[0059] A first user who has already created a linked virtual character in the Metaverse Surgical Planning 3D scene can invite other users (second users) to enter the Metaverse Surgical Planning 3D scene. The first user can, but is not limited to, triggering an invitation request through the virtual character's actions to generate an invitation control in the Metaverse Surgical Planning 3D scene, or providing the second user's account to generate an invitation request. The Metaverse Surgical System will then generate an invitation credential based on the invitation request and send the invitation credential to the second user. The invitation credential can be represented by, but is not limited to, a link URL, a QR code, a verification code, etc.

[0060] When a second user logs into the Metaverse Surgery System after receiving an invitation credential, the system can display a prompt asking whether to accept the invitation. If the second user chooses to accept the invitation, a virtual character that interacts with the second user's actions will be constructed in the Metaverse Surgery Planning 3D scene created by the first user. If the second user chooses not to accept the invitation, a Metaverse Surgery Planning 3D scene will be created for the second user, and a virtual character that interacts with the second user's actions will be constructed in the Metaverse Surgery Planning 3D scene.

[0061] In one embodiment, when a first user invites a second user to enter the Metaverse Surgical Planning 3D scene, operation permissions can also be set for the second user. Specifically, the Metaverse Surgical Planning 3D scene creation system also includes a permission setting module 104. The first user generates a permission authorization request by controlling a virtual character linked to their actions through gestures. The permission setting module 104 responds to the first user's permission authorization request and sets user permissions corresponding to the permission authorization request for the second user. User permissions include access to the planning sub-scene, viewing permissions, planning operation permissions, and interaction permissions with the Metaverse Surgical System.

[0062] The second user's permissions can be the same as or different from the first user's. The second user's permissions can be configured by the first user through the permission settings module 104, or they can be configured automatically by the system.

[0063] In one embodiment, the Metaverse Surgical Planning 3D Scene Creation System further includes a permission modification module 105. The permission modification module 105 responds to a permission modification request from a first user and modifies the user permissions of a second user according to the request. During the surgical planning process, the first user can modify the permissions of the second user according to actual needs.

[0064] In one embodiment, the scene creation module 101 further includes a sub-scene creation unit 1012, which is used to determine user permissions and cause the virtual character to enter a target planning sub-scene that matches the user permissions.

[0065] The planning sub-scenario includes at least one of the following: raw data warehouse sub-scenario, 3D model building sub-scenario, surgical planning sub-scenario, prosthesis warehouse sub-scenario, and simulation testing sub-scenario.

[0066] The sub-scene creation unit 1012 determines the target planning sub-scene that matches the user's permissions from the above-mentioned planning sub-scenes, and causes the user's virtual character to enter the determined target planning sub-scene.

[0067] In one embodiment, the scene creation module 101 responds to a scene switching instruction from a virtual character, causing the virtual character to enter another target planning sub-scene corresponding to the scene switching instruction.

[0068] For example, in planning a joint replacement surgery, assuming the planning process includes medical image data selection, 3D model creation, surgical planning, prosthesis selection, and simulation testing, see the corresponding documentation. Figure 3a The metaverse surgical planning 3D scene includes a raw data warehouse sub-scene, a 3D model building sub-scene, a surgical planning sub-scene, a prosthesis warehouse sub-scene, and a simulation testing sub-scene. Users can issue scene switching commands according to actual needs. The scene building module 101 determines the target planning sub-scene from multiple planning sub-scenes based on the scene switching command and displays it. At the same time, the user's virtual character enters the target planning sub-scene.

[0069] In one embodiment, the scene establishment module 101 sequentially determines the target planning sub-scene from multiple planning sub-scenes according to the order of planning process nodes, and when the target event is triggered, the virtual character enters the target planning sub-scene.

[0070] For example, suppose the order of the planning process nodes is: medical image data selection node → 3D model establishment node → surgical planning node → prosthesis selection node → simulation test node. If the target planning sub-scene is the 3D model establishment sub-scene used to realize the 3D model establishment node, when the target event is triggered, the scene establishment module 101 determines that the next process node of the 3D model establishment node is the surgical planning node. Then, the surgical planning sub-scene used to realize the surgical planning node is determined as the target planning sub-scene and displayed.

[0071] Among them, the target events can be set according to actual needs, such as the display duration of the target planning sub-scene reaching the duration threshold, or the surgical planning task in the current target planning sub-scene being determined to be completed, or the user receiving a scene switching instruction.

[0072] The primary storage area in the raw data warehouse sub-scene is the patient's medical image data. When a user enters the raw data warehouse sub-scene by controlling a virtual character through gestures, they can see the medical image data of multiple objects displayed in the raw data warehouse sub-scene. The user can select the medical image data of the target object and drag it into the 3D model to create a sub-scene.

[0073] The 3D model creation sub-scene primarily enables the 3D reconstruction of medical imaging data. It can, but is not limited to, presenting the reconstructed 3D model as a page-based display within the Metaverse surgical planning 3D scene. Users can also control a virtual character via gestures to select measurement tools such as lines, angles, and circles from the 3D model creation sub-scene to measure the 3D model and obtain relevant parameters. For example, in a femoral 3D model, relevant parameters include femoral length and eccentricity.

[0074] Users can also use gestures to control virtual characters to extract feature parameters from medical image data, generate a mesh model of the target object, assemble it into a Mesh view for display, and determine the key feature points of the mesh model.

[0075] The surgical planning sub-scene primarily provides surgical planning functions. Taking bone and joint replacement surgery planning as an example, after the user selects the planning type (hip joint planning or knee joint planning), the Metaverse surgical system automatically matches a suitable joint prosthesis based on the morphology and key features of the bone tissue, implants the target joint prosthesis into the object's 3D model, and registers and displays the object's 3D model after the target joint prosthesis is implanted. If the user finds the joint prosthesis installation unsuitable, they can manually select a suitable joint prosthesis for replacement or adjust the installation position of the joint prosthesis. During the planning and adjustment process, the Metaverse surgical planning 3D scene can also display various parameters of the current joint prosthesis position in real time, such as anteversion angle, abduction angle, coverage, and lower limb length difference.

[0076] The simulation test sub-scenario is used to simulate and test the surgical planning results. Taking the planning of a joint replacement surgery as an example, the simulation test can be used to see whether the movement of bone tissue after the joint prosthesis is implanted will cause collision.

[0077] It should be noted that the above sub-scene division is only an example. In actual applications, you can design each sub-scene according to your needs. For example, the 3D model creation sub-scene can be divided into the model segmentation sub-scene and the key point marking sub-scene.

[0078] The entire metaverse surgical planning 3D scene can also be constructed as a factory, see [link / reference]. Figure 3bThe factory comprises a main hall, three warehouses, and four workshops. Each warehouse or workshop corresponds to a planning sub-scenario. Users can complete an independent surgical planning task within each workshop. The workshops are unidirectionally connected; the planning process data output from the previous workshop (planning sub-scenario) serves as the input data for the next workshop. User virtual tasks and / or planning process data are sequentially transmitted between workshops via directional portals. For example, data from the image reading workshop can be directed to the original data warehouse, data from all workshops can be directed to the measurement tool library (i.e., the planning tool library), and data from the planning workshop can be directed to the prosthetic implant warehouse. This unidirectional connection between workshops ensures the sequential nature of the workflow, reduces the burden of user selection, and improves efficiency.

[0079] In one embodiment, each workshop (planning sub-scenario) shares planning process data. When a user adjusts the value of a planning parameter in one workshop, the corresponding planning parameter values ​​in other workshops will also be adjusted accordingly.

[0080] In one embodiment, each surgical planning sub-scene has a corresponding virtual entrance. Users can control a virtual character to enter the corresponding surgical planning sub-scene by using gestures, thus switching scenes. The virtual entrances displayed may differ for users with different user permissions, or the same virtual entrance may have different access permissions.

[0081] In one embodiment, after a user logs in, the surgical planning lobby is first displayed. Invited users (second users) are then invited to the surgical planning lobby to observe the overall process and status of the surgical planning. Upon further invitation to each room (or workshop), they can observe the surgical planning process up close and even participate in it. Each room has a door leading to the lobby. When a user in the surgical planning lobby receives an invitation from the primary user (first user and / or a second user who has been granted permission to further invite users), the door to the corresponding room is displayed or opened, allowing the user to enter the corresponding room; users who have not received an invitation cannot enter the corresponding room.

[0082] In one embodiment, the Metaverse Surgical System also provides a mode selection function, allowing the main user to choose between a workflow mode or a free mode. In workflow mode, the user must complete the work in the current workshop before proceeding to the next; in free mode, the user can freely choose to enter any room and perform any operation.

[0083] In one embodiment, the switching of planning sub-scenes in the metaverse surgical planning 3D scene can be achieved through a channel, which can be in the form of a door, corridor, teleportation point or teleportation vehicle, etc. Through the channel, one can enter each surgical planning sub-scene and realize scene switching.

[0084] In one embodiment, the 3D surgical planning scene in the metaverse can also display operation instructions to help users understand how to perform gestures to achieve their desired operational effects and complete the surgical planning. The operation instructions can be displayed through animation, text, voice, or a combination thereof; this embodiment of the invention does not impose any particular limitation on this method.

[0085] In one embodiment, surgical planning process data can also be saved. Specifically, users can control a virtual character to obtain a virtual memory card through gestures and store process data from the surgical planning process, such as 3D models and simulation test data, in the virtual memory card for later viewing and learning.

[0086] Figure 2 A flowchart illustrating a method for establishing a 3D scene for surgical planning in a metaverse, as provided in an exemplary embodiment of the present invention, is shown. This surgical planning method is applied to a metaverse surgical system and is suitable for preoperative and intraoperative planning of various surgeries. (See also...) Figure 2 The surgical planning method includes the following steps:

[0087] Step 201: Establish a metaverse surgical planning 3D scene, and establish a target planning sub-scene and a virtual character that interacts with the user's actions in the metaverse surgical planning 3D scene.

[0088] The surgical planning process includes multiple planning process nodes, each planning process node corresponds to a planning sub-scenario, and the target planning sub-scenario is the scene currently displayed to the user or the planning sub-scenario in which the user is currently located.

[0089] The Metaverse Surgical System supports the creation of multiple virtual characters in the 3D scene of Metaverse surgical planning, which can be linked with the actions of multiple users. It supports multi-person collaborative surgical planning, multi-angle observation of the surgical planning process, and provides users with an immersive surgical planning environment. Even if users participate remotely, they will feel as if they are there, enhancing the user experience.

[0090] In one embodiment, a virtual avatar that interacts with the user's actions is created based on the first user's surgical planning request. Specifically, in response to the surgical planning request, a virtual avatar that interacts with the first user's actions is created in the metaverse surgical planning 3D scene.

[0091] The first user, wearing a device capable of connecting to the internet and accessing a browser (such as VR glasses or a VR headset), performs corresponding operations, such as issuing voice commands, entering a relevant URL in the browser, or launching a surgical planning app. The Metaverse Surgical System then displays the Metaverse homepage, which provides user login and registration functionality. Surgical planning requests can be generated based on the first user's login request or upon successful verification of the first user's account and / or password. Once the user completes login, the Metaverse surgical planning 3D scene is created, and a virtual character that interacts with the user's actions is established within that scene.

[0092] In another implementation, based on the corresponding operation performed by the first user, the metaverse surgery system can directly create a metaverse surgery planning 3D scene. This metaverse surgery planning 3D scene can be a surgery planning scene lobby, where the first user can log in and register.

[0093] In one embodiment, user permissions can also be determined during the user login process. Within the metaverse surgical planning 3D scene, the areas a user can access, the tools they can obtain, and the types of operations they can perform using those tools are matched with their permissions. For example, suppose user A has permission to perform surgical planning operations. User A can enter the corresponding target planning sub-scene, obtain surgical planning tools, and use them to perform surgical planning. User B, however, only has permission to observe surgical planning and not to perform any planning operations. Therefore, user B can only enter the corresponding target planning sub-scene for observation but cannot obtain or operate the surgical planning tools.

[0094] In one embodiment, a virtual character is created based on an invitation credential. Specifically, a virtual character that interacts with the second user's actions is created in the metaverse surgical planning 3D scene based on the invitation credential provided by the second user. The invitation credential is generated based on the invitation request from the first user.

[0095] A first user who has already created a linked virtual character in the Metaverse Surgical Planning 3D scene can invite other users (second users) to enter the Metaverse Surgical Planning 3D scene. The first user can, but is not limited to, triggering an invitation request through the virtual character's actions to generate an invitation control in the Metaverse Surgical Planning 3D scene, or providing the second user's account to generate an invitation request. The Metaverse Surgical System will then generate an invitation credential based on the invitation request and send the invitation credential to the second user. The invitation credential can be represented by, but is not limited to, a link URL, a QR code, a verification code, etc.

[0096] When a second user logs into the Metaverse Surgery System after receiving an invitation credential, the system can display a prompt asking whether to accept the invitation. If the second user chooses to accept the invitation, a virtual character that interacts with the second user's actions will be constructed in the Metaverse Surgery Planning 3D scene created by the first user. If the second user chooses not to accept the invitation, a Metaverse Surgery Planning 3D scene will be created for the second user, and a virtual character that interacts with the second user's actions will be constructed in the Metaverse Surgery Planning 3D scene.

[0097] In one embodiment, when a first user invites a second user to enter the Metaverse Surgical Planning 3D scene, operation permissions can also be set for the second user. Specifically, the first user generates a permission authorization request by controlling a virtual character that is linked to their actions through gestures. In response to the first user's permission authorization request, the Metaverse Surgical System sets user permissions corresponding to the permission authorization request for the second user. These user permissions include permission to enter the planning sub-scene, viewing permission, planning operation permission, and interaction permission with the Metaverse Surgical System.

[0098] The second user's permissions can be the same as or different from the first user's. The second user's permissions can be configured by the first user or by the system. During surgical planning, the second user's permissions can also be modified based on a permission modification request initiated by the first user.

[0099] Step 202: Obtain the medical image data of the target object and construct a 3D model of the object corresponding to the medical image data in the metaverse surgical planning 3D scene.

[0100] In one embodiment, the Metaverse Surgical Planning 3D scene displays medical image data of multiple objects. The first user can select the medical image data of the target object from the multiple objects according to actual needs, so as to construct a 3D model of the object corresponding to the medical image data in the Metaverse Surgical Planning 3D scene.

[0101] Regarding the selection method for the medical image data of the target object, in one implementation, identifiers corresponding to the medical image data of each object are displayed in the metaverse surgical planning 3D scene. The identifier selected by the virtual character is determined as the medical image data of the target object, where the identifier can be, but is not limited to, icons and / or text identifiers. In another implementation, a virtual administrator is created in the metaverse surgical planning 3D scene. The virtual administrator can interact with the virtual character, and the interaction method can include, but is not limited to, voice interaction. The virtual character informs the virtual administrator of relevant information about the target object, such as at least one of name, number, age, and gender, and the virtual administrator retrieves the corresponding medical image data from the database.

[0102] Step 203: Based on the virtual motion signals of the virtual character manipulating the 3D model of the target object, realize the surgical planning for the target object.

[0103] The virtual character interacts with the user's actions. The user can generate virtual action signals for the virtual character to manipulate the 3D model of the object through gestures, thereby realizing surgical planning for the target object. The operation is convenient and can improve planning efficiency.

[0104] In one embodiment, the surgical planning process includes multiple planning process nodes, each planning process node corresponding to a planning sub-scene; the surgical planning method further includes: in response to the scene switching command of the virtual character, displaying the target planning sub-scene corresponding to the scene switching command in the metaverse surgical planning 3D scene.

[0105] For example, in planning a joint replacement surgery, assuming the planning process includes medical image data selection, 3D model creation, surgical planning, prosthesis selection, and simulation testing, see the corresponding documentation. Figure 3a The Metaverse surgical planning 3D scene includes a raw data warehouse sub-scene, a 3D model building sub-scene, a surgical planning sub-scene, a prosthesis warehouse sub-scene, and a simulation testing sub-scene. Users can issue scene switching commands according to actual needs, and the system will determine and display the target planning sub-scene from multiple planning sub-scenes based on the scene switching commands.

[0106] In one embodiment, the target planning sub-scenario is determined sequentially from multiple planning sub-scenarios according to the order of planning process nodes, and the target planning sub-scenario is displayed when the target event is triggered.

[0107] For example, suppose the order of the planning process nodes is: medical image data selection node → 3D model creation node → surgical planning node → prosthesis selection node → simulation test node. If the target planning sub-scene is the 3D model creation sub-scene used to realize the 3D model creation node, when the target event is triggered, the next process node of the 3D model creation node is determined to be the surgical planning node. Then, the surgical planning sub-scene used to realize the surgical planning node is determined as the target planning sub-scene and displayed.

[0108] Among them, the target events can be set according to actual needs, such as the display duration of the target planning sub-scene reaching the duration threshold, or the surgical planning task in the current target planning sub-scene being determined to be completed, or the user receiving a scene switching instruction.

[0109] The primary storage area in the raw data warehouse sub-scene is the patient's medical image data. When a user enters the raw data warehouse sub-scene by controlling a virtual character through gestures, they can see the medical image data of multiple objects displayed in the raw data warehouse sub-scene. The user can select the medical image data of the target object and drag it into the 3D model to create a sub-scene.

[0110] The 3D model creation sub-scene primarily enables the 3D reconstruction of medical imaging data. It can, but is not limited to, presenting the reconstructed 3D model as a page-based display within the Metaverse surgical planning 3D scene. Users can also control a virtual character via gestures to select measurement tools such as lines, angles, and circles from the 3D model creation sub-scene to measure the 3D model and obtain relevant parameters. For example, in a femoral 3D model, relevant parameters include femoral length and eccentricity.

[0111] Users can also use gestures to control virtual characters to extract feature parameters from medical image data, generate a mesh model of the target object, assemble it into a Mesh view for display, and determine the key feature points of the mesh model.

[0112] The surgical planning sub-scene primarily provides surgical planning functions. Taking bone and joint replacement surgery planning as an example, after the user selects the planning type (hip joint planning or knee joint planning), the Metaverse surgical system automatically matches a suitable joint prosthesis based on the morphology and key features of the bone tissue, implants the target joint prosthesis into the object's 3D model, and registers and displays the object's 3D model after the target joint prosthesis is implanted. If the user finds the joint prosthesis installation unsuitable, they can manually select a suitable joint prosthesis for replacement or adjust the installation position of the joint prosthesis. During the planning and adjustment process, the Metaverse surgical planning 3D scene can also display various parameters of the current joint prosthesis position in real time, such as anteversion angle, abduction angle, coverage, and lower limb length difference.

[0113] The simulation test sub-scenario is used to simulate and test the surgical planning results. Taking the planning of a joint replacement surgery as an example, the simulation test can be used to see whether the movement of bone tissue after the joint prosthesis is implanted will cause collision.

[0114] It should be noted that the above sub-scene division is only an example. In actual applications, you can design each sub-scene according to your needs. For example, the 3D model creation sub-scene can be divided into the model segmentation sub-scene and the key point marking sub-scene.

[0115] The entire metaverse surgical planning 3D scene can also be constructed as a factory, see [link / reference]. Figure 3bThe factory comprises a main hall, three warehouses, and four workshops. Each warehouse or workshop corresponds to a planning sub-scenario. Users can complete an independent surgical planning task within each workshop. The workshops are unidirectionally connected; the planning process data output from the previous workshop (planning sub-scenario) serves as the input data for the next workshop. User virtual tasks and / or planning process data are sequentially transmitted between workshops via directional portals. For example, data from the image reading workshop can be directed to the original data warehouse, data from all workshops can be directed to the measurement tool library (i.e., the planning tool library), and data from the planning workshop can be directed to the prosthetic implant warehouse. This unidirectional connection between workshops ensures the sequential nature of the workflow, reduces the burden of user selection, and improves efficiency.

[0116] In one embodiment, each workshop (planning sub-scenario) shares planning process data. When a user adjusts the value of a planning parameter in one workshop, the corresponding planning parameter values ​​in other workshops will also be adjusted accordingly.

[0117] In one embodiment, each surgical planning sub-scene has a corresponding virtual entrance. Users can control a virtual character to enter the corresponding surgical planning sub-scene by using gestures, thus switching scenes. The virtual entrances displayed may differ for users with different user permissions, or the same virtual entrance may have different access permissions.

[0118] In one embodiment, after a user logs in, the surgical planning lobby is first displayed. Invited users (second users) are then invited to the surgical planning lobby to observe the overall process and status of the surgical planning. Upon further invitation to each room (or workshop), they can observe the surgical planning process up close and even participate in it. Each room has a door leading to the lobby. When a user in the surgical planning lobby receives an invitation from the primary user (first user and / or a second user who has been granted permission to further invite users), the door to the corresponding room is displayed or opened, allowing the user to enter the corresponding room; users who have not received an invitation cannot enter the corresponding room.

[0119] In one embodiment, the Metaverse Surgical System also provides a mode selection function, allowing the main user to choose between a workflow mode or a free mode. In workflow mode, the user must complete the work in the current workshop before proceeding to the next; in free mode, the user can freely choose to enter any room and perform any operation.

[0120] In one embodiment, the switching of planning sub-scenes in the metaverse surgical planning 3D scene can be achieved through a channel, which can be in the form of a door, corridor, teleportation point or teleportation vehicle, etc. Through the channel, one can enter each surgical planning sub-scene and realize scene switching.

[0121] In one embodiment, the 3D surgical planning scene in the metaverse can also display operation instructions to help users understand how to perform gestures to achieve their desired operational effects and complete the surgical planning. The operation instructions can be displayed through animation, text, voice, or a combination thereof; this embodiment of the invention does not impose any particular limitation on this method.

[0122] In one embodiment, surgical planning process data can also be saved. Specifically, users can control a virtual character to obtain a virtual memory card through gestures and store process data from the surgical planning process, such as 3D models and simulation test data, in the virtual memory card for later viewing and learning.

[0123] The following section uses bone tissue as the target object for a bone and joint replacement surgery planning procedure as an example to further explain the virtual motion signals in the process of establishing a 3D scene for surgical planning in the metaverse. (See also...) Figure 4 The surgical planning method includes the following steps:

[0124] Step 401: Create a virtual character that interacts with the user's actions in the 3D scene of the metaverse surgical planning.

[0125] The specific implementation of step 401 is similar to that of step 201, and will not be described in detail here.

[0126] Step 402: Generate a bone tissue mesh model corresponding to the three-dimensional bone tissue model, and determine the key feature points of the bone tissue mesh model.

[0127] A three-dimensional model of bone tissue is also known as a three-dimensional model of an object.

[0128] In step 402, the user can control the virtual character to enter the 3D model creation sub-scene through gestures, select the medical image data of the patient for whom surgery is planned, and create a 3D bone tissue model corresponding to the medical image data in the 3D model creation sub-scene, generate a bone tissue mesh model corresponding to the 3D bone tissue model, and determine the key feature points of the bone tissue mesh model.

[0129] Users can control the virtual character to perform basic operations such as translation, rotation, scaling, and adjustment on the 3D model of bone tissue using gestures.

[0130] In one embodiment, the Metaverse Surgical System generates a bone tissue mesh model using a mesh generation model. Specifically, medical image data is input into a pre-trained mesh generation model, and a bone tissue mesh model corresponding to a three-dimensional bone tissue model is generated based on the mesh generation model. The mesh generation model is obtained by training a neural network using training samples.

[0131] In one embodiment, the Metaverse Surgical System determines the key features of a bone tissue mesh model using a key point determination model. Specifically, the bone tissue mesh model data is input into a pre-trained key point determination model, and the key features of the bone tissue mesh model are determined based on the key point determination model. The key point determination model is obtained by training a neural network using training samples.

[0132] Step 403: Determine candidate joint prostheses that match the three-dimensional model of bone tissue based on key feature points.

[0133] In one embodiment, the Metaverse Surgical System matches candidate joint prostheses from a joint prosthesis library based on key feature points, identifying those that match a 3D model of bone tissue. These candidate prostheses are then displayed to the user for selection. This automatic initial screening eliminates obviously unsuitable prostheses, reducing the range of options available to the user and improving efficiency.

[0134] Step 404: Select the target joint prosthesis from the candidate joint prostheses based on the virtual motion signal.

[0135] Users can select the target joint prosthesis from the candidate joint prostheses by controlling the virtual character through gestures.

[0136] Step 405: Implant the target joint prosthesis into the three-dimensional model of bone tissue, and register and display the three-dimensional model of bone tissue after implantation of the target joint prosthesis.

[0137] Users can also control the virtual character to adjust the position of the target joint prosthesis based on the 3D model of the implanted bone tissue through gestures.

[0138] Step 406: Perform motion simulation on the three-dimensional model of bone tissue after implantation of the target joint prosthesis to determine whether the various bone tissue sub-models contained in the three-dimensional model of bone tissue will collide.

[0139] Users can enter the simulated test sub-scene to simulate the motion of the 3D bone tissue model, determining whether the various bone tissue sub-models within the 3D bone tissue model will collide, thus allowing for early identification of the planning effect. If the planning effect is not ideal, users can use gestures to control the virtual character to reselect the target joint prosthesis and implant it into the 3D bone tissue model.

[0140] Users can also use gestures to control the virtual character to adjust the size, shape and other attribute parameters of the target joint prosthesis, so as to design the required joint prosthesis themselves when the joint prosthesis in the joint prosthesis library does not meet the requirements.

[0141] In one embodiment, the physiological parameters of the object's 3D model after implantation of the target joint prosthesis can also be displayed in the metaverse surgical planning 3D scene; the physiological parameters include at least one of the following: the anteversion angle of the target joint prosthesis, the abduction angle of the target joint prosthesis, the coverage of the target joint prosthesis, the lower limb length difference, and the eccentricity, so that the user can understand the surgical planning situation in real time.

[0142] Figure 5 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present invention, showing a block diagram of an exemplary electronic device 60 suitable for implementing embodiments of the present invention. Figure 5 The electronic device 60 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0143] like Figure 5 As shown, the electronic device 60 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0144] Bus 63 includes a data bus, an address bus, and a control bus.

[0145] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0146] The memory 62 may also include a program tool 625 (or utility) having a set (at least one) program module 624, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0147] The processor 61 performs various functional applications and data processing, such as the methods provided in any of the above embodiments, by running computer programs stored in the memory 62.

[0148] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 65. Furthermore, the model-generated electronic device 60 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of the model-generated electronic device 60 via bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0149] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0150] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0151] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0152] In a possible implementation, the present invention can also be implemented as a program product comprising program code, wherein when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the method implementing any of the above embodiments.

[0153] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0154] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A metaverse surgical planning 3D scene creation system, characterized in that, include: The scene creation module is used to create a metaverse surgical planning 3D scene, and to create a target planning sub-scene and a virtual character that interacts with the user's actions within the metaverse surgical planning 3D scene. The surgical planning process of the metaverse surgical planning three-dimensional scene includes multiple planning process nodes, and each planning process node corresponds to a planning sub-scene. The data acquisition module is used to acquire medical image data of the target object and construct a 3D model of the object corresponding to the medical image data in the target planning sub-scene; the target object includes bone tissue; The surgical planning module is used to plan the surgery for the target object based on the virtual action signals of the virtual character manipulating the three-dimensional model of the object. The planning sub-scenario includes a surgical planning sub-scenario and a simulation testing sub-scenario; The surgical planning sub-scene is used to match the target joint prosthesis according to the morphology and key features of the bone tissue, implant the target joint prosthesis into the object 3D model, and register and display the object 3D model after implantation of the target joint prosthesis. The simulation test sub-scenario is used to examine whether bone tissue movement after implantation of the target joint prosthesis will cause collisions through simulation testing.

2. The metaverse surgical planning 3D scene creation system according to claim 1, characterized in that, The scene creation module includes: The virtual character creation unit is used to create a virtual character that interacts with the user's actions in the metaverse surgical planning three-dimensional scene in response to a surgical planning request. And / or, the virtual character creation unit is used to create a virtual character in the three-dimensional scene of the metaverse surgical planning that is linked to the user actions of the second user, based on the invitation credential provided by the second user; the invitation credential is generated based on the invitation request of the first user.

3. The metaverse surgical planning 3D scene creation system according to claim 2, characterized in that, Also includes: The permission setting module is used to respond to the permission authorization request of the first user and set user permissions for the second user corresponding to the permission authorization request.

4. The metaverse surgical planning 3D scene creation system according to claim 1, characterized in that, The scene creation module includes: A switching unit is used to respond to a scene switching command of the virtual character and cause the virtual character to enter another target planning sub-scene corresponding to the scene switching command; Alternatively, the switching unit is used to sequentially determine the target planning sub-scene from multiple planning sub-scenes according to the order of planning process nodes, and when the target event is triggered, to cause the virtual character to enter the target planning sub-scene.

5. The metaverse surgical planning three-dimensional scene creation system according to claim 4, characterized in that, The planning process data of the previously shown target planning sub-scenario is used as the input data for the current target planning sub-scenario. Alternatively, planning process data can be shared across different planning sub-scenarios.

6. The metaverse surgical planning three-dimensional scene establishment system according to claim 1, characterized in that, The scene creation module also includes: The sub-scene creation unit is used to determine user permissions and cause the virtual character to enter the target planning sub-scene that matches the user permissions.

7. The metaverse surgical planning three-dimensional scene creation system according to claim 3, characterized in that, Also includes: The permission modification module is used to respond to the permission modification request of the first user and modify the user permissions of the second user according to the permission modification request.

8. The metaverse surgical planning three-dimensional scene establishment system according to claim 1, characterized in that, The planning sub-scenario also includes at least one of the following: The sub-scenes include: raw data warehouse, 3D model creation, and prosthetic warehouse.

9. The metaverse surgical planning three-dimensional scene establishment system according to claim 1, characterized in that, The metaverse surgical planning 3D scene also includes a surgical planning hall; The metaverse surgical planning 3D scene creation system also includes: The login module is used to complete user login in the surgical planning hall in response to login requests; And / or, a registration module, used to complete user registration in the surgical planning hall in response to a registration request.

10. A method for establishing a three-dimensional scene for metaverse surgical planning, characterized in that, include: Establish a metaverse surgical planning 3D scene, and in the metaverse surgical planning 3D scene, establish a target planning sub-scene and a virtual character that interacts with the user's actions; The surgical planning process includes multiple planning process nodes, and each planning process node corresponds to a planning sub-scenario; Acquire medical image data of the target object, and construct a 3D model of the object corresponding to the medical image data in the target planning sub-scene; the target object includes bone tissue; Based on the virtual action signals of the virtual character manipulating the three-dimensional model of the object, surgical planning for the target object is realized; Based on the virtual motion signals of the virtual character manipulating the 3D model of the object, surgical planning for the target object is realized, including: Determine the key features of the bone tissue mesh model, and then determine the candidate joint prosthesis that matches the three-dimensional bone tissue model based on the key features. Select the target joint prosthesis from the candidate joint prostheses based on virtual motion signals; The target joint prosthesis is implanted into a three-dimensional model of bone tissue, and the three-dimensional model of bone tissue after implantation of the target joint prosthesis is registered and displayed. Motion simulation was performed on a three-dimensional model of bone tissue after implantation of the target joint prosthesis to determine whether collisions would occur between the various bone tissue sub-models contained in the three-dimensional model of bone tissue.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for establishing a three-dimensional scene for metaverse surgical planning as described in claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for establishing a three-dimensional scene for metaverse surgical planning as described in claim 10.

Images