Three-dimensional visualization method for surgical interaction processes and head-mounted device

Abstract

The application is suitable for the medical technology field, and provides a three-dimensional visualization method for a surgical interaction process and a head-mounted device. The method comprises the following steps: a CPU in the head-mounted device determines a selected target virtual surgical object, a target virtual surgical tool and a target model deformation mode, and determines a target processing unit from the CPU and a GPU; the target processing unit performs face reduction preprocessing on a first original three-dimensional model of the target virtual surgical object and a second original three-dimensional model of the target virtual surgical tool respectively; a display module displays the obtained first target three-dimensional model and second target three-dimensional model, and when it is detected that the two models produce spatial overlap, a target deformation strategy corresponding to the target model deformation mode is adopted to perform deformation processing on an overlapping area on the first target three-dimensional model; and the display module displays the second target three-dimensional model and the deformed first target three-dimensional model, so that the fluency, authenticity and accuracy of the surgical interaction process can be considered.

Description

Technical Field

[0001] This application belongs to the field of medical technology, and in particular relates to a three-dimensional visualization method and head-mounted device for surgical interaction process. Background Technology

[0002] Currently, with the continuous development of digital medical technology, 3D visualization technology has been widely used in surgical planning, surgical navigation, and surgical simulation teaching. Surgical interaction systems based on 3D visualization technology are generally desktop platforms (such as computer platforms), requiring doctors to constantly monitor the desktop display screen during surgery to obtain information such as the patient's anatomical structures, lesion locations, or simulation feedback. This means the doctor's gaze needs to frequently switch between the surgical area and the desktop display screen, which not only distracts the doctor but also reduces the accuracy and efficiency of the surgical procedure.

[0003] To address the issue of surgeons' line-of-sight separation during surgery, head-mounted devices have been introduced into the medical field. Specifically, by loading 3D models of the surgical object and surgical tools into the head-mounted device, and updating the interaction between these two models in real time based on the actual interaction between the object and tools, surgeons can intuitively obtain surgical information without switching their gaze. However, head-mounted devices have limited computing power. Running high-precision model interaction algorithms directly within the device can cause lag; conversely, running relatively simple model interaction algorithms to improve smoothness reduces the realism and accuracy of the surgical interaction. Summary of the Invention

[0004] In view of this, embodiments of this application provide a three-dimensional visualization method and head-mounted device for surgical interaction processes, in order to solve the technical problem that traditional surgical interaction algorithms running on head-mounted devices cannot simultaneously ensure the smoothness, realism, and accuracy of the surgical interaction process.

[0005] In a first aspect, embodiments of this application provide a three-dimensional visualization method for a surgical interaction process, applied to a head-mounted device, the head-mounted device including a CPU and a GPU, the method comprising:

[0006] The CPU responds to selection operations for virtual surgical objects, virtual surgical tools, and model deformation modes, respectively, and determines the selected target virtual surgical object, target virtual surgical tool, and target model deformation mode; the target model deformation mode is either a precise deformation mode or a fast deformation mode.

[0007] The CPU obtains the total number of surface vertices of the three-dimensional model of the target virtual surgical object, and determines the target processing unit to be processed from the CPU and the GPU based on the relationship between the total number of surface vertices and a preset threshold.

[0008] The target processing unit acquires a first original 3D model of the target virtual surgical object and a second original 3D model of the target virtual surgical tool, and performs surface reduction preprocessing on the first original 3D model and the second original 3D model respectively to obtain a first target 3D model of the target virtual surgical object and a second target 3D model of the target virtual surgical tool, so that the display module of the head-mounted device displays the first target 3D model and the second target 3D model;

[0009] The target processing unit acquires in real time the first pose data of the real surgical object corresponding to the first target 3D model and the second pose data of the real surgical tool corresponding to the second target 3D model, and adjusts the spatial pose of the first target 3D model and / or the second target 3D model in real time based on the first pose data and the second pose data.

[0010] When the target processing unit detects that the second target 3D model and the first target 3D model overlap spatially, it adopts a target deformation strategy corresponding to the deformation mode of the target model to deform the overlapping area on the first target 3D model, so that the display module displays the second target 3D model and the deformed first target 3D model.

[0011] In one optional implementation of the first aspect, a target deformation strategy corresponding to the deformation mode of the target model is used to deform the overlapping regions on the first target 3D model, including:

[0012] When the deformation mode of the target model is the precise deformation mode, the target surface vertices in the overlapping area that need to be updated in position are determined according to the type of the target virtual surgical tool;

[0013] If the target virtual surgical tool is a drilling tool, then the center point of all the target surface vertices and the drilling direction corresponding to the second target 3D model are determined, and the displacement weight of each target surface vertex is determined according to the spatial distribution relationship of each target surface vertex relative to the center point, and the target position of each target surface vertex is determined according to the drilling direction and the displacement weight of each target surface vertex.

[0014] If the target virtual surgical tool is a grinding tool, then the grinding plane and grinding axis corresponding to the second target 3D model are determined, and each target surface vertex is projected along the grinding axis onto the grinding plane to determine the target position of each target surface vertex; all target positions are within the axis-aligned bounding box corresponding to the first target 3D model;

[0015] Each of the target surface vertices is moved to its respective target position.

[0016] In one alternative implementation of the first aspect, determining the target surface vertices in the overlapping region that require position updates based on the type of the target virtual surgical tool includes:

[0017] When the target virtual surgical tool is a drilling tool, the target surface vertices that need to be updated are determined based on the target vectors from each surface vertex in the overlapping region to the reference point of the second target 3D model and the radial distance between each surface vertex and the axis of the second target 3D model.

[0018] When the target virtual surgical tool is a grinding tool, the target surface vertices are determined based on the axial coordinates of each surface vertex in the overlapping region on the grinding axis corresponding to the second target 3D model and the projection coordinates of each surface vertex on the target plane perpendicular to the grinding axis.

[0019] In one optional implementation of the first aspect, determining the target surface vertices requiring position updates based on the target vectors from each surface vertex in the overlapping region to the reference point of the second target 3D model and the radial distances between each surface vertex and the axis of the second target 3D model includes:

[0020] For each surface vertex in the overlapping region, the target vector from the surface vertex to the reference point of the second target 3D model is projected onto the drilling direction corresponding to the second target 3D model to obtain the axial height value corresponding to the surface vertex, and the axial height value is within [0, ... L Within the range of 1], the surface vertex is determined as a candidate surface vertex; L 1 represents the current effective borehole length of the second target 3D model;

[0021] For each candidate surface vertex, if the radial distance between the candidate surface vertex and the axis of the second target 3D model is less than or equal to a first difference, then the candidate surface vertex is determined as the target surface vertex that needs to be updated in position; the first difference is the difference between the radius of the second target 3D model and the preset surface edge distance.

[0022] In an optional implementation of the first aspect, determining the target surface vertices based on the axial coordinates of each surface vertex in the overlapping region on the grinding axis corresponding to the second target 3D model and the projected coordinates of each surface vertex on a target plane perpendicular to the grinding axis includes:

[0023] For each surface vertex in the overlapping region, if the surface vertex is determined to be in front of the grinding direction of the second target three-dimensional model based on the axial coordinate of the surface vertex on the grinding axis corresponding to the second target three-dimensional model, then the surface vertex is determined as a candidate surface vertex.

[0024] For each candidate surface vertex, the projected coordinates of the candidate surface vertex on the target plane perpendicular to the grinding axis are calculated, and if the projected coordinates are within the preset cross-sectional boundary range corresponding to the second target three-dimensional model, the candidate surface vertex is determined as the target surface vertex.

[0025] In one optional implementation of the first aspect, the displacement weights of each of the target surface vertices are determined based on their spatial distribution relative to the center point, including:

[0026] For each vertex of the target surface, the target distance between the vertex and the center point is calculated, and the displacement weight of the vertex is calculated based on the target distance; the displacement weight is negatively correlated with the target distance.

[0027] In one optional implementation of the first aspect, a target deformation strategy corresponding to the deformation mode of the target model is used to deform the overlapping regions on the first target 3D model, including:

[0028] When the deformation mode of the target model is the rapid deformation mode, if the target virtual surgical tool is a drilling tool, then according to the current grayscale value of each target pixel corresponding to each surface vertex in the overlapping area in the two-dimensional displacement texture map corresponding to the first original three-dimensional model, the target displacement value of each surface vertex is determined respectively, and the three-dimensional coordinates of the target position of each surface vertex are determined respectively according to the target displacement value of each surface vertex.

[0029] When the deformation mode of the target model is the rapid deformation mode, if the target virtual surgical tool is a grinding tool, then based on the current displacement scalar of the second target three-dimensional model in the grinding direction, the displacement vector of each surface vertex in the overlapping area on the target coordinate axis corresponding to the grinding direction is determined, and the three-dimensional coordinates of the original position of each surface vertex are updated according to the displacement vector to obtain the three-dimensional coordinates of the target position of each surface vertex.

[0030] Each of the surface vertices is moved to its respective target position.

[0031] In one optional implementation of the first aspect, the target displacement value of each surface vertex is determined based on the current grayscale value of each target pixel corresponding to each surface vertex in the overlapping region in the two-dimensional displacement texture map corresponding to the first original three-dimensional model, including:

[0032] For each surface vertex in the overlapping region, if the current grayscale value of the target pixel corresponding to the surface vertex in the two-dimensional displacement texture map is a preset grayscale value, then the target displacement value of the surface vertex is determined to be 0.

[0033] For each surface vertex in the overlapping region, if the current gray value of the target pixel corresponding to the surface vertex is not the preset gray value, the current gray value of the target pixel is normalized to obtain a normalized displacement coefficient, and the target displacement value of the surface vertex is calculated based on the normalized displacement coefficient and the preset maximum deformation depth of the target virtual surgical object.

[0034] In one optional implementation of the first aspect, determining the three-dimensional coordinates of the target position of each of the surface vertices based on the target displacement value of each surface vertex includes:

[0035] Determine the current normal vector of the overlapping region;

[0036] For each of the surface vertices, based on the three-dimensional coordinates of the vertex's original position, the vertex's target displacement value, and the current normal vector, the three-dimensional coordinates of the vertex's target position are calculated using the following formula:

[0037] P new = P old +mul( N , d );

[0038] in, P newThe three-dimensional coordinates of the target position of the surface vertex. P old Let be the three-dimensional coordinates of the original position of the surface vertex, and mul(·) be used to represent the multiplication operation. N For the current normal vector, d The target displacement value is given.

[0039] Secondly, embodiments of this application provide a head-mounted device, including a memory and a computer program stored in the memory and executable on a processor. When the processor executes the computer program, it implements the steps corresponding to the head-mounted device in any optional implementation of the first aspect described above.

[0040] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the optional implementations of the first aspect above.

[0041] Fourthly, embodiments of this application provide a computer program product that, when run on a computer device, enables the computer device to implement the method described in any optional implementation of the first aspect.

[0042] The three-dimensional visualization method, head-mounted device, computer-readable storage medium, and computer program product for surgical interaction provided in the embodiments of this application have the following beneficial effects:

[0043] The 3D visualization method for surgical interaction provided in this application involves the CPU in a head-mounted device determining the target processing unit based on the relationship between the total number of surface vertices of the 3D model of the target virtual surgical object and a preset threshold number. The target processing unit then performs subsequent steps such as model reduction preprocessing and model deformation. This method fully leverages the hardware advantages of different processing units within the head-mounted device, achieving a dynamic and balanced distribution of computational load. The target processing unit performs reduction preprocessing on the first original 3D model of the target virtual surgical object and the second original 3D model of the target virtual surgical tool. Using the first and second target 3D models obtained from the reduction preprocessing as a reference, it performs model deformation processing. This reduces the massive amount of data brought by high-precision models at the data source, thereby reducing the computational load on the target processing unit during model deformation and effectively overcoming the hardware bottleneck of limited computing power in the head-mounted device. Furthermore, by providing fast deformation modes and precise deformation modes, and when spatial overlap is detected between the second target 3D model and the first target 3D model, a target deformation strategy corresponding to the user-selected target model deformation mode is used to deform the overlapping area on the first target 3D model. This allows for adaptation to different user needs regarding model deformation effects (i.e., the 3D visualization effect of model interaction). Specifically, by providing a fast deformation mode, high frame rates and smoothness can be maintained using a low-computational-cost target deformation strategy in scenarios where the user's requirements for the realism of model interaction effects are not high. By providing a precise deformation mode, a high-precision target deformation strategy can be used to restore the realism and accuracy of model interaction during surgery in scenarios where the user is performing delicate surgical operations. Therefore, the method provided in this application embodiment can balance the smoothness, realism, and accuracy of the surgical interaction process. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the structure of a head-mounted device provided in an embodiment of this application;

[0046] Figure 2 A schematic flowchart illustrating a three-dimensional visualization method for a surgical interaction process provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating another implementation process of S25 in a three-dimensional visualization method for a surgical interaction process provided in an embodiment of this application.

[0048] Figure 4 This is a schematic diagram illustrating an implementation process of S25 in a three-dimensional visualization method for a surgical interaction process provided in another embodiment of this application. Detailed Implementation

[0049] The following embodiments are only used to illustrate the technical solutions of this application more clearly, and are therefore only examples and should not be used to limit the scope of protection of this application.

[0050] In the description of the embodiments of this application, the technical terms "comprising," "including," "having," and any variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the description of the embodiments of this application, unless otherwise stated, the technical term "multiple" refers to two or more, and the technical terms "at least one" or "one or more" refer to one, two, or more than two. The technical terms "first," "second," etc., are only used to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary / secondary relationship of the indicated technical features. The technical term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0051] This application first provides a head-mounted device. Exemplarily, the head-mounted device can be smart glasses, a mixed reality (MR) device, a virtual reality (VR) device, or an augmented reality (AR) device, etc. This application does not limit the specific type of head-mounted device. Figure 1 This is a schematic diagram of the structure of a head-mounted device provided in an embodiment of this application. Figure 1 As shown, the head-mounted device 10 may include a processor 101, a memory 102, and a computer program 103 stored in the memory 102 and executable on the processor 101. The processor 101 executes the computer program 103 to implement the various steps in the subsequent method embodiments.

[0052] Optionally, the processor 101 may include a central processing unit (CPU) and a graphics processing unit (GPU), etc. For example, the CPU may also be other general-purpose processors (such as microprocessors), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0053] Optionally, the memory 102 can be an internal storage unit of the head-mounted device 10, such as a hard disk or memory of the head-mounted device 10. The memory 102 can also be an external storage device of the head-mounted device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the head-mounted device 10. Furthermore, the memory 102 can include both internal and external storage units of the head-mounted device 10. The memory 102 is used to store computer programs and other programs and data required by the head-mounted device 10. The memory 102 can also be used to temporarily store data that has been output or will be output.

[0054] Optionally, the computer program 103 can be divided into one or more modules / units, which can be stored in the memory 102 and executed by the processor 101. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 103 in the head-mounted device 10.

[0055] Optionally, the head-mounted device 10 may also include a display module 104. The display module 104 may be connected to the processor 101. The display module 104 may be used to implement display functions, such as displaying a surgical tool selection interface, a surgical object selection interface, and a model deformation mode selection interface, and may also be used to display a three-dimensional model of the surgical object and / or a three-dimensional model of the surgical tool.

[0056] It should be understood that Figure 1This is merely an illustrative description of the structure of the head-mounted device 10 and does not constitute a limitation on the head-mounted device 10. In other embodiments, the head-mounted device 10 may also include more or fewer components than illustrated, or combine certain components, or include different components, etc.

[0057] This application also provides a three-dimensional visualization method for surgical interaction processes, which is applied to... Figure 1 The head-mounted device shown. Figure 2 This is a schematic flowchart illustrating a three-dimensional visualization method for a surgical interaction process provided in an embodiment of this application. Figure 2 As shown, the method may include the following steps S21 to S25:

[0058] S21, the CPU responds to the selection operations for the virtual surgical object, the virtual surgical tool, and the model deformation mode, respectively, and determines the selected target virtual surgical object, the target virtual surgical tool, and the target model deformation mode.

[0059] In practical applications, when a user uses a real surgical tool (such as a drilling tool or a grinding tool) to perform surgery on a real surgical object (such as any part of the human body), the real surgical tool and the real surgical object will physically interact, causing the real surgical object to deform. For example, when a user uses a drilling tool to drill into any bone in the human body, as the drill bit penetrates the bone, a local area of ​​the bone will undergo local deformation, thus forming a hole structure corresponding to the drilling path.

[0060] To improve the accuracy of surgical procedures while ensuring operational safety, users can utilize the head-mounted device provided in this application embodiment to assist in viewing the simulated interaction between surgical tools and surgical objects during surgery. Optionally, to adapt to various surgical scenarios, the head-mounted device can pre-store 3D models of one or more virtual surgical tools corresponding to real surgical tools, and 3D models of one or more virtual surgical objects corresponding to real surgical objects. Optionally, these 3D models can be pre-imported from other electronic devices (e.g., surgical navigation devices) into the head-mounted device. Furthermore, the head-mounted device can also be configured with a virtual surgical object selection interface and a virtual surgical tool selection interface, providing multiple virtual surgical objects and multiple virtual surgical tools for the user to select from, respectively.

[0061] Based on this, when using a head-mounted device for assisted surgery, users can first select a target virtual surgical object corresponding to the real surgical object and a target virtual surgical tool corresponding to the real surgical tool within the head-mounted device. Specifically, the user can input a first selection operation for the virtual surgical object and a second selection operation for the virtual surgical tool. The CPU in the head-mounted device can respond to the first and second selection operations respectively, determining the selected target virtual surgical object and target virtual surgical tool.

[0062] It should be understood that when a head-mounted device simulates the interaction between surgical tools and the surgical object, it needs to deform the 3D model of the target virtual surgical object. Optionally, to adapt to different user needs, this application embodiment can provide two model deformation modes: a precise deformation mode and a rapid deformation mode. Specifically, in the precise deformation mode, the head-mounted device can use a deformation strategy based on surface vertex displacement to deform the 3D model of the target virtual surgical object, thereby achieving precise deformation and improving the accuracy of model deformation. In the rapid deformation mode, the head-mounted device can use a deformation strategy based on 2D textures to deform the 3D model of the target virtual surgical object, thereby achieving rapid deformation and improving the efficiency of model deformation. It should be noted that the specific deformation process corresponding to the above two deformation strategies can be referred to the relevant descriptions in the subsequent embodiments, and will not be detailed here.

[0063] Optionally, the head-mounted device may also be equipped with a model deformation mode selection interface to provide the user with the aforementioned two model deformation modes. Based on this, when using the head-mounted device for assisted surgery, in addition to selecting the target virtual surgical object and the target virtual surgical tool, the user can also select their desired model deformation mode. Specifically, the user can input a third selection operation for the model deformation mode. The CPU can respond to the third selection operation and determine the selected target model deformation mode. The target model deformation mode can be either a precise deformation mode or a rapid deformation mode.

[0064] S22, the CPU obtains the total number of surface vertices of the three-dimensional model of the target virtual surgical object, and determines the target processing unit to be processed from the CPU and GPU based on the relationship between the total number of surface vertices and the preset number threshold.

[0065] Optionally, the CPU can retrieve the total number of surface vertices of the 3D model of the target virtual surgical object from the memory of the head-mounted device. It should be understood that this total number of surface vertices may specifically refer to the total number of surface vertices of the original 3D model of the target virtual surgical object stored in memory without any modifications.

[0066] The preset quantity threshold can be set according to actual needs, and this application embodiment does not limit it.

[0067] For example, the relationship between the total number of surface vertices and the preset threshold may specifically include: the total number of surface vertices of the three-dimensional model of the target virtual surgical object is greater than the preset threshold; and the total number of surface vertices of the three-dimensional model of the target virtual surgical object is less than or equal to the preset threshold.

[0068] Based on this, S22 may specifically include the following steps 1.1 to 1.2:

[0069] Step 1.1: If the total number of surface vertices of the 3D model of the target virtual surgical object is greater than a preset threshold, the GPU is determined as the target processing unit for data processing.

[0070] Step 1.2: If the total number of surface vertices of the 3D model of the target virtual surgical object is less than or equal to a preset threshold, the CPU is determined as the target processing unit.

[0071] It should be noted that steps 1.1 and 1.2 are mutually exclusive steps, meaning that the target processing unit will only choose one of these two steps to execute each time. In other words, the target processing unit will not execute step 1.2 if it executes step 1.1, and will not execute step 1.1 if it executes step 1.2.

[0072] This application embodiment determines the GPU as the target processing unit for executing subsequent model deformation steps when the total number of surface vertices of the 3D model of the target virtual surgical object exceeds a preset threshold. This fully utilizes the parallel computing power of the GPU, significantly shortens the model deformation time, and improves the real-time deformation and frame rate stability of complex models during interaction. In addition, by determining the CPU as the target processing unit for executing subsequent model deformation steps when the total number of surface vertices of the 3D model of the target virtual surgical object is less than or equal to a preset threshold, unnecessary low-level data transmission and scheduling overhead can be avoided while ensuring the accuracy of model deformation and the smoothness of interaction.

[0073] S23, the target processing unit acquires the first original three-dimensional model of the target virtual surgical object and the second original three-dimensional model of the target virtual surgical tool, and performs surface reduction preprocessing on the first original three-dimensional model and the second original three-dimensional model respectively to obtain the first target three-dimensional model of the target virtual surgical object and the second target three-dimensional model of the target virtual surgical tool, so that the display module of the head-mounted device displays the first target three-dimensional model and the second target three-dimensional model.

[0074] Optionally, both the first and second original 3D models can be pre-stored in the memory of the head-mounted device. Based on this, the target processing unit can first obtain the first original 3D model of the target virtual surgical object and the second original 3D model of the target virtual surgical tool stored in the memory, and then perform surface reduction preprocessing on the first and second original 3D models respectively to obtain the first target 3D model of the target virtual surgical object and the second target 3D model of the target virtual surgical tool.

[0075] Optionally, when the target processing unit is a CPU, the CPU can directly obtain the first original three-dimensional model of the target virtual surgical object and the second original three-dimensional model of the target virtual surgical tool from the memory, and perform surface reduction preprocessing on the first original three-dimensional model and the second original three-dimensional model respectively.

[0076] Optionally, when the target processing unit is a GPU, the CPU can send the acquired first and second original 3D models to the GPU, so that the GPU can perform polygon reduction preprocessing on the first and second original 3D models respectively. That is, the GPU can first obtain the first original 3D model of the target virtual surgical object and the second original 3D model of the target virtual surgical tool from the CPU, and then perform polygon reduction preprocessing on the first and second original 3D models respectively.

[0077] For example, the target processing unit can use a model reduction algorithm to perform reduction preprocessing on an original 3D model and a second original 3D model. The model reduction algorithm may include, for example, an edge collapse algorithm, a vertex clustering algorithm, or a vertex deletion algorithm, etc., and the specific type is not limited in the embodiments of this application.

[0078] Optionally, after the target processing unit obtains the first target 3D model and the second target 3D model, it can render the first target 3D model and the second target 3D model through the graphics rendering module in the GPU so that the display module of the head-mounted device can display the first target 3D model and the second target 3D model.

[0079] The embodiments of this application first perform surface reduction preprocessing on the first original 3D model and the second original 3D model, and then apply the obtained first target 3D model and the second 3D model to subsequent model deformation and rendering steps, which can match the relatively weak computing and rendering capabilities of head-mounted devices.

[0080] S24, the target processing unit acquires in real time the first pose data of the real surgical object corresponding to the first target 3D model and the second pose data of the real surgical tool corresponding to the second target 3D model, and adjusts the spatial pose of the first target 3D model and / or the second target 3D model in real time based on the first pose data and the second pose data.

[0081] Optionally, the target processing unit can acquire in real time the first pose data of the real surgical object corresponding to the first target 3D model and the second pose data of the real surgical tool corresponding to the second target 3D model from the surgical navigation device. The first pose data may include the first position information and first pose information of the real surgical object in the world coordinate system. The second pose data may include the second position information and second pose information of the real surgical tool in the world coordinate system. For example, the first position information can be obtained by using the first 3D coordinates (or coordinates of the first reference point on the real surgical object, such as the centroid, geometric center, or anatomical feature point, etc.) in the world coordinate system. x 1, y 1, z 1) This indicates that the first pose information can be represented by a first rotation parameter (e.g., Euler angles) of the actual surgical object's own coordinate system relative to the world coordinate system. The second position information can be represented by the second three-dimensional coordinates (in world coordinates) of a second reference point on the actual surgical tool (e.g., the tool tip or the mounting point of the tracking target). x 2, y 2, z 2) indicates that the second pose information can be represented by the second rotation parameter of the actual surgical tool's own coordinate system relative to the world coordinate system.

[0082] Specifically, the target processing unit can respectively determine the first three-dimensional coordinates ( x 1, y 1, z 1) and second three-dimensional coordinates ( x 2, y 2, z 2) Construct a first translation matrix and a second translation matrix. A first rotation matrix and a second rotation matrix can be constructed based on the first rotation parameters, respectively. The first translation matrix and the first rotation matrix can be combined to obtain a first homogeneous transformation matrix describing the pose of the actual surgical object. The second translation matrix and the second rotation matrix can be combined to obtain a second homogeneous transformation matrix describing the pose of the actual surgical tool. Both the first and second homogeneous transformation matrices are affine transformation matrices of dimension 4×4 that include both translation and rotation components.

[0083] Optionally, the spatial pose of the first target 3D model may include its first spatial position and first spatial orientation in the head-mounted device coordinate system, and the spatial pose of the second target 3D model may include its second spatial position and second spatial orientation in the head-mounted device coordinate system.

[0084] Based on this, the target processing unit can translate and / or rotate the first target 3D model according to the translation and / or rotation components in the first homogeneous transformation matrix, respectively, to adjust the spatial pose of the first target 3D model in the head-mounted device coordinate system. Similarly, the target processing unit can translate and / or rotate the second target 3D model according to the translation and / or rotation components in the second homogeneous transformation matrix, respectively, to adjust the spatial pose of the second target 3D model in the head-mounted device coordinate system.

[0085] Optionally, each time the target processing unit adjusts the spatial pose of the first target 3D model and / or the second target 3D model, it can detect whether the second target 3D model and the first target 3D model spatially overlap. It should be noted that the specific process for detecting whether the two 3D models spatially overlap can be found in descriptions in related technologies, and will not be detailed in this embodiment.

[0086] Optionally, if the second target 3D model and the first target 3D model do not overlap spatially, the target processing unit may execute S24 cyclically. Optionally, if the second target 3D model and the first target 3D model overlap spatially, the target processing unit may execute S25.

[0087] S25, when the target processing unit detects that the second target 3D model and the first target 3D model overlap in space, it adopts a target deformation strategy corresponding to the deformation mode of the target model to deform the overlapping area on the first target 3D model so that the display module can display the second target 3D model and the deformed first target 3D model.

[0088] Optionally, when the target model deformation mode is a precise deformation mode, the target deformation strategy is a deformation strategy based on surface vertex displacement, which may specifically include, for example: Figure 3 As shown in S251~S254, that is, in S25, a target deformation strategy corresponding to the deformation mode of the target model is used to deform the overlapping area on the first target 3D model. Specifically, it can include S251~S254, which are detailed below:

[0089] S251, when the target model deformation mode is the precise deformation mode, determine the target surface vertices in the overlapping area that need to be updated in position according to the type of the target virtual surgical tool.

[0090] In this embodiment, the strategies for determining the target surface vertices differ for different types of virtual surgical tools. Therefore, step S251 may specifically include steps 2.1 to 2.2:

[0091] Step 2.1: When the target virtual surgical tool is a drilling tool, determine the target surface vertices that need to be updated in position based on the target vectors from each surface vertex in the overlapping area to the reference point of the second target 3D model and the radial distance between each surface vertex and the axis of the second target 3D model.

[0092] The aforementioned reference point can be the center point of the virtual drill bit base on the second target's 3D model. Based on this, the target vector from the surface vertex to the reference point can refer to the spatial vector pointing from the surface vertex to the reference point.

[0093] The aforementioned axis can refer to the central symmetry line parallel to the drilling direction (i.e., the feed direction of the second target 3D model) and passing through the aforementioned reference point. Based on this, the radial distance between the surface vertex and the aforementioned axis can refer to the shortest distance from the surface vertex to the aforementioned axis.

[0094] In one specific implementation, step 2.1 may include the following steps 2.11 to 2.12:

[0095] Step 2.11: For each surface vertex in the overlapping region, project the target vector from that surface vertex to the reference point of the second target 3D model onto the drilling direction corresponding to the second target 3D model to obtain the axial height value corresponding to that surface vertex, and determine the axial height value within the range [0, ...]. L If the condition is within the range of 1], then the surface vertex is determined as a candidate surface vertex.

[0096] The aforementioned axial height value is the projection length of the target vector from the surface vertex to the reference point in the drilling direction corresponding to the second target 3D model.

[0097] L 1 represents the current effective borehole length of the second target 3D model.

[0098] Step 2.12: For each candidate surface vertex, if the radial distance between the candidate surface vertex and the axis of the second target 3D model is less than or equal to the first difference, then the candidate surface vertex is determined as the target surface vertex that needs to be updated in position.

[0099] The radial distance between the candidate surface vertex and the axis of the second target 3D model is the shortest distance between the candidate surface vertex and the axis of the second target 3D model.

[0100] The first difference can be the difference between the radius of the second target 3D model and the preset surface margin. The preset surface margin can be a preset safety tolerance distance or a smooth transition distance, etc.

[0101] In this embodiment of the application, the axial height value corresponding to any surface vertex in the overlapping region is in the range [0, ...].L Within the range of 1], and when the radial distance between the surface vertex and the axis of the second target 3D model is less than or equal to the first difference, the surface vertex is determined as the target surface vertex. It can accurately lock the surface vertex on the first target 3D model that is actually affected by the drilling operation using the dual dimensions of axial and radial, thereby ensuring the visual smoothness of the hole wall while avoiding excessive deformation.

[0102] Step 2.2: When the target virtual surgical tool is a grinding tool, the target surface vertices are determined based on the axial coordinates of each surface vertex in the overlapping area on the grinding axis corresponding to the second target 3D model and the projection coordinates of each surface vertex on the target plane perpendicular to the grinding axis.

[0103] The axial coordinates of the surface vertex on the grinding axis corresponding to the second target 3D model can refer to the one-dimensional coordinates of the surface vertex in the direction of the grinding axis, which can be used to represent the relative depth of the surface vertex relative to the grinding plane in the grinding direction. The projected coordinates of the surface vertex on the target plane can refer to the two-dimensional coordinates obtained by orthogonally projecting the surface vertex onto the target plane along a direction parallel to the grinding direction, which can be used to represent the spatial distribution of the surface vertex on the target plane.

[0104] In one specific implementation, step 2.2 may include the following steps 2.21 to 2.22:

[0105] Step 2.21: For each surface vertex in the overlapping region, if the surface vertex is located in front of the grinding direction of the second target 3D model based on the axial coordinate of the surface vertex on the grinding axis corresponding to the second target 3D model, then the surface vertex is determined as a candidate surface vertex.

[0106] It should be understood that if a surface vertex is in front of the grinding direction of the second target 3D model, it means that the surface vertex is located on the moving path of the grinding surface and will be deformed by the grinding operation.

[0107] Step 2.22: For each candidate surface vertex, calculate the projected coordinates of the candidate surface vertex on the target plane perpendicular to the grinding axis, and determine the candidate surface vertex as the target surface vertex if the projected coordinates are within the preset cross-sectional boundary range corresponding to the second target 3D model.

[0108] The preset cross-sectional boundary range can refer to the two-dimensional projection contour of the effective working area of ​​the second target 3D model on the target plane. Based on this, if the projection coordinates of the candidate surface vertex on the target plane are within the preset cross-sectional boundary range, it means that the candidate surface vertex falls within the effective working area of ​​the second target 3D model on the target plane.

[0109] This application embodiment determines the surface vertex that needs to be updated in position when any surface vertex in the overlapping area is in front of the grinding direction of the second target 3D model and the projection coordinate of the surface vertex on the target plane is within the preset cross-sectional boundary range of the second target 3D model. This can accurately lock the surface vertex of the first target 3D model that is actually affected by the grinding operation, thereby improving the realism and accuracy of the grinding deformation of the first target 3D model during the surgical interaction.

[0110] S252, if the target virtual surgical tool is a drilling tool, then determine the center point of all target surface vertices and the drilling direction corresponding to the second target 3D model, and determine the displacement weight of each target surface vertex according to the spatial distribution relationship of each target surface vertex relative to the center point, and determine the target position of each target surface vertex according to the drilling direction and the displacement weight of each target surface vertex.

[0111] The center point of all target surface vertices is the centroid of all target surface vertices.

[0112] The drilling direction corresponding to the second target 3D model refers to the current feed direction of the second target 3D model.

[0113] The spatial distribution of the vertices on the target surface relative to the center point can be used to describe the orientation and distance of the vertices relative to the center point.

[0114] Optionally, in step S252, the displacement weight of each target surface vertex is determined according to the spatial distribution relationship of each target surface vertex relative to the center point. This may specifically include the following steps:

[0115] For each target surface vertex, calculate the target distance between the target surface vertex and the center point, and calculate the displacement weight of the target surface vertex based on the target distance.

[0116] For example, the target distance mentioned above can refer to Euclidean distance. Based on this, the target processing unit can calculate the target distance between the vertex and the center point of the target surface based on the Euclidean distance equation.

[0117] Displacement weights can be used to describe the degree of deformation of a target surface vertex caused by drilling operations. The displacement weight of a target surface vertex is negatively correlated with the target distance between the vertex and the center point. That is, the greater the target distance between the vertex and the center point, the smaller its displacement weight; the smaller the target distance between the vertex and the center point, the greater its displacement weight.

[0118] For example, the target processing unit can calculate the displacement weight of each target surface vertex using a linear decay function based on the target distance between each target surface vertex and the center point. The target processing unit can also calculate the displacement weight of each target surface vertex using other methods; however, this application embodiment does not limit the specific calculation method for the displacement weight of each target surface vertex.

[0119] S253, if the target virtual surgical tool is a grinding tool, then determine the grinding plane and grinding axis corresponding to the second target 3D model, and project each target surface vertex along the grinding axis onto the grinding plane to determine the target position of each target surface vertex respectively.

[0120] The grinding plane refers to the two-dimensional reference plane on the second target 3D model that corresponds to the actual physical grinding surface. The grinding axis refers to the spatial reference line that passes through the second target 3D model and is parallel to the current feed direction of the second target 3D model. It should be understood that the grinding plane is perpendicular to the grinding axis.

[0121] In this embodiment, the target positions of all target surface vertices are within the axis-aligned bounding box of the first target 3D model. This avoids the target surface vertices being excessively pushed outside the maximum physical boundary of the first target 3D model during deformation, thereby improving the stability and realism of the model interactive visualization.

[0122] It should be noted that S252 and S253 are mutually exclusive steps, meaning that the target processing unit will only choose one of these two steps to execute each time. In other words, the target processing unit will not execute S253 if it executes S252, and will not execute S252 if it executes S253.

[0123] S254 moves each target surface vertex to its respective target position.

[0124] Optionally, when the target model's deformation mode is a fast deformation mode, the target deformation strategy is a deformation strategy based on a 2D texture. This strategy may specifically include, for example: Figure 4 As shown in S255~S257, that is, in S25, a target deformation strategy corresponding to the deformation mode of the target model is used to deform the overlapping area on the first target 3D model. Specifically, it can include S255~S257, which are detailed below:

[0125] S255, when the target model deformation mode is fast deformation mode, if the target virtual surgical tool is a drilling tool, then according to the current grayscale value of each target pixel corresponding to each surface vertex in the two-dimensional displacement texture map corresponding to the first original three-dimensional model, the target displacement value of each surface vertex is determined respectively, and the three-dimensional coordinates of the target position of each surface vertex are determined respectively according to the target displacement value of each surface vertex.

[0126] In this embodiment, each pixel in the two-dimensional displacement texture map corresponds one-to-one with each surface vertex of the first original three-dimensional model. For ease of understanding, each pixel in the two-dimensional displacement texture map that corresponds one-to-one with each surface vertex in the overlapping area of ​​the first target three-dimensional model is referred to as a target pixel, that is, each target pixel in the two-dimensional displacement texture map corresponds one-to-one with each surface vertex in the overlapping area.

[0127] Optionally, the grayscale value of each pixel in the two-dimensional displacement texture map can be used to represent the displacement value of the corresponding surface vertex. Based on this, in S255, according to the current grayscale value of each target pixel corresponding to each surface vertex in the overlapping area of ​​the two-dimensional displacement texture map corresponding to the first original three-dimensional model, the target displacement value of each surface vertex is determined respectively, which may specifically include the following steps 3.1 and 3.2:

[0128] Step 3.1: For each surface vertex in the overlapping region, if the current grayscale value of the target pixel corresponding to the surface vertex in the two-dimensional displacement texture map is a preset grayscale value, then the target displacement value of the surface vertex is determined to be 0.

[0129] For example, the grayscale value of each pixel in the two-dimensional displacement texture map can be in the range of [0, 255]. Based on this, the preset grayscale value can be, for example, 0, which is the grayscale value corresponding to black.

[0130] Optionally, if the grayscale value of a pixel in a two-dimensional displacement texture map is 0, it can indicate that the displacement value of its corresponding surface vertex is 0 (i.e., the surface vertex is still in its original position and has not been displaced). In this case, the target processing unit can determine the target displacement value of the surface vertex as 0.

[0131] Step 3.2: For each surface vertex in the overlapping region, if the current gray value of the target pixel corresponding to the surface vertex is not a preset gray value, the current gray value of the target pixel is normalized to obtain a normalized displacement coefficient. Based on the normalized displacement coefficient and the preset maximum deformation depth of the target virtual surgical object, the target displacement value of the surface vertex is calculated.

[0132] Optionally, if the grayscale value of a pixel in the two-dimensional displacement texture map is 255, it can indicate that the displacement value of its corresponding surface vertex has reached the preset maximum deformation depth of the target virtual surgical object; if the grayscale value of a pixel in the two-dimensional displacement texture map is between 0 and 255, it can indicate that the displacement value of its corresponding surface vertex is between 0 and the preset maximum deformation depth.

[0133] The preset maximum deformation depth can be pre-set and can be determined based on the model material.

[0134] Based on this, if the current grayscale value of the target pixel corresponding to any surface vertex in the overlapping region is not a preset grayscale value, the target processing unit can normalize the current grayscale value to obtain the normalized displacement coefficient corresponding to that surface vertex. The normalized displacement coefficient is a dimensionless coefficient between 0 and 1, used to represent the relative proportion of surface vertex deformation or the degree of influence. For example, when the normalized displacement coefficient is 0, it can indicate that the surface vertex has not deformed (corresponding to black); when the normalized displacement coefficient is 1, it can indicate that the surface vertex has undergone the maximum allowable deformation (corresponding to white).

[0135] Optionally, for each surface vertex of the overlapping region, the target processing unit can determine its target displacement value by multiplying its corresponding normalized displacement coefficient by a preset maximum deformation depth.

[0136] It should be noted that steps 3.1 and 3.2 are mutually exclusive steps, meaning that the target processing unit will only choose one of these two steps to execute each time. In other words, the target processing unit will not execute step 3.2 if it executes step 3.1, and will not execute step 3.1 if it executes step 3.2.

[0137] Optionally, in step S255, the three-dimensional coordinates of the target position of each surface vertex are determined based on the target displacement value of each surface vertex. Specifically, this may include the following steps 3.3 to 3.4:

[0138] Step 3.3: Determine the current normal vector of the overlapping region.

[0139] It should be understood that the current normal vector of the overlapping region is the surface normal vector of the triangle corresponding to the second target 3D model in the overlapping region.

[0140] Step 3.4: For each surface vertex, based on the original 3D coordinates of the vertex's position, the target displacement value of the vertex, and the current normal vector, calculate the 3D coordinates of the vertex's target position using the following formula:

[0141] P new =P old +mul( N , d );

[0142] in, P new The three-dimensional coordinates of the target position at the vertex of this surface. P old Let represent the three-dimensional coordinates of the original position of the vertex on the surface, and mul(·) is used to represent the multiplication operation. N For the current normal vector, d This represents the target displacement value.

[0143] S256, when the target model deformation mode is rapid deformation mode, if the target virtual surgical tool is a grinding tool, then based on the current displacement scalar of the second target three-dimensional model in the grinding direction, determine the displacement vector of each surface vertex in the overlapping area on the target coordinate axis corresponding to the grinding direction, and update the three-dimensional coordinates of the original position of each surface vertex according to the displacement vector to obtain the three-dimensional coordinates of the target position of each surface vertex.

[0144] Optionally, the target processing unit can calculate the relative distance from each surface vertex to the grinding plane corresponding to the second target 3D model based on the 3D coordinates of the original position of each surface vertex in the overlapping region, and can determine each relative distance as the current displacement scalar of each surface vertex in the grinding direction.

[0145] Optionally, the target processing unit can perform scalar vector multiplication on the current displacement scalar of each surface vertex and the unit vector in the grinding direction, thereby obtaining the displacement vector of each surface vertex on the target coordinate axis corresponding to the grinding direction.

[0146] Optionally, for each surface vertex, the target processing unit can perform a vector addition operation between the three-dimensional coordinates of the original position of the surface vertex and the displacement vector of the surface vertex on the target coordinate axis to obtain the three-dimensional coordinates of the target position of the surface vertex.

[0147] It should be noted that S255 and S256 are mutually exclusive steps, meaning that the target processing unit will only choose one of these two steps to execute each time. In other words, the target processing unit will not execute S256 if it executes S255, and will not execute S255 if it executes S256.

[0148] S257 moves each surface vertex to its respective target position.

[0149] It should be noted that S251~S254 and S255~S256 are mutually exclusive processes, meaning that the target processing unit will only select one of these two processes to execute each time. In other words, the target processing unit will not execute S255~S256 when executing S251~S254, and will not execute S251~S254 when executing S255~S256.

[0150] As can be seen from the above, this embodiment determines the target processing unit by having the CPU in the head-mounted device determine the target processing unit based on the relationship between the total number of surface vertices of the target virtual surgical object's 3D model and a preset threshold number. The target processing unit then performs subsequent model reduction preprocessing and model deformation steps. This fully leverages the hardware advantages of different processing units in the head-mounted device, achieving a dynamic and balanced distribution of computational load. The target processing unit performs reduction preprocessing on the first original 3D model of the target virtual surgical object and the second original 3D model of the target virtual surgical tool. Using the first and second target 3D models obtained from the reduction preprocessing as a benchmark, it performs model deformation processing. This reduces the massive amount of data brought by high-precision models at the data source, thereby reducing the computational load on the target processing unit during the model deformation process and effectively overcoming the hardware bottleneck of limited computing power in the head-mounted device.

[0151] Furthermore, by providing fast deformation modes and precise deformation modes, and when spatial overlap is detected between the second target 3D model and the first target 3D model, a target deformation strategy corresponding to the user-selected target model deformation mode is used to deform the overlapping area on the first target 3D model. This allows for adaptation to different user needs regarding model deformation effects (i.e., the 3D visualization effect of model interaction). Specifically, by providing a fast deformation mode, high frame rates and smoothness can be maintained using a low-computational-cost target deformation strategy in scenarios where the user's requirements for the realism of model interaction effects are not high. By providing a precise deformation mode, a high-precision target deformation strategy can be used to restore the realism and accuracy of model interaction during surgery in scenarios where the user is performing delicate surgical operations. Therefore, the method provided in this application embodiment can balance the smoothness, realism, and accuracy of the surgical interaction process.

[0152] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0154] It should be noted that, unless otherwise specified, all technical terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The technical terms used in the embodiments of this application are only used to explain specific embodiments of this application and are not intended to limit this application.

[0155] The term "embodiment" as used in the description of embodiments in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0156] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A three-dimensional visualization method for surgical interaction processes, characterized in that, Applied to a head-mounted device, the head-mounted device including a CPU and a GPU, the method includes: The CPU responds to selection operations for virtual surgical objects, virtual surgical tools, and model deformation modes, respectively, and determines the selected target virtual surgical object, target virtual surgical tool, and target model deformation mode; the target model deformation mode is either a precise deformation mode or a fast deformation mode. The CPU obtains the total number of surface vertices of the three-dimensional model of the target virtual surgical object, and determines the target processing unit to be processed from the CPU and the GPU based on the relationship between the total number of surface vertices and a preset threshold. The target processing unit acquires a first original 3D model of the target virtual surgical object and a second original 3D model of the target virtual surgical tool, and performs surface reduction preprocessing on the first original 3D model and the second original 3D model respectively to obtain a first target 3D model of the target virtual surgical object and a second target 3D model of the target virtual surgical tool, so that the display module of the head-mounted device displays the first target 3D model and the second target 3D model; The target processing unit acquires in real time the first pose data of the real surgical object corresponding to the first target 3D model and the second pose data of the real surgical tool corresponding to the second target 3D model, and adjusts the spatial pose of the first target 3D model and / or the second target 3D model in real time based on the first pose data and the second pose data. When the target processing unit detects that the second target 3D model and the first target 3D model overlap spatially, it adopts a target deformation strategy corresponding to the deformation mode of the target model to deform the overlapping area on the first target 3D model, so that the display module displays the second target 3D model and the deformed first target 3D model.

2. The method according to claim 1, characterized in that, Using a target deformation strategy corresponding to the deformation pattern of the target model, deformation processing is performed on the overlapping regions of the first target 3D model, including: When the deformation mode of the target model is the precise deformation mode, the target surface vertices in the overlapping area that need to be updated in position are determined according to the type of the target virtual surgical tool; If the target virtual surgical tool is a drilling tool, then the center point of all the target surface vertices and the drilling direction corresponding to the second target 3D model are determined, and the displacement weight of each target surface vertex is determined according to the spatial distribution relationship of each target surface vertex relative to the center point, and the target position of each target surface vertex is determined according to the drilling direction and the displacement weight of each target surface vertex. If the target virtual surgical tool is a grinding tool, then the grinding plane and grinding axis corresponding to the second target 3D model are determined, and each target surface vertex is projected along the grinding axis onto the grinding plane to determine the target position of each target surface vertex; all target positions are within the axis-aligned bounding box corresponding to the first target 3D model; Each of the target surface vertices is moved to its respective target position.

3. The method according to claim 2, characterized in that, Determining the target surface vertices in the overlapping region that require position updates based on the type of the target virtual surgical tool includes: When the target virtual surgical tool is a drilling tool, the target surface vertices that need to be updated are determined based on the target vectors from each surface vertex in the overlapping region to the reference point of the second target 3D model and the radial distance between each surface vertex and the axis of the second target 3D model. When the target virtual surgical tool is a grinding tool, the target surface vertices are determined based on the axial coordinates of each surface vertex in the overlapping region on the grinding axis corresponding to the second target 3D model and the projection coordinates of each surface vertex on the target plane perpendicular to the grinding axis.

4. The method according to claim 3, characterized in that, Based on the target vectors from each surface vertex in the overlapping region to the reference point of the second target 3D model and the radial distances between each surface vertex and the axis of the second target 3D model, the target surface vertices that need to be updated in position are determined, including: For each surface vertex in the overlapping region, the target vector from the surface vertex to the reference point of the second target 3D model is projected onto the drilling direction corresponding to the second target 3D model to obtain the axial height value corresponding to the surface vertex, and the axial height value is within [0, ... L Within the range of 1], the surface vertex is determined as a candidate surface vertex; L 1 represents the current effective borehole length of the second target 3D model; For each candidate surface vertex, if the radial distance between the candidate surface vertex and the axis of the second target 3D model is less than or equal to a first difference, then the candidate surface vertex is determined as the target surface vertex that needs to be updated in position; the first difference is the difference between the radius of the second target 3D model and the preset surface edge distance.

5. The method according to claim 3, characterized in that, The target surface vertices are determined based on the axial coordinates of each surface vertex in the overlapping region on the grinding axis corresponding to the second target 3D model and the projected coordinates of each surface vertex on the target plane perpendicular to the grinding axis, including: For each surface vertex in the overlapping region, if the surface vertex is determined to be in front of the grinding direction of the second target three-dimensional model based on the axial coordinate of the surface vertex on the grinding axis corresponding to the second target three-dimensional model, then the surface vertex is determined as a candidate surface vertex. For each candidate surface vertex, the projected coordinates of the candidate surface vertex on the target plane perpendicular to the grinding axis are calculated, and if the projected coordinates are within the preset cross-sectional boundary range corresponding to the second target three-dimensional model, the candidate surface vertex is determined as the target surface vertex.

6. The method according to claim 2, characterized in that, Based on the spatial distribution relationship of each target surface vertex relative to the center point, the displacement weight of each target surface vertex is determined, including: For each vertex of the target surface, the target distance between the vertex and the center point is calculated, and the displacement weight of the vertex is calculated based on the target distance; the displacement weight is negatively correlated with the target distance.

7. The method according to any one of claims 1-6, characterized in that, Using a target deformation strategy corresponding to the deformation pattern of the target model, deformation processing is performed on the overlapping regions of the first target 3D model, including: When the deformation mode of the target model is the rapid deformation mode, if the target virtual surgical tool is a drilling tool, then according to the current grayscale value of each target pixel corresponding to each surface vertex in the overlapping area in the two-dimensional displacement texture map corresponding to the first original three-dimensional model, the target displacement value of each surface vertex is determined respectively, and the three-dimensional coordinates of the target position of each surface vertex are determined respectively according to the target displacement value of each surface vertex. When the deformation mode of the target model is the rapid deformation mode, if the target virtual surgical tool is a grinding tool, then based on the current displacement scalar of the second target three-dimensional model in the grinding direction, the displacement vector of each surface vertex in the overlapping area on the target coordinate axis corresponding to the grinding direction is determined, and the three-dimensional coordinates of the original position of each surface vertex are updated according to the displacement vector to obtain the three-dimensional coordinates of the target position of each surface vertex. Each of the surface vertices is moved to its respective target position.

8. The method according to claim 7, characterized in that, Based on the current grayscale value of each target pixel corresponding to each surface vertex in the overlapping region in the two-dimensional displacement texture map corresponding to the first original three-dimensional model, the target displacement value of each surface vertex is determined, including: For each surface vertex in the overlapping region, if the current grayscale value of the target pixel corresponding to the surface vertex in the two-dimensional displacement texture map is a preset grayscale value, then the target displacement value of the surface vertex is determined to be 0. For each surface vertex in the overlapping region, if the current gray value of the target pixel corresponding to the surface vertex is not the preset gray value, the current gray value of the target pixel is normalized to obtain a normalized displacement coefficient, and the target displacement value of the surface vertex is calculated based on the normalized displacement coefficient and the preset maximum deformation depth of the target virtual surgical object.

9. The method according to claim 7, characterized in that, Based on the target displacement value of each of the surface vertices, the three-dimensional coordinates of the target position of each surface vertex are determined, including: Determine the current normal vector of the overlapping region; For each of the surface vertices, based on the three-dimensional coordinates of the vertex's original position, the vertex's target displacement value, and the current normal vector, the three-dimensional coordinates of the vertex's target position are calculated using the following formula: P new = P old +I have( N , d ); in, P new The three-dimensional coordinates of the target position of the surface vertex. P old Let be the three-dimensional coordinates of the original position of the surface vertex, and mul(·) be used to represent the multiplication operation. N For the current normal vector, d The target displacement value is given.

10. A head-mounted device, characterized in that, It includes a memory and a computer program stored in the memory and executable on a processor, wherein the processor executes the computer program to implement the steps corresponding to the head-mounted device in the method as described in any one of claims 1-9.

Images