A medical augmented reality system

Through image acquisition and human body recognition modules, combined with three-dimensional medical human model, markless posture recognition and model control are achieved, which solves the problems of high hardware requirements and low adaptability of existing systems, and provides efficient augmented reality effects and is suitable for dynamics. Surgical scene.

CN118553400BActive Publication Date: 2024-11-26JILIN UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411009135.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-26
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

The existing medical augmented reality system relies on markers for tracking, which increases hardware requirements and costs, and the model has a fixed posture and low adaptability, which cannot meet the needs of dynamic surgical scenarios.

Method used

The image acquisition module, human body recognition module, model control module and image processing module are adopted to realize mark-free posture recognition and model attitude control through human body recognition and three-dimensional medical human model, and provide augmented reality images.

Benefits of technology

It reduces hardware requirements and costs, enhances the adaptability of the system, and can change the model posture in real time to adapt to changes in the human posture, providing augmented reality effects for the whole body and local visualization of the human body.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118553400B_ABST
    Figure CN118553400B_ABST
Patent Text Reader

Abstract

The present invention discloses a medical augmented reality system, comprising: an image acquisition module for collecting real-world images; a human body recognition module connected to the image acquisition module, for taking the real-world images as input for human body recognition, and outputting the position information of each part of the human body to obtain key point coordinate information; a model control module connected to the human body recognition module, for obtaining a model image based on a three-dimensional medical human body model and taking the key point coordinate information as input through model control and reading display window data; an image processing module connected to the model control module, for processing the model image and the real-world image, and outputting an augmented reality image; an image display module connected to the image processing module, for displaying the augmented reality image on a screen. The present invention can achieve an augmented reality effect of visualizing the entire skeleton of the human body, and has a local visualization function.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of medical application and augmented reality, and in particular relates to a medical augmented reality system. Background Art

[0002] With the development of medical care and technology, people have higher requirements for medical level and quality, and traditional medical education and practice have shown their shortcomings. With the advancement of science and technology, medical augmented reality plays an increasingly important role in the medical field. Medical augmented reality technology is a combination of medicine and augmented reality technology. The medical augmented reality system provides a new way of visualization for medical education. It enriches medical teaching by displaying clinical images in front of students and makes abstract content intuitive and easy to understand. In addition, interaction with the system can improve students' learning interest and efficiency. In terms of patient condition analysis and clinical treatment, augmented reality can provide information directly in the surgeon's field of vision, which provides augmented reality with the potential to disrupt surgery. Medical staff can clearly and intuitively identify the patient's lesion site through augmented reality technology, which is more conducive to analyzing the patient's condition. Surgical navigation realized by augmented reality technology enables doctors to accurately control the location and boundaries of the patient's lesion during surgery. Medical augmented reality systems are also used to train medical skills and complex surgical operations. Doctors can use augmented reality technology to develop surgical plans, accurately grasp the location of the patient's lesion, and predict the complexity of the operation.

[0003] At present, medical augmented reality technology has shown great application potential in medical education, surgical navigation and diagnostic assistance. However, despite significant progress, current medical augmented reality systems still face some challenges and limitations. Most existing medical augmented reality systems rely on tracking technology with markers, such as the enhanced display effect of the frog anatomy teaching system achieved by Sebastian Oberdörfer et al. using markers. These markers are usually pre-designed patterns or objects. The position and posture of these markers are captured by cameras or other sensors to achieve accurate superposition of virtual information. Although this method guarantees the accuracy of the system to a certain extent, it also has some problems: first, the need for additional hardware. Tracking with markers requires additional hardware equipment and markers, which not only increases the complexity and cost of the system, but may also cause inconvenience during surgery or diagnosis. Secondly, it is environmental dependence. The markers need to be clearly captured and identified during the use of the system, which requires the operating environment to have good light conditions and less occlusion. In addition, damage or displacement of the markers may cause system tracking failure, making the system less adaptable. Moreover, the models used in existing systems are usually reconstructed from medical images such as CT, and the changes in the model's posture are usually not considered, that is, the model's posture is fixed during use. This will limit the flexibility of the system in practical applications and cannot fully meet the needs of dynamic surgical scenarios. Summary of the invention

[0004] In order to solve the technical problems that the current medical augmented reality uses markers and tracking systems for positioning, which increases the hardware requirements and cost of the system, and the system uses a fixed model with low adaptability, the present invention provides a medical augmented reality system, including:

[0005] An image acquisition module, used to collect real-world images;

[0006] A human body recognition module, connected to the image acquisition module, for taking the real-world image as input to perform human body recognition, and outputting position information of various parts of the human body to obtain key point coordinate information;

[0007] A model control module, connected to the human body recognition module, is used to obtain a model image based on a three-dimensional medical human body model by taking the key point coordinate information as input through model control and reading display window data;

[0008] An image processing module, connected to the model control module, for processing the model image and the real-world image and outputting an augmented reality image;

[0009] An image display module is connected to the image processing module and is used to display the augmented reality image on a screen.

[0010] Preferably, the human body recognition module includes a posture recognition unit and an information storage unit;

[0011] The posture recognition unit is used to perform human posture recognition on the input real-world image to obtain key point position information of the human body;

[0012] The information storage unit is used to store the key point position information in an array, and the information stored in the array includes key point plane coordinate information x and y, and depth coordinate information z.

[0013] Preferably, the gesture recognition unit has a human body part recognition function;

[0014] The human body part recognition function includes face recognition and hand recognition.

[0015] Preferably, the model control module includes a position control unit and a posture control unit;

[0016] The position control unit is used to control the transformation of the spatial position of the model after the position of the model in the display window is identically registered with the position of the human body in the image captured by the camera;

[0017] The transformation of the model space position includes rotation, scaling and translation;

[0018] The posture control unit is used to control the posture of the model by binding bones and establishing a posture library, and the posture of the model is controlled to be the same as that of the human body in the real world image.

[0019] Preferably, the position control unit includes a coordinate unification unit, a reference point selection unit, a parameter calculation unit, a coordinate updating unit, and a model rendering unit;

[0020] The coordinate unification unit is used to set the pixel size of the image captured by the camera to be the same as the pixel size of the model display window, establish a two-dimensional unified coordinate system with the center of the image as the origin and pixels as units, and convert the key point coordinates x, y, and z in pixels;

[0021] The reference point selection unit is used to select reference points according to different modes;

[0022] The parameter calculation unit is used to calculate the parameters required for model position control;

[0023] The coordinate updating unit is used to represent the model vertex coordinates with a four-dimensional column vector, and to update the coordinates of all the model vertices by calculation:

[0024] The model rendering unit is used to perform re-rendering after the coordinates are updated, render the updated model in the model display window, and obtain image data of the model display window as the output of the model control module.

[0025] Preferably, the modes include a full body mode, a face mode, and a hand mode;

[0026] The whole body mode is that all or almost all of the human body appears in the camera field of view;

[0027] The face mode is that the face is in the camera field of view, but most of the body is not in the camera field of view;

[0028] The hand mode is that the hand appears in the camera field of view, and the face is not in the camera field of view;

[0029] When the system is in full body mode or face mode, two eyes are selected as reference points;

[0030] When the system is in hand mode, select the wrist and middle fingertips as reference points.

[0031] Preferably, the parameters include a rotation angle, a scaling factor, a translation amount, a rotation matrix, a scaling matrix, and a translation matrix.

[0032] Preferably, the posture control unit includes a skeleton posture adjustment unit, a posture information storage unit, a weight distribution unit, and a local posture control unit;

[0033] The skeleton posture adjustment unit is used to bind skeletons to the initial state model, obtain a default skeleton, and adjust the position and posture of the default skeleton according to the movement mode of the human body;

[0034] The posture information storage unit is used to store the coordinates of each bone in different postures and the offset information relative to the default bone as a posture file, and then load the posture file into the system;

[0035] The weight assignment unit is used to create the skeleton of the initial model and make a weight file according to the information of the default skeleton, and then the system assigns weights to the default skeleton and binds it to the model to obtain a default posture;

[0036] The local posture control unit is used to establish a local coordinate system for each bone when the model is in a default posture; when the model posture needs to be changed, the coordinates of the points controlled by the bones remain unchanged in the local coordinate system of the bones, and only the position and direction of the local coordinate system are changed to perform local posture control on the bones.

[0037] Preferably, the local posture control unit includes a first transformation matrix calculation unit, a second transformation matrix calculation unit, a space conversion unit, and a vertex update rendering unit;

[0038] The first transformation matrix calculation unit is used to obtain the model space coordinates of the control point in the default posture through the local space coordinates of the control point of the skeleton in the default posture, and calculate and obtain the first space transformation matrix for transforming the coordinates of the control point from the local space of the skeleton to the model space;

[0039] The second transformation matrix calculation unit is used to calculate and obtain the second space transformation matrix for transforming the control points from the local skeleton space to the model space under the new posture according to the loaded posture file information when the model applies the new posture;

[0040] The space conversion unit is used to first transfer the coordinates of the control point from the model space to the local space of the skeleton in the default posture by calculation, and then convert the coordinates of the vertex from the local space back to the model space based on the second space transformation matrix of the skeleton in the new posture and the coordinates of the vertex in the new posture:

[0041] The vertex update rendering unit is used to update all vertices of the model and re-render, and then a new posture is applied to the model, thereby achieving posture control of the model.

[0042] Preferably, the image processing module includes an image fusion unit and an image replacement unit;

[0043] The image fusion unit is used to fuse the model image and the real-world image through an image processing function;

[0044] The image replacement unit is used to take out the model area from the model image and remove the corresponding area in the real image, and then add the model area to the blank area of ​​the real world image to obtain an augmented reality image.

[0045] Compared with the prior art, the present invention has the following advantages and technical effects:

[0046] The medical augmented reality system of the present invention can achieve the augmented reality effect of visualizing the whole body skeleton of the human body, and has the function of local visualization. In addition, the hardware requirements and cost of the system are reduced, while the adaptability of the system is enhanced. The registration is performed in a markerless manner, which reduces the hardware requirements of the system. The model control method is adopted to change the model posture in real time to adapt to the changes in the human body posture, thereby enhancing the adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a system workflow according to an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the posture control principle of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] like Figure 1-3 As shown, in this embodiment, a medical augmented reality system is provided, which is composed of five modules: image acquisition module, human body recognition module, model control module, image processing module and image display module. The image acquisition module and the image display module are responsible for acquiring real-world images and displaying augmented reality images, respectively. The human body recognition module, the model control module and the image processing module are responsible for the implementation of key functions of the system. Specifically, they include:

[0054] An image acquisition module, used to collect real-world images;

[0055] The human body recognition module is connected to the image acquisition module and is used to use the real-world image as input for human body recognition, and output the position information of each part of the human body to obtain the coordinate information of key points;

[0056] The model control module is connected to the human body recognition module and is used to obtain the model image by controlling the model and reading the display window data based on the three-dimensional medical human body model and taking the key point coordinate information as input;

[0057] An image processing module, connected to the model control module, is used to process the model image and the real-world image and output an augmented reality image;

[0058] The image display module is connected to the image processing module and is used to display the augmented reality image on the screen. Specifically, a call is initiated to the image processing module to request the augmented reality image to be displayed on the screen.

[0059] Furthermore, the human body recognition module acquires the real-world image by calling the image acquisition module. Then, human body recognition is performed to obtain the key point coordinate information. Then, the key point coordinate information is sent to the model control module. Specifically, the real-world image captured by the camera is used as input, and the position information of each part of the human body is output. It includes a posture recognition unit and an information storage unit;

[0060] The posture recognition unit is used to perform human posture recognition on the input real-world image and obtain the key point position information of the human body;

[0061] The information storage unit is used to store the key point position information in an array, and the information stored in the array includes the key point plane coordinate information x and y, and the depth coordinate information z.

[0062] Furthermore, considering that when human body parts appear in the camera's field of view, gesture recognition may not be able to detect key points or the detection results may be seriously inaccurate. Therefore, a human body local recognition function is added to the gesture recognition unit, which includes face recognition and hand recognition. This method can provide the system with the location information of the human body in the real world without the need for additional hardware such as markers, freeing it from the constraints of markers, reducing the hardware requirements and cost of the system, and making the system more portable.

[0063] Furthermore, the model control module obtains the model image by controlling the model and reading the display window data. Before this process, the model control module needs to obtain the coordinate information of the key points of the human body in the real world from the human body recognition module. Therefore, the model control module initiates a call to the human body recognition module to request the key point coordinate information. After receiving the key point coordinate information, the model control module starts to calculate the parameters, perform model control, and obtain the model image. Then the model image data is sent to the image processing module as a reply.

[0064] It is specifically based on a three-dimensional medical human body model, and uses the key point position information of the human body as input to output the image of the model rendering window. The system imports the three-dimensional model during initialization, and the model needs to be in 3D Object format. The system reads the vertex and face information of the model and stores it for use in the rendering process. It includes a position control unit and a posture control unit;

[0065] The position control unit is used to control the transformation of the spatial position of the model after the position of the model in the display window is aligned with the position of the human body in the image captured by the camera;

[0066] Transformations of the model space position include rotation, scaling, and translation;

[0067] The posture control unit is used to control the posture of the model by binding bones and establishing a posture library, so that the posture of the model is the same as that of the human body in the real world image.

[0068] Further, the position control unit includes a coordinate unification unit, a reference point selection unit, a parameter calculation unit, a coordinate updating unit, and a model rendering unit;

[0069] The coordinate unification unit is used to set the pixel size of the image captured by the camera to be the same as the pixel size of the model display window, and set the size of the image captured by the camera to 960×720 pixels. The size of the model display window is also set to 960×720 pixels. With the center of the image as the origin and pixels as the unit, a two-dimensional unified coordinate system is established, and the key point coordinates x, y, and z are converted in pixels.

[0070] The reference point selection unit is used to select reference points according to different modes;

[0071] The parameter calculation unit is used to calculate the parameters required for model position control;

[0072] The coordinate update unit is used to represent the model vertex coordinates with a four-dimensional column vector and update the coordinates of all vertices of the model by calculation: the calculation formula is:

[0073]

[0074] The model rendering unit is used to perform re-rendering after the coordinates are updated, render the updated model in the model display window, and obtain the image data of the model display window as the output of the model control module.

[0075] Furthermore, in order to distinguish different situations, three modes of the system are defined, including full body mode, face mode, and hand mode;

[0076] Full body mode means that all or almost all of the human body appears in the camera's field of view;

[0077] Face mode is when the face is in the camera's field of view, but most of the body is not;

[0078] Hand mode is when the hands appear in the camera's field of view, but the face is not;

[0079] When the system is in full body mode or face mode, two eyes are selected as reference points;

[0080] When the system is in hand mode, select the wrist and middle fingertips as reference points.

[0081] Furthermore, the parameters include rotation angle, scaling factor, translation amount, rotation matrix, scaling matrix, and translation matrix.

[0082] By rotation angle , scaling factor And the translation and The rotation matrix R, scaling matrix S and translation matrix P can be obtained, and the formula expression is:

[0083]

[0084] Further, the posture control unit includes a skeleton posture adjustment unit, a posture information storage unit, a weight distribution unit, and a local posture control unit;

[0085] The skeleton posture adjustment unit is used to bind skeletons to the initial state model, obtain the default skeleton, and adjust the position and posture of the default skeleton according to the movement mode of the human body, so that the skeleton can be in different postures;

[0086] The posture information storage unit is used for storing the coordinates of each bone in a certain posture and the offset information relative to the default bone as a posture file when the posture is in a certain posture, and then loading the posture file into the system;

[0087] The weight assignment unit is used to create the skeleton of the initial model and make a weight file based on the information of the default skeleton. Then the system assigns weights to the default skeleton and binds it to the model to obtain the default posture. The system imports posture files of other postures and establishes a posture library to store posture information. When a posture needs to be applied, use Figure 3 The principle shown controls the attitude of the model through a local attitude control unit.

[0088] The local posture control unit is used to establish a local coordinate system for each bone when the model is in the default posture; when the model posture needs to be transformed, the coordinates of the points controlled by the bones remain unchanged in the local coordinate system of the bones, and only the position and direction of the local coordinate system are changed to perform local posture control on the bones.

[0089] Further, the local posture control unit includes a first transformation matrix calculation unit, a second transformation matrix calculation unit, a space conversion unit, and a vertex update rendering unit;

[0090] The first transformation matrix calculation unit is used to obtain the model space coordinates of the control point in the default posture through the local space coordinates of the control point in the default posture of the skeleton, and calculate and obtain the first space transformation matrix for transforming the coordinates of the control point from the local space of the skeleton to the model space;

[0091] The second transformation matrix calculation unit is used to calculate and obtain the second space transformation matrix for transforming the control points from the local space of the skeleton to the model space under the new posture according to the loaded posture file information when the model applies the new posture;

[0092] The space conversion unit is used to transfer the coordinates of the control points from the model space to the local space of the bones in the default pose by calculation, and then convert the coordinates of the vertices from the local space back to the model space based on the second space transformation matrix of the bones in the new pose and the coordinates of the vertices in the new pose:

[0093] The vertex update rendering unit is used to update all vertices of the model and re-render, and then the new posture is applied to the model, thereby realizing the posture control of the model.

[0094] To go a step further, take bone i as an example. First, multiply the local space coordinates of the point controlled by bone i in the default posture by the 4×4 matrix The model space coordinates of the point in the default posture can be obtained. And the transformation matrix that can transform the coordinates of the point from the local space of bone i to the model space is calculated When the model needs to apply a new posture, the transformation matrix that can transform points from the local space of the skeleton i to the model space under the new posture can be calculated based on the loaded posture file information. Then, for the point v controlled by bone i, the coordinates of point v are first transferred from the model space to the local space of bone i in the default posture by the following formula:

[0095]

[0096] Then, use the spatial transformation matrix of bone i in the new posture Multiply , transform the coordinates of the vertex from local space back to model space:

[0097]

[0098] at last, is the coordinate of vertex v in the new pose. After all vertices of the model are updated and re-rendered using this method, the new pose will be applied to the model, thus realizing the pose control of the model.

[0099] Furthermore, in order to respond to the request of the image display module, the image processing module needs to obtain the model image and the real-world image, and perform image processing. Therefore, the image processing module first initiates a call to the model control module to request the model image. After receiving the model image, the image processing module calls the image acquisition module and obtains the real-world image in order to perform image processing and obtain an augmented reality image. Finally, the augmented reality image is sent to the image display module to be displayed on the screen. Among them, the real-world image is obtained through a camera, and the model image is obtained by reading the image data displayed in the model display window. After the previous steps, the two images are the same size, and the positions of the human body and the model in the image are the same. It includes an image fusion unit and an image replacement unit;

[0100] The image fusion unit is used to fuse the model image and the real world image through an image processing function;

[0101] The image replacement unit is used to take out the model area from the model image and remove the corresponding area in the real image, and then add the model area to the blank area of ​​the real-world image to obtain an augmented reality image, which is finally displayed on the screen to achieve an augmented reality visual effect.

[0102] The medical augmented reality system of the present invention can achieve the augmented reality effect of visualizing the whole body skeleton of the human body, and has the function of local visualization. In addition, the hardware requirements and cost of the system are reduced, while the adaptability of the system is enhanced. The registration is performed in a markerless manner, which reduces the hardware requirements of the system. The model control method is adopted to change the model posture in real time to adapt to the changes in the human body posture, thereby enhancing the adaptability of the system.

[0103] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A medical augmented reality system, characterized in that: include: An image acquisition module, used to collect real-world images; A human body recognition module, connected to the image acquisition module, for taking the real-world image as input to perform human body recognition, and outputting position information of various parts of the human body to obtain key point coordinate information; A model control module, connected to the human body recognition module, is used to obtain a model image based on a three-dimensional medical human body model by taking the key point coordinate information as input through model control and reading display window data; An image processing module, connected to the model control module, for processing the model image and the real-world image and outputting an augmented reality image; An image display module, connected to the image processing module, and used to display the augmented reality image on a screen; The human body recognition module includes a posture recognition unit and an information storage unit; The posture recognition unit is used to perform human posture recognition on the input real-world image to obtain key point position information of the human body; The information storage unit is used to store the key point position information in an array, and the information stored in the array includes the key point plane coordinate information x and y, and the depth coordinate information z; The model control module includes a position control unit and a posture control unit; The position control unit includes a coordinate unification unit, a reference point selection unit, a parameter calculation unit, a coordinate updating unit, and a model rendering unit; The coordinate unification unit is used to set the pixel size of the image captured by the camera to be the same as the pixel size of the model display window, establish a two-dimensional unified coordinate system with the center of the image as the origin and pixels as units, and convert the key point coordinates x, y, and z in pixels; The reference point selection unit is used to select reference points according to different modes; The parameter calculation unit is used to calculate the parameters required for model position control; The coordinate updating unit is used to represent the model vertex coordinates with a four-dimensional column vector, and to update the coordinates of all the model vertices by calculation: The model rendering unit is used to perform re-rendering after the coordinates are updated, render the updated model in the model display window, and obtain image data of the model display window as the output of the model control module; The modes include a full body mode, a face mode, and a hand mode; The whole body mode is that the whole human body appears in the camera field of view; The face mode is that the face is in the camera field of view, but the body is not in the camera field of view; The hand mode is that the hand appears in the camera field of view, and the face is not in the camera field of view; When the system is in full body mode or face mode, two eyes are selected as reference points; When the system is in hand mode, the wrist and middle fingertips are selected as reference points; The posture control unit includes a skeleton posture adjustment unit, a posture information storage unit, a weight distribution unit, and a local posture control unit; The skeleton posture adjustment unit is used to bind skeletons to the initial state model, obtain a default skeleton, and adjust the position and posture of the default skeleton according to the movement mode of the human body; The posture information storage unit is used to store the coordinates of each bone in different postures and the offset information relative to the default bone as a posture file, and then load the posture file into the system; The weight assignment unit is used to create the skeleton of the initial model and make a weight file according to the information of the default skeleton, and then the system assigns weights to the default skeleton and binds it to the model to obtain the default posture; The local posture control unit is used to establish a local coordinate system for each bone when the model is in a default posture; when the model posture needs to be changed, the coordinates of the points controlled by the bones remain unchanged in the local coordinate system of the bones, and only the position and direction of the local coordinate system are changed to perform local posture control on the bones.

2. The medical augmented reality system according to claim 1, characterized in that: The gesture recognition unit has a human body part recognition function; The human body part recognition function includes face recognition and hand recognition.

3. The medical augmented reality system according to claim 1, characterized in that: The position control unit is used to control the transformation of the spatial position of the model after the position of the model in the display window is identically registered with the position of the human body in the image captured by the camera; The transformation of the model space position includes rotation, scaling and translation; The posture control unit is used to control the posture of the model by binding bones and establishing a posture library, and the posture of the model is controlled to be the same as that of the human body in the real world image.

4. The medical augmented reality system according to claim 1, characterized in that: The parameters include rotation angle, scaling factor, translation amount, rotation matrix, scaling matrix, and translation matrix.

5. The medical augmented reality system according to claim 1, characterized in that: The local posture control unit includes a first transformation matrix calculation unit, a second transformation matrix calculation unit, a space conversion unit, and a vertex update rendering unit; The first transformation matrix calculation unit is used to obtain the model space coordinates of the control point in the default posture through the local space coordinates of the control point of the skeleton in the default posture, and calculate and obtain the first space transformation matrix for transforming the coordinates of the control point from the local space of the skeleton to the model space; The second transformation matrix calculation unit is used to calculate and obtain a second space transformation matrix for transforming the control points from the local skeleton space to the model space under the new posture according to the loaded posture file information when the model applies the new posture; The space conversion unit is used to transfer the coordinates of the control points from the model space to the local space of the skeleton in the default posture by calculation, and then convert the coordinates of the vertex from the local space back to the model space based on the second space transformation matrix of the skeleton in the new posture and the coordinates of the vertex in the new posture: The vertex update rendering unit is used to update all vertices of the model and re-render, and then a new posture is applied to the model, thereby achieving posture control of the model.

6. The medical augmented reality system according to claim 1, characterized in that: The image processing module includes an image fusion unit and an image replacement unit; The image fusion unit is used to fuse the model image and the real-world image through an image processing function; The image replacement unit is used to take out the model area from the model image and remove the corresponding area in the real image, and then add the model area to the blank area of ​​the real world image to obtain an augmented reality image.

Citation Information

Patent Citations

  • Real-time interactive augmented reality system and method and recording medium storing program for implementing the method

    US20110216090A1

  • Three-dimensional (3D) integrated teaching field system based on flipped platform and method for operating same

    US20240038086A1