Medical image generation device and operating method based on image integration using augmented reality

Abstract

The method for operating a medical image processing device using augmented reality according to the present disclosure includes the steps of: acquiring a first image including vascular information and bone tissue information obtained by tomography from a first external device; acquiring a second image including facial structures and facial curvature obtained by 3D photography from a second external device; extracting facial feature points from facial specific information including the vascular information, the bone tissue information, the facial structures, and the facial curvature; aligning the first image to the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto a patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

Description

Technical Field

[0001] The present disclosure relates to a medical image generation device and an operation method thereof. More specifically, the present disclosure relates to a medical image generation device based on an image alignment infrastructure using augmented reality and an operation method thereof.

Background Art

[0002] Augmented Reality (AR) is a technology that overlays 3D virtual images on real images or backgrounds to present them as one image. In recent years, there have been attempts to integrate augmented reality technology into the medical field.

[0003] As a conventional technology, there is an augmented reality-based navigation technology that simply visualizes the result of rigid registration using an image (CT / MRI) taken before surgery through AR technology. Augmented reality navigation technology requires the alignment of the coordinate system between the patient's body and the 3D object. Therefore, conventional augmented reality navigation technology uses a marker as information in the real world to perform coordinate system alignment.

[0004] Marker-based AR has a problem in that the alignment of medical images is determined only based on markers, so the markers must be set precisely. Therefore, there is a current need for a new technology that can suppress the occurrence of human errors in all processes of using augmented reality without setting markers or maintaining the relative positions of the patient's body and markers after the markers are set.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide a medical image processing device using augmented reality (AR) and an operation method thereof.

[0006] The issues that this disclosure aims to address are not limited to those mentioned above, and other issues not mentioned can be clearly understood by an average engineer from the description below. [Means for solving the problem]

[0007] A method for operating a medical image processing apparatus using Augmented Reality (AR) according to an exemplary embodiment of the present disclosure for achieving the technical challenges described above includes the steps of acquiring a first image from a first external device, which includes vascular information and bone tissue information obtained by tomography, and acquiring a second external device, which includes 3D imaging of the facial structure and Facial curvature A step of acquiring a second image including the blood vessel information, the bone tissue information, the facial structure, and Facial curvature The process may include the steps of: extracting facial feature points from facial-specific information including; aligning the first image with the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto the patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0008] Furthermore, a medical image processing apparatus according to an exemplary embodiment of the present disclosure includes a feature extraction module configured to extract feature portions from an image, an image matching module configured to match at least two images, an image processing module configured to process data constituting an image using predefined operations, a memory for storing an augmented reality processing module configured to realize and display the processed image as augmented reality, and one or more cores, wherein the memory includes a first image from an external device containing vascular information and bone tissue information, and a facial structure and Facial curvatureThe processor obtains a second image including the blood vessel information, bone tissue information, facial structure, and Facial curvature The method is characterized by extracting facial feature points from facial-specific information including the facial features, aligning the first image with the second image based on the facial feature points, generating a composite image by 3D modeling the first image aligned with the second image, projecting the composite image onto the patient's face, and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0009] In addition, we can provide computer programs stored on computer-readable storage media for execution to realize this disclosure. When a computer program stored on a computer-readable storage medium according to an exemplary embodiment of this disclosure is executed on one or more processors, it performs the following operations for operating a medical image processing device using augmented reality (AR). These operations include vascular information, bone tissue information, facial structure, etc., obtained from an external device. Facial curvature The method may include the steps of: extracting facial feature points from facial-specific information including; aligning the first image with the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto the patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0010] In addition, computer-readable recording media for recording computer programs for performing methods to realize this disclosure can be further provided. [Effects of the Invention]

[0011] According to the solutions to the aforementioned problems described in this disclosure, it is possible to provide a facial augmentation technology that minimizes errors by taking into account the complex and delicate structure of the facial region through an augmentation technology centered on the facial region.

[0012] According to the solutions to the aforementioned problems of this disclosure, it is possible to provide an augmented reality-based medical image processing device, method, and computer program that can improve the accuracy of coordinate system alignment through non-marker-based coordinate system alignment.

[0013] The solution to the aforementioned problem described in this disclosure has the effect of using augmented reality, eliminating the need to set markers, or suppressing human error in all processes of maintaining the relative position of the patient's body and the markers after setting them.

[0014] According to the present invention, a 3D image, which is aligned with a CT image, is projected onto the patient's face, allowing for a visual representation of the location and depth of bone and vascular tissues within the face. This enables accurate and safe facial surgery.

[0015] The effects of this disclosure are not limited to those mentioned above, and any other effects not mentioned can be clearly understood by an ordinary engineer from the description below. [Brief explanation of the drawing]

[0016] [Figure 1] A block diagram showing a medical image processing system according to an exemplary embodiment of the present disclosure. [Figure 2] This flowchart shows the operation method of a medical image processing apparatus according to an exemplary embodiment of the present disclosure. [Figure 3] This is a detailed flowchart illustrating the operation method of a medical image processing apparatus according to an exemplary embodiment of the present disclosure. [Figure 4] This is a detailed flowchart illustrating the operation method of a medical image processing apparatus according to an exemplary embodiment of the present disclosure. [Figure 5]FIG. showing the synthesis of a tomographic image and a 3D image by the image alignment operation of a medical image processing apparatus according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. showing the projection of a bone tissue layer by the image synthesis operation of a medical image processing apparatus according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. showing the projection of a blood vessel layer by the image synthesis operation of a medical image processing apparatus according to an exemplary embodiment of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

[0017] Throughout the present disclosure, the same reference numerals indicate the same components. The present disclosure does not describe all the elements of the embodiments, and general content in the technical field to which the present disclosure pertains or overlapping content in the embodiments is omitted. The terms "section, module, member, block" used in the specification can be realized by software or hardware, and in an embodiment, a plurality of "sections, modules, members, blocks" can be realized as one component, or one "section, module, member, block" can also include a plurality of components.

[0018] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where they are directly connected but also the case where they are indirectly connected, and indirect connection includes being connected via a wireless communication network.

[0019] Also, when a part is said to "include" a certain component, this means that, unless otherwise stated to the contrary, it does not exclude other components but can further include other components.

[0020] Throughout the specification, when a member is said to be "above" another member, this includes not only the case where one member is in contact with the other member but also the case where there is another member between the two members.

[0021] Terms such as "first," "second," etc., are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0022] Unless otherwise clearly stated in the context, singular expressions include plural forms.

[0023] In each stage, the identification codes are used for explanatory purposes only and do not indicate the order of the stages. The stages may be performed in a different order than specified unless the context explicitly states otherwise.

[0024] The operating principle and embodiments of this disclosure will be described below with reference to the attached drawings.

[0025] In this specification, "medical image processing device" includes all diverse devices that can perform computational processing and provide results to a user. For example, the medical image processing device described herein may include, or take any one of the following forms: a computer, a server device, and a portable terminal.

[0026] Here, a computer can include, for example, a laptop computer, desktop computer, laptop computer, tablet PC, or slate PC, all equipped with a web browser.

[0027] A server device is a server that communicates with external devices to process information, and may include application servers, computing servers, database servers, file servers, game servers, mail servers, proxy servers, and web servers.

[0028] A portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0029] Augmented reality (AR) is a technology that overlays three-dimensional virtual images onto real-world images or backgrounds to create a single image. Such augmented reality technology is being applied to a variety of fields, including games, health, and mapping services, via various smart devices, and in recent years, it has been used in the medical field.

[0030] Among surgical procedures, head and neck and facial surgeries highlight the advantages of augmented reality. Due to the complex and delicate structures of the head, neck, and face, accurate anatomical information is crucial in these surgeries. By utilizing augmented reality, medical teams can intuitively review medical images. The face, in particular, has a greater number and clearer anatomical landmarks compared to other body parts, making augmented reality technology extremely helpful in determining the location of surgical targets and their distances to major landmarks.

[0031] The most commonly used augmented reality implementation method in the medical field is the "marker-based AR" method, which requires firmly setting artificial markers (fiducial markers) on the surgical body to serve as the reference point for projecting medical images. In orthopedic surgery, since the bone is the primary surgical target, markers can be set on the bone.

[0032] However, in head, neck, and facial surgeries, the target is often soft tissue such as the brain or skin, and even in facial fracture surgery, it is difficult to place markers on the skull. Even if markers are placed on the forehead, where the bone and epidermis are close together and there is little fat layer, there is a risk of errors due to human error during the marker placement process. Marker-based AR has the problem that the alignment of medical images is determined based solely on the markers, so the markers must be placed with great precision. Therefore, there is currently a need for new technologies that use augmented reality and either do not require marker placement, or suppress the occurrence of human error in all processes of maintaining the relative position of the patient's body and the markers after marker placement.

[0033] Figure 1 is a block diagram showing a medical image processing system 1 according to an exemplary embodiment of the present disclosure.

[0034] Referring to Figure 1, the medical image processing system 1 may include a medical image processing device 10, a tomography device 20, and a 3D imaging device 30.

[0035] The medical image processing system 1 can acquire medical information of a patient before surgery or procedure, extract feature data necessary for the procedure from the obtained medical information, and process it into data necessary for surgery or procedure. According to an exemplary embodiment, the tomography device 20 can reconstruct the human body using X-ray or ultrasound results via computed tomography (CT) and process cross-sectional images of the inside of the human body. For convenience of explanation, CT scanning is shown in this disclosure, but the tomography device 20 may mean imaging of the inside of the human body through computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET).

[0036] According to an exemplary embodiment, the 3D imaging device 30 can acquire three-dimensional image information of an object using a stereo camera and / or a depth camera.

[0037] According to exemplary embodiments of this disclosure, the medical image processing system 1 can perform marker-less medical image processing through data analysis of a medical image processing device 10 that has acquired tomographic images from a tomography device 20 and 3D images from a 3D imaging device 30, without the physical setting of artificial markers for extracting medical images. This has the effect that the medical image processing system 1 does not need to set markers, or can suppress the occurrence of human error in all processes of maintaining the relative position of the patient's body and the markers after setting them.

[0038] In exemplary embodiments of this disclosure, the medical image processing system 1 can project markerless-based medical images onto a patient's face using augmented reality technology. This allows the medical image processing system 1 to visually display the location and depth of bone and vascular tissues within the face, enabling medical personnel to perform accurate and safe facial surgery using the medical image processing system 1.

[0039] The medical image processing device 10 may include an input interface (I / F) 110, a feature extraction module 120, an image matching module 130, an image processing module 140, an augmented reality processing module 150, an output interface 160, and a database 170. The medical image processing device 10 shown in Figure 1 is merely one embodiment of the present invention, and therefore the present invention should not be construed as being limited solely by Figure 1.

[0040] The input interface 110 is for inputting image information (or signals), audio information (or signals), data, or information input by a user, and may include at least one of at least one camera, at least one microphone, and a user input interface. Audio data and image data collected by the input interface 110 can be analyzed and processed by user control commands.

[0041] In exemplary embodiments of this disclosure, the input interface 110 serves as a conduit to various types of external devices connected to the device. Such an interface may include at least one of the following: a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (input / output) port, a video I / O (input / output) port, or an earphone port. The device can perform appropriate control related to the external devices connected to the interface.

[0042] The camera processes image frames, such as still images or videos, obtained by the image sensor in shooting mode. The processed image frames can be displayed via the display unit, projected directly onto the patient's face as light signals, or stored in memory.

[0043] On the other hand, if there are multiple cameras, they can be arranged to form a matrix structure, and multiple image information with various angles or focal points can be input through the cameras forming this matrix structure. Furthermore, the cameras can be arranged in a streo structure to acquire left and right images for realizing a three-dimensional stereoscopic image.

[0044] A user input interface is for users to input information. When information is input via the user input interface, the control unit can control the operation of the device in accordance with the input information. Such a user input interface may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc., located on at least one of the front, rear, and side of the device) and software touch keys. For example, touch keys may consist of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display unit through software processing, or touch keys located in parts other than the touchscreen. On the other hand, the virtual keys or visual keys can be displayed on the touchscreen in various forms, and may consist of graphics, text, icons, videos, or combinations thereof.

[0045] The sensing unit senses at least one of the following: internal information of the device, information about the surrounding environment of the device, and user information, and generates a corresponding sensing signal. Based on such sensing signals, the control unit can control the driving or operation of the device, or process data, perform functions, or operate on application programs installed on the device.

[0046] The sensing unit described above may include at least one of the following: proximity sensor, illumination sensor, touch sensor, acceleration sensor, magnetic sensor, gravity sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor), fingerprint recognition sensor, ultrasonic sensor, optical sensor (e.g., camera), microphone, environmental sensor (e.g., at least one of a barometer, hygrometer, thermometer, radiation detection sensor, heat detection sensor, gas detection sensor), and chemical sensor (e.g., healthcare sensor, biorecognition sensor). On the other hand, this device can combine and utilize information sensed by at least two or more of these sensors.

[0047] According to exemplary embodiments of this disclosure, the input interface 110 can receive medical information images of the patient's face from an external device. For example, a camera among the external devices can acquire real-time images containing information about the space in which the patient is located. For example, the camera can acquire real-time images of the patient's face. For example, the camera can extract at least one feature point from the facial image and continuously calculate the position and orientation of the depth camera. The position and orientation of the camera can be used to track augmented reality-based medical images, as described later.

[0048] The feature extraction module 120 may be configured to extract feature parts from an image. According to an exemplary embodiment, the feature extraction module 120 may extract blood vessels, bone tissue, etc., from a medical image. Facial curvatureFeature points can be extracted from various bones and other characteristic or reference body tissues or body parts. In this disclosure, the information of feature points may include three-dimensional positional information in three-dimensional space. The three-dimensional positional information of feature points can function as a marker.

[0049] The image matching module 130 can be configured to match at least two images. According to an exemplary embodiment, the image matching module 130 can match (or match) at least one feature point extracted from two or more images. The image matching module 130 can use a widely known image matching algorithm, or it can pre-store a lookup table necessary for stereo conversion or the like for image matching and load the calculation result value as needed. According to an exemplary embodiment, the image matching module 130 can downsize sampling a first feature point, then select two points (point pair) and configure the module to index the difference between these two points, quantizing their position and direction (i.e., position and normal vector), and store this in a lookup table.

[0050] According to exemplary embodiments of this disclosure, the image matching module 130 can match a 3D image with a tomographic image based on feature points. This allows the image matching module 130 to perform coordinate system matching based on non-markers without using artificial markers. Specifically, feature points within the face are set to function as markers to perform coordinate system matching.

[0051] According to exemplary embodiments of this disclosure, the image matching module 130 can continuously calculate the matching of the composite image with the facial feature points in response to real-time fluctuations in the patient's facial image. Generally, since the patient is under anesthesia during surgery or procedure, the fluctuations are not large, but slight positional and angular changes may occur due to contact of the face with hands or medical devices during surgery or procedure by the medical team. The image matching module 130 can achieve accurate image projection by continuously calculating the matching of the composite image with the facial feature points in response to real-time fluctuations in the patient's facial image.

[0052] In an exemplary embodiment, the image matching module 130 can perform primary matching of the composite image and the real-time patient facial image based on portions corresponding to the positions of both eyes. In an exemplary embodiment, after performing primary matching to further improve the accuracy of matching, the image matching module 130 can perform secondary matching of the composite image and the real-time patient facial image based on portions corresponding to the face other than the positions of both eyes. That is, the image matching module 130 can improve the accuracy of matching by performing primary matching first based on information about relatively undeformable reference points (e.g., both eyes).

[0053] In an exemplary embodiment, the image matching module 130 can perform matching using a first matching algorithm and a second matching algorithm. For example, based on the first matching algorithm, the image matching module 130 can calculate the approximate positions of a first feature point from a first image including blood vessels and bone tissue of a facial CT substrate and a second feature point from a second image of a facial 3D imaging substrate. Here, the approximate position may mean both position and orientation.

[0054] The image matching module 130 can calculate the approximate positions of a first and second feature point using, for example, the PPF (Point Pair Feature) algorithm as a first matching algorithm. Specifically, the image matching module 130 can downsize sampling feature points and then select two points (Point Pair). Subsequently, the image matching module 130 can quantize the two selected points (where information in a certain interval is merged into one piece of information, rather than being continuous) and select two identical or similar points from a lookup table. For example, the image matching module 130 can select a group of candidates that have the highest similarity score to the two quantized points and calculate the approximate positions of the first and second feature points based on this. In the exemplary embodiment, the PPF algorithm is used after quantization (where information in a certain interval is merged into one piece of information, rather than being continuous), so improvement in accuracy is necessary. Therefore, in this invention, a second algorithm, described later, is used in addition to the PPF algorithm to improve the accuracy of matching.

[0055] The image matching module 130 can calculate the approximate positions of the first and second feature points, and then calculate the precise positions of multiple first and second feature points corresponding to the reference skin based on a second matching algorithm. The image matching module 130 can use the above-mentioned approximate positions as a starting point and calculate the precise positions of multiple first and second feature points using, for example, the ICP (Iterative Closest Point) algorithm as the second matching algorithm. Here, the precise position may mean translation and rotation.

[0056] In other words, the image matching module 130 detects position and orientation as approximate positions through the PPF algorithm, and detects movement and rotation as precise positions through the ICP algorithm to further reduce the distance between points. Thus, the present invention can improve the accuracy of matching by using the PPF algorithm to find an approximate position and performing matching through the ICP algorithm using this as a starting point.

[0057] In another embodiment, the image matching module 130 can perform matching using only the second matching algorithm. For example, the image matching module 130 can set initial values ​​by performing pre-matching at a number of already defined points. Here, pre-matching can be performed through the ICP algorithm. The image matching module 130 can, for example, perform pre-matching at the center of the top surface of the scene bounding box corresponding to the real-time image and at the center of the second bounding box. Subsequently, the image matching module 130 can set the position and direction of the point with the minimum residual error among the pre-matching results as the initial value. Then, the image matching module 130 can perform matching by applying the ICP algorithm from the initial value to calculate the precise positions of a number of first feature points and a number of second feature points.

[0058] The image processing module 140 is configured to perform image processing by processing the data that constitutes the image using predefined operations.

[0059] In an exemplary embodiment, the image processing module 140 can remove noise from medical images using an anisotropic diffusion filter (ADF). Unlike commonly used noise reduction filters, the anisotropic diffusion filter analyzes image information and effectively removes noise while preserving edge information. By removing noise from medical images, the error in facial feature point information caused by noise can be minimized when extracting feature points from facial images.

[0060] In an exemplary embodiment, the image processing module 140 can separate the background and patient region in a medical image using Otsu's Method. Otsu's Method is a technique that automatically finds an adaptive threshold for the input image for region division through analysis of the brightness distribution of the input image. This threadholding technique using Otsu's Method is a technique that uses only the brightness values ​​of the image and does not consider the topology of the image. Therefore, the image processing module 140 can sequentially perform Seeded Region Growing (SRG) and Morphology operations to remove mis-divided regions by considering the topology information of the image. SRG is a technique that uses a seed as a basis to track and expand adjacent pixels that satisfy the conditions and find them. Morphology operations remove mis-divided noise and empty space from the division result by iteratively performing erode and dilate.

[0061] In an exemplary embodiment, the image processing module 140 can divide the patient region in units of slices of the medical image. The entire divided image is used as volume data for generating a skin mesh. In an exemplary embodiment, a portion of the medical image may be data 3D modeled in the form of a skin mesh. For example, the 3D modeled medical image may be 3D mesh information including 3D vertex information, their connection information, and surface information generated thereby.

[0062] In an exemplary embodiment, the image processing module 140 can generate a skin mesh 205 using the Marching Cube technique. When generating the skin mesh, the image processing module 140 can apply a Gaussian filter in three dimensions to smooth the volume data. This achieves an effect similar to applying smoothing to a mesh generated via the Marching Cube.

[0063] The image processing module 140 can perform mesh simplification using the Fast-Quadric Mesh Simplification algorithm. By simplifying the mesh, the amount of computation and memory required can be reduced.

[0064] The augmented reality processing module 150 can be configured to materialize and display the processed image as augmented reality.

[0065] The augmented reality processing module 150 can generate augmented reality-based medical images that output medical information onto the patient's body (especially the face) in real time, based on the patient's facial feature points.

[0066] In an exemplary embodiment, the augmented reality processing module 150 can allow physicians to verify the accuracy of the fit by adjusting the transparency and contrast of the patient's body model in the real-time image. It can also adjust the brightness to more easily identify lesions and tissues of interest in the real-time image.

[0067] In an exemplary embodiment, the augmented reality processing module 150 can output at least one of the following as medical information on the patient's body in real-time images: for example, a CT plane, a lesion, location information of the skin closest to the lesion, location information of the skin relative to the lesion and the anatomical axis, and a scale for measuring the distance between the lesion and the skin.

[0068] In exemplary embodiments of this disclosure, the augmented reality processing module 150 can separately display a first layer corresponding to vascular information and a second layer containing bone tissue information on a projection image projected onto the patient's face. For example, if vascular information is projected onto the patient's face in augmented reality form as the first layer, the bone tissue information corresponding to the second layer can be displayed in sections where the first layer is not displayed or in areas where it is not displayed. In some embodiments, the first and second layers can also be displayed simultaneously.

[0069] In an exemplary embodiment of the present disclosure, when a physician selects a specific CT slice, the augmented reality processing module 150 can position the specific CT slice 401 on the patient's body in a real-time image at a location corresponding to the position of the selected CT slice (inside the patient's torso).

[0070] In exemplary embodiments of this disclosure, the augmented reality processing module 150 stacks CT slices 401 to construct data in three dimensions, generates an anatomical plane image in a desired direction from this data, and can represent it.

[0071] In the exemplary embodiments of this disclosure, the augmented reality processing module 150 can change and represent the first color (e.g., red) of the lesion in the cross-sectional image.

[0072] In exemplary embodiments of this disclosure, the augmented reality processing module 150 can represent lines, shapes, text, and the like on a patient's body in a real-time image based on physician input.

[0073] In an exemplary embodiment of this disclosure, the augmented reality processing module 150 can adjust in real time the transparency of at least one element displayed on the screen (a lesion, size, and position confirmation disc) when a physician inputs lines, shapes, text, etc., onto the patient's body in a real-time image. This allows the physician to easily confirm lines, shapes, text, etc., onto the patient's body in a real-time image.

[0074] In exemplary embodiments of this disclosure, the augmented reality processing module 150 can continuously output augmented reality-based medical images by tracking spatial information based on the position and orientation of a depth camera. Specifically, the augmented reality processing module 150 maintains the consistency between 3D modeled medical image data and real-time images by tracking spatial information for each frame using a third algorithm, after aligning the 3D modeled medical image data and real-time images at least once. Here, the third algorithm may be a SLAM (Simultaneous Localization and Mapping) algorithm. Alternatively, real-time images can also be aligned frame by frame for real-time tracking (sensing changes in the position and orientation of the depth camera, as described later). This solves the problem of increased computational complexity caused by aligning 3D modeled data and real-time images frame by frame.

[0075] In exemplary embodiments of this disclosure, the augmented reality processing module 150 can construct a space as three-dimensional data based on spatial image information (RGB information or depth information) and determine the position and orientation of a depth camera from the three-dimensional data. For example, in exemplary embodiments of this disclosure, the augmented reality processing module 150 can obtain point information of three-dimensional space from a depth image, or extract RGB feature points, and then analyze the movement between frames to determine the position and orientation of a depth camera.

[0076] The feature extraction module 120, the image matching module 130, the image processing module 140, and the augmented reality processing module 150 can each be implemented as an algorithm for controlling their operation or as a program that reproduces the algorithm. In this case, each program can be executed on a computer by at least one processor (not shown) that performs the aforementioned operations using the memory that stores each program and the data stored in the memory. In this case, the memory and the processor can each be implemented as separate chips. Alternatively, the memory and the processor can be implemented as a single chip.

[0077] The processor is configured to control the overall organic operation of various functional parts of the electronic device 100, including processing one or more instructions necessary for controlling the medical image processing device 10, performing calculations based on instructions, and making decisions based on program logic. The processor can provide or process appropriate information or functions to the user by processing signals, data, information, etc., that are input or output via input / output interfaces or various processing modules 120-150, or by driving application programs stored in the database 170. The processed data can be stored in memory, used to build the database 170, or transmitted externally via input / output interfaces or various processing modules 120-150. Such a processor can be implemented as a general-purpose processor, a dedicated processor, or an application processor. In exemplary embodiments, the processor can be implemented as an arithmetic processor (e.g., CPU (Central Processing Unit), GPU (Graphic Processing Unit), AP (Application Processor), etc.) including dedicated logic circuits (e.g., FPGA (Field Programmable Gate Array), ASICs (Application Specific Integrated Circuits), etc.), but is not limited thereto. In exemplary embodiments, the processor may be implemented as a DSP (Digital Signal Processor), an MCU (Micro Controller Unit), or an NPU (Neural Processing Unit) specialized for processing artificial neural networks, etc., which can convert analog signals to digital for high-speed processing.

[0078] The processor can control any one or more of the aforementioned components in combination to implement the various embodiments of the present disclosure described in Figures 2 to 7 below on the device.

[0079] The output interface (I / F) 160 is for generating outputs related to vision, hearing, or touch, and may include at least one of a display unit, an acoustic output interface, a haptic module, and an optical output interface. The display unit can realize a touchscreen by forming a layered structure with or integrating it with a touch sensor. Such a touchscreen can function as a user input unit providing an input interface between the device and the user, and at the same time provide an output interface between the device and the user.

[0080] The display unit displays (outputs) information processed by this device. For example, the display unit can display execution screen information of an application program (for example, an application) driven by this device, or UI (User Interface) and GUI (Graphical User Interface) information based on such execution screen information.

[0081] The acoustic output interface can output audio data received via the communication unit or stored in memory, or output acoustic signals related to the functions performed by this device. Such an acoustic output interface may include a receiver, speaker, buzzer, etc.

[0082] A haptic module generates a variety of tactile effects that the user can perceive. A typical example of a haptic module's tactile effect is vibration. The intensity and pattern of vibrations generated by a haptic module can be controlled by user selection or by settings in the control unit. In addition to vibration, a haptic module can generate a variety of tactile effects through other means, such as pin arrangements that move perpendicularly to the skin surface, air jets and suction forces through nozzles and inlets, friction against the skin surface, contact with electrodes, electrostatic forces, and the reproduction of hot and cold sensations using heat-absorbing or heat-generating elements.

[0083] Haptic modules can transmit tactile effects not only through direct contact, but also enable users to perceive tactile effects through muscle sensation in their fingers, arms, and other parts of their body. Depending on the configuration of the device, two or more haptic modules may be included.

[0084] The optical output interface can use the light from the device's light source to output a signal to indicate the occurrence of an event, or to output media such as an image to be displayed on an external screen, similar to a projector. In an exemplary embodiment, the optical output interface can adjust the wavelength, magnitude, and projection position of the light so that the image is projected three-dimensionally onto an object to represent the image in augmented reality.

[0085] The output interface 160 serves as a communication channel for various types of external devices connected to this device. Such an interface may include at least one of the following: a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (input / output) port, a video I / O (input / output) port, or an earphone port.

[0086] Database 170 refers to a collection of data that is centrally managed for the purpose of being shared and used by storing various types of information. Database 170 can store data temporarily or semi-permanently. For example, database 170 may store an operating system (OS) for driving at least one device, or data related to data and applications for hosting a website (e.g., a web application). Furthermore, as mentioned above, the database can store modules in the form of computer code. Database 170 is managed via middleware separate from application programs.

[0087] Database 170 includes relational databases (RDBs), key-value databases, object databases, document databases, and in-memory databases.

[0088] Database 170 can store data that supports the various functions of this device, programs for the operation of the control unit, input / output data (e.g., music files, still images, videos, etc.), and numerous application programs (applications) driven by this device, data for the operation of this device, and instruction words. At least some of these application programs can be downloaded from an external server via wireless communication.

[0089] The database 170 can be implemented as memory that is separate from the device or connected by wire or wireless. In this case, the memory may include at least one type of storage medium from among flash memory type, hard disk type, SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk.

[0090] Figure 2 is a flowchart showing the operation method of a medical image processing device (Figures 1 and 10) according to an exemplary embodiment of the present disclosure.

[0091] The augmented reality-based medical image processing method according to one embodiment shown in Figure 2 includes steps that are processed chronologically or in parallel in the medical image processing device 10 shown in Figure 1, and is referred to as the operation method of the medical image processing device.

[0092] In step S110, the medical image processing device 10 can acquire a first image from the first external device, including vascular information and bone tissue information obtained by tomography. In exemplary embodiments, the first external device may be a tomography device (Figures 1 and 20). For convenience of explanation, CT imaging is shown in this disclosure, but the tomography device 20 may mean imaging of the inside of the human body through computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET).

[0093] According to exemplary embodiments of this disclosure, the medical image processing device 10 can extract first facial feature points from a first image. The first facial feature points can be derived from vascular information, which includes information about the location, type, size, and extent of distribution of blood vessels distributed in the facial area, and bone tissue information, which includes information about the location, type, size, and distribution of bone tissue constituting the facial area.

[0094] In step S120, the medical image processing device 10 receives a 3D image of the facial structure from the second external device and Facial curvature A second image including the above can be obtained. According to an exemplary embodiment, the 3D imaging device 30 is a stereo camera and / or a depth camera. The medical image processing device 10 can acquire three-dimensional image information of an object using the imaging results of the stereo camera and / or depth camera.

[0095] According to exemplary embodiments of this disclosure, the medical image processing device 10 can extract second facial feature points from a second image. The second facial feature points represent the facial structure and that constitute the face. Facial curvature This includes not only the shape and structure of the bone, but also the resulting distribution of the skin covering the bone, subcutaneous fat and muscle, and the condition of the skin including the epidermis and dermis.

[0096] In step S130, the medical image processing device 10 can match the first image to the second image based on facial feature points. According to an exemplary embodiment, the medical image processing device 10 can extract facial feature points based on the first and second facial feature points. According to an exemplary embodiment, the medical image processing device 10 can extract vascular information, bone tissue information, facial structure, Facial curvature Facial feature points can be extracted from facial-specific information including these features. According to an exemplary embodiment, the medical image processing device 10 can match the first image to the second image based on the facial feature points.

[0097] In an exemplary embodiment, the medical image processing device 10 can perform matching using a first matching algorithm and a second matching algorithm. For example, based on the first matching algorithm, the medical image processing device 10 can calculate approximate positions representing the location and orientation of first feature points from a first image including blood vessels and bone tissue from a facial CT substrate, and second feature points from a second image including the shape of the eyes, nose, mouth, ears, facial bones, skeleton, and facial structure on a facial 3D imaging substrate. In an exemplary embodiment, the medical image processing device 10 can downsize sampling feature points using the Point Pair Feature (PPF) algorithm, then select and quantize two points (Point Pair) and calculate approximate positions based on similarity for two identical or similar points in a lookup table. In an exemplary embodiment, the medical image processing device 10 can calculate the precise positions of multiple first feature points and multiple second feature points using the Iterative Closest Point (ICP) algorithm.

[0098] In step S140, the medical image processing device 10 can generate a composite image by 3D modeling the first image, which has been aligned with the second image. In an exemplary embodiment, the medical image processing device 10 can generate a composite image, which is the object to be projected onto the patient's actual face as augmented reality, as a result of aligning the first image from the tomography substrate with the second image from the 3D imaging substrate. The image processing results from steps S130 and S140 will be explained in detail with reference to Figure 5.

[0099] In step S150, the medical image processing device 10 can project the composite image onto the patient's face. According to the exemplary embodiment, the medical image processing device 10 can acquire a real-time image of the patient's face from an external device. According to the exemplary embodiment, the medical image processing device 10 can search for facial feature points from the real-time image of the patient's face. According to the exemplary embodiment, the medical image processing device 10 can continuously calculate the matching between the composite image and facial feature points based on the real-time changes in the patient's face image.

[0100] According to an exemplary embodiment, the medical image processing device 10 can perform a primary alignment of the composite image and the real-time patient's facial image based on the portion corresponding to the position of both eyes, as a step in continuously calculating the alignment of the composite image and facial feature points in response to changes in the real-time patient's facial image.

[0101] According to an exemplary embodiment, the medical image processing device 10 can perform primary alignment and then perform secondary alignment of the composite image with a real-time patient facial image based on the portion of the face other than the position of the eyes.

[0102] In an exemplary embodiment, the medical image processing device 10 can perform primary alignment of a composite image and a real-time patient facial image based on the portion corresponding to the positions of both eyes. In an exemplary embodiment, the image alignment module 130 can perform secondary alignment of the composite image and a real-time patient facial image based on the portion corresponding to the face other than the positions of both eyes, after performing primary alignment to further improve the accuracy of alignment. That is, the image alignment module 130 can improve the accuracy of alignment by first performing primary alignment based on information about relatively undeformable reference points (e.g., both eyes).

[0103] In step S160, the medical image processing device 10 can display a first layer corresponding to vascular information and a second layer containing bone tissue information on the projected image. According to an exemplary embodiment, the medical image processing device 10 can display the first layer. According to an exemplary embodiment, the medical image processing device 10 can display the second layer separately at times or in areas where the first layer is not displayed. Steps S150 and S160 will be explained in detail with reference to Figures 6 and 7.

[0104] At least one component can be added or removed in accordance with the performance of the components shown in Figure 2. Furthermore, it is easily understood by a person with ordinary skill in the art that the relative positions of the components can be changed in accordance with the performance or structure of the system.

[0105] On the other hand, each component shown in Figure 2 represents software and / or hardware components such as Field Programmable Gate Arrays (FPGAs) and Application Specific Integrated Circuits (ASICs).

[0106] According to the solutions to the aforementioned problems described in this disclosure, it is possible to provide a facial augmentation technology that minimizes errors by taking into account the complex and delicate structure of the facial region through an augmentation technology centered on the facial region.

[0107] According to the solutions to the aforementioned problems of this disclosure, it is possible to provide an augmented reality-based medical image processing device, method, and computer program that can improve the accuracy of coordinate system alignment through non-marker-based coordinate system alignment.

[0108] The solution to the aforementioned problem described in this disclosure uses augmented reality, which eliminates the need to set markers, or reduces the occurrence of human error in all processes of maintaining the relative position between the patient's body and the markers after they have been set.

[0109] According to the present invention, an image in which 3D images and CT images are aligned is projected onto the patient's face, allowing for a visual representation of the location and depth of bone and vascular tissues within the face, thereby enabling accurate and safe facial surgery.

[0110] Figure 3 is a detailed flowchart of step S130 in the operation method of a medical image processing device (Figures 1, 10) according to an exemplary embodiment of the present disclosure.

[0111] After step S120 is performed, in step S210, the medical image processing device 10 can extract at least one first feature point corresponding to bone tissue and blood vessels from the first image. According to an exemplary embodiment, the medical image processing device 10 can extract first feature points from a first image containing vascular information and bone tissue information obtained from tomography from an external device.

[0112] In step S230, the medical image processing device 10 can extract at least one second feature point corresponding to the skin region of the eyes, nose, mouth, and ears of the human body based on anatomical features. According to an exemplary embodiment, the medical image processing device 10 can obtain a 3D image of the facial structure from an external device and Facial curvature A second image including can be acquired. The 3D imaging device 30 may be a stereo camera and / or a depth camera. The medical image processing device 10 can derive at least one second feature point, including the skull, nasal bone, earlobe and jawbone, based on anatomical features from the result of acquiring three-dimensional image information of an object using the imaging results of the stereo camera and / or depth camera. The skull, nasal bone, earlobe and jawbone are, Facial curvature The second feature point is an element that can determine facial structure, and it can be derived from the shape and structure of the bones, as well as the resulting distribution of the skin covering the bones, subcutaneous fat and muscle, and the skin condition including the epidermis and dermis. The second feature point can be derived from the characteristics of the image, or it can be extracted as result data through hidden layers via feature extraction of a vision-based artificial intelligence learning model (e.g., CNN (Convolutional Neural Network)).

[0113] According to exemplary embodiments of this disclosure, the medical image processing device 10 can extract a second feature point from a first image. The second feature point can be derived from vascular information, which includes information about the location, type, size, and extent of distribution of blood vessels distributed in the facial area, and bone tissue information, which includes information about the location, type, size, and distribution of bone tissue constituting the facial area.

[0114] In step S250, the medical image processing device 10 can extract facial feature points based on first and second feature points. According to an exemplary embodiment, the medical image processing device 10 can align the facial feature points based on the skin surface derived from the second image. According to an exemplary embodiment, the medical image processing device 10 can calculate the approximate positions of at least one second feature point, including the skull, nasal bone, earlobe, and jawbone, and at least one first feature point, including bone tissue and blood vessels, based on anatomical features using a first alignment algorithm. According to an exemplary embodiment, the medical image processing device 10 can calculate the precise positions of at least one first feature point and at least one second feature point based on a second alignment algorithm. The first and second alignment algorithms have been described in detail using Figures 1 and 2, so redundant explanations are omitted.

[0115] Figure 4 is a detailed flowchart of step S230 of the operation method of the medical image processing apparatus according to an exemplary embodiment of the present disclosure.

[0116] In step S231, the medical image processing device 10 can extract stable feature points within the face that exhibit little deformation. According to an exemplary embodiment, generally, patients are under anesthesia during surgery or procedure, so the degree of fluctuation is not large. However, slight changes in position and angle may occur when the face is touched by hands or medical devices during surgery or procedure by the medical team. The medical image processing device 10 can set both eyes as reference points, as they exhibit little deformation even when movement or fluctuation occurs and are easy to search, and can extract stable feature points from the ratio of the position of both eyes to the size of the entire face. In this disclosure, only both eyes are shown for explanatory purposes, but when observed in the upper part of the face, various tissues, blood vessels, muscles, structures, etc. within the body that are easily searchable and do not change or fluctuate significantly can be used as stable feature points.

[0117] In step S233, the medical image processing device 10 can adjust the second feature point based on the coordinates and contour of the stable feature point. According to an exemplary embodiment, the stable feature point may be the position and size relative to the entire facial contour of both eyes.

[0118] Figure 5 shows the synthesis of a tomographic image and a 3D image by image matching operation of a medical image processing apparatus according to an exemplary embodiment of the present disclosure.

[0119] Referring to Figure 2, in step S130, the medical image processing device 10 can match the first image to the second image based on facial feature points. According to an exemplary embodiment, the medical image processing device 10 can match vascular information, bone tissue information, facial structure, Facial curvature Facial feature points can be extracted from facial-specific information including the facial feature points. According to an exemplary embodiment, the medical image processing device 10 can match the first image to the second image based on the facial feature points. In step S140, the medical image processing device 10 can generate a composite image by 3D modeling the first image matched with the second image. In step S150, the medical image processing device 10 can project the composite image onto the patient's face.

[0120] In an exemplary embodiment, the medical image processing device 10 can acquire a real-time image of a patient's face from an external device and search for facial feature points from the real-time image of the patient's face. The medical image processing device 10 can continuously calculate the matching between the composite image and the facial feature points based on the changes in the real-time image of the patient's face.

[0121] In an exemplary embodiment, the medical image processing device 10 can first perform primary alignment of the composite image with the real-time patient's facial image based on the portions corresponding to the positions of both eyes, and then perform secondary alignment of the composite image with the real-time patient's facial image based on the portions of the face other than the positions of both eyes.

[0122] In an exemplary embodiment, the medical image processing device 10 can project a synthesized image as augmented reality onto the patient's actual face using a SLAM (Simultaneous Localization and Mapping) algorithm.

[0123] Figure 6 shows the projection of a bone tissue layer by the image synthesis operation of the medical image processing device 10 according to an exemplary embodiment of the present disclosure, and Figure 7 shows the projection of a blood vessel layer by the image synthesis operation of the medical image processing device according to an exemplary embodiment of the present disclosure.

[0124] The medical image processing device 10 can display the first layer as a step in displaying a first layer corresponding to vascular information and a second layer containing bone tissue information on the projected image. In the exemplary embodiment, the medical image processing device 10 can display the second layer separately at a point in time or in a region where the first layer is not displayed. In the exemplary embodiment, the medical image processing device 10 can acquire a real-time image of the patient's face from a second external device.

[0125] Referring to Figure 6, the medical image processing device 10 can project the bone tissue and / or facial skeletal structure captured from the patient onto a position corresponding to the patient's actual face by displaying the second layer.

[0126] Referring to Figure 7, the medical image processing device 10 can project the actual vascular tissue captured from the patient onto the patient's actual face by displaying the first layer.

[0127] On the other hand, the disclosed embodiment can be implemented in the form of a recording medium that stores computer-executable instruction words. The instruction words can be stored in the form of program code, and when executed by a processor, a program module can be generated to perform the operations of the disclosed embodiment. The recording medium can be implemented as a recording medium that can be read by a computer.

[0128] Computer-readable recording media include all types of recording media that store instruction words that can be deciphered by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0129] As described above, the embodiments disclosed have been explained with reference to the attached drawings. A person with ordinary skill in the art to which this disclosure belongs will understand that this disclosure may be carried out in forms different from the disclosed embodiments without altering the technical idea or essential features of this disclosure. The disclosed embodiments are illustrative and should not be construed as restrictive.

Claims

1. A method for operating a medical image processing device using Augmented Reality (AR), The first step is to acquire a first image from a first external device, which includes vascular information and bone tissue information obtained by tomography of the facial region. The steps include acquiring a second image from a second external device, including the facial structure and curvature of the face through three-dimensional imaging, A step of extracting facial feature points from the aforementioned vascular information, the aforementioned bone tissue information, the aforementioned facial structure, and facial curvature including facial curvature, A step of aligning the first image with the second image based on the facial feature points, A step of generating a composite image by 3D modeling a first image that is consistent with the second image, The steps include projecting the aforementioned composite image onto the patient's face, The steps include displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image, including, How medical image processing equipment operates.

2. The step of extracting facial feature points from the aforementioned facial-specific information is as follows: The steps include extracting at least one first feature point corresponding to bone tissue and blood vessels from the first image, The steps include extracting at least one second feature point from the second image based on anatomical features, corresponding to skin areas for the eyes, nose, mouth, and ears of the human body, A step of extracting the facial feature points based on the first feature point and the second feature point, A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

3. The step of aligning the first image with the second image based on the aforementioned facial feature points is: A method for operating a medical image processing apparatus according to claim 2, characterized by including the step of aligning the facial feature points based on the skin surface derived from the second image.

4. The step of extracting at least one second feature point is, A step of extracting stable feature points within the face that are relatively less deformed by contact with hands or medical devices, A step of adjusting the second feature point based on the coordinates and contour of the stable feature point, Includes, The method for operating a medical image processing apparatus according to claim 2, characterized in that the stable feature points are the position and size of both eyes relative to the entire facial contour.

5. The aforementioned facial feature points are, A method for operating a medical image processing apparatus according to claim 1, characterized in that it functions as a marker used for image matching.

6. The step of projecting the aforementioned synthesized image onto the patient's face is as follows: The steps include acquiring a real-time image of the patient's face from the second external device, The steps include: searching for the facial feature points from the real-time facial image of the patient; The steps include: continuously calculating the consistency between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image; Includes, The step of continuously calculating the matching between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image is as follows: A step of first aligning the composite image and the real-time patient facial image based on the portion corresponding to the position of both eyes, After the initial alignment is performed, the composite image and the real-time patient facial image are secondarily aligned based on the portion of the face other than the position of the eyes. A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

7. The step of aligning the first image with the second image based on the aforementioned facial feature points is: A step of calculating the approximate positions of at least one first feature point corresponding to bone tissue and blood vessels based on a first matching algorithm, and at least one second feature point corresponding to skin regions for the eyes, nose, mouth, and ears of the human body from the second image, A step of calculating the precise positions of the at least one first feature point and the at least one second feature point based on a second matching algorithm, A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

8. The step of displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image is: The step of displaying the first layer, The steps include: displaying the second layer separately at a point in time or in an area where the first layer is not displayed; A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

9. A medical image processing device using Augmented Reality (AR), A memory that stores a feature extraction module configured to extract features from an image, an image matching module configured to match at least two images, an image processing module configured to process the data constituting the image using predefined operations, and an augmented reality processing module configured to realize and display the processed image as augmented reality. A processor comprising one or more cores that controls the operation of the feature extraction module, the image matching module, the image processing module, and the augmented reality processing module, Includes, The aforementioned memory is An external device acquires a first image from tomography of the face, including vascular and bone tissue information, and a second image including facial structure and facial curvature. The aforementioned processor, A medical image processing apparatus characterized by extracting facial feature points from facial-specific information including vascular information, bone tissue information, facial structure, and facial curvature; aligning the first image with the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto the patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

10. When the aforementioned processor extracts the facial feature points, From the first image, at least one first feature point corresponding to bone tissue and blood vessels is extracted. From the aforementioned second image, at least one second feature point corresponding to the skin areas of the eyes, nose, mouth, and ears of the human body is extracted based on anatomical features. The medical image processing apparatus according to claim 9, characterized in that it extracts the facial feature points based on the first feature point and the second feature point.

11. When the processor matches the first image to the second image based on the facial feature points, The medical image processing apparatus according to claim 10, characterized in that it aligns the facial feature points based on the skin surface derived from the second image.

12. When the processor extracts the at least one second feature point, We extract stable feature points within the face that are relatively less deformed by contact with hands or medical devices. The second feature point is adjusted based on the coordinates and contour of the stable feature point. The aforementioned stable feature points are, The medical image processing apparatus according to claim 10, characterized in that the position and size of both eyes are relative to the entire facial contour.

13. The aforementioned facial feature points are, The medical image processing apparatus according to claim 9, characterized in that it functions as a marker used for image matching.

14. The aforementioned memory is Real-time facial images of the patient are further acquired from the aforementioned external device. When the processor projects the synthesized image onto the patient's face, The facial feature points are searched from the real-time facial image of the patient. The medical image processing apparatus according to claim 9, characterized in that it continuously calculates the matching between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image.

15. When the processor matches the first image to the second image, Based on the first matching algorithm, approximate positions of at least one first feature point corresponding to bone tissue and blood vessels, and at least one second feature point corresponding to skin regions for the eyes, nose, mouth, and ears of the human body are calculated from the second image. The medical image processing apparatus according to claim 9, characterized in that it calculates the precise positions of the at least one first feature point and the at least one second feature point based on a second matching algorithm.

Images