Augmented reality image matching-based medical image generation device and its operating method

Abstract

The method for operating a medical image processing device using augmented reality according to the present disclosure includes the steps of: acquiring a first image including vascular information and bone tissue information obtained by tomography from a first external device; acquiring a second image including facial structures and facial curvature obtained by 3D photography from a second external device; extracting facial feature points from facial specific information including the vascular information, the bone tissue information, the facial structures, and the facial curvature; aligning the first image to the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto a patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

Description

[Technical Field]

[0001] The present disclosure relates to a medical image generating device and an operating method thereof, and more particularly to an image matching-based medical image generating device using augmented reality and an operating method thereof. [Background technology]

[0002] Augmented reality (AR) is a technology that overlays a 3D virtual image onto a real image or background to create a single image. In recent years, there have been attempts to integrate AR technology into the medical field.

[0003] Conventional technology includes augmented reality-based navigation technology that simply visualizes the results of rigid registration using AR technology using images (CT / MRI) taken before surgery. Augmented reality navigation technology requires the alignment of coordinate systems between the patient's body and 3D objects, and conventional augmented reality navigation technology uses markers as real-world information to align the coordinate systems.

[0004] Marker-based AR has the problem of requiring precise marker placement, as the alignment of medical images is determined solely based on the marker. Therefore, there is a need for new technology that uses augmented reality to either not set markers or to reduce the occurrence of human error in the entire process of maintaining the relative position of the marker and the patient's body after the marker is set. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a medical image processing device using augmented reality (AR) and an operating method thereof.

[0006] The problems that the present disclosure aims to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description that follows. [Means for solving the problem]

[0007] To achieve the above-mentioned technical objectives, an operating method of a medical image processing device using Augmented Reality (AR) according to an exemplary embodiment of the present disclosure includes the steps of: acquiring a first image including vascular information and bone tissue information obtained by tomography from a first external device; acquiring a second image including facial structures and facial curvature obtained by 3D photography from a second external device; extracting facial feature points from facial specific information including the vascular information, the bone tissue information, the facial structures, and the facial curvature; aligning the first image to the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto a patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0008] Also, a medical image processing device according to an exemplary embodiment of the present disclosure includes a memory storing a feature extraction module configured to extract features from an image, an image matching module configured to match at least two images, an image processing module configured to perform image processing on data constituting the image by a predefined operation, and an augmented reality processing module configured to realize and display the processed image as an augmented reality, and one or more cores, and controls operations of the feature extraction module, the image matching module, the image processing module, and the augmented reality processing module. and a processor, wherein the memory acquires a first image including blood vessel information and bone tissue information and a second image including facial structures and facial curvature from an external device, the processor extracts facial feature points from facial specific information including the blood vessel information, the bone tissue information, the facial structures, and the facial curvature, aligns the first image to the second image based on the facial feature points, generates a composite image by 3D modeling the first image aligned with the second image, projects the composite image onto the patient's face, and displays a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image.

[0009] In addition, a computer program stored in a computer-readable recording medium for execution to realize the present disclosure may also be provided. When executed by one or more processors, the computer program stored in a computer-readable storage medium according to an exemplary embodiment of the present disclosure performs the following operations for performing a method for operating a medical image processing device using augmented reality (AR). The operations may include extracting facial feature points from facial specific information including blood vessel information, bone tissue information, facial structure, and facial curvature obtained from an external device, aligning the first image to the second image based on the facial feature points, generating a composite image by 3D modeling the first image aligned with the second image, projecting the composite image onto the patient's face, and displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image.

[0010] In addition, a computer-readable recording medium having a computer program for executing the method for realizing the present disclosure recorded thereon can also be provided. [Effects of the Invention]

[0011] According to the means for solving the above-mentioned problems of the present disclosure, a facial augmentation technique can be provided that minimizes errors by taking into account the complex and delicate structure of the face through an augmentation technique centered on the face.

[0012] According to the solution to the above-mentioned problem of the present disclosure, an augmented reality-based medical image processing device, method, and computer program can be provided that can improve the accuracy of coordinate system alignment through non-marker-based coordinate system alignment.

[0013] The solution to the above-mentioned problem of the present disclosure has the advantage that by using augmented reality, it is not necessary to set a marker, or the occurrence of human error can be reduced throughout the entire process of maintaining the relative position of the patient's body and the marker after the marker is set.

[0014] According to the present invention, a 3D image is projected onto the patient's face in a manner that is aligned with the CT image, thereby visually displaying the position and depth of bone tissue and vascular tissue located inside the face, thereby enabling accurate and safe facial surgery to be performed.

[0015] The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned above will be clearly understood by those skilled in the art from the description below. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating a medical imaging system according to an exemplary embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating a method of operating a medical imaging device according to an exemplary embodiment of the present disclosure. [Figure 3] 4 is a detailed flowchart illustrating a method of operation of a medical imaging device according to an exemplary embodiment of the present disclosure. [Figure 4] 4 is a detailed flowchart illustrating a method of operation of a medical imaging device according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates the merging of a tomographic image and a 3D image through an image matching operation of a medical imaging processing device according to an exemplary embodiment of the present disclosure. [Figure 6] 10A-10C illustrate projections of bone tissue layers through an image compositing operation of a medical imaging device according to an exemplary embodiment of the present disclosure. [Figure 7] 10A and 10B are diagrams illustrating the projection of a vessel layer through an image compositing operation of a medical imaging processor according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] The same reference numerals refer to the same components throughout this disclosure. This disclosure does not describe all elements of the embodiments, and general content in the technical field to which this disclosure pertains or overlapping content in the embodiments will be omitted. The terms "unit, module, component, block" used in the specification can be realized by software or hardware, and depending on the embodiment, multiple "units, modules, components, blocks" may be realized as a single component, or one "unit, module, component, block" may include multiple components.

[0018] Throughout this specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, including connection via a wireless communication network.

[0019] Furthermore, when a part is described as "comprising" a certain element, this does not mean that it excludes other elements, but that it may further include other elements, unless otherwise specified.

[0020] Throughout this specification, when an element is said to be "on" another element, this includes not only when the element is in contact with the other element, but also when there is another element between the two elements.

[0021] The terms "first," "second," etc. are used to distinguish one component from another, and the components are not limited to the terms described above.

[0022] The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0023] In each step, the identifying numbers are used for convenience of explanation, and the identifying numbers do not describe the order of each step, and each step may be performed in a different order than specified unless the context clearly dictates a specific order.

[0024] The working principle and embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0025] The term "medical image processing device" as used herein includes a variety of devices capable of performing computational processing and providing results to a user. For example, the medical image processing device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take any one of these forms.

[0026] Here, the computer may include, for example, a notebook computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, etc. equipped with a web browser.

[0027] The server device is a server that communicates with external devices and processes information, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, a web server, etc.

[0028] The portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0029] Augmented reality (AR) is a technology that overlays a 3D virtual image onto a real image or background to display it as a single image. This AR technology is being applied to various fields such as games, health, and map services through various smart devices, and in recent years, AR technology has been used in the medical field.

[0030] Among surgical procedures, the advantages of augmented reality are highlighted in head and neck and facial surgery. Due to the complex and delicate structure of the head, neck, and face, precise anatomical information is required for these surgeries, but by utilizing augmented reality, medical staff can intuitively view medical images. Compared to other body parts, the face has many distinct anatomical landmarks, making augmented reality technology extremely useful for determining the location of surgical targets and the distance between them.

[0031] The most commonly used augmented reality method in the medical field is the "marker-based AR" method, which requires a fiducial marker to be firmly attached to a rigid body as a reference for projecting medical images. In orthopedic surgery, bones are the main surgical targets, so markers can be attached to bones.

[0032] However, in head and neck and facial surgeries, soft tissues such as the brain or skin are often the target, making it difficult to place markers on the skull, even in cases of facial fracture surgery. Even if markers are placed on the forehead, where the bone and epidermis are close and there is little fat layer, there is a risk of human error during the marker placement process. Marker-based AR has the problem of requiring precise placement of markers because medical image alignment is determined solely based on the marker. Therefore, there is a need for new technology that uses augmented reality to either not place markers or to reduce human error in the entire process of maintaining the relative position of the marker and the patient's body after the marker is placed.

[0033] FIG. 1 is a block diagram illustrating a medical image processing system 1 according to an exemplary embodiment of the present disclosure.

[0034] Referring to FIG. 1, a medical image processing system 1 can include a medical image processing device 10 , a tomography device 20 , and a 3D imaging device 30 .

[0035] The medical image processing system 1 can acquire medical information of a patient before surgery or treatment, extract feature data necessary for treatment from the acquired medical information, and process the data into data necessary for surgery or treatment. According to an exemplary embodiment, the tomography device 20 can use a computer to reconstruct the results of X-rays or ultrasound of the human body through computed tomography (CT) and process cross-sections of the inside of the human body into images. Although CT imaging is shown for the convenience of explanation in this disclosure, the tomography device 20 can also refer to the acquisition of internal images of the human body through computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET).

[0036] According to an exemplary embodiment, the 3D image capture device 30 can capture three-dimensional image information for an object using a stereo camera and / or a depth camera.

[0037] According to an exemplary embodiment of the present disclosure, the medical image processing system 1 can perform marker-less medical image processing through data analysis by the medical image processing device 10 that acquires tomographic images from the tomography device 20 and 3D images from the 3D imaging device 30, without physically setting artificial markers for extracting medical images. As a result, the medical image processing system 1 has the advantage that it does not require setting markers, or that it can reduce the occurrence of human error in all processes of maintaining the relative positions of the patient's body and the markers after setting the markers.

[0038] In an exemplary embodiment of the present disclosure, the medical image processing system 1 can project a markerless medical image onto a patient's face using augmented reality technology, thereby allowing the medical image processing system 1 to visually display the positions and depths of bone tissue and vascular tissue located inside the face, and allowing medical staff to perform accurate and safe facial surgery using the medical image processing system 1.

[0039] The medical image processing device 10 may include an input interface (I / F) 110, a feature extraction module 120, an image matching module 130, an image processing module 140, an augmented reality processing module 150, an output interface 160, and a database 170. The medical image processing device 10 of Fig. 1 is merely one embodiment of the present invention, and the present invention should not be construed as being limited by Fig. 1.

[0040] The input interface 110 is for inputting image information (or signals), audio information (or signals), data, or information input by a user, and may include at least one of at least one camera, at least one microphone, and a user input interface. The audio data and image data collected by the input interface 110 may be analyzed and processed according to user control commands.

[0041] In an exemplary embodiment of the present disclosure, the input interface 110 serves as a passageway for various types of external devices connected to the device. Such an interface unit may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with a SIM (SIM Card), an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. The device may perform appropriate control related to the external device connected to the interface unit.

[0042] The camera processes image frames, such as still or video images, acquired by the image sensor in a photographing mode, and the processed image frames can be displayed via a display unit, projected directly onto the patient's face as optical signals, or stored in a memory.

[0043] Meanwhile, when there are multiple cameras, they can be arranged in a matrix structure, and multiple image information having various angles or focuses can be input through the cameras in this matrix structure. Also, the cameras can be arranged in a stereo structure to obtain left and right images to realize a three-dimensional image.

[0044] The user input interface is used by a user to input information. When information is input through the user input interface, the control unit can control the operation of the device in accordance with the input information. Such user input interfaces can include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc., located on at least one of the front, rear, and side of the device) and software touch keys. For example, the touch keys can be virtual keys, soft keys, or visual keys displayed on a touchscreen display through software processing, or touch keys located outside the touchscreen. Meanwhile, the virtual keys or visual keys can have various forms and be displayed on the touchscreen, and can be, for example, graphics, text, icons, videos, or combinations thereof.

[0045] The sensing unit senses at least one of internal information of the device, information about the surrounding environment of the device, and user information, and generates a corresponding sensing signal. Based on the sensing signal, the control unit can control the driving or operation of the device, or process data, functions, or operations related to application programs installed on the device.

[0046] The sensing unit may include at least one of a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint sensor, an ultrasonic sensor, an optical sensor (e.g., a camera), a microphone, an environmental sensor (e.g., at least one of a barometer, a hygrometer, a thermometer, a radiation sensor, a heat sensor, and a gas sensor), and a chemical sensor (e.g., a healthcare sensor, a biometric sensor, etc.). Meanwhile, the present device may utilize information sensed by at least two of these sensors in combination.

[0047] According to an exemplary embodiment of the present disclosure, the input interface 110 can receive a medical information image of a patient's face from an external device. For example, a camera in the external device can capture a real-time image including information about the space in which the patient is located. For example, the camera can capture a real-time image of the patient's face. For example, the camera can extract at least one feature point from the facial image and continuously calculate the position and orientation of a depth camera. The position and orientation of the camera can be used to track an augmented reality-based medical image, as described below.

[0048] The feature extraction module 120 may be configured to extract features from an image. According to an exemplary embodiment, the feature extraction module 120 may extract feature points from features or reference body tissues or body parts, such as blood vessels, bone tissue, facial flexion, and various bones, in a medical image. In the present disclosure, the information of the feature points may include three-dimensional position information in a three-dimensional space. The three-dimensional position information of the feature points may function as markers.

[0049] The image matching module 130 may be configured to match at least two images. According to an exemplary embodiment, the image matching module 130 may match at least one feature point extracted from two or more images. The image matching module 130 may use a well-known image matching algorithm, or may pre-store a lookup table required for stereo transformation for image matching and load the calculation result value whenever needed. According to an exemplary embodiment, the image matching module 130 may downsize sampling a first feature point, select two points (point pairs), and quantize and index the position and direction (i.e., position and normal vector) as the difference between the two points, and store the quantized value in the lookup table.

[0050] According to an exemplary embodiment of the present disclosure, the image alignment module 130 can align the 3D image and the tomographic image based on feature points. As a result, the image alignment module 130 performs coordinate system alignment based on a non-marker system without using artificial markers. Specifically, the image alignment module 130 performs coordinate system alignment by setting feature points within the face to function as markers.

[0051] According to an exemplary embodiment of the present disclosure, the image matching module 130 can continuously calculate the matching of the composite image and the facial feature points according to the fluctuation of the patient's facial image in real time. Generally, the patient is under anesthesia during surgery or treatment, so the fluctuation is not large. However, slight fluctuations in position and angle may occur due to contact of the face with the hands or medical equipment of medical staff during surgery or treatment. The image matching module 130 can achieve accurate image projection by continuously calculating the matching of the composite image and the facial feature points according to the fluctuation of the patient's facial image in real time.

[0052] In an exemplary embodiment, the image matching module 130 can perform primary matching between the composite image and the real-time patient's facial image based on portions corresponding to the positions of the eyes. In an exemplary embodiment, the image matching module 130 can perform secondary matching between the composite image and the real-time patient's facial image based on portions corresponding to the face other than the positions of the eyes after performing primary matching to further improve the accuracy of the matching. That is, the image matching module 130 can improve the accuracy of the matching by first performing primary matching based on information about relatively unchanging reference points (e.g., the eyes).

[0053] In an exemplary embodiment, the image matching module 130 may perform matching using a first matching algorithm and a second matching algorithm. For example, the image matching module 130 may calculate approximate positions of first feature points from a first image including blood vessels and bone tissue based on a facial CT scan and second feature points from a second image based on a facial 3D imaging scan based on the first matching algorithm. Here, approximate positions may refer to positions and directions.

[0054] The image matching module 130 may calculate the approximate positions of the first and second feature points using, for example, a Point Pair Feature (PPF) algorithm as a first matching algorithm. Specifically, the image matching module 130 may downsize the feature points and select two points (point pairs). The image matching module 130 may then quantize the selected points (merging information over a certain range, rather than continuous, into one piece of information) and select two identical or similar points from a lookup table. For example, the image matching module 130 may select a group of candidates with the highest similarity scores to the two quantized points and calculate the approximate positions of the first and second feature points based on the selected points. In an exemplary embodiment, the PPF algorithm requires quantization (merging information over a certain range, rather than continuous, into one piece of information), so accuracy needs to be improved. Therefore, in the present invention, a second algorithm (described below) is additionally used in addition to the PPF algorithm to improve the accuracy of matching.

[0055] After calculating the approximate positions of the first and second feature points, the image matching module 130 can calculate precise positions of the first feature points corresponding to the reference skin and the second feature points corresponding to the reference skin based on a second matching algorithm. The image matching module 130 can calculate precise positions of the first feature points and the second feature points using, for example, an Iterative Closest Point (ICP) algorithm as a second matching algorithm starting from the approximate positions. Here, the precise positions may include translation and rotation.

[0056] That is, the image matching module 130 detects the position and direction as an approximate position through the PPF algorithm, and detects the movement and rotation as a precise position through the ICP algorithm to further approximate the distance between points. In this way, the present invention can improve the accuracy of matching by searching for an approximate position using the PPF algorithm and performing matching using the ICP algorithm from this as a starting point.

[0057] In another embodiment, the image matching module 130 may perform matching using only the second matching algorithm. For example, the image matching module 130 may perform pre-matching at multiple previously set points to set initial values. Here, the pre-matching may be performed using the ICP algorithm. For example, the image matching module 130 may perform pre-matching at the center of the top surface of a scene bounding box corresponding to the real-time image and the center of a second bounding box. Then, the image matching module 130 may set the position and orientation of a point with the smallest residual error among the pre-matching results as the initial value. Next, the image matching module 130 may perform matching by applying the ICP algorithm to the initial value to calculate precise positions of the multiple first feature points and the multiple second feature points.

[0058] The image processing module 140 performs image processing configured to process the data that constitutes an image according to predefined operations.

[0059] In an exemplary embodiment, the image processing module 140 can remove noise from medical images using an anisotropic diffusion filter (ADF). Unlike commonly used noise removal filters, the anisotropic diffusion filter analyzes image information to effectively remove noise while preserving edge information. By removing noise from medical images, errors in facial feature point information due to noise can be minimized when extracting feature points from facial images.

[0060] In an exemplary embodiment, the image processing module 140 can segment background and patient regions in a medical image using Otsu's method. Otsu's method is a technique that automatically finds an adaptive threshold for an input image to segment regions through an analysis of the brightness distribution of the input image. The thresholding method using Otsu's method uses only the brightness value of the image and does not consider the topology of the image. Therefore, the image processing module 140 can sequentially perform a seeded region growing (SRG) operation and a morphology operation to remove mis-segmented regions while taking into account the topology information of the image. SRG is a technique that uses a process of tracing, expanding, and searching for neighboring pixels that satisfy a condition based on a seed. The morphology operation repeatedly performs erosion and dilation to remove mis-segmented noise and empty space from the segmentation result.

[0061] In an exemplary embodiment, the image processing module 140 can segment a patient region in units of slices of a medical image. The entire segmented image is used as volume data for generating a skin mesh. In an exemplary embodiment, a portion of the medical image can be 3D modeled data in the form of a skin mesh. For example, the 3D modeled medical image can be 3D mesh information including 3D vertex information, their connection information, and surface information generated thereby.

[0062] In an exemplary embodiment, the image processing module 140 can generate the skin mesh 205 using a marching cubes technique. When generating the skin mesh, the image processing module 140 can apply a Gaussian filter in three dimensions to smooth the volume data, thereby achieving the same effect as applying smoothing to a mesh generated via marching cubes.

[0063] The image processing module 140 can perform mesh simplification using a fast-quadric mesh simplification algorithm, which can reduce the amount of calculation and memory required.

[0064] The augmented reality processing module 150 can be configured to instantiate and display the processed images as an augmented reality.

[0065] The augmented reality processing module 150 can generate an augmented reality-based medical image that allows medical information to be displayed on the patient's body (particularly the face) in a real-time image based on the patient's facial feature points.

[0066] In an exemplary embodiment, the augmented reality processing module 150 can adjust the transparency and contrast of the patient's body modeling in the real-time image to allow the physician to verify the accuracy of the match, and can adjust the brightness to more easily identify lesions and structures of interest in the real-time image.

[0067] In an exemplary embodiment, the augmented reality processing module 150 can output at least one of the following medical information on the patient's body in the real-time image: a CT plane, a lesion, position information of the skin closest to the lesion, position information of the skin in the direction of the lesion and an anatomical axis, and a scale for measuring the distance between the lesion and the skin.

[0068] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may separately display a first layer corresponding to blood vessel information and a second layer including bone tissue information on a projection image projected onto a patient's face. For example, when blood vessel information is projected onto a patient's face in an augmented reality format as the first layer, bone tissue information corresponding to the second layer may be displayed separately at a time or in an area where the first layer is not displayed. Depending on the embodiment, the first and second layers may be displayed simultaneously.

[0069] In an exemplary embodiment of the present disclosure, when a doctor selects a particular CT slice, the augmented reality processing module 150 can place the particular CT slice 401 at a position (inside the patient's torso) on the patient's body in the real-time image that corresponds to the position of the particular CT slice.

[0070] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can stack CT slices 401 to construct data in three dimensions, and generate and display anatomical plane images from this data in any desired direction.

[0071] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can change the first color (eg, red) of the lesion area to represent it in the cross-sectional image.

[0072] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can render lines, shapes, text, etc. on the patient's body in the real-time image according to the physician's input.

[0073] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can adjust the transparency of at least one element (lesion, size, and location confirmation disk) displayed on the screen in real time when the doctor inputs a line, a figure, a text, etc. on the patient's body in the real-time image, so that the doctor can easily see the line, a figure, a text, etc. on the patient's body in the real-time image.

[0074] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may track spatial information based on the position and direction of a depth camera and continuously output an augmented reality-based medical image. That is, the augmented reality processing module 150 aligns the 3D modeled medical image data with the real-time image at least once, and then continuously maintains the alignment between the 3D modeling data and the real-time image by tracking spatial information for each frame using a third algorithm. Here, the third algorithm may be a simultaneous localization and mapping (SLAM) algorithm. Alternatively, the real-time image may be aligned for each frame for real-time tracking (detecting changes in the position and direction of the depth camera, which will be described later). This solves the problem of increased computational complexity due to aligning the 3D modeling data with the real-time image for each frame.

[0075] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can generate three-dimensional data of the space based on image information (RGB information or depth information) of the space and determine the position and direction of the depth camera from the three-dimensional data. For example, in an exemplary embodiment of the present disclosure, the augmented reality processing module 150 can obtain point information of the three-dimensional space from the depth image or extract RGB feature points, and then analyze movement between frames to determine the position and direction of the depth camera.

[0076] The feature extraction module 120, image matching module 130, image processing module 140, and augmented reality processing module 150 can be implemented as algorithms for controlling their respective operations or as programs that reproduce the algorithms. In this case, each program can be executed on a computer by a memory that stores each program and at least one processor (not shown) that performs the aforementioned operations using data stored in the memory. In this case, the memory and processor can be implemented as separate chips. Alternatively, the memory and processor can be implemented as a single chip.

[0077] The processor is configured to process one or more instructions required for controlling the medical image processing apparatus 10, perform calculations according to the instructions, make decisions according to program logic, and generally control the organic operation of various functional units of the electronic device 100. The processor processes signals, data, information, etc. input or output via the input / output interface or the various processing modules 120-150, or runs application programs stored in the database 170, thereby providing or processing appropriate information or functions to a user. The processed data can be stored in memory, used to construct the database 170, or transmitted to the outside via the input / output interface or the various processing modules 120-150. Such a processor can be realized as a general-purpose processor, a dedicated processor, an application processor, or the like. In an exemplary embodiment, the processor can be realized as an arithmetic processor (e.g., a central processing unit (CPU), a graphic processing unit (GPU), an application processor (AP), etc.) including a dedicated logic circuit (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), but is not limited thereto. In an exemplary embodiment, the processor may be implemented as a DSP (Digital Signal Processor) capable of converting analog signals to digital signals and performing high-speed processing, an MCU (Micro Controller Unit), or an NPU (Neural Processing Unit) specialized for processing artificial neural networks.

[0078] The processor may control any one or more of the aforementioned components in combination to implement on the device various embodiments according to the present disclosure, as described in Figures 2-7 below.

[0079] The output interface (I / F) 160 is for generating output related to vision, hearing, touch, or the like, and may include at least one of a display unit, an audio output interface, a haptic module, and an optical output interface. The display unit may be layered with a touch sensor or may be integrally formed with the touch sensor to realize a touch screen. Such a touch screen may function as a user input unit that provides an input interface between the device and a user, and may also provide an output interface between the device and a user.

[0080] The display unit displays (outputs) information processed by the device. For example, the display unit may display execution screen information of an application program (e.g., an application) running on the device, or UI (User Interface) or GUI (Graphical User Interface) information based on such execution screen information.

[0081] The audio output interface can output audio data received via the communication unit or stored in the memory, or output audio signals related to functions performed by the device. Such an audio output interface can include a receiver, a speaker, a buzzer, etc.

[0082] A haptic module generates various haptic effects that can be felt by the user. A typical example of a haptic effect generated by a haptic module is vibration. The strength and pattern of the vibration generated by the haptic module can be controlled by user selection or by settings in the control unit. In addition to vibration, the haptic module can generate various haptic effects, such as stimuli such as pin arrangements that move vertically relative to the contact surface of the skin, air ejection or suction force through an outlet or inlet, friction against the skin surface, electrode contact, and electrostatic force, as well as effects that reproduce the sensation of cold or warm using elements that can absorb or generate heat.

[0083] The haptic module can not only transmit haptic effects through direct contact, but also allow the user to feel the haptic effects through the muscle sensation of the fingers, arms, etc. Depending on the configuration of the device, two or more haptic modules can be provided.

[0084] The light output interface can output a signal to notify the occurrence of an event using light from the light source of the device, or output media such as images to be displayed on an external screen like a projector. In an exemplary embodiment, the light output interface can adjust the wavelength, size, and projection position of the light so that an image is projected three-dimensionally onto an object to represent the image in augmented reality.

[0085] The output interface 160 serves as a communication path for various types of external devices connected to the device, and may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with a SIM card, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port.

[0086] Database 170 refers to a collection of data that is integrated and managed for the purpose of sharing and using various types of information by storing them. Database 170 can store data temporarily or semi-permanently. For example, database 170 can store an operating system (OS) for running at least one device, data for hosting a website, and data related to applications (e.g., web applications). As described above, database 170 can also store modules in the form of computer code. Database 170 is managed via middleware that is separate from application programs.

[0087] The database 170 includes a relational database (RDB), a key-value database, an object database, a document database, a memory database, and the like.

[0088] The database 170 can store data supporting various functions of the device, programs for the operation of the control unit, input / output data (e.g., music files, still images, videos, etc.), a number of application programs (or applications) run by the device, and data and commands for the operation of the device. At least some of these application programs can be downloaded from an external server via wireless communication.

[0089] The database 170 can be realized as a memory separate from the device or connected by wire or wirelessly, where the memory can include at least one type of storage medium from the following: a flash memory type, a hard disk type, a solid state disk type (SSD type), a silicon disk drive type (SDD type), a multimedia card micro type, a card-type memory (e.g., SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0090] FIG. 2 is a flow chart illustrating a method of operation of a medical imaging device (FIGS. 1, 10) according to an exemplary embodiment of the present disclosure.

[0091] The augmented reality-based medical image processing method according to one embodiment shown in FIG. 2 includes steps that are processed in time series or in parallel in the medical image processing apparatus 10 shown in FIG. 1, and is referred to as an operating method of the medical image processing apparatus.

[0092] In step S110, the medical image processing device 10 can acquire a first image including vascular information and bone tissue information by tomography from a first external device. In an exemplary embodiment, the first external device can be a tomography device (FIGS. 1 and 20). Although CT imaging is shown for the convenience of explanation in this disclosure, the tomography device 20 can refer to imaging of the inside of the human body through computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET).

[0093] According to an exemplary embodiment of the present disclosure, the medical image processing apparatus 10 can extract first facial feature points from the first image. The first facial feature points can be derived from blood vessel information including information on the position, type, size, and distribution of blood vessels distributed in the facial region, and bone tissue information including information on the position, type, size, and distribution of bone tissue constituting the facial region.

[0094] In step S120, the medical image processing apparatus 10 can acquire a second image including facial structures and facial flexion through 3D imaging from a second external device. According to an exemplary embodiment, the 3D imaging device 30 is a stereo camera and / or a depth camera. The medical image processing apparatus 10 can acquire 3D image information of the object using the imaging results of the stereo camera and / or the depth camera.

[0095] According to an exemplary embodiment of the present disclosure, the medical image processing device 10 can extract second facial feature points from the second image, which can be derived from the facial structure and facial curvature that constitute the face, as well as the shape and structure of the bones, the resulting distribution of the skin overlying the bones, the subcutaneous fat and muscle, and the skin condition including the epidermis and dermis.

[0096] In step S130, the medical image processing device 10 can align the first image to the second image based on the facial feature points. According to an exemplary embodiment, the medical image processing device 10 can extract facial feature points based on the first facial feature points and the second facial feature points. According to an exemplary embodiment, the medical image processing device 10 can extract facial feature points from facial-specific information including blood vessel information, bone tissue information, facial structure, and facial curvature. According to an exemplary embodiment, the medical image processing device 10 can align the first image to the second image based on the facial feature points.

[0097] In an exemplary embodiment, the medical image processing device 10 can perform matching using a first matching algorithm and a second matching algorithm. For example, the medical image processing device 10 can calculate approximate positions, meaning the positions and orientations, of first feature points from a first image including blood vessels and bone tissue based on a facial CT scan and second feature points from a second image including the shape of the eyes, nose, mouth, ears, and facial bones on the face, as well as the skeletal structure and facial structure based on a facial 3D image, based on the first matching algorithm. In an exemplary embodiment, the medical image processing device 10 can downsize sampling the feature points using a PPF (Point Pair Feature) algorithm, select and quantize two points (point pairs), and calculate approximate positions based on the similarity of two identical or similar points in a lookup table. In an exemplary embodiment, the medical image processing device 10 can calculate precise positions of multiple first feature points and multiple second feature points using an ICP (Iterative Closest Point) algorithm.

[0098] In step S140, the medical image processing apparatus 10 can generate a composite image by 3D modeling the first image aligned with the second image. In an exemplary embodiment, the medical image processing apparatus 10 can generate a composite image that can be projected as an augmented reality image onto the patient's actual face as a result of aligning the first image based on tomography and the second image based on 3D imaging. The image processing results of steps S130 and S140 will be described in detail with reference to FIG. 5.

[0099] In step S150, the medical image processing device 10 can project the composite image onto the patient's face. According to an exemplary embodiment, the medical image processing device 10 can acquire a real-time image of the patient's face from an external device. According to an exemplary embodiment, the medical image processing device 10 can search for facial feature points from the real-time image of the patient's face. According to an exemplary embodiment, the medical image processing device 10 can continuously calculate the match between the composite image and the facial feature points according to the fluctuation of the real-time image of the patient's face.

[0100] According to an exemplary embodiment, the medical image processing device 10 can perform primary alignment between the composite image and the real-time patient's facial image based on the parts corresponding to the positions of both eyes, as a step of continuously calculating the alignment between the composite image and facial feature points according to the fluctuations of the real-time patient's facial image.

[0101] According to an exemplary embodiment, after performing the primary alignment, the medical imaging device 10 can perform a secondary alignment between the composite image and the real-time patient's facial image based on corresponding parts of the face other than the positions of the eyes.

[0102] In an exemplary embodiment, the medical imaging device 10 can perform primary alignment between the synthetic image and the real-time patient's facial image based on portions corresponding to the positions of the eyes. In an exemplary embodiment, the image alignment module 130 can perform secondary alignment between the synthetic image and the real-time patient's facial image based on portions corresponding to the face other than the positions of the eyes after performing primary alignment to further improve alignment accuracy. That is, the image alignment module 130 can improve alignment accuracy by first performing primary alignment based on information about relatively unchanging reference points (e.g., the eyes).

[0103] In step S160, the medical image processing apparatus 10 can display a first layer corresponding to blood vessel information and a second layer including bone tissue information on the projection image. According to an exemplary embodiment, the medical image processing apparatus 10 can display the first layer. According to an exemplary embodiment, the medical image processing apparatus 10 can display the second layer separately at times or in areas where the first layer is not displayed. Steps S150 and S160 will be described in detail with reference to FIGS. 6 and 7.

[0104] At least one component may be added or removed depending on the performance of the components shown in Figure 2. Furthermore, it will be readily understood by those skilled in the art that the relative positions of the components may be changed depending on the performance or structure of the system.

[0105] Meanwhile, each component shown in FIG. 2 represents software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0106] According to the means for solving the above-mentioned problems of the present disclosure, a facial augmentation technique can be provided that minimizes errors by taking into account the complex and delicate structure of the face through an augmentation technique centered on the face.

[0107] According to the above-mentioned solution of the problem of the present disclosure, it is possible to provide an augmented reality-based medical image processing device, method, and computer program that can improve the accuracy of coordinate system alignment through non-marker-based coordinate system alignment.

[0108] According to the solution to the above-mentioned problem of the present disclosure, by using augmented reality, there is no need to set a marker, or after the marker has been set, the occurrence of human error can be reduced throughout the entire process of maintaining the relative position of the patient's body and the marker.

[0109] According to the present invention, an image in which a 3D image and a CT image are aligned is projected onto the patient's face, so that the position and depth of bone tissue and vascular tissue located inside the face can be visually displayed, thereby achieving accurate and safe facial surgery.

[0110] FIG. 3 is a detailed flowchart of step S130 of the method of operation of the medical imaging device (FIGS. 1 and 10) according to an exemplary embodiment of the present disclosure.

[0111] After step S120 is performed, in step S210, the medical image processing apparatus 10 can extract at least one first feature point corresponding to bone tissue and blood vessels from the first image. According to an exemplary embodiment, the medical image processing apparatus 10 can extract the first feature point from the first image including blood vessel information and bone tissue information obtained by tomography from an external device.

[0112] In step S230, the medical image processing device 10 can extract at least one second feature point corresponding to the skin regions of the eyes, nose, mouth, and ears of the human body based on anatomical features. According to an exemplary embodiment, the medical image processing device 10 can acquire a second image including facial structures and facial flexion through 3D imaging from an external device. The 3D imaging device 30 can be a stereo camera and / or a depth camera. The medical image processing device 10 can derive at least one second feature point including the skull, nasal bone, earlobes, and jawbone based on anatomical features from the results of acquiring 3D image information of the object using the imaging results of the stereo camera and / or depth camera. The skull, nasal bone, earlobes, and jawbone are elements that can determine facial flexion and facial structure. The second feature point can be derived from the shape and structure of the bones, as well as the distribution of the skin covering the bones, subcutaneous fat and muscle, and skin conditions including the epidermis and dermis. The second feature point can be derived from the characteristics of the image, or can be extracted as result data through a hidden layer through feature extraction of a vision-based artificial intelligence learning model (e.g., a convolutional neural network (CNN)).

[0113] According to an exemplary embodiment of the present disclosure, the medical image processing device 10 can extract second feature points from the first image. The second feature points can be derived from blood vessel information including information on the position, type, size, and distribution of blood vessels distributed in the facial region, and bone tissue information including information on the position, type, size, and distribution of bone tissue constituting the facial region.

[0114] In step S250, the medical image processing device 10 can extract facial feature points based on the first feature points and the second feature points. According to an exemplary embodiment, the medical image processing device 10 can match the facial feature points based on the skin surface derived from the second image. According to an exemplary embodiment, the medical image processing device 10 can calculate approximate positions of at least one second feature point including the skull, nasal bone, earlobe, and jawbone and at least one first feature point including bone tissue and blood vessels based on anatomical features based on a first matching algorithm. According to an exemplary embodiment, the medical image processing device 10 can calculate precise positions of at least one first feature point and at least one second feature point based on a second matching algorithm. The first and second matching algorithms have been described in detail using Figures 1 and 2, so repeated description will be omitted.

[0115] FIG. 4 is a detailed flowchart of step S230 of the method of operation of a medical imaging device according to an exemplary embodiment of the present disclosure.

[0116] In step S231, the medical image processing apparatus 10 can extract stable feature points within the face that are less likely to deform. According to an exemplary embodiment, a patient is generally anesthetized during surgery or treatment, so fluctuations are limited. However, slight fluctuations in position and angle may occur due to contact of the face with the hands or medical instruments of medical staff during surgery or treatment. The medical image processing apparatus 10 can set the eyes, which are less likely to deform and are easy to search even when movement or fluctuation occurs, as reference points and extract stable feature points based on the position of the eyes and their size ratio to the entire face. Although only the eyes are shown in this disclosure for convenience of explanation, various internal body tissues, blood vessels, muscles, structures, etc., which are easily searchable and do not change or fluctuate significantly when observed from the upper part of the face, can also be used as stable feature points.

[0117] In step S233, the medical image processing apparatus 10 can adjust the second feature points based on the coordinates and contours of the stable feature points. According to an exemplary embodiment, the stable feature points can be the positions and sizes of the eyes relative to the overall facial contour.

[0118] FIG. 5 is a diagram illustrating the merging of a tomographic image and a 3D image through an image matching operation of a medical imaging processing device according to an exemplary embodiment of the present disclosure.

[0119] Referring again to FIG. 2 , in step S130, the medical image processing device 10 can align the first image to the second image based on facial feature points. According to an exemplary embodiment, the medical image processing device 10 can extract facial feature points from facial-specific information including vascular information, bone tissue information, facial structure, and facial curvature. According to an exemplary embodiment, the medical image processing device 10 can align the first image to the second image based on the facial feature points. In step S140, the medical image processing device 10 can generate a composite image by 3D modeling the first image aligned with the second image. In step S150, the medical image processing device 10 can project the composite image onto the patient's face.

[0120] In an exemplary embodiment, the medical image processing device 10 can acquire real-time facial images of a patient from an external device, search for facial feature points from the real-time facial images of the patient, and continuously calculate the matching between the composite image and the facial feature points according to the fluctuations of the real-time facial images of the patient.

[0121] In an exemplary embodiment, the medical image processing device 10 can perform primary alignment between the composite image and the real-time patient's facial image based on the parts corresponding to the positions of the eyes, and then perform secondary alignment between the composite image and the real-time patient's facial image based on the parts corresponding to the face other than the positions of the eyes.

[0122] In an exemplary embodiment, the medical imaging device 10 can use a Simultaneous localization and mapping (SLAM) algorithm to project the synthetic image onto the patient's actual facial area as an augmented reality.

[0123] FIG. 6 is a diagram showing the projection of a bone tissue layer by an image synthesis operation of a medical image processing device 10 according to an exemplary embodiment of the present disclosure, and FIG. 7 is a diagram showing the projection of a blood vessel layer by an image synthesis operation of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0124] The medical imaging device 10 can display the first layer, which corresponds to blood vessel information and the second layer including bone tissue information, on the projection image. In an exemplary embodiment, the medical imaging device 10 can display the second layer separately from the time or area where the first layer is not displayed. In an exemplary embodiment, the medical imaging device 10 can acquire real-time facial images of the patient from a second external device.

[0125] Referring to Figure 6, by displaying the second layer, the medical image processing device 10 can project the facial bone structure and / or facial bones constituting the facial structure photographed from the patient to a position corresponding to the patient's actual face.

[0126] Referring to FIG. 7, the medical image processing apparatus 10 can project the actual vascular tissue photographed from the patient onto a position corresponding to the actual face of the patient by displaying the first layer.

[0127] Meanwhile, the disclosed embodiments may be realized in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, which, when executed by a processor, generates program modules to perform the operations of the disclosed embodiments. The recording medium may be realized as a computer-readable recording medium.

[0128] Computer-readable recording media include all types of recording media that store computer-readable instructions, such as ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, and optical data storage devices.

[0129] The disclosed embodiments have been described above with reference to the accompanying drawings. Those skilled in the art will understand that the present disclosure may be embodied in forms different from the disclosed embodiments without changing the technical concept or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A method for operating a medical image processing device using Augmented Reality (AR), The first step is to acquire a first image from a first external device, which includes vascular information and bone tissue information obtained by tomography of the facial region. The steps include acquiring a second image from a second external device, including the facial structure and curvature of the face through three-dimensional imaging, A step of extracting facial feature points from the aforementioned vascular information, the aforementioned bone tissue information, the aforementioned facial structure, and facial curvature including facial curvature, A step of aligning the first image with the second image based on the facial feature points, A step of generating a composite image by 3D modeling a first image that is consistent with the second image, The steps include projecting the aforementioned composite image onto the patient's face, The steps include displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image, including, How medical image processing equipment operates.

2. The step of extracting facial feature points from the aforementioned facial-specific information is as follows: The steps include extracting at least one first feature point corresponding to bone tissue and blood vessels from the first image, The steps include extracting at least one second feature point from the second image that corresponds to the skin areas of the eyes, nose, mouth, and ears of the human body based on anatomical features, A step of extracting the facial feature points based on the first feature point and the second feature point, A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

3. The step of aligning the first image with the second image based on the aforementioned facial feature points is: A method for operating a medical image processing apparatus according to claim 2, characterized by including the step of aligning the facial feature points based on the skin surface derived from the second image.

4. The step of extracting at least one second feature point is, A step of extracting stable feature points within the face that are relatively less deformed by contact with hands or medical devices, A step of adjusting the second feature point based on the coordinates and contour of the stable feature point, Includes, The method for operating a medical image processing apparatus according to claim 2, characterized in that the stable feature points are the position and size of both eyes relative to the entire facial contour.

5. The aforementioned facial feature points are, A method for operating a medical image processing apparatus according to claim 1, characterized in that it functions as a marker used for image matching.

6. The step of projecting the aforementioned synthesized image onto the patient's face is as follows: The steps include acquiring a real-time image of the patient's face from the second external device, The steps include: searching for the facial feature points from the real-time facial image of the patient; The steps include: continuously calculating the consistency between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image; Includes, The step of continuously calculating the matching between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image is as follows: A step of first aligning the composite image and the real-time patient facial image based on the portion corresponding to the position of both eyes, After the initial alignment is performed, the composite image and the real-time patient facial image are secondarily aligned based on the portion of the face other than the position of the eyes. A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

7. The step of aligning the first image with the second image based on the aforementioned facial feature points is: A step of calculating the approximate positions of at least one first feature point corresponding to bone tissue and blood vessels, and at least one second feature point corresponding to skin regions for the eyes, nose, mouth, and ears of the human body, based on a first matching algorithm, A step of calculating the precise positions of the at least one first feature point and the at least one second feature point based on a second matching algorithm, A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

8. The step of displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image is: The step of displaying the first layer, The steps include: displaying the second layer separately at a point in time or in an area where the first layer is not displayed; A method for operating a medical image processing apparatus according to claim 1, characterized by including the following:

9. A medical image processing device using Augmented Reality (AR), A memory that stores a feature extraction module configured to extract features from an image, an image matching module configured to match at least two images, an image processing module configured to process the data constituting the image using predefined operations, and an augmented reality processing module configured to realize and display the processed image as augmented reality. A processor comprising one or more cores that controls the operation of the feature extraction module, the image matching module, the image processing module, and the augmented reality processing module, Includes, The aforementioned memory is An external device acquires a first image from tomography of the face, including vascular and bone tissue information, and a second image including facial structure and facial curvature. The aforementioned processor, A medical image processing apparatus characterized by extracting facial feature points from facial-specific information including vascular information, bone tissue information, facial structure, and facial curvature; aligning the first image with the second image based on the facial feature points; generating a composite image by 3D modeling the first image aligned with the second image; projecting the composite image onto the patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

10. When the aforementioned processor extracts the facial feature points, From the first image, at least one first feature point corresponding to bone tissue and blood vessels is extracted. From the aforementioned second image, at least one second feature point corresponding to the skin areas of the eyes, nose, mouth, and ears of the human body is extracted based on anatomical features. The medical image processing apparatus according to claim 9, characterized in that it extracts the facial feature points based on the first feature point and the second feature point.

11. When the processor matches the first image to the second image based on the facial feature points, The medical image processing apparatus according to claim 10, characterized in that it aligns the facial feature points based on the skin surface derived from the second image.

12. When the processor extracts the at least one second feature point, We extract stable feature points within the face that are relatively less deformed by contact with hands or medical devices. The second feature point is adjusted based on the coordinates and contour of the stable feature point. The aforementioned stable feature points are, The medical image processing apparatus according to claim 10, characterized in that the position and size of both eyes are relative to the entire facial contour.

13. The aforementioned facial feature points are, The medical image processing apparatus according to claim 9, characterized in that it functions as a marker used for image matching.

14. The aforementioned memory is Real-time facial images of the patient are further acquired from the aforementioned external device. When the processor projects the synthesized image onto the patient's face, The facial feature points are searched from the real-time facial image of the patient. The medical image processing apparatus according to claim 9, characterized in that it continuously calculates the matching between the synthesized image and the facial feature points based on the real-time changes in the patient's facial image.

15. When the processor matches the first image to the second image, Based on the first matching algorithm, approximate positions of at least one first feature point corresponding to bone tissue and blood vessels, and at least one second feature point corresponding to skin regions for the eyes, nose, mouth, and ears of the human body are calculated from the second image. The medical image processing apparatus according to claim 9, characterized in that it calculates the precise positions of the at least one first feature point and the at least one second feature point based on a second matching algorithm.

Images