Medical image generating apparatus based on image registration using augmented reality and method of operating same

Abstract

An operation method of a medical image processing apparatus using augmented reality according to the present disclosure comprises the steps of: acquiring a first image including blood vessel information and bone tissue information based on tomography from a first external apparatus; acquiring a second image containing the facial structure and the facial curvature based on three-dimensional imaging from a second external device; extracting facial feature points from the blood vessel information, the bone tissue information, and facial unique information including the facial structure and the facial curvature; registering the first image to the second image based on the facial feature points; generating a composite image by three-dimensional (3D) modeling of the first image registered with the second image; projecting the composite image to the face of the patient; and displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on a projection image.

Description

Technical Field

[0001] The present disclosure relates to a medical image generation device and an operation method thereof. More specifically, the present disclosure relates to a medical image generation device and an operation method thereof based on image registration using augmented reality. Background Art

[0002] Augmented Reality (AR) is a technology that superimposes a three-dimensional virtual image onto a real image or background and presents it as an image. Recently, attempts have been made to integrate augmented reality technology into the medical field.

[0003] As a prior art, there is an augmented reality-based navigation technology that uses pre-operative images (CT / MRI) and simply visualizes the result of rigid registration through AR technology. For augmented reality navigation technology, registration of the coordinate system between the patient's body and a three-dimensional (3D) object is necessary. Thus, existing augmented reality navigation technology uses markers as real-world information to perform coordinate system registration.

[0004] Since marker-based AR determines the registration of medical images only based on markers, there is a difficulty in that the markers must be precisely set. Therefore, there is a need for a new technology that can suppress the occurrence of human errors throughout the process of using augmented reality without setting markers or maintaining the relative position between the patient's body and the markers after setting the markers. Summary of the Invention

[0005] Technical Problem

[0006] An object of the embodiments disclosed in the present disclosure is to provide a medical image processing device using augmented reality (AR) and an operation method thereof.

[0007] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and those skilled in the art can clearly understand other technical problems not mentioned through the following description.

[0008] Technical Solution

[0009] According to an operation method of a medical image processing device using augmented reality (AR) according to an exemplary embodiment of the present disclosure for implementing the above technical problem, the method may include the following steps: obtaining a first image including vascular information and bone tissue information based on tomography from a first external device; obtaining a second image including a facial structure and facial curvature based on three-dimensional imaging from a second external device; extracting facial feature points from the vascular information, the bone tissue information, and facial intrinsic information including the facial structure and the facial curvature; registering the first image to the second image based on the facial feature points; generating a composite image by performing three-dimensional (3D) modeling on the first image registered to the second image; projecting the composite image onto a patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0010] In addition, a medical image processing device according to an exemplary embodiment of the present disclosure may include: a memory storing a feature extraction module, an image registration module, an image processing module, and an augmented reality processing module, the feature extraction module configured to extract a feature part from an image, the image registration module configured to register at least two images, the image processing module configured to perform image processing on data constituting an image according to a predefined calculation, and the augmented reality processing module configured to implement and display a processed image in an augmented reality manner; a processor including one or more cores and controlling operations of the feature extraction module, the image registration module, the image processing module, and the augmented reality processing module, wherein the memory obtains a first image including vascular information and bone tissue information and a second image including a facial structure and facial curvature from an external device, and the processor extracts facial feature points from the vascular information, the bone tissue information, and facial intrinsic information including the facial structure and the facial curvature, registers the first image to the second image based on the facial feature points, generates a composite image by performing 3D modeling on the first image registered to the second image, projects the composite image onto a patient's face, and displays a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0011] In addition, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be provided. When the computer program stored in the computer-readable storage medium according to an exemplary embodiment of the present disclosure runs on one or more processors, in order to execute an operation method of a medical image processing device using augmented reality (AR: Augmented Reality), the following operations are performed, and the operations may include the following steps: extracting facial feature points from vascular information, bone tissue information, and facial intrinsic information including facial structure and facial curvature obtained from an external device; registering the first image to the second image based on the facial feature points; generating a composite image by performing 3D modeling on the first image registered with the second image; projecting the composite image onto the patient's face; and displaying a first layer corresponding to the vascular information and a second layer including the bone tissue information on the projected image.

[0012] In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be provided.

[0013] Technical effects

[0014] According to the above technical solution of the present disclosure, through a face-centered enhancement technology, the complex and delicate structure of the face can be considered, thereby providing a face enhancement technology that minimizes errors.

[0015] According to the above technical solution of the present disclosure, an augmented reality-based medical image processing device, method, and computer program capable of improving the accuracy of coordinate system registration through non-marker-based coordinate system registration can be provided.

[0016] According to the above technical solution of the present disclosure, there is an effect of using augmented reality without setting markers, or an effect of suppressing human errors from occurring throughout the process of maintaining the relative position of the patient's body and the markers after setting markers when using augmented reality.

[0017] According to the present invention, since the image registered with the CT scan by 3D imaging is projected onto the patient's face, the positions and depths of the bone tissue and vascular tissue located inside the face can be visually displayed, thereby enabling precise and safe facial surgery.

[0018] The effects of the present disclosure are not limited to the above-mentioned effects, and those skilled in the art can clearly understand other effects not mentioned through the following description. Brief description of the drawings

[0019] Figure 1 It is a block diagram showing a medical image processing system according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 is a flowchart showing an operation method of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 is a detailed flowchart showing an operation method of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 is a detailed flowchart showing an operation method of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 is a diagram showing the synthesis of a tomographic image and a 3D image based on an image registration operation of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0024] Figure 6 is a diagram showing the projection of a bone tissue layer based on an image synthesis operation of a medical image processing device according to an exemplary embodiment of the present disclosure.

[0025] Figure 7 is a diagram showing the projection of a blood vessel layer based on an image synthesis operation of a medical image processing device according to an exemplary embodiment of the present disclosure. Detailed Description

[0026] Throughout the present disclosure, the same reference numerals refer to the same components. The present disclosure does not describe all elements of the embodiments and will omit general content within the technical field to which the present disclosure pertains or content repeated between embodiments. The terms "unit, module, component, block" used in the specification may be implemented as software or hardware, and according to an embodiment, a plurality of "units, modules, components, blocks" may be implemented as one component, or one "unit, module, component, block" may include a plurality of components.

[0027] Throughout the specification, when it is mentioned that a certain part is "connected" to another part, this includes not only the case of direct connection but also the case of indirect connection, and the indirect connection includes connection via a wireless communication network.

[0028] Also, when it is mentioned that a certain part "includes" a certain component, this means that unless there is a particularly contrary record, other components may also be included, rather than excluding other components.

[0029] Throughout the specification, when it is mentioned that a certain component is "on" another component, this includes not only the case where a certain component is in contact with another component but also the case where there are other components between the two components.

[0030] The terms "first", "second", etc. are used to distinguish one component from another, and the components are not limited by the foregoing terms.

[0031] Unless otherwise clearly stated in the context, singular expressions include plural expressions.

[0032] For each step, the identification symbols are used for convenience of explanation, and the identification symbols are not used to explain the order of each step. Unless a specific order is clearly recorded in the context, each step can be implemented in a different order from the order described above.

[0033] Hereinafter, the principle of operation and embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0034] In this specification, the "medical image processing device" includes all kinds of devices that can perform arithmetic processing and provide results to the user. For example, the medical image processing device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in any form.

[0035] Among them, the computer may include, for example, a notebook computer equipped with a web browser, a desktop computer, a laptop computer, a tablet personal computer, a touch tablet personal computer, etc.

[0036] The server device, as a server that communicates with external devices and processes information, may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a network server, etc.

[0037] For example, as a wireless communication device that ensures portability and mobility, a portable terminal may include all types of handheld-based wireless communication devices such as Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication - 2000, Code Division Multiple Access (CDMA) - 2000, Wideband Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (WiBro) terminals, Smart Phones, etc., as well as wearable devices such as watches, rings, bracelets, ankle bracelets, necklaces, glasses, contact lenses, or head-mounted devices (HMD).

[0038] Augmented Reality (AR) is a technology that superimposes three-dimensional virtual images onto real images or backgrounds and presents them as a single image. This augmented reality technology is applicable to various fields such as gaming, fitness, and map services through a variety of smart devices. Recently, augmented reality technology has been used in the medical field.

[0039] In surgical operations, the surgeries that highlight the advantages of augmented reality are head and neck and facial surgeries. Since the head and neck have complex and delicate structures, in surgeries that require precise anatomical information, by using augmented reality, medical staff can visually confirm medical images. Especially compared with other body parts, the face has clear and numerous anatomical landmarks. Therefore, augmented reality technology is very helpful in determining the distance between the location of the surgical target and the main landmarks.

[0040] The most commonly used implementation of augmented reality in the medical field is the "Marker-based AR" method, which requires firmly setting an artificial marker (Fiducial Marker), which serves as a reference for projecting medical images, on a rigid body. In orthopedic surgery, the surgical target is mainly bones, so markers can be set on the bones.

[0041] However, in head and neck surgery and facial surgery, the target is often soft tissues such as the brain or skin. Even in facial fracture surgery, it is difficult to set markers on the skull. Even when setting markers on the forehead where the bone is close to the epidermal layer and there is almost no fat layer, errors may occur due to human error during the marker setting process. Marker-based AR determines the registration of medical images only based on markers, so there is a difficulty in accurately setting markers. Therefore, a new technology is needed that can use augmented reality without setting markers, or can suppress the occurrence of human error throughout the process of maintaining the relative position of the patient's body and the markers after setting the markers when using augmented reality.

[0042] Figure 1 It is a block diagram showing a medical image processing system 1 according to an exemplary embodiment of the present disclosure.

[0043] Refer to Figure 1 , the medical image processing system 1 may include a medical image processing device 10, a tomography device 20, and a 3D imaging device 30.

[0044] The medical image processing system 1 can acquire medical information of a patient before surgery or treatment, extract feature data required for the treatment from the acquired medical information, and process it into data required during surgery or treatment. According to an exemplary embodiment, the tomography device 20 can reconstruct the human body scan results using X-rays or ultrasonic waves by computer tomography (CT: Computed Tomography), and thus process the internal cross-section of the human body into an image. In the present disclosure, for the sake of convenience of explanation, CT scanning is taken as an example, but the tomography device 20 can refer to the shooting of internal human body images by computer tomography (CT: Computed Tomography), magnetic resonance imaging (MRI: Magnetic Resonance Imaging), and positron emission tomography (PET: Positron Emission Tomography).

[0045] According to an exemplary embodiment, the 3D imaging device 30 can use a stereo camera and / or a depth camera to acquire three-dimensional image information about an object.

[0046] According to an exemplary embodiment of the present disclosure, the medical image processing system 1 can perform marker-less medical image processing by analyzing data of the medical image processing device 10 on tomographic images obtained from the tomographic scanning device 20 and 3D images obtained from the 3D imaging device 30 without the physical setting of artificial markers for extracting medical images. Thus, the medical image processing system 1 has the effect of not requiring marker setting, or has the effect of being able to suppress the occurrence of human errors throughout the process of maintaining the relative position between the patient's body and the marker after the marker is set.

[0047] In an exemplary embodiment of the present disclosure, the medical image processing system 1 can project marker-less medical images onto the patient's face by using augmented reality technology. Thus, the medical image processing system 1 can visually display the position and depth of the bone tissue and vascular tissue located inside the face, and medical staff can use the medical image processing system 1 to perform precise and safe facial surgeries.

[0048] The medical image processing device 10 may include an input interface (I / F: Interface) 110, a feature extraction module 120, an image registration module 130, an image processing module 140, an augmented reality processing module 150, an output interface 160, and a database 170. Figure 1 The medical image processing device 10 is only one embodiment of the present invention, and thus should not be used to Figure 1 make a limiting interpretation of the present invention.

[0049] The input interface 110 is used to input image information (or signals), audio information (or signals), data, or information input by the user, and may include at least one of at least one camera, at least one microphone, and the user input interface 110. The voice data or image data collected from the input interface 110 can be processed into a user control command through analysis.

[0050] In an exemplary embodiment of the present disclosure, the input interface 110 serves as an interface unit and functions as a passage between the device and various types of external devices connected to the device. Such an interface unit may include at least one of a wired / wireless headphone port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting to a device equipped with an identification module (SIM), an audio input / output (I / O: Input / Output) port, a video input / output (I / O) port, and a headphone port. In this device, appropriate control related to the external devices connected to the interface unit can be executed.

[0051] The camera processes image frames such as still images or videos obtained by an image sensor in a shooting mode. The processed image frames can be displayed through a display unit, or can be directly projected onto a patient's face as an optical signal, or can be stored in a memory.

[0052] In addition, in the case where there are multiple cameras, they can be arranged in a matrix structure, and multiple pieces of image information with various angles or focal points can be input through the cameras constituting the matrix structure as described above. Furthermore, the cameras can also be arranged in a stereo structure to obtain a left image and a right image for implementing a three-dimensional stereoscopic image.

[0053] The user input interface 110 serves as an interface for receiving information from the user. When information is input through the user input interface 110, the control unit can control the operation of the device in a manner corresponding to the input information. Such a user input interface 110 can include hardware physical keys (for example, buttons, dome switches, rollers, roller switches, etc., located at least at one of the front surface, rear surface, and side surface of the device) and software touch keys. As an example, the touch keys can be composed of virtual keys, soft keys, or visual keys displayed on a touch screen type display unit through software processing, or can be composed of touch keys arranged in a part other than the touch screen. In addition, the virtual keys or visual keys can have various forms and be displayed on the touch screen. For example, they can be composed of graphics, texts, icons, videos, or combinations thereof.

[0054] The detection unit detects at least one of the internal information of the device, the surrounding environment information around the device, and user information, and generates a corresponding detection signal. The control unit can control the driving or operation of the device based on such a detection signal, or execute data processing, functions, or operations related to application programs set in the device.

[0055] The detection unit as described above may include at least one of a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor: infrared sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera), a microphone, an environmental sensor (e.g., including at least one of a barometer, a hygrometer, a thermometer, a radioactive detection sensor, a thermal detection sensor, a gas detection sensor), a chemical sensor (e.g., a healthcare sensor, a biometric sensor, etc.). In addition, the device may combine and utilize information detected by two or more of these sensors.

[0056] According to an exemplary embodiment of the present disclosure, the input interface 110 may receive a medical information image of a patient's face from an external device. For example, a camera in the external device may acquire a real-time image including information about the space where the patient is located. For example, the camera may acquire a real-time image of the patient's face. For example, the camera may extract at least one feature point from the face image and continuously calculate the position and orientation of the depth camera. As described below, the position and orientation of the camera may be used to track an augmented reality-based medical image.

[0057] The feature extraction module 120 may be configured to extract a feature part from an image. According to an exemplary embodiment, the feature extraction module 120 may extract feature points from features such as blood vessels, bone tissues, facial curvature, various bones, etc. in a medical image, or from a reference body tissue or body part. In the present disclosure, the information of the feature points may include three-dimensional position information in a three-dimensional space, and the three-dimensional position information of the feature points may serve as a marker.

[0058] The image registration module 130 may be configured to register at least two images. According to an exemplary embodiment, the image registration module 130 may register (or match) at least one feature point extracted from two or more images with each other. The image registration module 130 may use a well-known image registration algorithm, or may pre-store a look-up table required for stereo transformation, etc. for image registration, and may load the calculated result value when needed. According to an exemplary embodiment, the image registration module 130 may be configured to downsize sample the first feature point, then select a set of point pairs and quantify the difference of the point pairs as position and direction (i.e., position and normal vector) so as to be indexed, and may store it in the look-up table.

[0059] According to an exemplary embodiment of the present disclosure, the image registration module 130 may register a 3D image and a tomographic image based on feature points. Thus, the image registration module 130 performs coordinate system registration based on unmarked features without using artificial markings. Specifically, feature points within the face may be set as markings to perform coordinate system registration.

[0060] According to an exemplary embodiment of the present disclosure, the image registration module 130 may continuously calculate the registration of the composite image and the facial feature points according to the change of the real-time patient facial image. Usually during surgery or a procedure, since the patient is under anesthesia, the change state is not significant. However, during the surgery or procedure by medical staff, the face may be touched by hands or medical devices, resulting in slight position and angle changes. The image registration module 130 may continuously calculate the registration of the composite image and the facial feature points according to the change of the real-time patient facial image, thereby enabling accurate image projection.

[0061] In an exemplary embodiment, the image registration module 130 may perform primary registration on the composite image and the real-time patient facial image based on the part corresponding to the positions of both eyes. In an exemplary embodiment, to further improve the accuracy of registration, after performing primary registration, the image registration module 130 may perform secondary registration on the composite image and the real-time patient facial image based on the part corresponding to the face other than the positions of both eyes. That is, the image registration module 130 may first perform primary registration based on information about relatively non-deformable reference points (e.g., both eyes), thereby improving the accuracy of registration.

[0062] In an exemplary embodiment, the image registration module 130 may perform registration by using a first registration algorithm and a second registration algorithm. For example, the image registration module 130 may calculate approximate positions of a first feature point and a second feature point based on the first registration algorithm. The first feature point is extracted from a first image based on facial CT and including vascular and bone tissues, and the second feature point is extracted from a second image based on facial 3D imaging. Herein, the approximate positions may refer to positions and orientations.

[0063] The image registration module 130 uses, for example, a Point Pair Feature (PPF) algorithm as the first registration algorithm to calculate approximate positions of the first feature point and the second feature point. Specifically, the image registration module 130 may select a point pair after downsizing sampling of the feature points. Then, the image registration module 130 may quantize the selected point pair (non-continuously combine information in a predetermined interval into single information) to select the same or similar point pair from a look-up table. For example, the image registration module 130 may select a candidate group with the highest similarity score to the quantized point pair, and calculate approximate positions of the first feature point and the second feature point based thereon. In an exemplary embodiment, since the PPF algorithm is used through quantization (non-continuously combine information in a predetermined interval into single information), it is necessary to improve accuracy. In this regard, in the present invention, a second algorithm described below is additionally used together with the PPF algorithm to improve the accuracy of registration.

[0064] After calculating approximate positions of the first feature point and the second feature point, the image registration module 130 may calculate exact positions of a plurality of first feature points corresponding to a reference skin and a plurality of second feature points corresponding to the reference skin based on the second registration algorithm. The image registration module 130 may use, for example, an Iterative Closest Point (ICP) algorithm as the second registration algorithm to calculate exact positions of the plurality of first feature points and the plurality of second feature points, starting from the above approximate positions. Herein, the exact positions may refer to translations and rotations.

[0065] That is, the image registration module 130 detects a position and an orientation as approximate positions through the PPF algorithm, and detects translations and rotations as exact positions through the ICP algorithm, so as to make the distance between points closer. As described above, the present invention uses the PPF algorithm to find an approximate position and uses it as a starting point to perform registration through the ICP algorithm, thereby being able to improve the accuracy of registration.

[0066] As another embodiment, the image registration module 130 may perform registration only using the second registration algorithm. For example, the image registration module 130 may perform pre-registration at a preset number of points to set initial values. Among them, the pre-registration may be performed by the ICP algorithm. For example, the image registration module 130 may perform pre-registration at the upper center of the bounding box (Scene bounding box) corresponding to the real-time image and at the center of the second bounding box. Then, the image registration module 130 may set the position and orientation of the point with the smallest residual error in the result of the pre-registration as the initial values. Subsequently, the image registration module 130 may calculate the exact positions of the plurality of first feature points and the plurality of second feature points by applying the ICP algorithm from the initial values to perform registration.

[0067] The image processing module 140 may perform image processing, which is configured to perform image processing on the data constituting the image according to predefined calculations.

[0068] In an exemplary embodiment, the image processing module 140 may use an anisotropic diffusion filter (ADF: Anisotropic Diffusion Filter) to remove noise from medical images. Different from common noise removal filters, the anisotropic diffusion filter effectively removes noise while retaining edge information by analyzing image information. When extracting feature points from a facial image, the error of facial feature point information caused by noise can be minimized by removing noise from the medical image.

[0069] In an exemplary embodiment, the image processing module 140 may use Otsu's Method to distinguish the background and the patient area in medical images. Otsu's Method is a technique for automatically finding an adaptive threshold for the input image by analyzing the luminance distribution of the input image to perform region division. The thresholding technique using Otsu's Method is a technique that only uses the luminance values of the image without considering the topology of the image. Therefore, the image processing module 140 may sequentially perform seeded region growing (SRG: Seeded Region Growing) calculations and morphology calculations to remove mis-segmented regions while considering the topological information of the image. SRG is a technique for searching by tracing and expanding adjacent pixels that meet the conditions based on a seed point. The morphology calculation performs processes such as removing noise of mis-segmentation and eliminating empty spaces on the segmentation result by repeatedly performing erosion and dilation.

[0070] In an exemplary embodiment, the image processing module 140 may perform patient area segmentation in units of slices of medical images. The entire segmented image is applied as volume data for generating a skin mesh. In an exemplary embodiment, a part of the medical image may be data modeled in the form of a skin mesh. For example, the 3D modeled medical image may be 3D mesh information including three-dimensional vertex information, their connection information, and surface information generated therefrom.

[0071] In an exemplary embodiment, the image processing module 140 may use the Marching Cube technique to generate the skin mesh 205. When generating the skin mesh, the image processing module 140 may apply a Gaussian filter in three dimensions to smooth the volume data. Thus, the effect can be achieved the same as the case of smoothing the mesh generated by the Marching Cube.

[0072] The image processing module 140 may use the Fast-Quadric Mesh Simplification algorithm to perform mesh simplification. By performing mesh simplification, the computational amount and memory can be reduced.

[0073] The augmented reality processing module 150 may be configured to implement and display the processed image in an augmented reality manner.

[0074] The augmented reality processing module 150 may generate an augmented reality-based medical image based on the facial feature points of the patient, so that medical information is output onto the patient's body (especially the face) in a real-time image.

[0075] In an exemplary embodiment, the augmented reality processing module 150 may enable a doctor to confirm the registration accuracy by adjusting the transparency and contrast of the patient body modeling in the real-time image. In addition, the brightness can be adjusted to more easily identify lesions and tissues of interest in the real-time image.

[0076] In an exemplary embodiment, the augmented reality processing module 150 may output at least one of medical information such as a CT plane, a lesion, the position information of the skin closest to the lesion, the position information of the skin in the direction of the lesion and the anatomical axis, a scale for measuring the distance between the lesion and the skin, etc. on the patient's body in the real-time image.

[0077] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may separate and display a first layer corresponding to vascular information and a second layer including bone tissue information on a projection image projected onto a patient's face. For example, when the vascular information as the first layer is projected onto the patient's face in the form of augmented reality, the bone tissue information corresponding to the second layer may be separated and displayed at a time when the first layer is not displayed or in an area where the first layer is not displayed. According to an embodiment, the first layer and the second layer may also be displayed simultaneously.

[0078] In an exemplary embodiment of the present disclosure, if a doctor selects a specific CT slice, the augmented reality processing module 150 may arrange the specific CT slice 401 at a position corresponding to the CT slice position (inside the patient's body) on the patient's body in the real-time image.

[0079] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may construct three-dimensional data by stacking CT slices 401, and may generate and display an anatomical plane image in a desired direction in the data.

[0080] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may change the first color (e.g., red) of a lesion site in the anatomical plane image and display it.

[0081] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may present lines, graphics, text, etc. on the patient's body in the real-time image according to the doctor's input.

[0082] In an exemplary embodiment of the present disclosure, when a doctor inputs lines, graphics, text, etc. on the patient's body in the real-time image, the augmented reality processing module 150 may adjust the transparency of at least one element (a disk for confirming the lesion, size, and position) displayed on the screen in real time. Thereby, the doctor can easily confirm lines, graphics, text, etc. on the patient's body in the real-time image.

[0083] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may track spatial information based on the position and orientation of the depth camera, so as to continuously output medical images based on augmented reality. That is, after registering the 3D modeled medical image data with the real-time image at least once, the augmented reality processing module 150 uses a third algorithm to track the spatial information of each frame, so as to continuously maintain the registration of the 3D modeled data and the real-time image. Among them, the third algorithm may be a simultaneous localization and mapping (SLAM) algorithm. In contrast, in order to track in real time (sense the changes in the position and orientation of the depth camera described later), the real-time image may also be registered frame by frame. Thus, the problem of increased computational complexity caused by registering the 3D modeled data and the real-time image in each frame can be solved.

[0084] In an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may construct the space into three-dimensional data based on the image information (RGB information or depth information) of the space, and may determine the position and orientation of the depth camera from the three-dimensional data. For example, in an exemplary embodiment of the present disclosure, the augmented reality processing module 150 may obtain point information in the three-dimensional space or extract feature points of RGB from the depth image, and then may determine the position and orientation of the depth camera by analyzing the movement between frames.

[0085] The feature extraction module 120, the image registration module 130, the image processing module 140, and the augmented reality processing module 150 may be respectively implemented as programs for controlling operations or reproducing algorithms. In this case, each program may be implemented on a computer through a memory storing each program and at least one processor (not shown) that performs the above operations using the data stored in the memory. At this time, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may also be implemented as a single chip.

[0086] The processor is configured to integrally control the organic operations of various functional units of the electronic device 100, such as processing one or more commands required for controlling the medical imaging processing device 10, performing calculations according to the commands, making judgments according to program logic, etc. The processor can process input or output signals, data, information, etc. through the input / output interface 110 or various processing modules 120, 130, 140, 150, or drive the application programs stored in the database 170 to provide appropriate information or functions to the user or process the same. The processed data can be stored in the memory or used to construct the database 170, or can be transmitted to the outside through the input / output interface 110 or various processing modules 120, 130, 140, 150. Such a processor can be implemented as a general-purpose processor, a dedicated processor, or an application processor, etc. In an exemplary embodiment, the processor can be implemented as a computing processor (such as a central processing unit (CPU), a graphic processing unit (GPU), an application processor (AP), etc.) including dedicated logic circuits (such as a field programmable gate array (FPGA), an application specific integrated circuits (ASIC), etc.), but is not limited thereto. In an exemplary embodiment, the processor also does not exclude being implemented as a digital signal processor (DSP) capable of converting analog signals into digital signals and performing high-speed processing, a micro controller unit (MCU), or a neural processing unit (NPU) dedicated to processing artificial neural networks, etc.

[0087] The processor can perform control through one or a combination of the above components to implement the following Figures 2 to 7 multiple embodiments according to the present disclosure described in

[0088] The output interface (I / F) 160 is used to generate outputs related to vision, audition, or touch, etc., and can include at least one of a display unit, an audio output interface, a tactile module, and an optical output interface. The display unit can implement a touch screen by forming a mutually stacked structure with a touch sensor or forming an integrated structure. Such a touch screen can be used as a user input unit providing an input interface between the device and the user, and can also provide an output interface between the device and the user.

[0089] The display unit displays (outputs) the information processed in this device. For example, the display unit can display the running screen information of an application (as an example, an app) driven in this device or a user interface (UI: User Interface), graphic user interface (GUI: Graphic User Interface) information based on such running screen information.

[0090] The sound output interface can output the audio data received through the communication unit or the audio data stored in the memory, or can output the sound signal related to the functions executed in this device. Such a sound output interface 160 can include a receiver, a speaker, a buzzer, etc.

[0091] The haptic module generates various tactile effects that can be perceived by the user. A representative example of the tactile effect generated by the haptic module can be vibration. The intensity and pattern, etc. of the vibration generated by the haptic module can be controlled by the user's selection or the setting of the control unit. In addition, in addition to vibration, the haptic module can also generate various tactile effects in the following ways: the effects generated by the stimulation of a pin array that moves vertically against the skin surface, the jet force or suction force of air through a jet port or a suction port, the friction against the skin surface, the contact of an electrode, the electrostatic force, etc., and the effects generated by using an element that can absorb or generate heat to reproduce the sense of cold or heat.

[0092] The haptic module can not only transmit the tactile effect through direct contact, but can also be implemented to allow the user to perceive the tactile effect through the muscle sense of a finger or an arm, etc. According to the configuration form of this device, two or more haptic modules can be provided.

[0093] The light output interface 160 can use the light of the light source of this device to output a signal for notifying the occurrence of an event, or, like a projector, output media such as an image to an external screen for display. In an exemplary embodiment, the light output interface 160 can adjust the wavelength, size, and projection position of the light so that the image is stereoscopically projected onto an object to present the image in augmented reality.

[0094] The output interface 160 serves as a passage to various types of external devices connected to this device. Such an interface unit may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio input / output (I / O) port, a video input / output (I / O) port, and a headphone port.

[0095] The database 170 refers to a collection of data that is comprehensively managed for the purpose of storing various types of information for shared use. The database 170 can store data temporarily or semi-permanently. For example, the database 170 can store an operating system (OS) for driving at least one device, data for hosting a website, or data regarding an application (e.g., a web application). Additionally, as described above, the database can store modules in the form of computer code. The database 170 is managed through middleware independent of the application program.

[0096] The database 170 includes a relational database (RDB), a key-value database, an object database, a document database, an in-memory database, etc.

[0097] The database 170 can store data supporting various functions of this device and programs for controlling the operation of the control unit, can store input / output data (e.g., music files, still images, videos, etc.), and can store multiple application programs (application program) or applications (application) driven in this device, data and instructions for the operation of this device. At least a part of these application programs can be downloaded from an external server through wireless communication.

[0098] The database 170 may be implemented as a memory that is separate from or connected to the present apparatus by wire or wirelessly. At this time, the memory may include at least one type of storage medium such as a flash memory type, a hard disk type, a solid state disk type (SSD), a silicon disk drive type (SDD), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0099] Figure 2 is a flowchart showing an operation method of a medical image processing apparatus ( Figure 1 10 in the present disclosure).

[0100] According to Figure 2 an embodiment shown, an augmented reality-based medical image processing method includes steps of performing processing in a time series or in parallel in the medical image processing apparatus 10 shown, and is referred to as an operation method of the medical image processing apparatus. Figure 1

[0101] Figure 1 In step S110, the medical image processing apparatus 10 may acquire a first image including blood vessel information and bone tissue information based on tomography from a first external device. In an exemplary embodiment, the first external device may be a tomography device ( 20 in the present disclosure). In the present disclosure, for ease of explanation, a CT scan is taken as an example, but the tomography device 20 may refer to the taking of internal images of the human body by computed tomography (CT), magnetic resonance imaging (MI), and positron emission tomography (PET).

[0102] According to an exemplary embodiment of the present disclosure, the medical image processing device 10 may extract first facial feature points from a first image. The first facial feature points may be derived from blood vessel information and bone tissue information, where the blood vessel information includes information on the position, type, size, and distribution degree of blood vessels distributed on the face, and the bone tissue information includes information on the position, type, size, and distribution information of the bone tissue constituting the face.

[0103] In step S120, the medical image processing device 10 may obtain a second image including facial structure and facial curvature based on three-dimensional imaging from a second external device. According to an exemplary embodiment, the 3D imaging device 30 may be a stereo camera and / or a depth camera, and the medical image processing device 10 may utilize the shooting results of the stereo camera and / or the depth camera to obtain three-dimensional image information of an object.

[0104] According to an exemplary embodiment of the present disclosure, the medical image processing device 10 may extract second facial feature points from the second image. The second facial feature points may include the facial structure and facial curvature constituting the face, and the second facial feature points may be derived not only from the shape and structure of the bones, but also from the result distribution of the skin covering the bones, subcutaneous fat and muscles, and the skin state including the epidermis and dermis.

[0105] In step S130, the medical image processing device 10 may register the first image to the second image based on the facial feature points. According to an exemplary embodiment, the medical image processing device 10 may extract facial feature points based on the first facial feature points and the second facial feature points. According to an exemplary embodiment, the medical image processing device 10 may extract facial feature points from the blood vessel information, bone tissue information, and facial intrinsic information including facial structure and facial curvature. According to an exemplary embodiment, the medical image processing device 10 may register the first image to the second image based on the facial feature points.

[0106] In an exemplary embodiment, the medical image processing device 10 may perform registration by using a first registration algorithm and a second registration algorithm. For example, the medical image processing device 10 may calculate an approximate position representing the positions and orientations of a first feature point and a second feature point based on the first registration algorithm, where the first feature point is from a first image including blood vessels and bone tissues based on a facial CT, and the second feature point is from a second image including the eyes, nose, mouth, ears, facial bone shape and bones, and facial structure of the face based on a facial 3D imaging. In an exemplary embodiment, after downsizing the feature points by using a Point Pair Feature (PPF) algorithm, the medical image processing device 10 may select and quantify point pairs, and calculate the approximate position based on the similarity between the same or similar point pairs in a look-up table. In an exemplary embodiment, the medical image processing device 10 may use an Iterative Closest Point (ICP) algorithm to calculate the exact positions of a plurality of first feature points and a plurality of second feature points.

[0107] In step S140, the medical image processing device 10 may generate a synthetic image by performing 3D modeling on the first image registered with the second image. In an exemplary embodiment, the medical image processing device 10 may generate a synthetic image as an object to be projected onto the actual face of the patient in augmented reality, as a registration result of the first image based on tomography and the second image based on 3D imaging. The image processing results according to step S130 and step S140 will be described in more detail in Figure 5 more detail.

[0108] In step S150, the medical image processing device 10 may project the synthetic image onto the face of the patient. According to an exemplary embodiment, the medical image processing device 10 may obtain a real-time patient face image from an external device. According to an exemplary embodiment, the medical image processing device 10 may detect facial feature points from the real-time patient face image. According to an exemplary embodiment, the medical image processing device 10 may continuously calculate the registration of the synthetic image and the facial feature points according to the change of the real-time patient face image.

[0109] According to an exemplary embodiment, as a step of continuously calculating the registration of the synthetic image and the facial feature points according to the change of the real-time patient face image, the medical image processing device 10 may perform a primary registration on the synthetic image and the real-time patient face image based on a part corresponding to the positions of the two eyes.

[0110] According to an exemplary embodiment, after performing one-time registration, the medical image processing device 10 may perform secondary registration on the synthesized image and the real-time patient facial image based on the part corresponding to the face other than the positions of the two eyes.

[0111] In an exemplary embodiment, the medical image processing device 10 may perform one-time registration on the synthesized image and the real-time patient facial image based on the part corresponding to the positions of the two eyes. In an exemplary embodiment, in order to further improve the accuracy of registration, the image registration module 130 may perform secondary registration on the synthesized image and the real-time patient facial image based on the part corresponding to the face other than the positions of the two eyes after performing one-time registration. That is, the image registration module 130 may first perform one-time registration based on the information of relatively non-deformable reference points (for example, the two eyes), thereby improving the accuracy of registration.

[0112] In step S160, the medical image processing device 10 may display a first layer corresponding to the blood vessel information and a second layer including bone tissue information on the projection image. According to an exemplary embodiment, the medical image processing device 10 may display the first layer. According to an exemplary embodiment, the medical image processing device 10 may separate and display the second layer at a time or in an area where the first layer is not displayed. Regarding steps S150 and S160, they will be described in more detail in Figure 6 and Figure 7 which will be described in more detail.

[0113] Corresponding to Figure 2 the performance of the components shown in, at least one component may be added or deleted. In addition, those skilled in the art with ordinary knowledge in this field will easily understand that the relative positions of the components may be changed corresponding to the performance or structure of the system.

[0114] In addition, Figure 2 each of the components shown in refers to software and / or hardware components such as a field programmable gate array (FPGA) and an application specific integrated circuit (ASIC).

[0115] According to the above technical solution of the present disclosure, a face enhancement technology with minimized errors can be provided by an enhancement technology centered on the face, taking into account the complex and delicate structure of the face.

[0116] According to the above technical solution of the present disclosure, an augmented reality-based medical image processing device, method, and computer program capable of improving the accuracy of coordinate system registration through non-marker-based coordinate system registration can be provided.

[0117] According to the above technical solution of the present invention, there is an effect of using augmented reality without setting a marker, or an effect of being able to suppress human error during the entire process of using augmented reality while maintaining the relative position of the patient's body and the marker after setting the marker.

[0118] According to the present invention, since the image registered with the 3D image and the CT image is projected onto the patient's face, the position and depth of the bone tissue and blood vessel tissue located inside the face can be visually displayed, thereby enabling precise and safe facial surgery.

[0119] Figure 3 It is a detailed flowchart showing step S130 in the operation method of a medical image processing device ( Figure 1 10 in).

[0120] After step S120 is executed, in step S210, the medical image processing device 10 may extract at least one first feature point corresponding to the bone tissue and blood vessels from the first image. According to an exemplary embodiment, the medical image processing device 10 may extract the first feature point from a first image based on tomography containing blood vessel information and bone tissue information from an external device.

[0121] In step S230, the medical image processing device 10 may extract at least one second feature point corresponding to the skin regions of the eyes, nose, mouth, and ears of the human body based on anatomical features. According to an exemplary embodiment, the medical image processing device 10 may obtain a second image containing facial structure and facial curvature according to three-dimensional imaging from an external device, and the 3D imaging device 30 may be a stereo camera and / or a depth camera. The medical image processing device 10 may use the shooting results of the stereo camera and / or the depth camera to derive at least one second feature point including the skull, nasal bone, auricle, and mandible based on anatomical features for the obtained three-dimensional image information result of the object. The skull, nasal bone, auricle, and mandible are elements that determine facial curvature and facial structure, and the second feature point may be derived from the shape and structure of the bone, the result distribution of the skin covering the bone, subcutaneous fat and muscle, and the skin state including the epidermis and dermis. The second feature point may be derived from the characteristics of the image, or may be extracted as result data processed through a hidden layer through feature extraction (Feature Extraction) of a vision-based artificial intelligence learning model (for example, a convolutional neural network (CNN: Convolutional Neural Network)).

[0122] According to an exemplary embodiment of the present disclosure, the medical image processing device 10 may extract second feature points from a first image. The second feature points may be derived from vascular information and skeletal tissue information. The vascular information includes information on the positions, types, sizes, and distribution degrees of blood vessels distributed on the face, and the skeletal tissue information may include information on the positions, types, sizes, and distribution information of the skeletal tissues constituting the face.

[0123] In step S250, the medical image processing device 10 may extract facial feature points based on the first feature points and the second feature points. According to an exemplary embodiment, the medical image processing device 10 may register the facial feature points based on the skin surface derived from the second image. According to an exemplary embodiment, the medical image processing device 10 may calculate the approximate positions of at least one second feature point including the skull, nasal bone, auricle, and mandible and at least one first feature point including skeletal tissue and blood vessels based on a first registration algorithm and anatomical features. According to an exemplary embodiment, the medical image processing device 10 may calculate the exact positions of at least one first feature point and at least one second feature point based on a second registration algorithm. Regarding the first registration algorithm and the second registration algorithm, detailed descriptions have been provided in Figure 1 and Figure 2 and thus repeated descriptions are omitted.

[0124] Figure 4 is a detailed flowchart showing step S230 in the operation method of the medical image processing device according to an exemplary embodiment of the present disclosure.

[0125] In step S231, the medical image processing device 10 may extract stable feature points within the face with less deformation. According to an exemplary embodiment, usually during surgery or a procedure, since the patient is under anesthesia, the state of change is not significant. However, during the surgery or procedure by medical staff, the face may undergo slight position and angle changes due to contact with hands or medical devices. Even in the case of movement or change, the medical image processing device 10 can set the eyes, which have less deformation and are easy to detect, as reference points, and extract stable feature points based on the positions of the eyes and the size ratio to the entire face. In the present disclosure, only the eyes are taken as an example for the sake of convenience of description, but when observing from above the face, any internal body tissues, blood vessels, muscles, structures, etc. that are easy to detect and have a small degree of change can be used as stable feature points.

[0126] In step S233, the medical image processing device 10 may adjust the second feature points based on the coordinates and contours of the stable feature points. According to an exemplary embodiment, the stable feature points may be the positions and sizes of the entire facial contour relative to the eyes.

[0127] Figure 5FIG. is a diagram showing a synthesis of a tomographic image and a 3D image of an image registration operation based on a medical image processing apparatus according to an exemplary embodiment of the present disclosure.

[0128] Referring also to Figure 2 , in step S130, the medical image processing apparatus 10 may register a first image to a second image based on facial feature points. According to an exemplary embodiment, the medical image processing apparatus 10 may extract facial feature points from among vascular information, bone tissue information, and facial intrinsic information including facial structure and facial curvature. According to an exemplary embodiment, the medical image processing apparatus 10 may register a first image to a second image based on facial feature points. In step S140, the medical image processing apparatus 10 may generate a synthesized image by performing 3D modeling on the first image registered to the second image. In step S150, the medical image processing apparatus 10 may project the synthesized image onto the patient's face.

[0129] In an exemplary embodiment, the medical image processing apparatus 10 may acquire a real-time patient facial image from an external device and may detect facial feature points from the real-time patient facial image. The medical image processing apparatus 10 may continuously calculate the registration of the synthesized image and the facial feature points according to changes in the real-time patient facial image.

[0130] In an exemplary embodiment, the medical image processing apparatus 10 may perform a primary registration of the synthesized image and the real-time patient facial image based on a portion corresponding to the positions of both eyes, and then may perform a secondary registration of the synthesized image and the real-time patient facial image based on a portion corresponding to the face other than the positions of both eyes.

[0131] In an exemplary embodiment, the medical image processing apparatus 10 may use a simultaneous localization and mapping (SLAM) algorithm to project the synthesized image as augmented reality onto the patient's actual face.

[0132] Figure 6 FIG. is a diagram showing a projection of a bone tissue layer of an image synthesis operation based on a medical image processing apparatus 10 according to an exemplary embodiment of the present disclosure, Figure 7 FIG. is a diagram showing a projection of a blood vessel layer of an image synthesis operation based on a medical image processing apparatus according to an exemplary embodiment of the present disclosure.

[0133] As a step of displaying a first layer corresponding to blood vessel information and a second layer including bone tissue information on a projected image, the medical image processing apparatus 10 may display the first layer. In an exemplary embodiment, the medical image processing apparatus 10 may separate the second layer and display it at a time or in an area where the first layer is not displayed. In an exemplary embodiment, the medical image processing apparatus 10 may obtain a real-time patient facial image from a second external device.

[0134] Referring to Figure 6 , the medical image processing apparatus 10 may project facial bones constituting bone tissue and / or facial structure photographed from a patient onto corresponding positions on the actual face of the patient by displaying the second layer.

[0135] Referring to Figure 7 , the medical image processing apparatus 10 may project actual blood vessel tissue photographed from a patient onto corresponding positions on the actual face of the patient by displaying the first layer.

[0136] In addition, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable commands. The commands may be stored in the form of program code, and when executed by a processor, may perform the operations of the disclosed embodiments by generating program modules. The recording medium may be implemented as a computer-readable recording medium.

[0137] The computer-readable recording medium includes all kinds of recording media storing computer-readable instructions. For example, there may be a read-only memory (ROM), a random access memory (RAM), magnetic tapes, magnetic disks, flash memories, optical data storage devices, and the like.

[0138] As described above, the disclosed embodiments have been illustrated with reference to the accompanying drawings. Those of ordinary skill in the technical field to which the present disclosure pertains should understand that the present disclosure may be implemented in a form different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are merely exemplary and should not be construed in a limiting sense.

Claims

1. A method for operating a medical image processing device, which is a method for operating a medical image processing device using augmented reality (AR: Augmented Reality), characterized in that, It includes the following steps: Obtain a first image based on tomography containing vascular information and bone tissue information from a first external device; Obtain a second image based on three-dimensional imaging containing facial structure and facial curvature from a second external device; Extract facial feature points from the vascular information, the bone tissue information, and the facial intrinsic information including the facial structure and the facial curvature; Register the first image to the second image based on the facial feature points; Generate a synthetic image by performing three-dimensional modeling on the first image registered to the second image; Project the synthetic image onto the patient's face; And Display a first layer corresponding to the vascular information and a second layer containing the bone tissue information on the projected image.

2. The method for operating a medical image processing device according to claim 1, characterized in that, The step of extracting facial feature points from the facial intrinsic information includes the following steps: Extract at least one first feature point corresponding to bone tissue and blood vessels from the first image; Based on anatomical features, extract at least one second feature point corresponding to the skin regions of the eyes, nose, mouth, and ears of the human body from the second image; And Extract the facial feature points based on the first feature points and the second feature points.

3. The method for operating a medical image processing device according to claim 2, characterized in that, The step of registering the first image to the second image based on the facial feature points includes the following steps: Register the facial feature points based on the skin surface derived from the second image.

4. The method for operating a medical image processing device according to claim 2, characterized in that, The step of extracting the at least one second feature point includes the following steps: Extract stable feature points within the face with less deformation; and Adjust the second feature points based on the coordinates and contours of the stable feature points, wherein the stable feature points are the positions and sizes of the eyes relative to the entire facial contour.

5. The method for operating a medical image processing device according to claim 1, characterized in that, The facial feature points function as markers for image registration.

6. The method for operating a medical image processing device according to claim 1, characterized in that, The step of projecting the synthetic image onto the patient's face includes the following steps: Obtain a real-time patient facial image from the second external device; Detect the facial feature points from the real-time patient facial image; and Continuously calculate the registration of the synthetic image and the facial feature points according to the changes of the real-time patient facial image, wherein the step of continuously calculating the registration of the synthetic image and the facial feature points according to the changes of the real-time patient facial image includes the following steps: Perform a primary registration on the synthetic image and the real-time patient facial image based on the part corresponding to the positions of the eyes; and After performing the primary registration, perform a secondary registration on the synthetic image and the real-time patient facial image based on the part corresponding to the face other than the positions of the eyes.

7. The operating method of the medical image processing device according to claim 1, characterized in that, The step of registering the first image to the second image based on the facial feature points includes the following steps: Calculate the approximate positions of at least one first feature point corresponding to bone tissue and blood vessels and at least one second feature point extracted from the second image corresponding to the skin regions of the eyes, nose, mouth, and ears of the human body based on a first registration algorithm; and Calculate the exact positions of the at least one first feature point and the at least one second feature point based on a second registration algorithm.

8. The operating method of the medical image processing device according to claim 1, characterized in that, The steps of displaying a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image include the following steps: Displaying the first layer; and Separating and displaying the second layer at a time or in an area where the first layer is not displayed.

9. A medical image processing device, as a medical image processing device using augmented reality (AR), characterized in that, Comprising: A memory storing a feature extraction module, an image registration module, an image processing module, and an augmented reality processing module, the feature extraction module configured to extract feature parts from an image, the image registration module configured to register at least two images, the image processing module configured to perform image processing on the data constituting the image according to predefined calculations, and the augmented reality processing module configured to implement and display the processed image in an augmented reality manner; A processor including more than one core and controlling the operations of the feature extraction module, the image registration module, the image processing module, and the augmented reality processing module, wherein the memory obtains a first image including blood vessel information and bone tissue information and a second image including facial structure and facial curvature from an external device, the processor extracts facial feature points from the blood vessel information, the bone tissue information, and the facial intrinsic information including the facial structure and the facial curvature, registers the first image to the second image based on the facial feature points, generates a composite image by performing three-dimensional modeling on the first image registered to the second image, projects the composite image onto the patient's face, and displays a first layer corresponding to the blood vessel information and a second layer including the bone tissue information on the projected image.

10. The medical image processing device according to claim 9, characterized in that, When the processor extracts the facial feature points, the processor extracts at least one first feature point corresponding to bone tissue and blood vessels from the first image, extracts at least one second feature point corresponding to the skin areas of the eyes, nose, mouth, and ears of the human body from the second image based on anatomical features, and extracts the facial feature points based on the first feature points and the second feature points.

11. The medical image processing device according to claim 10, characterized in that, When the processor registers the first image to the second image based on the facial feature points, the processor registers the facial feature points based on the skin surface derived from the second image.

12. The medical image processing device according to claim 10, characterized in that, When the processor extracts the at least one second feature point, the processor extracts stable feature points within the face with less deformation and adjusts the second feature points based on the coordinates and contours of the stable feature points, where the stable feature points are the positions and sizes of the eyes relative to the entire facial contour.

13. The medical image processing device according to claim 9, characterized in that, The facial feature points function as markers for image registration.

14. The medical image processing device according to claim 9, characterized in that, The memory further obtains a real-time patient facial image from the external device, and, when the processor projects the composite image onto the patient's face, the processor detects the facial feature points from the real-time patient facial image and continuously calculates the registration of the composite image and the facial feature points according to the changes in the real-time patient facial image.

15. The medical image processing device according to claim 9, characterized in that, When registering the first image to the second image, the processor calculates, based on a first registration algorithm, at least one first feature point corresponding to bone tissue and blood vessels and approximate positions of at least one second feature point extracted from the second image corresponding to skin regions of the eyes, nose, mouth, and ears of the human body, and calculates the exact positions of the at least one first feature point and the at least one second feature point based on a second registration algorithm.

Images