Medical image display device and method for displaying medical images using the same

Abstract

The disclosed embodiments relate to a medical image output device and a medical image output method, which match and output an ultrasound image and a pre-acquired medical image, wherein the medical image output method using the disclosed medical image output device may include the following steps: sending an ultrasound signal to an object by using an ultrasound probe of the medical image output device, and receiving an ultrasound echo signal from the object; obtaining a first ultrasound image by utilizing the ultrasound echo signal; performing image matching on the pre-acquired first medical image and the first ultrasound image; obtaining a second ultrasound image of the object by utilizing the ultrasound probe; obtaining a second medical image by modifying the first medical image to correspond to the second ultrasound image; and displaying the second medical image together with the second ultrasound image.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a medical image display apparatus and a method of displaying a medical image using the same. Background Art

[0002] The ultrasound diagnostic apparatus transmits ultrasound signals generated by a transducer of a probe to a subject and detects information about the signals reflected from the subject, thereby obtaining at least one image of an internal portion (eg, soft tissue or blood flow) of the subject.

[0003] Ultrasound imaging devices are compact and affordable, and can display images in real time. Furthermore, since such ultrasound imaging devices are very safe due to the lack of radiation exposure, such devices have been widely used along with other types of diagnostic imaging devices, such as X-ray diagnostic devices, computed tomography (CT) scanners, magnetic resonance imaging (MRI) devices, and nuclear medicine diagnostic devices.

[0004] Because ultrasound images have a low signal-to-noise ratio (SNR), limitations can be compensated by image registration with CT images or MR images. Image registration is performed by extracting features from ultrasound images and CT / MR images and matching the extracted features with each other.

[0005] However, when the shape of an organ is deformed or its position is moved by an ultrasound probe, it is difficult to easily perform image registration. Summary of the Invention

[0006] Technical issues

[0007] Provided are a medical image display apparatus and a medical image display method for registering an ultrasound image with a previously acquired medical image and outputting a registration result.

[0008] Technical Solution

[0009] According to one aspect of the present disclosure, a medical image display method includes: sending an ultrasonic signal to an object via an ultrasonic probe of a medical image display device and receiving an ultrasonic echo signal from the object; obtaining a first ultrasonic image based on the ultrasonic echo signal; performing image registration between the first ultrasonic image and a previously obtained first medical image; obtaining a second ultrasonic image of the object via the ultrasonic probe; obtaining a second medical image by transforming the first medical image to correspond to the second ultrasonic image; and displaying the second medical image together with the second ultrasonic image.

[0010] According to another aspect of the present disclosure, a medical image display device includes: a display; an ultrasound probe configured to send an ultrasound signal to an object and receive an ultrasound echo signal from the object; a memory storing one or more instructions; and a processor configured to execute the one or more instructions to perform the following operations: obtain a first ultrasound image based on the ultrasound echo signal; perform image registration between the first ultrasound image and a previously obtained first medical image; control the ultrasound probe to obtain a second ultrasound image of the object; obtain a second medical image by transforming the first medical image to correspond to the second ultrasound image; and control the display to display the second medical image together with the second ultrasound image.

[0011] According to another aspect of the present disclosure, a computer-readable recording medium has recorded thereon a program for executing at least one of the medical image display methods according to the embodiments presented in the present disclosure on a computer.

[0012] The application stored in the recording medium may be intended to perform a function according to at least one of the medical image display methods of the embodiments presented in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram illustrating an ultrasonic diagnostic apparatus according to an exemplary embodiment;

[0014] Figure 2a 、 Figure 2b and Figure 2c are diagrams illustrating an ultrasonic diagnostic apparatus according to an exemplary embodiment;

[0015] Figure 3 is a flowchart of a method for a medical image display apparatus to output a medical image according to an embodiment;

[0016] Figure 4 is a flowchart of a method for registering an ultrasound image with a medical image by a medical image display device according to an embodiment;

[0017] Figure 5 An example of applying an ultrasound image to an artificial intelligence (AI) model by a medical image display apparatus according to an embodiment is shown;

[0018] Figure 6 An example is shown in which the medical image display apparatus according to the embodiment displays information on whether a condition for image registration is satisfied;

[0019] Figure 7 An example is shown in which the medical image display apparatus according to the embodiment displays information on whether a condition for image registration is satisfied;

[0020] Figure 8 An example is shown in which the medical image display apparatus according to the embodiment identifies whether a condition for image registration is satisfied;

[0021] Figure 9 An example is shown in which the medical image display apparatus according to the embodiment identifies whether a condition for image registration is satisfied;

[0022] Figure 10 An example of applying a medical image to an AI model by a medical image display device according to an embodiment is shown;

[0023] Figure 11 An example is shown in which the medical image display apparatus according to the embodiment performs image registration between an ultrasound image and a medical image;

[0024] Figure 12 is a flowchart of a method in which a medical image display apparatus applies a correction operation to a medical image according to an embodiment; and

[0025] Figure 13 A result obtained when the medical image display apparatus applies the result of the correction operation to the medical image according to the embodiment is shown. DETAILED DESCRIPTION

[0026] This specification describes the principles of the present disclosure and illustrates its embodiments to clarify the scope of the claims of the present disclosure and enable those skilled in the art to implement the embodiments of the present disclosure.The embodiments of the present disclosure may have different forms.

[0027] Certain exemplary embodiments are described in more detail below with reference to the accompanying drawings.

[0028] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the description, such as specific configurations and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. Therefore, it is readily understood that the exemplary embodiments may be practiced without those specifically defined matters. In addition, well-known functions or configurations are not described in detail because they may obscure the exemplary embodiments with unnecessary detail.

[0029] Terms such as "component" and "part" used herein indicate terms that can be implemented by software or hardware. According to exemplary embodiments, multiple components or parts may be implemented by a single unit or element, or a single component or part may include multiple elements.

[0030] Expressions such as “at least one of,” preceding a series of elements, modify the entire series of elements and do not modify the individual elements of the series.

[0031] Some embodiments of the present disclosure can be described in terms of functional block components and various processing operations. Some or all of these functional blocks can be implemented by any number of hardware components and / or software components that perform specific functions. For example, the functional blocks of the present disclosure can be implemented by one or more microprocessors or by circuit components for performing specific functions. For example, the functional blocks according to the present disclosure can be implemented using any programming or scripting language. Various algorithms executed on one or more processors can be used to implement the functional blocks. In addition, the present disclosure can adopt technologies in related fields for electronic configuration, signal processing and / or data processing. The terms "mechanism", "element", "device" and "construction" are used in a broad sense and are not limited to mechanical embodiments or physical embodiments.

[0032] Throughout the specification, it will be understood that when a component is referred to as being “connected” or “coupled” to another component, it may be “directly connected” to the other component, or “electrically coupled” to the other component with one or more intermediate elements interposed therebetween. Throughout the specification, when a component “includes” or “comprising” an element, unless there is a specific description to the contrary, the component may also include other elements without excluding other elements.

[0033] In addition, the connecting lines or connectors shown in the various figures are intended to represent exemplary functional relationships and / or physical or logical couplings between the components in the figures. In actual devices, the connections between components may be represented by alternative or additional functional relationships, physical connections or logical connections.

[0034] In an exemplary embodiment, the image may include any medical image acquired by various medical imaging devices, such as a magnetic resonance imaging (MRI) device, a computed tomography (CT) device, an ultrasound imaging device, or an X-ray device.

[0035] In addition, in this specification, the "object" as a thing to be imaged may include a person, an animal, or a part thereof. For example, the object may include a part of a person (ie, an organ or tissue) or a phantom.

[0036] Throughout the specification, an ultrasound image refers to an image of an object processed based on an ultrasound signal transmitted to and reflected from the object.

[0037] In this specification, a medical image display apparatus is an electronic device capable of outputting at least one of a medical image stored therein, a medical image received via a network, and a medical image obtained from a subject.

[0038] For example, the medical image display device may include a medical imaging device (such as an ultrasound imaging device capable of obtaining ultrasound images). Alternatively, the medical image display device may include a computing device (such as a general-purpose computer (e.g., PC) and a mobile device (e.g., a smartphone, a tablet personal computer (PC), etc.)) that outputs a medical image obtained from a server (e.g., a medical image transmission system (such as a picture archiving and communication system (PACS))) via a network.

[0039] According to an embodiment, the image registration presented in the specification may be performed using a visual understanding technology of an AI technology.

[0040] AI technology includes machine learning (deep learning) technology, which uses algorithms for autonomously classifying / learning features of input data and elements, and technologies for simulating the functions of the human brain (such as cognition and decision-making) by using machine learning algorithms. Within the field of AI, visual understanding is a technology for recognizing and processing objects in the same way as the human visual system, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, image enhancement, etc.

[0041] According to the present disclosure, functions related to AI can be operated via a processor and a memory. The processor can be configured as one or more processors. In this case, the one or more processors can be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) or a visual processing unit (VPU)), or a dedicated AI processor (such as a neural processing unit (NPU)). One or more processors control the input data to be processed according to predefined operating rules or AI models stored in the memory. Optionally, when one or more processors are dedicated AI processors, the dedicated AI processor may be designed with a hardware structure specifically for processing a specific AI model.

[0042] Predefined operating rules or AI models can be created through a training process. This means that predefined operating rules or AI models configured to perform desired characteristics (or purposes) are created by training a basic AI model based on a large amount of training data using a learning algorithm. The training process can be performed by the device in which the AI ​​is executed itself or via a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0043] The AI ​​model may be composed of multiple neural network layers. Each neural network layer has multiple weight values, and the neural network calculation may be performed via calculations between the calculation results in the previous layer and the multiple weight values. The multiple weight values ​​assigned to each neural network layer may be optimized by training the results of the AI ​​model. For example, the multiple weight values ​​may be modified to reduce or minimize the loss value or cost value obtained by the AI ​​model during the training process. The artificial neural network may include a deep neural network (DNN), and may be, for example, a convolutional neural network (CNN), a DNN, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive DNN (BRDNN), or a deep Q network (DQN), but is not limited thereto.

[0044] The AI ​​model presented herein can be created by learning a plurality of text data and image data as training data input according to predefined criteria. The AI ​​model can generate result data by executing a learning function in response to the input data and output the result data.

[0045] Furthermore, the AI ​​model may include a plurality of AI models trained to perform at least one function.

[0046] According to an embodiment, an AI model may be constructed in a medical image display apparatus. The medical image display apparatus may perform registration between ultrasound images and CT / MR images by using the AI ​​model and display the registered images.

[0047] According to an embodiment, the medical image display apparatus may transmit the obtained medical image to an electronic device (e.g., a server) that constructs an AI model, and output the medical image by using data received from the electronic device.

[0048] For example, the medical image display apparatus may transmit an ultrasound image obtained from a subject to a server, receive a CT / MR image registered with the ultrasound image from the server, and display the resulting CT / MR image.

[0049] It will be understood that although terms including ordinal numbers such as "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another.

[0050] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0051] Figure 1 is a block diagram illustrating a configuration of an ultrasonic diagnostic apparatus 100 (ie, a diagnostic apparatus) according to an exemplary embodiment.

[0052] Reference Figure 1The ultrasound diagnostic apparatus 100 may include a probe 20, an ultrasound transceiver 110, a controller 120, an image processor 130, one or more displays 140, a storage 150 (e.g., memory), a communicator 160 (i.e., a communication device or communication interface), and an input interface 170.

[0053] The ultrasound diagnostic apparatus 100 may be a cart-type ultrasound diagnostic apparatus or a portable ultrasound diagnostic apparatus (which is portable, movable, mobile, or handheld). Examples of the portable ultrasound diagnostic apparatus 100 may include a smartphone, a laptop computer, a personal digital assistant (PDA), and a tablet personal computer (PC), each of which may include a probe and a software application, but the embodiment is not limited thereto.

[0054] The probe 20 may include a plurality of transducers. The plurality of transducers may transmit an ultrasonic signal to the object 10 in response to a transmission signal received by the probe 20 from the transmitter 113. The plurality of transducers may receive the ultrasonic signal reflected from the object 10 to generate a reception signal. In addition, the probe 20 and the ultrasonic diagnostic apparatus 100 may be formed in one body (for example, provided in a single housing), or the probe 20 and the ultrasonic diagnostic apparatus 100 may be formed separately (for example, provided separately in separate housings) but linked wirelessly or via a wire. In addition, according to an embodiment, the ultrasonic diagnostic apparatus 100 may include one or more probes 20.

[0055] The controller 120 may control the transmitter 113 to cause the transmitter 113 to generate a transmission signal to be applied to each of the plurality of transducers based on the positions and focal points of the plurality of transducers included in the probe 20 .

[0056] The controller 120 may control the receiver 115 based on the positions and focal points of the plurality of transducers to generate ultrasound data by converting reception signals received from the probe 20 from analog signals to digital signals and summing the reception signals converted into digital form.

[0057] The image processor 130 may generate an ultrasound image by using the ultrasound data generated from the receiver 115 .

[0058] The display 140 may display a generated ultrasound image and various information processed by the ultrasound diagnostic apparatus 100. According to the present exemplary embodiment, the ultrasound diagnostic apparatus 100 may include two or more displays 140. The display 140 may include a touch screen combined with a touch panel.

[0059] The controller 120 may control the operation of the ultrasonic diagnostic apparatus 100 and the signal flow between the internal elements of the ultrasonic diagnostic apparatus 100. The controller 120 may include a memory for storing programs or data for performing the functions of the ultrasonic diagnostic apparatus 100 and a processor and / or microprocessor (not shown) for processing the programs or data. For example, the controller 120 may control the operation of the ultrasonic diagnostic apparatus 100 by receiving a control signal from the input interface 170 or an external device.

[0060] Additionally, the controller 120 may include an AI model 125 that registers ultrasound images with medical images from different modalities.

[0061] The AI ​​model 125 may be trained to identify whether a condition for performing image registration between an ultrasound image and a medical image is satisfied and output a result of the identification.

[0062] The AI ​​model 125 may be trained to obtain features in ultrasound images and perform image registration by comparing and matching the obtained features with corresponding features in medical images.

[0063] The AI ​​model 125 can be trained to perform correction operations on the medical image by comparing features in the ultrasound image with corresponding features in the medical image. The AI ​​model 125 can be trained to perform shape correction on the medical image by applying the results of the correction operations to the medical image.

[0064] The ultrasound diagnostic apparatus 100 may include a communicator 160 and may be connected to external apparatuses (eg, servers, medical apparatuses, and portable devices such as smartphones, tablet personal computers (PCs), wearable devices, etc.) via the communicator 160 .

[0065] The communicator 160 may include at least one element capable of communicating with an external device. For example, the communicator 160 may include at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0066] The communicator 160 may receive a control signal and data from an external device and transmit the received control signal to the controller 120 , so that the controller 120 may control the ultrasound diagnostic apparatus 100 in response to the received control signal.

[0067] The controller 120 may transmit a control signal to the external device via the communicator 160 , so that the external device may be controlled in response to the control signal of the controller 120 .

[0068] For example, the external device connected to the ultrasound diagnostic apparatus 100 may process data of the external device in response to a control signal of the controller 120 received via the communicator 160 .

[0069] A program for controlling the ultrasonic diagnostic apparatus 100 may be installed in an external device. The program may include a command language for executing a portion of the operation of the controller 120 or the entire operation of the controller 120.

[0070] The program may be pre-installed in the external device, or may be installed by a user of the external device by downloading the program from a server providing the application. The server providing the application may include a recording medium storing the program.

[0071] The memory 150 may store various data or programs for driving and controlling the ultrasound diagnostic apparatus 100 , input ultrasound data and / or output ultrasound data, ultrasound images, applications, and the like.

[0072] The input interface 170 may receive user input for controlling the ultrasonic diagnostic apparatus 100 and may include a keyboard, a button, a keypad, a mouse, a trackball, a jog switch, a knob, a touchpad, a touch screen, a microphone, a motion input device, a biometric input device, etc. For example, the user input may include an input for manipulating a button, a keypad, a mouse, a trackball, a jog switch, or a knob, an input for touching a touchpad or a touch screen, a voice input, a motion input, and a biometric information input (e.g., iris recognition or fingerprint recognition), but exemplary embodiments are not limited thereto.

[0073] The sensor 190 may include at least one sensor capable of obtaining information about the position of the probe 20. For example, the sensor 190 may include a magnetic field generator for generating a magnetic field within a specific range, an electromagnetic sensor for detecting electromagnetic induction in the magnetic field, and a position tracker for tracking the position of the electromagnetic sensor.

[0074] Refer to the following Figure 2a 、 Figure 2b and Figure 2c An example of the ultrasonic diagnostic apparatus 100 according to the present exemplary embodiment is described.

[0075] Figure 2a 、 Figure 2b and Figure 2c is a diagram illustrating an ultrasonic diagnostic apparatus according to an exemplary embodiment.

[0076] Reference Figure 2a and Figure 2bUltrasonic diagnostic devices 100a and 100b may include a main display 121 and a sub-display 122. At least one of the main display 121 and the sub-display 122 may include a touch screen. The main display 121 and the sub-display 122 may display ultrasonic images and / or various information processed by the ultrasonic diagnostic devices 100a and 100b. The main display 121 and the sub-display 122 may provide a graphical user interface (GUI) to receive user data input for controlling the ultrasonic diagnostic devices 100a and 100b. For example, the main display 121 may display an ultrasonic image, and the sub-display 122 may display a control panel for controlling the display of the ultrasonic image as a GUI. The sub-display 122 may receive data input for controlling the display of the image via the control panel displayed as a GUI. The ultrasonic diagnostic devices 100a and 100b may control the display of the ultrasonic image on the main display 121 by using the input control data.

[0077] Reference Figure 2b The ultrasound diagnostic apparatus 100b may include a control panel 165. The control panel 165 may include buttons, a trackball, a push switch, or a knob, and may receive data from a user for controlling the ultrasound diagnostic apparatus 100b. For example, the control panel 165 may include a time gain compensation (TGC) button 171 and a freeze button 172. The TGC button 171 is used to set a TGC value for each depth of the ultrasound image. In addition, when an input of the freeze button 172 is detected during scanning of the ultrasound image, the ultrasound diagnostic apparatus 100b may continue to display the frame image at that time point.

[0078] Buttons, a trackball, a jog switch, and a knob included in the control panel 165 may be provided as a GUI to the main display 121 or the sub display 122 .

[0079] Reference Figure 2c The ultrasonic diagnostic apparatus 100c may include a portable device. Examples of the portable ultrasonic diagnostic apparatus 100c may include, for example, a smartphone, a laptop computer, a personal digital assistant (PDA), or a tablet PC including a probe and an application, but exemplary embodiments are not limited thereto.

[0080] The ultrasonic diagnostic apparatus 100c may include a probe 20 and a body 40. The probe 20 may be connected to one side of the body 40 in a wired or wireless manner. The body 40 may include a touch screen 145. The touch screen 145 may display an ultrasonic image, various information processed by the ultrasonic diagnostic apparatus 100c, and a GUI.

[0081] Figure 3 is a flowchart of a method for a medical image display apparatus to output a medical image according to an embodiment.

[0082] The medical image display apparatus may obtain a first ultrasound image (Operation 310 ).

[0083] According to an embodiment, the medical image display apparatus may obtain an ultrasound image via the probe 20 connected thereto.

[0084] For example, the medical image display apparatus may be the ultrasound diagnostic apparatus 100. The ultrasound diagnostic apparatus 100 may transmit ultrasound signals to the subject 10 and receive ultrasound echo signals from the subject 10. The ultrasound diagnostic apparatus 100 may obtain an ultrasound image based on the ultrasound echo signals.

[0085] According to an embodiment, the medical image display apparatus may obtain an ultrasound image from at least one of an ultrasound diagnostic apparatus and a server via the communicator 160 .

[0086] For example, the medical image display device may receive ultrasound image data obtained by an ultrasound diagnostic device from an ultrasound diagnostic device connected via a network. As another example, the medical image display device may receive ultrasound image data from a server (e.g., a medical image transmission system such as PACS).

[0087] The medical image display apparatus may perform image registration between the first ultrasound image obtained in operation 310 and the first medical image (operation 330 ).

[0088] According to an embodiment, the first medical image may be a previously obtained medical image.

[0089] According to an embodiment, the first medical image may be an image stored in a memory of the medical image display device.

[0090] According to an embodiment, the first medical image may be a medical image received by the medical image display apparatus from a server (eg, a medical image transmission system such as PACS).

[0091] According to an embodiment, the first medical image may be at least one of a CT image, an MR image, and a three-dimensional (3D) ultrasound image. Alternatively, the first medical image may be at least one of an X-ray image and a two-dimensional (2D) ultrasound image.

[0092] The medical image display apparatus may perform image registration by applying at least one of the first ultrasound image and the first medical image to an AI model.

[0093] According to an embodiment, the medical image display apparatus may obtain features in the first ultrasound image by applying the first ultrasound image to an AI model.

[0094] For example, the medical image display device may apply the first ultrasound image to the AI ​​model built therein. Alternatively, the medical image display device may apply the first ultrasound image to the AI ​​model by transmitting data related to the first ultrasound image to a server on which the AI ​​model is built.

[0095] The AI ​​model may identify features in the first ultrasound image by analyzing the first ultrasound image applied thereto. The AI ​​model may also identify an object in the first ultrasound image based on the features in the first ultrasound image.

[0096] According to an embodiment, the medical image display apparatus may perform image registration between the first ultrasound image and the first medical image by comparing and matching features in the first ultrasound image with corresponding features in the first medical image.

[0097] For example, the medical image display apparatus may identify and match features in the first ultrasound image and corresponding features in the first medical image by using an AI model built therein.

[0098] In addition, the features in the first medical image may be obtained in advance. Alternatively, the features in the first medical image may be obtained by applying the first medical image to an AI model built in the medical image display device.

[0099] According to an embodiment, the medical image display device may transmit a first ultrasound image to a server and perform image registration by matching corresponding features in the first ultrasound image received from the server with features in the first medical image. In this case, the server may obtain the features in the first ultrasound image by applying the first ultrasound image to an AI model built on the server.

[0100] According to an embodiment, the medical image display device may transmit the first ultrasound image and the first medical image to a server and perform image registration by matching features in the first ultrasound image received from the server with features in the first medical image. In this case, the server may obtain features from the first ultrasound image by applying the first ultrasound image to an AI model built on the server. Furthermore, the server may obtain features from the first medical image by applying the first medical image to the AI ​​model built on the server.

[0101] According to an embodiment, the medical image display apparatus may transmit the first ultrasound image and the first medical image to a server, and receive data for performing image registration (e.g., position information of features in the first ultrasound image and position information of features in the first medical image) from the server. The medical image display apparatus may perform image registration between the first medical image and the first ultrasound image using the data received from the server.

[0102] The medical image display apparatus may obtain a second ultrasound image (Operation 350 ).

[0103] According to an embodiment, the medical image display apparatus may obtain an ultrasound image via the probe 20 connected to the medical image display apparatus.

[0104] According to an embodiment, the medical image display apparatus may obtain an ultrasound image from at least one of an ultrasound diagnostic apparatus and a server via the communicator 160 .

[0105] According to an embodiment, the second ultrasound image may be an ultrasound image newly obtained as the probe 20 moves. In this case, the medical image display apparatus may obtain a second ultrasound image in which the shape of the object changes as the probe 20 presses the object.

[0106] Repeated descriptions regarding operation 310 are omitted to avoid redundancy.

[0107] The medical image display apparatus may obtain a second medical image by transforming the first medical image to correspond to the second ultrasound image (operation 370 ).

[0108] According to an embodiment, the medical image display apparatus may obtain features in the second ultrasound image by applying the second ultrasound image to an AI model.

[0109] For example, the medical image display device may apply the second ultrasound image to an AI model built on the medical image display device. Alternatively, the medical image display device may apply the second ultrasound image to the AI ​​model by transmitting data related to the second ultrasound image to a server on which the AI ​​model is built.

[0110] The AI ​​model may identify features in the second ultrasound image. The AI ​​model may identify an object in the second ultrasound image based on the features in the second ultrasound image.

[0111] According to an embodiment, the AI ​​model may identify differences between the second ultrasound image and the first medical image by comparing features in the second ultrasound image with features in the first medical image.

[0112] For example, the AI ​​model may identify differences between the second ultrasound image and the first medical image by comparing the positions of features in the second ultrasound image with the positions of features in the first medical image, respectively. Furthermore, the AI ​​model may compare the positions of features in the second ultrasound image with the positions of features in the first medical image, thereby obtaining first information regarding changes in the positions of matching features in the second ultrasound image and the first medical image.

[0113] According to an embodiment, the AI ​​model may obtain a second medical image by transforming the first medical image based on matching features in the second ultrasound image and the first medical image.

[0114] For example, the AI ​​model can obtain the second medical image by moving features in the first medical image so that they correspond to the positions of features in the second ultrasound image. In this case, the AI ​​model can move the features in the first medical image by applying the first information to the features in the first medical image. The second medical image can be generated by moving the features in the first medical image.

[0115] According to an embodiment, the AI ​​model may identify differences between the first ultrasound image and the second ultrasound image by comparing features in the first ultrasound image with features in the second ultrasound image.

[0116] For example, the AI ​​model may track an object in the second ultrasound image by respectively comparing the locations of features in the first ultrasound image with the locations of features in the second ultrasound image.

[0117] As another example, the AI ​​model may compare the position of a feature in a first ultrasound image with the position of a feature in a second ultrasound image, thereby obtaining second information about changes in the positions of matching features in the first ultrasound image and the second ultrasound image.

[0118] According to an embodiment, the AI ​​model may obtain a second medical image by transforming the first medical image based on matching features in the first ultrasound image and the second ultrasound image.

[0119] For example, the AI ​​model may move a feature in the first medical image by applying the second information to the feature in the first medical image. The second medical image may be generated by moving the feature in the first medical image. In this case, because image registration is performed in operation 330 to match the feature in the first ultrasound image with the corresponding feature in the first medical image, the second information may be applied to the feature in the first medical image.

[0120] The medical image display device may obtain the second medical image from the AI ​​neural network. For example, the AI ​​neural network implemented in the medical image display device may store the second medical image in a memory of the medical image display device. As another example, the AI ​​neural network implemented on a server may transmit the second medical image to the medical image display device via a network.

[0121] The medical image display apparatus may display the second medical image (operation 390 ).

[0122] According to an embodiment, the medical image display apparatus may display the second medical image together with the second ultrasound image.

[0123] According to an embodiment, the medical image display apparatus may display the second medical image after performing image processing so that objects in the second ultrasound image and the second medical image may be easily recognized.

[0124] For example, the medical image display apparatus may perform image processing to color-code the object. Alternatively, the medical image display apparatus may perform image processing to indicate a boundary of the object with a predefined color.

[0125] Figure 4 is a flowchart of a method for registering an ultrasound image with a medical image by a medical image display apparatus according to an embodiment.

[0126] The medical image display device may apply the first ultrasound image to the AI ​​model ( Figure 1 125)(operation 331).

[0127] Since the method of applying the first ultrasound image to the AI ​​model, which has been described above with reference to operation 330 , may be similarly applied in operation 331 , a repeated description thereof will be omitted here.

[0128] The AI ​​model 125 may obtain features in the first ultrasound image (operation 333 ).

[0129] The AI ​​model 125 may be trained to obtain features from ultrasound images by learning a plurality of ultrasound images as training data, each ultrasound image including an object (eg, prostate, liver, kidney, etc.).

[0130] According to an embodiment, the AI ​​model 125 may be trained to obtain the boundaries of an object (eg, prostate, liver, kidney, etc.) as features.

[0131] According to an embodiment, the AI ​​model 125 may be trained to obtain the Euclidean distance used in semantic segmentation as a feature, which classifies each of the ultrasound image and the medical image in a pixel-by-pixel manner.

[0132] The AI ​​model 125 may obtain the Euclidean distance from the boundary of the object to each image pixel.

[0133] According to an embodiment, the AI ​​model 125 may be trained to recognize the boundary of an object and obtain a distance map image indicating the Euclidean distance from the boundary of the object (as described below with reference to FIG. Figure 5 described in more detail).

[0134] According to an embodiment, the AI ​​model 125 may output features in the obtained first ultrasound image to a medical image display apparatus.

[0135] The medical image display apparatus may identify whether a condition for image registration is satisfied (Operation 335 ).

[0136] According to an embodiment, the medical image display apparatus may identify whether a condition for image registration is satisfied based on features in the first ultrasound image obtained in operation 333 .

[0137] For example, the medical image display apparatus may identify the total width of the object based on the boundary of the object, and identify whether the image registration condition is satisfied based on the ratio of the width of the object in the first ultrasound image to the identified total width of the object (as described below with reference to Figure 6 and Figure 7 described in more detail).

[0138] As another example, the medical image display apparatus may identify whether a condition for image registration is satisfied based on a ratio of the object occupying the first ultrasound image (as described below with reference to FIG. Figure 8 described in more detail).

[0139] As another example, the medical image display apparatus may identify whether a condition for image registration is satisfied based on a direction in which an ultrasound signal transmitted from a probe is positioned (as described below with reference to FIG. Figure 9 described in more detail).

[0140] The medical image display apparatus may display information about the image registration (operation 337 ).

[0141] According to an embodiment, the medical image display apparatus may display information on whether a condition for image registration is satisfied.

[0142] For example, the medical image display apparatus may display information indicating that the proportion of an object being displayed in an ultrasound image is less than a certain percentage value.

[0143] As another example, the medical image display apparatus may display information indicating that the size of an object in an ultrasound image is smaller than a preset size.

[0144] As another example, the medical image display apparatus may display information indicating that an ultrasound signal is not transmitted to the subject.

[0145] According to an embodiment, the medical image display apparatus may display information on whether the shape of the registered medical image has been corrected (as described below with reference to FIG. Figure 6 and Figure 7 described in more detail).

[0146] Figure 5 An example is shown in which the medical image display apparatus according to the embodiment applies an ultrasound image to an AI model.

[0147] Reference Figure 5 , the medical image display device can apply the ultrasound image 510 to the AI ​​model 125.

[0148] According to an embodiment, the AI ​​model 125 may be constructed in the medical image display device. Alternatively, the AI ​​model 125 may be constructed on a server connected to the medical image display device via a network. For convenience, the embodiment in which the AI ​​model 125 is constructed in the medical image display device is described. It will be readily understood by those skilled in the art that the details of the AI ​​model 125 described below are similarly applicable to AI models according to other embodiments of the present disclosure.

[0149] According to an embodiment, the AI ​​model 125 may be trained by learning a plurality of ultrasound images (each of which includes an object) as training data to obtain features from the ultrasound images.

[0150] For example, multiple ultrasound images (each including a prostate) may be input as training data to the AI ​​model 125. Each of the ultrasound images may be segmented into multiple segments. Each of the ultrasound images may represent the boundary of the prostate as a feature. The AI ​​model 125 may obtain a feature vector from each of the multiple segmented ultrasound images using a CNN.

[0151] According to an embodiment, the AI ​​model 125 may be trained to recognize the boundary of an object as a feature. In addition, the AI ​​model 125 may be trained to obtain an object contour map image based on the recognized contour of the object.

[0152] For example, the AI ​​model 125 may identify the boundary of an object (e.g., a prostate) in an ultrasound image 510 applied to the AI ​​model 125 based on feature vectors obtained from a plurality of training data. The AI ​​model 125 may obtain feature vectors by segmenting the ultrasound image 510 and identify the boundary of the object based on the obtained feature vectors. The AI ​​model 125 may obtain an object contour map image 530 from the ultrasound image 510 based on the identified object contour.

[0153] According to an embodiment, the AI ​​model 125 may be trained to obtain Euclidean distance as a feature used in semantic segmentation, which classifies ultrasound images in a pixel-by-pixel manner.

[0154] For example, the AI ​​model 125 may obtain the Euclidean distance from the boundary of the object to each image pixel. The AI ​​model 125 may obtain a distance map image 550 representing the Euclidean distance from the boundary of the object. The AI ​​model 125 may generate the distance map image 550 by color-coding regions with equal Euclidean distances from the boundary of the object with the same color.

[0155] Figure 6 and Figure 7 Examples of displaying information on whether a condition for image registration is satisfied by the medical image display apparatus according to the embodiments are respectively shown.

[0156] Reference Figure 6 and Figure 7 , the medical image display apparatus may respectively display a pair of an ultrasound image 610 including an object 611 and a medical image 630 including an object 631, and a pair of an ultrasound image 710 including an object 711 and a medical image 730 including an object 731 on the display 140. In addition, the medical image display apparatus may display information regarding whether a condition for performing image registration between the pair of ultrasound image 610 and medical image 630 or the pair of ultrasound image 710 and medical image 730 is satisfied by using various methods.

[0157] The positions of the objects 611 and 711 in the ultrasound images 610 and 710 change according to the direction in which the ultrasound signal transmitted by the ultrasound probe is directed.

[0158] When the general Figure 6 The ultrasound image 610 is Figure 7 When comparing the ultrasound image 710, Figure 6 The object 611 is shown to be completely contained within the ultrasound image 610, while Figure 7 Only a portion of the object 711 is shown included in the ultrasound image 710. In other words, Figure 7 The object 711 is partially outside the ultrasound image 710. As the objects 611 and 711 partially fall outside the ultrasound images 610 and 710 to a greater extent, respectively, the accuracy of image registration decreases.

[0159] According to an embodiment, the medical image display apparatus may identify whether the contours of the objects 611 and 711 identified using the AI ​​model 125 are completely included in the ultrasound images 610 and 710, respectively. In other words, the medical image display apparatus may identify the extent to which the objects 611 and 711 fall outside the ultrasound images 610 and 710, respectively.

[0160] For example, the medical image display apparatus may identify regions of objects 611 and 711 in the ultrasound images 610 and 710, respectively, based on the boundaries of the objects 611 and 711 identified using the AI ​​model 125. The medical image display apparatus may identify the extent to which the objects 611 and 711 are included in the ultrasound images 610 and 710, respectively, based on the regions of the objects 611 and 711 in the ultrasound images 610 and 710.

[0161] Specifically, the medical image display apparatus may identify the entire region of the object 611 in the ultrasound image 610 that completely includes the object 611. The medical image display apparatus may identify the region of the object 711 in the ultrasound image 710 that includes only a portion of the object 711. The medical image display apparatus may obtain a ratio of the region of the object 711 included in the ultrasound image 710 relative to the entire region of the object 711. The medical image display apparatus may identify the extent to which the object 711 is included in the ultrasound image 710 based on the obtained ratio.

[0162] As another example, the medical image display apparatus may determine the positions of objects 611 and 711 in the ultrasound images 610 and 710, respectively, based on the boundaries of the objects 611 and 711 identified using the AI ​​model 125. The medical image display apparatus may identify the extent to which the objects 611 and 711 are included in the ultrasound images 610 and 710, respectively, based on the positions of the objects 611 and 711 in the ultrasound images 610 and 710.

[0163] According to an embodiment, the medical image display apparatus may determine whether the conditions for performing image registration are met by comparing the extent to which the object is included in the ultrasound image with a preset threshold. For example, when the ratio of the area of ​​the object 711 included in the ultrasound image 710 to the entire area of ​​the object 711 is less than 50%, the medical image display apparatus may determine that the conditions for performing image registration are not met.

[0164] According to an embodiment, the medical image display apparatus may identify whether a condition for performing image registration is satisfied based on information about the position of the probe 20 obtained using a sensor. For example, the medical image display apparatus may track the movement of the probe 20 using an electromagnetic sensor within a specific range of a magnetic field generated by a magnetic field generator, thereby identifying the extent to which the objects 611 and 711 are outside the ultrasound images 610 and 710, respectively.

[0165] According to an embodiment, the medical image display apparatus may display information indicating that a condition for performing image registration is not satisfied. For example, the medical image display apparatus may display a notification indicating that the object 711 falls outside the ultrasound image 710.

[0166] According to an embodiment, the medical image display apparatus may obtain information on the reliability of image registration by comparing the extent to which the object is included in the ultrasound image with a preset threshold value. The medical image display apparatus may display the information on the reliability of the image registration.

[0167] For example, the medical image display apparatus may display information on the reliability of image registration by displaying bar graphs 635 and 735 corresponding to preset thresholds, respectively, on the display 140 .

[0168] Specifically, the medical image display apparatus may display a result of comparing a ratio of a region of the object 711 included in the ultrasound image 710 relative to the entire region of the object 711 with a threshold value set to 30%, 50%, or 70% as a bar graph 735 .

[0169] Figure 8 An example is shown in which the medical image display apparatus according to the embodiment identifies whether a condition for image registration is satisfied.

[0170] When the object 811 in the ultrasound image 810 is too small or too large, conditions for performing image registration may not be satisfied.

[0171] Therefore, refer to Figure 8 , the medical image display apparatus may identify whether a condition for performing image registration is satisfied based on a ratio at which the object 811 occupies the ultrasound image 810 .

[0172] According to an embodiment, the medical image display apparatus may obtain a ratio at which the object 811 occupies the ultrasound image 810 based on a boundary of the object 811 recognized using the AI ​​model 125 .

[0173] For example, the medical image display apparatus may identify a region of the object 811 in the ultrasound image 810 based on the boundary of the object 811 identified using the AI ​​model 125. The medical image display apparatus may obtain a ratio at which the object 811 occupies the ultrasound image 810 based on the region of the object 811.

[0174] According to an embodiment, the medical image display apparatus may identify whether a condition for performing image registration is satisfied by comparing a ratio at which the object 811 occupies the ultrasound image 810 with a preset threshold.

[0175] For example, when the proportion of the object 811 occupying the ultrasound image 810 is less than 20%, the medical image display apparatus may recognize that a condition for performing image registration is not satisfied.

[0176] As another example, when the object 811 occupies a proportion of 90% or more of the ultrasound image 810 , the medical image display apparatus may recognize that a condition for performing image registration is not satisfied.

[0177] According to an embodiment, the medical image display apparatus may display information indicating that a condition for performing image registration is not satisfied. For example, the medical image display apparatus may display a notification indicating that the object 811 in the ultrasound image 810 is too small or too large.

[0178] According to an embodiment, the medical image display apparatus may obtain information on the reliability of image registration by comparing a ratio of the object 811 occupying the ultrasound image 810 with a preset threshold value. The medical image display apparatus may display the information on the reliability of image registration.

[0179] Figure 9 An example is shown in which the medical image display apparatus according to the embodiment identifies whether a condition for image registration is satisfied.

[0180] When the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed is not toward the subject 10 , it is difficult to perform image registration.

[0181] Therefore, refer to Figure 9 , the medical image display apparatus may identify whether a condition for performing image registration is satisfied based on the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed.

[0182] According to an embodiment, the medical image display apparatus may identify whether a condition for performing image registration is satisfied by identifying a direction 25 in which an ultrasound signal transmitted from the probe 20 is directed based on information about the position of the probe 20 obtained via a sensor.

[0183] For example, the medical image display apparatus may identify the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed by tracking the movement of the probe 20 via the electromagnetic sensor within a specific range of the magnetic field generated by the magnetic field generator.

[0184] According to an embodiment, the medical image display apparatus may identify the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed, based on the boundary of the object 10 identified in the ultrasound image by using the AI ​​model 125 .

[0185] For example, the medical image display apparatus may identify the position of the object 10 in the ultrasound image based on the boundary of the object 10 identified using the AI ​​model 125. The medical image display apparatus may identify the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed based on the position of the object 10 in the ultrasound image.

[0186] According to an embodiment, the medical image display apparatus may display information indicating that a condition for performing image registration is not satisfied. For example, the medical image display apparatus may display information indicating that the direction 25 in which the ultrasound signal transmitted from the probe 20 is directed is not toward the subject 10.

[0187] Figure 10 An example is shown in which the medical image display apparatus according to the embodiment applies a medical image to an AI model.

[0188] Reference Figure 10 , the medical image display device can apply the medical image 1010 to the AI ​​model 125.

[0189] Depending on the embodiment, the AI ​​model 125 may be implemented in the medical image display device. Alternatively, the AI ​​model 125 may be implemented on a server connected to the medical image display device via a network. For convenience, the embodiment in which the AI ​​model 125 is implemented in the medical image display device is described. It will be readily understood by those skilled in the art that the details of the AI ​​model 125 described below are similarly applicable to the AI ​​models of other embodiments of the present disclosure.

[0190] According to an embodiment, the medical image 1010 may be at least one of a CT image, an MR image, and a 3D ultrasound image.

[0191] According to an embodiment, the AI ​​model 125 may be trained by learning a plurality of medical images (each of which includes an object) as training data to obtain features from the medical images.

[0192] For example, multiple medical images (each including a prostate) may be input to AI model 125 as training data. Each medical image may include a slice image. The slice image may be segmented into multiple segments. Each medical image may represent the boundary of the prostate as a feature. AI model 125 may obtain a feature vector from each of the multiple segmented cross-sectional medical images using a CNN.

[0193] According to an embodiment, the AI ​​model 125 may be trained to recognize object boundaries as features. For example, the AI ​​model 125 may be trained to recognize object boundaries in 3D medical images as features. As another example, the AI ​​model 125 may be trained to recognize object boundaries in cross-sectional images as features.

[0194] Furthermore, the AI ​​model 125 may be trained to obtain an object contour map image based on the contours of the identified objects.

[0195] For example, the AI ​​model 125 may identify the boundary of an object (e.g., a prostate) in the medical image 1010 applied to the AI ​​model 125 based on feature vectors obtained from a plurality of training data. The AI ​​model 125 may obtain feature vectors by segmenting the medical image 1010 and identify the boundary of the object based on the obtained feature vectors. The AI ​​model 125 may obtain an object contour map image from the medical image 1010 based on the identified object contour.

[0196] According to an embodiment, the AI ​​model 125 may be trained to obtain Euclidean distance as a feature used in semantic segmentation, which classifies ultrasound images in a pixel-by-pixel manner.

[0197] For example, the AI ​​model 125 may obtain the Euclidean distance from the boundary of the object. The AI ​​model 125 may obtain a 3D distance map image 1030 representing the Euclidean distance from the boundary of the object. The AI ​​model 125 may generate the 3D distance map image 1030 by color-coding regions having equal Euclidean distances from the boundary of the object with the same color.

[0198] According to an embodiment, the AI ​​model 125 may generate a cross-sectional distance map image 1050 by slicing the 3D distance map image 1030 .

[0199] For example, the AI ​​model 125 may generate a cross-sectional distance map image 1050 by slicing the 3D distance map image 1030 based on information about the position of the probe.

[0200] According to an embodiment, features in the medical image 1010 may be pre-acquired. For example, the features in the medical image 1010 may be acquired from the pre-acquired medical image 1010 by an AI neural network built in a server, and then transmitted to a medical image display device. As another example, the features in the medical image 1010 may be acquired from the pre-acquired medical image 1010 by an AI neural network built in a medical image display device.

[0201] Figure 11 An example is shown in which the medical image display apparatus according to the embodiment performs image registration between an ultrasound image and a medical image.

[0202] Reference Figure 11 , the medical image display apparatus may perform image registration between the ultrasound image 1110 and the medical image 1120 by using the AI ​​model 125 .

[0203] According to an embodiment, the medical image display apparatus may perform image registration between a 3D ultrasound image and a 3D medical image.

[0204] According to an embodiment, the medical image display apparatus may perform image registration between a 2D ultrasound image and a cross-sectional image generated by slicing a 3D medical image. For example, the medical image display apparatus may extract a cross-sectional image from the 3D medical image based on information regarding the position of the probe 20 obtained via a sensor. The medical image display apparatus may perform image registration between the selected cross-sectional image and the 2D ultrasound image.

[0205] According to an embodiment, the image registration between the ultrasound image 1110 and the medical image 1120 performed by the AI ​​model 125 may be divided into two stages.

[0206] For example, the AI ​​model 125 may perform an overall registration between the ultrasound image 1110 and the medical image 1120 (stage 1 image registration), and then perform a precise registration between the object 1111 in the ultrasound image 1110 and the object 1121 in the medical image 1120 (stage 2 image registration).

[0207] According to an embodiment, the AI ​​model 125 may perform image registration between the ultrasound image 1110 and the medical image 1120 by comparing and matching features from the ultrasound image 1110 with corresponding features from the medical image 1120 .

[0208] For example, the AI ​​model 125 may perform image registration between the ultrasound image 1110 and the medical image 1120 by comparing and matching the locations of features from the ultrasound image 1110 with the locations of corresponding features from the medical image 1120 .

[0209] As another example, the AI ​​model 125 may perform image registration between the ultrasound image 1110 and the medical image 1120 by comparing and matching the boundary of the object 1111 in the ultrasound image 1110 with the boundary of the object 1121 in the medical image 1120 .

[0210] As another example, the AI ​​model 125 may perform image registration between the ultrasound image 1110 and the medical image 1120 by comparing and matching a distance map image of the object 1111 from the ultrasound image 1110 with a distance map image of the object 1121 from the medical image 1120 .

[0211] According to an embodiment, the AI ​​model 125 may perform image registration between the ultrasound image 1110 and the medical image 1120 by rotating at least one of the ultrasound image 1110 and the medical image 1120 so as to compare and match features from the ultrasound image 1110 with corresponding features from the medical image 1120.

[0212] For example, the AI ​​model 125 may rotate at least one of the ultrasound image 1110 and the medical image 1120 to perform overall image registration between the ultrasound image 1110 and the medical image 1120. The AI ​​model 125 may rotate at least one of the ultrasound image 1110 and the medical image 1120 to match features of the object 1111 in the ultrasound image 1110 with corresponding features of the object 1121 in the medical image 1120.

[0213] According to an embodiment, the AI ​​model 125 may perform shape correction on at least one of the object 1111 in the ultrasound image 1110 and the object 1121 in the medical image 1120 by comparing and matching features from the ultrasound image 1110 with corresponding features from the medical image 1120 .

[0214] For example, the AI ​​model 125 may perform shape correction on at least one of the object 1111 in the ultrasound image 1110 and the object 1121 in the medical image 1120 in order to accurately register the ultrasound image 1110 and the medical image 1120 (as described below with reference to FIG. Figure 12 described in more detail).

[0215] Figure 12 is a flowchart of a method for applying a correction operation to a medical image by a medical image display apparatus according to an embodiment.

[0216] The medical image display apparatus may apply the second ultrasound image to the AI ​​model 125 (operation 351). The second ultrasound image is Figure 3 The ultrasound image is obtained in operation 350 .

[0217] According to an embodiment, the second ultrasound image may be an ultrasound image newly obtained as the probe 20 moves. In this case, the medical image display apparatus may obtain a second ultrasound image in which the shape of the object changes due to pressure applied by the probe 20.

[0218] The AI ​​model 125 may obtain features in the second ultrasound image (operation 352 ) and may output the obtained features in the second ultrasound image to a medical image display apparatus.

[0219] Because operation 352 and reference Figure 4 The described operation 333 is similar, so its repeated description will be omitted here.

[0220] The medical image display apparatus may identify whether there is a result of performing a correction operation on the first medical image (operation 353 ).

[0221] According to an embodiment, the medical image display apparatus may identify, via the AI ​​model 125 , whether there is a result of performing a correction operation for changing the shape of an object in the first medical image.

[0222] The medical image display apparatus may not perform an additional correction operation by applying the result of the correction operation performed on the first medical image to the first medical image. Therefore, the medical image display apparatus may quickly perform image registration and display the result of applying the shape correction to the first medical image.

[0223] When there is a result of performing the correction operation on the first medical image, the medical image display apparatus may proceed to operation 354. On the other hand, when there is no result of performing the correction operation on the first medical image, the medical image display apparatus may proceed to operation 355.

[0224] The medical image display apparatus may identify whether the position of the probe 20 has changed (Operation 354 ).

[0225] According to an embodiment, the medical image display apparatus may identify whether the position of the probe 20 has changed based on information about the position of the probe 20 obtained via a sensor. For example, the medical image display apparatus may identify whether the position of the probe 20 has changed by tracking the movement of the probe 20 using an electromagnetic sensor within a specific range of the magnetic field generated by the magnetic field generator.

[0226] When the position of the probe 20 is changed, the medical image display apparatus may proceed to operation 355. Otherwise, when the position of the probe 20 is not changed, the medical image display apparatus may proceed to operation 371.

[0227] In other words, when the position of the probe 20 is not changed, the medical image display apparatus proceeds to operation 371 , thereby quickly performing image registration and displaying a medical image to which shape correction has been applied.

[0228] The medical image display apparatus may perform a correction operation for transforming the first medical image (operation 355 ).

[0229] According to an embodiment, the medical image display apparatus may perform a correction operation for transforming the first medical image by comparing and matching features in the second ultrasound image obtained in operation 352 with corresponding features in the first medical image via the AI ​​model 125 .

[0230] For example, the AI ​​model 125 may perform a correction operation to obtain a vector for correcting differences, each difference between the position of each feature from the second ultrasound image and the position of its corresponding feature from the first medical image.

[0231] As another example, the AI ​​model 125 may perform a correction operation to obtain a vector for correcting a difference between contours of an object respectively obtained from the second ultrasound image and the first medical image.

[0232] As another example, the AI ​​model 125 may perform a correction operation to obtain a vector for correcting a difference between distance map images of the object obtained from the second ultrasound image and the first medical image, respectively.

[0233] The medical image display apparatus may obtain a second medical image by applying a result of the correction operation to the first medical image (operation 371 ).

[0234] According to an embodiment, the medical image display apparatus may apply the result of the correction operation performed in operation 355 to the first medical image by using the AI ​​model 125 , thereby obtaining a second medical image including the object to which shape correction has been applied.

[0235] For example, the AI ​​model 125 can obtain a second medical image including an object to which shape correction has been applied by applying a vector for correcting differences to the first medical image, each difference between the position of each feature from the second ultrasound image and the position of its corresponding feature from the first medical image.

[0236] As another example, the AI ​​model 125 may obtain a second medical image including an object to which shape correction has been applied by applying a vector for correcting a difference between contours of the object obtained from the second ultrasound image and the first medical image, respectively, to the first medical image.

[0237] As another example, the AI ​​model 125 may obtain a second medical image including an object to which shape correction has been applied by applying a vector for correcting the difference between distance map images of the object obtained from the second ultrasound image and the first medical image, respectively, to the first medical image.

[0238] According to an embodiment, the AI ​​neural network built in the medical image display device may store the second medical image in the memory of the medical image display device. As another example, the AI ​​neural network built on the server may transmit the second medical image to the medical image display device via a network.

[0239] Figure 13 A result obtained when the medical image display apparatus applies the result of the correction operation to the medical image according to the embodiment is shown.

[0240] Reference Figure 13 ,exist Figure 3 The object 1311 is identified in the first ultrasound image obtained in operation 310. Figure 3 An object 1313 is identified in the second ultrasound image 1310 obtained in operation 350. Figure 3 An object 1331 is identified in the first medical image 1330 registered with the first ultrasound image in operation 330. The object 1353 is included in Figure 3 The second medical image 1350 is obtained in operation 370 .

[0241] When the first ultrasound image is registered with the first medical image 1330, the object 1311 in the first ultrasound image matches the object 1331 in the first medical image 1330. Therefore, the shape of the object 1311 in the first ultrasound image is similar to the shape of the object 1331 in the first medical image 1330.

[0242] In addition, the object 1353 in the second medical image 1350 has a shape obtained by changing the shape of the object 1331 in the first medical image 1330 based on the object 1313 in the second ultrasound image 1310. Therefore, the shape of the object 1313 in the second ultrasound image 1310 is similar to the shape of the object 1353 in the second medical image 1350.

[0243] According to experimental results, when the medical image display device is a computer including an i7-4790 central processing unit (CPU) with 8 logical cores (running at 3.60GHz), 16 gigabytes (GB) of random access memory (RAM), and an Nvidia Quadro K2200 graphics processing unit (GPU), it takes an average of 115ms per frame to correct the medical image. Therefore, according to embodiments of the present disclosure, the medical image display device can correct the medical image to correspond to the ultrasound image in real time and display the resulting image.

[0244] Furthermore, experimental results show that the results of correcting medical images according to embodiments of the present disclosure have an error of 1.595±1.602 mm (average trans-registration error of Euclidean distance) in a phantom experiment and an error of 3.068±1.599 mm (average trans-registration error of Euclidean distance) in a clinical experiment. Therefore, the medical image display device of the disclosed embodiments can correct a medical image that has been registered with an ultrasound image to correspond to the ultrasound image and display the medical image to which the correction results have been applied, thereby clearly providing information about the subject to the user.

[0245] The embodiments of the present disclosure may be implemented via a non-transitory computer-readable recording medium having computer-executable instructions and data stored therein. The instructions may be stored in the form of program code and, when executed by a processor, generate a predefined program module to perform a preset operation. Furthermore, the instructions, when executed by the processor, may perform the preset operation according to the embodiments.

Claims

1. A method for displaying a medical image by using a medical image display device, the method comprising: transmitting an ultrasonic signal to a subject and receiving an ultrasonic echo signal from the subject using an ultrasonic probe of the medical image display device; obtaining a first ultrasound image based on the ultrasound echo signal; identifying a first contour of the object included in the first ultrasound image as a feature of the first ultrasound image by applying the first ultrasound image to an artificial intelligence model; identifying, using the artificial intelligence model, a second contour of the object included in a pre-acquired first medical image as a feature of the first medical image; performing image registration between the first ultrasound image and the first medical image by matching features of the first ultrasound image with features of the first medical image; obtaining a second ultrasound image of the object using the ultrasound probe; obtaining a second medical image by transforming the first medical image to correspond to the second ultrasound image; as well as displaying the second medical image together with the second ultrasound image, The step of performing the image registration includes: obtaining a first proportion occupied by the object in the first ultrasound image based on the first outline of the object; and Based on whether the first ratio is smaller than or larger than a preset threshold, it is identified whether a condition for image registration is satisfied.

2. The method according to claim 1, wherein The steps of performing the image registration include: Information on the reliability of the image registration is displayed based on the result of the identification of whether the conditions for image registration are satisfied.

3. The method according to claim 1, wherein The step of obtaining features in the first ultrasound image comprises: obtaining a first distance map indicating a distance from the first contour of the object to each pixel in the first ultrasound image, and The step of matching features of the first ultrasound image with features of the first medical image includes comparing the first distance map with a second distance map previously obtained from the first medical image by using the artificial intelligence model.

4. The method according to claim 1, wherein The step of obtaining the second ultrasound image includes obtaining the second ultrasound image including the object deformed by pressing the object with the ultrasound probe, and The step of obtaining the second medical image includes: obtaining, by applying the second ultrasound image to the artificial intelligence model, a third contour of the object included in the second ultrasound image as a feature of the second ultrasound image; and The second medical image is obtained by transforming the first medical image based on a result of comparing features in the first medical image with features in the second ultrasound image.

5. A medical image display device, comprising: monitor; an ultrasound probe configured to transmit ultrasound signals to a subject and receive ultrasound echo signals from the subject; a memory storing one or more instructions; and A processor configured to execute the one or more instructions to perform the following operations: obtaining a first ultrasound image based on the ultrasound echo signal; identifying a first contour of the object included in the first ultrasound image as a feature of the first ultrasound image by applying the first ultrasound image to an artificial intelligence model; identifying, using the artificial intelligence model, a second contour of the object included in a pre-acquired first medical image as a feature of the first medical image; performing image registration between the first ultrasound image and the first medical image by matching features of the first ultrasound image with features of the first medical image; controlling the ultrasound probe to obtain a second ultrasound image of the object; obtaining a second medical image by transforming the first medical image to correspond to the second ultrasound image; and controlling the display to display the second medical image together with the second ultrasound image, The processor is further configured to: obtaining a first proportion occupied by the object in the first ultrasound image based on the first outline of the object; and Based on whether the first ratio is smaller than or larger than a preset threshold, it is identified whether a condition for image registration is satisfied.

6. The medical image display device according to claim 5, wherein: The processor is further configured to execute the one or more instructions to: The display is controlled to display information on reliability of the image registration based on a result of the identification of whether a condition for image registration is satisfied.

7. The medical image display device according to claim 6, wherein: The processor is further configured to execute the one or more instructions to: obtaining a first distance map indicating a distance from the first contour of the object to each pixel in the first ultrasound image; The first distance map is compared to a second distance map previously obtained from the first medical image using the artificial intelligence model.

8. The medical image display device according to claim 6, wherein: The processor is further configured to execute the one or more instructions to: obtaining a second ultrasound image of the object including deformation caused by pressing the object with the ultrasound probe; obtaining, by applying the second ultrasound image to the artificial intelligence model, a third contour of the object included in the second ultrasound image as a feature of the second ultrasound image; and The second medical image is obtained by transforming the first medical image based on a result of comparing features in the first medical image with features in the second ultrasound image.

Images