Medical imaging device and medical image processing method
The method and apparatus prioritize and organize lesion information in medical images using contours and arrows, addressing the issue of information overload and improving diagnosis efficiency.
Patent Information
- Application Number
- JP2022552179
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-28
- Filing Date
- 2021-02-25
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2041-02-25
AI Technical Summary
Existing medical imaging technologies display too much information, leading to obstruction of the original image and increased diagnosis time, with a risk of missed lesion information due to overwhelming data.
A method and apparatus that generate contours based on lesion information, considering overlap, similarity, and probability to prioritize and display relevant lesion data, using contours and arrows to guide medical personnel.
Enhances the clarity of medical images by prioritizing and organizing lesion information, reducing obstruction and diagnosis time, and minimizing the risk of missed information.
Smart Images

Figure 0007591221000001 
Figure 0007591221000002 
Figure 0007591221000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an apparatus and method for acquiring lesion information from a medical image and displaying the lesion information on the medical image. [Background technology]
[0002] With the development of technologies such as big data and artificial intelligence, the reliability of technologies for automatically detecting lesions from medical images is increasing. As a result, medical devices are now able to detect multiple lesions from medical images. However, when information about multiple detected lesions is displayed in one medical image, the amount of information displayed in the medical image may be too much. If too much information is displayed in a medical image, not only may the original image be blocked, but it may also take a long time for medical personnel to make a diagnosis using the multiple lesion information displayed. In addition, there is a risk that medical personnel may overlook some of the multiple lesion information displayed.
[0003] Therefore, a technique for effectively displaying lesions on an image is needed. Summary of the Invention [Problem to be solved by the invention]
[0004] The present disclosure provides a medical image processing method, a computer program stored on a recording medium, and a medical imaging device (system) for solving the above problems. [Means for solving the problem]
[0005] An operating method of a medical imaging device according to one embodiment of the present disclosure includes the steps of acquiring lesion information regarding at least one lesion included in a medical image, generating at least one contour corresponding to the at least one lesion in the medical image based on the acquired lesion information, and outputting the at least one contour generated in the medical image.
[0006] The step of generating at least one contour in an operating method of a medical imaging device according to one embodiment of the present disclosure is characterized in that it includes a step of generating a contour based on at least one of information relating to overlapping lesion areas between multiple lesions contained in the medical image or information relating to the correlation between multiple lesions based on acquired lesion information.
[0007] The step of generating at least one contour in an operating method of a medical imaging device according to one embodiment of the present disclosure is characterized by including the steps of: determining a size of an overlapping area of a first lesion area and a second lesion area based on first lesion information and second lesion information contained in a plurality of lesions; determining information related to the probability that a second lesion is present in the medical image based on the second lesion information; and generating at least one contour surrounding the first lesion if the size of the overlapping area is greater than a first threshold and the information related to the probability that a second lesion is present in the medical image is less than a second threshold.
[0008] The step of generating at least one contour in the operating method of the medical imaging device according to one embodiment of the present disclosure is characterized by including the steps of determining whether there is pathological similarity between the first lesion and the second lesion, determining whether the first lesion and the second lesion coexist in the same area, and generating at least one contour surrounding the first lesion and not generating at least one contour surrounding the second lesion if the similarity indicates that the first lesion and the second lesion are similar, or if the coexistence indicates that the first lesion and the second lesion cannot coexist with each other.
[0009] An operating method of a medical imaging device according to one embodiment of the present disclosure further includes a step of sorting the plurality of lesion information in order of the probability of being present in the medical image based on the plurality of lesion information, and is characterized in that a first lesion has a higher probability of being present in the medical image than a second lesion.
[0010] The step of outputting at least one contour in an operating method of a medical imaging device according to one embodiment of the present disclosure includes the steps of: determining first candidate arrow information for a plurality of first candidate arrows indicating a first contour among the at least one contour; determining an arrow-text area in a frame of the first contour; determining a contact area in the arrow-text area where one side of a text box corresponding to the plurality of first candidate arrows and displaying lesion information related to the first contour meets one end of the plurality of first candidate arrows included in the first candidate arrow information; generating at least one arrow set for a displayable position of a text box and arrow for the first contour based on the determined contact area; obtaining a score for each of the at least one arrow set; selecting one of the at least one arrow set based on the obtained score; and outputting the arrow and text in the text box together with the at least one contour based on the selected arrow set.
[0011] In one embodiment of the operating method of the medical imaging device of the present disclosure, the step of determining first candidate arrow information includes a step of obtaining modified first candidate arrow information by moving a start point or an end point of a candidate arrow included in the first candidate arrow information so that the first text box does not overlap with the second text box or the second contour when a first text box for a first contour overlaps with a second text box for a second contour or the second contour, and the step of generating at least one arrow set includes a step of generating at least one arrow set based on the modified first candidate arrow information.
[0012] An operating method of a medical imaging device according to one embodiment of the present disclosure further includes a step of determining second candidate arrow information for a plurality of second candidate arrows indicating a second contour among at least one contour, wherein the step of determining first candidate arrow information includes a step of obtaining modified first candidate arrow information by moving a start point or an end point of one of the plurality of first candidate arrows so that one of the plurality of first candidate arrows does not intersect with one of the plurality of second candidate arrows, and the step of generating at least one arrow set includes a step of generating at least one arrow set based on the modified first candidate arrow information.
[0013] The step of determining first candidate arrow information in the operating method of a medical imaging device according to one embodiment of the present disclosure includes a step of obtaining an intersection point between the first contour and a second contour of the at least one contour, a step of obtaining a first junction point where one of the plurality of first candidate arrows meets the first contour, and a step of modifying a position of a start point or an end point of one of the plurality of first candidate arrows so that the first junction point is away from the intersection point by more than a threshold value to obtain modified first candidate arrow information, and the step of generating at least one arrow set includes a step of generating at least one arrow set based on the modified first candidate arrow information.
[0014] The arrow-text region of the operating method of the medical imaging apparatus according to one embodiment of the present disclosure is characterized in that it is a ring-shaped region surrounding the first contour without contacting the first contour.
[0015] The score of the operating method of the medical imaging device according to one embodiment of the present disclosure is characterized in that the score is higher the greater the distance between at least one text box corresponding to at least one contour, the shorter the length of at least one arrow corresponding to at least one contour, and the greater the distance between the intersection of two contours contained in at least one contour and the tangent point between at least one contour and at least one arrow.
[0016] The step of generating at least one contour in an operating method of a medical imaging device according to one embodiment of the present disclosure is characterized by including a step of determining the position and shape of at least one contour based on acquired lesion information.
[0017] The step of determining the position and shape of at least one contour in an operating method of a medical imaging device according to one embodiment of the present disclosure is characterized by including a step of obtaining a probability that each pixel in the medical image is included in at least one lesion area based on the lesion information, and a step of determining the thickness of at least one contour based on the probability that it is included in at least one lesion area.
[0018] A medical imaging device according to one embodiment of the present disclosure includes a processor and a memory, and the processor acquires lesion information regarding at least one lesion included in the medical image according to an instruction code included in the memory, generates at least one contour corresponding to the at least one lesion in the medical image based on the acquired lesion information, and outputs the generated at least one contour to the medical image.
[0019] The processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that, based on the acquired lesion information, a contour is generated based on at least one of information relating to overlapping lesion areas between multiple lesions included in the medical image, or information relating to the relationship between multiple lesions, according to an instruction code included in the memory.
[0020] A processor of a medical imaging device according to an embodiment of the present disclosure determines a size of an overlapping area between a first lesion area and a second lesion area based on first lesion information and second lesion information included in a plurality of lesions, and determines information related to a probability that a second lesion is present in the medical image based on the second lesion information, in accordance with an instruction code included in a memory. If the size of the overlapping area is greater than a first threshold and the information related to the probability that a second lesion is present in the medical image is less than a second threshold, the processor generates at least one contour surrounding the first lesion.
[0021] A processor of a medical imaging device according to an embodiment of the present disclosure is characterized in that, according to instructions stored in a memory, it determines whether there is a pathological similarity between a first lesion and a second lesion, determines whether the first lesion and the second lesion coexist in the same area, and generates at least one contour surrounding the first lesion and does not generate at least one contour surrounding the second lesion if the similarity indicates that the first lesion and the second lesion are similar, or if the coexistence indicates that the first lesion and the second lesion cannot coexist with each other.
[0022] A processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that, based on a plurality of pieces of lesion information, the plurality of pieces of lesion information are sorted in order of the probability of being present in the medical image in accordance with an instruction word contained in a memory, and a first lesion has a higher probability of being present in the medical image than a second lesion.
[0023] A processor of a medical imaging device according to an embodiment of the present disclosure, in accordance with an instruction included in a memory, determines first candidate arrow information for a plurality of first candidate arrows indicating a first contour among at least one contour, determines an arrow-text area in a frame of the first contour, determines a contact area in the arrow-text area where one side of a text box corresponding to the plurality of first candidate arrows and displaying lesion information related to the first contour meets one end of the plurality of first candidate arrows included in the first candidate arrow information, generates at least one arrow set for a displayable position of a text box and arrow for the first contour based on the determined contact area, obtains a score for each of the at least one arrow set, selects one of the at least one arrow sets based on the obtained score, and outputs the arrow and text in the text box together with the at least one contour based on the selected arrow set.
[0024] A processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that, when a first text box for a first contour overlaps with a second text box for a second contour or with the second contour, a start point or an end point of a candidate arrow included in first candidate arrow information is moved so that the first text box does not overlap with the second text box or the second contour, thereby obtaining modified first candidate arrow information, and generating at least one arrow set based on the modified first candidate arrow information, in accordance with instructions included in a memory.
[0025] A processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that it determines second candidate arrow information for a plurality of second candidate arrows indicating a second contour among at least one contour according to an instruction word included in a memory, moves a start point or an end point of one of the plurality of first candidate arrows so that one of the plurality of first candidate arrows does not intersect with one of the plurality of second candidate arrows, thereby obtaining modified first candidate arrow information, and generates at least one arrow set based on the modified first candidate arrow information.
[0026] A processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that, according to instructions included in a memory, it obtains an intersection point between the first contour and a second contour of the at least one contour, obtains a first junction point where one of the plurality of first candidate arrows meets the first contour, modifies a position of a start point or an end point of one of the plurality of first candidate arrows so that the first junction point is away from the intersection point by more than a threshold, thereby obtaining modified first candidate arrow information, and generating at least one arrow set based on the modified first candidate arrow information.
[0027] The arrow-text region of the medical imaging device according to one embodiment of the present disclosure is characterized in that it is a ring-shaped region surrounding the first contour without contacting the first contour.
[0028] The score of the medical imaging device according to one embodiment of the present disclosure is characterized in that the greater the distance between at least one text box corresponding to at least one contour, the shorter the length of at least one arrow corresponding to at least one contour, and the greater the distance between the intersection of two contours contained in at least one contour and the tangent point between at least one contour and at least one arrow.
[0029] A processor of a medical imaging device according to an embodiment of the present disclosure is characterized in that it determines the position and shape of at least one contour based on acquired lesion information according to instructions stored in the memory.
[0030] A processor of a medical imaging device according to one embodiment of the present disclosure is characterized in that, in accordance with instructions contained in a memory, a probability that each pixel in a medical image is included in at least one lesion area is obtained based on lesion information, and a thickness of at least one contour is determined based on the probability that each pixel is included in at least one lesion area.
[0031] An operating method of a medical imaging device according to one embodiment of the present disclosure includes the steps of: acquiring lesion information regarding at least one lesion detected from a medical image; determining a shape and position of at least one contour corresponding to the at least one lesion based on the acquired lesion information; determining a position of at least one text area including text indicating lesion information regarding the at least one lesion within the medical image; and displaying the at least one contour and the text contained in the at least one text area on the medical image based on the determined shape and position of the at least one contour and the determined position of the at least one text area.
[0032] The step of determining the position of at least one text area in one embodiment of the present disclosure includes a step of determining the position of the at least one text area based on at least one of a distance between the at least one contour and the at least one text area, or whether or not there is an overlap between the at least one contour and the at least one text area.
[0033] According to an embodiment of the present disclosure, the obtaining step includes obtaining lesion information for a plurality of lesions detected from the medical image. The determining step of determining a form and a position of at least one contour includes determining a portion of the plurality of lesions to be displayed in the medical image, and determining a form and a position of at least one contour for the determined portion of the lesions. The determining step of determining a position of at least one text area includes determining a position of at least one text area including lesion information for the determined portion of the lesions.
[0034] The step of determining some lesions according to one embodiment of the present disclosure includes the steps of identifying lesions, etc., among the multiple lesions, whose lesion areas overlap, and determining some of the multiple lesions based on at least one of the size of the overlapping area between the overlapping lesions, etc., the lesion probability of each of the overlapping lesions, etc., the association between the overlapping lesions, etc., or the probability that some of the overlapping lesions, etc. are present in one medical image.
[0035] The method according to an embodiment of the present disclosure further includes generating at least one arrow indicating the at least one contour, and the displaying step includes displaying the generated at least one arrow on the medical image so as to connect between the at least one contour and the at least one text region.
[0036] According to an embodiment of the present disclosure, the obtaining step includes obtaining lesion information related to a plurality of lesions detected from the medical image, the generating step includes generating an arrow for each of the plurality of lesions, and the displaying step includes displaying the generated arrows for each of the plurality of lesions on the medical image such that the generated arrows for each of the plurality of lesions do not cross each other.
[0037] According to an embodiment of the present disclosure, the obtaining step includes obtaining lesion information related to a plurality of lesions detected from the medical image, the generating step includes generating an arrow for each of the plurality of lesions, and the displaying step includes displaying the generated arrow for each of the plurality of lesions on the medical image such that the generated arrow for each of the plurality of lesions does not cross a contour corresponding to each of the plurality of lesions.
[0038] A step of displaying the generated arrow in a medical image according to one embodiment of the present disclosure includes a step of determining at least one contact area where the generated arrow contacts at least one contour, and a step of displaying the generated at least one arrow so that it is connected to the at least one contact area.
[0039] The step of determining at least one contact area according to one embodiment of the present disclosure includes determining at least one contact area based on a distance between a plurality of contact areas among a plurality of contact areas that are tangent to at least one contour.
[0040] In one embodiment of the present disclosure, the at least one text area includes a plurality of text areas, and the step of determining the position of the at least one text area includes a step of determining the position of each of the plurality of text areas based on a distance between the plurality of text areas.
[0041] An electronic device according to one embodiment of the present disclosure includes a memory for storing one or more instructions; and a processor configured to execute the one or more stored instructions to acquire lesion information for at least one lesion detected from a medical image, determine a shape and position of a contour corresponding to the at least one lesion based on the acquired lesion information, determine a position of at least one text area containing text indicating lesion information for the at least one lesion within the medical image, and display the at least one contour and the text contained in the at least one text area on the medical image based on the determined shape and position of the at least one contour and the determined position of the at least one text area.
[0042] The processor according to one embodiment of the present disclosure is further configured to determine a position of the at least one text region based on at least one of a distance between the at least one contour and the at least one text region, or whether or not there is an overlap between the at least one contour and the at least one text region.
[0043] A processor according to one embodiment of the present disclosure is further configured to obtain lesion information for a plurality of lesions detected from the medical image, determine a portion of the plurality of lesions to be displayed in the medical image, determine a shape and position of at least one contour for the determined portion of the lesions, and determine a position of at least one text region including lesion information for the determined portion of the lesions.
[0044] The processor according to one embodiment of the present disclosure is further configured to identify lesions, etc., among the multiple lesions, whose lesion areas overlap, and determine a lesion that is a part of the multiple lesions based on at least one of the size of the overlapping area between the overlapping lesions, etc., the lesion probability of each of the overlapping lesions, etc., the association between the overlapping lesions, etc., or the probability that a part of the overlapping lesions, etc. is present in one medical image.
[0045] The processor according to one embodiment of the present disclosure is further configured to generate at least one arrow indicating the at least one contour and display the generated at least one arrow on the medical image so as to connect between the at least one contour and the at least one text region.
[0046] A processor according to one embodiment of the present disclosure is further configured to acquire lesion information regarding a plurality of lesions detected from the medical image, generate an arrow for each of the plurality of lesions, and display the arrow for each of the plurality of lesions on the medical image such that the generated arrows for each of the plurality of lesions do not cross each other.
[0047] A processor according to one embodiment of the present disclosure is further configured to acquire lesion information regarding a plurality of lesions detected from the medical image, generate an arrow for each of the plurality of lesions, and display the arrow for each of the plurality of lesions on the medical image such that the generated arrow for each of the plurality of lesions does not intersect with each of the contours corresponding to each of the plurality of lesions.
[0048] The processor according to one embodiment of the present disclosure is further configured to determine at least one contact area where the generated arrow touches the at least one contour, and display the generated arrow as being connected to the at least one contact area.
[0049] The processor according to an embodiment of the present disclosure is further configured to determine at least one of a plurality of contact regions tangent to the at least one contour based on a distance between the plurality of contact regions.
[0050] According to an embodiment of the present disclosure, the at least one text region includes a plurality of text regions, and the processor is further configured to determine a position of each of the plurality of text regions based on a distance between the plurality of text regions.
[0051] In addition, a program for implementing the above-described method of operating the medical imaging apparatus may be recorded in a computer-readable recording medium. [Brief description of the drawings]
[0052] [Figure 1] FIG. 1 illustrates a medical imaging device according to one embodiment of the present disclosure. [Diagram 2] 10 is a flowchart illustrating an operation of a medical imaging apparatus according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates a medical image according to one embodiment of the present disclosure. [Figure 4] FIG. 2 illustrates pseudocode for a method of operation of a medical imaging device according to one embodiment of the present disclosure. [Diagram 5] 1 is a diagram showing a medical image displayed by a medical imaging device according to an embodiment of the present disclosure. [Figure 6] 1 is a diagram showing a medical image displayed by a medical imaging device according to an embodiment of the present disclosure. [Figure 7] 1 is a diagram showing a medical image displayed by a medical imaging device according to an embodiment of the present disclosure. [Figure 8] 1 is a diagram showing a medical image displayed by a medical imaging device according to an embodiment of the present disclosure. [Figure 9] 10 is a flowchart illustrating an operation of a medical imaging apparatus according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 11] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 13] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 14] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 15]FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 16] FIG. 2 is a diagram for explaining the operation of a medical imaging device according to an embodiment of the present disclosure. [Figure 17] FIG. 13 is a diagram for explaining a method of obtaining a score according to one embodiment of the present disclosure. [Figure 18] FIG. 13 is a diagram for explaining a method of obtaining a score according to one embodiment of the present disclosure. [Figure 19] FIG. 2 illustrates a medical image according to one embodiment of the present disclosure. [Figure 20] FIG. 2 illustrates a medical image according to one embodiment of the present disclosure. [Figure 21] FIG. 1 is a block diagram showing an internal configuration of an electronic device according to an embodiment of the present disclosure. [Figure 22] 4 is a flowchart illustrating a method of operating a medical imaging device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0053] The advantages and features of the disclosed embodiments and the methods for achieving them will become clearer with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in various different forms. However, the embodiments are provided only to make the disclosure complete and to allow those skilled in the art to accurately recognize the category of the invention.
[0054] The terms used in this specification are briefly explained, and the disclosed embodiments are specifically described.
[0055] The terms used in this specification are currently selected as widely used as possible while taking into consideration the function of this disclosure, but these may change depending on the intentions or precedents of engineers engaged in the relevant field, the emergence of new technology, etc. In addition, there may be terms arbitrarily selected by the applicant in specific cases, but the meanings of these will be described in detail in the description of the invention. Therefore, the terms used in this disclosure should be defined based on the meanings that the terms have and the overall content of this disclosure, rather than simply the names of the terms.
[0056] In this specification, unless otherwise clearly specified in the context, a singular expression can include a plural expression, and a plural expression can include a singular expression.
[0057] Throughout the specification, when a part "comprises" a certain element, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.
[0058] Additionally, the term "module" as used herein means a software or hardware component, and a "module" performs a certain function. However, the term "module" is not limited to software or hardware. A "module" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, a "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and "modules" may be combined into fewer components and "modules" or further separated into additional components and "modules" within which functionality is provided.
[0059] According to one embodiment of the present disclosure, a "unit" may be embodied with a processor and memory. A "processor" should be broadly construed to include a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, a "processor" may also refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. A "processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0060] The term "memory" should be interpreted broadly to include any electronic component capable of storing electronic information. "Memory" may also refer to various types of processor-readable media, such as Random Access Memory (RAM), Read Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read / write information from / to the memory. Memory integrated into a processor is in electronic communication with the processor.
[0061] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present disclosure. In the drawings, parts that are not relevant to the description will be omitted in order to clearly explain the present disclosure.
[0062] FIG. 1 is a diagram illustrating a medical imaging device according to an embodiment of the present disclosure.
[0063] The medical imaging apparatus 100 may include a controller 110, a database 120, and an output unit 130. The controller 110 may include a processor and a memory. The processor may execute commands stored in the memory.
[0064] The database 120 may store various data. For example, the database 120 may store at least one of medical images or analysis results of medical personnel and analysis results of a medical analysis device. The database 120 may also include a medical image analysis model for analyzing the medical images. The medical image analysis model may be a rule-based model or a machine learning model.
[0065] The output unit 130 may include at least one of a video output unit and an audio output unit, and may be controlled by the control unit 110. The output unit 130 may output at least one of a medical image and an analysis result.
[0066] The operation of the medical imaging device 100 will be described in detail below.
[0067] FIG. 2 is a flowchart showing the operation of the medical imaging apparatus according to an embodiment of the present disclosure.
[0068] The medical imaging device 100 may perform step 210 of acquiring lesion information related to at least one lesion included in the medical image. The medical image may be an image of a part of a patient's body. The medical image may be an image captured by various medical imaging devices. For example, the medical image may be one of CT, MRI, X-RAY, Mammography, or Ultrasound images. The medical image may be received from the database 120. The medical image may also be received from an external device. For example, the medical image may be received from an external medical imaging device that is a different device than the medical imaging device 100.
[0069] The lesion information may be information about a lesion included in a medical image. The lesion information may be information about a lesion detected by an automatic analysis performed by a medical image analysis device. However, the lesion information is not limited thereto, and may be an analysis result by a medical professional.
[0070] The medical imaging device 100 may receive the lesion information from an external device or from the database 120 of the medical imaging device 100. The medical imaging device 100 may also acquire the lesion information by analyzing the medical image based on an internal analysis tool. The internal analysis tool may be a rule-based model or a machine learning model for detecting the lesion. For example, the internal analysis tool may include a medical image analysis model trained based on machine learning.
[0071] The lesion information may include at least one of information on a lesion type, information on a lesion location, information on a probability that a lesion exists in a medical image, information on a lesion morphology, information on a lesion size, and information on a lesion area.
[0072] The information on the lesion type may include at least one of a lesion name or a lesion classification information. The information on the lesion type may be information diagnosed by a medical professional or information diagnosed based on a self-analysis tool. As described above, the medical image analysis device may acquire the information on the lesion type from the medical image based on a rule-based model or a machine learning model.
[0073] The lesion location information may indicate at least one point where the lesion is most likely to exist in the medical image. The lesion location information may be expressed by at least one coordinate value. The lesion location information may be the coordinate value of a pixel where the lesion is located in the medical image. If the lesion occupies a portion of the medical image, the lesion location information may be expressed by a plurality of coordinate values. In addition, the lesion location information may include the coordinate value of one lesion point and radius information from the lesion point. However, the lesion location information is not limited thereto and may be expressed in various manners.
[0074] The probability information of a lesion existing in a medical image indicates a probability of a lesion existing in a part of a patient's body corresponding to the medical image based on the medical image. The medical image analysis device may detect a lesion existing in the medical image using a rule-based model or a machine learning model. The medical image analysis device may also predict the probability of the lesion existing in the medical image based on the rule-based model or the machine learning model. The probability information of a lesion existing in a medical image may include a probability value corresponding to each pixel included in the medical image. The probability information of a lesion existing in a medical image may include a probability value corresponding to a specific region of the medical image. Here, the specific region is an area included in the medical image and may be an image that is the same as or smaller than the medical image. The probability value corresponding to the specific region of the medical image may be acquired by the medical image analysis device or received by a medical professional.
[0075] In addition, the medical imaging device 100 can determine a probability value corresponding to a specific region based on the probability values corresponding to pixels included in the specific region. For example, the medical imaging device 100 can obtain a representative probability value corresponding to the specific region using the maximum, average, median, or minimum value of the probability values of the pixels, etc. included in the specific region. Medical personnel can reduce the probability of misdiagnosis by making a diagnosis based on the probability information.
[0076] The lesion morphology information may indicate information related to the lesion morphology. The lesion morphology information may be information related to the border morphology of the lesion. The lesion morphology information may have a circular, elliptical or atypical shape. The lesion morphology information may be determined by a medical professional's diagnosis. In addition, the medical image analysis device may acquire the lesion morphology information based on a rule-based model or a machine learning model.
[0077] The lesion size information is information regarding the size of the lesion, and may be the number of pixels included in the lesion area in the medical image. Furthermore, the lesion size information may be the horizontal or vertical length of the lesion in the medical image. Furthermore, the lesion size information may be the width of the lesion area in the medical image. The lesion size information may indicate the number of pixels included in the lesion area. Furthermore, the lesion size information may indicate a radius. That is, the inner area of a circle based on the center point and radius of the lesion may indicate the lesion area.
[0078] The lesion area information may indicate an area where a lesion is highly likely to exist in the medical image. The medical imaging device 100 may receive the lesion area information from a medical professional. The medical imaging device 100 may also automatically acquire the lesion area information based on a rule-based model or a machine learning model. The medical imaging device 100 may calculate the probability that a lesion exists for each pixel of the medical image. Alternatively, the medical imaging device 100 may calculate the probability that a lesion exists for a pixel group. For example, the pixel group may include a plurality of pixels. The pixel group may indicate a certain area in the medical image. The medical imaging device 100 may also calculate the probability that a lesion exists for the pixel group. If the probability that a lesion exists in a pixel is equal to or greater than a threshold, the medical imaging device 100 may determine that the pixel is included in the lesion area. That is, the probability that a lesion exists in a pixel included in the lesion area may be equal to or greater than a threshold. The threshold may be predetermined information.
[0079] The medical imaging device 100 may perform step 220 of generating at least one contour corresponding to at least one lesion in the medical image based on the acquired lesion information.
[0080] The contour may indicate one of the position or shape of a lesion. The medical imaging device 100 may display at least one lesion for one medical image. The multiple lesions may have different positions and shapes. The medical imaging device 100 may acquire the position and shape of at least one lesion based on the lesion information. The medical imaging device 100 may generate a contour reflecting the position and shape of each lesion. That is, the medical imaging device 100 may determine the position and shape of at least one contour based on the acquired lesion information. That is, the medical imaging device 100 may determine the position of the contour based on the position of the lesion included in the lesion information of the medical imaging device 100. Also, the medical imaging device 100 may determine the shape of the contour based on the shape of the lesion included in the lesion information.
[0081] The position of the contour may correspond to the lesion position information included in the lesion information. The medical imaging device 100 may determine the shape of the contour based on the lesion information. The medical imaging device 100 may determine the contour through the following process.
[0082] More specifically, the medical imaging device 100 may determine the contour based on the border of the lesion area included in the lesion information. As described above, the medical imaging device 100 may determine that a pixel is included in the lesion area if the probability that a lesion exists in the pixel is equal to or greater than a threshold. That is, the probability that a lesion exists in a pixel, etc., within the lesion area may be equal to or greater than a threshold. The medical imaging device 100 may determine the border of the lesion area included in the lesion information as the contour. The contour may surround the lesion area.
[0083] In addition, the medical imaging device 100 can expand the border of the lesion area by A times to obtain a contour including the lesion. A can be a real number greater than 1. That is, the contour can be the same as or larger than the lesion area. Expanding the border of the lesion area by A times means that the horizontal length of the lesion area becomes A times and the vertical length becomes A times. Therefore, the inner area of the contour can be A^2 times the area of the lesion area.
[0084] In addition, the medical imaging device 100 can determine a polygon, ellipse, or circle including a lesion area included in the lesion information as a contour. According to the present disclosure, medical personnel can focus on observing the periphery of the contour, which is a part of the medical image, and can finally easily determine whether a lesion exists inside the contour.
[0085] The shape of the contour may further include a thickness of the contour line. The medical imaging device 100 may obtain the thickness of the contour line based on the lesion information. For example, the medical imaging device 100 may obtain a probability that each pixel in the medical image is included in at least one lesion area based on the lesion information. In addition, the medical imaging device 100 may determine the thickness of at least one contour based on the probability that a pixel included within the contour is included in at least one lesion area. For example, the thickness of the contour may be thicker as the probability increases.
[0086] The medical imaging device 100 can obtain the probability that each pixel included in the medical image is included in the lesion. In addition, the medical imaging device 100 can determine a set of pixels, etc., whose probability of being included in the lesion is equal to a threshold value, as a contour. Since the probability of being included in the lesion is discontinuous, there may not be a value equal to the threshold value. In this case, if the probability corresponding to one of two adjacent pixels is greater than the threshold value and the probability corresponding to the other pixel is less than the threshold value, one of the two adjacent pixels can be included in the contour. The probability that a pixel inside the contour is included in the lesion area may be greater than the threshold value. Here, the threshold value may be a preset value.
[0087] In addition, the medical imaging device 100 may determine a set of pixels, etc., whose probability of being included in the lesion falls within a threshold range as a contour. If the threshold range is too narrow, the set of pixels, etc., may appear as a line. The probability that a pixel inside the contour is included in the lesion area may be greater than the minimum value of the threshold range. Here, the threshold range may be a predetermined range.
[0088] The above describes a configuration in which the medical imaging device 100 generates one contour for one lesion. When multiple lesions are present in a medical image, the medical imaging device 100 can generate multiple contours corresponding to the multiple lesions. This will be described with reference to FIG. 3.
[0089] FIG. 3 is a diagram illustrating a medical image according to one embodiment of the present disclosure.
[0090] The medical imaging device 100 may generate at least one contour for each lesion included in the medical image. The medical imaging device 100 may generate at least one contour, such as a contour line, for each lesion. The probability that a pixel within one of a plurality of lines included in the contour line is included in the lesion area may be greater than a threshold.
[0091] 3, the medical image may include a first contour line 311 and a second contour line 312 for one lesion. A probability that a pixel included in an inner region of the first contour line 311 is included in the lesion region may be equal to or greater than a first threshold. A probability that a pixel included in an inner region of the second contour line 312 is included in the lesion region may be equal to or greater than a second threshold. A probability that a pixel included in an inner region of the second contour line 312 and an outer region of the first contour line 311 is included in the lesion region may be lower than the first threshold and equal to or greater than the second threshold. Here, the first threshold and the second threshold may be predetermined values. The first threshold may be greater than the second threshold.
[0092] The medical imaging device 100 can generate a thicker contour line when the probability that the pixels inside the contour line are included in the lesion area is higher. For example, the probability that the pixels included in the inner area of the first contour line 311 are included in the lesion area may be high. However, the probability that the pixels included between the first contour line 311 and the second contour line 312 are included in the lesion area may be relatively low. Therefore, the medical imaging device 100 can display the first contour line 311 thicker than the second contour line 312.
[0093] The medical imaging device 100 can generate different colors for the contour lines according to the probability that the pixels inside the contour lines are included in the lesion area. For example, the probability that the pixels included in the inner area of the first contour line 311 are included in the lesion area may be high. However, the probability that the pixels included between the first contour line 311 and the second contour line 312 are included in the lesion area may be relatively low. Thus, the medical imaging device 100 can display the first contour line 311 in red and the second contour line 312 in orange.
[0094] The medical imaging device 100 may display the area between the contour lines in different colors based on the probability that the pixels inside the contour lines are included in the lesion area. For example, the probability that the pixels included in the area inside the first contour line 311 are included in the lesion area may be high. However, the probability that the pixels included between the first contour line 311 and the second contour line 312 are included in the lesion area may be relatively low. Thus, the medical imaging device 100 may display the inside of the first contour line 311 in red and the area between the second contour line 312 and the first contour line 311 in orange. The medical imaging device 100 may adjust the transparency so that the displayed colors do not block the original medical image.
[0095] The probability that a pixel included in one of the multiple lines included in the contour line is included in the lesion area may be included in a specific threshold range. The specific threshold range may be a predetermined range. Referring to FIG. 3, the probability that a pixel included in the first contour line 311 is included in the lesion area may be included in a first threshold range. Also, the probability that a pixel included in the second contour line 312 is included in the lesion area may be included in a second threshold range. The minimum value of the first threshold range may be greater than the maximum value of the second threshold range.
[0096] In addition, the medical imaging device 100 can generate a plurality of contours by expanding the border of the lesion area included in the lesion information. For example, the first contour line 311 can be the border of the lesion area. The medical imaging device 100 can expand the first contour line 311 to obtain the second contour line 312. The second contour line 312 can be thinner than the first contour line 311.
[0097] The medical imaging device 100 may perform a step 230 of outputting at least one contour generated on the medical image. The output unit 130 may output the medical image and at least one contour.
[0098] To generate at least one contour, the medical imaging device 100 may perform the following steps.
[0099] The medical imaging device 100 can perform a step of generating a contour based on the acquired lesion information, based on at least one of information relating to overlapping lesion areas between multiple lesions contained in the medical image, or information relating to the relationship between multiple lesions.
[0100] A plurality of lesions may be detected in a medical image. Lesion information may be obtained for each of the plurality of lesions. As described above, the lesion information may include lesion area information.
[0101] The medical imaging device 100 can obtain information on overlapping lesion areas between multiple lesions based on the lesion information, and can also obtain information on the association between multiple lesions.
[0102] The information on the association between multiple lesions is information indicating whether or not there is a medical association between multiple lesions. For example, the information on the association may include information on the presence or absence of coexistence, indicating whether or not multiple lesions appear in the same patient at the same time. For example, if a first lesion does not appear simultaneously with a second lesion, there is no coexistence, and if a first lesion appears simultaneously with a second lesion, there is coexistence.
[0103] The information on the association may also include information on the presence or absence of similarity, which indicates whether or not a plurality of lesions are likely to be manifested simultaneously. A plurality of lesions may have names that can be distinguished, but may be substantially similar. For example, when a first lesion is manifested medically, a second lesion is also manifested, there is a similarity. When a first lesion is manifested, a second lesion is also manifested, which does not mean that the first lesion is manifested when the second lesion is manifested. However, without being limited thereto, when a first lesion is manifested, a second lesion is also manifested, which may mean that the first lesion is manifested when the second lesion is manifested.
[0104] In addition, since the appearance of the first lesion and the appearance of the second lesion are independent, if the appearance of the first lesion has no effect on the appearance of the second lesion, the first lesion and the second lesion are dissimilar. In addition, if the appearance of the second lesion has no effect on the appearance of the first lesion, the first lesion and the second lesion are dissimilar.
[0105] The medical imaging device 100 can acquire information related to the association based on a rule-based model or a machine learning model. The medical imaging device 100 can acquire information related to the association based on the database 120. The medical imaging device 100 can acquire first lesion information related to a first lesion. Also, the medical imaging device 100 can acquire second lesion information related to a second lesion. The medical imaging device 100 can acquire a first lesion identifier including at least one of a code, a name, and a type of the first lesion from the first lesion information. Also, the medical imaging device 100 can acquire a second lesion identifier including at least one of a code, a name, and a type of the second lesion from the second lesion information. The medical imaging device 100 can acquire information related to the association between the first lesion and the second lesion from the database based on the first lesion identifier and the second lesion identifier. For example, the database can store information related to the association between the first lesion identifier and the second lesion identifier in advance. The medical imaging device 100 can derive association information corresponding to the first lesion identifier and the second lesion identifier from the database.
[0106] The medical imaging device 100 can obtain the information on the association from something not in the database 120. The medical imaging device 100 can obtain the information on the association by applying the first lesion information and the second lesion information to a machine learning model. The machine learning model can be a model in which the association between the first lesion and the second lesion is learned by machine learning. The medical imaging device 100 can obtain the information on the association from a medical professional. Also, the medical imaging device 100 can obtain the information on the association from the lesion information.
[0107] The first lesion information related to the first lesion may include information on a lesion, etc. that is similar to the first lesion, and information on a lesion, etc. that is coexistent with the first lesion. The medical imaging device 100 can determine whether the second lesion is similar to the first lesion and whether there is coexistence based on the first lesion information.
[0108] The medical imaging device 100 can obtain information related to the association based on an input signal from a medical professional, and can also obtain information related to the association from an external device.
[0109] The medical imaging device 100 can perform a step of determining the size of an overlapping area of a first lesion area and a second lesion area based on first lesion information and second lesion information contained in multiple lesions in order to generate at least one contour.
[0110] As described above, the medical imaging device 100 can obtain first lesion information related to a first lesion and second lesion information related to a second lesion. The medical imaging device 100 can obtain first lesion area information in the medical image from the first lesion information, and obtain second lesion area information in the medical image from the second lesion information. The medical imaging device 100 can also determine a size of an overlapping area between the first lesion area and the second lesion area. The size of the overlapping area can be represented as at least one of the number of pixels in the overlapping area, the width of the overlapping area, the horizontal length of the overlapping area, or the vertical length of the overlapping area.
[0111] Here, the first lesion area and the second lesion area may simply mean the area of the first lesion and the area of the second lesion, but are not limited thereto. The first lesion area may also mean the area inside the first contour corresponding to the first lesion. Also, the second lesion area may also mean the area inside the second contour corresponding to the second lesion.
[0112] The medical imaging device 100 may perform a step of determining information related to a probability that the second lesion is present in the medical image based on the second lesion information. As described above, the information related to the probability that the second lesion is present in the medical image may be a probability value corresponding to each pixel value included in the medical image or a probability value corresponding to a specific region included in the medical image. Here, the specific region may correspond to the generated contour.
[0113] In addition, the medical imaging device 100 may perform a step of generating at least one contour surrounding the first lesion when the size of the overlapping region is greater than a first threshold and the information related to the probability that the second lesion exists in the medical image is less than a second threshold. The first threshold and the second threshold may be predetermined values. The second threshold may be, but is not limited to, the information related to the probability that the first lesion exists in the medical image.
[0114] The contour may be the same as or larger than the area of the first lesion. The method for generating at least one contour surrounding the first lesion has been described above, and therefore will not be described again.
[0115] When the size of the overlapping area is greater than the first threshold and the information regarding the probability that the second lesion exists in the medical image is less than the second threshold, the medical imaging device 100 may only generate at least one contour surrounding the first lesion, but not generate at least one contour surrounding the second lesion, because the size of the overlapping area is greater than the first threshold, the first lesion and the second lesion overlap to a significant extent, and the information regarding the probability that the second lesion exists in the medical image is less than the second threshold, so the importance of the second lesion in the medical image is relatively low.
[0116] The medical imaging device 100 may perform a step of determining whether there is a pathological similarity between the first lesion and the second lesion in order to generate at least one contour. Whether there is a similarity may be included in the information related to the association. The information related to the similarity has been described above, and therefore a duplicated description will be omitted. In addition, the medical imaging device 100 may determine whether there is a similarity using a rule-based model based on a database, obtain the information from a machine learning model, receive the information from a medical professional, or obtain the information from the lesion information. The process of obtaining the information related to the association has been described above, and therefore a duplicated description will be omitted.
[0117] The medical imaging device 100 may perform a step of determining whether the first lesion and the second lesion coexist in the same region. The information regarding the presence or absence of coexistence has been described above, and therefore a duplicated description will be omitted. In addition, the medical imaging device 100 may determine whether or not there is coexistence using a rule-based model based on a database, obtain the information from a machine learning model, receive the information from a medical professional, or obtain the information from lesion information. The process of obtaining the information regarding the association has been described above, and therefore a duplicated description will be omitted.
[0118] The medical imaging device 100 may perform a step of generating at least one contour surrounding the first lesion and not generating at least one contour surrounding the second lesion if the similarity determination indicates that the first lesion and the second lesion are similar, or if the coexistence determination indicates that the first lesion and the second lesion cannot coexist with each other.
[0119] When the similarity indicates that the first lesion and the second lesion are similar, the medical imaging device 100 can generate at least one contour surrounding only the first lesion. Similarity means that the second lesion may also appear when the first lesion appears. When the similarity indicates that the first lesion and the second lesion are not similar, the medical imaging device 100 can generate at least one contour corresponding to the first lesion and at least one contour corresponding to the second lesion.
[0120] In addition, the medical imaging device 100 may generate at least one contour surrounding only one of the first lesion or the second lesion when the coexistence indicates that the first lesion and the second lesion cannot coexist with each other. In order to determine whether to generate a contour for one of the first lesion or the second lesion, the medical imaging device 100 may compare the probability that the first lesion and the second lesion exist in the medical image. In addition, if the probability that the first lesion exists in the medical image is higher than that of the second lesion, the medical imaging device 100 may generate at least one contour surrounding the first lesion. The medical imaging device 100 may not generate at least one contour surrounding the second lesion. Conversely, if the probability that the first lesion exists in the medical image is lower than that of the second lesion, the medical imaging device 100 may generate at least one contour surrounding the second lesion. The medical imaging device 100 may not generate at least one contour surrounding the first lesion.
[0121] The medical imaging device 100 may further perform a step of sorting the plurality of pieces of lesion information in order of a probability of existence in the medical image based on the plurality of pieces of lesion information. By sorting the plurality of pieces of lesion information in order of a probability of existence in one medical image, the medical imaging device 100 may not output a contour corresponding to a lesion having a low probability of existence when there is no coexistence between two lesions.
[0122] In addition, if the coexistence determination indicates that the first lesion and the second lesion can coexist with each other, the medical imaging device 100 can generate at least one contour corresponding to the first lesion and at least one contour corresponding to the second lesion.
[0123] FIG. 4 illustrates pseudocode for a method of operation of a medical imaging device according to one embodiment of the present disclosure.
[0124] In line 410, C may denote a set of contours, etc., corresponding to multiple detected lesions. C may contain a total of M elements, i.e., C={c1, c2, . . . , c M}.
[0125] The medical imaging device 100 can execute the for statement while the index (i) increases from 1 to M-1. In line 420, R(i) is a function that arranges the contour elements in the set C in order of the probability of being present in the medical image. That is, R(1) can output at least one contour for a lesion with the highest probability of being present in the medical image. Also, R(M-1) can output at least one contour for a lesion with the lowest probability of being present in the medical image. The medical imaging device 100 can select a contour corresponding to the lesion with the i-th highest probability according to R(i). p The medical imaging device 100 assigns a contour for a lesion that is present in the medical image with a low probability as i increases according to R(i). p can be assigned to
[0126] At line 430, D(c p ) is the contour cp The medical imaging device 100 outputs the lesion information corresponding to D(c p ) by c p The corresponding lesion information is p can be assigned to
[0127] The medical imaging device 100 can execute the for statement while the index (j) increases from 2 to M. In FIG 4, the index (j) increases from 2, but is not limited thereto. The index (j) can also increase from i+1 to M.
[0128] In line 440, R(j) is a function that arranges the contour elements in the set C in order of the probability that they are present in the medical image. The medical imaging device 100 uses R(j) to arrange the contour corresponding to the lesion with the jth highest probability as c q can be assigned to
[0129] At line 450, D(c q ) is the contour c q The medical imaging device 100 outputs the lesion information corresponding to D(c q ) by c q The corresponding lesion information is q can be assigned to
[0130] At line 460, IoU(c p , c q ) is the contour c p and contour c q The size of the overlapping area can be shown by S(c q ) is the contour c q It can indicate the probability that a pixel inside is included in the lesion area.
[0131] At line 460, the medical imaging device 100 p and contour c q The size of the overlapping region is the threshold (MTd q ) and the contour c qThe probability that the pixels inside the lesion area are included in the lesion area is the threshold (STd q ), line 470 can be performed.
[0132] At line 470, Sim(d p , d q ) is the lesion information d p and lesion information d q It can output information about whether or not the similarity of Sim(d p , d q ) is 1, then the lesion information d p and lesion information d q It can be shown that there is a similarity between Sim(d p , d q ) is 0, then the lesion information d p and lesion information d q It can be shown that there is no similarity between them.
[0133] CoOcc(d p , d q ) is the lesion information d p and lesion information d q CoOcc(d p , d q ) is 1, then the lesion information d p and lesion information d q It can be shown that there is a compatibility between CoOcc(d p , d q ) is 0, then the lesion information d p and lesion information d q It can be shown that there is no compatibility between them.
[0134] At line 470, the medical imaging device 100 receives the lesion information d p and lesion information d q There is a similarity between the lesion information p and lesion information d q If there is no compatibility between them, line 480 can be performed.
[0135] If the above condition is satisfied, the medical imaging device 100 extracts the contour c from the set C.q That is, the medical imaging device 100 can remove the contour c p and contour c q The size of the overlapping region is the threshold (MTd q ) and the contour c q The probability that the pixels inside the lesion area are included in the lesion area is the threshold (STd q ) and lesion information d p and lesion information d q There is a similarity between the lesion information p and lesion information d q If there is no compatibility between them, then the set C to the contour c q The medical imaging device 100 can output the contours included in the set C to the output unit 130. That is, the medical imaging device 100 can output the contours c q can be suppressed.
[0136] 5 to 8 are diagrams showing medical images displayed by a medical imaging device according to an embodiment of the present disclosure.
[0137] The medical imaging device 100 can display an arrow or text near the contour. The text can include content related to the contour. Figures 5 to 8 show examples of the arrangement of arrows or text located near the contour. As shown in Figures 5 to 8, the medical imaging device 100 can arrange the arrows and text so that the user can clearly understand the meaning of the contour.
[0138] As shown in Fig. 5, the medical imaging device 100 can output a contour along with a medical image. The medical imaging device 100 can also display an arrow and text adjacent to the contour. The arrow can connect between the contour and the text. The text can describe a lesion included in the contour. The medical imaging device 100 can obtain the text based on the lesion information.
[0139] A medical image 510 shows an example of an incorrect display of arrows and text. Referring to box 511, multiple texts are displayed overlapping each other. Also, multiple arrows may cross each other at least in part. Therefore, it is difficult for medical personnel to confirm the contents of the text.
[0140] A medical image 520 shows a medical image normally displayed by the medical imaging device 100 according to the present disclosure. Referring to box 521, the arrows included in the medical image 520 do not cross each other, and the text does not overlap. Therefore, medical personnel can clearly understand the meaning of the contours.
[0141] As shown in FIG 6, a medical image 610 shows an example of an arrow and text being displayed incorrectly. Referring to box 611, an arrow crosses a contour. When an arrow crosses a contour, the length of the arrow becomes longer and the distance between the contour and the text that the arrow points to may become farther. Therefore, it is difficult for medical personnel to confirm which contour the text is describing.
[0142] A medical image 620 shows a medical image normally displayed by the medical imaging device 100 according to the present disclosure. Referring to box 621, the arrow included in the medical image 620 does not cross the contour, so that medical personnel can clearly understand the meaning of the contour.
[0143] As shown in Fig. 7, a medical image 710 shows an example in which an arrow and text are displayed incorrectly. Referring to box 711, the arrow and text are displayed on a contour. When the arrow and text are displayed on a contour, the readability of the text is reduced. Therefore, it is difficult for medical personnel to confirm the content of the text.
[0144] A medical image 720 shows a medical image normally displayed by the medical imaging device 100 according to the present disclosure. Referring to box 721, arrows and texts included in the medical image 720 are not displayed on the contours, so that medical personnel can clearly understand the meaning of the contours.
[0145] As shown in Figure 8, a medical image 810 shows an example of an arrow and text being displayed incorrectly. Referring to box 811, an arrow is displayed at the intersection of two contours. When an arrow is displayed at the intersection of two contours, it is difficult to know which contour the arrow is pointing to. Therefore, it is difficult for medical personnel to check the text corresponding to the contour.
[0146] A medical image 820 shows a medical image normally displayed by the medical imaging device 100 according to the present disclosure. Referring to box 821, an arrow included in the medical image 820 is not displayed at the intersection of two contours, so that medical personnel can clearly see the text corresponding to the contours.
[0147] Fig. 9 is a flowchart showing the operation of the medical imaging apparatus according to an embodiment of the present disclosure, and Figs. 10 to 13 are diagrams for explaining the operation of the medical imaging apparatus according to an embodiment of the present disclosure.
[0148] The medical imaging apparatus 100 may perform step 910 of determining first candidate arrow information related to a plurality of first candidate arrows indicating a first contour among the at least one contour to output the at least one contour.
[0149] The candidate arrow information may include at least one of position information regarding the start point of the arrow, position information regarding the end point of the arrow, information regarding the direction of the arrow, and information regarding the length of the arrow.
[0150] Referring to the upper left diagram of FIG. 10, the medical imaging apparatus 100 may obtain a plurality of first candidate arrows 1021, 1022, 1023 indicating a first contour 1010 among at least one contour. The plurality of first candidate arrows 1021, 1022, 1023 may be approximately perpendicular to the first contour 1010. The plurality of first candidate arrows 1021, 1022, 1023 may include a start point and an end point. In the present disclosure, the start point of an arrow may mean a point on the arrow that is closest to the contour. In the present disclosure, the end point may mean a point on the arrow opposite to the start point. The start point of an arrow may be, but is not limited to, the head of the arrow, and the end point of the arrow may be the head of the arrow.
[0151] The positions of the starting points of the plurality of first candidate arrows 1021, 1022, 1023 may be randomly determined on the contour 1010. Also, the number of the plurality of first candidate arrows 1021, 1022, 1023 may be a predetermined number. Although not all of the plurality of first candidate arrows are given reference numerals in FIG. 10, the number of the plurality of first candidate arrows 1021, 1022, 1023 may be 10. The lengths of the plurality of first candidate arrows 1021, 1022, 1023 may be predetermined. Also, the lengths of the plurality of first candidate arrows 1021, 1022, 1023 may be randomly determined within a predetermined range.
[0152] 11, the medical imaging apparatus 100 may obtain a plurality of first candidate arrows 1121, 1122, 1123 indicating a first contour 1110 among at least one contour. Text boxes 1131, 1132, 1133 may be located at the end points of the plurality of first candidate arrows 1121, 1122, 1123. Text may be displayed in the text boxes 1131, 1132, 1133. The plurality of first candidate arrows 1121, 1122, 1123 and the text boxes 1131, 1132, 1133 may have a 1:1 correspondence.
[0153] The positions of the first candidate arrows 1121, 1122, 1123 and the text boxes 1131, 1132, 1133 may be set in a predetermined manner. The positions of the text boxes 1131, 1132, 1133 may be set based on the directions of the first candidate arrows 1121, 1122, 1123. In addition, the medical imaging apparatus 100 may set the position of the text box so that one of the first candidate arrows 1121, 1122, 1123 is located at an edge or side of the text box. For example, if the start point of the first candidate arrow 1121 is at the upper left of the end point, the upper left vertex of the text box 1131 may be adjacent to the end point of the first candidate arrow 1121. In addition, if the start point of the first candidate arrow 1122 is at the left of the end point, the left side of the text box 1132 may be adjacent to the end point of the first candidate arrow 1122. FIG. 11 explains the relationship between the arrows and the text, so further explanation is omitted.
[0154] The positional relationship between the candidate arrows and the text boxes shown in FIG. 11 is only one embodiment, and various other positional relationships between the candidate arrows and the text boxes are possible.
[0155] The head of the candidate arrow 1160 may be located at an end point, i.e., the starting point of the candidate arrow 1160 may be close to or abut the contour 1150, and the end point of the candidate arrow 1160 may be close to or abut the edge or side of the text region 1170 (e.g., a text region in the form of a box, etc.).
[0156] Still referring to FIG. 9, the medical imaging system 100 can perform a step 920 of determining an arrow-text region 1030 on the periphery of the first contour 1010 .
[0157] 10, an arrow-text area 1030 may be an area that includes the end of an arrow and a part of a text box. In the arrow-text area 1030, the end of an arrow and a text box may be adjacent to each other.
[0158] 12, the medical imaging device 100 can generate a contour 1210. The medical imaging device 100 can generate a first extended contour 1221 by extending the contour 1210. The medical imaging device 100 can also generate a second extended contour 1222 by extending the contour 1210. The second extended contour 1222 can include the first extended contour 1221. The medical imaging device 100 can use the contour 1210 at a predetermined multiple to obtain the first extended contour 1221 and the second extended contour 1222. The medical imaging device 100 can determine an arrow-text region 1230 between the first extended contour 1221 and the second extended contour 1222. The arrow-text region 1230 can be a ring-shaped region surrounding the first contour 1210 without contacting the first contour 1210.
[0159] If the end points of the multiple first candidate arrows generated in step 910 are not included in the arrow-text region 1030, the medical imaging device 100 can adjust the start points or end points of the multiple first candidate arrows so that the end points of the multiple first candidate arrows are included in the arrow-text region 1030.
[0160] 9, the medical imaging apparatus 100 may perform step 930 of determining a contact area where one side of a text box that corresponds to the first candidate arrows and displays lesion information related to the first contour meets one end of the first candidate arrows included in the first candidate arrow information in the arrow-text area. Here, the one side may indicate an edge or side of the text box, and the one end may be a start point or an end point of the candidate arrow.
[0161] 10, the medical imaging apparatus 100 can determine the text to be inserted into the text box based on the lesion information. Also, the medical imaging apparatus 100 can determine the size of the text box based on the font and content of the text. The medical imaging apparatus 100 can place the text box at the end points of the first candidate arrows 1041, 1042, and 1043.
[0162] The text content may include at least one of information about a lesion type, information about the probability that a lesion exists in a medical image, information about the lesion morphology, information about the lesion size, and information about the probability that a lesion exists in a unit area of a medical image.
[0163] The medical imaging device 100 may determine information about the text box. The information about the text box may include the size of the text box, the height of the text box, the width of the text box, or the position information of the text box.
[0164] The medical imaging device 100 can determine the location of the text box based on information about the arrow. For example, the medical imaging device 100 can determine that one edge or side of the text box is located within a predetermined distance from the end point of the arrow.
[0165] The medical imaging device 100 may determine whether text boxes corresponding to the plurality of first candidate arrows 1041, 1042, and 1043 can be located within the medical image. In addition, the medical imaging device 100 may determine an area of the arrow-text area 1070 that allows the text boxes to be located within the medical image as a contact area. For example, a part of the text box corresponding to the first candidate arrow 1041 may be located outside the medical image 1050. The medical imaging device 100 may determine the contact area by removing an area in the arrow-text area 1070 where the first candidate arrow 1041 is located.
[0166] The medical imaging device 100 may also determine whether text boxes corresponding to the first candidate arrows 1041, 1042, and 1043 overlap with an object. The medical imaging device 100 may also determine an area where the text boxes do not overlap with the object as a contact area. Here, the object may include at least one of a contour, another text box, or another arrow. For example, a part of the text box corresponding to the first candidate arrow 1042 may overlap with another contour 1060. The medical imaging device 100 may determine the contact area by removing an area where the first candidate arrow 1042 exists in the arrow-text area 1070.
[0167] If the text box corresponding to the first candidate arrow 1043 is located within the medical image and does not overlap with an object, the medical imaging device 100 can obtain the contact area using the area including the end point of the first candidate arrow 1043.
[0168] 10, through the above process, the medical imaging apparatus 100 can obtain a part of the arrow-text region 1070 as a contact region 1080. The medical imaging apparatus 100 can select a plurality of candidate arrows whose end points are included in the contact region, and can generate at least one arrow set using the selected candidate arrows.
[0169] Still referring to FIG. 9, the medical imaging device 100 can perform step 940 of generating a text box for the first contour and at least one arrow set for displayable positions of the arrows based on the determined contact area.
[0170] 13, a medical image may include a plurality of contours 1310, 1320, and 1330. The medical imaging device 100 may determine a contact area for each of the plurality of contours 1310, 1320, and 1330 based on steps 910 to 930. The medical imaging device 100 may also determine a plurality of candidate arrows for each of the plurality of contours 1310, 1320, and 1330. The medical imaging device 100 may select a plurality of candidate arrows whose end points are included in the contact area. The medical imaging device 100 may also remove candidate arrows whose end points are not included in the contact area.
[0171] The medical imaging device 100 may generate an arrow set by selecting one candidate arrow per contour from among a plurality of candidate arrows at least a portion of which is included in the contact area. That is, the medical imaging device 100 may generate an arrow set by selecting one of a plurality of candidate arrows corresponding to one contour.
[0172] For example, referring to SET1 in Fig. 13, the medical imaging device 100 can select a 1-1 arrow 1311 from among a plurality of candidate arrows for a first contour 1310. Also, referring to SET1, the medical imaging device 100 can select a 2-1 arrow 1321 from among a plurality of candidate arrows for a second contour 1320. Also, referring to SET1, the medical imaging device 100 can select a 3-1 arrow 1331 from among a plurality of candidate arrows for a third contour 1330. The 1-1 arrow 1311, the 2-1 arrow 1321, and the 3-1 arrow 1331 can be included in SET1.
[0173] 13, the medical imaging apparatus 100 may select a 1-2 arrow 1312 from among a plurality of candidate arrows for a first contour 1310. Also, with reference to SET2, the medical imaging apparatus 100 may select a 2-2 arrow 1322 from among a plurality of candidate arrows for a second contour 1320. Also, with reference to SET2, the medical imaging apparatus 100 may select a 3-2 arrow 1332 from among a plurality of candidate arrows for a third contour 1330. The 1-2 arrow 1312, the 2-2 arrow 1322, and the 3-2 arrow 1332 may be included in SET2.
[0174] The medical imaging device 100 may generate a predetermined number of sets. The medical imaging device 100 may randomly select arrows to be included in the set from among a plurality of candidate arrows. The medical imaging device 100 may position a text box at an end point of an arrow in a predetermined manner. The medical imaging device 100 may include arrow information in the arrow set. That is, the medical imaging device 100 may include arrow information corresponding to each of a plurality of contours in the arrow set and store it.
[0175] At least one arrow set may include at least one of arrow information or information about a text box. As described above, the information about a text box may include a size of a text box, a height of a text box, a width of a text box, or a position information of a text box.
[0176] 9, the medical imaging device 100 may perform step 950 of acquiring a score for each of the at least one arrow set. The score indicates the degree to which the medical practitioner can stably review at least one of the medical image, the contours, the arrows, and the text. A higher score indicates that the medical practitioner can stably review at least one of the medical image, the contours, the arrows, and the text. The step of acquiring a score will be described in more detail in conjunction with FIGS. 17 and 18.
[0177] The medical imaging device 100 may perform a step of selecting one of the at least one arrow set based on the obtained score. The medical imaging device 100 may select the arrow set with the highest score. For example, in FIG. 13, if SET1 has the highest score among SET1 and SET2, the medical imaging device 100 may select SET1. In addition, the medical imaging device 100 may not select the arrow set with the highest score if the arrows in the arrow set cross each other.
[0178] The medical imaging device 100 may not select an arrow set when a first arrow corresponding to a first contour and a second arrow corresponding to a second contour intersect with each other in the arrow set, or may remove at least one arrow set when a first arrow corresponding to a first contour and a second arrow corresponding to a second contour intersect with each other in the arrow set.
[0179] 9, the medical imaging device 100 may perform step 960 of outputting an arrow and text in a text box together with at least one contour based on a selected arrow set. The medical imaging device 100 may output an arrow based on a plurality of arrow information included in the selected arrow set. The medical imaging device 100 may also determine the content of the text based on the lesion information. The medical imaging device 100 may also position the text box adjacent to the arrow based on a predetermined method. The medical imaging device 100 may also display text in the text box. The text box may not be displayed.
[0180] For example, in FIG. 13, when the medical imaging device 100 selects SET1, the medical imaging device 100 can output a medical image 1300, a plurality of contours 1310, 1320, 1330, a plurality of arrows 1311, 1321, 1331, and text, as in SET1.
[0181] FIG. 14 is a diagram for explaining the operation of the medical imaging apparatus according to an embodiment of the present disclosure.
[0182] The medical imaging apparatus 100 may perform the following process to determine the first candidate arrow information: The following steps may be performed together with one of steps 910 to 940 of FIG.
[0183] When a first text box for a first contour overlaps with a second text box for a second contour or with a second contour, the medical imaging device 100 may perform a step of acquiring modified first candidate arrow information by moving a start point or an end point of a candidate arrow included in the first candidate arrow information so that the first text box does not overlap with the second text box or with the second contour.
[0184] With reference to the medical image 1410, the medical imaging device 100 may determine that a first text box 1411 for a first contour overlaps with a second text box 1412 for a second contour. The medical imaging device 100 may obtain modified first candidate arrow information by moving a start point or an end point of a candidate arrow 1413 included in the first candidate arrow information so that the first text box 1411 does not overlap with the second text box 1412. For example, the medical imaging device 100 may move the candidate arrow 1413 by an amount where the first text box 1411 and the second text box 1412 overlap.
[0185] The medical image 1420 shows a candidate arrow 1423 based on the modified first candidate arrow information. The medical imaging device 100 can obtain a modified first text box 1421 based on the candidate arrow 1423. The modified first text box 1421 can be arranged not to overlap with a second text box 1422.
[0186] With reference to the medical image 1430, the medical imaging device 100 may determine that a first text box 1431 for a first contour overlaps with a second contour 1432. The medical imaging device 100 may move a start point or an end point of a candidate arrow 1433 included in the first candidate arrow information so that the first text box 1431 does not overlap with the second contour 1432, thereby obtaining modified first candidate arrow information.
[0187] The medical image 1440 shows a candidate arrow 1443 based on the modified first candidate arrow information. The medical imaging device 100 can obtain a modified first text box 1441 based on the candidate arrow 1443. The modified first text box 1441 can be arranged not to overlap with the second contour 1442.
[0188] The medical imaging device 100 may perform a step of generating at least one set of arrows based on the modified first candidate arrow information. This step may be performed in conjunction with step 940 of FIG.
[0189] FIG. 15 is a diagram for explaining the operation of the medical imaging apparatus according to an embodiment of the present disclosure.
[0190] The medical imaging apparatus 100 may perform the following process to determine the first candidate arrow information: The following steps may be performed together with one of steps 910 to 940 of FIG.
[0191] The medical imaging device 100 may perform a step of determining first candidate arrow information for a plurality of first candidate arrows indicating a first contour 1521 among the at least one contour. The plurality of first candidate arrows may include a first candidate arrow 1522. The medical imaging device 100 may also perform a step of determining second candidate arrow information for a plurality of second candidate arrows indicating a second contour 1531 among the at least one contour. The plurality of second candidate arrows may include a second candidate arrow 1532.
[0192] The medical imaging device 100 may determine whether a first candidate arrow 1522 of the plurality of first candidate arrows intersects with a second candidate arrow 1532 of the plurality of second candidate arrows. In addition, when the first candidate arrow 1522 and the second candidate arrow 1532 intersect as in the medical image 1510, the medical imaging device 100 may perform a step of moving a start point or an end point of the first candidate arrow 1522 so that the first candidate arrow 1522 of the plurality of first candidate arrows does not intersect with the second candidate arrow 1532 of the plurality of second candidate arrows, and acquiring modified first candidate arrow information.
[0193] Like the medical image 1550, the medical imaging device 100 may move a start point and an end point of the first candidate arrow 1522 corresponding to the first contour 1561 to obtain a modified first candidate arrow 1562. The medical imaging device 100 may obtain modified first candidate arrow information based on the modified first candidate arrow 1562.
[0194] The medical imaging device 100 may perform a step of generating at least one set of arrows based on the modified first candidate arrow information. This step may be performed in conjunction with step 940 of FIG.
[0195] FIG. 16 is a diagram for explaining the operation of the medical imaging apparatus according to an embodiment of the present disclosure.
[0196] The medical imaging apparatus 100 may perform the following process to determine the first candidate arrow information: The following steps may be performed together with one of steps 910 to 940 of FIG.
[0197] Referring to the medical image 1610, the medical imaging device 100 may perform a step of obtaining an intersection 1640 between a first contour 1621 and a second contour 1631 of the at least one contour. The medical imaging device 100 may perform a step of obtaining a first junction 1623 where a first candidate arrow 1622 of the plurality of first candidate arrows and the first contour 1621 meet.
[0198] The medical imaging device 100 may determine whether the first junction 1623 is away from the intersection 1640 by a threshold or more. Here, the threshold may be a predetermined value. If the distance between the first junction 1623 and the intersection 1640 is less than the threshold, the medical imaging device 100 may modify the position of a start point or an end point of a first candidate arrow 1622 of one of the plurality of first candidate arrows such that the first junction 1623 is away from the intersection 1640 by the threshold or more. The medical imaging device 100 may perform a step of modifying the position of the start point or the end point of the first candidate arrow 1622 to obtain modified first candidate arrow information. The medical imaging device 100 may move the start point of the first candidate arrow 1622 along the first contour such that the first junction 1623 is away from the intersection 1640 by the threshold or more. Additionally, the medical imaging device 100 may move the end point of the first candidate arrow 1622 so that the arrow is perpendicular to the contour.
[0199] The medical image 1650 shows a modified first candidate arrow 1662. A second tangent point may be a tangent point between the modified first candidate arrow 1662 and the first contour 1661. Also, an intersection point 1680 may be a point where the first contour 1661 intersects with the second contour 1671. The second tangent point 1663 may be away from the intersection point 1680 by more than a threshold amount.
[0200] The medical imaging device 100 may perform a step of generating at least one set of arrows based on the modified first candidate arrow information. This step may be performed in conjunction with step 940 of FIG.
[0201] 17 and 18 are diagrams for explaining a method of obtaining a score according to one embodiment of the present disclosure.
[0202] As described in Fig. 9, the medical imaging device 100 may perform step 950 of obtaining a score. The medical imaging device 100 may calculate the score according to a predetermined method. The medical imaging device 100 may calculate the score based on the distance between the text boxes.
[0203] As shown in FIG. 17, the medical imaging apparatus 100 may determine a higher score as the distance between at least one of the text boxes 1711, 1721, 1731 corresponding to at least one of the contours 1710, 1720, 1730 increases.
[0204] For example, the medical imaging device 100 can determine the closest distance between the text boxes. The medical imaging device 100 can determine a first distance 1741 between the first text box 1711 and the second text box 1721. The medical imaging device 100 can determine a second distance 1742 between the second text box 1721 and the third text box 1731. The medical imaging device 100 can determine a third distance 1743 between the first text box 1711 and the third text box 1731. The medical imaging device 100 can determine a score based on the first distance 1741, the second distance 1742, and the third distance 1743. The medical imaging device 100 can determine a higher score as the sum of the first distance 1741, the second distance 1742, and the third distance 1743 increases.
[0205] In addition, the medical imaging device 100 may determine a higher score for at least one arrow 1712, 1722, 1732 corresponding to at least one contour 1710, 1720, 1730, as the length of the at least one arrow is shorter. The medical imaging device 100 may determine the length of at least one arrow based on at least one candidate arrow information corresponding to at least one contour, respectively. As described above, the candidate arrow information may include a start point position, an end point position, or an arrow length. The medical imaging device 100 may obtain the length of the arrow from the candidate arrow information.
[0206] More specifically, the medical imaging device 100 can obtain the length of the first arrow 1712, the length of the second arrow 1722, and the length of the third arrow 1732 based on the first candidate arrow information corresponding to the first contour 1710, the second candidate arrow information corresponding to the second contour 1720, and the third arrow information corresponding to the third contour 1730. The medical imaging device 100 can determine a higher score as the sum of the length of the first arrow 1712, the length of the second arrow 1722, and the length of the third arrow 1732 is smaller.
[0207] 18, the medical imaging device 100 may determine a score based on the distance between intersections 1851, 1852 of two contours 1810, 1820 included in at least one contour 1810, 1820 and between contacts 1812, 1822 of at least one contour 1810, 1820 and at least one arrow 1811, 1821. More specifically, the medical imaging device 100 may determine a higher score as the distance between intersections 1851, 1852 of two contours 1810, 1820 included in at least one contour 1810, 1820 and between contacts 1812, 1822 of at least one contour 1810, 1820 and at least one arrow 1811, 1821 increases.
[0208] The medical imaging device 100 can obtain a first arrow 1811 corresponding to a first contour 1810 included in at least one contour based on the arrow information. Also, the medical imaging device 100 can obtain a second arrow 1821 corresponding to a second contour 1820 included in at least one contour based on the arrow information.
[0209] The medical imaging device 100 can obtain a first intersection point 1851 and a second intersection point 1852 of the first contour 1810 and the second contour 1820 .
[0210] Furthermore, the medical imaging apparatus 100 may determine a point where the first contour 1810 and the first arrow 1811 are closest to each other as a first tangent point 1812. The first tangent point 1812 may be a starting point of the first arrow 1811. Furthermore, the first tangent point 1812 may be a point of the first contour 1810 that is closest to the first arrow 1811.
[0211] Furthermore, the medical imaging apparatus 100 may determine a point where the second contour 1820 and the second arrow 1821 are closest to each other as a second tangent point 1822. The second tangent point 1822 may be a starting point of the second arrow 1821. Furthermore, the second tangent point 1822 may be a point of the second contour 1820 that is closest to the second arrow 1821.
[0212] The medical imaging device 100 can determine a score based on a distance 1831 between the first intersection 1851 and the first junction 1812, a distance 1841 between the first intersection 1851 and the second junction 1822, a distance 1832 between the second intersection 1852 and the first junction 1812, and a distance 1842 between the second intersection 1852 and the second junction 1822. The medical imaging device 100 can determine a higher score as the sum of the distance 1831 between the first intersection 1851 and the first junction 1812, the distance 1841 between the first intersection 1851 and the second junction 1822, the distance 1832 between the second intersection 1852 and the first junction 1812, and the distance 1842 between the second intersection 1852 and the second junction 1822 is larger.
[0213] FIG. 19 is a diagram illustrating a medical image according to one embodiment of the present disclosure.
[0214] 2, the medical imaging device 100 may perform step 210 of acquiring lesion information for at least one lesion included in the medical image 1900. The medical imaging device 100 may also perform step 210 of generating a hit map 1920 corresponding to at least one lesion in the medical image based on the acquired lesion information.
[0215] The hit map 1920 can be displayed on the lesion area included in the lesion information, and can be displayed larger than the lesion area included in the lesion information.
[0216] The medical imaging device 100 may generate the hit map 1920 by varying the color and transparency based on the probability that the pixel of the medical image 1900 is included in the lesion area. For example, the pixel may be represented in a reddish color if the pixel has a higher probability of being included in the lesion area, and the pixel may be represented in a bluer color if the pixel has a lower probability of being included in the lesion area. Also, the pixel may be represented in an opaque color if the pixel has a higher probability of being included in the lesion area, and the pixel may be represented in a transparent color if the pixel has a lower probability of being included in the lesion area.
[0217] The medical imaging device 100 can obtain the maximum probability that a pixel on the medical image 1900 is included in the lesion area. The medical imaging device 100 can also determine whether the maximum probability is lower than a threshold value. Here, the threshold value can be a predetermined value. If the probability that a pixel is included in the lesion area is low, the hit map 1920 is highly transparent or is displayed in an inconspicuous color, making it difficult for medical personnel to check. For example, as shown in FIG. 19, if the probability that a pixel is included in the lesion area is low at 27%, it is difficult for medical personnel to check the hit map. The medical imaging device 100 can determine whether the hit map 1920 is easy for medical personnel to see by determining whether the maximum probability is lower than a threshold value.
[0218] The medical imaging device 100 may perform a step of generating a contour 1910 surrounding at least a portion of the hit map 1920 if the maximum value of the probability is lower than a threshold. The contour 1910 may be the same as or smaller than the hit map 1920. However, without being limited thereto, the contour 1910 may be larger than the hit map 1920. The medical imaging device 100 may generate a contour when it is difficult for a medical professional to see the hit map 1920. When it is difficult for a medical professional to see the hit map 1920, the medical imaging device 100 may display only the contour or simultaneously display the contour and the hit map, so that the medical professional can easily check the lesion information displayed on the display.
[0219] The above describes a configuration for obtaining the probability that each pixel on the medical image 1900 is included in a lesion area. However, the present invention is not limited to this. The medical imaging device 100 can obtain the probability that each unit area including at least one pixel is included in a lesion area. The same description can be applied to the case where the medical imaging device 100 uses a unit area, and therefore, a duplicated description will be omitted.
[0220] FIG. 20 is a diagram illustrating a medical image according to one embodiment of the present disclosure.
[0221] 2, the medical imaging device 100 may perform step 210 of acquiring lesion information regarding at least one lesion included in the medical image 2000. The medical imaging device 100 may also perform step 220 of generating at least one contour corresponding to at least one lesion in the medical image based on the acquired lesion information. Also, similar to FIG 9, the medical imaging device 100 may generate arrows and text, although in certain cases the medical imaging device 100 may not generate arrows.
[0222] The medical imaging device 100 may determine whether the area of the lesion or the interior area of a contour is greater than a threshold value, and may not generate an arrow for the contour if the area of the lesion or the interior area of the contour is greater than a threshold value.
[0223] Alternatively, the medical imaging device 100 may determine whether the ratio of the lesion area or the inner area of the contour to the medical image 2000 is greater than a threshold value. If the ratio of the lesion area or the inner area of the contour to the medical image 2000 is greater than a threshold value, the medical imaging device 100 may not generate an arrow for the contour.
[0224] In addition, the medical imaging device 100 may not generate an arrow when the type of lesion included in the lesion information needs to be displayed large on the medical image. Examples of such lesions may include mediastinal widening (MW) or cardiomegaly (Cm).
[0225] If no arrow is generated, the medical imaging apparatus 100 may display text related to the lesion information in a part of the medical image 2000. For example, the medical imaging apparatus 100 may display text related to the lesion information in a bottom edge or corner of the medical image 2000.
[0226] 20 , since the contour 2020 is a relatively small area, an arrow and text can be displayed. However, since the contour 2010 is a relatively large area, an arrow cannot be displayed and text 2011 can be displayed. The medical imaging device 100 can display the text 2011 for the contour 2010 in a partial area or at a partial position of the medical image 2000. For example, the medical imaging device 100 can display the text 2011 for the contour 2010 in a corner or at the bottom edge of the medical image 2000.
[0227] To indicate that there is a relationship between the contour 2010 and the text 2011, the medical imaging device 100 may display the line shape of the contour 2010 in front of the text 2011. In addition, the medical imaging device 100 may set the color of the contour 2010 and the color of the text 2011 to be the same.
[0228] Fig. 21 is a block diagram showing an internal configuration of an electronic device 2100 according to an embodiment of the present disclosure. The electronic device 2100 may include a memory 2110, a processor 2120, a communication module 2130, and an input / output interface 2140. For example, the electronic device 2100 may include the medical imaging device 100 described above. As shown in Fig. 21, the electronic device 2100 may be configured to communicate information and / or data over a network using the communication module 2130.
[0229] The memory 2110 may include any non-transitory computer-readable recording medium. According to an embodiment, the memory 2110 may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), and a flash memory. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, and a disk drive may be included in the electronic device 2100 as a separate permanent storage device separate from the memory. In addition, the memory 2110 may store an operating system and at least one program code (e.g., code for acquiring lesion information, determining the shape and position of a contour, determining the position of a text area, etc., which is installed and operated in the electronic device 2100).
[0230] Such software components may be loaded from a computer-readable recording medium separate from the memory 2110. Such separate computer-readable recording medium may include a recording medium directly connectable to such electronic device 2100, and may include computer-readable recording media such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and a memory card. As another example, the software components may be loaded into the memory 2110 via the communication module 2130, rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 2110 based on a computer program (e.g., a program for determining the shape and position of a contour, determining the position of a text area, generating an arrow, determining the position of an arrow, etc.) installed by a file provided via the communication module 2130 by a developer or a file distribution system that distributes an installation file of an application.
[0231] The processor 2120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to a user terminal (not shown) or other external system via the memory 2110 or the communication module 2130. For example, the processor 2120 may receive a medical image and acquire lesion information regarding at least one lesion detected from the medical image. The processor 2120 may also determine a shape and position of at least one contour corresponding to the at least one lesion based on the acquired lesion information, and determine a position of at least one text area including text indicating lesion information regarding the at least one lesion in the medical image. In this case, the processor 2120 may provide the determined shape and position of the contour, the position of the text area, etc. to the input / output interface 2140, the user terminal (not shown), and / or other external system.
[0232] The communication module 2130 may provide a configuration or function for a user terminal (not shown) and the electronic device 2100 to communicate with each other via a network, and may provide a configuration or function for the electronic device 2100 to communicate with an external system (e.g., a separate cloud system, etc.). For example, control signals, commands, data, etc. provided under the control of the processor 2120 of the electronic device 2100 may be transferred to the user terminal and / or the external system via the communication module of the user terminal and / or the external system via the communication module 2130 and the network. For example, the user terminal and / or the external system may receive the medical image, the shape and position of the contour, the content of the text, the position of the text area, etc. transmitted from the electronic device 2100.
[0233] In addition, the input / output interface 2140 of the electronic device 2100 may be a means for interfacing with a device for input or output (not shown) that may be connected to the electronic device 2100 or may be included in the electronic device 2100. For example, the input / output interface 2140 may include a means for interfacing with a display configured to display a medical image. In FIG. 21, the input / output interface 2140 is illustrated as an element configured separately from the processor 2120, but is not limited thereto, and the input / output interface 2140 may be configured to be included in the processor 2120. The electronic device 2100 may include more components than those illustrated in FIG. 21. However, it is not necessary to explicitly illustrate most of the prior art components.
[0234] The processor 2120 of the electronic device 2100 may be configured to manage, process and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems. According to an embodiment, the processor 2120 may acquire lesion information regarding at least one lesion detected from the medical image, and determine a shape and a position of at least one contour corresponding to the at least one lesion based on the acquired lesion information. The processor 2120 may also determine a position of at least one text area including text indicating lesion information regarding the at least one lesion in the medical image, and display the at least one contour and the text included in the at least one text area on the medical image based on the determined shape and position of the at least one contour and the determined position of the at least one text area.
[0235] 22 is a flow chart illustrating a method 2200 of operating a medical imaging device according to one embodiment of the present disclosure. According to one embodiment, the method 2200 of operating a medical imaging device may be performed by a processor (e.g., at least one processor of a medical imaging device (electronic device)). As shown, the method 2200 of operating a medical imaging device may begin by the processor acquiring lesion information for at least one lesion detected from a medical image (S2210). Additionally or alternatively, the processor may acquire lesion information for multiple lesions detected from a medical image.
[0236] The processor may determine a shape and a position of at least one contour corresponding to at least one lesion based on the acquired lesion information (S2220). For example, the processor may determine a part of the lesions to be displayed in the medical image among the plurality of lesions, and determine a shape and a position of at least one contour for the determined part of the lesions. The processor may also determine a position of at least one text area including text indicating lesion information related to the at least one lesion in the medical image (S2230). For example, the processor may determine a position of the at least one text area based on at least one of a distance between the at least one contour and the at least one text area, or whether or not the at least one contour and the at least one text area overlap. The processor may also determine a position of the at least one text including lesion information related to the determined part of the lesion. The at least one text area may include a plurality of text areas, and in this case, the processor may determine a position of each of the plurality of text areas based on a distance between the plurality of text areas.
[0237] The processor may display the at least one contour and text included in the at least one text region on the medical image based on the shape and position of the determined at least one contour and the determined position of the at least one text region (S2240). The processor may also generate at least one arrow indicating the at least one contour, and display the at least one arrow generated to connect the at least one contour and the at least one text region on the medical image. In this case, the processor may obtain lesion information regarding a plurality of lesions detected from the medical image, and generate an arrow for each of the plurality of lesions. Thereafter, the processor may display the arrow for each of the plurality of lesions on the medical image so that the arrows for each of the plurality of lesions do not cross each other. Additionally or alternatively, the processor may display the arrow for each of the plurality of lesions on the medical image so that the arrows for each of the plurality of lesions do not cross each other with the contours corresponding to each of the plurality of lesions. Additionally or alternatively, the processor may determine at least one contact area tangent to the at least one contour, and display the at least one arrow generated to be connected to the at least one contact area. For example, the at least one contact area may include a plurality of contact areas that are tangent to the at least one contour, and the processor may determine the at least one contact area based on a distance between the plurality of contact areas.
[0238] According to one embodiment, the processor can determine a part of the plurality of lesions to be displayed in the medical image by checking the lesions, etc., among the plurality of lesions, that have overlapping lesion areas, and determining the part of the plurality of lesions based on at least one of the size of the overlapping area between the overlapping lesions, the lesion probability of each of the overlapping lesions, the association between the overlapping lesions, or the probability that a part of the overlapping lesions is present in one medical image. According to such an embodiment, the medical personnel can focus on observing the periphery of the contour, which is a part of the medical image, and finally can easily determine whether a lesion is present inside the contour.
[0239] The above description has focused on various embodiments. It should be understood by those skilled in the art that the present invention can be embodied in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not the above description, and all differences within the scope of the equivalents should be interpreted as being included in the present invention.
[0240] Meanwhile, the above-described embodiments of the present invention may be created as a program executable by a computer and may be embodied in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, flexible disk, hard disk, etc.) and optically readable media (e.g., CD-ROM, DVD, etc.).
Claims
1. 1. A method for operating a computer-assisted medical imaging system, comprising: The computer obtaining a plurality of lesion information related to a plurality of lesions detected from the medical image; extracting overlapping lesions in which partial regions of the plurality of lesions overlap based on the acquired plurality of lesion information; determining at least one of the plurality of lesions to be displayed based on at least one of a size, including at least one of a number of pixels, width, horizontal length, vertical length, or radius, of an overlapping region between the overlapping lesions, a lesion probability, which is a probability that each of the overlapping lesions exists in the medical image, obtained based on a database or a rule-based or machine learning model, or an association, including a similarity or coexistence, between the overlapping lesions; determining the morphology and location of at least one contour corresponding to said at least one displayed lesion; determining a location within the medical image of at least one text area containing text indicating lesion information relating to the at least one displayed lesion; and displaying the at least one contour and text contained in the at least one text area on the medical image based on a shape and a position of the determined at least one contour and a position of the determined at least one text area; The step of determining a position of the at least one text region comprises: If the at least one text region includes only one text region, determining a position of the at least one text region based on at least one of a distance between the at least one contour and the at least one text region or whether or not the at least one contour and the at least one text region overlap; A method for operating a medical imaging device, when the at least one text area includes multiple text areas, determining a position of the at least one text area based on at least one of a distance between the at least one text areas, whether or not there is overlap between the at least one text area, a distance between the at least one contour and the at least one text area, or whether or not there is overlap between the at least one contour and the at least one text area.
2. The step of determining at least one display target lesion comprises: identifying overlapping lesions, which are lesion areas that overlap, among the plurality of lesions based on contours corresponding to each of the plurality of lesions; and determining at least one of the plurality of lesions to be displayed based on at least one of a size of an overlapping area between corresponding contours of the overlapping lesions, a lesion probability that pixels inside corresponding contours of the overlapping lesions are included in the lesion area, an association between lesion information of the overlapping lesions based on a database or a rule-based or machine learning model, or probability information that some of the overlapping lesions exist simultaneously in one medical image.
3. generating at least one arrow indicating the at least one contour; The displaying step includes: The method of claim 1 , further comprising the step of displaying the generated at least one arrow on the medical image so as to connect between the at least one contour and the at least one text region.
4. The step of displaying at least one arrow on the medical image includes:
4. The method for operating a medical imaging device according to claim 3, further comprising the steps of: when the at least one arrow includes a plurality of arrows, moving a start point or an end point of the at least one arrow so that the at least one arrow does not cross each other, thereby obtaining modified arrow information; and displaying an arrow based on the modified arrow information for each of the at least one display target lesion in the medical image.
5. The step of displaying at least one arrow on the medical image includes: The method for operating a medical imaging device according to claim 3, further comprising the steps of: obtaining corrected arrow information by moving a starting point or an end point of the at least one arrow so that the at least one arrow does not cross the at least one contour corresponding to each of the at least one display target lesion; and displaying an arrow based on the corrected arrow information on the medical image.
6. The step of displaying the generated at least one arrow on the medical image includes: determining at least one contact area of an outer periphery of the at least one contour where one end of the at least one arrow meets one side of the at least one text area, and where the at least one text area is located within the medical image; The method of claim 3, further comprising the step of displaying said at least one arrow such that one end of said at least one arrow is included in said at least one contact area.
7. In an electronic device, a memory for storing one or more instructions; Executing the one or more stored instructions, Obtaining multiple lesion information regarding multiple lesions detected from the medical image; extracting overlapping lesions in which partial regions of the plurality of lesions overlap based on the acquired plurality of lesion information; determining at least one of the plurality of lesions to be displayed based on at least one of a size, including at least one of a number of pixels, width, horizontal length, vertical length, or radius, of an overlapping region between the overlapping lesions, a lesion probability, which is a probability that each of the overlapping lesions is present in the medical image, obtained based on a database or a rule-based or machine learning model, or an association, including a similarity or coexistence, between the overlapping lesions; determining the shape and location of at least one contour corresponding to said at least one displayed lesion; determining a location within the medical image of at least one text region containing text indicating lesion information relating to the at least one displayed lesion; a processor configured to display the at least one contour and text contained in the at least one text region on the medical image based on a shape and position of the determined at least one contour and a position of the determined at least one text region; The processor, If the at least one text region includes only one text region, determining a position of the at least one text region based on at least one of a distance between the at least one contour and the at least one text region or whether or not the at least one contour and the at least one text region overlap; When the at least one text area includes multiple text areas, the electronic device is further configured to determine a position of the at least one text area based on at least one of a distance between the at least one text areas, whether or not there is overlap between the at least one text areas, a distance between the at least one contour and the at least one text area, or whether or not there is overlap between the at least one contour and the at least one text area.
8. The processor, identifying overlapping lesions having overlapping lesion areas among the plurality of lesions based on the contours corresponding to each of the plurality of lesions; The electronic device of claim 7, further configured to determine the at least one lesion to be displayed among the plurality of lesions based on at least one of a size of an overlapping area between each corresponding contour of the overlapping lesions, a lesion probability that pixels inside each corresponding contour of the overlapping lesions are included in the lesion area, an association between lesion information of each of the overlapping lesions based on a database or a rule-based or machine learning model, or probability information that some of the overlapping lesions exist simultaneously in one medical image.
9. The processor, generating at least one arrow pointing to said at least one contour; The electronic device of claim 7 , further configured to display the generated at least one arrow on the medical image so as to connect between the at least one contour and the at least one text region.
10. The processor, When the at least one arrow includes a plurality of arrows, moving a start point or an end point of the at least one arrow so that the generated at least one arrow does not cross each other to obtain modified arrow information; The electronic device of claim 9 , further configured to display an arrow based on the modified arrow information for each of the at least one displayed lesion in the medical image.
11. The processor, obtaining modified arrow information by moving a start point or an end point of the at least one arrow such that the at least one arrow does not cross the at least one contour corresponding to each of the at least one displayed lesions; The electronic device of claim 9 , further configured to display an arrow based on the modified arrow information on the medical image.
12. The processor, determining at least one contact area of an outer periphery of the at least one contour where one end of the at least one arrow meets one side of the at least one text area, the at least one text area being located within the medical image; The electronic device of claim 9 , further configured to display one end of the at least one arrow within the at least one contact area.
Citation Information
Patent Citations
Medical report system, medical report viewer, medical report program, and method of viewing medical report
JP2009238037A
Additional information display apparatus and additional information display program
JP2015138428A
Information processing device and method thereof, information processing system, and computer program
JP2017051591A
