Image processing device, medical diagnostic device, ultrasonic endoscope device, image processing method, and program
By detecting the location and the lesion area in the medical image processing device and determining the display method based on the positional relationship and confidence, the problem of insufficient accuracy of diagnosis results in the prior art is solved, and high-precision lesion recognition and display are achieved.
Patent Information
- Application Number
- CN202380030763.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-29
- Filing Date
- 2023-02-14
- Publication Date
- 2024-11-12
AI Technical Summary
Existing medical image processing devices are difficult to identify and display the matching and positional relationship between human parts and lesions with high accuracy, resulting in insufficient accuracy and reliability of diagnostic results.
By detecting the location and lesion area in the medical image in the processor, the display method is determined based on positional relationship, confidence and matching, including overlap, contrast display and hidden site areas, ensuring the accuracy and visibility of the lesion area. .
It improves the identification accuracy of the lesion area and the reliability of diagnostic results, reduces the possibility of doctors' misjudgment, and enhances the visibility and user experience of image processing.
Smart Images

Figure CN118946315A_ABST
Abstract
Description
Technical Field
[0001] The technology of the present invention relates to an image processing device, a medical diagnostic device, an ultrasonic endoscope device, an image processing method, and a program. Background Art
[0002] Japanese Patent Application Publication No. 2021-100555 discloses a medical image processing device having at least one processor. In the medical image processing device described in Japanese Patent Application Publication No. 2021-100555, at least one processor acquires a medical image, acquires part information representing a part of a human body of a subject displayed in the medical image, detects a lesion from the medical image and acquires lesion category information representing the category of the lesion, determines whether there is any contradiction between the part information and the lesion category information, and determines a notification method of the part information and the lesion category information based on the result of the determination. Summary of the invention
[0003] Technical issues to be solved by the invention
[0004] One embodiment of the technology of the present invention provides an image processing device, a medical diagnosis device, an ultrasonic endoscope device, an image processing method, and a program that enable a user or the like to grasp a lesion with high accuracy.
[0005] Means for solving technical problems
[0006] A first method of the technology of the present invention is an image processing device comprising a processor, which detects a first image area representing a part and a second image area representing the lesion from a medical image obtained by photographing an observation object area including a part of a human body and a lesion, and causes a display device to display the results of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
[0007] A second aspect of the technology of the present invention is an image processing device according to the first aspect, wherein the display method is determined according to the part, the lesion, and the positional relationship.
[0008] A third aspect of the technology of the present invention is an image processing device according to the first aspect or the second aspect, wherein the display method is determined based on the positional relationship and the matching between the part and the lesion.
[0009] The fourth mode of the technology of the present invention is an image processing device according to any one of the first mode to the third mode, wherein the display method of the first image area varies depending on the location, lesion, and positional relationship, and the display method of the second image area is a method of displaying the second image area on a display device.
[0010] The fifth mode of the technology of the present invention is an image processing device according to the fourth mode, wherein, when the part and the lesion do not match, the display method of the first image area is a method of not displaying the first image area on the display device, and the display method of the second image area is a method of displaying the second image area on the display device.
[0011] The sixth mode of the technology of the present invention is an image processing device according to the fourth mode or the fifth mode, wherein, when the part and the lesion match, the display method of the first image area is a method of displaying the first image area on a display device and is determined according to the positional relationship, and the display method of the second image area is a method of displaying the second image area on a display device.
[0012] A seventh aspect of the technology of the present invention is an image processing device according to any one of the first to sixth aspects, wherein the positional relationship is defined by an overlap degree or a distance between the first image area and the second image area.
[0013] The eighth mode of the present invention is an image processing device according to the seventh mode, wherein the positional relationship is determined by an overlap degree, and when the overlap degree is greater than a first degree, the display mode is a mode of displaying the second image area in a specific manner within the medical image.
[0014] The ninth method of the technology of the present invention is an image processing device according to the seventh method, wherein the positional relationship is determined by an overlap degree, and when the overlap degree is greater than a first degree, the display method is a method of displaying the second image area in a specific manner within the medical image and displaying the first image area in a manner that can be compared with the second image area.
[0015] The tenth method of the technology of the present invention is an image processing device according to any one of the first to ninth methods, wherein the processor obtains the confidence of the result of detecting the first image area, i.e., the first confidence, and the confidence of the result of detecting the second image area, i.e., the second confidence, and the display method is determined according to the first confidence, the second confidence, and the positional relationship.
[0016] An eleventh aspect of the technology of the present invention is an image processing device according to the tenth aspect, wherein the display method is determined according to the magnitude relationship and positional relationship between the first certainty and the second certainty.
[0017] The twelfth mode of the technology of the present invention is an image processing device according to any one of the first to eleventh modes, wherein a display mode is determined according to a plurality of positional relationships, and the plurality of positional relationships are positional relationships between a plurality of first image areas and a second image area of a plurality of types of parts.
[0018] A thirteenth aspect of the present technology is the image processing device according to the twelfth aspect, wherein the display mode for each of the plurality of first image regions is different for each of the plurality of positional relationships.
[0019] A fourteenth aspect of the present technology is the image processing device according to the twelfth aspect or the thirteenth aspect, wherein the display mode for each of the plurality of first image areas differs according to the positional relationship between the plurality of first image areas.
[0020] The fifteenth embodiment of the technology of the present invention is an image processing device according to any one of the first to fourteenth embodiments, wherein the medical image is an image defined by multiple frames, the processor detects the first image area and the second image area for each frame, and the display method is determined for each frame.
[0021] The sixteenth embodiment of the technology of the present invention is an image processing device according to the fifteenth embodiment, wherein the processor determines whether the combination of the first image area and the second image area is correct for each frame based on the correspondence between multiple types of parts and lesions corresponding to each part, and corrects the display mode corresponding to the frame used as the judgment object when it is judged that the combination of the first image area and the second image area is incorrect, based on the display mode corresponding to the frame used as the judgment object when it is judged that the combination of the first image area and the second image area is correct.
[0022] A seventeenth aspect of the present technology is a medical diagnostic apparatus including the image processing apparatus according to any one of the first to sixteenth aspects, and an imaging device that captures an image of an observation target area.
[0023] An eighteenth aspect of the technology of the present invention is an ultrasonic endoscope device including an image processing device according to any one of the first to sixteenth aspects, and an ultrasonic device for acquiring an ultrasonic image as a medical image.
[0024] The nineteenth embodiment of the technology of the present invention is an image processing method, which includes: detecting a first image area representing a part and a second image area representing a lesion from a medical image obtained by imaging an observation object area including a part of a human body and a lesion; and causing a display device to display the results of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
[0025] The twentieth method of the technology of the present invention is a program for causing a computer to execute the following processing, the processing comprising: detecting a first image area representing a part and a second image area representing a lesion from a medical image obtained by imaging an observation object area including a part of a human body and a lesion, and causing a display device to display the results of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
[0026] The twenty-first method of the technology of the present invention is an image processing device, which has a processor, which detects a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation object area including a part of the human body and a lesion, and determines the accuracy of the second image area based on the positional relationship between the first image area and the second image area.
[0027] The twenty-second method of the technology of the present invention is an image processing device according to the twenty-first method, wherein the processor determines the accuracy based on the positional relationship and the relationship between the confidence of the result of detecting the first image area, i.e., the first confidence, and the confidence of the result of detecting the second image area, i.e., the second confidence.
[0028] The twenty-third method of the technology of the present invention is an image processing device according to the twenty-second method, wherein, when the positional relationship is in a predetermined positional relationship, the first image area and the second image area do not match, and the relationship between the first confidence and the second confidence is in a predetermined confidence relationship, the processor determines that the second image area is accurate.
[0029] The twenty-fourth embodiment of the technology of the present invention is an image processing device according to the twenty-first embodiment, wherein when the positional relationship is a predetermined positional relationship and the first image area and the second image area match, the processor determines that the second image area is correct.
[0030] A twenty-fifth aspect of the technology of the present invention is an image processing device according to any one of the twenty-first to twenty-third aspects, wherein the processor determines the accuracy of the first image area.
[0031] The twenty-sixth embodiment of the technology of the present invention is an image processing device according to any one of the twenty-first to twenty-fifth embodiments, wherein the processor causes the display device to display the medical image, and when it is determined that the second image area is accurate, causes the display to display information indicating that a lesion has been detected.
[0032] A twenty-seventh aspect of the present invention is an image processing device according to the twenty-sixth aspect. In the image processing device of the twenty-seventh aspect, the position where information indicating that a lesion is detected is displayed is an area corresponding to the second image area in the display area where the medical image is displayed. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is a conceptual diagram showing an example of a situation in which an ultrasonic endoscope system is used.
[0034] Figure 2 This is a conceptual diagram showing an example of the overall configuration of an ultrasonic endoscope system.
[0035] Figure 3 This is a conceptual diagram showing an example of a state in which the insertion portion of an ultrasonic endoscope is inserted into the stomach of a subject.
[0036] Figure 4 This is a block diagram showing an example of the hardware configuration of an endoscope processing device.
[0037] Figure 5 This is a block diagram showing an example of the hardware configuration of the ultrasonic processing device.
[0038] Figure 6 This is a block diagram showing an example of the hardware configuration of the display control device.
[0039] Figure 7 This is a block diagram showing an example of main functions of a processor of a display control device.
[0040] Figure 8 This is a conceptual diagram showing an example of the processing content of the acquisition unit.
[0041] Fig. 9 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the detection unit, and the determination unit.
[0042] Fig.10 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, and the control unit.
[0043] Fig.11 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, and the positional relationship identification unit.
[0044] Fig.12 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the detection unit, the positional relationship specifying unit, and the control unit when the overlap degree is smaller than a predetermined overlap degree.
[0045] Fig.13 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the detection unit, the positional relationship specifying unit, and the control unit when the overlap degree is equal to or greater than a predetermined overlap degree.
[0046] Fig.14 This is a flowchart showing an example of the flow of display control processing.
[0047] Fig.15This is a conceptual diagram showing an example of the processing content of the first modification example.
[0048] Fig.16 This is a conceptual diagram showing an example of the processing content of the second modification example.
[0049] Fig.17 This is a conceptual diagram showing an example of the processing content of the third modification example.
[0050] Fig.18 This is a conceptual diagram showing an example of the processing contents of the fourth modification example of the detection unit and the determination unit.
[0051] Fig.19 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is incorrect and the overlap degree is smaller than a predetermined overlap degree.
[0052] Fig. 20 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is incorrect and the overlap degree is equal to or greater than a predetermined overlap degree.
[0053] Fig.21 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is incorrect and the second certainty is lower than the first certainty.
[0054] Fig. 22 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is correct and the overlap degree is less than a predetermined overlap degree.
[0055] Fig.23 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is correct and the overlap degree is equal to or greater than a predetermined overlap degree.
[0056] Fig.24 This is a conceptual diagram showing an example of the processing contents of the acquisition unit, the determination unit, the positional relationship specifying unit, and the control unit when the combination of the part region and the lesion region is correct and the second certainty is less than the first certainty.
[0057] Fig.25A This is a flowchart showing an example of the flow of the display control process according to the fourth modification.
[0058] Fig.25B yes Fig.25A Continuation of the flowchart shown.
[0059] Fig.26 This is a conceptual diagram showing an example of the processing contents of the fifth modification example of the detection unit and the determination unit.
[0060] Fig.27A This is a flowchart showing an example of the flow of the display control process according to the fifth modification.
[0061] Fig.27B yes Fig.27A Continuation of the flowchart shown.
[0062] Fig.28 1 is a conceptual diagram showing an example of an ultrasonic image according to a conventional example and an example of an ultrasonic image after the display control process according to the fifth modification example has been performed.
[0063] Fig.29 1 is a conceptual diagram showing an example of an ultrasonic image according to a conventional example and an example of an ultrasonic image after the display control process according to the sixth modification example has been performed.
[0064] Fig. 30A This is a flowchart showing an example of the flow of the display control process according to the seventh modification example.
[0065] Fig. 30B yes Fig. 30A Continuation of the flowchart shown.
[0066] Fig.31A This is a flowchart showing an example of the flow of the display control process according to the eighth modification example.
[0067] Fig.31B yes Fig.31A Continuation of the flowchart shown.
[0068] Fig.32 This is a conceptual diagram showing an example of the processing content of the ninth modification example of the control unit.
[0069] Fig.33A This is a flowchart showing an example of the flow of the display control process according to the tenth modification.
[0070] Fig.33B yes Fig.33A Continuation of the flowchart shown. DETAILED DESCRIPTION
[0071] Hereinafter, an example of an embodiment of an image processing device, a medical diagnostic device, an ultrasonic endoscope device, an image processing method, and a program according to the technology of the present invention will be described with reference to the drawings.
[0072] First, the words and phrases used in the following description are explained.
[0073] CPU is the abbreviation of "Central Processing Unit". GPU is the abbreviation of "Graphics Processing Unit". RAM is the abbreviation of "Random Access Memory". NVM is the abbreviation of "Non-volatile memory". EEPROM is the abbreviation of "Electrically Erasable Programmable Read-Only Memory". ASIC is the abbreviation of "Application Specific Integrated Circuit". PLD is the abbreviation of "Programmable Logic Device". FPGA is the abbreviation of "Field-Programmable Gate Array". SoC is the abbreviation of "System-on-a-chip". SSD is the abbreviation of "Solid State Drive". USB is the abbreviation of "Universal Serial Bus". HDD is the abbreviation of "Hard Disk Drive". EL is the abbreviation of "Electro-Luminescence". I / F is the abbreviation of "Interface". CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor". CCD is the abbreviation of "Charge Coupled Device". CT is the abbreviation of "Computed Tomography". MRI is the abbreviation of "Magnetic Resonance Imaging". AI is the abbreviation of "Artificial Intelligence". FIFO is the abbreviation of "First In First Out". FPC is the abbreviation of "Flexible Printed Circuit". IoU is the abbreviation of "Intersection over Union".
[0074] As an example, Figure 1As shown, the ultrasonic endoscope system 10 includes an ultrasonic endoscope device 12 and a display device 14. The ultrasonic endoscope device 12 is used by medical personnel such as doctors 16, nurses, and / or technicians (hereinafter referred to as "users"). The ultrasonic endoscope device 12 includes an ultrasonic endoscope 18, and is a device for performing internal diagnosis and treatment of a subject 20 (for example, a patient) via the ultrasonic endoscope 18. The ultrasonic endoscope device 12 is an example of a "medical diagnostic device" and an "ultrasonic endoscope device" of the present invention. The ultrasonic endoscope 18 is an example of an "imaging device" of the present invention.
[0075] The ultrasonic endoscope 18 acquires an image showing the internal condition of the subject 20 by the doctor 16, and outputs the image. Figure 1 In the example shown, an ultrasonic endoscope 18 is inserted into a body cavity from the mouth of a subject 20. Figure 1 In the example shown, the ultrasonic endoscope 18 is inserted into the body cavity from the mouth of the subject 20 , but this is merely an example, and the ultrasonic endoscope 18 may be inserted into the body cavity from the nostril, anus, or perforation of the subject 20 .
[0076] The display device 14 displays various information including images. An example of the display device 14 is a liquid crystal display or an EL display. A plurality of screens are displayed side by side on the display device 14. Figure 1 In the example shown, a first screen 22 and a second screen 24 are displayed as examples of a plurality of screens.
[0077] Different types of images obtained by the ultrasonic endoscope device 12 are displayed in the first screen 22 and the second screen 24. An ultrasonic dynamic image 26 is displayed in the first screen 22. The ultrasonic dynamic image 26 is a dynamic image generated based on a reflected wave, and the reflected wave is obtained by irradiating an ultrasonic wave toward an observation target area in the body of the subject 20, and the ultrasonic wave is reflected in the observation target area. The ultrasonic dynamic image 26 is displayed in the first screen 22 in a real-time display mode. Here, a real-time display mode is illustrated, but this is only an example, and other display modes such as a post view mode may also be used. As an example of an observation target area irradiated with ultrasonic waves, an area including organs and lesions of the subject 20 can be cited.
[0078] Here, the observation target area irradiated with ultrasound is an example of the "observation target area" of the present invention. In addition, the organs and lesions of the subject are examples of the "body parts and lesions" of the present invention. In addition, the ultrasonic dynamic image 26 (i.e., a dynamic image generated based on the reflected waves obtained by the ultrasound being reflected in the observation target area) is an example of the "medical image obtained by photographing the observation target area" of the present invention.
[0079] An endoscope dynamic image 28 is displayed on the second screen 24. As an example of the endoscope dynamic image 28, a dynamic image generated by capturing visible light or near-infrared light can be cited. The endoscope dynamic image 28 is displayed in real time on the second screen 24. In addition, in the present embodiment, the endoscope dynamic image 28 is also illustrated together with the ultrasonic dynamic image 26, but this is only an example, and the technology of the present invention is also applicable even without the endoscope dynamic image 28.
[0080] As an example, Figure 2 As shown, the ultrasonic endoscope 18 includes an operation section 30 and an insertion section 32. The insertion section 32 is formed in a tubular shape. The insertion section 32 includes a front end section 34, a curved section 36, and a soft section 37. The front end section 34, the curved section 36, and the soft section 37 are arranged in the order of the front end section 34, the curved section 36, and the soft section 37 from the front end side to the base end side of the insertion section 32. The soft section 37 is formed of a long strip of flexible material and connects the operation section 30 and the curved section 36. The curved section 36 is partially bent or rotated around the axis of the insertion section 32 by the operation of the operation section 30. As a result, the insertion section 32 is sent into the inner side of the body cavity while being bent or rotated around the axis of the insertion section 32 according to the shape of the body cavity (for example, the shape of the digestive tract such as the esophagus, stomach, duodenum, small intestine, and large intestine, or the shape of the duct of the bronchus).
[0081] The front end portion 34 is provided with an illumination device 38, an endoscope oscilloscope 40, an ultrasonic probe 42, and an opening 44 for a treatment device. The illumination device 38 has an illumination window 38A and an illumination window 38B. The illumination device 38 irradiates light (for example, white light or near-infrared light composed of three primary colors) through the illumination window 38A and the illumination window 38B. The endoscope oscilloscope 40 uses an optical method to photograph the inside of the body. As an example of the endoscope oscilloscope 40, a CMOS camera can be cited. The CMOS camera is only an example, and other types of cameras such as a CCD camera can also be used.
[0082] The ultrasonic probe 42 is provided at the distal end side of the distal end portion 34. The outer surface 42A of the ultrasonic probe 42 is curved outwardly in a convex shape from the base end side toward the distal end side of the ultrasonic probe 42. The ultrasonic probe 42 transmits ultrasonic waves through the outer surface 42A, and receives reflected waves obtained by the transmitted ultrasonic waves being reflected in the observation target area through the outer surface 42A.
[0083] The treatment instrument opening 44 is formed on the proximal side of the front end portion 34 of the ultrasonic probe 42. The treatment instrument opening 44 is an opening for allowing a treatment instrument 46 to protrude from the front end portion 34. A treatment instrument insertion opening 48 is formed in the operation portion 30, and the treatment instrument 46 is inserted into the insertion portion 32 from the treatment instrument insertion opening 48. The treatment instrument 46 passes through the insertion portion 32 and protrudes into the body from the treatment instrument opening 44. Figure 2 In the example shown, as the treatment tool 46, a puncture needle 50 with a guide sheath protrudes from the treatment tool opening 44. The puncture needle 50 with a guide sheath includes a puncture needle 50A and a guide sheath 50B. The puncture needle 50A passes through the guide sheath 50B and protrudes from the guide sheath 50B. Here, as the treatment tool 46, the puncture needle 50 with a guide sheath is illustrated, but this is only an example, and the treatment tool 46 may also be a gripping forceps, a scalpel, and / or a snare. In addition, the treatment tool opening 44 also functions as a suction port for sucking blood and body waste.
[0084] The ultrasonic endoscope device 12 includes a universal cord 52, an endoscope treatment device 54, a light source device 56, an ultrasonic treatment device 58, and a display control device 60. The universal cord 52 includes a base end portion 52A and first to third distal end portions 52B to 52D. The base end portion 52A is connected to the operation unit 30. The first distal end portion 52B is connected to the endoscope treatment device 54. The second distal end portion 52C is connected to the light source device 56. The third distal end portion 52D is connected to the ultrasonic treatment device 58.
[0085] The ultrasonic endoscope system 10 includes a receiving device 62. The receiving device 62 receives instructions from the user. Examples of the receiving device 62 include an operation panel having multiple hard keyboards and / or a touch screen, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.
[0086] The endoscope processing device 54 is connected to a receiving device 62. The endoscope processing device 54 transmits and receives various signals to and from the endoscope scope 40 or controls the light source device 56 according to the instructions received by the receiving device 62. The endoscope processing device 54 causes the endoscope scope 40 to take an image and obtains an endoscope dynamic image 28 (see FIG. 2 ) from the endoscope scope 40. Figure 1 The light source device 56 emits light under the control of the endoscope processing device 54 and supplies light to the lighting device 38. The lighting device 38 has a built-in light guide, and the light supplied from the light source device 56 is irradiated from the lighting windows 38A and 38B via the light guide.
[0087] The ultrasonic processing device 58 is connected to a receiving device 62. The ultrasonic processing device 58 transmits and receives various signals to and from the ultrasonic probe 42 in accordance with instructions received by the receiving device 62. The ultrasonic processing device 58 causes the ultrasonic probe 42 to transmit ultrasonic waves, and generates and outputs an ultrasonic dynamic image 26 based on the reflected waves received by the ultrasonic probe 42.
[0088] The display control device 60 is connected to the display device 14, the endoscope processing device 54, the ultrasonic processing device 58, and the receiving device 62. The display control device 60 controls the display device 14 according to the instruction received by the receiving device 62. The display control device 60 acquires the endoscope dynamic image 28 from the endoscope processing device 54 and causes the display device 14 to display the acquired endoscope dynamic image 28 (see Figure 1 ). In addition, the display control device 60 acquires the ultrasonic dynamic image 26 from the ultrasonic processing device 58, and causes the display device 14 to display the acquired ultrasonic dynamic image 26 (see Figure 1 ).
[0089] As an example, Figure 3 As shown, the insertion portion 32 of the ultrasonic endoscope 18 is inserted into the stomach 64 of the subject 20. The endoscopic oscilloscope 40 captures the inside of the stomach 64 at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second, etc.), thereby generating a real-time display image showing the condition inside the stomach 64 as an endoscopic dynamic image 28. In addition, when the front end portion 34 reaches the target position inside the stomach 64, the outer surface 42A of the ultrasonic probe 42 contacts the inner wall 64A of the stomach 64.
[0090] The ultrasonic probe 42 transmits ultrasonic waves through the outer surface 42A while the outer surface 42A is in contact with the inner wall 64A. The ultrasonic waves are irradiated to the observation target area including organs such as the pancreas 65 and the kidney 66 and the lesion through the inner wall 64A. The reflected waves obtained by the ultrasonic waves being reflected in the observation target area are received by the ultrasonic probe 42 through the outer surface 42A. Then, the reflected waves received by the ultrasonic probe 42 are visualized into a real-time display image showing the status of the observation target area at a predetermined frame rate, thereby obtaining an ultrasonic dynamic image 26.
[0091] In addition, Figure 3 In the example shown, an example of irradiating ultrasound from the stomach 64 toward organs such as the pancreas 65 and the kidney 66 in order to detect a lesion in the pancreas 65 is given, but this is only one example. For example, in order to detect a lesion in the pancreas 65, ultrasound may be irradiated toward organs such as the pancreas 65 and the kidney 66 from the duodenum.
[0092] In addition, Figure 3In the illustrated example, the ultrasonic endoscope 18 is inserted into the stomach 64 , but this is only an example, and the ultrasonic endoscope 18 may be inserted into organs such as the intestines and / or bronchus.
[0093] As an example, Figure 4 As shown, the endoscope processing device 54 includes a computer 67 and an input / output interface 68. The computer 67 includes a processor 70, a RAM 72, and an NVM 74. The input / output interface 68, the processor 70, the RAM 72, and the NVM 74 are connected to a bus 76.
[0094] For example, the processor 70 includes a CPU and a GPU, and controls the entire endoscope processing device 54. The GPU operates under the control of the CPU and is responsible for executing various graphics-related processing. In addition, the processor 70 may be one or more CPUs that integrate the GPU function, or one or more CPUs that do not integrate the GPU function.
[0095] RAM 72 is a memory for temporarily storing information and is used as a working memory by processor 70. NVM 74 is a non-volatile storage device for storing various programs and various parameters. As an example of NVM 74, a flash memory (for example, EEPROM and / or SSD) can be cited. In addition, the flash memory is only an example, and it can be other non-volatile storage devices such as HDD, or a combination of two or more non-volatile storage devices.
[0096] The receiving device 62 is connected to the input / output interface 68, and the processor 70 obtains the instruction received by the receiving device 62 via the input / output interface 68, and executes processing corresponding to the obtained instruction. In addition, the endoscope scope 40 is connected to the input / output interface 68. The processor 70 controls the endoscope scope 40 via the input / output interface 68 or obtains the endoscope dynamic image 28 obtained by photographing the body of the subject 20 through the endoscope scope 40 via the input / output interface 68. In addition, the light source device 56 is connected to the input / output interface 68. The processor 70 supplies light to the lighting device 38 or adjusts the light amount of light supplied to the lighting device 38 by controlling the light source device 56 via the input / output interface 68. In addition, the display control device 60 is connected to the input / output interface 68. The processor 70 transmits and receives various signals with the display control device 60 via the input / output interface 68.
[0097] As an example, Figure 5 As shown, the ultrasonic processing device 58 includes a computer 78 and an input / output interface 80. The computer 78 includes a processor 82, a RAM 84, and an NVM 86. The input / output interface 80, the processor 82, the RAM 84, and the NVM 86 are connected to a bus 88. The processor 82 controls the entire ultrasonic processing device 58. In addition, Figure 5The computer 78 shown includes a plurality of hardware resources (processor 82, RAM 84, and NVM 86) and Figure 4 The plurality of hardware resources included in the computer 67 shown are of the same type, and thus description thereof is omitted here.
[0098] The receiving device 62 is connected to the input / output interface 80. The processor 82 obtains the instruction received by the receiving device 62 via the input / output interface 80 and executes processing corresponding to the obtained instruction. In addition, the display control device 60 is connected to the input / output interface 80. The processor 82 transmits and receives various signals with the display control device 60 via the input / output interface 80.
[0099] The ultrasonic processing device 58 includes a multiplexer 90, a transmission circuit 92, a reception circuit 94, and an analog-to-digital converter 96 (hereinafter referred to as "ADC96"). The multiplexer 90 is connected to the ultrasonic probe 42. The input end of the transmission circuit 92 is connected to the input-output interface 80, and the output end of the transmission circuit 92 is connected to the multiplexer 90. The input end of the ADC96 is connected to the output end of the reception circuit 94, and the output end of the ADC96 is connected to the input-output interface 80. The input end of the reception circuit 94 is connected to the multiplexer 90.
[0100] The ultrasonic probe 42 includes a plurality of ultrasonic transducers 98. The ultrasonic probe 42 is formed by stacking a backing layer, an ultrasonic transducer unit (i.e., a unit in which a plurality of ultrasonic transducers 98 are arranged in a one-dimensional or two-dimensional array), an acoustic matching layer, and an acoustic lens from the inside to the outside of the ultrasonic probe 42.
[0101] The backing layer supports each ultrasonic vibrator 98 included in the ultrasonic vibrator unit from the back side. In addition, the backing layer has the function of attenuating ultrasonic waves propagating from the ultrasonic vibrator 98 to the backing layer side. The backing is made of a rigid material such as hard rubber, and an ultrasonic attenuating material (for example, ferrite and ceramics) is added as needed.
[0102] The acoustic matching layer is laminated on the ultrasonic transducer unit and is provided to achieve acoustic impedance matching between the subject 20 and the ultrasonic transducer 98. The acoustic matching layer can improve the transmittance of ultrasonic waves. The material of the acoustic matching layer can be any organic material as long as the value of the acoustic impedance is closer to that of the subject 20 than that of the piezoelectric element included in the ultrasonic transducer 98. For example, epoxy resin, silicone rubber, polyimide, and / or polyethylene can be cited as the material of the acoustic matching layer.
[0103] The acoustic lens is stacked on the acoustic matching layer and is a lens that focuses the ultrasonic waves emitted from the ultrasonic transducer unit toward the observation target area. The acoustic lens is composed of silicone resin, liquid silicone rubber, butadiene resin, and / or polyurethane resin, etc., and is mixed with powders of titanium oxide, aluminum oxide, or silicon dioxide as needed.
[0104] Each of the plurality of ultrasonic vibrators 98 is formed by placing electrodes on both sides of a piezoelectric element. As an example of a piezoelectric element, barium titanate, lead zirconate titanate, or potassium niobate can be cited. The electrode is composed of individual electrodes individually provided for the plurality of ultrasonic vibrators 98 and a vibrator ground commonly used for the plurality of ultrasonic vibrators 98. The electrode is electrically connected to the ultrasonic processing device 58 via an FPC and a coaxial cable.
[0105] The ultrasonic probe 42 is a convex probe in which a plurality of ultrasonic vibrators 98 are arranged in an arc shape. The plurality of ultrasonic vibrators 98 are arranged along an outer surface 42A (see Figure 2 and Figure 3 ) are arranged. That is, the plurality of ultrasonic vibrators 98 are arranged at equal intervals in a convex curved shape along the axial direction of the front end portion 34 (i.e., the long-side axis direction of the insertion portion 32). Therefore, the ultrasonic probe 42 transmits ultrasonic waves radially by operating the plurality of ultrasonic vibrators 98. In addition, a convex probe is illustrated here, but this is only an example, and a radial, linear, or fan-shaped probe may also be used. In addition, the scanning method of the ultrasonic probe 42 is not particularly limited.
[0106] The transmission circuit 92 and the reception circuit 94 are electrically connected to each of the plurality of ultrasonic vibrators 98 via the multiplexer 90. The multiplexer 90 selects at least one of the plurality of ultrasonic vibrators 98 and opens the channel of the selected ultrasonic vibrator 98, that is, the selected ultrasonic vibrator.
[0107] The transmission circuit 92 is controlled by the processor 82 via the input / output interface 80. The transmission circuit 92 supplies a drive signal (e.g., a plurality of pulse-shaped signals) for ultrasonic transmission to the selected ultrasonic vibrator under the control of the processor 82. The drive signal is generated according to the transmission parameters set by the processor 82. The transmission parameters are the number of drive signals supplied to the selected ultrasonic vibrator, the supply time of the drive signal, the amplitude of the drive vibration, and the like.
[0108] The transmission circuit 92 transmits ultrasonic waves from the selected ultrasonic vibrator by supplying a driving signal to the selected ultrasonic vibrator. That is, when the driving signal is supplied to the electrode included in the selected ultrasonic vibrator, the piezoelectric element included in the selected ultrasonic vibrator expands and contracts, and the selected ultrasonic vibrator vibrates. As a result, a pulsed ultrasonic wave is output from the selected ultrasonic vibrator. The output intensity of the selected ultrasonic vibrator is defined by the amplitude of the ultrasonic wave output from the selected ultrasonic vibrator (that is, the magnitude of the sound pressure of the ultrasonic wave).
[0109] The reflected wave obtained by transmitting ultrasonic waves and reflecting from the observation object area is received by the ultrasonic vibrator 98. The ultrasonic vibrator 98 outputs an electrical signal representing the received reflected wave to the receiving circuit 94 via the multiplexer 90. Specifically, the piezoelectric element included in the ultrasonic vibrator 98 outputs an electronic signal. The size of the electrical signal output from the ultrasonic vibrator 98 (i.e., the voltage value) is a size corresponding to the receiving sensitivity of the ultrasonic vibrator 98. The receiving sensitivity of the ultrasonic vibrator 98 is defined as the ratio of the amplitude of the electrical signal output by the ultrasonic vibrator 98 by receiving ultrasonic waves to the amplitude of the ultrasonic waves transmitted by the ultrasonic vibrator 98. The receiving circuit 94 receives the electrical signal from the ultrasonic vibrator 98, amplifies the received electrical signal and outputs it to the ADC 96. The ADC 96 digitizes the electrical signal input from the receiving circuit 94. The processor 82 generates an ultrasonic dynamic image 26 (refer to FIG. 26 ) by acquiring the electrical signal digitized by the ADC 96 and based on the acquired electrical signal. Figure 1 and Figure 3 ) to obtain ultrasonic dynamic image 26.
[0110] In this embodiment, the combination of the ultrasonic probe 42 and the ultrasonic treatment device 58 is an example of the "imaging device" of the present invention. In this embodiment, the combination of the ultrasonic probe 42 and the ultrasonic treatment device 58 is an example of the "ultrasonic device" of the present invention.
[0111] As an example, Figure 6 As shown, the display control device 60 includes a computer 100 and an input / output interface 102 . The computer 100 includes a processor 104 , a RAM 106 , and an NVM 108 . The input / output interface 102 , the processor 104 , the RAM 106 , and the NVM 108 are connected to a bus 110 .
[0112] Here, the display control device 60 is an example of the “image processing device” of the present invention. In addition, the computer 100 is an example of the “computer” of the present invention. In addition, the processor 104 is an example of the “processor” of the present invention.
[0113] The processor 104 controls the entire display control device 60. Figure 6 The computer 100 shown includes a plurality of hardware resources (processor 104, RAM 106, and NVM 108) and Figure 4 The plurality of hardware resources included in the computer 67 are of the same type, and therefore their description is omitted here.
[0114] The receiving device 62 is connected to the input / output interface 102, and the processor 104 obtains the instruction received by the receiving device 62 via the input / output interface 102, and executes the processing according to the obtained instruction. In addition, the display device 14 is connected to the input / output interface 102. In addition, the endoscope processing device 54 is connected to the input / output interface 102, and the processor 104 transmits and receives various signals with the processor 70 of the endoscope processing device 54 via the input / output interface 102. In addition, the ultrasonic processing device 58 is connected to the input / output interface 102, and the processor 104 transmits and receives various signals with the processor 82 of the ultrasonic processing device 58 via the input / output interface 102.
[0115] The display device 14 is connected to the input / output interface 102, and the processor 104 controls the display device 14 via the input / output interface 102 to display various information on the display device 14. For example, the processor 104 acquires the endoscope dynamic image 28 (see Figure 1 and Figure 3 ), and obtains the ultrasonic dynamic image 26 from the ultrasonic processing device 58 (refer to Figure 1 and Figure 3 ) so that the display device 14 displays the ultrasonic dynamic image 26 and the endoscopic dynamic image 28.
[0116] However, the ultrasonic endoscope 18 irradiates ultrasonic waves into the body of the subject 20, and images the reflected waves obtained by the ultrasonic waves being reflected in the observation target area into an ultrasonic dynamic image 26, thereby suppressing the load on the body of the subject 20 and detecting the lesions contained in the observation target area. However, the ultrasonic dynamic image 26 has lower recognition than the visible light image (for example, the endoscope dynamic image 28), and there is a fear that the lesions may be missed or the diagnosis results may differ due to different doctors 16. Therefore, in recent years, in order to suppress the missed diagnosis of lesions and / or the deviation of the diagnosis results due to different doctors 16, AI-based image recognition processing has been used.
[0117] However, even if a lesion is detected by the AI-based image recognition processing, multiple organs may be captured together with the lesion in the ultrasonic dynamic image 26, making it difficult to identify which of the multiple organs the lesion corresponds to. Furthermore, when multiple organs are captured in an overlapping manner in the ultrasonic dynamic image 26, it becomes more difficult to identify which organ the lesion corresponds to.
[0118] Therefore, in view of such a situation, in this embodiment, as an example, Figure 7As shown, in the display control device 60, the display control process is performed by the processor 104. The display control process program 112 is stored in the NVM 108. The display control process program 112 is an example of a "program" of the present invention. The processor 104 reads the display control process program 112 from the NVM 108, and performs the display control process by executing the read display control process program 112 on the RAM 106. The display control process is realized by the processor 104 operating as the acquisition unit 104A, the detection unit 104B, the determination unit 104C, the positional relationship specifying unit 104D, and the control unit 104E according to the display control process program 112.
[0119] As an example, Figure 8 As shown, the acquisition unit 104A acquires the endoscope dynamic image 28 from the endoscope processing device 54. The endoscope dynamic image 28 is an image defined by a plurality of frames. That is, the endoscope dynamic image 28 is configured to include a plurality of endoscope images 114, and the plurality of endoscope images 114 are obtained by the endoscope processing device 54 as a plurality of frames in a time series at a predetermined frame rate. In addition, the acquisition unit 104A acquires the ultrasonic dynamic image 26 from the ultrasonic processing device 58. The ultrasonic dynamic image 26 is an image defined by a plurality of frames. That is, the ultrasonic dynamic image 26 is configured to include a plurality of ultrasonic images 116, and the plurality of ultrasonic images 116 are obtained by the ultrasonic processing device 58 as a plurality of frames in a time series at a predetermined frame rate.
[0120] As an example, Fig. 9 As shown, the detection unit 104B detects a part region 116A and a lesion region 116B from the ultrasonic image 116 acquired by the acquisition unit 104A. The part region 116A is an image region representing an organ (e.g., pancreas) displayed in the ultrasonic image 116. The lesion region 116B is an image region representing a lesion (e.g., tumor) displayed in the ultrasonic image 116. The part region 116A is an example of the "first image region" of the present invention, and the lesion region 116B is an example of the "second image region" of the present invention.
[0121] The detection unit 104B detects the part region 116A and the lesion region 116B for each frame (i.e., for each of the plurality of ultrasonic images 116 included in the ultrasonic dynamic image 26). Then, the display control process is performed for each frame. In the following, in order to facilitate understanding of the technology of the present invention, the case where the display control process is performed for one frame is described.
[0122] The detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasonic image 116 by performing AI-based image recognition processing on the ultrasonic image 116. Here, the AI-based image recognition processing is illustrated, but this is only an example, and the part region 116A and the lesion region 116B may be detected by template matching-based image recognition processing instead of AI-based image recognition processing. In addition, the detection unit 104B may use both AI-based image recognition processing and template matching-based image recognition processing.
[0123] The detection unit 104B generates part region information 118 and lesion region information 120. The part region information 118 is information about the part region 116A detected by the detection unit 104B. The part region information 118 includes coordinate information 118A and part name information 118B. The coordinate information 118A is information including coordinates (for example, two-dimensional coordinates) that can identify the position of the part region 116A in the ultrasonic image 116 (for example, the position of the contour of the part region 116A). The part name information 118B is information (for example, information indicating the name of the organ itself or an identification mark that can uniquely identify the type of the organ, etc.) that can identify the name of the part represented by the part region 116A detected by the detection unit 104B (that is, the type of the part).
[0124] The lesion region information 120 is information about the lesion region 116B detected by the detection unit 104B. The lesion region information 120 includes coordinate information 120A and lesion name information 120B. The coordinate information 120A is information including coordinates (for example, two-dimensional coordinates) that can identify the position of the lesion region 116B in the ultrasonic image 116 (for example, the position of the outline of the lesion region 116B). The lesion name information 120B is information (for example, information indicating the name of the lesion or an identification mark that can uniquely identify the type of the lesion, etc.) that can identify the name of the lesion represented by the lesion region 116B detected by the detection unit 104B (that is, the type of the lesion).
[0125] The NVM 112 stores a matching determination table 122. In the matching determination table 122, a plurality of part name information 118B and a plurality of lesion name information 118B correspond one to one. That is, the matching determination table 122 specifies the name of the part specified by the part name information 120B and the name of the lesion specified by the lesion name information 120B. In other words, the matching determination table 122 specifies the correct combination of the part and the lesion. Fig. 9 In the example shown, a combination of pancreas and pancreatic cancer, and a combination of kidney and renal cancer are shown as a part of the correct combination of the site and the lesion. The matching determination table is an example of the "correspondence relationship" of the present invention.
[0126] The determination unit 104C acquires the part name information 118B and the lesion name information 120B from the part region information 118. Then, the determination unit 104C determines the matching of the combination of the part name information 118B and the lesion name information 120B by referring to the matching determination table 122 stored in the NVM 112, and determines the matching of the part region 116A and the lesion region 116B (in other words, whether the combination of the part and the lesion is correct or not). That is, the determination unit 104C refers to the matching determination table 122 to determine whether the combination of the name of the part specified by the part name information 118B and the name of the lesion specified by the lesion name information 120B is correct or not. Thus, the correctness of the combination of the part region 116A and the lesion region 116B detected by the detection unit 104B (that is, whether the combination of the part region 116A and the lesion region 116B matches or does not match) is determined by the determination unit 104C.
[0127] As an example, Fig.10 As shown, the control unit 104E acquires the endoscopic image 114 and the ultrasonic image 116 set as the determination object of the determination unit 104C, displays the acquired ultrasonic image 116 on the first screen 22, and displays the acquired endoscopic image 114 on the second screen 24. Here, when the determination unit 104C determines that the part area 116A and the lesion area 116B do not match, the control unit 104E displays the ultrasonic image 116 displayed on the first screen 22 in the first display mode. The first display mode refers to a display mode in which the part area 116A is not displayed and the lesion area 116B is displayed. Fig.10 In the example shown, in the ultrasonic image 116 in the first screen 22 , the part region 116A is not displayed, but the lesion region 116B is displayed.
[0128] As an example, Fig.11As shown, when the determination unit 104C determines that the site area 116A and the lesion area 116B match, the positional relationship specifying unit 104D obtains the positional relationship between the site area 116A and the lesion area 116B. As an example, the positional relationship between the site area 116A and the lesion area 116B is specified by the overlap, which is the degree of overlap between the site area 116A and the lesion area 116B. The positional relationship specifying unit 104D obtains the coordinate information 118A from the site area information 118 and obtains the coordinate information 120A from the lesion area information 120, and calculates the overlap 124 between the site area 116A and the lesion area 116B using the coordinate information 118A and 120A. As an example of an indicator of the overlap 124, IoU can be cited. In this case, for example, the overlap 124 is the ratio of the area of the region where the lesion area 116B and the site area 116A overlap relative to the area of the region where the lesion area 116B and the site area 116A are combined. Fig.11 In the example shown, the state where the entire lesion region 116B overlaps with the site region 116A is represented as "overlapping degree = 1.0", and the state where a part of the lesion region 116B overlaps with the site region 116A is represented as "overlapping degree = 0.4".
[0129] In addition, although IoU is exemplified here, the technology of the present invention is not limited thereto, and the overlap degree 124 may be a ratio of the area of the region where the lesion region 116B and the part region 116A overlap with respect to the lesion region 116B.
[0130] As an example, Fig.12 and Fig.13 As shown, the control unit 104E causes the display device 14 to display the result of the detection of the part region 116A and the lesion region 116B by the detection unit 104B in a display mode corresponding to the positional relationship between the part region 116A and the lesion region 116B (here, as an example, the overlap 124). In the present embodiment, as an example, the display mode of displaying the result of the detection of the part region 116A and the lesion region 116B by the detection unit 104B on the display device 14 is determined according to the part shown in the part region 116A, the type of lesion (hereinafter, also simply referred to as "lesion"), and the positional relationship between the part region 116A and the lesion region 116B (for example, the matching degree and the overlap 124 between the part shown in the part region 116A and the lesion).
[0131] More specifically, for example, the display method for causing the display device 14 to display the result of the detection unit 104B detecting the part region 116A is different depending on the part indicated by the part region 116A, the lesion, and the positional relationship between the part region 116A and the lesion region 116B (for example, the matching degree and overlap degree 124 between the part indicated in the part region 116A and the lesion). In addition, for example, the display method for causing the display device 14 to display the result of the detection unit 104B detecting the lesion region 116B is a method of displaying the lesion region 116B on the first screen 22.
[0132] As an example, Fig.12 As shown, the positional relationship specifying unit 104D determines whether the overlap degree 124 is greater than a predetermined overlap degree (here, "0.5" as an example). The predetermined overlap degree may be a fixed value or a variable value that can be changed according to instructions received by the receiving device 62 and / or various conditions. The predetermined overlap degree is an example of the "first degree" of the technology of the present invention.
[0133] The control unit 104E acquires the endoscopic image 114 and the ultrasonic image 116 set as the determination target of the determination unit 104C, displays the acquired ultrasonic image 116 on the first screen 22, and displays the acquired endoscopic image 114 on the second screen 24. Here, when the positional relationship specifying unit 104D determines that the overlap degree 124 is less than the predetermined overlap degree, the control unit 104E displays the ultrasonic image 116 displayed on the first screen 22 in the first display mode.
[0134] As an example, Fig.13 As shown, the control unit 104E acquires the endoscopic image 114 and the ultrasonic image 116 set as the determination object of the determination unit 104C, and displays the acquired ultrasonic image 116 on the first screen 22, and displays the acquired endoscopic image 114 on the second screen 24. Here, when the position relationship specifying unit 104D determines that the overlap degree 124 is greater than a predetermined overlap degree, the control unit 104E displays the ultrasonic image 116 displayed on the first screen 22 in a second display mode. The second display mode refers to a mode of displaying the lesion area 116B in a specific manner within the ultrasonic image 116, and displaying the site area 116A in a manner that can be compared with the lesion area 116B. Fig.13In the example shown, as an example of the second display method, the contour of the part region 116A and the contour of the lesion region 116B are outlined with a curve, and the contour of the lesion region 116B is outlined with a thicker line than the contour of the part region 116A. In other words, the position of the part region 116A and the position of the lesion region 116B in the ultrasonic image 116 are displayed in a specific manner, and the part region 116A and the lesion region 116B are displayed in a distinguishable state by highlighting the lesion region 116B compared with the part region 116A. Highlighting the lesion region 116B compared with the part region 116A means displaying the lesion region 116B in a manner that is more eye-catching than the part region 116A.
[0135] The control unit 104E controls the display device 14 to display in the second display mode (for example, Fig.13 The control unit 104E displays the ultrasonic image 116 in the display mode shown in the figure, acquires the coordinate information 118A from the part region information 118, and acquires the coordinate information 120A from the lesion region information 120. Then, the control unit 104E processes the ultrasonic image 116 based on the coordinate information 118A and 120A, and displays it on the first screen 22. The processing of the ultrasonic image 116 by the control unit 104E is as follows: the contour of the part region 116A is specified based on the coordinate information 118A and outlined with a curve, the contour of the lesion region 116B is specified based on the coordinate information 120A and outlined with a curve, and the contour of the lesion region 116B is made thicker than the contour of the part region 116A.
[0136] Here, an example of making the outline of the lesion area 116B thicker than the outline of the part area 116A is given for explanation, but this is only an example. For example, the brightness of the outline of the lesion area 116B can also be made higher than the brightness of the outline of the part area 116A. In addition, for example, a pattern or color can be applied to the lesion area 116B, and a pattern or color that is not as eye-catching as that of the lesion area 116B can be applied to the part area 116A. In addition, for example, a pattern or color can be applied only to the lesion area 116B among the part area 116A and the lesion area 116B, and the outline of the part area 116A can be outlined with a curve. The part area 116A can also be made more eye-catching than the lesion area 116B by making the line type of the curve outlining the outline of the part area 116A different from the line type of the curve outlining the outline of the lesion area 116B. In this way, any display method can be used as long as the part area 116A and the lesion area 116B are displayed in a manner that can be specified and compared (that is, can be distinguished).
[0137] Next, refer to Fig.14An example of the flow of the display control process performed by the processor 104 of the display control device 60 when the receiving device 62 receives an instruction to start the execution of the display control process will be described. Fig.14 The process flow represented by the flowchart is an example of the “image processing method” of the technology of the present invention.
[0138] exist Fig.14 In the display control process shown in FIG. 1 , first, in step ST10, the acquisition unit 104A acquires the endoscopic image 114 from the endoscope processing device 54 and acquires the ultrasonic image 116 from the ultrasonic processing device 58 (see FIG. 1 ). Figure 8 ). In step ST10, the endoscopic image 114 acquired by the acquisition unit 104A is one frame of endoscopic image 114 that has not been used in the processing of step ST22 or step ST24 among the multiple endoscopic images 114 included in the endoscopic dynamic image 28. In addition, in step ST10, the ultrasonic image 116 acquired by the acquisition unit 104A is one frame of ultrasonic image 11 that has not been used in the processing after step ST12 among the multiple ultrasonic images 116 included in the ultrasonic dynamic image 26. After the processing of step ST10 is executed, the display control processing is transferred to step ST12.
[0139] In step ST12, the detection unit 104B performs AI-based image recognition processing on the ultrasonic image 116 acquired in step ST10, thereby detecting the part region 116A and the lesion region 116B (see FIG. Fig. 9 ). After the process of step ST12 is executed, the display control process moves to step ST14.
[0140] In step ST14, the detection unit 104B generates part region information 118 which is information related to the part region 116A and lesion region information 120 which is information related to the lesion region 116B (see Fig. 9 ). After the process of step ST14 is executed, the display control process moves to step ST16.
[0141] In step ST16, the determination unit 104C refers to the matching determination table 122 and determines whether the part region 116A and the lesion region 116B match based on the part region information 118 and the lesion region information 120 generated in step ST14 (refer to Fig. 9 ). In step ST16, if the part region 116A and the lesion region 116B do not match, the determination is negative, and the display control process moves to step ST22. In step ST16, if the part region 116A and the lesion region 116B match, the determination is positive, and the display control process moves to step ST18.
[0142] In step ST18, the positional relationship specifying unit 104D acquires the coordinate information 118A from the part region information 118 generated in step ST14, and acquires the coordinate information 120A from the lesion region information 120 generated in step ST14 (see Fig.11 Then, the positional relationship specifying unit 104D calculates the overlap degree 124 using the coordinate information 118A and 120A (see Fig.11 ). After the process of step ST18 is executed, the display control process shifts to step ST20.
[0143] In step ST20, the positional relationship specifying unit 104D determines whether the overlap degree 124 calculated in step ST18 is equal to or greater than a predetermined overlap degree (see Fig.12 and Fig.13 ). In step ST20, when the overlap degree 124 is less than the predetermined overlap degree, the determination is negative, and the display control process moves to step ST22. In step ST20, when the overlap degree 124 is greater than the predetermined overlap degree, the determination is positive, and the display control process moves to step ST24.
[0144] Furthermore, by executing the processing of step ST16 to step ST20, the accuracy of the lesion 116 is determined. If the determination in step ST16 is affirmative (i.e., the part region 116A and the lesion region 116B match), and the determination in step ST20 is affirmative (i.e., the part region 116A and the lesion region 116B are in a known positional relationship), the lesion region 116 is determined to be accurate.
[0145] In step ST22, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasonic image 116 in the first display mode. That is, the control unit 104E does not display the part area 116A in the ultrasonic image 116, and displays the lesion area 116B (see FIG. Fig.10 and Fig.12 ). After executing the process of step ST22, the display control process moves to step ST26.
[0146] In step ST24, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasonic image 116 in the second display mode. That is, the control unit 104E displays the part area 116A and the lesion area 116B in the ultrasonic image 116 in a manner that allows comparison and distinction (see FIG. 1 ). Fig.13 ). After executing the process of step ST24, the display control process moves to step ST26.
[0147] In step ST26, the control unit 104E determines whether a condition for terminating the display control process (hereinafter referred to as "display control process termination condition") is satisfied. As an example of the display control process termination condition, a condition in which an instruction to terminate the display control process is received by the receiving device 62 can be cited. In step ST26, if the display control process termination condition is not satisfied, the determination is negative, and the display control process is transferred to step ST10. In step ST26, if the display control process termination condition is satisfied, the determination is positive, and the display control process is terminated.
[0148] As described above, in the ultrasonic endoscope system 10, the detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasonic image 116. Here, for example, when the part region 116A and the lesion region 116B do not overlap, the lesion shown in the lesion region 116B may be a lesion that is unrelated to the part shown in the part region 116A. In addition, when the part region 116A and the lesion region 116B do not overlap at all, it is highly likely that the lesion shown in the lesion region 116B is a lesion that is unrelated to the part shown in the part region 116A. On the contrary, when the entire lesion region 116B overlaps with the part region 116A, it can be said that the lesion shown in the lesion region 116B is a lesion that is highly related to the part shown in the part region 116A.
[0149] Therefore, in the ultrasonic endoscope system 10, the accuracy of the lesion region 116B is determined based on the positional relationship between the part region 116A and the lesion region 116B (see Fig.14 Then, the determination result is displayed on the second screen 22 (refer to step ST20). Fig.12 and Fig.13 That is, the result of the detection of the part region 116A and the lesion region 116B from the ultrasonic image 116 by the detection unit 104B is displayed on the first screen 22 (see Fig.12 and Fig.13 ). Thus, the user can grasp the lesion with high accuracy. For example, compared with the case where the results of detecting the part area 116A and the lesion area 116B are always displayed in a constant display mode regardless of the positional relationship between the part area 116A and the lesion area 116B, the user can grasp the lesion with high accuracy.
[0150] In the ultrasonic endoscope system 10, the display method of the result of the detection of the site region 116A and the lesion region 116B by the detection unit 104B from the ultrasonic image 116 on the first screen 22 is determined according to the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B. Therefore, the result of the detection of the site region 116A and the lesion region 116B by the detection unit 104B from the ultrasonic image 116 is displayed on the display device 14 in a display method corresponding to the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B (see FIG. 1 ). Fig.10 , Fig.12 and Fig.13 ). Thus, the user can grasp the lesion with high accuracy. For example, compared with the case where the results of detecting the part area 116A and the lesion area 116B are always displayed in a constant display mode regardless of the part, the lesion, and the positional relationship between the part area 116A and the lesion area 116B, the user can grasp the lesion with high accuracy.
[0151] In the ultrasonic endoscope system 10, the display method of the result of the detection of the part region 116A and the lesion region 116B by the detection unit 104B from the ultrasonic image 116 on the first screen 22 is determined according to the positional relationship between the part region 116A and the lesion region 116B and the matching degree between the part and the lesion. Therefore, the result of the detection of the part region 116A and the lesion region 116B by the detection unit 104B from the ultrasonic image 116 is displayed on the display device 14 in a display method corresponding to the positional relationship between the part region 116A and the lesion region 116B and the matching degree between the part and the lesion (see FIG. 1 ). Fig.10 , Fig.12 and Fig.13 ). Thus, the user can grasp the lesion with high accuracy. For example, compared with the case where the results of detecting the part area 116A and the lesion area 116B are always displayed in a constant display mode regardless of the positional relationship between the part area 116A and the lesion area 116B and the matching between the part and the lesion, the user can grasp the lesion with high accuracy.
[0152] In the ultrasonic endoscope system 10, the display method of the part region 116A varies depending on the part, the lesion, and the positional relationship between the part region 116A and the lesion region 116B. The lesion region 116B is displayed on the first screen 22 (see Fig.10 , Fig.12 and Fig.13 ). For example, in Fig.10 and Fig.12 In the example shown, the lesion area 116B is displayed on the first screen 22, but the part area 116A is not displayed on the first screen 22. Fig.13In the example shown, the part region 116A and the lesion region 116B are displayed on the first screen 22. Therefore, the user can easily recognize the difference between the part and the lesion. For example, compared with the case where the display method of the part region 116A is always constant regardless of the part, the lesion, and the positional relationship between the part region 116A and the lesion region 116B, the user can easily recognize the difference between the part and the lesion.
[0153] In addition, in the ultrasonic endoscope system 10, when the site and the lesion do not match, the site area 116A is not displayed on the first screen 22, but the lesion area 116B is displayed on the first screen 22 (see Fig.10 ). Therefore, it is possible to suppress the situation where a part is mistaken for a lesion or a lesion is mistaken for a part. For example, compared with the situation where both the part area 116A and the lesion area 116B are displayed even though the part and the lesion do not match, it is possible to suppress the situation where a part is mistaken for a lesion or a lesion is mistaken for a part.
[0154] In the ultrasonic endoscope system 10, the positional relationship between the site region 116A and the lesion region 116B is defined by the overlap degree 124. Therefore, the result of detecting the site region 116A and the lesion region 116B from the ultrasonic image 116 by the detection unit 104B can be displayed on the first screen 22 in a display mode corresponding to the overlap degree 124 (see FIG. Fig.12 and Fig.13 ).
[0155] In the ultrasonic endoscope system 10, the positional relationship between the part region 116A and the lesion region 116B is defined by the overlap degree 124. When the overlap degree 124 is equal to or greater than a predetermined overlap degree, the lesion region 116B is displayed in a specific manner in the ultrasonic image 116 (see Fig.13 ). Therefore, it is possible for a user or the like to understand a lesion that is highly related to the part displayed on the ultrasonic image 116.
[0156] In addition, in the ultrasonic endoscope system 10, the positional relationship between the part region 116A and the lesion region 116B is defined by the overlap 124. When the overlap 124 is equal to or greater than the predetermined overlap, the lesion region 116B is displayed in a manner that can be identified in the ultrasonic image 116, and the part region 116A is displayed in a manner that can be compared with the lesion region 116B. Therefore, the user or the like can understand the positional relationship between the part and the lesion that is highly related to the part.
[0157] In the ultrasonic endoscope system 10, the ultrasonic dynamic image 26 is a dynamic image defined by a plurality of ultrasonic images 116, the detection unit 104B detects the part region 116A and the lesion region 116B for each ultrasonic image 116, and the display method of the part region 116A and the lesion region 116B is determined for each ultrasonic image 116. Therefore, even if the ultrasonic dynamic image 26 is a dynamic image defined by a plurality of ultrasonic images 116, the user or the like can grasp the lesion part for each ultrasonic image 116.
[0158] In addition, in the ultrasonic endoscope system 10, the accuracy of the lesion 116 is determined by performing the processing of steps ST16 to ST20. For example, when the part region 116A and the lesion region 116B match and the part region 116A and the lesion region 116B are in a known positional relationship (for example, step ST20: Y), it is determined that the lesion region 116 is accurate. Then, the part region 116A and the lesion region 116B in the ultrasonic image 116 are displayed in a manner that allows comparison and distinction (see FIG. 1 ). Fig.13 ). This enables users and the like to understand highly correlated parts and lesions.
[0159] [First Modification]
[0160] In the above-mentioned embodiment, an example of displaying or not displaying the part region 116A is given when the combination of the part region 116A and the lesion region 116B matches (see Figure 11 to Figure 13 ), but the technology of the present invention is not limited thereto. For example, in the case where the combination of the part region 116A and the lesion region 116B matches, the display method of the part region 116A may also be a method in which the part region 116A is displayed in the first screen 22 and is determined according to the positional relationship between the part region 116A and the lesion region 116B.
[0161] In this case, as an example, Fig.15 As shown, when the positional relationship specifying unit 104D determines that the overlap degree 124 is greater than or equal to a predetermined overlap degree, the control unit 104E displays the ultrasonic image 116 in the first screen 22 in the second display mode, and sets the strength of the contour of the part region 116A to a strength corresponding to the overlap degree 124. For example, the contour is made more conspicuous as the overlap degree 124 increases.
[0162] As a method of making the outline more conspicuous, for example, a method of increasing the brightness of the outline or increasing the thickness of the outline can be cited. Fig.15In the example shown, the outline of the part region 116A displayed on the first screen 22 when the overlap degree is "1.0" is thicker than the outline of the part region 116A displayed on the first screen 22 when the overlap degree is "0.6", so the part region 116A displayed on the first screen 22 when the overlap degree is "1.0" is more eye-catching than the part region 116A displayed on the first screen 22 when the overlap degree is "0.6". As a result, the user or the like can recognize the positional relationship between the matched part and the lesion (for example, the overlap degree 124).
[0163] [Second Modification]
[0164] In the above embodiment, the name of the part shown in the part area 116A is not displayed in the first screen 22, but the technology of the present invention is not limited to this. Fig.16 As shown, information indicating the name of the part indicated in the part area 116A may be displayed on the first screen 22 .
[0165] In this case, the control unit 104E acquires the part name information 118B from the part region information 118, and displays the information indicating the name of the part specified by the part name information 118B in the part region 116A of the first screen 22 in an overlapping manner. Fig.16 In the example shown, the text "pancreas" is displayed as information indicating the name of the part in an overlapping manner in the part area 116A. The information indicating the name of the part may be a number, a symbol, or a graphic that can specify the name of the part (i.e., the type of the part) instead of text. Fig.16 As shown, the information indicating the name of the part ( Fig.16 In the example shown, the text "pancreas" is displayed as a pop-up from the part area 116A. In addition, the control unit 104E may switch between displaying and non-displaying the information indicating the name of the part according to the instruction received by the receiving device 62. In this way, by displaying the information indicating the name of the part in a state associated with the part area 116A, the user or the like can understand the name of the part indicated in the part area 116A displayed on the first screen 22.
[0166] In addition, the control unit 104E may also obtain the lesion name information 120B from the lesion region information 120, and display the information indicating the name of the lesion specified by the lesion name information 120B in an overlapping manner on the lesion region 116B of the first screen 22. In addition, the information indicating the name of the lesion may be displayed in a pop-up manner from the lesion region 116B, without being limited to the overlapping display. In addition, the control unit 104E may also switch between displaying and not displaying the information indicating the name of the lesion according to the instruction received by the receiving device 62. In this way, by displaying the information indicating the name of the lesion in a state associated with the lesion region 116B, the user or the like can understand the name of the lesion shown in the lesion region 116B displayed on the first screen 22.
[0167] [Third Modification]
[0168] In the above embodiment, the overlap degree 124 is taken as an example for explanation, but this is only an example. Fig.17 As shown, the distance 126 may be calculated by the positional relationship specifying unit 104D instead of the overlap 124. The distance 126 is also calculated using the coordinate information 118A and 120A in the same manner as the overlap 124. When the overlap 124 is "1.0", that is, when the entire lesion region 116B overlaps with the part region 116A, the distance 126 is 0 mm. When there is a non-overlapping region between the part region 116A and the lesion region 116B, the distance 126 exceeds 0 mm. As an example of the distance 126, the distance between a portion of the contour of the region that does not overlap with the part region 116A in the lesion region 116B and the part region 116A can be cited. For example, the portion of the contour of the region that does not overlap with the part region 116A refers to the position 116B1 that is farthest from the part region 116A in the contour of the region that does not overlap with the part region 116A. In this way, even if the distance 126 is used instead of the overlap 124, the same effect as the above-mentioned embodiment can be obtained.
[0169] [Fourth Modification]
[0170] In the above embodiment, the display mode of the ultrasonic image 116 is described by taking as an example the mode in which the display mode of the ultrasonic image 116 is determined according to the part, the lesion, and the positional relationship between the part region and the lesion region 116B, but the technology of the present invention is not limited thereto. For example, the display mode of the ultrasonic image 116 may be determined according to the confidence of the result of detecting the part region 116A by the image recognition processing of the AI method, the confidence of the result of detecting the lesion region 116B by the image recognition processing of the AI method, and the positional relationship between the part region 116A and the lesion region 116B.
[0171] Here, as an example of the confidence of the result of detecting the part region 116A by the image recognition processing in the AI mode, a value corresponding to the maximum score among the multiple scores obtained according to the learning model can be cited, and the learning model is obtained by making the neural network perform machine learning for detecting the part region 116A. In addition, as an example of the confidence of the result of detecting the lesion region 116B by the image recognition processing in the AI mode, a value corresponding to the maximum score among the multiple scores obtained according to the learning model can be cited, and the learning model is obtained by making the neural network perform machine learning for detecting the lesion region 116B. As an example of the value corresponding to the score, a value obtained by converting the score by the activation function used as the output layer of the neural network (that is, a probability expressed by a value in the range of 0 to 1) can be cited. As an example of the activation function, the Softmax function used as the output layer of the multi-classification can be cited.
[0172] As an example, Fig.18 As shown, the detection unit 104B obtains the first confidence 118C and the second confidence 120C used in the AI-based image recognition processing of the ultrasonic image 116. The first confidence 118C is the confidence of the result of detecting the part area 116A by the AI-based image recognition processing. The second confidence 120C is the confidence of the result of detecting the lesion area 116B by the AI-based image recognition processing. The detection unit 104B generates information including coordinate information 118A, part name information 118B, and the first confidence 118C as part area information 118. In addition, the detection unit 104B generates information including coordinate information 120A, lesion name information 120B, and the second confidence 120C as lesion area information 120. The determination unit 104C determines the matching between the part area 116A and the lesion area 116B in the same way as in the above-mentioned embodiment.
[0173] As an example, Figure 19 to Figure 24 As shown, the display method of the ultrasonic image 116 is determined according to the magnitude relationship between the first confidence level 118C and the second confidence level 1120C, and the positional relationship between the part area 116A and the lesion area 116B.
[0174] As an example, Fig.19 and Fig. 20As shown, when the determination unit 104C determines that the part region 116A and the lesion region 116B do not match, the positional relationship specifying unit 104D determines whether the second confidence 120C included in the lesion region information 120 is greater than the first confidence 118C included in the part region information 118. When the second confidence 120C is greater than the first confidence 118C, the positional relationship specifying unit 104D calculates the overlap 124 in the same manner as in the above embodiment, and determines whether the overlap 124 is greater than a predetermined overlap.
[0175] Here, as an example, Fig.19 As shown, when the positional relationship specifying unit 104D determines that the overlap degree 124 is less than the predetermined overlap degree, the control unit 104E displays the endoscopic image 114 on the second screen 24 and displays the ultrasonic image 116 in the first display mode on the first screen 22. Fig. 20 As shown, when the positional relationship specifying unit 104D determines that the overlap degree 124 is greater than the predetermined overlap degree, the control unit 104E displays the endoscopic image 114 on the second screen 24 and displays the ultrasonic image 116 in the third display mode on the first screen 22. The third display mode refers to a method of highlighting the lesion area 116B. As a method of highlighting the lesion area 116B, for example, a method of increasing the brightness of the outline of the lesion area 116B, a method of giving a color or pattern to the lesion area 116B, or a method of not displaying the area other than the lesion area 116B in the ultrasonic image 116, etc. can be cited. In this way, the highlighting of the lesion area 116B can be achieved by displaying it in a manner that can be distinguished from other areas in the ultrasonic image 116.
[0176] As an example, Fig.21 As shown, when the determination unit 104C determines that the part region 116A and the lesion region 116B do not match, the positional relationship specifying unit 104D determines whether the second certainty 120C included in the lesion region information 120 is greater than the first certainty 118C included in the part region information 118. Here, when the positional relationship specifying unit 104D determines that the second certainty 120C is less than the first certainty, the control unit 104E displays the endoscopic image 114 acquired by the acquisition unit 104A on the second screen 24, and displays the ultrasonic image 116 acquired by the acquisition unit 104A on the first screen 22.
[0177] As an example, Fig. 22 and Fig.23As shown, when the determination unit 104C determines that the part region 116A and the lesion region 116B match, the positional relationship specifying unit 104D determines whether the second confidence 120C included in the lesion region information 120 is greater than the first confidence 118C included in the part region information 118. When the second confidence 120C is greater than the first confidence, the positional relationship specifying unit 104D calculates the overlap 124 in the same manner as in the above embodiment, and determines whether the overlap 124 is greater than a predetermined overlap.
[0178] Here, as an example, Fig. 22 As shown, when the positional relationship specifying unit 104D determines that the overlap degree 124 is less than the predetermined overlap degree, the control unit 104E displays the endoscopic image 114 on the second screen 24 and displays the ultrasonic image 116 in the first display mode on the first screen 22. Fig.23 As shown, when the positional relationship specifying unit 104D determines that the overlap degree 124 is equal to or greater than a predetermined overlap degree, the ultrasonic image 116 is displayed on the first screen 22 in the second display format.
[0179] As an example, Fig.24 As shown, when the determination unit 104C determines that the part region 116A and the lesion region 116B match, the positional relationship specifying unit 104D determines whether the second certainty 120C included in the lesion region information 120 is greater than the first certainty 118C included in the part region information 118. Here, when the detection unit 104B determines that the second certainty 120C is less than the first certainty, the control unit 104E displays the endoscopic image 114 acquired by the acquisition unit 104A on the second screen 24, and displays the ultrasonic image 116 acquired by the acquisition unit 104A on the first screen 22.
[0180] Next, refer to Fig.25A and Fig.25B An example of the flow of the display control process performed by the processor 104 of the display control device 60 when an instruction to start execution of the display control process according to the fourth modification is received by the receiving device 62 will be described.
[0181] Fig.25A and Fig.25B The flowchart shown is similar to Fig.14 Compared with the flowchart shown in FIG. 1 , the difference is that steps ST50 to ST64 are applied instead of steps ST14 and ST16. Fig.14 The same steps in the flowchart shown are marked with the same step numbers and the description is omitted.
[0182] exist Fig.25AIn the display control process shown, in step ST50, the detection unit 104B generates information including coordinate information 118A, part name information 118B, and first confidence level 118C as part region information 118. In addition, the detection unit 104B generates information including coordinate information 120A, lesion name information 120B, and second confidence level 120C as lesion region information 120. After the process of step ST50 is executed, the display control process moves to step ST52.
[0183] In step ST52, the determination unit 104C refers to the matching determination table 122 to determine whether the part region 116A and the lesion region 116B match based on the part region information 118 and the lesion region information 120 generated in step ST50 (see Fig.18 ). In step ST52, if the part region 116A and the lesion region 116B do not match, the determination is negative, and the display control process is transferred to Fig.25B In step ST52, when the part region 116A and the lesion region 116B match, the determination is affirmative, and the display control process shifts to step ST54.
[0184] In step ST54, the positional relationship specifying unit 104D obtains the first confidence 118C from the part region information 118 generated in step ST50, and obtains the second confidence 120C from the lesion region information 120 generated in step ST50. Then, the positional relationship specifying unit 104D determines whether the second confidence 120C is greater than the first confidence 118C. In step ST54, if the second confidence 120C is less than the first confidence 118C, the determination is negative, and the process is transferred to step ST54. Fig.25B In step ST54, when the second certainty 120C is greater than the first certainty 118C, the determination is affirmative, and the display control process shifts to step ST18.
[0185] exist Fig.25B In step ST56 shown, the positional relationship specifying unit 104D obtains the first certainty 118C from the part region information 118 generated in step ST50, and obtains the second certainty 120C from the lesion region information 120 generated in step ST50. Then, the positional relationship specifying unit 104D determines whether the second certainty 120C is greater than the first certainty 118C. In step ST56, if the second certainty 120C is less than the first certainty 118C, the determination is negative, and the process moves to step ST64. In step ST56, if the second certainty 120C is greater than the first certainty 118C, the determination is positive, and the display control process moves to step ST58.
[0186] In step ST58, the positional relationship specifying unit 104D acquires the coordinate information 118A from the part region information 118 generated in step ST50, and acquires the coordinate information 120A from the lesion region information 120 generated in step ST50 (see Fig.19 and Fig. 20 Then, the positional relationship specific information 104D calculates the overlap degree 124 using the coordinate information 118A and 120A (see Fig.19 and Fig. 20 ). After the process of step ST58 is executed, the display control process shifts to step ST60.
[0187] In step ST60, the positional relationship specifying unit 104D determines whether the overlap degree 124 calculated in step ST58 is greater than a predetermined overlap degree. In step ST60, if the overlap degree 124 is less than the predetermined overlap degree, the determination is negative, and the display control process is transferred to Fig.25A In step ST60, when the overlap degree 124 is equal to or greater than the predetermined overlap degree, the determination is affirmative, and the display control process shifts to step ST62.
[0188] In step ST62, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasonic image 116 in the third display mode. That is, the control unit 104E highlights the lesion area 116B in the ultrasonic image 116 (see FIG. Fig. 20 After executing step ST62, the display control process moves to Fig.25A Step ST26 is shown.
[0189] In step ST64, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. After executing the process of step ST64, the display control process shifts to step ST64. Fig.25A Step ST26 is shown.
[0190] In the fourth modification as described above, the result of detecting the part area 116A and the lesion area 116B from the ultrasonic image 116 by the detection unit 104B is displayed on the first screen 22 in a display manner corresponding to the first confidence 118C, the second confidence 120C, and the positional relationship between the part area 116A and the lesion area 116B (for example, the overlap 124). Therefore, it is possible to suppress the occurrence of a situation where the user or the like recognizes a part and a lesion that have a low correlation with each other. For example, compared to a situation where the display manner of the ultrasonic image 116 is determined without considering the first confidence 118C and the second confidence 120C at all, it is possible to suppress the occurrence of a situation where the user or the like recognizes a part and a lesion that have a low correlation with each other.
[0191] In addition, in the fourth modification, the result of detecting the part area 116A and the lesion area 116B from the ultrasonic image 116 by the detection unit 104B is displayed on the first screen 22 in a display manner corresponding to the magnitude relationship between the first confidence 118C and the second confidence 120C, and the positional relationship between the part area 116A and the lesion area 116B (for example, the overlap 124). Therefore, it is possible to suppress the occurrence of a situation where the user or the like recognizes a part and a lesion that have a low correlation with each other. For example, compared to a situation where the display manner of the ultrasonic image 116 is determined without considering the magnitude relationship between the first confidence 118C and the second confidence 120C and the positional relationship between the part area 116A and the lesion area 116B, it is possible to suppress the occurrence of a situation where the user or the like recognizes a part and a lesion that have a low correlation with each other. In addition, the magnitude relationship between the first confidence 118C and the second confidence 120C can be made known to the user or the like by the display manner of the first screen 22. Furthermore, since the object to be compared with the second certainty 120C is the first certainty 118C, there is no need to prepare in advance a threshold value for comparison with the second certainty 120C.
[0192] Furthermore, in this fourth variant, the display mode is determined based on the magnitude relationship between the first confidence 118C and the second confidence 120C, but the display mode is not limited thereto. The display mode may also be determined based on whether the second confidence 120C is above a predetermined confidence (e.g., 0.7). The predetermined confidence may be a fixed value or a variable value that may be changed based on instructions and / or various conditions received by the receiving device 62. If the second confidence 120C is above the predetermined confidence, the lesion area 116B is displayed more prominently than when the second confidence 120C is less than the predetermined confidence. In addition, the display intensity of the lesion area 116B may also be determined based on the magnitude of the second confidence 120C. For example, the larger the second confidence 120C is, the higher the display intensity of the lesion area 116B is.
[0193] In addition, when the display intensity of the lesion region 116B is increased according to the overlap 124, it is possible to identify from the display method (for example, the color given to the outline of the part region 116A and / or the outline of the lesion region 116B, etc.) whether the display intensity is determined according to the size of the second certainty 120C or the display intensity is determined according to the overlap 124. In addition, the display intensity of the part region 116A can also be determined by the same method as that of the first certainty 128C.
[0194] [Fifth Modification]
[0195] In the above embodiment, the display mode of the ultrasonic image 116 is described by taking an example in which the display mode of the ultrasonic image 116 is determined according to the positional relationship between a part area 116A and the lesion area 116B, but the technology of the present invention is not limited to this. For example, the display mode of the ultrasonic image 116 may also be determined according to multiple positional relationships. Here, multiple positional relationships refer to the positional relationships between multiple part areas of multiple types of parts and the lesion area 116B.
[0196] As an example, Fig.26 As shown, the detection unit 104B detects the site regions 116A and 116C and the lesion region 116B from the ultrasonic image 116 by the same method as the above-mentioned embodiment. The site shown in the site region 116A and the site shown in the site region 116C are sites of different types. For example, the site shown in the site region 116A is the pancreas, and the site shown in the site region 116C is the duodenum.
[0197] The detection unit 104B generates the lesion region information 120 by the same method as in the above-mentioned embodiment. In addition, the detection unit 104B generates the site region information 118 for each of the plurality of sites. Fig.26 In the illustrated example, the detection unit 104B generates the part region information 118 related to the part region 116A and the part region information 118 related to the part region 116C.
[0198] The determination unit 104C determines the matching between the part region 116A and the lesion region 116B and the matching between the part region 116C and the lesion region 116B with reference to the matching determination table 122. In the ultrasonic endoscope system 10 of the fifth modified example, display control processing is performed based on the plurality of part region information 118 generated by the detection unit 104B, the lesion region information 120 generated by the detection unit 104B, and the determination result of the determination unit 104C.
[0199] Next, refer to Fig.27A and Fig.27BAn example of the flow of the display control process performed by the processor 104 of the display control device 60 when an instruction to start execution of the display control process according to the fifth modification is received by the receiving device 62 will be described.
[0200] Fig.27A and Fig.27B The flowchart shown is similar to Fig.25A and Fig.25B Compared with the flowchart shown in FIG. 1 , the difference is that step ST80 is applied instead of step ST12, step ST82 is inserted between step ST80 and step ST50, and step ST84 and step ST86 are inserted between step ST22 and step ST26. Fig.25A and Fig.25B The same steps in the flowchart shown are marked with the same step numbers and the description is omitted.
[0201] exist Fig.27A In the display control process shown, in step ST80, the detection unit 104B detects a plurality of part regions (here, as an example, part regions 116A and 116C) from the ultrasonic image 116 by performing an AI-based image recognition process, and detects the lesion region 116B. After executing the process of step ST80, the display control process moves to step ST82.
[0202] In step ST82, the detection unit 104B obtains one part region that is not used in the processing after step ST50 from the plurality of part regions detected in step ST80. After the processing of step ST82 is executed, the display control processing is transferred to step ST50. After step ST50, the processing using the part region detected in step ST82 or Fig.27B The processing of one part area acquired in step ST86 is shown.
[0203] exist Fig.27B In step ST84 shown in FIG. 1 , control unit 104E determines whether the processing after step ST50 has been executed for all the part regions detected in step ST80. In step ST84, if the processing after step ST50 has not been executed for all the part regions detected in step ST80, the determination is negative, and the display control process shifts to step ST86. In step ST84, if the processing after step ST50 has been executed for all the part regions detected in step ST80, the determination is positive, and the display control process shifts to step ST26.
[0204] In step ST86, the detection unit 104B acquires one part region that has not been used in the processing after step ST50 from the plurality of part regions detected in step ST80. After the processing of step ST86 is executed, the display control processing shifts to the display control processing. Fig.27A Step ST50.
[0205] In this way, by conducting Fig.27A and Fig.27B The display control process shown displays the ultrasonic image 116 in a display format determined based on the positional relationship between each of the plurality of part regions and the lesion region 116B.
[0206] For example, when it is determined that the overlap degree 124 between the part region 116A and the lesion region 116B and the overlap degree 124 between the part region 116C and the lesion region 116B are both less than the predetermined overlap degree (step ST20: N), as an example, Fig.28 As shown, the ultrasonic image 116 is displayed in the first display mode. In the conventional example, the part regions 116A and 116C are displayed together with the lesion region 116B, and it is unclear which of the part regions 116A and 116C the lesion region 116B is associated with. In contrast, in the fifth modification, by executing Fig.27A The process of step ST22 shown does not display the part regions 116A and 116C but displays the lesion region 116B, thereby preventing a part region with low correlation with the lesion region 116B from being mistakenly recognized by the user as a part region with correlation with the lesion region 116B.
[0207] In addition, when it is determined that the overlap degree 124 between the part region 116A and the lesion region 116B is greater than the predetermined overlap degree, and when it is determined that the overlap degree 124 between the part region 116C and the lesion region 116B is less than the predetermined overlap degree, as an example, Fig.28 As shown, the part area 116A and the lesion area 116B are displayed in the second display mode, and the part area 116C and the lesion area 116B are displayed in the first display mode. In the conventional example, the part areas 116A and 116C are displayed together with the lesion area 116B, and it is unclear which of the part areas 116A and 116C the lesion area 116B is associated with. In contrast, in the fifth modification example, by executing Fig.27A In the processing of step ST22 shown in FIG. 1 , the lesion region 116B is displayed instead of the part region 116C, thereby preventing the part region 116C having a low correlation with the lesion region 116B from being mistakenly recognized by the user as a part region having a correlation with the lesion region 116B. Fig.27AThe process of step ST24 shown above displays the part region 116A and the lesion region 116B in a manner that allows comparison and distinction, so that the user or the like can recognize that the part region that is highly related to the lesion region 116B is the part region 116A.
[0208] In addition, the display that can be contrasted and distinguished refers to, for example, a display method that highlights the distinguishability of the part area 116A and the lesion area 116B. The highlighting of the distinguishability is achieved, for example, by the color difference and / or brightness difference between the part area 116A and the lesion area 116B. Here, the color difference refers to, for example, a complementary color relationship in the color wheel. In addition, regarding the brightness difference, for example, when the lesion area 116B is expressed in the brightness range of "150 to 255 grayscale", the part area 116A can also be expressed in the brightness range of "0 to 50 grayscale". In addition, regarding the highlighting of the distinguishability, for example, it is also achieved by differentiating the display method of a frame (for example, a circumscribed frame or an outer contour) that surrounds the position of the part area 116A in a specific manner and the display method of a frame (for example, a circumscribed frame or an outer contour) that surrounds the position of the lesion area 116B in a specific manner.
[0209] [Sixth Modification]
[0210] In the fifth modification, when it is determined that the overlap degree 124 between the part region 116C and the lesion region 116B is less than the predetermined overlap degree, the part region 116C is not displayed, but the technology of the present invention is not limited thereto. For example, the display method for each of the plurality of part regions may be different for each of the plurality of positional relationships. Here, the plurality of positional relationships refers to the positional relationship between the plurality of part regions of the plurality of types of parts and the lesion region 116B.
[0211] For example, Fig.29 As shown in FIG. 1 , when the overlap 124 between the site region 116A and the lesion region 116B is greater than the predetermined overlap, the site region 116A and the lesion region 116B are displayed in the second display mode. Furthermore, when the overlap 124 between the site region 116C and the lesion region 116B is less than the predetermined overlap, the site region 116C is displayed with the overlap 124 being “0” as a condition. In addition, even if the overlap 124 between the site region 116C and the lesion region 116B is less than the predetermined overlap, if the overlap 124 is a value greater than “0”, the site region 116C is displayed. Fig.28 Similarly, in the example shown in the lower section, the part area 116C is not displayed.
[0212] Thus, it is possible to prevent the user from mistaking the part region 116C having a low correlation with the lesion region 116B as the part region having a correlation with the lesion region 116B, and to make the user recognize that the part region having a high correlation with the lesion region 116B is the part region 116A. In addition, when the part region 116C is present at a position where it is unlikely to be misidentified by the user as the part region having a correlation with the lesion region 116B, the part region 116C is displayed, so that the user can understand the positional relationship between the part region 116A and the part region 116C, and the positional relationship between the part region 116C and the lesion region 116B.
[0213] In addition, under the condition that the overlap degree 124 between the site area 116C and the lesion area 116B is "0", the display intensity of the site area 116C (for example, the brightness of the outline of the site area 116C and / or the thickness of the outline) may be increased as the distance between the site area 116C and the lesion area 116B becomes longer.
[0214] In addition, the display method of each of the plurality of part regions may be different according to the positional relationship between the plurality of part regions. For example, when the part region 116A overlapping with the lesion region 116B by more than a predetermined overlap degree and the part region 116C overlapping less than a predetermined overlap degree, the part region 116C may not be displayed; when the part region 116A overlapping with the lesion region 116B by more than a predetermined overlap degree and the part region 116C do not overlap, the part region 116C may be displayed. In addition, when the part region 116A overlapping with the lesion region 116B by more than a predetermined overlap degree and the part region 116C do not overlap, the longer the distance between the part region 116A and the part region 116C, the higher the display intensity of the part region 116C.
[0215] This can prevent the user from mistaking the part region 116C, which has a low correlation with the lesion region 116B, for a part region related to the lesion region 116B. In addition, the user can understand the positional relationship between the part region 116A and the part region 116C.
[0216] [Seventh Modification]
[0217] exist Fig.27A and Fig.27B In the display control process shown, an example is given in which the degree of overlap 124 with the lesion region 116B is calculated for each of the plurality of region regions and all the region regions are displayed based on the calculated degree of overlap 124, but the technology of the present invention is not limited thereto. For example, the degree of overlap 124 with the lesion region 116B may be calculated for each of the plurality of region regions and only the region region associated with the maximum degree of overlap among the calculated degrees of overlap 124 may be displayed.
[0218] For example, in this case, Fig. 30A and Fig. 30B The display control process shown is performed by processor 104 . Fig. 30A and Fig. 30B The flowchart shown is similar to Fig.27A and Fig.27B Compared with the flowchart shown in FIG. 1 , the difference is that steps ST100 to ST106 are applied instead of step ST20. Fig.27A and Fig.27B The same steps in the flowchart shown are marked with the same step numbers and the description is omitted.
[0219] exist Fig.27A In the display control process shown, in step ST100, the positional relationship specifying unit 104D associates the overlap degree 124 calculated in step ST18 with the part region information 118 generated in step ST50 and stores them in RAM 106. In the process of this step ST100, the overlap degree 124 and the part region information 118 for each of the plurality of part regions (for example, the part regions 116A and 116C) detected in step ST80 are associated and stored in RAM 106. That is, the plurality of overlap degrees 124 and the plurality of part region information 118 are stored in RAM 106 in a state of being associated one-to-one. After the process of step ST100 is executed, the display control process shifts to step ST102.
[0220] In step ST102, the positional relationship specifying unit 104D determines whether the processing after step ST50 has been executed for all the part regions detected in step ST80. In step ST102, if the processing after step ST50 has not been executed for all the part regions detected in step ST80, the determination is negative, and the display control process is transferred to step ST86. In step ST102, if the processing after step ST50 has been executed for all the part regions detected in step ST80, the determination is positive, and the display control process is transferred to step ST86. Fig.27B Step ST104 is shown.
[0221] exist Fig.27B In step ST104 shown, positional relationship specifying unit 104D acquires the maximum overlap degree 124 and part region information 118 associated with the maximum overlap degree 124 stored in RAM 106 from RAM 106. After executing step ST104, display control processing proceeds to step ST106.
[0222] In step ST106, the positional relationship specifying unit 104D determines whether the maximum overlap degree obtained in step ST104 is greater than or equal to a predetermined overlap degree. In step ST106, if the maximum overlap degree is less than the predetermined overlap degree, the determination is negative, and the display control process is transferred to step ST22. Then, after step ST22, the process using the part region information 118 obtained in step ST104 and the lesion region information 120 generated in step ST50 is executed. On the other hand, in step ST106, if the maximum overlap degree is greater than or equal to the predetermined overlap degree, the determination is positive, and the display control process is transferred to step ST24. Then, after step ST24, the process using the part region information 118 obtained in step ST104 and the lesion region information 120 generated in step ST50 is executed.
[0223] In this way, by conducting Fig. 30A and Fig. 30B The display control process shown displays the ultrasonic image 116 in a display mode corresponding to the positional relationship between the largest part region and the lesion region 116B, the largest part region being the part region that overlaps the lesion region 116B to the greatest extent among the plurality of part regions. Therefore, even if a plurality of part regions are detected, the user or the like can grasp the part regions and the lesion region 116B that are highly correlated with each other.
[0224] [Eighth Modification]
[0225] In the seventh modification, the ultrasonic image 116 is displayed in a display mode corresponding to the positional relationship between the largest part region among the plurality of part regions and the lesion region 116B, but the technology of the present invention is not limited thereto. For example, the ultrasonic image 116 may be displayed in a display mode corresponding to the positional relationship between the part region specified using the part region information 118 and the lesion region information 120, wherein the part region information 118 includes the largest first certainty 118C among the plurality of first certainties 118C obtained from the plurality of part region information 118.
[0226] For example, in this case, Fig.31A and Fig.31B The display control process shown is performed by processor 104 . Fig.31A and Fig.31B The flowchart shown is similar to Fig.27A and Fig.27B Compared with the flowchart shown in FIG. 1 , the difference is that steps ST110 to ST114 are applied instead of step ST82 and step ST50. Fig.27A and Fig.27B The same steps in the flowchart shown are marked with the same step numbers and the description is omitted.
[0227] exist Fig.31A In the display control process shown, in step ST110, detection unit 104B generates a plurality of part region information 118 related to a plurality of part regions (for example, part regions 116A and 116C) detected in step ST80. After executing the process of step ST110, the display control process shifts to step ST112.
[0228] In step ST112, the detection unit 104B compares the plurality of first certainties 118C included in the plurality of part region information 118 generated in step ST110, and identifies the part region information 118 including the largest first certainty 118C from the plurality of part region information 118 generated in step ST110. Then, after step ST52, processing using the part region information 118 identified in step ST112 is performed. After the processing in step ST112 is performed, the display control processing shifts to step ST114.
[0229] In step ST114, the detection unit 104B generates lesion region information 120 related to the lesion region 116B detected in step ST80. After step ST52, processing using the lesion region information 120 generated in step ST114 is performed.
[0230] In addition, after executing Fig.31A After the processing of step ST22 shown in FIG. Fig.31B In step ST26 shown in FIG. Fig.31B In step ST26 shown in FIG. 1 , if the determination is negative, the process proceeds to Fig.31A In step ST10 shown in the figure, if the determination is affirmative in step ST26, the display control process ends.
[0231] In this way, by conducting Fig.31A and Fig.31B The display control processing shown displays the ultrasonic image 116 in a display mode corresponding to the positional relationship between the part region specified using the part region information 118 and the lesion region information 120, the part region information 118 including the largest first certainty 118C among the plurality of first certainties 118C acquired from the plurality of part region information 118. Therefore, even if a plurality of part regions are detected, it is possible for the user or the like to grasp the part region and the lesion region 116B that are highly correlated with each other.
[0232] [Ninth Modification Example]
[0233] In the above embodiment, the example of performing display control processing on the plurality of ultrasonic images 116 included in the ultrasonic dynamic image 26 frame by frame and displaying the ultrasonic images 116 in the display mode determined for each frame is described. In this case, as an example, Fig.32 As shown, if the display control process is performed on the plurality of ultrasonic images 116 in time series, the display mode of the ultrasonic images 116 may be different between the case where the part region 116A and the lesion region 116B match and the case where they do not match. For example, there may be a case where the ultrasonic image 116 is displayed in a first display mode when the part region 116A and the lesion region 116B do not match, and the ultrasonic image 116 is displayed in a second display mode when the part region 116A and the lesion region 116B match.
[0234] Here, when several frames of ultrasonic image 116 adjacent to the ultrasonic image 116 displayed in the first display mode before and after in time series are displayed in the second display mode, the ultrasonic image 116 displayed in the first display mode is likely to be the ultrasonic image 116 originally displayed in the second display mode. That is, although the part region 116A and the lesion region 116B do not match, since the determination unit 104C determines that the part region 116A and the lesion region 116B match, there is a high possibility that the ultrasonic image 116 is displayed in the second display mode.
[0235] Therefore, in the ultrasonic endoscope system 10 of the ninth modified example, the control unit 104E corrects the display mode of the ultrasonic image 116 that is likely to be misjudged by the judgment unit 104C, based on the display mode of the ultrasonic image 116 that is correctly judged by the judgment unit 104C. The display mode of the ultrasonic image 116 that is likely to be misjudged refers to the display mode corresponding to the ultrasonic image 116 used as the judgment target when the judgment unit 104C judges that the combination of the part region 116A and the lesion region 116B is incorrect (that is, does not match). In addition, the display mode of the ultrasonic image 116 that is correctly judged refers to the display mode corresponding to the ultrasonic image 116 set as the judgment target when the judgment unit 104C judges that the combination of the part region 116A and the lesion region 116B is correct (that is, matches) (that is, the display mode determined by the same method as the above-mentioned embodiment).
[0236] exist Fig.32In the example shown, the control unit 104E determines the display mode for each of the plurality of ultrasonic images 116 by the method described in the above embodiment, and stores the plurality of ultrasonic images 116 in a time series in the order in which the display modes were determined. The control unit 104E stores the plurality of ultrasonic images 116 in a FIFO mode. That is, the control unit 104E outputs the oldest frame to the display device 14 each time a new frame is added. Fig.32 In the illustrated example, for convenience of illustration, the ultrasonic images 116 from the first frame to the seventh frame are held in time series by the control unit 104E.
[0237] The first to third frames and the fifth to seventh frames of the ultrasonic image 116 are determined by the determination unit 104C to be a correct combination of the site region 116A and the lesion region 116B and are determined to be displayed in the second display mode. The fourth frame of the ultrasonic image 116 is determined by the determination unit 104C to be an incorrect combination of the site region 116A and the lesion region 116B and is determined to be displayed in the first display mode.
[0238] Here, although the three frames of ultrasonic images 116 before and after the ultrasonic image 116 (i.e., the fourth frame of ultrasonic image 116) determined to have an incorrect combination of the part region 116A and the lesion region 116B in the time series are determined to have a correct combination of the part region 116A and the lesion region 116B, this is merely an example. Four or more frames of ultrasonic images 116 determined to have a correct combination of the part region 116A and the lesion region 116B may be adjacent to the ultrasonic image 116 determined to have an incorrect combination of the part region 116A and the lesion region 116B in the time series.
[0239] In addition, here, there is only one frame of the ultrasonic image 116 in which the combination of the part region 116A and the lesion region 116B is determined to be incorrect, but this is only an example. For example, as long as the number of frames of the ultrasonic image 116 in which the combination of the part region 116A and the lesion region 116B is determined to be correct, the number of frames of the ultrasonic image 116 in which the combination of the part region 116A and the lesion region 116B is determined to be incorrect is sufficiently small. For example, the sufficiently small number of frames refers to a number of frames that is about a fraction to a few hundredths of the number of frames of the ultrasonic image 116 in which the combination of the part region 116A and the lesion region 116B is determined to be correct. The sufficiently small number of frames may be a fixed value or a variable value that changes according to the instructions received by the receiving device 62 and / or various conditions.
[0240] exist Fig.32 In the example shown, the control unit 104E refers to the display modes determined for the first to third frames and the fifth to seventh frames of the ultrasonic images 116 among the plurality of ultrasonic images 116 stored in time series, and corrects the display mode of the fourth frame ultrasonic image 116. Fig.32 In the example shown, since the second display mode is determined for the first to third frames and the fifth to seventh frames of the ultrasonic image 116, the first display mode determined for the fourth frame of the ultrasonic image 116 is corrected to the second display mode. Thus, the display mode of all the ultrasonic images 116 stored in time series can be made consistent with the second display mode. The control unit 104E makes the display mode of the first to seventh frames of the ultrasonic image 116 consistent with the second display mode, and outputs the ultrasonic image 116 to the display device 14 in time series, and displays the ultrasonic image 116 on the first screen 22. Thus, the incorrect combination of the part region 116A and the lesion region 116B can be prevented from being mistaken by the user or the like as the correct combination of the part region 116A and the lesion region 116B.
[0241] [Tenth Modification]
[0242] In this tenth variation, referring to Fig.33A and Fig.33B The display control process described in the eighth modification example (see Fig.31A and Fig.31B ) The display control processing of different algorithms is described. Fig.33A and Fig.33B The flowchart shown is similar to Fig.31A and Fig.31B Compared with the flowchart shown in FIG. 1 , the difference is that the processing from step ST146 to step ST170 is applied instead of the processing from step ST80 to step ST64. Fig.31A and Fig.31B The different processing shown in the flowchart is explained.
[0243] exist Fig.33A In step ST146 shown, the detection unit 104B detects a plurality of site regions (e.g., pancreas and kidney) and a plurality of lesion regions 116B (e.g., pancreatic cancer and kidney cancer) from the ultrasonic image 116 by performing an AI-based image recognition process. After executing the process of step ST146, the display control process moves to step ST148.
[0244] In step ST148, the detection unit 104B generates a plurality of part region information 118 corresponding to the plurality of part regions detected in step ST80. In addition, the detection unit 104B generates a plurality of lesion region information 120 corresponding to the plurality of lesion regions 116B detected in step ST80. After the process of step ST148 is executed, the display control process shifts to step ST149.
[0245] In step ST149, the positional relationship specifying unit 104D selects one lesion region 116B that has not been processed after step ST150, that is, a processing target lesion region, from the plurality of lesion regions 116B detected in step ST146. After executing the process of step ST149, the display control process shifts to step ST150.
[0246] In step ST150, the positional relationship specifying unit 104D acquires coordinate information 118A from each of the plurality of part region information 118 generated in step ST110. In addition, the positional relationship specifying unit 104D acquires coordinate information 120A from the lesion region information 120 corresponding to the lesion region to be processed selected in step ST149 among the plurality of lesion region information 120 generated in step ST148. Then, the positional relationship specifying unit 104D calculates the overlap 124 between each part region detected in step ST80 and the lesion region to be processed. For example, if the plurality of part regions are the pancreas and the kidney, the overlap 124 between the pancreas and the lesion region to be processed and the overlap 124 between the kidney and the lesion region to be processed are calculated. Thus, in this step ST150, a plurality of overlaps 124 are calculated. After the process of step ST150 is executed, the display control process moves to step ST152.
[0247] In step ST152, the positional relationship specifying unit 104D determines whether there is a maximum overlap degree 124 among the plurality of overlap degrees 124 calculated in step ST150. In step ST152, if there is no maximum overlap degree 124 among the plurality of overlap degrees 124, the determination is negative, and the display control process is transferred to step ST153. Fig.33B In step ST152 , when the largest overlap degree 124 exists among the plurality of overlap degrees 124 , the determination is affirmative, and the display control process shifts to step ST156 .
[0248] In addition, here, the presence or absence of the maximum overlap degree 124 is determined, but the technology of the invention is not limited to this, and it may be determined whether the overlap degree 124 greater than a predetermined reference value exists.
[0249] exist Fig.33B In step ST154 shown, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. After executing the process of step ST154, the display control process shifts to step ST170.
[0250] exist Fig.33AIn step ST156 shown, the positional relationship specifying unit 104D acquires the maximum overlap degree 124 which is the largest overlap degree 124 among the plurality of overlap degrees 124 calculated in step ST150 and the part region information 118 associated with the maximum overlap degree. After executing the process of step ST156, the display control process shifts to step ST158.
[0251] In step ST158, the positional relationship specifying unit 104D determines whether the maximum overlap degree obtained in step ST156 is greater than a predetermined overlap degree. In step ST158, if the maximum overlap degree is less than the predetermined overlap degree, the determination is negative, and the display control process is transferred to Fig.33B In step ST106, when the maximum overlap degree is equal to or greater than the predetermined overlap degree, the determination is affirmative, and the display control process shifts to step ST160.
[0252] In step ST160, the determination unit 104C determines Fig.14 The same method as the processing of step ST16 shown in the figure is used to determine whether the processing target part area and the processing target lesion area match. Here, the processing target part area refers to the part area corresponding to the part area information 118 obtained in step ST156 (that is, the part area specified according to the part area information 118 obtained in step ST156). As for the determination of whether the matching between the processing target part area and the processing target lesion area is correct or not, the part area information 118 obtained in step ST156 and the lesion area information 120 corresponding to the processing target lesion area selected in step ST149 are used. The lesion area information 120 corresponding to the processing target lesion area selected in step ST149 refers to the lesion area information 120 corresponding to the processing target lesion area selected in step ST149 among the multiple lesion area information 120 generated in step ST148.
[0253] In step ST160, if the processing target part region matches the processing target lesion region, the determination is negative and the process moves to step ST161. Fig.33B In step ST160, if the treatment target part region and the treatment target lesion region do not match, the determination is affirmative, and the process proceeds to step ST162.
[0254] In step ST162, the positional relationship specifying unit 104D obtains the first confidence 118C from the part region information 118 obtained in step ST156. In addition, the positional relationship specifying unit 104D obtains the lesion region information 120 corresponding to the lesion region of the processing object selected in step ST149 from the multiple lesion region information 120 generated in step ST148, and obtains the second confidence 120C from the obtained lesion region information 120. Then, the positional relationship specifying unit 104D determines whether the second confidence 120C is greater than the first confidence 118C. In step ST162, if the second confidence 120C is less than the first confidence 118C, the determination is denied and the process is transferred to step ST163. Fig.33B In step ST162, if the second confidence level 120C is greater than the first confidence level 118C, the determination is affirmative, and the display control process is transferred to step ST164. Fig.33B Step ST168 shown.
[0255] Here, it is determined whether the magnitude relationship of "first certainty 118C>second certainty 120C" holds, but the technology of the present invention is not limited thereto. For example, it may be determined whether the difference between first certainty 118C and second certainty 120C exceeds a threshold.
[0256] In addition, the conditions for comparing the first confidence 118C and the second confidence 120C may be changed according to the type of the processing target region. For example, a threshold value may be set according to the type of the processing target region, and the first confidence 118C and the second confidence 120C may be compared if the threshold value exceeds the threshold value.
[0257] exist Fig.33B In step ST164 shown, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the processing target site region in the ultrasonic image 116, and does not display the processing target lesion region in the ultrasonic image 116. For example, when the processing target lesion region is a region indicating pancreatic cancer, and the processing target site region is a region indicating kidneys, the region indicating kidneys is displayed, and the region indicating pancreatic cancer is not displayed.
[0258] In addition, in the present specification, the concept of non-display includes not only a method of not displaying at all but also a method of reducing the display intensity (e.g., brightness and / or shade, etc.) to a perceptual level (e.g., a perceptual level known in advance through a sensory test using an actual device and / or a computer simulation) to a degree that will not be misidentified by the doctor 16. After the processing of step ST164 is executed, the display control processing moves to step ST170.
[0259] exist Fig.33B In step ST168 shown, the control unit 104E displays the ultrasonic image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasonic image 116 displayed on the first screen 22 in the first display mode. That is, the control unit 104E displays the treatment target lesion region in the ultrasonic image 116, and does not display the treatment target site region in the ultrasonic image 116. For example, when the treatment target lesion region is a region indicating pancreatic cancer and the treatment target site region is a region indicating kidneys, the region indicating pancreatic cancer is displayed, and the region indicating kidneys is not displayed. After the process of step ST168 is executed, the display control process moves to step ST170.
[0260] In addition, in step ST154, step ST162, and / or step ST168, whether to finally display or not display the processing target area may be determined based on the type of the processing target lesion area and / or the type of the processing target area.
[0261] In addition, it is preferable that a part region representing a specific organ (for example, the kidney 66 when examining the pancreas 65) is not always displayed. However, even when a part region representing a specific organ is not always displayed, the part region representing the specific organ is used in the process related to the overlap degree 124 and the process related to the determination of the magnitude relationship between the first certainty 118C and the second certainty 120C (that is, used in the processes of steps ST149 to ST162).
[0262] In step ST170, the positional relationship specifying unit 104D determines whether all of the plurality of lesion regions 116B detected in step ST146 are used in the processing after step ST150. In step ST170, if all of the plurality of lesion regions 116B detected in step ST146 are not used in the processing after step ST150, the determination is negative, and the display control processing is transferred to step ST150. Fig.33A In step ST170, if all of the plurality of lesion regions 116B detected in step ST146 are used in the processing after step ST150, the determination is affirmative, and the display control processing shifts to step ST26.
[0263] As described above, in the tenth variation, the accuracy of the treatment target lesion region is determined based on the positional relationship between the treatment target lesion region and the treatment target part region, and the relationship between the first certainty 118C and the second certainty 120C. Fig.33AThe processing of steps ST149 to ST162 shown in the figure determines the accuracy of the treatment target lesion region. Then, the determination result is displayed on the first screen 22 (refer to steps ST164 and ST168). Therefore, it is possible to prevent the display of the treatment target part region that is incorrectly combined with the treatment target lesion region, or the display of the treatment target lesion region that is incorrectly combined with the treatment target part region. As a result, it is possible to prevent the user from misidentifying.
[0264] In addition, in the present tenth modified example, when the positional relationship between the treatment target lesion region and the treatment target part region is in a predetermined positional relationship (e.g., step ST152: Y), the treatment target part region and the treatment target lesion region do not match (e.g., step ST160: N), and the relationship between the first certainty 118C and the second certainty 120C is in a predetermined certainty relationship (e.g., step ST162: Y), it is determined that the treatment target lesion region is accurate. Fig.33A As a result of the processing of steps ST156 to ST164 shown in the figure, if the determination is affirmative in step ST162, it is determined that the processing target lesion region is correct. Then, if it is determined that the processing target lesion region is correct, the determination result is displayed on the first screen 22 (step ST168). Therefore, the processing target lesion region determined to be correct is displayed. As a result, the user can grasp the processing target lesion region determined to be correct.
[0265] In addition, in the tenth variation, the accuracy of the processing target area is determined. Fig.33A The accuracy of the processing target region is determined by the processing of steps ST149 to ST162 shown in the figure. Then, the determination result is displayed on the first screen 22 (refer to steps ST164 and ST168). Therefore, it is possible to prevent the display of the processing target region that is incorrectly combined with the processing target lesion region, or the display of the processing target lesion region that is incorrectly combined with the processing target region. As a result, it is possible to prevent the user from misidentifying.
[0266] In addition, when it is determined that the lesion region to be processed is correct, information indicating that the lesion has been detected may be displayed on the display 14. In this case, for example, in the third display mode (see Fig. 20 ) The ultrasonic image 116 is displayed on the first screen 22. Thus, the user or the like can visually recognize the position of the treatment target lesion region on the first screen 22.
[0267] [Other Modifications]
[0268] In the above-mentioned embodiment, the display mode of the part area 116A and the lesion area 116B is determined for each frame, but the technology of the present invention is not limited to this. For example, the display mode of the part area 116A and the lesion area 116B can also be determined for each predetermined number of frames (for example, every several frames or every dozens of frames). In this case, the number of times the display control process is performed is reduced, so the load of the processor 104 can be reduced compared to the case where the display control process is performed for each frame. In addition, in this way, when the display mode of the part area 116A and the lesion area 116B is determined for each predetermined number of frames, the display mode of the part area 116A and the lesion area 116B can also be determined using the frame interval in which the display mode (for example, the first to third display modes, etc.) is visually perceived through the afterimage phenomenon.
[0269] In the above embodiment, an example of displaying the outline of the part area 116A and the outline of the lesion area 116B is given for explanation, but the technology of the present invention is not limited to this. For example, the position of each of the part area 116A and the lesion area 116B can also be displayed in a specific manner by using a bounding box used in the image recognition processing of the AI method. In this case, the control unit 104E can display the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B on the display device 14 in the same display method as the display method applied to the part area 116A and the lesion area 116B. For example, it is also possible to switch between displaying and not displaying the bounding box that specifies the part area 116A, or change the display method of the outline of the bounding box that specifies the part area 116A under the same conditions as the above embodiment, or change the display method of the outline of the bounding box that specifies the lesion area 116B under the same conditions as the above embodiment.
[0270] In addition, the positional relationship specifying unit 104D may calculate the overlap 124 and / or the distance 126 using the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B. As an example of the overlap 124 in this case, the IoU using the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B may be cited. The IoU in this case refers to the ratio of the area of overlap between the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B to the area of the area that combines the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B. In addition, the overlap 124 may also be the ratio of the area of overlap between the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B to the total area of the bounding box that specifies the lesion area 116B. In this case, an example of the distance 126 is the distance between a portion of the outline of the region that does not overlap with the bounding box that specifies the part region 116A in the bounding box that specifies the lesion region 116B and the bounding box that specifies the part region 116A. For example, the portion of the outline of the region that does not overlap with the part region 116A refers to a position in the outline of the region that does not overlap with the part region 116A that is farthest from the bounding box that specifies the part region 116A.
[0271] In the above embodiment, as the first display mode, an example of not displaying the site areas 116A and 116C is given, but this is only an example, and the display intensity of the site areas 116A and / or 116C may be lower than the display intensity of the lesion area 116B, and the site areas 116A and / or 116C may not be displayed.
[0272] In the above embodiment, an example of displaying the ultrasonic image 116 in any of the first to third display modes when the second confidence level 120C is greater than the first confidence level 118C is given, but the technology of the present invention is not limited thereto. For example, when the first confidence level 118C is greater than the second confidence level 120C, the part region 116A or 116C and the lesion region 116B may be displayed in a manner that allows comparison, provided that the overlap level 124 is less than a predetermined overlap level or that the overlap level 124 is "0". In addition, the part region 116A or 116C and the lesion region 116B may be displayed in the second display mode.
[0273] In the above-mentioned embodiment, an example is given of setting the strength of the outline of the part area 116A to an intensity corresponding to the overlap degree 124, but the technology of the present invention is not limited to this. When the overlap degree 124 is above a predetermined overlap degree, the greater the overlap degree 124, the higher the display intensity of both the part area 116A and the lesion area 116B.
[0274] In the above embodiment, the ultrasonic image 116 is displayed on the first screen 22 , but this is only an example. For example, the ultrasonic image 116 may be displayed on the entire screen of the display device 14 .
[0275] In the above-mentioned embodiment, the processor 104 causes the display device 14 to display the ultrasonic image 116 by directly operating the display device 14, but this is only an example. For example, the processor 104 may also cause the display device 14 to display the ultrasonic image 116 by indirectly operating the display device 14. For example, in this case, screen information representing the screen to be displayed on the display device 14 (for example, at least the first screen 22 of the first screen 22 and the second screen 24) is temporarily stored in an external memory (not shown). Then, the processor 104 or a processor other than the processor 104 acquires the screen information from the external memory, and based on the acquired screen information, causes the display device 14 or a display device other than the display device 14 to display at least the first screen 22 of the first screen 22 and the second screen 24. As a specific example of this case, there can be cited an example in which the processor 104 uses cloud computing to cause the display device 14 or a display device other than the display device 14 to display at least the first screen 22 of the first screen 22 and the second screen 24.
[0276] In the above-described embodiment, an example in which the display control process is executed on the ultrasonic moving image 26 is described. However, the display control process may be executed on an ultrasonic still image.
[0277] In the above-mentioned embodiment, an example of performing display control processing on the ultrasonic image 116 acquired by the ultrasonic endoscope device 12 is given for explanation, but the technology of the present invention is not limited to this. For example, display control processing may also be performed on an ultrasonic image acquired by an extracorporeal ultrasonic diagnostic device using an extracorporeal ultrasonic probe. Display control processing may also be performed on a medical image obtained by photographing the observation object area of the subject 20 by various modalities such as an X-ray diagnostic device, a CT diagnostic device, and / or an MRI diagnostic device. In addition, the extracorporeal ultrasonic diagnostic device, the X-ray diagnostic device, the CT diagnostic device, and / or the MRI diagnostic device are examples of "imaging devices" of the technology of the present invention.
[0278] In the above-mentioned embodiment, an example of a method of performing display control processing by the processor 104 of the display control device 60 is given for explanation, but the technology of the present invention is not limited to this. For example, the device for performing display control processing may also be arranged outside the display control device 60. As an example of a device arranged outside the display control device 60, an endoscope processing device 54 and / or an ultrasound processing device 58 can be cited. In addition, as another example of a device arranged outside the display control device 60, a server can be cited. For example, the server is implemented by cloud computing. Here, cloud computing is illustrated, but this is only an example. For example, the server can be implemented by a main frame, or by network computing such as fog computing, edge computing, or grid computing. The server is only an example, and instead of a server, at least one personal computer or the like may be used. In addition, the display control processing may also be performed in a distributed manner by a plurality of devices including the display control device 60 and at least one device arranged outside the display control device 60.
[0279] In addition, in the above embodiment, the display control processing program 112 is stored in the NVM 108, but the technology of the present invention is not limited to this. For example, the display control processing program 112 can also be stored in a portable storage medium such as an SSD or a USB memory. The storage medium is a non-temporary, computer-readable storage medium. The display control processing program 112 stored in the storage medium is installed in the computer 100 of the display control device 60. The processor 104 performs display control processing according to the display control processing program 112.
[0280] In the above embodiment, the computer 100 is illustrated, but the technology of the present invention is not limited thereto, and a device including ASIC, FPGA, and / or PLD may be applied instead of the computer 100. In addition, a combination of hardware structure and software structure may be used instead of the computer 100.
[0281] As hardware resources for executing the display control processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a general-purpose processor that functions as a hardware resource for executing the display control processing by executing software, that is, a program, can be cited. In addition, as a processor, for example, a processor, that is, a dedicated electronic circuit, such as an FPGA, a PLD, or an ASIC, which has a circuit structure specially designed for executing a specific process can be cited. A memory is built into or connected to any processor, and any processor executes the display control processing by using the memory.
[0282] The hardware resource for executing the display control processing may be composed of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a processor and an FPGA). In addition, the hardware resource for executing the display control processing may also be a processor.
[0283] As an example of a system composed of one processor, there is a first method in which a processor is composed of a combination of one or more processors and software, and the processor functions as a hardware resource for executing display control processing. There is a second method in which a processor is used that implements the functions of the entire system including multiple hardware resources for executing display control processing using one integrated circuit (IC) chip, as represented by SoC, etc. In this way, the display control processing is implemented using one or more of the above-mentioned processors as hardware resources.
[0284] As the hardware structure of these various processors, more specifically, an electronic circuit composed of a combination of circuit elements such as semiconductor elements can also be used. In addition, the above-mentioned display control processing is only an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps can be added, or the processing order can be changed within the scope of the main purpose.
[0285] The above-mentioned contents and illustrated contents are detailed descriptions of the part of the technology of the present invention, which are only an example of the technology of the present invention. For example, the description of the above-mentioned structure, function, action, and effect is a description of an example of the structure, function, action, and effect of the part of the technology of the present invention. Therefore, it goes without saying that, it is also possible to delete unnecessary parts, or add new elements, or replace with new elements for the above-mentioned contents and illustrated contents without departing from the scope of the main purpose of the technology of the present invention. In addition, in order to avoid complexity and make it easy to understand the part of the technology of the present invention, in the above-mentioned contents and illustrated contents, the description of technical common sense, etc., which does not need to be particularly explained when implementing the technology of the present invention, is omitted.
[0286] In this specification, "A and / or B" is synonymous with "at least one of A and B". That is, it means that "A and / or B" can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are expressed by connecting "and / or", the same concept as "A and / or B" is also applied.
[0287] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually described as being incorporated by reference.
[0288] The following additional notes are disclosed in relation to the above-mentioned embodiments.
[0289] (Note 1)
[0290] An image processing device, wherein:
[0291] With processor,
[0292] The above processors
[0293] A first image region representing a part of a human body and a lesion are detected from a medical image obtained by photographing an observation object region including the part of the human body and the lesion, and a second image region representing the lesion is detected.
[0294] The display device is caused to display the result of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
[0295] (Note 2)
[0296] An image processing device according to Supplement 1, wherein:
[0297] The processor obtains a first confidence level for a result of detecting the first image region and a second confidence level for a result of detecting the second image region,
[0298] The display method is determined according to the first confidence level, the second confidence level, and the positional relationship.
[0299] When the first confidence level is greater than the second confidence level and the first image area and the second image area do not overlap, the display method is a method of displaying the first image area and the second image area in a comparative manner within the medical image.
[0300] (Note 3)
[0301] An image processing device according to Supplement 1, wherein:
[0302] The processor determines whether the combination of the first image region and the second image region is correct or not based on the correspondence between the plurality of types of the parts and the lesions corresponding to each of the parts.
[0303] When the processor determines that the combination of the first image area and the second image area is correct, the first confidence level is greater than the second confidence level, and the first image area and the second image area do not overlap, the display method is a method of displaying the first image area and the second image area in a comparable manner within the medical image.
[0304] (Note 4)
[0305] An image processing device according to Supplement 2 or 3, wherein:
[0306] The display intensity of the first image area is determined according to the distance between the first image area and the second image area.
[0307] (Note 5)
[0308] An image processing device according to Supplement 1, wherein:
[0309] The observation target area includes a plurality of types of the above-mentioned parts and the above-mentioned lesions.
[0310] The above processors
[0311] Detecting a plurality of the first image regions and the second image regions representing a plurality of types of the above-mentioned parts from the above-mentioned medical image,
[0312] When the second image area overlaps with multiple first image areas in the medical image, the positional relationship is the relationship between the position of the maximum overlapping image area among the multiple first image areas that overlaps the second image area to the greatest extent and the position of the second image area.
[0313] (Note 6)
[0314] An image processing device according to Supplement 1, wherein:
[0315] The processor obtains a first confidence level for a result of detecting the first image region and a second confidence level for a result of detecting the second image region,
[0316] The display method is determined according to the first confidence level, the second confidence level, and the positional relationship.
[0317] The observation target area includes a plurality of types of the above-mentioned parts and the above-mentioned lesions.
[0318] The above processors
[0319] Detecting a plurality of the first image regions and the second image regions representing a plurality of types of the above-mentioned parts from the above-mentioned medical image,
[0320] In the case where the second image region overlaps with a plurality of the first image regions in the medical image, the first confidence level is the maximum confidence level among a plurality of confidence levels for a plurality of results of detecting the plurality of the first image regions.
[0321] (Note 7)
[0322] An image processing device according to Supplement 1, wherein:
[0323] The above processors
[0324] Obtaining a confidence level, that is, a first confidence level, for a result of detecting the first image area.
[0325] The above display method is determined according to the above first confidence level and the above positional relationship.
[0326] (Note 8)
[0327] An image processing device according to Supplement 1, wherein:
[0328] The processor obtains a second confidence level as a result of detecting the second image area.
[0329] The above display method is determined according to the above second certainty and the above positional relationship.
Claims
1. An image processing device comprising a processor, the processor, A first image region representing a part of a human body and a lesion is detected from a medical image obtained by imaging an observation object region including the part of the human body and the lesion, and a second image region representing the lesion is detected. The display device is caused to display the result of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
2. The image processing device according to claim 1, wherein: The display method is determined according to the part, the lesion, and the positional relationship.
3. The image processing device according to claim 1 or 2, wherein: The display method is determined according to the positional relationship and the matching between the part and the lesion.
4. The image processing device according to any one of claims 1 to 3, wherein: The display method of the first image area varies according to the part, the lesion, and the positional relationship. The display method regarding the second image area is a method of displaying the second image area on the display device.
5. The image processing device according to claim 4, wherein: In case the site and the lesion do not match, The display mode of the first image area is a mode in which the first image area is not displayed on the display device, The display method regarding the second image area is a method of displaying the second image area on the display device.
6. The image processing device according to claim 4 or 5, wherein: In case the site and the lesion match, The display method of the first image area is a method of displaying the first image area on the display device and is determined according to the positional relationship. The display method regarding the second image area is a method of displaying the second image area on the display device.
7. The image processing device according to any one of claims 1 to 6, wherein: The positional relationship is defined by an overlap degree or a distance between the first image region and the second image region.
8. The image processing device according to claim 7, wherein: The positional relationship is determined by the overlap degree, and when the overlap degree is greater than a first degree, the display method is a method of displaying the second image area in a specific manner within the medical image.
9. The image processing apparatus according to claim 7, wherein: The positional relationship is determined by the overlap degree, and when the overlap degree is above a first degree, the display method is a method of displaying the second image area in the medical image in a specific manner and displaying the first image area in a manner that can be compared with the second image area.
10. The image processing device according to any one of claims 1 to 9, wherein: The processor obtains a first confidence level for a result of detecting the first image area and a second confidence level for a result of detecting the second image area, The display method is determined according to the first confidence level, the second confidence level, and the positional relationship.
11. The image processing device according to claim 10, wherein: The display method is determined according to the magnitude relationship and the position relationship between the first confidence level and the second confidence level.
12. The image processing device according to any one of claims 1 to 11, wherein: The display mode is determined according to the plurality of positional relationships. The plurality of positional relationships are positional relationships between a plurality of the first image regions and the second image regions for a plurality of types of the parts.
13. The image processing apparatus according to claim 12, wherein: The display manner regarding each of the plurality of first image areas is different according to each of the plurality of positional relationships.
14. The image processing device according to claim 12 or 13, wherein: The display manner for each of the plurality of first image regions differs according to a first image region positional relationship among the plurality of first image regions.
15. The image processing device according to any one of claims 1 to 14, wherein: The medical image is an image defined by a plurality of frames, the processor, detecting the first image area and the second image area for each of the frames, The display mode is determined for each of the frames.
16. The image processing apparatus according to claim 15, wherein: the processor, Based on the correspondence between the plurality of types of the parts and the lesions corresponding to each of the parts, determining for each of the frames whether the combination of the first image area and the second image area is correct or not, Based on the display mode corresponding to the frame used as the judgment object when it is judged that the combination of the first image area and the second image area is correct, the display mode corresponding to the frame used as the judgment object when it is judged that the combination of the first image area and the second image area is incorrect is corrected.
17. A medical diagnostic device comprising: The image processing device according to any one of claims 1 to 16; and A camera device is used to capture images of the observation target area.
18. An ultrasonic endoscope device comprising: The image processing device according to any one of claims 1 to 16; and An ultrasonic device acquires an ultrasonic image as the medical image.
19. An image processing method, comprising: Detecting, from a medical image obtained by imaging an observation target area including a part of a human body and a lesion, a first image area representing the part and a second image area representing the lesion; as well as The display device is caused to display the result of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
20. A program for causing a computer to execute the following processing, the processing comprising: detecting, from a medical image obtained by imaging an observation target region including a human body part and a lesion, a first image region representing the part and a second image region representing the lesion, and The display device is caused to display the result of detecting the first image area and the second image area in a display manner corresponding to the positional relationship between the first image area and the second image area.
21. An image processing device comprising a processor, the processor, A first image region representing a part of a human body and a lesion is detected from a medical image obtained by imaging an observation object region including the part of the human body and the lesion, and a second image region representing the lesion is detected. The accuracy of the second image area is determined according to the positional relationship between the first image area and the second image area.
22. The image processing apparatus according to claim 21, wherein: The processor determines the accuracy based on the positional relationship and a relationship between a first confidence level, which is a confidence level for a result of detecting the first image area, and a second confidence level, which is a confidence level for a result of detecting the second image area.
23. The image processing apparatus according to claim 22, wherein: When the positional relationship is in a predetermined positional relationship, the first image area and the second image area do not match, and the relationship between the first confidence and the second confidence is in a predetermined confidence relationship, the processor determines that the second image area is accurate.
24. The image processing apparatus according to claim 21, wherein: When the positional relationship is a predetermined positional relationship and the first image region matches the second image region, the processor determines that the second image region is correct.
25. The image processing device according to any one of claims 21 to 23, wherein: The processor determines an accuracy of the first image region.
26. The image processing device according to any one of claims 21 to 25, wherein: the processor, causing a display device to display the medical image, When it is determined that the second image area is correct, information indicating that the lesion is detected is displayed on a display.
27. The image processing apparatus according to claim 26, wherein: The location where the information indicating that the lesion has been detected is displayed is an area corresponding to the second image area in the display area where the medical image is displayed.
