Image processing device, medical diagnosis device, endoscopic ultrasonography device, image processing method, and program

Abstract

This image processing device is provided with a processor. The processor detects a first image region that shows a part of a human body and a second image region that shows a lesion from a medical image obtained by imaging an observation target region including the part and the lesion. The processor allows a display device to display a result of the detection of the first image region and the second image region in a display mode corresponding to the positional relationship between the first image region and the second image region.

Description

Image processing device, medical diagnostic device, ultrasonic endoscope device, image processing method, and program

[0001] The technology of the present disclosure relates to an image processing device, a medical diagnostic device, an ultrasonic endoscope device, an image processing method, and a program.

[0002] Japanese Patent Laid-Open Publication No. 2021-100555 discloses a medical image processing device having at least one processor. In the medical image processing device described in Japanese Patent Laid-Open Publication No. 2021-100555, the at least one processor acquires a medical image, acquires part information indicating a part of the human body of the subject shown in the medical image, detects a lesion from the medical image and acquires lesion type information indicating the type of the lesion, determines whether there is a contradiction between the part information and the lesion type information, and determines a notification mode for the part information and the lesion type information based on the determination result.

[0003] One embodiment of the technique of the present disclosure provides an image processing device, a medical diagnostic device, an ultrasonic endoscope device, an image processing method, and a program that enable a user or the like to accurately grasp a lesion.

[0004] A first aspect of the technology of the present disclosure is an image processing device that includes a processor, which detects a first image area indicating a part and a second image area indicating a lesion from a medical image obtained by capturing an image of an observation area including a part of a human body and a lesion, and displays the results of detecting the first image area and the second image area on a display device in a display mode that corresponds to the positional relationship between the first image area and the second image area.

[0005] A second aspect of the technology of the present disclosure is the image processing device according to the first aspect, in which the display mode is determined according to the site, the lesion, and the positional relationship.

[0006] A third aspect of the technology of the present disclosure is the image processing device according to the first or second aspect, in which the display mode is determined according to a positional relationship and consistency between a site and a lesion.

[0007] A fourth aspect of the technology of the present disclosure is an image processing device according to any one of the first to third aspects, in which the display mode for the first image area varies depending on the part, the lesion, and the positional relationship, and the display mode for the second image area is a mode in which the second image area is displayed on the display device.

[0008] A fifth aspect of the technology of the present disclosure is an image processing device according to the fourth aspect, in which, when the area and the lesion do not match, the display mode for the first image area is such that the first image area is not displayed on the display device, and the display mode for the second image area is such that the second image area is displayed on the display device.

[0009] A sixth aspect of the technology of the present disclosure is an image processing device according to the fourth or fifth aspect, in which, when the part and the lesion are aligned, the display mode for the first image area is such that the first image area is displayed on the display device and is determined according to the positional relationship, and the display mode for the second image area is such that the second image area is displayed on the display device.

[0010] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to sixth aspects, in which the positional relationship is defined by the degree of overlap or distance between the first image area and the second image area.

[0011] An eighth aspect of the technology of the present disclosure is an image processing device according to the seventh aspect, in which the positional relationship is determined by the degree of overlap, and when the degree of overlap is a first degree or higher, the display mode is such that the second image area is displayed in an identifiable manner within the medical image.

[0012] A ninth aspect of the technology of the present disclosure is an image processing device according to the seventh aspect, in which the positional relationship is determined by the degree of overlap, and when the degree of overlap is a first degree or higher, the display mode is such that the second image area is displayed so as to be identifiable within the medical image, and the first image area is displayed so as to be contrasted with the second image area.

[0013] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to ninth aspects, in which a processor acquires a first confidence level, which is a confidence level for the result of detecting a first image area, and a second confidence level, which is a confidence level for the result of detecting a second image area, and the display mode is determined according to the first confidence level, the second confidence level, and the positional relationship.

[0014] An eleventh aspect according to the technology of the present disclosure is the image processing device according to the tenth aspect, in which the display mode is determined according to the magnitude relationship between the first certainty degree and the second certainty degree and the positional relationship.

[0015] A twelfth aspect of the technology of the present disclosure is an image processing device according to any one of the first to eleventh aspects, in which the display mode is determined according to a plurality of positional relationships, and the plurality of positional relationships are positional relationships between a plurality of first image areas and a plurality of second image areas for a plurality of types of body parts.

[0016] A thirteenth aspect of the technology of the present disclosure is the image processing device according to the twelfth aspect, in which a display mode for each of the plurality of first image regions differs depending on each of a plurality of positional relationships.

[0017] A fourteenth aspect of the technology of the present disclosure is an image processing device according to the twelfth or thirteenth aspect, in which the display mode for each of the plurality of first image areas differs depending on the first image area positional relationship between the plurality of first image areas.

[0018] A fifteenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourteenth aspects, in which the medical image is an image defined by a plurality of frames, the processor detects a first image area and a second image area for each frame, and the display mode is determined for each frame.

[0019] A sixteenth aspect of the technology of the present disclosure is an image processing device according to the fifteenth aspect, in which the processor determines whether the combination of the first image area and the second image area is correct for each frame based on the correspondence between multiple types of parts and the lesions corresponding to each part, and corrects the display mode corresponding to the frame used as the judgment target when it is determined that the combination of the first image area and the second image area is incorrect, based on the display mode corresponding to the frame used as the judgment target when it is determined that the combination of the first image area and the second image area is correct.

[0020] A seventeenth aspect of the technology of the present disclosure is a medical diagnostic device comprising an image processing device according to any one of the first to sixteenth aspects and an imaging device that images an observation target area.

[0021] An 18th aspect of the technology of the present disclosure is an ultrasonic endoscopic device comprising an image processing device according to any one of the first to sixteenth aspects and an ultrasonic device that acquires ultrasonic images as medical images.

[0022] A nineteenth aspect of the technology of the present disclosure is an image processing method that includes detecting a first image area indicating a part and a second image area indicating a lesion from a medical image obtained by imaging an observation area including a part and a lesion of a human body, and displaying the results of detecting the first image area and the second image area on a display device in a display mode that corresponds to the positional relationship between the first image area and the second image area.

[0023] A twentieth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including detecting a first image area indicating a part and a second image area indicating a lesion from a medical image obtained by imaging an observation area including a part and a lesion of a human body, and displaying the results of detecting the first image area and the second image area on a display device in a display mode corresponding to the positional relationship between the first image area and the second image area.

[0024] A 21st aspect of the technology of the present disclosure is an image processing device that includes a processor, which detects a first image area indicating a part and a second image area indicating a lesion from a medical image obtained by imaging an observation area including a part and a lesion of a human body, and determines the likelihood of the second image area based on the positional relationship between the first image area and the second image area.

[0025] A 22nd aspect of the technology of the present disclosure is an image processing device according to the 21st aspect, in which a processor determines the likelihood based on the positional relationship and the relationship between a first certainty factor, which is the certainty factor for the result of detecting the first image area, and a second certainty factor, which is the certainty factor for the result of detecting the second image area.

[0026] A 23rd aspect of the technology of the present disclosure is an image processing device according to the 22nd aspect, in which the processor determines that the second image area is certain when the positional relationship is a predetermined positional relationship, the first image area and the second image area are not consistent, and the relationship between the first certainty degree and the second certainty degree is a predetermined certainty degree relationship.

[0027] A 24th aspect of the technology of the present disclosure is an image processing device according to the 21st aspect, in which the processor determines that the second image area is correct if the positional relationship is a predetermined positional relationship and the first image area and the second image area are consistent.

[0028] A twenty-fifth aspect of the technique of the present disclosure is the image processing device according to any one of the twenty-first to twenty-third aspects, in which the processor determines the likelihood of the first image region.

[0029] A 26th aspect of the technology of the present disclosure is an image processing device relating to any one of the 21st to 25th aspects, in which the processor displays a medical image on a display device and, if it determines that the second image area is reliable, displays information on the display indicating that a lesion has been detected.

[0030] A 27th aspect of the technique of the present disclosure is the image processing device according to the 26th aspect. In the image processing device according to the 27th aspect, the position where information indicating that a lesion has been detected is displayed is an area corresponding to the second image area within the display area where the medical image is displayed.

[0031] 1 is a conceptual diagram showing an example of an aspect in which an ultrasound endoscopic system is used. FIG. 1 is a conceptual diagram showing an example of the overall configuration of an ultrasound endoscopic system. FIG. 1 is a conceptual diagram showing an example of an aspect in which an insertion section of an ultrasound endoscope is inserted into the stomach of a subject. FIG. 2 is a block diagram showing an example of the hardware configuration of an endoscopic processing device. FIG. 3 is a block diagram showing an example of the hardware configuration of an ultrasound processing device. FIG. 4 is a block diagram showing an example of the hardware configuration of a display control device. FIG. 5 is a block diagram showing an example of main functions of a processor of the display control device. FIG. 6 is a conceptual diagram showing an example of the processing content of an acquisition section. FIG. 7 is a conceptual diagram showing an example of the processing content of an acquisition section, a detection section, and a determination section. FIG. 8 is a conceptual diagram showing an example of the processing content of an acquisition section, a determination section, and a control section. FIG. 9 is a conceptual diagram showing an example of the processing content of an acquisition section, a determination section, and a positional relationship identification section. FIG. 10 is a conceptual diagram showing an example of the processing content of an acquisition section, a detection section, a positional relationship identification section, and a control section when the degree of overlap is less than a default degree of overlap. FIG. 11 is a conceptual diagram showing an example of the processing content of an acquisition section, a detection section, a positional relationship identification section, and a control section when the degree of overlap is equal to or greater than the default degree of overlap. FIG. 12 is a flowchart showing an example of the flow of display control processing. FIG. 13 is a conceptual diagram showing an example of the processing content of a first modified example. FIG. 14 is a conceptual diagram showing an example of the processing content of a second modified example. FIG. 15 is a conceptual diagram showing an example of the processing content of a third modified example. FIG. 10 is a conceptual diagram showing an example of processing details relating to a fourth modified example of the detection unit and the determination unit. FIG. 11 is a conceptual diagram showing an example of processing details of the acquisition unit, determination unit, positional relationship identification unit, and control unit when the combination of the site area and the lesion area is incorrect and the overlapping degree is less than a default overlapping degree. FIG. 12 is a conceptual diagram showing an example of processing details of the acquisition unit, determination unit, positional relationship identification unit, and control unit when the combination of the site area and the lesion area is incorrect and the overlapping degree is equal to or greater than the default overlapping degree. FIG. 13 is a conceptual diagram showing an example of processing details of the acquisition unit, determination unit, positional relationship identification unit, and control unit when the combination of the site area and the lesion area is incorrect and the second certainty degree is equal to or less than the first certainty degree. FIG. 14 is a conceptual diagram showing an example of processing details of the acquisition unit, determination unit, positional relationship identification unit, and control unit when the combination of the site area and the lesion area is correct and the overlapping degree is less than the default overlapping degree.27A . FIG. 27A is a conceptual diagram showing an example of processing contents of the acquisition unit, determination unit, positional relationship identification unit, and control unit when the combination of the site region and the lesion region is correct and the second certainty degree is equal to or lower than the first certainty degree. FIG. 27B is a conceptual diagram showing an example of processing contents of the detection unit and determination unit according to a fifth modified example. FIG. 27B is a flowchart showing an example of the flow of display control processing according to the fifth modified example. FIG. 27A is a conceptual diagram showing an example of an ultrasound image of a conventional example and an ultrasound image on which display control processing according to the fifth modified example has been performed. FIG. 27B is a conceptual diagram showing an example of an ultrasound image of a conventional example and an ultrasound image on which display control processing according to the sixth modified example has been performed. FIG. 27B is a flowchart showing an example of the flow of display control processing according to the seventh modified example. FIG. 27B is a flowchart showing an example of the flow of display control processing according to the eighth modified example. FIG. 27B is a conceptual diagram showing an example of processing contents of the control unit according to a ninth modified example. FIG. 27C is a flowchart showing an example of the flow of display control processing according to the tenth modified example. FIG. 27A is a conceptual diagram showing an example of an ultrasound image of a conventional example and an ultrasound image on which display control processing according to the fifth modified example has been performed.

[0032] Hereinafter, examples of embodiments of an image processing device, a medical diagnostic device, an ultrasonic endoscope device, an image processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0033] First, the terms used in the following description will be explained.

[0034] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". CT is an abbreviation for "Computed Tomography". MRI is an abbreviation for "Magnetic Resonance Imaging". AI is an abbreviation for "Artificial Intelligence". FIFO is an abbreviation for "First In First Out". FPC is an abbreviation for "Flexible Printed Circuit". IoU is an abbreviation for "Intersection over Union".

[0035] As an example, as shown in FIG. 1 , an ultrasound endoscopic system 10 includes an ultrasound endoscopic device 12 and a display device 14. The ultrasound endoscopic device 12 is used by medical professionals such as doctors 16, nurses, and / or technicians (hereinafter referred to as "users"). The ultrasound endoscopic device 12 includes an ultrasound endoscope 18 and is an apparatus for performing medical examinations on the inside of a subject 20 (e.g., a patient) via the ultrasound endoscope 18. The ultrasound endoscopic device 12 is an example of a "medical diagnostic device" and an "ultrasound endoscopic device" according to the technology of the present disclosure. The ultrasound endoscope 18 is an example of an "imaging device" according to the technology of the present disclosure.

[0036] The ultrasound endoscope 18 allows the doctor 16 to capture an image of the subject 20, thereby acquiring and outputting an image showing the state inside the body. In the example shown in Fig. 1, the ultrasound endoscope 18 is shown inserted into a body cavity through the mouth of the subject 20. Note that, although the ultrasound endoscope 18 is inserted into the body cavity through the mouth of the subject 20 in the example shown in Fig. 1, this is merely an example, and the ultrasound endoscope 18 may also be inserted into the body cavity through the nostrils, anus, perforation, or the like of the subject 20.

[0037] The display device 14 displays various information including images. Examples of the display device 14 include a liquid crystal display and an EL display. The display device 14 displays a plurality of screens arranged side by side. In the example shown in Fig. 1, a first screen 22 and a second screen 24 are shown as examples of the plurality of screens.

[0038] The first screen 22 and the second screen 24 display different types of images obtained by the ultrasound endoscopic device 12. An ultrasound video 26 is displayed on the first screen 22. The ultrasound video 26 is a video generated based on reflected waves obtained when ultrasound is irradiated toward an observation region inside the body of the subject 20 and the ultrasound is reflected from the observation region. The ultrasound video 26 is displayed on the first screen 22 using a live view method. Although the live view method is illustrated here, this is merely an example, and other display methods such as a post view method may also be used. An example of an observation region to be irradiated with ultrasound is a region including an organ or lesion of the subject 20.

[0039] Here, the observation region to which ultrasound is irradiated is an example of an "observation region" according to the technology of the present disclosure. Furthermore, the subject's organs and lesions are examples of "body parts and lesions" according to the technology of the present disclosure. Furthermore, the ultrasound video image 26 (i.e., a video image generated based on reflected waves obtained by ultrasound being reflected from the observation region) is an example of a "medical image obtained by capturing an image of the observation region" according to the technology of the present disclosure.

[0040] An endoscopic moving image 28 is displayed on the second screen 24. An example of the endoscopic moving image 28 is a moving image generated by capturing visible light, near-infrared light, or the like. The endoscopic moving image 28 is displayed on the second screen 24 using a live view method. Note that in this embodiment, the endoscopic moving image 28 is also shown as an example together with the ultrasound moving image 26, but this is merely an example, and the technology of the present disclosure is applicable even without the endoscopic moving image 28.

[0041] As shown in FIG. 2 , the ultrasonic endoscope 18 includes a control section 30 and an insertion section 32. The insertion section 32 is tubular. The insertion section 32 includes a distal end section 34, a bending section 36, and a flexible section 37. The distal end section 34, the bending section 36, and the flexible section 37 are arranged in this order from the distal end to the proximal end of the insertion section 32. The flexible section 37 is made of a long, flexible material and connects the control section 30 and the bending section 36. The bending section 36 partially bends and rotates around the axis of the insertion section 32 when the control section 30 is operated. As a result, the insertion section 32 is advanced deeper into the body cavity while bending and rotating around the axis of the insertion section 32 in accordance with the shape of the body cavity (e.g., the shape of the digestive tract, such as the esophagus, stomach, duodenum, small intestine, and large intestine, or the shape of the bronchial duct).

[0042] The distal end portion 34 is provided with an illumination device 38, an endoscope 40, an ultrasound probe 42, and a treatment tool opening 44. The illumination device 38 has illumination windows 38A and 38B. The illumination device 38 emits light (e.g., white light or near-infrared light composed of three primary colors) through the illumination windows 38A and 38B. The endoscope 40 captures images of the inside of the body using an optical method. An example of the endoscope 40 is a CMOS camera. The CMOS camera is merely an example, and other types of cameras such as a CCD camera may also be used.

[0043] The ultrasonic probe 42 is provided on the distal end side of the distal end portion 34. The outer surface 42A of the ultrasonic probe 42 is curved in a convex shape that extends outward from the base end side to the distal end side of the ultrasonic probe 42. The ultrasonic probe 42 transmits ultrasonic waves via the outer surface 42A and receives reflected waves obtained when the transmitted ultrasonic waves are reflected from the observation target region via the outer surface 42A.

[0044] The treatment tool opening 44 is formed closer to the proximal end of the distal end portion 34 than the ultrasonic probe 42. This opening allows a treatment tool 46 to protrude from the distal end portion 34. A treatment tool insertion port 48 is formed in the operating unit 30, and the treatment tool 46 is inserted into the insertion section 32 from the treatment tool insertion port 48. The treatment tool 46 passes through the insertion section 32 and protrudes into the body from the treatment tool opening 44. In the example shown in FIG. 2 , a puncture needle with a guide sheath 50 protrudes from the treatment tool opening 44 as the treatment tool 46. The puncture needle with a guide sheath 50 has a puncture needle 50A and a guide sheath 50B. The puncture needle 50A passes through the guide sheath 50B and protrudes from the guide sheath 50B. Here, the treatment tool 46 is illustrated as a puncture needle with a guide sheath 50, but this is merely an example, and the treatment tool 46 may be a grasping forceps, a scalpel, a snare, or the like. The treatment tool opening 44 also functions as a suction port for sucking blood, internal waste, and the like.

[0045] The ultrasonic endoscope device 12 includes a universal cord 52, an endoscope processing device 54, a light source device 56, an ultrasonic processing device 58, and a display control device 60. The universal cord 52 has a base end 52A and first to third tip ends 52B to 52D. The base end 52A is connected to the operation unit 30. The first tip end 52B is connected to the endoscope processing device 54. The second tip end 52C is connected to the light source device 56. The third tip end 52D is connected to the ultrasonic processing device 58.

[0046] The ultrasound endoscope system 10 includes a reception device 62. The reception device 62 receives instructions from a user. Examples of the reception device 62 include an operation panel having multiple hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0047] A reception device 62 is connected to the endoscope processing device 54. The endoscope processing device 54 exchanges various signals with the endoscope 40 and controls the light source device 56 in accordance with instructions received by the reception device 62. The endoscope processing device 54 causes the endoscope 40 to capture images and acquires and outputs endoscopic moving images 28 (see FIG. 1 ) from the endoscope 40. The light source device 56 emits light under the control of the endoscope processing device 54 and supplies the light to the illumination device 38. A light guide is built into the illumination device 38, and the light supplied from the light source device 56 passes through the light guide and is irradiated from illumination windows 38A and 38B.

[0048] The ultrasonic processing device 58 is connected to a reception device 62. The ultrasonic processing device 58 exchanges various signals with the ultrasonic probe 42 in accordance with instructions received by the reception device 62. The ultrasonic processing device 58 causes the ultrasonic probe 42 to transmit ultrasonic waves, and generates and outputs an ultrasonic video image 26 based on the reflected waves received by the ultrasonic probe 42.

[0049] The display device 14, the endoscope processing device 54, the ultrasound processing device 58, and the reception device 62 are connected to the display control device 60. The display control device 60 controls the display device 14 in accordance with instructions received by the reception device 62. The display control device 60 acquires endoscopic moving images 28 from the endoscope processing device 54 and displays the acquired endoscopic moving images 28 on the display device 14 (see FIG. 1). The display control device 60 also acquires ultrasound moving images 26 from the ultrasound processing device 58 and displays the acquired ultrasound moving images 26 on the display device 14 (see FIG. 1).

[0050] 3, the insertion section 32 of the ultrasonic endoscope 18 is inserted into the stomach 64 of the subject 20. The endoscope 40 captures images of the inside of the stomach 64 at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second) to generate a live view image showing the state of the inside of the stomach 64 as an endoscopic moving image 28. In addition, when the tip 34 reaches a target position in the stomach 64,

[0051] When the ultrasonic probe 42 reaches the stomach 64, the outer surface 42A of the ultrasonic probe 42 comes into contact with the inner wall 64A of the stomach 64. With the outer surface 42A in contact with the inner wall 64A, the ultrasonic probe 42 transmits ultrasonic waves via the outer surface 42A. The ultrasonic waves are irradiated via the inner wall 64A onto an observation region including organs such as the pancreas 65 and kidneys 66 and lesions. The reflected waves obtained by the ultrasonic waves being reflected from the observation region are received by the ultrasonic probe 42 via the outer surface 42A. The reflected waves received by the ultrasonic probe 42 are then visualized as a live view image showing the state of the observation region at a predetermined frame rate, thereby obtaining an ultrasonic video image 26.

[0052] 3 shows an example in which ultrasonic waves are irradiated from inside the stomach 64 toward organs such as the pancreas 65 and kidneys 66 in order to detect a lesion in the pancreas 65, but this is merely one example. For example, ultrasonic waves may be irradiated from inside the duodenum toward organs such as the pancreas 65 and kidneys 66 in order to detect a lesion in the pancreas 65.

[0053] In addition, in the example shown in Figure 3, the ultrasonic endoscope 18 is shown inserted into the stomach 64, but this is merely one example, and the ultrasonic endoscope 18 may be inserted into organs such as the intestines and / or bronchi.

[0054] 4 , the endoscope processing device 54 includes a computer 67 and an input / output interface 68. The computer 67 includes a processor 70, a RAM 72, and an NVM 74. The input / output interface 68, the processor 70, the RAM 72, and the NVM 74 are connected to a bus 76.

[0055] For example, the processor 70 has a CPU and a GPU, and controls the entire endoscope processing device 54. The GPU operates under the control of the CPU and is responsible for executing various graphic-related processes. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0056] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0057] The input / output interface 68 is connected to the reception device 62, and the processor 70 acquires instructions accepted by the reception device 62 via the input / output interface 68 and executes processing in accordance with the acquired instructions. The input / output interface 68 is also connected to the endoscope 40. The processor 70 controls the endoscope 40 via the input / output interface 68 and acquires, via the input / output interface 68, endoscopic moving images 28 obtained by imaging the inside of the body of the subject 20 with the endoscope 40. The light source device 56 is also connected to the input / output interface 68. The processor 70 controls the light source device 56 via the input / output interface 68 to supply light to the illumination device 38 and adjust the amount of light supplied to the illumination device 38. The input / output interface 68 is also connected to the display control device 60. The processor 70 exchanges various signals with the display control device 60 via the input / output interface 68.

[0058] As an example, as shown in Fig. 5, the ultrasound processing device 58 includes a computer 78 and an input / output interface 80. The computer 78 includes a processor 82, a RAM 84, and a NVM 86. The input / output interface 80, the processor 82, the RAM 84, and the NVM 86 are connected to a bus 88. The processor 82 controls the entire ultrasound processing device 58. Note that the multiple hardware resources (processor 82, RAM 84, and NVM 86) included in the computer 78 shown in Fig. 5 are the same type as the multiple hardware resources included in the computer 67 shown in Fig. 4, and therefore a description thereof will be omitted here.

[0059] The input / output interface 80 is connected to the reception device 62, and the processor 82 acquires instructions accepted by the reception device 62 via the input / output interface 80 and executes processing according to the acquired instructions. The input / output interface 80 is also connected to the display control device 60. The processor 82 exchanges various signals with the display control device 60 via the input / output interface 80.

[0060] The ultrasound processing device 58 includes a multiplexer 90, a transmission circuit 92, a reception circuit 94, and an analog-to-digital converter 96 (hereinafter referred to as "ADC 96"). The multiplexer 90 is connected to the ultrasound probe 42. The input end of the transmission circuit 92 is connected to the input / output interface 80, and the output end of the transmission circuit 92 is connected to the multiplexer 90. The input end of the ADC 96 is connected to the output end of the reception circuit 94, and the output end of the ADC 96 is connected to the input / output interface 80. The input end of the reception circuit 94 is connected to the multiplexer 90.

[0061] The ultrasonic probe 42 includes a plurality of ultrasonic transducers 98. The ultrasonic probe 42 is formed by laminating, from the inside to the outside of the ultrasonic probe 42, a backing material layer, an ultrasonic transducer unit (i.e., a unit in which a plurality of ultrasonic transducers 98 are arranged in a one-dimensional or two-dimensional array), an acoustic matching layer, and an acoustic lens.

[0062] The backing material layer supports each ultrasonic vibrator 98 included in the ultrasonic vibrator unit from the back surface side. The backing material layer also has the function of attenuating ultrasonic waves propagated from the ultrasonic vibrators 98 to the backing material layer side. The backing material is made of a rigid material such as hard rubber, and an ultrasonic attenuation material (for example, ferrite, ceramics, etc.) is added as needed.

[0063] The acoustic matching layer is overlaid on the ultrasound transducer unit and is provided to achieve acoustic impedance matching between the subject 20 and the ultrasound transducer 98. The provision of the acoustic matching layer makes it possible to increase the transmittance of ultrasound. The material of the acoustic matching layer may be an organic material whose acoustic impedance value is closer to that of the subject 20 than that of the piezoelectric element included in the ultrasound transducer 98. For example, the material of the acoustic matching layer may be an epoxy resin, silicone rubber, polyimide, and / or polyethylene.

[0064] The acoustic lens is layered on the acoustic matching layer and focuses the ultrasonic waves emitted from the ultrasonic transducer unit toward the observation area. The acoustic lens is made of a material such as silicone resin, liquid silicone rubber, butadiene resin, and / or polyurethane resin, and may contain powders of titanium oxide, alumina, silica, or the like, as needed.

[0065] Each of the multiple ultrasonic transducers 98 is formed by arranging electrodes on both sides of a piezoelectric element. Examples of piezoelectric elements include barium titanate, lead zirconate titanate, and potassium niobate. The electrodes consist of individual electrodes provided individually for the multiple ultrasonic transducers 98 and a transducer ground common to the multiple ultrasonic transducers 98. The electrodes are electrically connected to the ultrasonic processing device 58 via an FPC and a coaxial cable.

[0066] The ultrasonic probe 42 is a convex probe in which multiple ultrasonic transducers 98 are arranged in an arc. The multiple ultrasonic transducers 98 are arranged along the outer surface 42A (see FIGS. 2 and 3 ). That is, the multiple ultrasonic transducers 98 are arranged at equal intervals in a convex curve along the axial direction of the tip portion 34 (i.e., the longitudinal axis direction of the insertion section 32). Therefore, the ultrasonic probe 42 transmits ultrasonic waves radially by activating the multiple ultrasonic transducers 98. Note that, although a convex probe is illustrated here, this is merely an example, and a radial, linear, or sector type probe may also be used. The scanning method of the ultrasonic probe 42 is not particularly limited.

[0067] The transmitting circuit 92 and the receiving circuit 94 are electrically connected to each of the plurality of ultrasonic transducers 98 via the multiplexer 90. The multiplexer 90 selects at least one of the plurality of ultrasonic transducers 98 and opens a channel of the selected ultrasonic transducer 98.

[0068] The transmission circuit 92 is controlled by the processor 82 via the input / output interface 80. The transmission circuit 92 supplies drive signals for ultrasonic transmission (e.g., multiple pulsed signals) to the selected ultrasonic transducer under the control of the processor 82. The drive signals are generated in accordance with transmission parameters set by the processor 82. The transmission parameters include the number of drive signals to be supplied to the selected ultrasonic transducer, the supply time of the drive signals, and the amplitude of the drive vibration.

[0069] The transmission circuit 92 supplies a drive signal to the selected ultrasonic transducer, causing the selected ultrasonic transducer to transmit ultrasonic waves. That is, when a drive signal is supplied to an electrode included in the selected ultrasonic transducer, the piezoelectric element included in the selected ultrasonic transducer expands and contracts, causing the selected ultrasonic transducer to vibrate. As a result, pulsed ultrasonic waves are output from the selected ultrasonic transducer. The output strength of the selected ultrasonic transducer is defined by the amplitude of the ultrasonic waves output from the selected ultrasonic transducer (i.e., the magnitude of the sound pressure of the ultrasonic waves).

[0070] The ultrasonic transducer 98 receives reflected waves obtained by transmitting ultrasonic waves and reflecting them from the observation area. The ultrasonic transducer 98 outputs an electrical signal indicative of the received reflected waves to the receiving circuit 94 via the multiplexer 90. Specifically, a piezoelectric element included in the ultrasonic transducer 98 outputs an electronic signal. The magnitude (i.e., voltage value) of the electrical signal output from the ultrasonic transducer 98 corresponds to the receiving sensitivity of the ultrasonic transducer 98. The receiving sensitivity of the ultrasonic transducer 98 is defined as the ratio of the amplitude of the electrical signal output by the ultrasonic transducer 98 when it receives ultrasonic waves to the amplitude of the ultrasonic waves transmitted by the ultrasonic transducer 98. The receiving circuit 94 receives the electrical signal from the ultrasonic transducer 98, amplifies the received electrical signal, and outputs it to the ADC 96. The ADC 96 digitizes the electrical signal input from the receiving circuit 94. The processor 82 acquires the electrical signal digitized by the ADC 96 and generates an ultrasonic video image 26 (see FIGS. 1 and 3 ) based on the acquired electrical signal, thereby acquiring the ultrasonic video image 26.

[0071] In this embodiment, the combination of the ultrasonic probe 42 and the ultrasonic processing device 58 is an example of an "imaging device" according to the technology of the present disclosure. Also, in this embodiment, the combination of the ultrasonic probe 42 and the ultrasonic processing device 58 is an example of an "ultrasound device" according to the technology of the present disclosure.

[0072] 6 , the display control device 60 includes a computer 100 and an input / output interface 102. The computer 100 includes a processor 104, a RAM 106, and an NVM 108. The input / output interface 102, the processor 104, the RAM 106, and the NVM 108 are connected to a bus 110.

[0073] Here, the display control device 60 is an example of an "image processing device" according to the technology of the present disclosure. The computer 100 is an example of a "computer" according to the technology of the present disclosure. The processor 104 is an example of a "processor" according to the technology of the present disclosure.

[0074] The processor 104 controls the entire display control device 60. Note that the multiple hardware resources (processor 104, RAM 106, and NVM 108) included in the computer 100 shown in Fig. 6 are the same type as the multiple hardware resources included in the computer 67 shown in Fig. 4, and therefore a description thereof will be omitted here.

[0075] The input / output interface 102 is connected to the reception device 62, and the processor 104 acquires instructions accepted by the reception device 62 via the input / output interface 102 and executes processing in accordance with the acquired instructions. The display device 14 is also connected to the input / output interface 102. The input / output interface 102 is also connected to the endoscope processing device 54, and the processor 104 exchanges various signals with the processor 70 of the endoscope processing device 54 via the input / output interface 102. The input / output interface 102 is also connected to the ultrasound processing device 58, and the processor 104 exchanges various signals with the processor 82 of the ultrasound processing device 58 via the input / output interface 102.

[0076] The display device 14 is connected to the input / output interface 102, and the processor 104 controls the display device 14 via the input / output interface 102 to display various information on the display device 14. For example, the processor 104 acquires endoscopic moving images 28 (see FIGS. 1 and 3) from the endoscopic processing device 54, acquires ultrasound moving images 26 (see FIGS. 1 and 3) from the ultrasound processing device 58, and causes the display device 14 to display the ultrasound moving images 26 and the endoscopic moving images 28.

[0077] The ultrasound endoscope 18 emits ultrasound waves inside the body of the subject 20 and converts the reflected waves obtained by the ultrasound waves being reflected off the observation region into ultrasound video images 26, thereby detecting lesions in the observation region while minimizing the physical burden on the subject 20. However, the ultrasound video images 26 have lower visibility than visible light images (e.g., endoscopic video images 28), which may result in lesions being overlooked or in variations in diagnostic results between physicians 16. Therefore, in recent years, AI-based image recognition processing has been used to prevent lesions from being overlooked and / or to reduce variations in diagnostic results between physicians 16.

[0078] However, even if a lesion is detected by AI-based image recognition processing, multiple organs may appear along with the lesion in the ultrasound video image 26, making it difficult to identify which of the multiple organs the lesion is in. Furthermore, when multiple organs overlap and appear in the ultrasound video image 26, it becomes even more difficult to identify which organ the lesion is in.

[0079] In view of these circumstances, in this embodiment, as shown in FIG. 7 as an example, display control processing is performed by the processor 104 in the display control device 60. A display control processing program 112 is stored in the NVM 108. The display control processing program 112 is an example of a "program" according to the technology of the present disclosure. The processor 104 reads the display control processing program 112 from the NVM 108 and executes the read display control processing program 112 on the RAM 106 to perform the display control processing. The display control processing is realized by the processor 104 operating as an acquisition unit 104A, a detection unit 104B, a determination unit 104C, a positional relationship identification unit 104D, and a control unit 104E in accordance with the display control processing program 112.

[0080] As an example, as shown in FIG. 8 , the acquisition unit 104A acquires an endoscopic moving image 28 from the endoscopic processing device 54. The endoscopic moving image 28 is an image defined by a plurality of frames. That is, the endoscopic moving image 28 includes a plurality of endoscopic images 114 obtained as a plurality of frames in time series by the endoscopic processing device 54 according to a predetermined frame rate. The acquisition unit 104A also acquires an ultrasound moving image 26 from the ultrasound processing device 58. The ultrasound moving image 26 is an image defined by a plurality of frames. That is, the ultrasound moving image 26 includes a plurality of ultrasound images 116 obtained as a plurality of frames in time series by the ultrasound processing device 58 according to a predetermined frame rate.

[0081] 9 , detection unit 104B detects site region 116A and lesion region 116B from ultrasound image 116 acquired by acquisition unit 104A. Site region 116A is an image region that indicates an organ (e.g., the pancreas) that appears in ultrasound image 116. Lesion region 116B is an image region that indicates a lesion (e.g., a tumor) that appears in ultrasound image 116. Site region 116A is an example of a "first image region" according to the technology of the present disclosure, and lesion region 116B is an example of a "second image region" according to the technology of the present disclosure.

[0082] The detection unit 104B detects the site region 116A and the lesion region 116B for each frame (i.e., for each of the multiple ultrasound images 116 included in the ultrasound video image 26). Then, display control processing is performed for each frame. In the following, to facilitate understanding of the technology of the present disclosure, a case where display control processing is performed for one frame will be described.

[0083] Detection unit 104B performs AI-based image recognition processing on ultrasound image 116 to detect site region 116A and lesion region 116B from ultrasound image 116. While AI-based image recognition processing is illustrated here, this is merely an example, and site region 116A and lesion region 116B may be detected by template matching-based image recognition processing instead of AI-based image recognition processing. Furthermore, detection unit 104B may use both AI-based image recognition processing and template matching-based image recognition processing.

[0084] Detection unit 104B generates part region information 118 and lesion region information 120. Part region information 118 is information related to part region 116A detected by detection unit 104B. Part region information 118 includes coordinate information 118A and part name information 118B. Coordinate information 118A is information including coordinates (e.g., two-dimensional coordinates) capable of identifying the position of part region 116A within ultrasound image 116 (e.g., the position of the outline of part region 116A). Part name information 118B is information capable of identifying the name of the part (i.e., the type of part) indicated by part region 116A detected by detection unit 104B (e.g., information indicating the name of the organ itself or an identifier capable of uniquely identifying the type of organ).

[0085] Lesion area information 120 is information related to lesion area 116B detected by detection unit 104B. Lesion area information 120 includes coordinate information 120A and lesion name information 120B. Coordinate information 120A is information including coordinates (e.g., two-dimensional coordinates) capable of identifying the position of lesion area 116B within ultrasound image 116 (e.g., the position of the contour of lesion area 116B). Lesion name information 120B is information capable of identifying the name of the lesion (i.e., the type of lesion) indicated by lesion area 116B detected by detection unit 104B (e.g., information indicating the name of the lesion or an identifier capable of uniquely identifying the type of lesion).

[0086] A consistency determination table 122 is stored in the NVM 112. The consistency determination table 122 associates a plurality of pieces of site name information 118B with a plurality of pieces of lesion name information 118B in a one-to-one correspondence. That is, the consistency determination table 122 defines the names of sites identified from the site name information 120B and the names of lesions identified from the lesion name information 120B. In other words, the consistency determination table 122 defines correct combinations of sites and lesions. In the example shown in FIG. 9 , the combination of pancreas and pancreatic cancer and the combination of kidney and kidney cancer are shown as examples of some of the correct combinations of sites and lesions. The consistency determination table is an example of a "correspondence" according to the technology of the present disclosure.

[0087] Determination unit 104C acquires region name information 118B and lesion name information 120B from region region information 118. Then, determination unit 104C references consistency determination table 122 stored in NVM 112 to determine the consistency of the combination of region name information 118B and lesion name information 120B, thereby determining the consistency between region region 116A and lesion region 116B (in other words, whether the combination of the region and the lesion is correct or not). That is, determination unit 104C references consistency determination table 122 to determine whether the combination of the region name identified from region name information 118B and the lesion name identified from lesion name information 120B is correct or not. In this way, determination unit 104C determines whether the combination of region region 116A and lesion region 116B detected by detection unit 104B is correct or not (i.e., whether the combination of region region 116A and lesion region 116B is consistent or not).

[0088] As an example, as shown in FIG. 10 , control unit 104E acquires endoscopic image 114 and ultrasound image 116 that was the subject of judgment by judgment unit 104C, displays acquired ultrasound image 116 on first screen 22, and displays acquired endoscopic image 114 on second screen 24. Here, if judgment unit 104C determines that part region 116A and lesion region 116B are not consistent, control unit 104E displays ultrasound image 116 on first screen 22 in a first display mode. The first display mode refers to a display mode in which part region 116A is not displayed and lesion region 116B is displayed. In the example shown in FIG. 10 , part region 116A is not displayed in ultrasound image 116 on first screen 22, and lesion region 116B is displayed.

[0089] 11 , when determination unit 104C determines that site region 116A and lesion region 116B are consistent, positional relationship determination unit 104D acquires the positional relationship between site region 116A and lesion region 116B. As an example, the positional relationship between site region 116A and lesion region 116B is defined by an overlap, which is the degree to which site region 116A and lesion region 116B overlap. Positional relationship determination unit 104D acquires coordinate information 118A from site region information 118, acquires coordinate information 120A from lesion region information 120, and calculates overlap 124 between site region 116A and lesion region 116B using coordinate information 118A and 120A. An example of an index of overlap 124 is IoU. In this case, for example, overlap 124 is the ratio of the area of ​​the overlapping region of lesion region 116B and site region 116A to the area of ​​the combined region of lesion region 116B and site region 116A. In the example shown in Figure 11, the state in which the entire lesion region 116B overlaps with site region 116A is shown as "overlap = 1.0," and the state in which only a portion of lesion region 116B overlaps with site region 116A is shown as "overlap = 0.4."

[0090] Although IoU is used as an example here, the technology of the present disclosure is not limited to this, and overlap 124 may be the ratio of the area of ​​the area where lesion area 116B and site area 116A overlap to lesion area 116B.

[0091] 12 and 13 , control unit 104E causes display device 14 to display the results of detection of site region 116A and lesion region 116B by detection unit 104B in a display format that corresponds to the positional relationship between site region 116A and lesion region 116B (here, as an example, overlapping degree 124). In the present embodiment, as an example, the display format in which the results of detection of site region 116A and lesion region 116B by detection unit 104B are displayed on display device 14 is determined according to the site indicated by site region 116A, the type of lesion (hereinafter also simply referred to as "lesion"), and the positional relationship between site region 116A and lesion region 116B (for example, the consistency between the site indicated in site region 116A and the lesion and overlapping degree 124).

[0092] More specifically, for example, the display mode in which the results of detection of site area 116A by detection unit 104B are displayed on display device 14 varies depending on the site and lesion indicated by site area 116A, and the positional relationship between site area 116A and lesion area 116B (e.g., consistency and overlap 124 between the site and lesion indicated in site area 116A). Also, for example, the display mode in which the results of detection of lesion area 116B by detection unit 104B are displayed on display device 14 is a mode in which lesion area 116B is displayed on first screen 22.

[0093] 12 , the positional relationship identification unit 104D determines whether the overlapping degree 124 is equal to or greater than a default overlapping degree (here, for example, "0.5"). The default overlapping degree may be a fixed value, or may be a variable value that changes depending on instructions and / or various conditions accepted by the accepting device 62. The default overlapping degree is an example of a "first degree" according to the technology of the present disclosure.

[0094] The control unit 104E acquires the endoscopic image 114 and the ultrasound image 116 that was the subject of determination by the determination unit 104C, displays the acquired ultrasound image 116 on the first screen 22, and displays the acquired endoscopic image 114 on the second screen 24. Here, when the positional relationship identification unit 104D determines that the overlapping degree 124 is less than the default overlapping degree, the control unit 104E displays the ultrasound image 116 displayed on the first screen 22 in the first display mode.

[0095] As an example, as shown in FIG. 13 , control unit 104E acquires endoscopic image 114 and ultrasound image 116 that was the subject of determination by determination unit 104C, displays acquired ultrasound image 116 on first screen 22, and displays acquired endoscopic image 114 on second screen 24. Here, if positional relationship determination unit 104D determines that overlap 124 is equal to or greater than a default overlap, control unit 104E displays ultrasound image 116 displayed on first screen 22 in a second display mode. The second display mode refers to a mode in which lesion region 116B is displayed so as to be identifiable within ultrasound image 116, and site region 116A is displayed so as to be contrasted with lesion region 116B. In the example shown in FIG. 13 , as an example of the second display mode, the outlines of site region 116A and lesion region 116B are bordered by curved lines, and the outline of lesion region 116B is bordered thicker than the outline of site region 116A. In other words, the positions of site region 116A and lesion region 116B are displayed so as to be identifiable within ultrasound image 116, and lesion region 116B is displayed more highlighted than site region 116A, thereby enabling site region 116A and lesion region 116B to be distinguished from each other. Displaying lesion region 116B more highlighted than site region 116A means that lesion region 116B is displayed more prominently than site region 116A.

[0096] Control unit 104E acquires coordinate information 118A from site region information 118 and acquires coordinate information 120A from lesion region information 120 in order to display ultrasound image 116 in the second display mode (the display mode shown in FIG. 13 as an example) on display device 14. Control unit 104E then processes ultrasound image 116 based on coordinate information 118A and 120A, and displays the result on first screen 22. The processing performed on ultrasound image 116 by control unit 104E refers to processing in which the outline of site region 116A is identified from coordinate information 118A and outlined with a curve, the outline of lesion region 116B is identified from coordinate information 120A and outlined with a curve, and the outline of lesion region 116B is made thicker than the outline of site region 116A.

[0097] Here, an example has been described in which the outline of lesion region 116B is thicker than the outline of site region 116A, but this is merely one example. For example, the brightness of the outline of lesion region 116B may be higher than the brightness of the outline of site region 116A. Furthermore, for example, a pattern or color may be applied to lesion region 116B, and a pattern or color that is less noticeable than that of lesion region 116B may be applied to site region 116A. Furthermore, for example, a pattern or color may be applied to only lesion region 116B of site region 116A and lesion region 116B, and the outline of site region 116A may be outlined with a curved line. The line types of the curves that outline the outline of site region 116A and the outline of lesion region 116B may be different to make site region 116A more noticeable than lesion region 116B. In this way, any display mode may be used as long as it allows site region 116A and lesion region 116B to be displayed in a manner that allows them to be identified and compared (i.e., distinguished).

[0098] Next, an example of the flow of the display control process performed by the processor 104 of the display control device 60 when an instruction to start execution of the display control process is accepted by the accepting device 62 will be described with reference to Fig. 14. Note that the flow of the process shown in the flowchart of Fig. 14 is an example of an "image processing method" according to the technique of the present disclosure.

[0099] In the display control process shown in Fig. 14, first, in step ST10, the acquisition unit 104A acquires an endoscopic image 114 from the endoscopic processing device 54 and acquires an ultrasound image 116 from the ultrasound processing device 58 (see Fig. 8). In step ST10, the endoscopic image 114 acquired by the acquisition unit 104A is a single-frame endoscopic image 114 that has not yet been used in the processes of step ST22 or step ST24, among the multiple endoscopic images 114 included in the endoscopic moving image 28. In addition, in step ST10, the ultrasound image 116 acquired by the acquisition unit 104A is a single-frame ultrasound image 116 that has not yet been used in the processes of step ST12 and subsequent steps, among the multiple ultrasound images 116 included in the ultrasound moving image 26. After the process of step ST10 is executed, the display control process proceeds to step ST12.

[0100] In step ST12, detection unit 104B performs AI image recognition processing on ultrasound image 116 acquired in step ST10 to detect site region 116A and lesion region 116B from ultrasound image 116 (see FIG. 9 ). After the processing of step ST12 is executed, the display control processing proceeds to step ST14.

[0101] In step ST14, detection unit 104B generates part area information 118, which is information about part area 116A, and lesion area information 120, which is information about lesion area 116B (see FIG. 9 ). After the processing of step ST14 is executed, the display control processing proceeds to step ST16.

[0102] In step ST16, determination unit 104C refers to consistency determination table 122 and determines whether or not site area 116A and lesion area 116B are consistent based on site area information 118 and lesion area information 120 generated in step ST14 (see FIG. 9 ). If site area 116A and lesion area 116B are not consistent in step ST16, the determination is negative, and the display control process proceeds to step ST22. If site area 116A and lesion area 116B are consistent in step ST16, the determination is positive, and the display control process proceeds to step ST18.

[0103] In step ST18, positional relationship identification unit 104D acquires coordinate information 118A from site region information 118 generated in step ST14, and acquires coordinate information 120A from lesion region information 120 generated in step ST14 (see FIG. 11 ). Positional relationship identification unit 104D then calculates overlap 124 using coordinate information 118A and 120A (see FIG. 11 ). After the processing of step ST18 is executed, the display control processing proceeds to step ST20.

[0104] In step ST20, the positional relationship specification unit 104D calculates the overlapping degree calculated in step ST18.

[0105] In step ST20, it is determined whether the multiplicity 124 is equal to or greater than the default multiplicity (see FIGS. 12 and 13). If the multiplicity 124 is less than the default multiplicity, the determination is negative, and the display control process proceeds to step ST22. If the multiplicity 124 is equal to or greater than the default multiplicity, the determination is positive, and the display control process proceeds to step ST24.

[0106] The processes of steps ST16 to ST20 are executed to determine the likelihood of lesion 116. If the determination in step ST16 is affirmative (i.e., site area 116A and lesion area 116B are consistent) and if the determination in step ST20 is affirmative (i.e., site area 116A and lesion area 116B are in a known positional relationship), lesion area 116 is determined to be certain.

[0107] In step ST22, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasound image 116 in the first display mode. That is, the control unit 104E hides the site region 116A in the ultrasound image 116, and displays the lesion region 116B (see FIGS. 10 and 12). After the processing of step ST22 is executed, the display control processing proceeds to step ST26.

[0108] In step ST24, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasound image 116 in the second display mode. That is, the control unit 104E displays the site region 116A and the lesion region 116B in the ultrasound image 116 so that they can be compared and distinguished from each other (see FIG. 13 ). After the processing of step ST24 is executed, the display control processing proceeds to step ST26.

[0109] In step ST26, the control unit 104E determines whether a condition for terminating the display control process (hereinafter referred to as a "display control process termination condition") has been satisfied. One example of the display control process termination condition is a condition in which an instruction to terminate the display control process has been accepted by the acceptance device 62. If the display control process termination condition has not been satisfied in step ST26, the determination is negative, and the display control process proceeds to step ST10. If the display control process termination condition has been satisfied in step ST26, the determination is positive, and the display control process ends.

[0110] As described above, in ultrasound endoscopic system 10, detection unit 104B detects site region 116A and lesion region 116B from ultrasound image 116. Here, for example, if site region 116A does not overlap with lesion region 116B, the lesion indicated by lesion region 116B may be unrelated to the site indicated by site region 116A. Furthermore, if site region 116A and lesion region 116B do not overlap at all, the lesion indicated by lesion region 116B is likely to be unrelated to the site indicated by site region 116A. Conversely, if lesion region 116B entirely overlaps with site region 116A, the lesion indicated by lesion region 116B can be said to be a lesion that is closely related to the site indicated by site region 116A.

[0111] Therefore, in ultrasound endoscopic system 10, the likelihood of lesion region 116B is determined based on the positional relationship between part region 116A and lesion region 116B (see step ST20 in FIG. 14 ). The determination result is then displayed on second screen 22 (see FIGS. 12 and 13 ). That is, the detection results of part region 116A and lesion region 116B from ultrasound image 116 by detector 104B are displayed on first screen 22 in a display format that corresponds to the positional relationship between part region 116A and lesion region 116B (see FIGS. 12 and 13 ). This allows the user, etc., to accurately grasp the lesion. For example, compared to a case where the detection results of part region 116A and lesion region 116B are always displayed in a constant display format regardless of the positional relationship between part region 116A and lesion region 116B, the user can accurately grasp the lesion.

[0112] Furthermore, in the ultrasound endoscopic system 10, the display mode for displaying the results of detection of the site region 116A and the lesion region 116B from the ultrasound image 116 by the detector 104B on the first screen 22 is determined according to the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B. Therefore, the display device 14 displays the results of detection of the site region 116A and the lesion region 116B from the ultrasound image 116 by the detector 104B in a display mode according to the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B (see FIGS. 10 , 12 , and 13 ). This allows the user to accurately grasp the lesion. For example, compared to a case where the results of detection of the site region 116A and the lesion region 116B are always displayed in a fixed display mode regardless of the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B.

[0113] Furthermore, in ultrasound endoscopic system 10, the display mode for displaying the results of detection of part region 116A and lesion region 116B from ultrasound image 116 by detector 104B on first screen 22 is determined based on the positional relationship between part region 116A and lesion region 116B and the consistency between the part and the lesion. Therefore, on display device 14, the results of detection of part region 116A and lesion region 116B from ultrasound image 116 by detector 104B are displayed in a display mode based on the positional relationship between part region 116A and lesion region 116B and the consistency between the part and the lesion (see FIGS. 10 , 12 , and 13 ). This allows the user, etc., to accurately grasp the lesion. For example, compared to a case where the results of detection of part region 116A and lesion region 116B are always displayed in a fixed display mode regardless of the positional relationship between part region 116A and lesion region 116B and the consistency between the part and the lesion, the user can accurately grasp the lesion.

[0114] Furthermore, in the ultrasound endoscopic system 10, the display mode of the site region 116A varies depending on the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B, and the lesion region 116B is displayed on the first screen 22 (see FIGS. 10 , 12 , and 13 ). For example, in the examples shown in FIGS. 10 and 12 , the lesion region 116B is displayed on the first screen 22, but the site region 116A is not, whereas in the example shown in FIG. 13 , the site region 116A and the lesion region 116B are displayed on the first screen 22. This makes it easier for the user to recognize the difference between the site and the lesion. For example, this makes it easier for the user to recognize the difference between the site and the lesion, compared to when the display mode of the site region 116A is always constant regardless of the site, the lesion, and the positional relationship between the site region 116A and the lesion region 116B.

[0115] Furthermore, in ultrasound endoscopic system 10, when the site and the lesion do not match, site region 116A is not displayed on first screen 22, and lesion region 116B is displayed on first screen 22 (see FIG. 10 ). This prevents the site from being mistakenly recognized as a lesion, or vice versa. For example, compared to when both site region 116A and lesion region 116B are displayed even though the site and the lesion do not match, this prevents the site from being mistakenly recognized as a lesion, or vice versa.

[0116] Furthermore, in ultrasound endoscopic system 10, the positional relationship between part region 116A and lesion region 116B is defined by overlapping degree 124. Therefore, the results of detection of part region 116A and lesion region 116B from ultrasound image 116 by detector 104B can be displayed on first screen 22 in a display mode according to overlapping degree 124 (see FIGS. 12 and 13 ).

[0117] Furthermore, in the ultrasound endoscopic system 10, the positional relationship between the part region 116A and the lesion region 116B is defined by the overlapping degree 124, and when the overlapping degree 124 is equal to or greater than a predetermined overlapping degree, the lesion region 116B is displayed so as to be identifiable within the ultrasound image 116 (see FIG. 13 ). This allows the user to grasp lesions that are highly related to the part shown in the ultrasound image 116.

[0118] Furthermore, in ultrasound endoscopic system 10, the positional relationship between site region 116A and lesion region 116B is defined by overlapping degree 124, and when overlapping degree 124 is equal to or greater than a predetermined overlapping degree, lesion region 116B is displayed so as to be identifiable within ultrasound image 116, and site region 116A is displayed so as to be contrasted with lesion region 116B. This allows the user to grasp the positional relationship between a site and a lesion that is highly associated with the site.

[0119] Furthermore, in the ultrasound endoscopic system 10, the ultrasound video 26 is a video defined by a plurality of ultrasound images 116, and the detection unit 104B detects a part area 116A and a lesion area 116B for each ultrasound image 116, and the display mode of the part area 116A and the lesion area 116B is determined for each ultrasound image 116. Therefore, even if the ultrasound video 26 is a video defined by a plurality of ultrasound images 116, the user or the like can grasp the lesion location for each ultrasound image 116.

[0120] Furthermore, in ultrasound endoscopic system 10, the reliability of lesion 116 is determined by performing the processes of steps ST16 to ST20. For example, if part region 116A and lesion region 116B are consistent and have a known positional relationship (e.g., step ST20: Y), lesion region 116 is determined to be reliable. Then, part region 116A and lesion region 116B are displayed in ultrasound image 116 so that they can be compared and distinguished (see FIG. 13 ). This allows the user to grasp which parts and lesions are highly related.

[0121] [First Modification] In the above embodiment, an example (see FIGS. 11 to 13 ) has been described in which part region 116A is displayed or hidden when the combination of part region 116A and lesion region 116B is consistent, but the technology of the present disclosure is not limited to this. For example, when the combination of part region 116A and lesion region 116B is consistent, the display mode of part region 116A may be such that part region 116A is displayed on first screen 22 and is determined depending on the positional relationship between part region 116A and lesion region 116B.

[0122] 15 as an example, when the positional relationship specifying unit 104D determines that the overlapping degree 124 is equal to or greater than the default overlapping degree, the control unit 104E displays the ultrasound image 116 on the first screen 22 in the second display mode, and adjusts the intensity of the outline of the part region 116A to an intensity corresponding to the overlapping degree 124. For example, the larger the overlapping degree 124, the more noticeable the outline becomes.

[0123] Examples of methods for making the contours more prominent include increasing the brightness of the contours or increasing the thickness of the contours. In the example shown in Fig. 15, the contour of the site region 116A displayed on the first screen 22 when the overlapping degree is "1.0" is thicker than the contour of the site region 116A displayed on the first screen 22 when the overlapping degree is "0.6." Therefore, the site region 116A displayed on the first screen 22 when the overlapping degree is "1.0" stands out more than the site region 116A displayed on the first screen 22 when the overlapping degree is "0.6." This allows the user to recognize the positional relationship between the matching site and the lesion (e.g., overlapping degree 124).

[0124] [Second Variant] In the above embodiment, the name of the part indicated by the part area 116A is not displayed on the first screen 22, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 16 , information indicating the name of the part indicated by the part area 116A may be displayed on the first screen 22.

[0125] In this case, control unit 104E acquires part name information 118B from part region information 118 and superimposes information indicating the name of the part identified from part name information 118B on part region 116A of first screen 22. In the example shown in FIG. 16 , the word "pancreas" is superimposed on part region 116A as information indicating the part name. Instead of letters, numbers, marks, or figures that can identify the part name (i.e., the type of part) may be used. Furthermore, instead of superimposed display, information indicating the part name (the word "pancreas" in the example shown in FIG. 16 ) may be displayed as a pop-up from part region 116A, as shown in FIG. 16 . Control unit 104E may also switch between displaying and hiding information indicating the part name in response to an instruction received by reception device 62. In this way, by displaying information indicating the part name in association with part region 116A, a user or the like can understand the name of the part indicated by part region 116A displayed on first screen 22.

[0126] Furthermore, the control unit 104E may acquire lesion name information 120B from the lesion area information 120 and superimpose information indicating the name of the lesion identified from the lesion name information 120B on the lesion area 116B of the first screen 22. Instead of superimposing the information indicating the name of the lesion, the information may be displayed as a pop-up from the lesion area 116B. The control unit 104E may also switch between displaying and hiding the information indicating the name of the lesion in response to an instruction received by the reception device 62. Displaying the information indicating the name of the lesion in association with the lesion area 116B in this manner allows the user or the like to understand the name of the lesion indicated by the lesion area 116B displayed on the first screen 22.

[0127] [Third Modification] In the above embodiment, overlapping degree 124 has been described as an example. However, this is merely an example. For example, as shown in FIG. 17 , distance 126 may be calculated by positional relationship identification unit 104D instead of overlapping degree 124. Distance 126 is calculated using coordinate information 118A and 120A, similar to overlapping degree 124. When overlapping degree 124 is 1.0, i.e., when the entire lesion region 116B overlaps with site region 116A, distance 126 is 0 millimeters. When a non-overlapping region exists between site region 116A and lesion region 116B, distance 126 exceeds 0 millimeters. An example of distance 126 is the distance between site region 116A and a portion of the outline of a region of lesion region 116B that does not overlap with site region 116A. For example, a portion of the outline of a region that does not overlap with part region 116A refers to position 116B1 of the outline of a region that does not overlap with part region 116A that is farthest from part region 116A. In this way, even if distance 126 is used instead of overlap 124, the same effect as in the above embodiment can be obtained.

[0128] [Fourth Modification] In the above embodiment, an example was described in which the display mode of ultrasound image 116 was determined depending on the site, the lesion, and the positional relationship between the site region and lesion region 116B, but the technology of the present disclosure is not limited to this. For example, the display mode of ultrasound image 116 may be determined depending on the confidence level of the result of detection of site region 116A by AI-based image recognition processing, the confidence level of the result of detection of lesion region 116B by AI-based image recognition processing, and the positional relationship between site region 116A and lesion region 116B.

[0129] Here, an example of the confidence level for the result of detecting part region 116A by AI-based image recognition processing is a value corresponding to the maximum score among multiple scores obtained from a trained model obtained by having a neural network perform machine learning to detect part region 116A. Also, an example of the confidence level for the result of detecting lesion region 116B by AI-based image recognition processing is a value corresponding to the maximum score among multiple scores obtained from a trained model obtained by having a neural network perform machine learning to detect lesion region 116B. An example of a value corresponding to the score is a value obtained by converting the score using an activation function used as the output layer of the neural network (i.e., a probability expressed as a value ranging from 0 to 1). An example of an activation function is a softmax function used as the output layer of multi-class classification.

[0130] As an example, as shown in FIG. 18 , detection unit 104B acquires first certainty factor 118C and second certainty factor 120C used in the AI-based image recognition processing of ultrasound image 116. First certainty factor 118C is the certainty factor for the result of detection of site region 116A by the AI-based image recognition processing. Second certainty factor 120C is the certainty factor for the result of detection of lesion region 116B by the AI-based image recognition processing. Detection unit 104B generates information including coordinate information 118A, site name information 118B, and first certainty factor 118C as site region information 118. Detection unit 104B also generates information including coordinate information 120A, lesion name information 120B, and second certainty factor 120C as lesion region information 120. Determination unit 104C determines the consistency between site region 116A and lesion region 116B in the same manner as in the above embodiment.

[0131] As an example, as shown in FIGS. 19 to 24, the display mode of ultrasound image 116 is determined according to the magnitude relationship between first certainty factor 118C and second certainty factor 1120C, and the positional relationship between site region 116A and lesion region 116B.

[0132] 19 and 20 , when determination unit 104C determines that part area 116A and lesion area 116B do not match, positional relationship identification unit 104D determines whether second certainty factor 120C included in lesion area information 120 is greater than first certainty factor 118C included in part area information 118. When second certainty factor 120C is greater than first certainty factor 118C, positional relationship identification unit 104D calculates overlapping factor 124 in the same manner as in the above embodiment, and determines whether overlapping factor 124 is greater than or equal to a default overlapping factor.

[0133] 19 , when the positional relationship specifying unit 104D determines that the overlapping degree 124 is less than the default overlapping degree, the control unit 104E displays the endoscopic image 114 on the second screen 24 and the ultrasound image 116 in the first display mode on the first screen 22. On the other hand, when the positional relationship specifying unit 104D determines that the overlapping degree 124 is equal to or greater than the default overlapping degree, as shown in FIG. 20 , the control unit 104E displays the endoscopic image 114 on the second screen 24 and the ultrasound image 116 in the third display mode on the first screen 22. The third display mode refers to a mode in which the lesion region 116B is highlighted. Examples of methods for highlighting the lesion region 116B include increasing the brightness of the outline of the lesion region 116B, adding a color or pattern to the lesion region 116B, or hiding the region of the ultrasound image 116 other than the lesion region 116B. In this way, the lesion area 116B is highlighted by being displayed in a manner that allows it to be distinguished from other areas in the ultrasound image 116.

[0134] 21 , when determination unit 104C determines that part region 116A and lesion region 116B do not match, positional relationship identification unit 104D determines whether second certainty factor 120C included in lesion region information 120 is greater than first certainty factor 118C included in part region information 118. Here, when positional relationship identification unit 104D determines that second certainty factor 120C is equal to or less than the first certainty factor, control unit 104E displays endoscopic image 114 acquired by acquisition unit 104A on second screen 24, and displays ultrasound image 116 acquired by acquisition unit 104A on first screen 22.

[0135] 22 and 23 , when determination unit 104C determines that part area 116A and lesion area 116B are consistent, positional relationship determination unit 104D determines whether second certainty factor 120C included in lesion area information 120 is greater than first certainty factor 118C included in part area information 118. When second certainty factor 120C is greater than the first certainty factor, positional relationship determination unit 104D calculates overlapping factor 124 in the same manner as in the above embodiment, and determines whether overlapping factor 124 is equal to or greater than a default overlapping factor.

[0136] 22 as an example, when the positional relationship specifying unit 104D determines that the degree of overlap 124 is less than the default degree of overlap, the control unit 104E displays the endoscopic image 114 on the second screen 24 and displays the ultrasound image 116 in the first display mode on the first screen 22. On the other hand, when the positional relationship specifying unit 104D determines that the degree of overlap 124 is equal to or greater than the default degree of overlap, as shown in FIG. 23 as an example, the control unit 104E displays the ultrasound image 116 in the second display mode on the first screen 22.

[0137] 24 , when determination unit 104C determines that part region 116A and lesion region 116B are consistent, positional relationship identification unit 104D determines whether second certainty factor 120C included in lesion region information 120 is greater than first certainty factor 118C included in part region information 118. Here, when detection unit 104B determines that second certainty factor 120C is equal to or less than the first certainty factor, control unit 104E displays endoscopic image 114 acquired by acquisition unit 104A on second screen 24, and displays ultrasound image 116 acquired by acquisition unit 104A on first screen 22.

[0138] Next, an example of the flow of the display control processing performed by the processor 104 of the display control device 60 when an instruction to start executing the display control processing related to this fourth variant is accepted by the acceptance device 62 will be described with reference to Figures 25A and 25B.

[0139] The flowcharts shown in Figures 25A and 25B differ from the flowchart shown in Figure 14 in that steps ST50 to ST64 are applied instead of steps ST14 and ST16. Note that the same steps as those in the flowchart shown in Figure 14 are given the same step numbers and will not be described here.

[0140] 25A , in step ST50, detection unit 104B generates information including coordinate information 118A, region name information 118B, and first certainty factor 118C as region region information 118. Detection unit 104B also generates information including coordinate information 120A, lesion name information 120B, and second certainty factor 120C as lesion region information 120. After the processing of step ST50 is executed, the display control processing proceeds to step ST52.

[0141] In step ST52, determination unit 104C refers to consistency determination table 122 and determines whether or not site area 116A and lesion area 116B are consistent based on site area information 118 and lesion area information 120 generated in step ST50 (see FIG. 18 ). If site area 116A and lesion area 116B are not consistent in step ST52, the determination is negative, and the display control process proceeds to step ST56 shown in FIG. 25B . If site area 116A and lesion area 116B are consistent in step ST52, the determination is positive, and the display control process proceeds to step ST54.

[0142] In step ST54, positional relationship identification unit 104D acquires first certainty factor 118C from site region information 118 generated in step ST50, and acquires second certainty factor 120C from lesion region information 120 generated in step ST50. Then, positional relationship identification unit 104D determines whether second certainty factor 120C is greater than first certainty factor 118C. In step ST54, if second certainty factor 120C is equal to or less than first certainty factor 118C, the determination is negative, and the process proceeds to step ST64 shown in FIG. 25B. In step ST54, if second certainty factor 120C is greater than first certainty factor 118C, the determination is positive, and the display control process proceeds to step ST18.

[0143] 25B , positional relationship identification unit 104D acquires first certainty factor 118C from site region information 118 generated in step ST50, and acquires second certainty factor 120C from lesion region information 120 generated in step ST50. Positional relationship identification unit 104D then determines whether second certainty factor 120C is greater than first certainty factor 118C. If second certainty factor 120C is equal to or less than first certainty factor 118C in step ST56, the determination is negative, and the process proceeds to step ST64. If second certainty factor 120C is greater than first certainty factor 118C in step ST56, the determination is positive, and the display control process proceeds to step ST58.

[0144] In step ST58, positional relationship specifying unit 104D acquires coordinate information 118A from site region information 118 generated in step ST50, and acquires coordinate information 120A from lesion region information 120 generated in step ST50 (see FIGS. 19 and 20). Positional relationship specifying unit 104D then calculates overlap 124 using coordinate information 118A and 120A (see FIGS. 19 and 20). After the processing of step ST58 is executed, the display control processing proceeds to step ST60.

[0145] In step ST60, the positional relationship identification unit 104D determines whether the multiplicity 124 calculated in step ST58 is equal to or greater than a default multiplicity. If the multiplicity 124 is less than the default multiplicity in step ST60, the determination is negative, and the display control process proceeds to step ST22 shown in Fig. 25A. If the multiplicity 124 is equal to or greater than the default multiplicity in step ST60, the determination is positive, and the display control process proceeds to step ST62.

[0146] In step ST62, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and also displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasound image 116 in the third display mode. That is, the control unit 104E highlights the lesion area 116B in the ultrasound image 116 (see FIG. 20 ). After the processing of step ST62 is executed, the display control processing proceeds to step ST26 shown in FIG. 25A.

[0147] In step ST64, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. After the processing of step ST64 is executed, the display control processing proceeds to step ST26 shown in FIG.

[0148] As described above, in the fourth modified example, the result of detection of part region 116A and lesion region 116B from ultrasound image 116 by detection unit 104B is displayed on first screen 22 in a display mode according to first certainty factor 118C, second certainty factor 120C, and the positional relationship (e.g., overlapping factor 124) between part region 116A and lesion region 116B. Therefore, it is possible to prevent a situation in which a user or the like recognizes a part and a lesion that are not closely related to each other. For example, compared to when the display mode of ultrasound image 116 is determined without any consideration of first certainty factor 118C and second certainty factor 120C, it is possible to prevent a user or the like from recognizing a part and a lesion that are not closely related to each other.

[0149] This can prevent the occurrence of such a situation.

[0150] Furthermore, in the fourth modification, the results of detection of site region 116A and lesion region 116B by detector 104B from ultrasound image 116 are displayed on first screen 22 in a display mode that corresponds to the magnitude relationship between first certainty factor 118C and second certainty factor 120C and the positional relationship between site region 116A and lesion region 116B (e.g., overlap 124). This prevents a user from recognizing a site and a lesion that are less related to each other. For example, compared to a case in which the display mode of ultrasound image 116 is determined without any consideration of the magnitude relationship between first certainty factor 118C and second certainty factor 120C and the positional relationship between site region 116A and lesion region 116B, this prevents a user from recognizing a site and a lesion that are less related to each other. Furthermore, the magnitude relationship between first certainty factor 118C and second certainty factor 120C can be perceived by a user through the display mode of first screen 22. Furthermore, since the object to be compared with second certainty degree 120C is first certainty degree 118C, there is no need to prepare a threshold value in advance for comparison with second certainty degree 120C.

[0151] In the fourth modification, the display mode is determined according to the magnitude relationship between first certainty level 118C and second certainty level 120C. However, the present invention is not limited to this. The display mode may be determined according to whether second certainty level 120C is equal to or greater than a predetermined certainty level (e.g., 0.7). The predetermined certainty level may be a fixed value or a variable value that changes according to instructions received by reception device 62 and / or various conditions. If second certainty level 120C is equal to or greater than the predetermined certainty level, lesion area 116B is displayed more emphasized than when second certainty level 120C is less than the predetermined certainty level. Furthermore, the display intensity of lesion area 116B may be determined according to the magnitude of second certainty level 120C. For example, the display intensity of lesion area 116B may be increased as second certainty level 120C increases.

[0152] Furthermore, when the display intensity of lesion region 116B is increased according to overlapping degree 124, the display mode (e.g., the color applied to the outline of site region 116A and / or the outline of lesion region 116B) may be used to distinguish whether the display intensity is determined according to the magnitude of second certainty degree 120C or according to overlapping degree 124. Note that the display intensity of site region 116A may also be determined using a similar method using first certainty degree 128C.

[0153] [Fifth Modification] In the above embodiment, an example was described in which the display mode of the ultrasound image 116 was determined based on the positional relationship between one site region 116A and the lesion region 116B, but the technology of the present disclosure is not limited to this. For example, the display mode of the ultrasound image 116 may be determined based on multiple positional relationships. Here, multiple positional relationships refer to the positional relationships between multiple site regions and the lesion region 116B for multiple types of sites.

[0154] 26 , detector 104B detects part regions 116A and 116C and lesion region 116B from ultrasound image 116 in a manner similar to that of the above embodiment. The part indicated by part region 116A and the part indicated by part region 116C are different types of parts. For example, the part indicated by part region 116A is the pancreas, and the part indicated by part region 116C is the duodenum.

[0155] Detection unit 104B generates lesion area information 120 in the same manner as in the above embodiment. Detection unit 104B also generates part area information 118 for each of the multiple parts. In the example shown in Fig. 26, detection unit 104B generates part area information 118 for part area 116A and part area information 118 for part area 116C.

[0156] Determination unit 104C refers to consistency determination table 122 to determine the consistency between part region 116A and lesion region 116B, and the consistency between part region 116C and lesion region 116B. In ultrasound endoscopic system 10 according to the fifth modification, display control processing is performed based on the plurality of part region information 118 generated by detection unit 104B, lesion region information 120 generated by detection unit 104B, and the determination result by determination unit 104C.

[0157] Next, an example of the flow of the display control processing performed by the processor 104 of the display control device 60 when an instruction to start executing the display control processing related to this fifth variant is accepted by the accepting device 62 will be described with reference to Figures 27A and 27B.

[0158] The flowcharts shown in Figures 27A and 27B differ from the flowcharts shown in Figures 25A and 25B in that step ST80 is applied instead of step ST12, step ST82 is inserted between step ST80 and step ST50, and steps ST84 and ST86 are inserted between step ST22 and step ST26. Note that the same steps as those in the flowcharts shown in Figures 25A and 25B are assigned the same step numbers and their explanations will be omitted.

[0159] 27A , in step ST80, detection unit 104B performs AI-based image recognition processing to detect multiple part regions (here, as an example, part regions 116A and 116C) and lesion region 116B from ultrasound image 116. After the processing of step ST80 is executed, the display control processing proceeds to step ST82.

[0160] In step ST82, detection unit 104B acquires one body part area that has not been used in the processing from step ST50 onward from the multiple body part areas detected in step ST80. After the processing of step ST82 is executed, the display control processing proceeds to step ST50. From step ST50 onward, processing is executed using the one body part area acquired in step ST82 or step ST86 shown in FIG. 27B.

[0161] 27B, the control unit 104E determines whether or not the processing from step ST50 onwards has been performed on all of the part regions detected in step ST80. If the processing from step ST50 onwards has not been performed on all of the part regions detected in step ST80 in step ST84, the determination is negative, and the display control processing proceeds to step ST86. If the processing from step ST50 onwards has been performed on all of the part regions detected in step ST80 in step ST84, the determination is positive, and the display control processing proceeds to step ST26.

[0162] In step ST86, detection unit 104B acquires one part area that has not yet been used in the processing from step ST50 onwards from the multiple part areas detected in step ST80. After the processing of step ST86 is executed, the display control processing proceeds to step ST50 shown in FIG. 27A.

[0163] In this way, by performing the display control processing shown in FIGS. 27A and 27B, ultrasound image 116 is displayed in a display mode determined according to the positional relationship between each of the multiple site regions and lesion region 116B, etc.

[0164] For example, if it is determined that both overlapping degree 124 between site region 116A and lesion region 116B and overlapping degree 124 between site region 116C and lesion region 116B are less than the predetermined overlapping degree (step ST20: N), ultrasound image 116 is displayed in the first display mode, as shown in FIG. 28 . In the conventional example, site regions 116A and 116C are displayed together with lesion region 116B, but it is unclear which of site regions 116A and 116C lesion region 116B is associated with. In contrast, in the fifth modified example, by performing the processing of step ST22 shown in FIG. 27A , site regions 116A and 116C are hidden and lesion region 116B is displayed, thereby preventing a user from misidentifying a site region that is less associated with lesion region 116B as a site region that is associated with lesion region 116B.

[0165] Furthermore, if overlap 124 between site region 116A and lesion region 116B is determined to be equal to or greater than the default overlap, and overlap 124 between site region 116C and lesion region 116B is determined to be less than the default overlap, site region 116A and lesion region 116B are displayed in the second display mode, and site region 116C and lesion region 116B are displayed in the first display mode, as shown in FIG. 28 . In the conventional example, site regions 116A and 116C are displayed together with lesion region 116B, but it is unclear which of site regions 116A and 116C lesion region 116B is associated with. In contrast, in the fifth modified example, by executing the processing of step ST22 shown in FIG. 27A , site region 116C is hidden and lesion region 116B is displayed, thereby preventing a user from misidentifying site region 116C, which is less associated with lesion region 116B, as a site region associated with lesion region 116B. Furthermore, by executing the processing of step ST24 shown in FIG. 27A, the part area 116A and the lesion area 116B are displayed in a manner that allows them to be compared and distinguished, so that the user or the like can recognize that the part area that is highly related to the lesion area 116B is the part area 116A.

[0166] Note that a contrastable and distinguishable display refers to, for example, a display mode in which the distinctiveness of site region 116A and lesion region 116B is emphasized. The emphasis on distinctiveness is achieved, for example, by the color difference and / or luminance difference between site region 116A and lesion region 116B. Here, color difference refers, for example, to a complementary color relationship on a color wheel. Regarding luminance difference, for example, if lesion region 116B is expressed within a luminance range of "150 to 255 gradations," site region 116A may be expressed within a luminance range of "0 to 50 gradations." The emphasis on distinctiveness can also be achieved, for example, by differentiating the display mode of a frame (e.g., a circumscribing frame or outer contour) that surrounds site region 116A so that the position can be identified from the display mode of a frame (e.g., a circumscribing frame or outer contour) that surrounds lesion region 116B so that the position can be identified.

[0167] [Sixth Modification] In the fifth modification, when overlap 124 between site region 116C and lesion region 116B is determined to be less than a predetermined overlap, site region 116C is hidden. However, the technology of the present disclosure is not limited to this. For example, the display mode for each of multiple site regions may differ depending on each of multiple positional relationships. Here, multiple positional relationships refer to the positional relationships between multiple site regions and lesion region 116B for multiple types of sites.

[0168] For example, as shown in Figure 29, when overlap 124 between site region 116A and lesion region 116B is equal to or greater than the default overlap, site region 116A and lesion region 116B are displayed in the second display mode. When overlap 124 between site region 116C and lesion region 116B is less than the default overlap, site region 116C is displayed provided that overlap 124 is "0." Furthermore, even if overlap 124 between site region 116C and lesion region 116B is less than the default overlap, if overlap 124 is greater than "0," site region 116C is hidden, as in the example shown in the lower part of Figure 28.

[0169] This makes it possible to prevent the user or the like from mistakenly recognizing site area 116C, which has a low correlation with lesion area 116B, as a site area that is correlated with lesion area 116B, and also makes it possible for the user or the like to recognize that the site area that is highly correlated with lesion area 116B is site area 116A. Furthermore, when site area 116C is located in a position where it is unlikely to be mistakenly recognized by the user or the like as a site area that is correlated with lesion area 116B, site area 116C is displayed, making it possible for the user or the like to grasp the positional relationship between site area 116A and site area 116C, and the positional relationship between site area 116C and lesion area 116B.

[0170] In addition, provided that the overlap 124 between the part area 116C and the lesion area 116B is "0", the display intensity of the part area 116C (for example, the brightness of the outline and / or the thickness of the outline of the part area 116C) may be increased as the distance between the part area 116C and the lesion area 116B increases.

[0171] The display mode for each of the multiple part regions may also differ depending on the positional relationship between the multiple part regions. For example, if part region 116A and part region 116C overlap with lesion region 116B at a predetermined overlapping level or higher, but overlap at a lower overlapping level, part region 116C may be hidden. Alternatively, if part region 116A and part region 116C overlap with lesion region 116B at a predetermined overlapping level or higher but do not overlap, part region 116C may be displayed. Furthermore, if part region 116A and part region 116C overlap with lesion region 116B at a predetermined overlapping level or higher but do not overlap, the display intensity of part region 116C may be increased as the distance between part region 116A and part region 116C increases.

[0172] This makes it possible to prevent the user or the like from mistakenly recognizing site area 116C, which has a low relevance to lesion area 116B, as a site area that is relevant to lesion area 116B. It also makes it possible for the user or the like to understand the positional relationship between site area 116A and site area 116C.

[0173] 27A and 27B , an example has been described in which overlap 124 with lesion area 116B is calculated for each of a plurality of part areas, and all part areas are displayed depending on the calculated overlap 124. However, the technology of the present disclosure is not limited to this. For example, overlap 124 between each of a plurality of part areas and lesion area 116B may be calculated, and only the part area associated with the greatest overlap of the calculated overlaps 124 may be displayed.

[0174] For example, in this case, the display control process shown in Figures 30A and 30B is executed by the processor 104. The flowcharts shown in Figures 30A and 30B differ from the flowcharts shown in Figures 27A and 27B in that steps ST100 to ST106 are applied instead of step ST20. Note that the same steps as those in the flowcharts shown in Figures 27A and 27B are given the same step numbers, and their explanations will be omitted.

[0175] 27A , in step ST100, positional relationship identification unit 104D associates overlapping degrees 124 calculated in step ST18 with part area information 118 generated in step ST50 and stores them in RAM 106. After the process of step ST100, overlapping degrees 124 and part area information 118 for each of the multiple part areas (e.g., part areas 116A and 116C) detected in step ST80 are associated and stored in RAM 106. That is, multiple overlapping degrees 124 and multiple part area information 118 are stored in one-to-one correspondence in RAM 106. After the process of step ST100 is executed, the display control process proceeds to step ST102.

[0176] In step ST102, positional relationship identification unit 104D determines whether the processing from step ST50 onwards has been performed on all of the part regions detected in step ST80. If the processing from step ST50 onwards has not yet been performed on all of the part regions detected in step ST80 in step ST102, the determination is negative, and the display control processing proceeds to step ST86. If the processing from step ST50 onwards has been performed on all of the part regions detected in step ST80 in step ST102, the determination is positive, and the display control processing proceeds to step ST104 shown in FIG. 27B.

[0177] 27B , positional relationship identification section 104D acquires, from RAM 106, the maximum overlapping degree 124, which is the largest of the multiple overlapping degrees 124 stored in RAM 106, and part region information 118 associated with the maximum overlapping degree. After the processing of step ST104 is executed, the display control processing proceeds to step ST106.

[0178] In step ST106, positional relationship identification unit 104D determines whether the maximum overlap acquired in step ST104 is equal to or greater than the default overlap. If the maximum overlap is less than the default overlap in step ST106, the determination is negative, and the display control process proceeds to step ST22. Then, from step ST22 onwards, processing is performed using site area information 118 acquired in step ST104 and lesion area information 120 generated in step ST50. On the other hand, if the maximum overlap is equal to or greater than the default overlap in step ST106, the determination is positive, and the display control process proceeds to step ST24. Then, from step ST24 onwards, processing is performed using site area information 118 acquired in step ST104 and lesion area information 120 generated in step ST50.

[0179] 30A and 30B, ultrasound image 116 is displayed in a display mode that corresponds to the positional relationship between lesion area 116B and a maximum region region that is the region region that has the greatest degree of overlap with lesion area 116B among the multiple region regions. Therefore, even if multiple region regions are detected, the user can recognize region regions and lesion area 116B that are highly related to each other.

[0180] [Eighth Modification] In the seventh modification, an example was described in which ultrasound image 116 is displayed in a display mode that corresponds to the positional relationship between the largest part of the plurality of part regions and lesion region 116B, but the technology of the present disclosure is not limited to this. For example, ultrasound image 116 may be displayed in a display mode that corresponds to the positional relationship between lesion region 116B and a part region identified using part region information 118 that includes the largest first certainty factor 118C among multiple first certainty factors 118C acquired from multiple part region information 118 and lesion region information 120.

[0181] For example, in this case, the display control process shown in Figures 31A and 31B is executed by the processor 104. The flowcharts shown in Figures 31A and 31B differ from the flowcharts shown in Figures 27A and 27B in that steps ST110 to ST114 are applied instead of steps ST82 and ST50. Note that the same steps as those in the flowcharts shown in Figures 27A and 27B are given the same step numbers, and descriptions thereof will be omitted.

[0182] 31A , in step ST110, detection unit 104B generates a plurality of pieces of part area information 118 relating to the plurality of part areas (e.g., part areas 116A and 116C) detected in step ST80. After the processing of step ST110 is executed, the display control processing proceeds to step ST112.

[0183] In step ST112, detection unit 104B compares the multiple first certainty factors 118C included in the multiple pieces of part region information 118 generated in step ST110, and identifies, from the multiple pieces of part region information 118 generated in step ST110, part region information 118 that includes the largest first certainty factor 118C. Then, in step ST52 and subsequent steps, processing is performed using part region information 118 identified in step ST112. After the processing of step ST112 is performed, the display control processing proceeds to step ST114.

[0184] In step ST114, detection section 104B generates lesion area information 120 regarding lesion area 116B detected in step ST80. From step ST52 onwards, processing is performed using lesion area information 120 generated in step ST114.

[0185] After the process of step ST22 shown in Fig. 31A is executed, the display control process proceeds to step ST26 shown in Fig. 31B. If the determination in step ST26 shown in Fig. 31B is negative, the process proceeds to step ST10 shown in Fig. 31A. If the determination in step ST26 is positive, the display control process ends.

[0186] 31A and 31B is performed, ultrasound image 116 is displayed in a display mode that corresponds to the positional relationship between lesion area 116B and the part area identified using part area information 118 that includes the largest first certainty factor 118C among multiple first certainty factors 118C acquired from multiple part area information 118 and lesion area information 120. Therefore, even if multiple part areas are detected, the user or the like can grasp part areas and lesion area 116B that are highly related to each other.

[0187] [Ninth Modification] In the above embodiment, an example has been described in which the display control process is executed frame by frame on the plurality of ultrasound images 116 included in the ultrasound video 26, thereby displaying the ultrasound images 116 in a display mode determined for each frame. In this case, as shown in FIG. 32 as an example, when the display control process is executed in chronological order on the plurality of ultrasound images 116, the display mode of the ultrasound images 116 may differ depending on whether the part region 116A and the lesion region 116B are aligned or not. For example, when the part region 116A and the lesion region 116B are not aligned, the ultrasound image 116 may be displayed in a first display mode, and when the part region 116A and the lesion region 116B are aligned, the ultrasound image 116 may be displayed in a second display mode.

[0188] Here, when ultrasound image 116 displayed in the first display mode and several frames of ultrasound image 116 adjacent to each other in the time series are displayed in the second display mode, it is highly likely that ultrasound image 116 displayed in the first display mode is ultrasound image 116 that would normally be displayed in the second display mode. In other words, it is highly likely that ultrasound image 116 was displayed in the second display mode because determination unit 104C determined that part area 116A and lesion area 116B are consistent, even though they are not.

[0189] Therefore, in the ultrasound endoscopic system 10 according to the ninth modification, the control unit 104E corrects the display mode of the ultrasound image 116 that may have been erroneously determined by the determination unit 104C based on the display mode of the ultrasound image 116 that has been correctly determined by the determination unit 104C. The display mode of the ultrasound image 116 that may have been erroneously determined refers to the display mode corresponding to the ultrasound image 116 used as the determination target when the determination unit 104C determines that the combination of the part region 116A and the lesion region 116B is incorrect (i.e., inconsistent). Furthermore, the display mode of the ultrasound image 116 that has been correctly determined refers to the display mode corresponding to the ultrasound image 116 that has been used as the determination target when the determination unit 104C determines that the combination of the part region 116A and the lesion region 116B is correct (i.e., consistent) (i.e., the display mode determined in the same manner as in the above embodiment).

[0190] In the example shown in Fig. 32, the control unit 104E determines the display mode for each of the multiple ultrasound images 116 in the manner described in the above embodiment, and stores the multiple ultrasound images 116 in chronological order in the order in which the display modes were determined. The control unit 104E stores the multiple ultrasound images 116 in a FIFO manner. That is, the control unit 104E outputs the oldest frame to the display device 14 every time a new frame is added. In the example shown in Fig. 32, for convenience of illustration, the first to seventh frames of the ultrasound images 116 are stored in chronological order by the control unit 104E.

[0191] The first to third and fifth to seventh frames of ultrasound image 116 are determined by determination unit 104C to have a correct combination of part region 116A and lesion region 116B, and are therefore to be displayed in the second display mode. The fourth frame of ultrasound image 116 is determined by determination unit 104C to have an incorrect combination of part region 116A and lesion region 116B, and are therefore to be displayed in the first display mode.

[0192] Here, the combination of part region 116A and lesion region 116B in each of the three ultrasound images 116 frames preceding and following in the chronological order with respect to ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be incorrect (i.e., the fourth ultrasound image 116), is determined to be correct, but this is merely an example. Four or more ultrasound images 116 in which the combination of part region 116A and lesion region 116B is determined to be correct may be adjacent in the chronological order to ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be incorrect.

[0193] Furthermore, although ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be incorrect is one frame here, this is merely an example. For example, the number of frames of ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be incorrect may be sufficiently smaller than the number of frames of ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be correct. For example, a sufficiently small number of frames refers to a number of frames that is approximately one-fifth to one-hundredth of the number of frames of ultrasound image 116 in which the combination of part region 116A and lesion region 116B is determined to be correct. The sufficiently small number of frames may be a fixed value or a variable value that is changed according to instructions and / or various conditions received by reception device 62.

[0194] In the example shown in Fig. 32 , the control unit 104E corrects the display mode of the fourth frame of the ultrasound image 116 by referring to the display modes determined for the first to third and fifth to seventh frames of the ultrasound images 116 among the multiple ultrasound images 116 stored in chronological order. In the example shown in Fig. 32 , the second display mode has been determined for the first to third and fifth to seventh frames of the ultrasound images 116, so the first display mode determined for the fourth frame of the ultrasound image 116 is corrected to the second display mode. As a result, the display modes of all ultrasound images 116 stored in chronological order are aligned to the second display mode. The control unit 104E adjusts the display modes of the first to seventh frames of the ultrasound images 116 to the second display mode and outputs the ultrasound images 116 to the display device 14 in chronological order, thereby displaying the ultrasound images 116 on the first screen 22. This makes it possible to prevent a user or the like from mistakenly recognizing an incorrect combination of site region 116A and lesion region 116B as a correct combination of site region 116A and lesion region 116B.

[0195] [Tenth Modification] In this tenth modification, a display control process having a different algorithm from the display control process described in the eighth modification (see Figures 31A and 31B) will be described with reference to Figures 33A and 33B. The flowcharts shown in Figures 33A and 33B differ from the flowcharts shown in Figures 31A and 31B in that processes from steps ST146 to ST170 are applied instead of processes from steps ST80 to ST64. In this tenth modification, only processes different from the flowcharts shown in Figures 31A and 31B will be described.

[0196] 33A, the detection unit 104B performs AI image recognition processing to detect multiple site regions (e.g., pancreas and kidney) and multiple lesion regions 116B (e.g., pancreatic cancer and kidney cancer) from the ultrasound image 116. After the processing of step ST146 is executed, the display control processing proceeds to step ST148.

[0197] In step ST148, detection unit 104B generates a plurality of pieces of site area information 118 corresponding to the plurality of site areas detected in step ST80. Detection unit 104B also generates a plurality of pieces of lesion area information 120 corresponding to the plurality of lesion areas 116B detected in step ST80. After the processing of step ST148 is executed, the display control processing proceeds to step ST149.

[0198] In step ST149, positional relationship specifying unit 104D selects a processing target lesion area, which is one lesion area 116B that has not been processed from step ST150 onwards, from among the plurality of lesion areas 116B detected in step ST146. After the processing of step ST149 is executed, the display control processing proceeds to step ST150.

[0199] In step ST150, positional relationship identification unit 104D acquires coordinate information 118A from each of the multiple pieces of site region information 118 generated in step ST110. Furthermore, positional relationship identification unit 104D acquires coordinate information 120A from lesion region information 120 corresponding to the target lesion region selected in step ST149 from the multiple pieces of lesion region information 120 generated in step ST148. Positional relationship identification unit 104D then calculates overlapping degrees 124 for each site region and target lesion region detected in step ST80. For example, if the multiple site regions are the pancreas and kidney, overlapping degrees 124 for the pancreas and the target lesion region, and overlapping degrees 124 for the kidney and the target lesion region are calculated. Thus, multiple overlapping degrees 124 are calculated in step ST150. After the processing of step ST150 is executed, the display control processing proceeds to step ST152.

[0200] In step ST152, the positional relationship identification unit 104D determines whether or not the maximum multiplicity 124 exists among the multiple multiplicities 124 calculated in step ST150. If the maximum multiplicity 124 does not exist among the multiple multiplicities 124 in step ST152, the determination is negative, and the display control process proceeds to step ST154 shown in FIG. 33B. If the maximum multiplicity 124 exists among the multiple multiplicities 124 in step ST152, the determination is positive, and the display control process proceeds to step ST156.

[0201] Here, the presence or absence of the maximum degree of overlap 124 is determined, but the disclosed technology is not limited to this, and it may also be determined whether or not there is a degree of overlap 124 that is equal to or greater than a certain reference value.

[0202] 33B, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. After the processing of step ST154 is executed, the display control processing proceeds to step ST170.

[0203] 33A , the positional relationship identification unit 104D acquires the maximum overlapping degree 124, which is the largest of the multiple overlapping degrees 124 calculated in step ST150, and the part region information 118 associated with the maximum overlapping degree. After the processing of step ST156 is executed, the display control processing proceeds to step ST158.

[0204] In step ST158, the positional relationship identification unit 104D determines whether the maximum overlapping degree acquired in step ST156 is equal to or greater than the default overlapping degree. If the maximum overlapping degree is less than the default overlapping degree in step ST158, the determination is negative, and the display control process proceeds to step ST154 shown in Fig. 33B. If the maximum overlapping degree is equal to or greater than the default overlapping degree in step ST106, the determination is positive, and the display control process proceeds to step ST160.

[0205] In step ST160, determination unit 104C determines whether the processing target site region and the processing target lesion region are consistent, in a manner similar to the processing in step ST16 shown in FIG. 14 . Here, the processing target site region refers to the site region corresponding to site region information 118 acquired in step ST156 (i.e., the site region identified from site region information 118 acquired in step ST156). The determination of whether the processing target site region and the processing target lesion region are consistent is made using site region information 118 acquired in step ST156 and lesion region information 120 corresponding to the processing target lesion region selected in step ST149. Lesion region information 120 corresponding to the processing target lesion region selected in step ST149 refers to the lesion region information 120 corresponding to the processing target lesion region selected in step ST149 from the plurality of lesion region information 120 generated in step ST148.

[0206] In step ST160, if the processing target region and the processing target lesion region match, the determination is negative and the process proceeds to step ST154 shown in Fig. 33B. In step ST160, if the processing target region and the processing target lesion region do not match, the determination is positive and the process proceeds to step ST162.

[0207] In step ST162, positional relationship identification unit 104D acquires first certainty factor 118C from site region information 118 acquired in step ST156. Positional relationship identification unit 104D also acquires lesion region information 120 corresponding to the processing target lesion region selected in step ST149 from the plurality of lesion region information 120 generated in step ST148, and acquires second certainty factor 120C from the acquired changed region information 120. Positional relationship identification unit 104D then determines whether second certainty factor 120C is greater than first certainty factor 118C. In step ST162, if second certainty factor 120C is equal to or less than first certainty factor 118C, the determination is negative, and the process proceeds to step ST164 shown in FIG. 33B. In step ST162, if second certainty factor 120C is greater than first certainty factor 118C, the determination is positive, and the display control process proceeds to step ST168 shown in FIG. 33B.

[0208] Although it is determined here whether the magnitude relationship "first confidence level 118C > second confidence level 120C" holds, the technology of the present disclosure is not limited to this. For example, it may be determined whether the difference between first confidence level 118C and second confidence level 120C exceeds a threshold value.

[0209] Furthermore, the conditions for comparing first confidence level 118C and second confidence level 120C may be changed depending on the type of treatment target region. For example, different thresholds may be set depending on the type of treatment target region, and first confidence level 118C and second confidence level 120C that exceed the thresholds may be compared.

[0210] 33B, the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the processing target site region within the ultrasound image 116, and does not display the processing target lesion region within the ultrasound image 116. For example, if the processing target lesion region is a region indicative of pancreatic cancer and the processing target site region is a region indicative of a kidney, the region indicative of the kidney is displayed, and the region indicative of pancreatic cancer is not displayed.

[0211] In this specification, the concept of non-display includes not only a state in which the display is not displayed at all, but also a state in which the display intensity (e.g., brightness and / or shading) is reduced to a perceptual level that will not cause a misidentification by the doctor 16 (e.g., a perceptual level known in advance by a sensory test using an actual device and / or a computer simulation). After the processing of step ST164 is executed, the display control processing proceeds to step ST170.

[0212] In step ST168 shown in FIG. 33B , the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24. Here, the control unit 104E displays the ultrasound image 116 displayed on the first screen 22 in the first display mode. That is, the control unit 104E displays the processing target lesion region in the ultrasound image 116, and hides the processing target site region in the ultrasound image 116. For example, if the processing target lesion region is a region indicating pancreatic cancer and the processing target site region is a region indicating a kidney, the region indicating pancreatic cancer is displayed, and the region indicating the kidney is hidden. After the processing of step ST168 is executed, the display control processing proceeds to step ST170.

[0213] In addition, in steps ST154, ST162, and / or ST168, whether the processing target area is ultimately displayed or not may be determined depending on the type of processing target lesion area and / or the type of processing target area.

[0214] Furthermore, it is preferable to always hide part regions that represent specific organs (for example, kidneys 66 in a scene where pancreas 65 is examined). However, even if part regions that represent specific organs are always hidden, the part regions that represent specific organs are used in processes related to overlapping degree 124 and processes related to determining the magnitude relationship between first certainty factor 118C and second certainty factor 120C (i.e., used in the processes of steps ST149 to ST162).

[0215] In step ST170, positional relationship specifying unit 104D determines whether all of multiple lesion areas 116B detected in step ST146 have been used in the processing from step ST150 onwards. If in step ST170, all of multiple lesion areas 116B detected in step ST146 have not been used in the processing from step ST150 onwards, the determination is negative, and the display control processing proceeds to step ST149 shown in Figure 33A. If in step ST170, all of multiple lesion areas 116B detected in step ST146 have been used in the processing from step ST150 onwards, the determination is positive, and the display control processing proceeds to step ST26.

[0216] As described above, in the tenth modified example, the certainty of the processing target lesion region is determined based on the positional relationship between the processing target lesion region and the processing target site region, and the relationship between first certainty factor 118C and second certainty factor 120C. That is, the certainty of the processing target lesion region is determined by performing the processes of steps ST149 to ST162 shown in FIG. 33A . The determination results are then displayed on first screen 22 (see steps ST164 and ST168). This makes it possible to prevent the display of a processing target site region that is incorrectly combined with the processing target lesion region, or the display of a processing target lesion region that is incorrectly combined with the processing target site region. As a result, it is possible to prevent erroneous identification by the user, etc.

[0217] Furthermore, in the tenth modified example, if the positional relationship between the processing target lesion area and the processing target site area is a predetermined positional relationship (e.g., step ST152: Y), the processing target site area and the processing target lesion area are inconsistent (e.g., step ST160: N), and the relationship between first certainty factor 118C and second certainty factor 120C is a predetermined certainty factor relationship (e.g., step ST162: Y), the processing target lesion area is determined to be certain. That is, as a result of the processing of steps ST156 to ST164 shown in FIG. 33A, if the determination in step ST162 is positive, the processing target lesion area is determined to be certain. Then, if the processing target lesion area is determined to be certain, the determination result is displayed on first screen 22 (see step ST168). Therefore, the processing target lesion area determined to be certain is displayed. As a result, the user or the like can grasp the processing target lesion area determined to be certain.

[0218] Furthermore, in the tenth modified example, the likelihood of the processing target region is determined. That is, the likelihood of the processing target region is determined by performing the processes of steps ST149 to ST162 shown in FIG. 33A. The determination result is then displayed on the first screen 22 (see steps ST164 and ST168). This makes it possible to prevent the display of a processing target region that is incorrectly combined with the processing target lesion region, or the display of a processing target lesion region that is incorrectly combined with the processing target region. As a result, it is possible to prevent erroneous identification by the user, etc.

[0219] If the target lesion area is determined to be accurate, information indicating that a lesion has been detected may be displayed on display 14. In this case, for example, ultrasound image 116 may be displayed on first screen 22 in the third display mode (see FIG. 20 ). This allows the user or the like to visually recognize the location of the target lesion area on first screen 22.

[0220] [Other Modifications] In the above embodiment, the display modes of site region 116A and lesion region 116B are determined for each frame, but the technology of the present disclosure is not limited to this. For example, the display modes of site region 116A and lesion region 116B may be determined every predetermined number of frames (e.g., every few frames or every few tens of frames). In this case, the number of times the display control process is performed is reduced, thereby reducing the load on processor 104 compared to when the display control process is performed every frame. Furthermore, when the display modes of site region 116A and lesion region 116B are determined every predetermined number of frames, the display modes of site region 116A and lesion region 116B may be determined at a frame interval at which the display modes (e.g., first to third display modes) are visually perceived due to the persistence of vision phenomenon.

[0221] In the above embodiment, an example in which the outline of site region 116A and the outline of lesion region 116B are displayed has been described, but the technology of the present disclosure is not limited thereto. For example, the positions of site region 116A and lesion region 116B may be displayed so as to be identifiable using bounding boxes used in AI-based image recognition processing. In this case, control unit 104E may cause display device 14 to display the bounding boxes identifying site region 116A and the bounding boxes identifying lesion region 116B in the same display format as that applied to site region 116A and lesion region 116B. For example, control unit 104E may switch between displaying and hiding the bounding box identifying site region 116A, change the display format of the outline of the bounding box identifying site region 116A, and change the display format of the outline of the bounding box identifying lesion region 116B under the same conditions as in the above embodiment.

[0222] Furthermore, positional relationship identification unit 104D may calculate overlap 124 and / or distance 126 using a bounding box identifying site region 116A and a bounding box identifying lesion region 116B. An example of overlap 124 in this case is the IoU using the bounding box identifying site region 116A and the bounding box identifying lesion region 116B. In this case, IoU refers to the ratio of the area of ​​overlap between the bounding box identifying site region 116A and the bounding box identifying lesion region 116B to the area of ​​the combined region of the bounding box identifying site region 116A and the bounding box identifying lesion region 116B. Furthermore, overlap 124 may be the ratio of the area of ​​overlap between the bounding box identifying site region 116A and the bounding box identifying lesion region 116B to the total area of ​​the bounding box identifying lesion region 116B. An example of distance 126 in this case is the distance between a portion of the outline of a region of the bounding box that identifies lesion region 116B that does not overlap with the bounding box that identifies site region 116A and the bounding box that identifies site region 116A. For example, the portion of the outline of the region that does not overlap with site region 116A refers to the position of the outline of the region that does not overlap with site region 116A that is farthest from the bounding box that identifies site region 116A.

[0223] In the above embodiment, an example was given of a first display mode in which the part areas 116A and 116C are not displayed, but this is merely one example, and the part areas 116A and / or 116C may not be displayed, and the display intensity of the part areas 116A and / or 116C may be made lower than the display intensity of the lesion area 116B.

[0224] In the above embodiment, an example was described in which ultrasound image 116 is displayed in one of the first to third display modes when second certainty factor 120C is greater than first certainty factor 118C, but the technology of the present disclosure is not limited to this. For example, when first certainty factor 118C is greater than second certainty factor 120C, site region 116A or 116C and lesion region 116B may be displayed in a comparable manner on the condition that overlap factor 124 is less than a default overlap factor or on the condition that overlap factor 124 is "0." Alternatively, site region 116A or 116C and lesion region 116B may be displayed in the second display mode.

[0225] In the above embodiment, an example was given in which the intensity of the contour of the part area 116A is set to an intensity according to the overlapping degree 124, but the technology of the present disclosure is not limited to this, and when the overlapping degree 124 is equal to or greater than a default overlapping degree, the display intensity of both the part area 116A and the lesion area 116B may be increased as the overlapping degree 124 increases.

[0226] In the above embodiment, an example in which the ultrasound image 116 is displayed on the first screen 22 has been described, but this is merely an example. For example, the ultrasound image 116 may be displayed on the entire screen of the display device 14.

[0227] In the above embodiment, the processor 104 directly influences the display device 14 to display the ultrasound image 116 on the display device 14, but this is merely an example. For example, the processor 104 may indirectly influence the display device 14 to display the ultrasound image 116 on the display device 14. In this case, for example, screen information indicating a screen to be displayed on the display device 14 (e.g., at least the first screen 22 of the first screen 22 and the second screen 24) is temporarily stored in external storage (not shown). Then, the processor 104 or a processor other than the processor 104 acquires the screen information from the external storage, and based on the acquired screen information, at least the first screen 22 of the first screen 22 and the second screen 24 is displayed on the display device 14 or a display device other than the display device 14. As a specific example of this case, the processor 104 may use cloud computing to display at least the first screen 22 of the first screen 22 and the second screen 24 on the display device 14 or a display device other than the display device 14.

[0228] Examples include:

[0229] In the above embodiment, an example in which the display control process is performed on the ultrasonic moving image 26 has been described, but the display control process may be performed on an ultrasonic still image.

[0230] In the above embodiment, an example in which display control processing is performed on the ultrasound image 116 acquired by the ultrasound endoscopic device 12 has been described, but the technology of the present disclosure is not limited to this. For example, display control processing may be performed on ultrasound images acquired by an extracorporeal ultrasound diagnostic device using an extracorporeal ultrasound probe. Display control processing may be performed on medical images obtained by imaging the observation region of the subject 20 using various modalities such as an X-ray diagnostic device, a CT diagnostic device, and / or an MRI diagnostic device. The extracorporeal ultrasound diagnostic device, the X-ray diagnostic device, the CT diagnostic device, and / or the MRI diagnostic device are examples of the "imaging device" according to the technology of the present disclosure.

[0231] In the above embodiment, an example in which the display control processing is performed by the processor 104 of the display control device 60 has been described. However, the technology of the present disclosure is not limited to this. For example, the device performing the display control processing may be provided external to the display control device 60. Examples of devices provided external to the display control device 60 include the endoscope processing device 54 and / or the ultrasound processing device 58. Another example of a device provided external to the display control device 60 is a server. For example, the server may be implemented by cloud computing. While cloud computing is used here as an example, this is merely an example. For example, the server may be implemented by a mainframe or by network computing such as fog computing, edge computing, or grid computing. The server is merely an example, and at least one personal computer may be used instead of the server. Furthermore, the display control processing may be distributed among multiple devices, including the display control device 60 and at least one device provided external to the display control device 60.

[0232] Furthermore, in the above embodiment, an example has been described in which the display control processing program 112 is stored in the NVM 108, but the technology of the present disclosure is not limited to this. For example, the display control processing program 112 may be stored in a portable storage medium such as an SSD or a USB memory. The storage medium is a non-transitory computer-readable storage medium. The display control processing program 112 stored in the storage medium is installed in the computer 100 of the display control device 60. The processor 104 executes display control processing in accordance with the display control processing program 112.

[0233] In the above embodiment, the computer 100 is exemplified, but the technology of the present disclosure is not limited to this, and devices including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 100. Furthermore, instead of the computer 100, a combination of a hardware configuration and a software configuration may be used.

[0234] The hardware resources for executing the display control process described in the above embodiments can be various processors, as listed below. Examples of processors include general-purpose processors that function as hardware resources for executing display control processes by executing software, i.e., programs. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute the display control process.

[0235] The hardware resource that executes the display control process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a processor and an FPGA). Also, the hardware resource that executes the display control process may be a single processor.

[0236] As an example of a configuration using a single processor, first, one processor is configured by combining one or more processors with software, and this processor functions as a hardware resource that executes display control processing. Second, there is a configuration in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute display control processing, on a single IC chip, as typified by SoC. In this way, the display control processing is realized using one or more of the various processors described above as hardware resources.

[0237] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above display control process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0238] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0239] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0240] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0241] The following additional notes are provided regarding the above-described embodiments.

[0242] (Supplementary Note 1) An image processing device comprising a processor that detects a first image area indicating a part of a human body and a second image area indicating the lesion from a medical image obtained by capturing an image of an observation area including the part and the lesion, and causes a display device to display the results of detecting the first image area and the second image area in a display mode corresponding to the positional relationship between the first image area and the second image area.

[0243] (Supplementary Note 2) The image processing device described in Supplementary Note 1, wherein the processor acquires a first certainty factor that is a certainty factor for the result of detecting the first image area and a second certainty factor that is a certainty factor for the result of detecting the second image area, the display mode is determined according to the first certainty factor, the second certainty factor, and the positional relationship, and when the first certainty factor is greater than the second certainty factor and the first image area and the second image area do not overlap, the display mode is a mode in which the first image area and the second image area are displayed in a manner that allows them to be compared within the medical image.

[0244] (Supplementary Note 3) The image processing device described in Supplementary Note 1, wherein the processor determines whether the combination of the first image area and the second image area is correct based on the correspondence between the multiple types of parts and the lesions corresponding to each of the parts, and when the processor determines that the combination of the first image area and the second image area is correct, the first certainty factor is greater than the second certainty factor, and the first image area and the second image area do not overlap, the display mode is a mode in which the first image area and the second image area are displayed in a manner that allows them to be compared within the medical image.

[0245] (Supplementary Note 4) The image processing device according to Supplementary Note 2 or Supplementary Note 3, wherein a display intensity of the first image area is determined according to a distance between the first image area and the second image area.

[0246] (Supplementary Note 5) The image processing device described in Supplementary Note 1, wherein the observation target area includes a plurality of types of the site and the lesion, and the processor detects a plurality of the first image areas and the second image areas from the medical image, each of which indicates a plurality of types of the site, and when the second image area overlaps with a plurality of the first image areas in the medical image, the positional relationship is the relationship between the position of the second image area and the position of the maximum overlapping image area, which has the greatest degree of overlap with the second image area among the plurality of the first image areas.

[0247] (Supplementary Note 6) The image processing device described in Supplementary Note 1, wherein the processor obtains a first certainty level which is a certainty level for the result of detecting the first image area and a second certainty level which is a certainty level for the result of detecting the second image area, the display mode is determined according to the first certainty level, the second certainty level, and the positional relationship, the observation area includes multiple types of the site and the lesion, the processor detects multiple first image areas and second image areas which indicate multiple types of the site from the medical image, and when the second image area overlaps with multiple first image areas in the medical image, the first certainty level is the largest certainty level among multiple certainty levels for multiple results of detecting the multiple first image areas.

[0248] (Supplementary Note 7) The image processing device according to Supplementary Note 1, wherein the processor acquires a first certainty factor that is a certainty factor for a result of detecting the first image region, and the display mode is determined according to the first certainty factor and the positional relationship.

[0249] (Supplementary Note 8) The image processing device according to Supplementary Note 1, wherein the processor acquires a second certainty factor that is a certainty factor for a result of detecting the second image region, and the display mode is determined according to the second certainty factor and the positional relationship.

Claims

1. A processor is provided. The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; The display mode is determined according to the positional relationship and the consistency between the site and the lesion. Image processing device.

2. the display mode of the first image region varies depending on the site, the lesion, and the positional relationship; The display mode for the second image area is a mode in which the second image area is displayed on the display device. The image processing device according to claim 1 .

3. If the site and the lesion do not match, the display mode for the first image area is a mode in which the first image area is not displayed on the display device, The display mode for the second image area is a mode in which the second image area is displayed on the display device. The image processing device according to claim 2 .

4. If the site and the lesion match, the display mode of the first image region is a mode in which the first image region is displayed on the display device and is determined according to the positional relationship; The display mode for the second image area is a mode in which the second image area is displayed on the display device.

4. The image processing device according to claim 2 or 3.

5. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; The positional relationship is defined by the degree of overlap between the first image area and the second image area. Image processing device.

6. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; The positional relationship is defined by an overlapping degree between the first image region and the second image region, and when the overlapping degree is a first degree or more, the display manner is a manner in which the second image region is displayed so as to be identifiable within the medical image. Image processing device.

7. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; the positional relationship is defined by an overlapping degree between the first image area and the second image area, When the degree of overlap is equal to or greater than a first degree, the display manner is such that the second image region is identifiable within the medical image, and the first image region is displayed in a manner that can be contrasted with the second image region. Image processing device.

8. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; obtaining a first confidence level, which is a confidence level for a result of detecting the first image region, and a second confidence level, which is a confidence level for a result of detecting the second image region; The display mode is determined according to the first confidence level, the second confidence level, and the positional relationship. Image processing device.

9. The display mode is determined according to the magnitude relationship between the first confidence level and the second confidence level and the positional relationship. The image processing device according to claim 8.

10. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; the display mode is determined in accordance with a plurality of the positional relationships; The plurality of positional relationships are positional relationships between the first image region and the second image region for a plurality of types of the site. Image processing device.

11. The display mode for each of the plurality of first image regions differs depending on each of the plurality of positional relationships. The image processing device according to claim 10.

12. The display mode for each of the plurality of first image regions differs depending on a first image region positional relationship between the plurality of first image regions. The image processing device according to claim 10 or 11.

13. A processor comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; the medical image being an image defined by a plurality of frames; The processor, Detecting the first image area and the second image area for each frame; The display mode is determined for each frame. Image processing device.

14. The processor, determining whether a combination of the first image region and the second image region is correct for each of the frames based on a correspondence relationship between the plurality of types of the sites and a lesion corresponding to each of the sites; The display mode corresponding to the frame used as a determination target when it is determined that the combination of the first image region and the second image region is incorrect is corrected based on the display mode corresponding to the frame used as a determination target when it is determined that the combination of the first image region and the second image region is correct. The image processing device according to claim 13.

15. An image processing device according to any one of claims 1 to 14, and an imaging device for imaging the observation target area. Medical diagnostic equipment.

16. An image processing device according to any one of claims 1 to 14, and an ultrasound device for acquiring an ultrasound image as the medical image. Ultrasound endoscopic device.

17. Detecting a first image area representing a part of a human body and a second image area representing the lesion from a medical image obtained by imaging the part of the human body and an observation target area including the lesion; and displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; The display mode is determined according to the positional relationship and the consistency between the site and the lesion. Image processing methods.

18. A program for causing a computer to execute a process, comprising: The process comprises: Detecting a first image area representing a part of a human body and a second image area representing the lesion from a medical image obtained by imaging the part of the human body and an observation target area including the lesion; and displaying a result of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area; The display mode is determined according to the positional relationship and the consistency between the site and the lesion. program.

19. A processor is provided. The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; determining a likelihood of the second image area according to a positional relationship between the first image area and the second image area; The positional relationship is defined by the degree of overlap between the first image area and the second image area. Image processing device.

20. The processor determines the likelihood according to the positional relationship and a relationship between a first confidence level, which is a confidence level for a result of detecting the first image region, and a second confidence level, which is a confidence level for a result of detecting the second image region. The image processing device according to claim 19.

21. A method for controlling a computer-implemented program comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; The certainty of the second image area is determined according to a positional relationship between the first image area and the second image area, and a relationship between a first certainty factor, which is a certainty factor for a result of detecting the first image area, and a second certainty factor, which is a certainty factor for a result of detecting the second image area. Image processing device.

22. A method for controlling a computer-implemented program comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; determining the certainty of the second image area according to a positional relationship between the first image area and the second image area, and a relationship between a first certainty factor, which is a certainty factor for a result of detecting the first image area, and a second certainty factor, which is a certainty factor for a result of detecting the second image area; When the positional relationship is a predetermined positional relationship, the first image area and the second image area are not consistent, and the relationship between the first certainty degree and the second certainty degree is a predetermined certainty degree relationship, the second image area is determined to be certain. Image processing device.

23. A method for controlling a computer-implemented programmable logic device comprising: The processor, detecting a first image area representing the part and a second image area representing the lesion from a medical image obtained by imaging an observation target area including the part and the lesion of a human body; determining a likelihood of the second image area according to a positional relationship between the first image area and the second image area; If the positional relationship is a predetermined positional relationship and the first image area and the second image area match, the second image area is determined to be correct. Image processing device.

24. The processor determines a likelihood of the first image region.

23. The image processing device according to any one of claims 19 to 22.

25. The processor, Displaying the medical image on a display device; When the second image region is determined to be correct, information indicating that the lesion has been detected is displayed on the display device.

25. The image processing device according to any one of claims 19 to 24.

26. The position where the information indicating that the lesion has been detected is displayed is an area corresponding to the second image area in the display area where the medical image is displayed. The image processing device according to claim 25.

Images