Diagnosis support device, ultrasonic endoscope, diagnosis support method, and program

Abstract

The diagnosis support device includes a processor. The processor acquires an ultrasonic image, displays the acquired ultrasonic image on the display device, and displays, in the ultrasonic image, a first marker capable of specifying a lesion region detected from the ultrasonic image in the ultrasonic image and a second marker capable of specifying an organ region detected from the ultrasonic image in the ultrasonic image. The first mark is displayed in a more emphasized state than the second mark.

Description

Technical Field

[0001] The technology of the present invention relates to a diagnosis support device, an ultrasonic endoscope, a diagnosis support method, and a program. Background Art

[0002] Japanese Patent Application Publication No. 2021-185970 discloses an image processing device for processing medical images. The image processing device described in Japanese Patent Application Publication No. 2021-185970 includes: a detection unit for detecting a lesion candidate region; a validity evaluation unit for evaluating the validity of the lesion candidate region using a normal tissue region corresponding to the detected lesion candidate region; and a display unit for determining display content for a user using the evaluation result.

[0003] Japanese Patent Application Publication No. 2015-154918 discloses a lesion detection device. The lesion detection device described in Japanese Patent Application Publication No. 2015-154918 includes: a lesion candidate detector for detecting lesion candidates in a medical image; a peripheral object detector for detecting anatomical objects in the medical image; a lesion candidate verifier for verifying the lesion candidate based on anatomical context information including relationship information between the position of the lesion candidate and the position of the anatomical object; and a candidate remover for removing false positive lesion candidates from the detected lesion candidates based on the verification result of the lesion candidate verifier.

[0004] An ultrasonic diagnostic device is disclosed in Japanese Patent Publication No. 2021-180730. The ultrasonic diagnostic device described in Japanese Patent Publication No. 2021-180730 includes: a detection unit, which detects a candidate lesion part based on a frame data column obtained by transmitting and receiving ultrasonic waves; and a notification unit, which displays a mark notifying the candidate lesion part on an ultrasonic image generated by the frame data column based on the detection result of the detection unit, and changes the display mode of the mark according to the possibility that the candidate lesion part is a lesion part. In addition, the notification unit includes: a calculation unit, which calculates the reliability of the possibility that the candidate lesion part is a lesion part based on the frame data column; and a control unit, which changes the display mode of the mark according to the reliability. In addition, when the reliability is low, the control unit changes the display mode to make the mark inconspicuous compared to when the reliability is high. Summary of the invention

[0005] One embodiment of the technology according to the present invention provides a diagnosis support device, an ultrasonic endoscope, a diagnosis support method, and a program that can suppress omission of a lesion region in diagnosis using an ultrasonic image.

[0006] Means for solving technical problems

[0007] The first method involved in the technology of the present invention is a diagnosis support device, which includes a processor, and the processor performs the following processing: acquiring an ultrasonic image; causing a display device to display the acquired ultrasonic image; displaying a first mark in the ultrasonic image that can identify a lesion area detected from the ultrasonic image and a second mark in the ultrasonic image that can identify an organ area detected from the ultrasonic image, wherein the first mark is displayed in a state where it is more emphasized than the second mark.

[0008] A second aspect according to the technology of the present invention is the diagnosis support device according to the first aspect, wherein the first mark is a mark capable of identifying an outer edge of a first range where the lesion region exists.

[0009] A third aspect according to the technology of the present invention is the diagnosis support device according to the second aspect, wherein the first range is defined by a first rectangular frame surrounding the lesion region.

[0010] A fourth aspect according to the technology of the present invention is the diagnosis support device according to the third aspect, wherein the first rectangular frame is a rectangular frame circumscribing the lesion region.

[0011] A fifth aspect according to the technology of the present invention is the diagnosis support device according to the third aspect or the fourth aspect, wherein the first mark is a mark formed so that at least a portion of the first rectangular frame can be visually identified.

[0012] The sixth mode involved in the technology of the present invention is a diagnostic support device involved in any one of the third to fifth modes, wherein the first rectangular frame surrounds the lesion area in a rectangle when observed from the front, and the first mark is a plurality of first images assigned to at least a plurality of diagonal corners among the four corners of the first rectangular frame.

[0013] A seventh aspect according to the technology of the present invention is the diagnosis support device according to any one of the first to sixth aspects, wherein the second mark is a mark capable of identifying an outer edge of a second range where the organ region exists.

[0014] An eighth aspect according to the technology of the present invention is the diagnosis support device according to the seventh aspect, wherein the second range is defined by a second rectangular frame surrounding the organ region.

[0015] A ninth aspect according to the technology of the present invention is the diagnosis support device according to the eighth aspect, wherein the second rectangular frame is a rectangular frame circumscribing the organ region.

[0016] A tenth aspect according to the technology of the present invention is the diagnosis support device according to the eighth aspect or the ninth aspect, wherein the second mark is a mark formed so that at least a portion of the second rectangular frame can be visually identified.

[0017] The 11th method involved in the technology of the present invention is a diagnostic support device involved in any one of the 8th to 10th methods, wherein the second rectangular frame surrounds the organ area in a rectangle when observed from the front, and the second mark is a plurality of second images assigned to the central parts of a plurality of sides of at least the opposite sides among the four sides of the second rectangular frame.

[0018] The 12th method involved in the technology of the present invention is a diagnostic support device involved in any one of the 1st to 11th methods, wherein the ultrasonic image is a dynamic image including multiple frames, and when N is set to a natural number greater than 2, the processor performs the following processing: when a lesion area is detected in N consecutive frames from multiple frames, a first mark is displayed in the ultrasonic image.

[0019] The 13th method involved in the technology of the present invention is a diagnostic support device involved in any one of the 1st to 12th methods, wherein the ultrasonic image is a dynamic image including multiple frames, and when M is set to a natural number greater than 2, the processor performs the following processing: when an organ area is detected from M consecutive frames from multiple frames, a second mark is displayed in the ultrasonic image.

[0020] The 14th method involved in the technology of the present invention is a diagnostic support device involved in any one of the 1st to 11th methods, wherein the ultrasonic image is a dynamic image including multiple frames, and when N and M are set to natural numbers greater than 2, the processor performs the following processing: when a lesion area is detected from N consecutive frames from multiple frames, a first mark is displayed in the ultrasonic image; when an organ area is detected from M consecutive frames from multiple frames, a second mark is displayed in the ultrasonic image, and N is a value less than M.

[0021] The 15th mode involved in the technology of the present invention is a diagnostic support device involved in any one of the 1st to 14th modes, wherein the processor performs the following processing: when a lesion area is detected, the detection of the lesion area is notified by causing the sound regeneration device to output sound and / or causing the vibration generator to generate vibration.

[0022] The 16th method involved in the technology of the present invention is a diagnostic support device involved in any one of the 1st to 15th methods, wherein the processor performs the following processing: causing the display device to display multiple screens including a first screen and a second screen; displaying an ultrasonic image on the first screen and the second screen; and displaying a first mark and a second mark in the ultrasonic image of the first screen and in the ultrasonic image of the second screen, respectively.

[0023] A seventeenth aspect according to the technology of the present invention is the diagnosis support device according to any one of the first to sixteenth aspects, wherein the processor performs processing for detecting a lesion region and an organ region from an ultrasonic image.

[0024] An eighteenth aspect according to the technology of the present invention is an ultrasonic endoscope including: the diagnosis support device according to any one of the first to seventeenth aspects; and an ultrasonic endoscope body to which the diagnosis support device is connected.

[0025] The 19th method involved in the technology of the present invention is a diagnosis support method, which includes the following steps: acquiring an ultrasonic image; causing a display device to display the acquired ultrasonic image; and displaying a first mark in the ultrasonic image that can identify a lesion area detected from the ultrasonic image and a second mark in the ultrasonic image that can identify an organ area detected from the ultrasonic image, wherein the first mark is displayed in a more emphasized state than the second mark.

[0026] The 20th method involved in the technology of the present invention is a program for causing a computer to execute processing, wherein the processing includes the following steps: acquiring an ultrasonic image; causing a display device to display the acquired ultrasonic image; and displaying a first mark capable of identifying a lesion area detected from the ultrasonic image and a second mark capable of identifying an organ area detected from the ultrasonic image in the ultrasonic image, wherein the first mark is displayed in a state where it is more emphasized than the second mark. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a conceptual diagram showing an example of a method of using an endoscope system.

[0028] Figure 2 This is a conceptual diagram showing an example of the overall configuration of an endoscope system.

[0029] Figure 3 This is a block diagram showing an example of the structure of an ultrasonic endoscope.

[0030] Figure 4 This is a conceptual diagram showing an example of a method of generating a learned model by causing a model to learn teacher data.

[0031] Figure 5 This is a conceptual diagram showing an example of the processing content of the generation unit.

[0032] Figure 6 This is a conceptual diagram showing an example of the processing contents of the generating unit and the detecting unit.

[0033] Figure 7 This is a conceptual diagram showing an example of the processing content of the control unit to generate a marker based on the detection frame.

[0034] Figure 8 This is a conceptual diagram showing an example of a manner in which an ultrasonic image to which a mark is assigned is displayed on a screen of a display device.

[0035] Fig.9A This is a flowchart showing an example of the flow of the diagnosis support process.

[0036] Fig. 9B yes Fig.9A A continuation of the flowchart shown.

[0037] Fig.10 This is a conceptual diagram showing an example of the processing content involved in the first modification.

[0038] Fig.11A This is a flowchart showing an example of the flow of the diagnosis support process according to the first modification.

[0039] Fig. 11B yes Fig.11A A continuation of the flowchart shown.

[0040] Fig.12 This is a conceptual diagram showing an example of a manner in which an ultrasonic image to which a first mark is assigned and an ultrasonic image to which a second mark is assigned are displayed on different screens in an endoscope system according to a second modification.

[0041] Fig.13 This is a conceptual diagram showing an example of the manner in which a control unit controls a sound reproducing device and a vibration generator in an endoscope system according to a third modification. DETAILED DESCRIPTION

[0042] Hereinafter, an example of an embodiment of a diagnosis support device, an ultrasonic endoscope, a diagnosis support method, and a program according to the technology of the present invention will be described with reference to the drawings.

[0043] First, the words and phrases used in the following description are explained.

[0044] CPU is short for "Central Processing Unit". GPU is short for "Graphics Processing Unit". TPU is short for "Tensor Processing Unit". RAM is short for "Random Access Memory". NVM is short for "Non-volatile memory". EEPROM is short for "Electrically Erasable Programmable Read-Only Memory". ASIC is short for "Application Specific Integrated Circuit". PLD is short for "Programmable Logic Device". FPGA is short for "Field-Programmable Gate Array". SoC is short for "System-on-a-chip". SSD is short for "Solid State Drive". USB is short for "Universal Serial Bus". HDD is the abbreviation of “Hard Disk Drive”. EL is the abbreviation of “Electro-Luminescence”. CMOS is the abbreviation of “Complementary Metal Oxide Semiconductor”. CCD is the abbreviation of “Charge Coupled Device”. PC is the abbreviation of “Personal Computer”. LAN is the abbreviation of “Local Area Network”. WAN is the abbreviation of “Wide Area Network”. AI is the abbreviation of “Artificial Intelligence”. BLI is the abbreviation of “Blue Light Imaging”. LCI is the abbreviation of “Linked Color Imaging”. NN is the abbreviation of “Neural Network”. CNN is the abbreviation of “Convolutional neural network”.R-CNN is the abbreviation of "Region based Convolutional Neural Network". YOLO is the abbreviation of "You only Look Once". RNN is the abbreviation of "Recurrent Neural Network". FCN is the abbreviation of "Fully Convolutional Network".

[0045] As an example, Figure 1 As shown, the endoscope system 10 includes an ultrasonic endoscope 12 and a display device 14. The ultrasonic endoscope 12 is a convex ultrasonic endoscope, and includes an ultrasonic endoscope body 16 and a processing device 18. The ultrasonic endoscope 12 is an example of an "ultrasonic endoscope" involved in the technology of the present invention. The processing device 18 is an example of a "diagnosis support device" involved in the technology of the present invention. The ultrasonic endoscope body 16 is an example of an "ultrasonic endoscope body" involved in the technology of the present invention. The display device 14 is an example of a "display device" involved in the technology of the present invention.

[0046] In the present embodiment, a convex type ultrasonic endoscope is cited as an example of the ultrasonic endoscope 12 , but this is only an example, and the technology of the present invention is also applicable to a radial type ultrasonic endoscope.

[0047] The ultrasonic endoscope body 16 is used by, for example, a doctor 20. The processing device 18 is connected to the ultrasonic endoscope body 16, and performs various signal exchanges with the ultrasonic endoscope body 16. That is, the processing device 18 controls the operation of the ultrasonic endoscope body 16 by outputting signals to the ultrasonic endoscope body 16, or performs various signal processing on signals input from the ultrasonic endoscope body 16.

[0048] The ultrasonic endoscope 12 is a device for performing treatment (e.g., diagnosis and / or treatment, etc.) of a treatment target site (e.g., organs such as the pancreas) in the body of a subject 22, and generates and outputs an ultrasonic image 24 representing an observation target area including the treatment target site.

[0049] For example, when the doctor 20 observes the observation target area in the body of the subject 22, the doctor 20 moves the ultrasonic endoscope body 16 from the mouth or nose of the subject 22 (at Figure 1 The ultrasonic endoscope body 16 (in the example shown, the mouth) is inserted into the body of the subject 22 and radiates ultrasonic waves at a position such as the stomach or duodenum. The ultrasonic endoscope body 16 radiates ultrasonic waves to an observation target area in the body of the subject 22 and detects reflected waves obtained by the radiated ultrasonic waves reflecting in the observation target area.

[0050] In addition, Figure 1 In the example shown, an upper digestive organ endoscopy is shown, but the technology of the present invention is not limited thereto, and the technology of the present invention can also be applied to lower digestive organ endoscopy, bronchial endoscopy, and the like.

[0051] The processing device 18 generates an ultrasonic image 24 based on the reflected waves detected by the ultrasonic endoscope body 16 and outputs the image to the display device 14 or the like.

[0052] The display device 14 displays various information including images under the control of the processing device 18. An example of the display device 14 is a liquid crystal display or an EL display. On the screen 26 of the display device 14, the ultrasound image 24 generated by the processing device 18 is displayed as a dynamic image. The dynamic image is generated at a predetermined frame rate (for example, several tens of frames per second) and displayed on the screen 26. Figure 1 In the example shown, an ultrasonic image 24 in a screen 26 includes a lesion region 25 indicating a portion corresponding to a lesion and an organ region 27 indicating a portion corresponding to an organ (i.e., a lesion and an organ are shown in the ultrasonic image 24). The lesion region 25 is an example of a "lesion region" involved in the technology of the present invention. The organ region 27 is an example of an "organ region" involved in the technology of the present invention.

[0053] In addition, Figure 1 In the example shown, the ultrasonic image 24 is displayed on the screen 26 of the display device 14, but this is only an example, and the ultrasonic image 24 may be displayed on a display device other than the display device 14 (for example, a display of a tablet terminal). In addition, the ultrasonic image 24 may be stored in a non-temporary storage medium (for example, a flash memory, a HDD, and / or a magnetic tape, etc.) that can be read by a computer.

[0054] As an example, Figure 2 As shown, the ultrasonic endoscope body 16 includes an operation portion 28 and an insertion portion 30. The insertion portion 30 is formed in a tubular shape. The insertion portion 30 includes a front end portion 32, a bending portion 34, and a soft portion 36. The front end portion 32, the bending portion 34, and the soft portion 36 are arranged in order from the front end side to the base end side of the insertion portion 30. The soft portion 36 is formed of a long strip of flexible raw material and connects the operation portion 28 and the bending portion 34. The bending portion 34 is partially bent or rotated around the axis of the insertion portion 30 by operating the operation portion 28. As a result, the insertion portion 30 is sent to the inner side of the lumen organ while being bent according to the shape of the lumen organ (for example, the shape of the duct of the duodenum) or rotating around the axis of the insertion portion 30.

[0055] The distal end portion 32 is provided with an ultrasonic probe 38 and a treatment instrument opening 40. The ultrasonic probe 38 is provided at the distal end side of the distal end portion 32. The ultrasonic probe 38 is a convex ultrasonic probe that radiates ultrasonic waves and receives reflected waves obtained when the radiated ultrasonic waves are reflected in the observation target area.

[0056] The treatment instrument opening 40 is formed on the proximal side of the front end portion 32 relative to the ultrasonic probe 38. The treatment instrument opening 40 is an opening for protruding the treatment instrument 42 from the front end portion 32. A treatment instrument insertion port 44 is formed in the operation portion 28, and the treatment instrument 42 is inserted into the insertion portion 30 from the treatment instrument insertion port 44. The treatment instrument 42 passes through the insertion portion 30 and protrudes from the treatment instrument opening 40 to the outside of the ultrasonic endoscope body 16. In addition, the treatment instrument opening 40 also functions as a suction port for sucking blood and body waste.

[0057] exist Figure 2 In the example shown, a puncture needle is shown as the treatment instrument 42. This is just an example, and the treatment instrument 42 may be a grasping forceps and / or a sheath, etc.

[0058] exist Figure 2 In the example shown, the front end portion 32 is provided with an illumination device 46 and a camera 48. The illumination device 46 irradiates light. Examples of the type of light irradiated from the illumination device 46 include visible light (e.g., white light), non-visible light (e.g., near infrared light), and / or special light. Examples of the special light include BLI light and / or LCI light.

[0059] The camera 48 takes pictures of the inside of the luminal organ by an optical method. As an example of the camera 48, a CMOS camera can be cited. The CMOS camera is only an example, and other types of cameras such as a CCD camera can also be used. In addition, the image obtained by taking pictures by the camera 48 is displayed on the display device 14, or displayed on a display device other than the display device 14 (for example, a display of a tablet terminal), or stored in a storage medium (for example, a flash memory, an HDD and / or a magnetic tape, etc.).

[0060] The ultrasonic endoscope 12 includes a treatment device 18 and a universal cord 50. The universal cord 50 has a base end 50A and a distal end 50B. The base end 50A is connected to the operation unit 28. The distal end 50B is connected to the treatment device 18. That is, the ultrasonic endoscope body 16 and the treatment device 18 are connected via the universal cord 50.

[0061] The endoscope system 10 includes a receiving device 52. The receiving device 52 is connected to the processing device 18. The receiving device 52 receives instructions from the user. Examples of the receiving device 52 include an operation panel having a plurality of hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0062] The processing device 18 performs various signal processing according to the instructions received from the receiving device 52, or exchanges various signals with the ultrasonic endoscope body 16, etc. For example, the processing device 18 causes the ultrasonic probe 38 to emit ultrasonic waves according to the instructions received from the receiving device 52, and generates an ultrasonic image 24 (see Figure 1 ) and output.

[0063] The display device 14 is also connected to the processing device 18. The processing device 18 controls the display device 14 according to the instruction received by the receiving device 52. Thus, for example, the ultrasonic image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 (see Figure 1 ).

[0064] As an example, Figure 3 As shown, the processing device 18 includes a computer 54, an input / output interface 56, a transceiver circuit 58, and a communication module 60. The computer 54 is an example of a "computer" involved in the technology of the present invention.

[0065] The computer 54 includes a processor 62 , a RAM 64 , and an NVM 66 . The input / output interface 56 , the processor 62 , the RAM 64 , and the NVM 66 are connected to a bus 68 .

[0066] The processor 62 controls the processing device 18 as a whole. For example, the processor 62 has a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for performing image processing. In addition, the processor 62 may be one or more CPUs with integrated GPU functions, or one or more CPUs without integrated GPU functions. Furthermore, the processor 62 may include a multi-core CPU or a TPU. The processor 62 is an example of a "processor" involved in the technology of the present invention.

[0067] RAM 64 is a memory for temporarily storing information, and is used as a working memory by processor 62. NVM 66 is a non-volatile storage device for storing various programs and various parameters. As an example of NVM 66, a flash memory (e.g., EEPROM) and / or an SSD can be cited. In addition, flash memory and SSD are only examples, and other non-volatile storage devices such as HDD can be used, or a combination of two or more non-volatile storage devices can be used.

[0068] The receiving device 52 is connected to the input / output interface 56 , and the processor 62 acquires an instruction received by the receiving device 52 via the input / output interface 56 , and executes a process corresponding to the acquired instruction.

[0069] The transceiver circuit 58 is connected to the input / output interface 56. The transceiver circuit 58 generates an ultrasonic wave emission signal 70 of a pulse waveform according to an instruction from the processor 62, and outputs the signal to the ultrasonic probe 38. The ultrasonic probe 38 converts the ultrasonic wave emission signal 70 input from the transceiver circuit 58 into ultrasonic waves, and radiates the ultrasonic waves to an observation target region 72 of the subject 22. The ultrasonic probe 38 receives a reflected wave obtained by the ultrasonic wave emitted from the ultrasonic probe 38 being reflected by the observation target region 72, converts the reflected wave into a reflected wave signal 74 as an electrical signal, and outputs the reflected wave to the transceiver circuit 58. The transceiver circuit 58 digitizes the reflected wave signal 74 input from the ultrasonic probe 38, and outputs the digitized reflected wave signal 74 to the processor 62 via the input / output interface 56. The processor 62 generates an ultrasonic image 24 (see FIG. 24 ) indicating the state of the observation target region 72 based on the reflected wave signal 74 input from the transceiver circuit 58 via the input / output interface 56. Figure 1 ).

[0070] Although in Figure 3 Although not shown in the figure, the input / output interface 56 is also connected to the lighting device 46 (refer to Figure 2 ). The processor 62 controls the lighting device 46 via the input / output interface 56 to change the type of light emitted from the lighting device 46 or adjust the light amount. Figure 3 Although not shown in the figure, the camera 48 is also connected to the input / output interface 56 (see Figure 2 The processor 62 controls the camera 48 via the input / output interface 56 , or acquires an image obtained by the camera 48 capturing the inside of the subject 22 via the input / output interface 56 .

[0071] The communication module 60 is connected to the input / output interface 56. The communication module 60 is an interface including a communication processor and an antenna, etc. The communication module 60 is connected to a network (not shown) such as a LAN or a WAN, and is responsible for communication between the processor 62 and external devices.

[0072] The display device 14 is connected to the input / output interface 56 , and the processor 62 controls the display device 14 via the input / output interface 56 to cause the display device 14 to display various information.

[0073] The receiving device 52 is connected to the input / output interface 56 , and the processor 62 acquires an instruction received by the receiving device 52 via the input / output interface 56 , and executes a process corresponding to the acquired instruction.

[0074] The NVM 66 stores a diagnosis support program 76 and a learned model 78 . The diagnosis support program 76 is an example of a “program” according to the technique of the present invention. The learned model 78 is a learned model having a data structure for processing for detecting lesions and organs from the ultrasonic image 24 .

[0075] The processor 62 reads the diagnosis support program 76 from the NVM 66 and executes the read diagnosis support program 76 on the RAM 64, thereby performing the diagnosis support process. The diagnosis support process is a process of detecting lesions and organs from the observation target area 72 using an AI method and supporting the doctor 20 (refer to Figure 1 ) Diagnosis is performed by using the learned model 78. Detection of lesions and organs using AI is achieved by using the learned model 78.

[0076] The processor 62 performs the diagnosis support processing based on the learned model 78 from the ultrasonic image 24 (reference Figure 1 ) detects a site corresponding to a lesion and a site corresponding to an organ, thereby detecting the lesion and the organ from the observation target area 72. The diagnosis support processing is implemented by the processor 62 operating as the generation unit 62A, the detection unit 62B, and the control unit 62C according to the diagnosis support program 76 executed on the RAM 64.

[0077] As an example, Figure 4 As shown, the learned model 78 is generated by learning the unlearned model 80. The model 80 is learned using the teacher data 82. The teacher data 82 includes a plurality of ultrasonic images 84 that are different from each other. For example, the ultrasonic image 84 is different from the ultrasonic image 24 (see Figure 1 )Similarly, this is an ultrasonic image generated by a convex ultrasonic endoscope.

[0078] The plurality of ultrasonic images 84 include ultrasonic images showing organs (for example, the pancreas, etc.), ultrasonic images showing lesions, and ultrasonic images showing organs and lesions. Figure 4 In the illustrated example, an embodiment in which an organ region 86 representing a portion corresponding to an organ is included in the ultrasonic image 84 and an embodiment in which a lesion region 88 representing a portion corresponding to a lesion is included in the ultrasonic image 84 is illustrated.

[0079] As an example of model 80, a mathematical model using NN can be cited. As the type of NN, YOLO, R-CNN, or FCN can be cited. In addition, the NN used in model 80 can be a combination of YOLO, R-CNN, or FCN and RNN. RNN is suitable for learning multiple images obtained in time series. In addition, the type of NN cited here is only an example, and other types of NN that can detect objects by learning images can also be used.

[0080] exist Figure 4 In the example shown, an organ annotation 90 is assigned to an organ region 86 in an ultrasonic image 84. The organ annotation 90 is information that can identify the position of the organ region 86 in the ultrasonic image 84 (for example, information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing the organ region 86). Here, for convenience of description, as an example of the organ annotation 90, information that can identify the position of the organ region 86 in the ultrasonic image 84 is illustrated, but this is only an example. For example, the organ annotation 90 may include other types of information that identify the organ shown in the ultrasonic image 84, such as information that can identify the type of the organ shown in the ultrasonic image 84.

[0081] exist Figure 4 In the example shown, a lesion annotation 92 is assigned to the lesion region 88. The lesion annotation 92 is information that can identify the position of the lesion region 88 in the ultrasonic image 84 (for example, information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing the lesion region 88). Here, for the sake of convenience, as an example of the lesion annotation 92, information that can identify the position of the lesion region 88 in the ultrasonic image 84 is illustrated, but this is only an example. For example, the lesion annotation 92 may include other types of information that identify the lesion that appears in the ultrasonic image 84, such as information that can identify the type of the lesion that appears in the ultrasonic image 84.

[0082] In addition, for the sake of convenience, in the following, when it is not necessary to distinguish between the organ annotation 90 and the lesion annotation 92, they are referred to as "annotations" without being marked with symbols. Also, in the following, for the sake of convenience, the processing using the learned model 78 is described as processing that is actively performed by the learned model 78 as the main body. That is, for the sake of convenience, the learned model 78 is described as a function of processing the input information and outputting the processing results. Also, in the following, for the sake of convenience, a part of the processing for learning the model 80 is also described as processing that is actively performed by the model 80 as the main body. That is, for the sake of convenience, the model 80 is described as a function of processing the input information and outputting the processing results.

[0083] The training data 82 is input to the model 80. That is, each ultrasonic image 84 is input to the model 80. The model 80 predicts the position of the organ region 86 and / or the position of the lesion region 88 based on the input ultrasonic image 84, and outputs a prediction result. The prediction result includes information that can identify the position predicted by the model 80 as the position of the organ region 86 in the ultrasonic image 84 and / or information that can identify the position predicted by the model 80 as the position of the lesion region 88 in the ultrasonic image 84.

[0084] Here, as an example of information capable of specifying the position predicted by the model 80 as the position of the organ region 86 in the ultrasonic image 84, for example, information including a plurality of coordinates capable of specifying the position of a bounding box surrounding the region predicted to be the position of the organ region 86 (i.e., the position of the bounding box in the ultrasonic image 84) can be cited. Furthermore, as an example of information capable of specifying the position predicted by the model 80 as the position of the lesion region 88 in the ultrasonic image 84, for example, information including a plurality of coordinates capable of specifying the position of a bounding box surrounding the region predicted to be the position of the lesion region 88 (i.e., the position of the bounding box in the ultrasonic image 84) can be cited.

[0085] The model 80 is adjusted according to the error between the annotations given to the ultrasonic images 84 input to the model 80 and the prediction results output from the model 80. That is, the model 80 is optimized by adjusting a plurality of optimization variables (e.g., a plurality of connection loads and a plurality of offset values) in the model 80 so as to minimize the error, thereby generating the learned model 78. That is, the data structure of the learned model 78 is obtained by causing the model 80 to learn a plurality of ultrasonic images 84 that are different from each other and to which annotations are given.

[0086] According to the conventionally known technology, the lesion area 25 is detected by learning the model 78 (refer to Figure 1 ) and the organ region 27 detected by the learning model 78 (reference Figure 1 ) is visualized by displaying the result as a mark such as a detection frame on the screen 26. The mark such as the detection frame indicates the positions of the lesion region 25 and the organ region 27. The frequency of displaying the lesion region 25 on the screen 26 (in other words, the frequency of appearance of the lesion region 25) is lower than the frequency of displaying the organ region 27 (in other words, the frequency of appearance of the organ region 27). This means that when the doctor 20 uses the ultrasonic image 24 for diagnosis, the possibility of missing the lesion region 25 is higher than the possibility of missing the organ region 27.

[0087] Furthermore, if the marks such as the detection frame attached to the organ region 27 and the marks such as the detection frame attached to the lesion region 25 are displayed on the screen 26 in a mixed state, the presence of the marks such as the detection frame attached to the organ region 27 may hinder the diagnosis. This may also be one of the reasons for increasing the possibility of overlooking the lesion region 25.

[0088] Therefore, in view of this situation, in the processing device 18 involved in this embodiment, as an example, Figures 5 to 9B As shown, the diagnosis support process is performed. Hereinafter, an example of the diagnosis support process will be specifically described.

[0089] As an example, Figure 5 As shown, the generator 62A acquires the reflected wave signal 74 from the transmission and reception circuit 58, and generates the ultrasonic image 24 based on the acquired reflected wave signal 74, thereby acquiring the ultrasonic image 24. The ultrasonic image 24 is an example of an “ultrasonic image” according to the technique of the present invention.

[0090] As an example, Figure 6 As shown, the detection unit 62B detects the lesion region 25 from the ultrasonic image 24 generated by the generator 62A based on the learned model 78, thereby detecting the lesion. That is, the detection unit 62B determines whether the lesion region 25 exists in the ultrasonic image 24 based on the learned model 78, and generates lesion position identification information 94 (for example, information including a plurality of coordinates that identify the position of the lesion region 25) that identifies the position of the lesion region 25 when the lesion region 25 exists in the ultrasonic image 24. Here, if the processing of detecting the lesion by the detection unit 62B is mainly explained with the learned model 78, when the ultrasonic image 24 generated by the generator 62A is input, the learned model 78 determines whether the lesion region 25 exists in the input ultrasonic image 24. When the learned model 78 determines that the lesion region 25 exists in the ultrasonic image 24 (that is, when the lesion reflected in the ultrasonic image 24 is detected), the learned model 78 outputs the lesion position identification information 94.

[0091] Then, the detector 62B detects the organ region 27 from the ultrasonic image 24 generated by the generator 62A based on the learned model 78, thereby detecting the organ. That is, the detector 62B determines whether the organ region 27 is present in the ultrasonic image 24 based on the learned model 78, and generates the organ position identification information 96 (for example, information including a plurality of coordinates identifying the position of the organ region 27) that identifies the position of the organ region 27 when the organ region 27 exists in the ultrasonic image 24. Here, the processing of detecting the organ by the detector 62B will be described mainly with the learned model 78 as the main body. When the ultrasonic image 24 generated by the generator 62A is input, the learned model 78 determines whether the organ region 27 is present in the input ultrasonic image 24. When the learned model 78 determines that the organ region 27 is present in the ultrasonic image 24 (that is, when the organ shown in the ultrasonic image 24 is detected), the learned model 78 outputs the organ position identification information 96.

[0092] The detection unit 62B generates the detection frames 98 and 100 and superimposes the generated detection frames 98 and 100 on the ultrasonic image 24 , thereby adding the detection frames 98 and 100 to the ultrasonic image 24 .

[0093] The detection frame 98 is a rectangular frame corresponding to the bounding frame used when the learned model 78 detects the lesion area 25 from the ultrasonic image 24 (for example, the bounding frame with the highest reliability score related to the lesion area 25). In other words, the detection frame 98 is a frame surrounding the range 25A where the lesion area 25 detected by the learned model 78 exists. The range 25A is a rectangular range and is defined by the detection frame 98. Figure 6 In the example shown, a rectangular frame circumscribing the lesion region 25 is shown as an example of the detection frame 98. The rectangular frame circumscribing the lesion region 25 is just an example, and the technology of the present invention is also applicable even if the rectangular frame does not circumscribing the lesion region 25.

[0094] The detection unit 62B assigns the detection frame 98 to the ultrasonic image 24 corresponding to the lesion position identification information 94 output from the learned model 78 (that is, the ultrasonic image 24 input to the learned model 78 in order to output the lesion position identification information 94) based on the lesion position identification information 94. That is, the detection unit 62B assigns the detection frame 98 to the ultrasonic image 24 by overlapping the detection frame 98 on the ultrasonic image 24 corresponding to the lesion position identification information 94 output from the learned model 78 so as to surround the lesion region 25.

[0095] The detection frame 100 is a rectangular frame corresponding to the bounding frame used when the learned model 78 detects the organ region 27 from the ultrasonic image 24 (for example, the bounding frame with the highest reliability score related to the organ region 27). That is, the detection frame 100 is a frame surrounding the lesion region 25 detected by the learned model 78. That is, the detection frame 100 is a frame surrounding the range 27A where the organ region 27 detected by the learned model 78 exists. The range 27A is a rectangular range and is defined by the detection frame 100. Figure 6 In the example shown, a rectangular frame circumscribing the organ region 27 is shown as an example of the detection frame 100. The rectangular frame circumscribing the organ region 27 is just an example, and the technology of the present invention is also applicable even if the rectangular frame does not circumscribing the organ region 27.

[0096] The detection unit 62B assigns the detection frame 100 to the ultrasonic image 24 corresponding to the organ position identification information 96 output from the learned model 78 (that is, the ultrasonic image 24 input to the learned model 78 in order to output the organ position identification information 96) based on the organ position identification information 96. That is, the detection unit 62B assigns the detection frame 100 to the ultrasonic image 24 by overlapping the detection frame 100 on the ultrasonic image 24 corresponding to the organ position identification information 96 output from the learned model 78 so as to surround the organ region 27.

[0097] In this embodiment, detection frame 98 is an example of the "first rectangular frame" involved in the technology of the present invention. Range 25A is an example of the "first range" involved in the technology of the present invention. Detection frame 100 is an example of the "second rectangular frame" involved in the technology of the present invention. Range 27A is an example of the "second range" involved in the technology of the present invention. In addition, for the sake of convenience, in the following, when there is no need to distinguish between detection frames 98 and 100, they are referred to as "detection frames" without being marked with symbols.

[0098] As an example, Figure 7 As shown in FIG. 1 , the control unit 62C acquires the ultrasonic image 24 reflecting the detection result from the detection unit 62B. Figure 7 In the illustrated example, a mode is shown in which the ultrasonic image 24 to which the detection frames 98 and 100 are assigned is acquired and processed by the control unit 62C.

[0099] In the ultrasonic image 24, the detection frame 98 surrounds the lesion region 25 in a rectangular shape when viewed from the front. In addition, in the ultrasonic image 24, the detection frame 100 surrounds the organ region 27 in a rectangular shape when viewed from the front. Here, the front view refers to the state of the screen 26 when the ultrasonic image 24 is displayed on the display device 14, for example, when viewed from the front side.

[0100] The control unit 62C generates the first marker 102 based on the detection frame 98. The first marker 102 is a marker that can identify the lesion region 25 detected by the detection unit 62B from the ultrasonic image 24 in the ultrasonic image 24. The first marker 102 is formed so as to identify the outer edge of the range 25A.

[0101] The first mark 102 is composed of four images. Figure 7 In the example shown, the four images are L-shaped pieces 102A to 102D. In each of the L-shaped pieces 102A to 102D, a portion of the detection frame 98 is imaged. That is, the L-shaped pieces 102A to 102D are marks formed in a manner that allows visual identification of a portion of the detection frame 98. Figure 7 In the illustrated example, the L-shaped pieces 102A to 102D are formed in the same shape and the same size.

[0102] exist Figure 7 In the example shown, the positions of the L-shaped pieces 102A to 102D correspond to the positions of the four corners of the detection frame 98. The L-shaped pieces 102A to 102D are formed in the shape of the corners of the detection frame 98, respectively. That is, the L-shaped pieces 102A to 102D are formed in an L shape, respectively. In this way, the position of the range 25A in the ultrasonic image 24 can be determined by assigning the L-shaped pieces 102A to 102D to the four corners of the detection frame 98. The L-shaped pieces 102A to 102D are an example of the "plurality of first images" involved in the technology of the present invention.

[0103] The control unit 62C generates the second marker 104 based on the detection frame 100. The second marker 104 is a marker capable of specifying the organ region 27 detected by the detection unit 62B from the ultrasonic image 24 within the ultrasonic image 24. The second marker 104 is formed so as to specify the outer edge of the range 27A.

[0104] The second mark 104 is composed of four images. Figure 7 In the example shown, the four images constituting the second mark 104 are T-shaped pieces 104A to 104D. In each of the T-shaped pieces 104A to 104D, a portion of the detection frame 100 is imaged. That is, the T-shaped pieces 104A to 104D are marks formed in a manner that allows visual identification of a portion of the detection frame 100. Figure 7 In the example shown, the positions of the T-shaped pieces 104A to 104D correspond to the positions of the central portions of the sides 100A to 100D constituting the detection frame 100. The T-shaped pieces 104A to 104D are each formed in a T shape. Figure 7In the example shown, T-shaped pieces 104A to 104D are formed in the same shape and the same size. T-shaped pieces 104A to 104D respectively include straight lines 106 and 108. One end of straight line 108 is located at the midpoint of straight line 106, and straight line 108 is arranged perpendicular to straight line 106.

[0105] The straight line 106 of the T-shaped piece 104A is formed at a position parallel to and overlapping the side 100A. The straight line 108 of the T-shaped piece 104A extends from the midpoint of the side 100A to the lower side when viewed from the front. The straight line 106 of the T-shaped piece 104B is formed at a position parallel to and overlapping the side 100B. The straight line 108 of the T-shaped piece 104B extends from the midpoint of the side 100B to the left side when viewed from the front. The straight line 106 of the T-shaped piece 104C is formed at a position parallel to and overlapping the side 100C. The straight line 108 of the T-shaped piece 104C extends from the midpoint of the side 100C to the upper side when viewed from the front. The straight line 106 of the T-shaped piece 104D is formed at a position parallel to and overlapping the side 100D. The straight line 108 of the T-shaped piece 104D extends from the midpoint of the side 100D to the right side when viewed from the front.

[0106] Thus, the position of the range 27A in the ultrasonic image 24 can be determined by allocating the T-shaped slices 104A to 104D to the respective central portions of the sides 100A to 100D of the detection frame 100. The T-shaped slices 104A to 104D are examples of the "plurality of second images" involved in the technique of the present invention.

[0107] exist Figure 7 In the example shown, the first mark 102 is formed in a state where it is emphasized more than the second mark 104. Here, the emphasized state means that when the first mark 102 and the second mark 104 are displayed in a mixed state on the screen 26, the first mark 102 is more visually conspicuous than the second mark 104. Figure 7 In the example shown, the first mark 102 is formed of a thicker line than the second mark 104, and the L-shaped pieces 102A to 102D are formed of a larger size than the T-shaped pieces 104A to 104D. As a result, the first mark 102 is emphasized more than the second mark 104. Hereinafter, for the sake of convenience, when it is not necessary to distinguish between the first mark 102 and the second mark 104, they are referred to as "marks" without being given symbols.

[0108] As an example, Figure 8As shown, the control unit 62C causes the display device 14 to display the ultrasonic image 24 generated by the generating unit 62A. At this time, when the detecting unit 62B does not detect the lesion and the organ, the control unit 62C displays the ultrasonic image 24 without the mark on the screen 26 of the display device 14. Furthermore, when the detecting unit 62B detects the lesion and / or the organ, the control unit 62C displays the ultrasonic image 24 with the mark on the screen 26 of the display device 14. On the screen 26, the first marker 102 is displayed at a position corresponding to the lesion area 25 in the ultrasonic image 24. That is, the L-shaped pieces 102A to 102D are displayed so as to surround the lesion area 25. In other words, the L-shaped pieces 102A to 102D are displayed so as to be able to determine the range 25A (refer to Figure 6 and Figure 7 ) The outer edge of the ultrasound image 24. Thus, the position of the lesion area 25 in the ultrasound image 24 can be visually grasped.

[0109] Furthermore, on the screen 26, the second marker 104 is displayed at a position corresponding to the organ region 27 in the ultrasonic image 24. That is, the T-shaped pieces 104A to 104D are displayed so as to surround the organ region 27. In other words, the T-shaped pieces 104A to 104D are displayed so as to identify the range 27A (see FIG. 2 ). Figure 6 and Figure 7 )’s outer edge.

[0110] Furthermore, in the screen 26, the first mark 102 is displayed in a state of being emphasized more than the second mark 104. Thus, the position of the lesion region 25 and the position of the organ region 27 can be visually distinguished.

[0111] Next, refer to Fig.9A and Fig. 9B , the function of the endoscope system 10 is described.

[0112] exist Fig.9A and Fig. 9B 2 shows an example of a flow of diagnosis support processing performed by the processor 62 of the processing device 18 on the condition that diagnosis using the endoscope system 10 has started (for example, ultrasound emission by the ultrasound endoscope 12 has started). Fig.9A and Fig. 9B The illustrated flow of the diagnosis support process is an example of a “diagnosis support method” according to the technique of the present invention.

[0113] exist Fig.9A In the diagnostic support process shown in FIG. 1 , first, in step ST10, the generator 62A determines whether the image display time has arrived. The image display time is, for example, a time divided by a time interval specified by the inverse of the frame rate. In step ST10, if the image display time has not arrived, the determination is negative, and the diagnostic support process is transferred to step ST20. Fig. 9BStep ST36 is shown. In step ST10, if the image display time has come, the determination is affirmative, and the diagnosis support process moves to step ST12.

[0114] In step ST12, the generator 62A generates the ultrasonic image 24 (see FIG. 24 ) based on the reflected wave signal 74 input from the transmission and reception circuit 58. Figure 5 ). After executing the process of step ST12, the diagnosis support process transfers to step ST14.

[0115] In step ST14, the detection unit 62B inputs the ultrasonic image 24 generated in step ST12 to the learned model 78. After the process of step ST14 is executed, the diagnosis support process proceeds to step ST16.

[0116] In step ST16, the detector 62B uses the learned model 78 to determine whether the ultrasonic image 24 input to the learned model 78 in step ST14 includes the lesion area 25. If the ultrasonic image 24 includes the lesion area 25, the learned model 78 outputs the lesion position identification information 94 (see Figure 6 ).

[0117] In step ST16, if the ultrasonic image 24 does not include the lesion area 25, the determination is negative and the diagnosis support process is transferred to step ST16. Fig. 9B In step ST24 shown in the figure, if the lesion area 25 is included in the ultrasonic image 24 in step ST16, the determination is affirmative, and the diagnosis support process moves to step ST18.

[0118] In step ST18, the detector 62B uses the learned model 78 to determine whether the ultrasonic image 24 input to the learned model 78 in step ST14 includes the organ region 27. If the ultrasonic image 24 includes the organ region 27, the learned model 78 outputs the organ position identification information 96 (see Figure 6 ).

[0119] In step ST18, if the organ region 27 is not included in the ultrasonic image 24, the determination is negative, and the diagnosis support process is transferred to step ST19. Fig. 9B Step ST32 is shown. In step ST18, when the organ region 27 is included in the ultrasonic image 24, the determination is affirmative, and the diagnosis support process shifts to step ST20.

[0120] In step ST20, the control unit 62C determines the position of the lesion based on the information 94 (reference Figure 6 ) to generate the first marker 102 (reference Figure 7Specifically, the control unit 62C generates a detection frame 98 (see Figure 6 ), and generates the first marker 102 (reference Figure 7 ). Then, the control unit 62C determines the organ position information 96 (refer to Figure 6 ) to generate the second marker 104 (reference Figure 7 Specifically, the control unit 62C generates a detection frame 100 based on the organ position identification information 96 (see Figure 6 ), and generates a second marker 104 based on the detection frame 100 (reference Figure 7 ). The first marker 102 and the second marker 104 generated in this way are superimposed on the ultrasonic image 24 generated in step ST12, and are thereby added to the ultrasonic image 24. After the process of step ST20 is executed, the diagnosis support process proceeds to step ST22.

[0121] In step ST22, the control unit 62C displays the ultrasonic image 24 on which the first marker 102 and the second marker 104 are superimposed on the screen 26 of the display device 14 (see FIG. 26 ). Figure 8 ). The first mark 102 is displayed in a more emphasized state than the second mark 104. After the process of step ST22 is executed, the diagnosis support process is transferred to Fig. 9B Step ST36 is shown.

[0122] exist Fig. 9B In step ST24 shown in FIG. 14 , the detector 62B uses the learned model 78 to determine whether the ultrasonic image 24 input to the learned model 78 in step ST14 includes the organ region 27. If the ultrasonic image 24 includes the organ region 27, the learned model 78 outputs the organ position identification information 96 (see FIG. 14 ). Figure 6 ).

[0123] In step ST24, if the ultrasonic image 24 does not include the organ region 27, the determination is negative and the diagnosis support process moves to step ST30. In step ST24, if the ultrasonic image 24 includes the organ region 27, the determination is positive and the diagnosis support process moves to step ST26.

[0124] In step ST26, the control unit 62C determines the position of the organ based on the information 96 (reference Figure 6 ) to generate the second marker 104 (reference Figure 7 ). The second marker 104 is superimposed on the ultrasonic image 24 generated in step ST12, and thereby is added to the ultrasonic image 24. After the process of step ST26 is executed, the diagnosis support process proceeds to step ST28.

[0125] In step ST28, the control unit 62C displays the ultrasonic image 24 on which the second mark 104 is superimposed on the screen 26 of the display device 14. After the process of step ST28 is executed, the diagnosis support process proceeds to step ST36.

[0126] In step ST30, the control unit 62C displays the ultrasonic image 24 generated in step ST12 on the screen 26 of the display device 14. After the process of step ST30 is executed, the diagnosis support process proceeds to step ST36.

[0127] In step ST32, the control unit 62C determines the position of the lesion based on the information 94 (reference Figure 6 ) to generate the first marker 102 (reference Figure 7 ). The first marker 102 is superimposed on the ultrasonic image 24 generated in step ST12, and thereby is added to the ultrasonic image 24. After the process of step ST32 is executed, the diagnosis support process proceeds to step ST34.

[0128] In step ST34, the control unit 62C displays the ultrasonic image 24 on which the first mark 102 is superimposed on the screen 26 of the display device 14. After the process of step ST34 is executed, the diagnosis support process proceeds to step ST36.

[0129] In step ST36, the control unit 62C determines whether the condition for terminating the diagnostic support process (hereinafter referred to as the "diagnostic support termination condition") is satisfied. An example of the diagnostic support termination condition is that the receiving device 52 receives an instruction to terminate the diagnostic support process. In step ST36, if the diagnostic support termination condition is not satisfied, the determination is negative, and the diagnostic support process is transferred to step ST37. Fig.9A In step ST10 shown in the figure, in step ST36, if the diagnosis support end condition is satisfied, the determination is affirmative, and the diagnosis support process ends.

[0130] As described above, in the endoscope system 10, when the lesion region 25 is detected, the first marker 102 (see Figure 7 ), when the organ region 27 is detected, a second marker 104 (reference Figure 7). Then, the ultrasonic image 24 with the first mark 102 and the second mark 104 superimposed thereon is displayed on the screen 26 of the display device 14. The first mark 102 is displayed in a more emphasized state than the second mark 104. Thus, the doctor 20 can easily visually distinguish the lesion area 25 from the organ area 27. Furthermore, the visual expression intensity of the second mark 104 is weaker than that of the first mark 102, so that the first mark 102 can be suppressed from being overlooked due to the second mark 104 being too conspicuous. Thus, in the diagnosis using the ultrasonic image 24, it is possible to suppress the overlooking of the lesion area 25.

[0131] The first mark 102 is a mark that can identify the outer edge of the range 25A where the lesion region 25 exists. Therefore, the doctor 20 can visually recognize the outer edge of the range 25A where the lesion region 25 exists from the ultrasonic image 24.

[0132] The range 25A where the lesion region 25 exists is defined by the detection frame 98 which is a rectangular frame surrounding the lesion region 25. Therefore, the range 25A where the lesion region 25 exists can be processed in units of the detection frame 98 which is a rectangular frame.

[0133] The detection frame 98 is a rectangular frame circumscribing the lesion region 25. Therefore, compared with using a rectangular frame that does not circumscribing the lesion region 25 (for example, a rectangular frame that is further outside than the lesion region 25), the doctor 20 can determine the range 25A where the lesion region 25 exists with high accuracy.

[0134] The first mark 102 is a mark formed so as to be visually identified as a part of the detection frame 98. Therefore, the doctor 20 can visually recognize the range 25A where the lesion region 25 exists in units of the detection frame 98.

[0135] The first marker 102 includes L-shaped pieces 102A to 102D disposed at four corners of the detection frame 98. Therefore, compared with the case where the entire detection frame 98 is displayed, when the doctor 20 observes the ultrasonic image 24, the number of elements that hinder observation can be reduced.

[0136] The second mark 104 is a mark that can identify the outer edge of the range 27A where the organ region 27 exists. Therefore, the doctor 20 can visually recognize the outer edge of the range 27A where the organ region 27 exists from the ultrasonic image 24 .

[0137] The range 27A where the organ region 27 exists is defined by the detection frame 100 which is a rectangular frame surrounding the organ region 27. Therefore, the range 27A where the organ region 27 exists can be processed in units of the detection frame 100 which is a rectangular frame.

[0138] The detection frame 100 is a rectangular frame circumscribing the organ region 27. Therefore, the doctor 20 can identify the range 27A where the organ region 27 exists with higher accuracy than when using a rectangular frame that does not circumscribing the organ region 27 (for example, a rectangular frame that is farther outside than the organ region 27).

[0139] The second mark 104 is a mark formed so as to visually identify a part of the detection frame 100 . Therefore, the doctor 20 can visually recognize the range 27A where the organ region 27 exists in units of the detection frame 100 .

[0140] The second marker 104 includes T-shaped pieces 104A to 104D disposed at the center of each of the four sides of the detection frame 100. Therefore, compared with the case where the entire detection frame 100 is displayed, when the doctor 20 observes the ultrasonic image 24, fewer elements that hinder observation can be provided.

[0141] Furthermore, when the T-shaped pieces 104A to 104D are displayed as the second markers 104, the distance between the adjacent T-shaped pieces in the T-shaped pieces 104A to 104D becomes shorter than when the markers are arranged at the four corners of the detection frame 100, so that the second markers 104 (i.e., the T-shaped pieces 104A to 104D) can be less likely to be missed than when the markers are arranged at the four corners of the detection frame 100. In particular, the larger the organ region 27 is and the smaller the markers arranged at the four corners of the detection frame 100 are, the easier it is to miss the markers arranged at the four corners of the detection frame 100. However, even under such conditions, the second markers 104 (i.e., the T-shaped pieces 104A to 104D) can be less likely to be missed than when the markers are arranged at the four corners of the detection frame 100.

[0142] Moreover, as an example, Figure 7 and Figure 8 As shown, the intersection point of the straight line connecting the midpoint of the T-shaped piece 104A and the midpoint of the T-shaped piece 104C and the straight line connecting the midpoint of the T-shaped piece 104B and the midpoint of the T-shaped piece 104D (hereinafter, simply referred to as the "intersection point") is a point included in the central part of the range 27A where the organ region 27 exists. The direction of the straight line 108 of the T-shaped piece 104A, the direction of the straight line 108 of the T-shaped piece 104B, the direction of the straight line 108 of the T-shaped piece 104C, and the direction of the straight line 108 of the T-shaped piece 104D point to the position of the intersection point. Therefore, the doctor 20 can visually estimate the position of the central part of the range 27A where the organ region 27 exists based on the positions of the T-shaped pieces 104A to 104D.

[0143] The lesion region 25 and the organ region 27 are detected by the processor 62. Therefore, the doctor 20 can visually recognize the lesion region 25 detected by the processor 62 through the first mark 102, and can visually recognize the organ region 27 detected by the processor 62 through the second mark 104.

[0144] [First Modification]

[0145] In the above embodiment, the detection results of the lesion region 25 and the organ region 27 by the detection unit 62B are displayed as marks in units of frames (i.e., each time the ultrasonic image 24 is generated), but the technology of the present invention is not limited to this. For example, the control unit 62C may be as follows: when the lesion region 25 is detected for N consecutive images from a plurality of frames, the first mark 102 is displayed in the ultrasonic image 24, and when the organ region 27 is detected for M consecutive images from a plurality of frames, the second mark 104 is displayed in the ultrasonic image 24. Here, N and M are natural numbers that satisfy the magnitude relationship of "N<M", and a plurality of frames refer to a plurality of ultrasonic images 24 in time series (for example, a plurality of ultrasonic images 24 constituting a dynamic image).

[0146] Here, when N is set to "2" and M is set to "5", as an example, Fig.10 As shown, the first marker 102 is displayed in the ultrasonic image 24 on the condition that the lesion region 25 is detected from two consecutive frames, and the second marker 104 is displayed in the ultrasonic image 24 on the condition that the organ region 27 is detected from five consecutive frames.

[0147] exist Fig.10 In the example shown, when the organ region 27 is continuously detected from time t0 to time t4, the second marker 104 is displayed at time t4. Fig.10 In the example shown, the following method is shown: when the lesion area 25 is continuously detected at time t2 and time t3, the first mark 102 is displayed at time t3, and when the lesion area 25 is continuously detected at time t3 and time t4, the first mark 102 is displayed at time t4.

[0148] exist Fig.11A and Fig. 11B An example of the flow of the diagnosis support process according to the first modification is shown in FIG. Fig.11A and Fig. 11B The flowchart shown is similar to Fig.9A and Fig. 9B Compared with the flowchart shown in FIG. 1 , the processing of steps ST100 to ST116 is added.

[0149] The processing of step ST100 and the processing of step ST102 are provided between the processing of step ST16 and the processing of step ST18. The processing of step ST104 and the processing of step ST106 are provided between the processing of step ST18 and the processing of step ST20. The processing of step ST108 is provided at the previous stage of the processing of step ST24, and is executed when the determination in step ST16 is negative. The processing of step ST110 and the processing of step ST112 are provided between the processing of step ST24 and the processing of step ST26. The processing of step ST114 is provided at the previous stage of the processing of step ST30, and is executed when the determination in step ST24 is negative. The processing of step ST116 is provided at the previous stage of the processing of step ST32, and is executed when the determination in step ST18 is negative.

[0150] exist Fig.11A In the diagnosis support device shown, in step ST100 , the detection unit 62B adds 1 to the first variable whose initial value is “0.” After the process of step ST100 is executed, the diagnosis support process moves to step ST102 .

[0151] In step ST102, the detection unit 62B determines whether the first variable is greater than or equal to N. In step ST102, if the first variable is less than N, the determination is negative, and the diagnostic support process is transferred to step ST103. Fig. 11B Step ST24 is shown. In step ST102, when the first variable is equal to or greater than N, the determination is affirmative, and the diagnosis support process moves to step ST18.

[0152] In step ST104 , the detection unit 62B adds 1 to the second variable whose initial value is “0.” After the process of step ST102 is executed, the diagnosis support process proceeds to step ST106 .

[0153] In step ST106, the detection unit 62B determines whether the second variable is greater than M. In step ST106, if the second variable is less than M, the determination is negative, and the diagnosis support process is transferred to step ST106. Fig. 11B In step ST106, when the second variable is equal to or greater than M, the determination is affirmative, and the diagnosis support process moves to step ST20.

[0154] exist Fig. 11B In step ST108 shown in FIG. 1 , the detection unit 62B resets the first variable. That is, the first variable returns to the initial value. After the process of step ST108 is executed, the diagnosis support process moves to step ST24.

[0155] In step ST110 , the detection unit 62B adds 1 to the second variable. After the process of step ST110 is executed, the diagnosis support process proceeds to step ST112 .

[0156] In step ST112, the detection unit 62B determines whether the second variable is greater than or equal to M. In step ST112, if the second variable is less than M, the determination is negative, and the diagnostic support process moves to step ST30. In step ST112, if the second variable is greater than or equal to M, the determination is positive, and the diagnostic support process moves to step ST26.

[0157] In step ST114, the detection unit 62B resets the second variable. That is, the second variable returns to the initial value. After the process of step ST114 is executed, the diagnosis support process moves to step ST30.

[0158] In step ST116, the detection unit 62B resets the second variable. After the process of step ST116 is executed, the diagnosis support process proceeds to step ST32.

[0159] As described above, in the first modification, when the lesion region 25 is detected for N consecutive images from a plurality of frames, the first marker 102 is displayed in the ultrasonic image 24. Therefore, compared with the case where the ultrasonic image 24 reflecting the detection result is displayed for each frame, the detection result with high reliability is visualized as the first marker 102, so that it is possible to suppress the area that is not the lesion region 25 from being mistakenly identified as the lesion region 25 by the doctor 20. Furthermore, when the organ region 27 is detected for M consecutive images from a plurality of frames, the second marker 104 is displayed in the ultrasonic image 24. Therefore, compared with the case where the ultrasonic image 24 reflecting the detection result is displayed for each frame, the detection result with high reliability is visualized as the second marker 104, so it is possible to suppress the area that is not the organ region 27 from being mistakenly identified as the organ region 27 by the doctor 20.

[0160] Furthermore, in the first modification, natural numbers satisfying the magnitude relationship of N<M are used as N and M. In the first modification, the first marker 102 is displayed on the condition that the lesion region 25 is detected for N consecutive images, so that, for example, the frequency of visualizing the detection result of the lesion region 25 as the first marker 102 becomes higher than when the first marker 102 is visualized on the condition that the lesion region 25 is detected for M consecutive images. Therefore, the risk of missing the lesion region 25 can be reduced. Furthermore, in the first modification, the second marker 104 is displayed on the condition that the organ region 27 is detected for M consecutive images, so, for example, the frequency of visualizing the detection result of the organ region 27 as the second marker 104 becomes lower than when the second marker 104 is visualized on the condition that the organ region 27 is detected for N consecutive images. Therefore, it is possible to suppress the second marker 104 from hindering the diagnosis due to the second marker 104 being displayed at a high frequency.

[0161] In the first modification, "2" is cited as N and "5" is cited as M, but this is only an example, and N and M can be natural numbers greater than 2, and are preferably natural numbers for which the magnitude relationship of "N<M" holds.

[0162] [Second Modification]

[0163] In the above embodiment, an example is given in which only the ultrasonic image 24 with the markers is displayed on the screen 26, but the technology of the present invention is not limited thereto. For example, an ultrasonic image 24 with only the first marker 102 of the first marker 102 and the second marker 104 and an ultrasonic image 24 with only the second marker 104 of the first marker 102 and the second marker 104 may be displayed separately.

[0164] At this time, for example, Fig.12 As shown in FIG. 1 , the control unit 62C causes the display device 14 to display the first screen 26A and the second screen 26B side by side. Then, the control unit 62C displays the ultrasonic image 24 with only the first mark 102 of the first mark 102 and the second mark 104 marked on the first screen 26A. And, the control unit 62C displays the ultrasonic image 24 with only the second mark 104 of the first mark 102 and the second mark 104 marked on the second screen 26B. As a result, the visual recognition of the ultrasonic image 24 is improved compared to the case where the first mark 102 and the second mark 104 are mixed in one ultrasonic image 24. In addition, the first screen 26A is an example of the "first screen" involved in the technology of the present invention, and the second screen 26B is an example of the "second screen" involved in the technology of the present invention.

[0165] exist Fig.12In the example shown, the control unit 62C causes the display device 14 to display the first screen 26A and the second screen 26B side by side, but this is only an example, and the control unit 62C may cause the display device 14 to display a screen equivalent to the first screen 26A, and cause a display device other than the display device 14 to display a screen equivalent to the second screen 26B.

[0166] Furthermore, according to the provided conditions, the ultrasonic image 24 marked with the first marker 102 and the second marker 104, the ultrasonic image 24 marked with only the first marker 102 of the first marker 102 and the second marker 104, and the ultrasonic image 24 marked with only the second marker 104 of the first marker 102 and the second marker 104 may be selectively displayed on the screen 26 (see Figure 1 Here, as a first example of the provided condition, the condition that the receiving device 52 receives an instruction from the user can be cited. As a second example of the provided condition, the condition that at least one designated lesion is detected can be cited. As a third example of the provided condition, the condition that at least one designated organ is detected can be cited.

[0167] [Third Modification]

[0168] In the above embodiment, the doctor 20 grasps the detection of the lesion region 25 by visually recognizing the first mark 102 displayed in the ultrasonic image 24, and grasps the detection of the organ region 27 by visually recognizing the second mark 104 displayed in the ultrasonic image 24, but the technology of the present invention is not limited to this. For example, the doctor 20 may be notified of the detection of the lesion region 25 and / or the organ region 27 by outputting sound and / or generating vibration.

[0169] At this time, for example, Fig.13As shown, the endoscope system 10 includes a sound reproducing device 110 and a vibration generator 112, and the control unit 62C controls the sound reproducing device 110 and the vibration generator 112. For example, when the lesion region 25 is detected by the detection unit 62B, the control unit 62C reproduces a sound expressing information indicating that the lesion region 25 is detected. Also, when the organ region 27 is detected by the detection unit 62B, the control unit 62C reproduces a sound expressing information indicating that the organ region 27 is detected. Also, when the lesion region 25 is detected by the detection unit 62B, the control unit 62C generates a vibration expressing information indicating that the lesion region 25 is detected. Also, when the organ region 27 is detected by the detection unit 62B, the control unit 62C generates a vibration expressing information indicating that the organ region 27 is detected. For example, the vibration generator 112 is worn by the doctor 20 in a state of contact with the body of the doctor 20, and the doctor 20 understands that the lesion region 25 and / or the organ region 27 are detected by sensing the vibration generated by the vibration generator 112.

[0170] In this way, when the lesion area 25 is detected, the detection of the lesion area 25 is notified by causing the sound reproducing device 110 to reproduce sound or causing the vibration generator 112 to generate vibration, thereby reducing the risk of missing the lesion area 25 shown in the ultrasonic image 24.

[0171] Furthermore, the control unit 62C may cause the vibration generator 112 to change the magnitude of vibration and / or the interval at which vibration is generated, etc., when the lesion region 25 is detected, when the organ region 27 is detected, or when both the lesion region 25 and the organ region 27 are detected. Furthermore, the magnitude of vibration and / or the interval at which vibration is generated may be changed according to the type of lesion detected, and the magnitude of vibration and / or the interval at which vibration is generated may be changed according to the type of organ detected.

[0172] In the third modification, an example is given in which sound is reproduced or vibration is generated even when the organ region 27 is detected. However, sound may not be reproduced or vibration may not be generated when the organ region 27 is detected.

[0173] [Other Modifications]

[0174] In the above embodiment, L-shaped pieces 102A to 102D are exemplified as the first marker 102, but the technology of the present invention is not limited thereto. For example, a pair of first images can be used that can identify the positions corresponding to the diagonal corners of the four corners of the detection frame 98. As an example of a pair of first images, a combination of L-shaped pieces 102A and 102C or a combination of L-shaped pieces 102B and 102D can be cited. Moreover, the L-shaped pieces 102A to 102D are only examples, and pieces of shapes other than the L-shaped shape such as the I-shaped shape can also be used. Furthermore, as the first marker 102, the detection frame 98 itself can also be used. Furthermore, as the first marker 102, a marker of a shape in which a part of the detection frame 98 is missing can also be used. In this way, the first marker 102 can be used as long as it can identify the position of the lesion area 25 in the ultrasonic image 24 and at least a part of the detection frame 98 is formed in a manner that can be visually identified.

[0175] In the above embodiment, a rectangular frame is exemplified as the detection frame 98 , but this is only an example, and a frame of another shape may be used.

[0176] In the above embodiment, T-shaped pieces 104A to 104D are exemplified as the second marker 104, but the technology of the present invention is not limited thereto. For example, a pair of second images can be used that can identify the position corresponding to the opposite side of the four sides of the detection frame 100. As an example of a pair of second images, a combination of T-shaped pieces 104A and 104C or a combination of T-shaped pieces 104B and 104D can be cited. In addition, the T-shaped pieces 104A to 104D are only examples, and pieces of shapes other than T, such as I-shaped pieces, can also be used. In addition, as the second marker 104, the detection frame 100 itself can also be used. In addition, as the second marker 104, a marker of a shape in which a part of the detection frame 100 is missing can also be used. In this way, the second marker 104 can be any marker as long as it can identify the position of the organ region 27 in the ultrasonic image 24 and at least a part of the detection frame 100 is formed in a manner that can be visually identified.

[0177] In the above embodiment, a rectangular frame is exemplified as the detection frame 100 , but this is only an example, and a frame of another shape may be used.

[0178] In the above embodiment, the first mark 102 is emphasized more than the second mark 104 by making the line of the first mark 102 thicker than the line of the second mark 104 and making the size of the L-shaped pieces 102A to 102D larger than the size of the T-shaped pieces 104A to 104D, but this is only an example. For example, the second mark 104 may be displayed thinner than the first mark 102. In addition, the first mark 102 may be displayed in color and the second mark 104 may be displayed in a neutral color. In addition, the first mark 102 may be displayed in a more conspicuous line type than the second mark 104. In this way, any display method may be used as long as the first mark 102 is emphasized more than the second mark 104.

[0179] In the above embodiment, an example of detecting the lesion region 25 and the organ region 27 by an AI method (i.e., an example of detecting the lesion region 25 and the organ region 27 based on the learned model 78) is given for explanation, but the technology of the present invention is not limited thereto, and the lesion region 25 and the organ region 27 may be detected by a non-AI method. As a detection method by a non-AI method, for example, a detection method using template matching may be given.

[0180] In the above embodiment, the lesion region 25 and the organ region 27 are detected based on the learned model 78, but the lesion region 25 and the organ region 27 may be detected based on different learned models. In this case, for example, the lesion region 25 may be detected based on the learned model obtained by learning the model specifically for detecting the lesion region 25, and the organ region 27 may be detected based on the learned model obtained by learning the model specifically for detecting the organ region 27.

[0181] In the above embodiment, the ultrasonic endoscope 12 is exemplified, but the technology of the present invention can also be applied to an external ultrasonic diagnostic apparatus.

[0182] In the above-mentioned embodiment, an example of displaying the ultrasonic image 24 and the mark generated by the processing device 18 on the screen 26 of the display device 14 is given, but the ultrasonic image 24 marked with the mark may be sent to various devices such as a server, a PC and / or a tablet terminal and stored in the memory of the various devices. Furthermore, the ultrasonic image 24 marked with the mark may be recorded in a report. Furthermore, the detection frame 98 and / or the detection frame 100 may also be stored in the memory of various devices, or may be recorded in a report. Furthermore, the lesion location determination information 94 and / or the organ location determination information 96 may be stored in the memory of various devices, or may be recorded in a report. The ultrasonic image 24, the mark, the lesion location determination information 94, the organ location determination information 96, the detection frame 98 and / or the detection frame 100 are preferably stored in the memory or recorded in the report for each subject 22.

[0183] In the above-mentioned embodiment, the diagnosis support processing is performed by the processing device 18 as an example, but the technology of the present invention is not limited to this. The diagnosis support processing can be performed by the processing device 18 and at least one device provided outside the processing device 18, or it can be performed only by at least one device provided outside the processing device 18 (for example, an auxiliary processing device connected to the processing device 18 and used to expand the functions of the processing device 18).

[0184] As an example of at least one device disposed outside the processing device 18, a server can be cited. The server can be implemented by cloud computing. Cloud computing is only an example, and network computing such as fog computing, edge computing, or grid computing can also be used. In addition, the server cited as at least one device disposed outside the processing device 18 is only an example, and instead of the server, at least one PC and / or at least one host, etc., can be used, or at least one server, at least one PC and / or at least one host, etc. can also be used.

[0185] In the above embodiment, the doctor 20 is made to perceive the presence or absence of a lesion and the location of the lesion, but the doctor 20 may also be made to perceive the type of lesion and / or the degree of progression of the lesion. In this case, the model 80 may learn the ultrasonic image 24 in a state where the lesion annotation 92 includes information that can identify the type of lesion and / or the degree of progression of the lesion.

[0186] In the above embodiment, the doctor 20 is made to perceive the presence and position of an organ, but the doctor 20 may also be made to perceive the type of organ, etc. In this case, the model 80 may be made to learn the ultrasonic image 24 in a state where the organ annotation 90 includes information that can identify the type of organ, etc.

[0187] In the above embodiment, the example in which the processing device 18 detects lesions and organs is described, but the lesions and / or organs may be detected by a device other than the processing device 18 (for example, a server or PC, etc.).

[0188] In the above embodiment, an example in which the diagnostic support program 76 is stored in the NVM 66 is given for explanation, but the technology of the present invention is not limited to this. For example, the diagnostic support program 76 can be stored in a portable storage medium such as an SSD or a USB memory. The storage medium is a non-temporary computer-readable storage medium. The diagnostic support program 76 stored in the storage medium is installed in the computer 54. The processor 62 performs diagnostic support processing according to the diagnostic support program 76.

[0189] In the above embodiment, the computer 54 is exemplified, but the technology of the present invention is not limited thereto, and a device including ASIC, FPGA and / or PLD may be applied instead of the computer 54. Furthermore, a combination of hardware structure and software structure may be used instead of the computer 54.

[0190] As hardware resources for executing the diagnostic support processing described in the above-mentioned embodiments, various processors as shown below can be used. As a processor, for example, a general-purpose processor that functions as a hardware resource for executing the diagnostic support processing by executing software, that is, a program can be cited. In addition, as a processor, for example, a processor having a circuit structure specially designed for executing a specific processing, that is, a dedicated circuit, such as an FPGA, a PLD, or an ASIC can be cited. A memory is built in or connected to any processor, and any processor executes the diagnostic support processing by using the memory.

[0191] The hardware resource for performing the diagnosis support processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (e.g., a combination of multiple FPGAs, or a combination of a processor and an FPGA). Furthermore, the hardware resource for performing the medical diagnosis processing may be a single processor.

[0192] As an example of a system composed of one processor, there is a method in which one processor is composed of a combination of one or more processors and software, and the processor functions as a hardware resource for executing the diagnostic support process. There is a method in which a processor is used, such as a SoC, to realize the functions of an entire system including multiple hardware resources for executing the diagnostic support process by using one IC chip. In this way, the diagnostic support process is realized by using one or more of the above-mentioned various processors as hardware resources.

[0193] Moreover, as the hardware structure of these various processors, more specifically, a circuit composed of a combination of circuit elements such as semiconductor elements can be used. In addition, the above-mentioned diagnostic support processing is only an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced within the scope of the main purpose.

[0194] The above-described contents and illustrated contents are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-described structure, function, action, and effect are descriptions related to an example of the structure, function, action, and effect of the parts involved in the technology of the present invention. Therefore, it is self-evident that unnecessary parts can be deleted, new elements can be added, or replacement can be made to the above-described contents and illustrated contents without departing from the scope of the main purpose of the technology of the present invention. In addition, in order to avoid complexity and facilitate understanding of the parts involved in the technology of the present invention, in the above-described contents and illustrated contents, on the basis of being able to implement the technology of the present invention, descriptions related to technical common sense that does not need to be specifically explained are omitted.

[0195] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are connected and expressed with "and / or", the same thinking as "A and / or B" is also applied.

[0196] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually stated.

Claims

1. A diagnostic support device comprising a processor, The processor performs the following processing: Acquiring ultrasound images; causing a display device to display the acquired ultrasonic image; displaying, in the ultrasonic image, a first marker capable of identifying a lesion region detected from the ultrasonic image and a second marker capable of identifying an organ region detected from the ultrasonic image, in the ultrasonic image; The first mark is displayed in a more emphasized state than the second mark.

2. The diagnostic support device according to claim 1, wherein: The first mark is a mark capable of identifying the outer edge of a first range where the lesion area exists.

3. The diagnostic support device according to claim 2, wherein: The first range is defined by a first rectangular frame surrounding the lesion area.

4. The diagnostic support device according to claim 3, wherein: The first rectangular frame is a rectangular frame circumscribing the lesion area.

5. The diagnostic support device according to claim 3, wherein: The first mark is a mark formed so that at least a portion of the first rectangular frame can be visually identified.

6. The diagnostic support device according to claim 3, wherein: The first rectangular frame surrounds the lesion area in a frontal view rectangle. The first markers are a plurality of first images assigned to a plurality of corners including at least diagonal corners among four corners of the first rectangular frame.

7. The diagnostic support device according to claim 1, wherein: The second marker is a marker capable of identifying an outer edge of a second range in which the organ region exists.

8. The diagnostic support device according to claim 7, wherein: The second range is defined by a second rectangular frame surrounding the organ region.

9. The diagnostic support device according to claim 8, wherein: The second rectangular frame is a rectangular frame circumscribing the organ region.

10. The diagnostic support device according to claim 8, wherein: The second mark is a mark formed so that at least a portion of the second rectangular frame can be visually identified.

11. The diagnostic support device according to claim 8, wherein: The second rectangular frame surrounds the organ region in a frontal view rectangle. The second markers are a plurality of second images assigned to central portions of a plurality of sides including at least the opposite sides among the four sides of the second rectangular frame.

12. The diagnostic support device according to claim 1, wherein: The ultrasonic image is a dynamic image including a plurality of frames. When N is a natural number greater than or equal to 2, The processor performs processing for displaying the first mark in the ultrasonic image when the lesion area is detected in N consecutive frames from among the plurality of frames.

13. The diagnostic support device according to claim 1, wherein: The ultrasonic image is a dynamic image including a plurality of frames. When M is a natural number greater than or equal to 2, The processor performs processing for displaying the second mark in the ultrasonic image when the organ region is detected in M ​​consecutive frames from among the plurality of frames.

14. The diagnostic support device according to claim 1, wherein: The ultrasonic image is a dynamic image including a plurality of frames. When N and M are natural numbers greater than or equal to 2, The processor performs the following processing: When the lesion area is detected in N consecutive frames from among the plurality of frames, displaying the first mark in the ultrasonic image; When the organ region is detected in M ​​consecutive frames from among the plurality of frames, the second mark is displayed in the ultrasonic image. N is a value smaller than M.

15. The diagnostic support device according to claim 1, wherein: The processor performs the following processing: when the lesion area is detected, the processor notifies the detection of the lesion area by causing the sound reproducing device to output sound and / or causing the vibration generator to generate vibration.

16. The diagnostic support device according to claim 1, wherein: The processor performs the following processing: causing the display device to display a plurality of pictures including a first picture and a second picture; Displaying the ultrasonic image on the first screen and the second screen; The first mark and the second mark are displayed in the ultrasonic image on the first screen and in the ultrasonic image on the second screen, respectively.

17. The diagnostic support device according to claim 1, wherein: The processor detects the lesion region and the organ region from the ultrasonic image.

18. An ultrasonic endoscope comprising: The diagnostic support device according to claim 1; and The ultrasonic endoscope main body is connected to the diagnosis support device.

19. A diagnostic support method comprising the following steps: Acquiring ultrasound images; causing a display device to display the acquired ultrasonic image; and displaying, in the ultrasonic image, a first marker capable of identifying a lesion region detected from the ultrasonic image and a second marker capable of identifying an organ region detected from the ultrasonic image, in the ultrasonic image; The first mark is displayed in a more emphasized state than the second mark.

20. A program for causing a computer to execute a process, wherein: The process comprises the following steps: Acquiring ultrasound images; causing a display device to display the acquired ultrasonic image; and displaying, in the ultrasonic image, a first marker capable of identifying a lesion region detected from the ultrasonic image and a second marker capable of identifying an organ region detected from the ultrasonic image, in the ultrasonic image; The first mark is displayed in a more emphasized state than the second mark.

Images