Control device, medical support system, and medical support method

Abstract

Provided are a control device, a medical support system, and a medical support method that make it possible to efficiently create examination reports. The control device determines whether a second examination image captured after the time when a first examination image was captured has been acquired, and, if the second examination image has been acquired, determines whether the second examination image includes second organ-recognition information in which a portion of an organ included in the second examination image is recognized, and, if it is determines that the second organ-recognition information is present, switches from a first input screen to a second input screen, and displays the second input screen on a display unit, the second input screen including the second examination image, one or more attachment candidate images to be attached to an examination report according to the second organ-recognition information, and qualitative diagnostic item information regarding multiple finding items corresponding to the second organ-recognition information.

Description

Control Device, Medical Support System, and Medical Support Method

[0001] The present disclosure relates to a control device, a medical support system, and a medical support method.

[0002] Conventionally, in an endoscope system, there is a technique for outputting an inspection report simultaneously with an endoscope examination. In Patent Document 1, based on a captured image, a site in the large intestine is identified, the number of polyps is aggregated for each site, and then an endoscope image based on the captured image and the aggregation result of the polyps aggregated for each site are displayed on a display device (see, for example, Patent Document 1).

[0003] Japanese Patent No. 6883662

[0004] By the way, in an endoscope examination, a medical practitioner such as a doctor makes a diagnosis during the examination of a patient, and after the examination, the medical practitioner creates an examination report by inputting data from a terminal such as a personal computer. The creation work of this examination report depends on the memory of the medical practitioner.

[0005] Furthermore, in an endoscope examination, in addition to the medical practitioner, an assistant such as a nurse who assists the medical practitioner conducts the examination of the patient using an endoscope while cooperating with the medical practitioner. This assistant does not assist the medical practitioner at all times during the examination, and there has been a demand for a technology that can efficiently create examination support during the examination while relying on the help of the assistant.

[0006] However, in the above-mentioned Patent Document 1, neither an endoscope image based on a captured image nor an aggregation result of polyps aggregated for each site is output, and no consideration has been given to improving the efficiency of creating an examination report.

[0007] The present disclosure has been made in view of the above, and an object thereof is to provide a control device, a medical support system, and a medical support method capable of efficiently creating an examination report.

[0008] To solve the above-mentioned problems and achieve the objective, the control device according to the present disclosure is a control device that assists in inputting findings information to a patient's examination report, and comprises a processor, which causes the display unit to display a first input screen including one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the parts of organs included in the first examination images are recognized, determines whether a second examination image taken after the imaging time of the first examination image has been acquired, and if the second examination image has been acquired, determines whether the second examination image contains second organ recognition information in which the parts of organs included in the second examination image are recognized, and if it is determined that the second organ recognition information exists, switches from the first input screen to display on the display unit a second input screen including the second examination image, one or more attachment candidate images corresponding to the second organ recognition information for attachment to the examination report, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

[0009] Furthermore, in the control device relating to the present disclosure, if the processor determines that there is no second organ recognition information, it deems that the same organ region as the first examination image has been imaged and displays a third input screen on the display unit, which includes a second examination image in a identifiable display mode different from the display mode of the first examination image, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

[0010] Furthermore, in the control device relating to this disclosure, the processor causes the display unit to display one of the text, graphic, or icon corresponding to the first organ recognition information superimposed on the second examination image.

[0011] Furthermore, in the control device relating to the present disclosure, when the processor receives a selection signal to select one of the plurality of finding items, it obtains diagnostic items and numerical conditions corresponding to the finding items corresponding to the selection signal based on the trained model recorded in the trained model database, the image analysis assumption information for each qualitative diagnosis recorded in the image analysis assumption database, and the selection signal, extracts one or more candidate images to be attached according to the diagnostic items and numerical information from the group of examination images recorded in the examination image database, and switches from the first input screen to the display unit to display a fourth input screen including the candidate images to be attached.

[0012] Furthermore, in the control device relating to this disclosure, the processor extracts candidate images from the inspection image database that are not attached to other findings information.

[0013] Furthermore, in the control device relating to the present disclosure, the processor extracts from the examination image database the candidate images to be attached to other findings information, which have multiple lesions, and switches from the first input screen to display the fifth input screen, which includes the candidate images to be attached having multiple lesions, on the display unit.

[0014] Furthermore, in the control device relating to the present disclosure, the processor extracts from the inspection image database at least one of inspection images taken within a predetermined time from the acquisition time of the second inspection image and a predetermined number of inspection images from the second inspection image as the attached candidate images.

[0015] Furthermore, the control device relating to this disclosure is a medical support device as described in the above disclosure.

[0016] Furthermore, the medical support system relating to this disclosure includes a control device equipped with a processor that assists in inputting findings information for a patient's examination report, and a terminal device usable by an assistant assisting the surgeon. The processor displays a first input screen on the display unit of the terminal device, which includes one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the parts of organs included in the first examination images are recognized. The processor determines whether a second examination image taken after the imaging time of the first examination image has been acquired. If the second examination image has been acquired, the processor determines whether second organ recognition information, in which the parts of organs included in the second examination image are recognized, is present in the second examination image. If it is determined that the second organ recognition information is present, the processor switches from the first input screen to display a second input screen on the display unit, which includes the second examination image, one or more candidate images to be attached to the examination report according to the second organ recognition information, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

[0017] Furthermore, in the medical support system relating to this disclosure, if the processor determines that there is no second organ recognition information, it deems that the same organ region as the first examination image has been imaged and displays a third input screen on the display unit, which includes a second examination image in a identifiable display mode different from the display mode of the first examination image, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

[0018] Furthermore, in the medical support system relating to this disclosure, the processor overlays one of the text, graphic, or icon corresponding to the first organ recognition information onto the second examination image and displays it on the display unit.

[0019] Furthermore, in the medical support system relating to this disclosure, when the processor receives a selection signal to select one of the plurality of finding items, it obtains diagnostic items and numerical conditions corresponding to the finding items corresponding to the selection signal based on the trained model recorded in the trained model database, the image analysis assumption information for each qualitative diagnosis recorded in the image analysis assumption database, and the selection signal, extracts one or more candidate images to attach according to the diagnostic items and numerical information from the group of examination images recorded in the examination image database, and switches from the first input screen to the display unit to display a fourth input screen including the candidate images to attach.

[0020] Furthermore, in the medical support system relating to this disclosure, the processor extracts candidate images from the examination image database that are not yet attached to other findings information.

[0021] Furthermore, in the medical support system relating to this disclosure, the processor extracts from the examination image database the candidate images that have been attached to other findings information and have multiple lesions, and switches from the first input screen to display a fifth input screen containing the candidate images having multiple lesions on the display unit.

[0022] Furthermore, in the medical support system relating to this disclosure, the processor extracts from the examination image database at least one of the examination images taken within a predetermined time from the acquisition time of the second examination image and a predetermined number of examination images from the second examination image as the attached candidate images.

[0023] Furthermore, the medical support method relating to this disclosure is a medical support method executed by a control device equipped with a processor that assists in inputting findings information for a patient's examination report, wherein the processor displays a first input screen on a display unit, which includes one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the parts of organs included in the first examination images are recognized; determines whether a second examination image taken after the imaging time of the first examination image has been acquired; if the second examination image has been acquired, determines whether second organ recognition information in which the parts of organs included in the second examination image are recognized is present in the second examination image; if it is determined that the second organ recognition information is present, switches from the first input screen to display a second input screen on the display unit, which includes the second examination image, one or more attachment candidate images corresponding to the second organ recognition information for attachment to the examination report, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

[0024] Furthermore, in the medical support method relating to the present disclosure, if the processor determines that there is no second organ recognition information, it will switch from the first input screen to the display unit and display a third input screen on the display unit, which includes the second examination image in a identifiable display mode different from the display mode of the first examination image, which is assumed to be an image of the same organ part as the first examination image, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

[0025] Furthermore, in the medical support method relating to this disclosure, the processor superimposes one of the following onto the second examination image and displays it on the display unit: text, a figure, or an icon corresponding to the first organ recognition information.

[0026] Furthermore, in the medical support method relating to this disclosure, when the processor receives a selection signal to select one of the plurality of finding items, it obtains diagnostic items and numerical conditions corresponding to the finding items corresponding to the selection signal based on the trained model recorded in the trained model database, the image analysis assumption information for each qualitative diagnosis recorded in the image analysis assumption database, and the selection signal, extracts one or more candidate images to attach according to the diagnostic items and numerical information from the group of examination images recorded in the examination image database, and switches from the first input screen to the display unit to display a fourth input screen including the candidate images to attach.

[0027] Furthermore, in the medical support method relating to this disclosure, the processor extracts candidate images from the examination image database that are not attached to other findings information.

[0028] According to this disclosure, the process of creating inspection reports can be made more efficiently.

[0029] Figure 1 is a diagram showing the overall configuration of a medical support system according to one embodiment. Figure 2 is a block diagram showing the functional configuration of the main parts of an endoscope and control device according to one embodiment. Figure 3 is a block diagram showing the functional configuration of an image diagnostic device according to one embodiment. Figure 4 is a block diagram showing the functional configuration of a medical support device according to one embodiment. Figure 5 is a block diagram showing the functional configuration of a terminal device according to one embodiment. Figure 6 is a flowchart showing an overview of the processing performed by the medical support device according to one embodiment. Figure 7 is a diagram showing a subflowchart showing an overview of the recognition display processing in Figure 6. Figure 8 is a diagram showing an example of a display screen displayed by the terminal device according to one embodiment before the display is switched by the medical support device. Figure 9 is a diagram showing an example of a display screen displayed by the terminal device according to one embodiment after the display is switched by the medical support device. Figure 10 is a diagram showing another example of a display screen displayed by the terminal device according to one embodiment after the display is switched by the medical support device. Figure 11 is a diagram showing an overview of the extraction method extracted by the extraction unit of the medical support device according to one embodiment. Figure 12 is a diagram showing a subflowchart showing an overview of the unrecognizable display processing in Figure 6. Figure 13 is a diagram showing an example of a display screen displayed by the terminal device according to one embodiment after the display is switched by the medical support device. Figure 14 is a subflowchart showing an overview of the qualitative diagnostic item display process in Figure 6. Figure 15 is an example of a display screen displayed by a terminal device according to one embodiment before the display is switched by the medical support device. Figure 16 is an example of image analysis assumption information for each qualitative diagnosis recorded by the image analysis assumption DB of the medical support device according to one embodiment. Figure 17 is an overview of the extraction method extracted by the extraction unit of the medical support device according to one embodiment. Figure 18 is an example of a display screen displayed by a terminal device according to one embodiment after the display is switched by the medical support device. Figure 19 is an example of image analysis assumption information for each qualitative diagnosis recorded by the image analysis assumption DB of the medical support device according to one embodiment. Figure 20 is an overview of the extraction method extracted by the extraction unit of the medical support device according to one embodiment. Figure 21 is an example of a display screen displayed by a terminal device according to one embodiment after the display is switched by the medical support device.

[0030] The embodiments for implementing this disclosure will be described in detail below with reference to drawings. However, this disclosure is not limited to the embodiments described below. Furthermore, the figures referenced in the following description only provide a schematic representation of the shape, size, and positional relationships to the extent that the content of this disclosure can be understood. In other words, this disclosure is not limited to the shapes, sizes, and positional relationships exemplified in the figures. In addition, the same parts will be denoted by the same reference numerals in the drawings.

[0031] [Overall Configuration of the Medical Support System] Figure 1 is a diagram showing the overall configuration of a medical support system according to one embodiment. The medical support system 1 shown in Figure 1 continuously images the inside of a subject by inserting the insertion part 21 of the endoscope 2 through the mouth of a subject such as a person or animal into the esophagus or through the anus of the subject into the large intestine, and displays the captured image data in chronological order on the display device 3.

[0032] Furthermore, the medical support system 1 detects (recognizes or estimates) feature regions including organs, body parts, polyps, or cancers in multiple observation areas captured in the image data, based on the image data group or still image data (examination images) input from the control device 4 and a trained model (inference model) that has been trained using training data obtained by previously imaging the subject's organs, body parts, and lesions, and outputs the detection results to the control device 4. The operator, such as a doctor, observes and treats the subject while confirming the displayed image on the display device 3 and the detection results (estimated results) from the image diagnostic device 5. Here, the trained model is a deep learning model that is trained using training data in which feature regions such as the subject's organs, body parts, and lesions are added to image data of the subject, such as a patient, and outputs organs, body parts, tumors, non-tumors, inflammatory activity, invasive cancers, and feature regions contained in the image data and examination images.

[0033] Furthermore, the medical support system 1 acquires image data input from the control device 4 via the network N100 to the medical support device 6, as well as examination information including the detection results (estimated results) from the image diagnostic device 5, and records the acquired image data and examination information in association. The medical support system 1 then outputs various data in response to requests from terminal devices 7 that can be operated by surgeons, assistants, etc., via the network N100.

[0034] Furthermore, the medical support system 1 generates report information such as examination reports of the subject by the surgeon and assistants at the terminal device 7, and this report information is output to the medical support device 6 and recorded. Subsequently, the surgeon makes a diagnosis of the subject and conducts conferences, etc., based on the report information and the diagnostic results of living cells obtained by biopsy during the observation of the subject.

[0035] As shown in Figure 1, the medical support system 1 comprises an endoscope 2, a display device 3, a control device 4, an image diagnostic device 5, a medical support device 6, and a terminal device 7.

[0036] [Endoscope Configuration] The endoscope 2 continuously generates image data (RAW data) by imaging the inside of the subject and sequentially outputs this image data to the control device 4. As shown in Figure 1, the endoscope 2 comprises an insertion unit 21, an operating unit 22, and a universal code 23.

[0037] The insertion portion 21 is flexible in at least a portion and is inserted into the subject. As shown in Figure 1, the insertion portion 21 comprises a tip portion 24 provided at the tip of the insertion portion 21, a bendable portion 25 connected to the base end side (operating portion 22 side) of the tip portion 24 and configured to bendable, and a long, flexible tube portion 26 connected to the base end side of the bendable portion 25 and having flexibility.

[0038] The operating section 22 is connected to the proximal end portion of the insertion section 21. The operating section 22 receives various operations on the endoscope 2. As shown in Figure 1, the operating section 22 is provided with a bending knob 221, an insertion opening 222, and a plurality of operating members 223.

[0039] The bending knob 221 is configured to be rotatable in response to user operation by a user such as a surgeon. By rotating the bending knob 221, a bending mechanism (not shown) made of metal or resin wire disposed within the insertion portion 21 is activated. As a result, the bending portion 25 bends.

[0040] The insertion port 222 communicates with a treatment instrument channel (not shown), which is a conduit extending from the tip of the insertion section 21, and is an insertion port for inserting treatment instruments, etc., into the treatment instrument channel from outside the endoscope 2.

[0041] Multiple operating members 223 are composed of buttons and the like that accept various operations from a user such as an operator, and output operation signals corresponding to these operations to the control device 4 via the universal code 23. Examples of these operations include a release operation to instruct still images from the endoscope 2, and an operation to switch the observation mode of the endoscope 2 to normal light observation mode or special observation mode.

[0042] The universal cord 23 extends from the operation section 22 in a direction different from the extension direction of the insertion section 21 and is a cord on which a light guide 231 (see Figure 2) made of optical fiber or the like, a first signal line 232 (see Figure 2) for transmitting the image data mentioned above, a second signal line 233 (see Figure 2) for transmitting the operation signals mentioned above, etc., are arranged. A first connector section 27 is provided at the base end of the universal cord 23, as shown in Figure 1. The first connector section 27 is detachably connected to the control device 4.

[0043] The display device 3 is composed of a display monitor such as a liquid crystal or organic EL (Electro-Luminescence), and under the control of the control device 4, it displays a display image based on image data processed by the control device 4, as well as various information related to the endoscope 2.

[0044] The control device 4 is realized by using a processor which is a processing device having hardware such as a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or a CPU (Central Processing Unit), and a memory which is a temporary storage area used by the processor. The control device 4 comprehensively controls the operations of each part of the endoscope 2 according to the program recorded in the memory.

[0045] The image diagnostic device 5 (CAD) uses the image data group input from the control device 4 and the learned model learned in advance with learning data, and detects features including organs, parts, polyps or tumors such as cancer and non-tumors in the image data (abnormal areas or lesion candidate areas), calculates inflammation activity evaluation information for evaluating inflammation activity, and outputs to the control device 4 the inspection information in which the estimation result including the determination result of whether it is invasive cancer or not is associated with the estimation time of this estimation result. The image diagnostic device 5 is realized by using a processor which is a processing device having hardware such as a GPU, an FPGA, or a CPU, and a memory which is a temporary storage area used by the processor.

[0046] The medical support device 6 sequentially records various data input from the control device 4 and the recognition results of the image diagnostic device 5, and transmits data according to requests from the terminal device 7. The medical support device 6 is realized by using a processor which is a processing device having hardware such as a GPU, an FPGA, or a CPU, and a memory which is a temporary storage area used by the processor. Further, the medical support device 6 is configured by using an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0047] The terminal device 7 is configured by using a mobile phone, a tablet terminal, etc., and displays various information transmitted from the medical support device 6 via the network N100 on the display unit 71. Also, the terminal device 7 transmits various information input by the user to the medical support device 6. Note that the terminal device 7 may be a laptop personal computer or the like as long as it can be connected to the medical support device 6 via the network N100.

[0048] [Functional Configuration of the Main Parts of the Medical Support System] Next, the functional configuration of the main parts of the above-described medical support system 1 will be described. FIG. 2 is a block diagram showing the functional configuration of the main parts of the endoscope 2 and the control device 4. Hereinafter, the endoscope 2 and the control device 4 will be described in this order, and then the image diagnostic device 5 and the medical support device 6 will be described.

[0049] [Functional Configuration of the Endoscope] First, the configuration of the endoscope 2 will be described. As shown in FIG. 2, the endoscope 2 includes an illumination optical system 201, an imaging optical system 202, an imaging element 203, an A / D conversion unit 204, a P / S conversion unit 205, an imaging recording unit 206, and an imaging control unit 207. Here, each of the illumination optical system 201, the imaging optical system 202, the imaging element 203, the A / D conversion unit 204, the P / S conversion unit 205, the imaging recording unit 206, and the imaging control unit 207 is disposed inside the distal end portion 24.

[0050] The illumination optical system 201 is composed of one or more lenses or the like, and irradiates the illumination light supplied from the light guide 231 toward the subject.

[0051] The imaging optical system 202 is composed of one or more lenses or the like, and forms a subject image on the light receiving surface of the imaging element 203 by condensing the reflected light reflected from the subject, the return light from the subject, the fluorescence or the like emitted from the subject, and the like. Further, the imaging optical system 202 can change the focal length and the angle of view by moving the lens along the optical axis direction O1.

[0052] The image sensor 203 is configured using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, in which one of the color filters constituting a Bayer array (RGB) is placed at each of a plurality of pixels arranged in a two-dimensional matrix. Under the control of the imaging control unit 207, the image sensor 203 receives the subject image formed by the imaging optical system 202, performs photoelectric conversion, and generates an captured image (analog signal). In this embodiment, the image sensor 203 may be configured integrally with an image sensor and a TOF (Time Of Flight) sensor that acquires subject distance information (hereinafter referred to as depth map information) using the TOF method. Depth map information is information in which the subject distance from the position of the image sensor 203 (position of the tip 24) to the corresponding position on the observation target corresponding to the pixel position in the captured image is detected for each pixel position. The configuration for generating depth map information is not limited to the TOF sensor described above, but may also include an image sensor including a phase difference sensor. In the following, depth map information and captured images will be collectively referred to as image data. The image sensor 203 outputs the image data to the A / D conversion unit 204.

[0053] The A / D conversion unit 204 is configured using an A / D conversion circuit and the like. Under the control of the imaging control unit 207, the A / D conversion unit 204 performs A / D conversion processing on the analog image data input from the image sensor 203 and outputs it to the P / S conversion unit 205.

[0054] The P / S conversion unit 205 is configured using a P / S conversion circuit or the like, and under the control of the imaging control unit 207, it performs parallel / serial conversion of the digital image data input from the A / D conversion unit 204 and outputs it to the control device 4 via the first signal line 232. Alternatively, instead of the P / S conversion unit 205, an E / O conversion unit that converts image data into an optical signal may be provided, and the image data may be output to the control device 4 using the optical signal. Furthermore, the image data may be transmitted to the control device 4 by wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0055] The imaging and recording unit 206 is composed of non-volatile memory and volatile memory, and records various information related to the endoscope 2 (for example, pixel information of the image sensor 203). The imaging and recording unit 206 also records various setting data and control parameters transmitted from the control device 4 via the second signal line 233.

[0056] The imaging control unit 207 is implemented using a TG (Timing Generator), a processor which is a processing unit having hardware such as a CPU, and memory which is a temporary storage area used by the processor. Based on the setting data received from the control unit 4 via the second signal line 233, the imaging control unit 207 controls the operation of the image sensor 203, the A / D conversion unit 204, and the P / S conversion unit 205, respectively.

[0057] [Configuration of the control device] Next, the configuration of the control device 4 will be described. As shown in Figure 2, the control device 4 includes a condensing lens 401, a first light source unit 402, a second light source unit 403, a light source control unit 404, an S / P conversion unit 405, an image processing unit 406, an input unit 407, a recording unit 408, a communication unit 409, and a control unit 410.

[0058] The condensing lens 401 collects the light emitted by the first light source unit 402 and the second light source unit 403, and emits it to the light guide 231.

[0059] The first light source unit 402, under the control of the light source control unit 404, emits visible white light (normal light) to supply the light guide 231 as illumination light. This first light source unit 402 is composed of a collimating lens, a white LED (Light Emitting Diode) lamp, and a drive driver, etc. The first light source unit 402 may also be composed of a red LED lamp, a green LED lamp, and a blue LED lamp that emit light simultaneously to supply visible white light. Alternatively, the first light source unit 402 may be composed of a halogen lamp or a xenon lamp, etc.

[0060] The second light source unit 403, under the control of the light source control unit 404, emits special light having a predetermined wavelength band to supply special light to the light guide 231 as illumination light. Here, the special light is light used for narrow-band imaging (NBI) using narrow-band light including 390-445 nm and 530-550 nm. Of course, in addition to narrow-band light, amber-colored light (600 nm and 630 nm) used for red dichromatic imaging (RDI) may also be used as special light.

[0061] The light source control unit 404 is implemented using a processor having hardware such as an FPGA or CPU, and memory, which is a temporary storage area used by the processor. Based on control data input from the control unit 410, the light source control unit 404 controls the light emission timing and light emission duration of the first light source unit 402 and the second light source unit 403, respectively.

[0062] The S / P conversion unit 405, under the control of the control unit 410, performs serial / parallel conversion on the image data received from the endoscope 2 via the first signal line 232 and outputs it to the image processing unit 406. Note that if the endoscope 2 outputs image data as an optical signal, an O / E conversion unit that converts optical signals to electrical signals may be provided instead of the S / P conversion unit 405. Also, if the endoscope 2 transmits image data via wireless communication, a communication module capable of receiving wireless signals may be provided instead of the S / P conversion unit 405.

[0063] The image processing unit 406 is implemented using a processor with hardware such as a GPU or FPGA, and memory, which is a temporary storage area used by the processor. Under the control of the control unit 410, the image processing unit 406 performs predetermined image processing on the parallel data image data input from the S / P conversion unit 405 and outputs it to the display device 3. Examples of predetermined image processing include demosaicing, white balance processing, gain adjustment processing, gamma correction processing, and format conversion processing.

[0064] The input unit 407 is configured using a mouse, foot switch, keyboard, buttons, switches, and touch panel, and receives user operations from a user such as a surgeon, and outputs an operation signal corresponding to the user operation to the control unit 410.

[0065] The recording unit 408 is configured using recording media such as volatile memory, non-volatile memory, SSDs, HDDs, and memory cards. The recording unit 408 records data including various parameters necessary for the operation of the control device 4 and the endoscope 2. The recording unit 408 also includes a program recording unit 408a for recording various programs for operating the endoscope 2 and the control device 4, and an image data recording unit 408b for recording image data.

[0066] The image data recording unit 408b, under the control of the control unit 410, records the image data group generated by the endoscope 2 continuously imaging multiple observation sites of the subject, still image data, and the estimation results of characteristic regions including organs, body parts, polyps or cancers in the observation site in the examination image based on still image data (captured image data and examination images) estimated by the image diagnostic device 5, as well as the imaging time, in association with patient information, etc.

[0067] The communication unit 409, under the control of the control unit 410, transmits various information to the medical support device 6 via the image diagnostic device 5 and the network N100, and also receives various information from the image diagnostic device 5 and the medical support device 6 and outputs it to the control unit 410. Specifically, under the control of the control unit 410, the communication unit 409 transmits a series of temporally continuous image data processed by the image processing unit 406 to the image diagnostic device 5, and receives the estimation results, including organs, body parts, and characteristic regions estimated by the image diagnostic device 5, and outputs them to the control unit 410. The communication unit 41, under the control of the control unit 410, transmits the image data series, still image data, and examination information, including the estimation results from the image diagnostic device 5, to the medical support device 6. Here, the examination information includes the estimation results estimated by the image diagnostic device 5, examination images, and information including the time the examination images were captured. The communication unit 409 is configured using a communication module or the like.

[0068] The control unit 410 corresponds to a second processor according to this disclosure. This control unit 410 is implemented using a second processor, which is a processing device having hardware such as an FPGA or CPU, and memory, which is a temporary storage area used by the second processor. The control unit 410 comprehensively controls each part that constitutes the endoscope 2 and the control device 4. The control unit 410 includes an imaging control unit 410a, a display control unit 410b, and a communication control unit 410c.

[0069] The imaging control unit 410a controls the imaging of the endoscope 2. The imaging control unit 410a generates a series of temporally continuous image data by causing the endoscope 2 to continuously capture images. In addition, when a release signal is input from the operation unit 22, the imaging control unit 410a causes the image sensor 203 to capture still image data. Furthermore, the imaging control unit 410a acquires image data from the endoscope 2 via the S / P conversion unit 405 or the image processing unit 406 and records it in the image data recording unit 408b.

[0070] The display control unit 410b displays the observation image corresponding to the image data captured by the endoscope 2 on the display device 3.

[0071] The communication control unit 410e causes the communication unit 409 to transmit various information to the medical support device 6 and the image diagnostic device 5 via the network N100, and also causes the communication unit 409 to receive various information from the medical support device 6 and the image diagnostic device 5.

[0072] [Functional Configuration of the Medical Imaging Device] Next, the functional configuration of the medical imaging device 5 will be described. Figure 3 is a block diagram showing the functional configuration of the medical imaging device 5. The medical imaging device 5 shown in Figure 3 comprises a communication unit 51, an input unit 52, a recording unit 53, a display unit 54, and a control unit 55.

[0073] The communication unit 51, under the control of the control unit 55, receives a group of image data generated by the endoscope 2 from the control device 4 via the network N100 and outputs it to the control unit 55. The communication unit 51 also, under the control of the control unit 55, outputs examination information to the control device 4 via the network N100, including the detection results of the detected organs of the subject, and the detection time at which these results were detected. The communication unit 51 is configured using a communication module or the like.

[0074] The input unit 52 is configured using buttons, switches, and a touch panel, and receives various inputs and outputs them to the control unit 55.

[0075] The recording unit 53 is configured using recording media such as volatile memory, non-volatile memory, SSDs, HDDs, and memory cards. The recording unit 53 records data including various parameters necessary for the operation of the medical imaging device 5. The recording unit 53 also includes a program recording unit 531 for recording various programs for operating the medical imaging device 5, and an inference model recording unit 532.

[0076] The inference model recording unit 532 records an inference model used to detect organs, body parts, and feature regions based on image data. Here, the inference model takes image data from the endoscope 2 as input and outputs organs, body parts, tumors, non-tumors, inflammatory activity, invasive cancer, and feature regions contained in the image data and examination images. This inference model is, for example, a model generated by machine learning using artificial intelligence (AI). Specifically, the inference model is a deep learning model that learns using training data in which feature regions such as organs, body parts, and lesions of a subject are added to image data of a subject such as a patient, and outputs organs, body parts, tumors, non-tumors, inflammatory activity, invasive cancer, and feature regions contained in the image data and examination images.

[0077] The display unit 54 displays various information related to the medical imaging device 5 under the control of the control unit 55. The display unit 54 is configured using a liquid crystal display, an organic EL display, or the like.

[0078] The control unit 55 is implemented using a processor having hardware such as an FPGA or CPU, and memory, which is a temporary storage area used by the processor. The control unit 55 comprehensively controls each part that constitutes the medical support device 6. The control unit 55 includes an acquisition unit 551, an organ / site estimation unit 552, a tumor / non-tumor estimation unit 533, an inflammation activity evaluation unit 554, and an invasive cancer estimation unit 555.

[0079] The acquisition unit 551 acquires image data or still image data that functions as an examination image, sequentially generated by the endoscope 2 from the control device 4 via the communication unit 51.

[0080] The organ / part estimation unit 552 detects the organs, parts, and characteristic regions of the subject contained in the image data and examination images, based on the image data acquired by the acquisition unit 551 and the inference model recorded by the inference model recording unit 532.

[0081] The tumor / non-tumor estimation unit 533 detects whether the subject has tumors or not in the image data or examination images based on the image data acquired by the acquisition unit 551 and the inference model recorded by the inference model recording unit 532.

[0082] The inflammation activity evaluation unit 554 evaluates the inflammatory activity of the subject contained in the image data and examination images based on the image data acquired by the acquisition unit 551 and the inference model recorded by the inference model recording unit 532.

[0083] The invasive cancer estimation unit 555 estimates the presence of invasive cancer in the subject contained in the image data and examination images, based on the image data acquired by the acquisition unit 551 and the inference model recorded by the inference model recording unit 532.

[0084] [Functional Configuration of the Medical Support Device] Next, the functional configuration of the medical support device 6 will be described. Figure 4 is a block diagram illustrating the functional configuration of the medical support device 6. The medical support device 6 shown in Figure 4 comprises a communication unit 61, a display unit 62, an input unit 63, a database 64 (hereinafter simply referred to as "DB64"), a program recording unit 65, and a control unit 66. In one embodiment, the medical support device 6 functions as a control device.

[0085] The communication unit 61, under the control of the control unit 66, transmits various information to the control device 4 and the image diagnostic device 5 via the network N100, and receives various information from the control device 4 and the image diagnostic device 5. The communication unit 61 is configured using a communication module.

[0086] The display unit 62 displays various information related to the medical support device 6 under the control of the control unit 66. The display unit 62 is constructed using a liquid crystal display, an organic EL display, or the like.

[0087] The input unit 63 receives various inputs in response to external operations and outputs them to the control unit 66. The input unit 63 is configured using buttons, a keyboard, switches, a touch panel, etc.

[0088] DB64 records various types of information. DB64 is configured using HDDs and SSDs, etc. DB64 includes patient information DB641, findings information DB642, examination image DB643, image analysis prediction DB644, and trained model DB645.

[0089] The patient information database 641 records, for each patient, the patient's identifying patient ID, date of birth, attending physician, address, and allergy information, all associated with each patient.

[0090] The findings information DB642 records findings information from doctors, etc., along with examination images and qualitative diagnostic items, for each patient ID.

[0091] The examination image database 643 records, for each patient ID, the examination image, organ recognition information related to the organs in the examination image obtained by the image diagnostic device 5, and the time of acquisition, in association with each patient ID.

[0092] The image analysis database 644 records qualitative diagnostic item information associated with each organ recognition information. Specifically, the image analysis database 644 records qualitative diagnostic item information, consisting of multiple predicted diagnostic names, associated with each of the organ recognition information for the esophagus, stomach, and large intestine. Furthermore, the image analysis database 644 records AI diagnostic items and conditions associated with each qualitative diagnostic item.

[0093] The trained model DB645 records an inference model used to estimate diagnostic items and numerical conditions based on organ recognition information associated with examination images. Here, the inference model takes organ recognition information as input and outputs diagnostic items and numerical conditions. This inference model is constructed using the deep learning model described above.

[0094] The program recording unit 65 is configured using volatile memory, non-volatile memory, and an SSD, and records various programs for operating the medical support device 6.

[0095] The control unit 66 corresponds to the first processor according to this disclosure. This control unit 66 is implemented using a first processor having hardware such as an FPGA or CPU, and memory which is a temporary storage area used by the first processor. The control unit 66 comprehensively controls each part that constitutes the medical support device 6. The control unit 66 includes an acquisition unit 661, a determination unit 662, a setting unit 663, an extraction unit 664, a display control unit 665, and an estimation unit 666.

[0096] The acquisition unit 661 acquires image data and inspection information from the control device 4 via the network N100.

[0097] The determination unit 662 determines whether a user, such as an assistant, is inputting findings information using the terminal device 7. Specifically, the determination unit 662 determines whether it has received various operation signals for inputting findings information from the terminal device 7. If it has received operation signals, it determines that the user is inputting findings information using the terminal device 7. On the other hand, if it has not received various operation signals for inputting findings information from the terminal device 7, it determines that the user is not inputting findings information using the terminal device 7.

[0098] The extraction unit 664 extracts one or more examination images with the same organ recognition information from the group of examination images recorded by the examination image DB 643 as candidate images to be attached to the examination report.

[0099] The display control unit 665 switches the display of the attached candidate images by switching from the first input screen displayed by the display unit 71 of the terminal device 7 to a second input screen which includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664.

[0100] The estimation unit 666 estimates diagnostic items and numerical conditions based on the trained model recorded in the trained model DB 645, the image analysis assumption information for each qualitative diagnosis recorded in the image analysis assumption DB 644, the qualitative diagnosis set by the user, and the organ recognition information of the examination image.

[0101] [Functional Configuration of Terminal Device] Next, the functional configuration of the terminal device 7 will be described. Figure 5 is a block diagram showing the functional configuration of the terminal device 7. The terminal device 7 comprises a display unit 71, an input unit 72, an output unit 73, a communication unit 74, a recording unit 75, and a control unit 76.

[0102] The display unit 71, under the control of the control unit 76, displays various information corresponding to various information related to the terminal device 7 and information received from the medical support device 6. The display unit 71 is constructed using a liquid crystal display or an organic EL display, etc.

[0103] The input unit 72 receives various inputs in response to external operations and outputs them to the control unit 76. The input unit 72 is configured using input interfaces such as buttons, switches, and touch panels.

[0104] The output unit 73 outputs various information under the control of the control unit 76. The output unit 73 is configured using, for example, a speaker.

[0105] The communication unit 74, under the control of the control unit 76, transmits various information to the medical support device 6 via the network N100 and receives various information from the medical support device 6. The communication unit 11 is configured using a communication module.

[0106] The recording unit 75 is composed of volatile memory, non-volatile memory, memory cards, and HDDs, SSDs, etc. The recording unit 75 records various programs executed by the terminal device 7 and data being processed.

[0107] The control unit 76 is implemented using a processor having hardware such as an FPGA, GPU, or CPU, and memory, which is a temporary storage area used by the processor. The control unit 76 controls each part that makes up the terminal device 7.

[0108] [Processing by the Medical Support Device] Next, we will explain the processing performed by the medical support device 6. Figure 6 is a flowchart showing an overview of the processing performed by the medical support device 6.

[0109] As shown in Figure 6, first, the determination unit 662 determines whether or not a user, such as an assistant, is inputting findings information using the terminal device 7 (step S101). Specifically, the determination unit 662 determines whether or not it has received various operation signals for inputting findings information from the terminal device 7. If it has received operation signals, it determines that the user is inputting findings information using the terminal device 7. On the other hand, if it has not received various operation signals for inputting findings information from the terminal device 7, it determines that the user is not inputting findings information using the terminal device 7. If the determination unit 662 determines that the user is inputting findings information using the terminal device 7 (step S101: Yes), the medical support device 6 proceeds to step S107, which will be described later. Conversely, if the determination unit 662 determines that the user is not inputting findings information using the terminal device 7 (step S101: No), the medical support device 6 proceeds to step S102, which will be described later.

[0110] In step S102, the determination unit 662 determines whether the acquisition unit 661 has acquired a newly captured examination image from the control device 4. Specifically, the determination unit 662 determines whether the acquisition unit 661 has acquired the image by receiving a newly captured examination image from the control device 4. In this case, the control device 4 may transmit organ recognition information detected by the image diagnostic device 5 for the examination image along with the examination image. If the determination unit 662 determines that the acquisition unit 661 has acquired a newly captured examination image from the control device 4 (step S102: Yes), the medical support device 6 proceeds to step S103. On the other hand, if the determination unit 662 determines that the acquisition unit 661 has not acquired a newly captured examination image from the control device 4 (step S102: No), the medical support device 6 proceeds to step S105, which will be described later.

[0111] In step S103, the determination unit 662 determines whether or not the examination image contains organ recognition information from the image diagnostic device 5. For example, the determination unit 662 determines whether or not organ recognition information from the image diagnostic device 5 is stored in the header information of the examination image. If the determination unit 662 determines that the examination image contains organ recognition information from the image diagnostic device 5 (step S103: Yes), the medical support device 6 proceeds to step S104, which will be described later. On the other hand, if the determination unit 662 determines that the examination image does not contain organ recognition information from the image diagnostic device 5 (step S103: No), the device proceeds to step S106.

[0112] In step S104, the medical support device 6 performs a recognition and display process that inputs and displays findings information according to the organ location information.

[0113] [Recognition and Display Processing] Figure 7 is a subflowchart showing an overview of the recognition and display processing in step S104.

[0114] As shown in Figure 7, the acquisition unit 661 records the organ recognition information and examination images, which are included in the newly acquired examination images from the control device 4 and estimated by the image diagnostic device 5, in the examination image DB 643 in association with patient information (step S201). Specifically, the organ recognition information and examination images are recorded in the examination image DB 643 in association with the patient ID.

[0115] Next, the setting unit 663 acquires qualitative diagnostic item information for the target organ based on the qualitative diagnostic item information recorded by the image analysis assumption DB 644 and the organ recognition information acquired by the acquisition unit 661 (step S202).

[0116] Subsequently, the extraction unit 664 extracts one or more examination images with the same organ recognition information from the group of examination images recorded in the examination image DB 643 as candidate images to be attached to the examination report (step S203).

[0117] Next, the display control unit 665 switches the display of the attached candidate images by switching from the first input screen displayed by the display unit 71 of the terminal device 7 to a second input screen that includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664 (step S204). After step S204, the medical support device 6 returns to the main routine in Figure 6 and proceeds to step S105.

[0118] Here, we will explain the screen transitions of the terminal device 7 in the recognition and display process described above. Figure 8 is an example of a display screen that the terminal device 7 displays before the display switching by the medical support device 6. Figure 9 is an example of a display screen that the terminal device 7 displays after the display switching by the medical support device 6. Figure 10 is another example of a display screen that the terminal device 7 displays after the display switching by the medical support device 6.

[0119] As shown in Figure 8, the terminal device 7 displays an input screen P1 (first input screen) that includes patient information T1, organ recognition information B1 for inputting findings for an examination report, an attachment image area W1 containing at least one attachment candidate image P10, and a qualitative diagnostic item area M1 containing qualitative diagnostic item information Q1 of the target organ, in accordance with the information transmitted from the medical support device 6. Furthermore, organ information A1 is superimposed on each attachment candidate image. Organ information A1 can be an icon, text, or graphic. The attachment image area W1 also includes a cursor K1 that accepts an instruction signal that allows transition to displaying other attachment candidate images. The user displays the attachment candidate images present in the display area of ​​the display unit 71 by operating the cursor K1.

[0120] When the terminal device 7 is displaying the input screen P1 shown in Figure 8, and the medical support device 6 acquires a new examination image from the control device 4, the terminal device 7 switches from the input screen P1 to the input screen P2 and displays it on the display unit 71, as shown in Figure 9, according to the information transmitted from the medical support device 6.

[0121] The input screen P2 includes an attached image area W1 which includes organ recognition information B2 estimated by the image diagnostic device 5, an examination image P21 and at least one or more attached candidate images P22 extracted by the extraction unit 664 according to the organ recognition information B2, and a qualitative diagnostic item area M1 which includes qualitative diagnostic item information Q2 of the target organ according to the organ recognition information B2 acquired by the setting unit 663. Furthermore, organ-related information A2 is superimposed on each attached candidate image P21, P22.

[0122] Furthermore, if the organ is the same as the organ recognition information B2 in Figure 8, and the image diagnostic device 5 recognizes a specific region of that organ, then, as shown in Figure 10, the terminal device 7 switches from input screen P1 to input screen P3 and displays the information on the display unit 71 according to the information transmitted from the medical support device 6.

[0123] The input screen P3 includes an attached image area W1 which includes organ recognition information B3 estimated by the image diagnostic device 5, an examination image P21 and at least one attached candidate image P22 extracted by the extraction unit 664 according to the organ recognition information B2, and a qualitative diagnostic item area M1 which includes qualitative diagnostic item information Q3 of the target organ according to the organ recognition information B3 acquired by the setting unit 663.

[0124] Next, we will describe an extraction method in which the extraction unit 664 extracts at least one or more candidate images. Figure 11 is a diagram showing an overview of the extraction method performed by the extraction unit 664.

[0125] As shown in Figure 11, the extraction unit 664 extracts, based on the imaging time of the inspection images, inspection images from the group of inspection images recorded by the inspection image DB 643 that satisfy at least one of the following conditions: inspection images taken within a predetermined time period (e.g., within 10 minutes) from the imaging time of the inspection image, and inspection images taken within a predetermined number of images (e.g., within 5 images) from the inspection image taken immediately before, as candidate images to attach. In the case shown in Figure 11, the extraction unit 664 extracts a group of 5 inspection images G1 that are within a predetermined number of images from the inspection image taken immediately before, and then extracts a group of 4 inspection images G2 taken within a predetermined time period (e.g., within 10 minutes) from the imaging time (11:03:10) as candidate images to attach to the inspection image group G1.

[0126] In Figure 11, the extraction unit 664 extracts a predetermined number of inspection image group G1 from the inspection image captured immediately before, and then extracts four inspection image group G2 from the inspection image group G1 within a predetermined time from the time of acquisition, for example, within 10 minutes. However, the system is not limited to this, and the system may also extract a predetermined number of inspection image group G1 from the inspection image captured immediately before as attachment candidate images, or extract an inspection image group G2 within a predetermined time from the time of acquisition as attachment candidate images.

[0127] In this way, the display control unit 665 switches the display of the attached candidate images from the input screen P1 (first input screen) displayed by the display unit 71 of the terminal device 7 to an input screen P2 (second input screen) which includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664, each time the acquisition unit 661 acquires a new examination image from the control device 4. As a result, each time a new examination image is captured, the medical support device 6 switches the input screen (first input screen) displayed by the display unit 71 of the terminal device 7 to an input screen P2 (second input screen) which includes qualitative diagnostic item information of the target organ according to the organ recognition information of the examination image and one or more attached candidate images, thereby reducing the burden of data input on the user during the examination.

[0128] Returning to Figure 6, we will continue the explanation from step S105 onwards. In step S105, if the input of findings information is completed (step S105: Yes), the medical support device 6 terminates this process. On the other hand, if the input of findings information is not completed (step S105: No), the medical support device 6 returns to step S101 as described above.

[0129] In step S106, the medical support device 6 performs an unrecognizable display process that estimates organ location information and displays the input of findings information.

[0130] [Recognition Unavailable Display Process] Figure 12 is a subflowchart showing an overview of the recognition unavailable display process in step S106.

[0131] As shown in Figure 12, the acquisition unit 661 considers the organ currently displayed by the display unit 71 of the terminal device 7 to be the organ of the current examination image, records the organ recognition information of this organ in the examination image DB 643 in association with patient information, and also records the examination image in the examination image DB 643 in association with patient information (step S301).

[0132] Next, the setting unit 663 acquires qualitative diagnostic item information for the target organ based on the qualitative diagnostic item information recorded by the image analysis assumption DB 644 and the organ recognition information B1 (e.g., esophagus) of the previous examination image (step S302). In this case, since the determination unit 662 cannot recognize the organ recognition information of the examination image acquired by the acquisition unit 661, the setting unit 663 considers the organ recognition information of the current examination image as the organ recognition information B1 (e.g., esophagus) of the previous examination image and acquires qualitative diagnostic item information for the target organ from the qualitative diagnostic item information recorded by the image analysis assumption DB 644. That is, the setting unit 663 acquires qualitative diagnostic item information corresponding to the organ recognition information of the current examination image from the image analysis assumption DB 644.

[0133] Subsequently, the extraction unit 664 extracts one or more examination images with the same organ recognition information from the group of examination images recorded in the examination image DB 643 as candidate images to be attached to the examination report (step S303).

[0134] Next, the display control unit 665 switches the display of the attached candidate images by switching the input screen (third input screen) displayed by the display unit 71 of the terminal device 7 to an input screen (first input screen) that includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664 (step S304). After step S304, the medical support device 6 returns to the main routine in Figure 6 and proceeds to step S105.

[0135] Here, we will explain the screen transitions of the terminal device 7 in the unrecognizable display process described above. Figure 13 shows an example of a display screen that the terminal device 7 displays after the display switching by the medical support device 6.

[0136] When the terminal device 7 is displaying the input screen P1 in Figure 8, and the medical support device 6 acquires a new examination image from the control device 4, as shown in Figure 13, the terminal device 7 switches from the input screen P1 (first input screen) to the input screen P4 (third input screen) and displays it on the display unit 71 according to the information transmitted from the medical support device 6.

[0137] The input screen P4 includes an attached image area W1 containing organ recognition information B1 about the organ in the immediately preceding examination image estimated by the image diagnostic device 5, the examination image P31 and at least one or more attached candidate images P32 extracted by the extraction unit 664 according to the organ recognition information B1, and a qualitative diagnostic item area M1 containing qualitative diagnostic item information Q1 of the target organ according to the organ recognition information B1 acquired by the setting unit 663. In this case, the icon A3 superimposed on the examination image P31 is displayed in a different display mode than the display mode of other icons A1. This allows the user to intuitively understand that the examination image P31 has not had organ recognition information recognized by the image diagnostic device 5.

[0138] In this way, the display control unit 665 switches the display of the attached candidate images by switching the input screen P4 (third input screen), which includes the qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664, from the input screen (first input screen) displayed by the display unit 71 of the terminal device 7. As a result, each time a new examination image is captured, the medical support device 6 switches the input screen displayed by the display unit 71 of the terminal device 7 to an input screen that includes the qualitative diagnostic item information of the target organ corresponding to the organ recognition information of the examination image and one or more attached candidate images, thereby reducing the burden of data input on the user during the examination. Furthermore, if there is no organ recognition information for the examination image from the image diagnostic device 5, the medical support device 6 considers the organ recognition information of the previous examination image as the current organ recognition information and sets the qualitative diagnostic item information of the target organ corresponding to this considered organ recognition information, thereby reducing the burden of data input on the user during the examination.

[0139] Returning to Figure 6, we will continue the explanation from step S107 onwards. In step S107, the determination unit 662 determines whether or not the user has already selected candidate images to attach to the inspection report. If the determination unit 662 determines that the user has already selected candidate images to attach to the inspection report (step S107: Yes), the medical support device 6 proceeds to step S105. On the other hand, if the determination unit 662 determines that the user has not yet selected candidate images to attach to the inspection report (step S107: No), the medical support device 6 proceeds to step S108, which will be described later.

[0140] In step S108, the medical support device 6 performs qualitative diagnostic item processing, which involves inputting and displaying findings information associated with a qualitative diagnosis.

[0141] [Qualitative Diagnostic Item Processing] Figure 14 is a subflowchart showing an overview of the qualitative diagnostic item display processing in step S108.

[0142] As shown in Figure 14, the acquisition unit 661 records the organ recognition information and examination images, which are included in the newly acquired examination images from the control device 4 and estimated by the image diagnostic device 5, in the examination image DB 643 in association with patient information (step S401).

[0143] Next, the acquisition unit 661 acquires the qualitative diagnostic item information of the input target organ set by the setting unit 663 and records it in the examination image DB 643 (step S402).

[0144] Subsequently, the estimation unit 666 estimates diagnostic items and numerical conditions based on the trained model recorded in the trained model DB 645, the image analysis assumption information for each qualitative diagnosis recorded in the image analysis assumption DB 644, the qualitative diagnosis set by the user, and the organ recognition information of the examination image (step S403).

[0145] Next, the extraction unit 664 extracts images that meet the conditions from the group of examination images recorded in the examination image DB 643, based on the diagnostic items, numerical conditions, and organ recognition information estimated by the estimation unit 666, as candidate images to be attached (step S404).

[0146] Subsequently, the display control unit 665 switches the display of the attached candidate images by switching the input screen (fourth input screen) displayed by the display unit 71 of the terminal device 7 to an input screen (first input screen) that includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664 (step S204). After step S204, the medical support device 6 returns to the main routine in Figure 6 and proceeds to step S105.

[0147] Here, we will explain the screen transitions of the terminal device 7 during the qualitative diagnostic item processing described above. Figure 15 shows an example of a display screen that the terminal device 7 displays before the display is switched by the medical support device 6.

[0148] As shown in Figure 15, the terminal device 7 displays an input screen P4 (first input screen) in accordance with the information transmitted from the medical support device 6, which includes patient information T1, organ recognition information B1 for inputting findings for the examination report, an attached image area H1 including at least one candidate attached image P10, and a qualitative diagnostic item area M1 including qualitative diagnostic item information of the target organ.

[0149] Under the circumstances shown in Figure 15, when the target organ recognition information B1 is "stomach", and the user operates the terminal device 7 and selects a gastric fundic gland polyp in the qualitative diagnostic item area M1, the estimation unit 666 estimates the diagnostic items and numerical conditions based on the trained model recorded by the trained model DB 645, the qualitative diagnosis set by the user, and the organ recognition information of the examination image.

[0150] Figure 16 shows an example of image analysis prediction information for each qualitative diagnosis recorded by the image analysis prediction DB 644. In the image analysis prediction information U1 shown in Figure 16, AI diagnostic items and conditions are associated with each qualitative diagnostic item. In the case shown in Figure 16, when the user selects a gastric fundic gland polyp in the qualitative diagnostic item area M1, the estimation unit 666 estimates the AI ​​diagnostic item as "tumor / non-tumor discrimination" and the condition "the probability of tumor is 50% or more" based on the trained model recorded by the trained model DB 645, the qualitative diagnosis set by the user, and the organ recognition information of the examination image.

[0151] Next, we will describe an extraction method in which the extraction unit 664 extracts at least one or more attached candidate images. Figure 17 is a diagram showing an overview of the extraction method performed by the extraction unit 664.

[0152] As shown in Figure 17, the extraction unit 664 extracts images that satisfy each condition from the group of examination images recorded in the examination image DB 643 as candidate images to be attached, based on the AI ​​qualitative items, numerical conditions, and organ recognition information estimated by the estimation unit 666. In the case shown in Figure 17, the extraction unit 664 extracts examination images P103 and P105 from the group of examination images P100 as candidate images to be attached, which are examination images not attached to the findings information of the examination report, whose organ recognition information is "stomach", and which have a probability of tumor of 50% or more.

[0153] Subsequently, when the terminal device 7 is displaying the input screen P5 in Figure 15, it switches from the input screen P5 to an input screen corresponding to the qualitative diagnosis selected by the user, according to the information transmitted from the medical support device 6.

[0154] Figure 18 shows an example of a display screen that the terminal device 7 displays after the display is switched by the medical support device 6. As shown in Figure 18, the terminal device 7 switches from input screen P5 to input screen P6 according to the information transmitted from the display control unit 665 of the medical support device 6.

[0155] The input screen P6 includes organ recognition information B2 estimated by the image diagnostic device 5, attached candidate images P51 and P52 extracted by the extraction unit 664 in the attached image area W1, and a qualitative diagnostic item area M1 containing qualitative diagnostic item information of the target organ selected by the user. In this case, the display control unit 665 transmits the qualitative diagnostic item information selected by the user to the terminal device 7 in a display manner that can be distinguished from other qualitative diagnostic item information, while maintaining the display manner from the state of the input screen P5. Furthermore, the display control unit 665 superimposes organ-related information A2 onto each of the attached candidate images P51 and P52.

[0156] As a result, the medical support device 6, according to the qualitative diagnostic information selected by the user, extracts images that satisfy each condition from the group of examination images recorded in the examination image DB 643, based on the AI ​​qualitative items, numerical conditions, and organ recognition information estimated by the estimation unit 666, as attached candidate images P51 and P52, and the display control unit 665 switches to the input screen P6 which has been switched to the attached candidate images P51 and P52 extracted by the extraction unit 664, thereby reducing the burden of data input for the user during the examination.

[0157] The extraction unit 664 extracted unattached images, specifically examination image P103 and examination image P105, each showing a single lesion, as candidate images for attachment. However, it is not limited to this; for example, even attached images may be extracted as candidate images if they show multiple lesions.

[0158] Figure 19 shows an example of image analysis prediction information for each qualitative diagnosis recorded by the image analysis prediction DB 644. In the image analysis prediction information U2 shown in Figure 19, AI diagnostic items and conditions are associated with each qualitative diagnostic item. In the case shown in Figure 19, when the user selects a gastric fundic gland polyp in the qualitative diagnostic item area M1, the estimation unit 666 estimates the AI ​​diagnostic item as "tumor / non-tumor discrimination" and the condition "the probability of tumor is 50% or more" based on the trained model recorded by the trained model DB 645, the qualitative diagnosis set by the user, and the organ recognition information of the examination image.

[0159] Next, we will describe an extraction method in which the extraction unit 664 extracts at least one or more attached candidate images. Figure 20 is a diagram showing an overview of the extraction method performed by the extraction unit 664.

[0160] As shown in Figure 20, the extraction unit 664 extracts images that satisfy each condition from the group of examination images recorded in the examination image DB 643 as candidate images to be attached, based on the AI ​​qualitative items, numerical conditions, and organ recognition information estimated by the estimation unit 666. In the case shown in Figure 17, the extraction unit 664 extracts from the group of examination images P100, even though it has already been attached to the findings information of the examination report, examination image P104 which contains multiple lesions, and examination images P103 and P105 whose organ recognition information is "stomach" and for which the possibility of tumor is 50% or more, as candidate images to be attached.

[0161] Subsequently, when the terminal device 7 is displaying the input screen P5 in Figure 15, it switches from the input screen P5 to an input screen corresponding to the qualitative diagnosis selected by the user, according to the information transmitted from the medical support device 6.

[0162] Figure 21 shows an example of a display screen that the terminal device 7 displays after the display is switched by the medical support device 6. As shown in Figure 21, the terminal device 7 switches from input screen P5 to input screen P7 according to the information transmitted from the display control unit 665 of the medical support device 6.

[0163] The input screen P7 includes an attached image area W1 containing organ recognition information B2 estimated by the image diagnostic device 5, attached candidate images P61 and P62 extracted by the extraction unit 664 in the attached image area W1, and a qualitative diagnostic item area M1 containing qualitative diagnostic item information of the target organ selected by the user.

[0164] In this way, the display control unit 665 displays on the input screen P7 whether it is an already attached image or a candidate image for attachment when there are multiple lesions. As a result, the medical support device 6 can reduce the burden on the user in terms of data input during the examination.

[0165] According to the embodiment described above, the display control unit 665 switches the display of the attached candidate images by switching the input screen P2 (second input screen), which includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664, from the input screen P1 (first input screen) displayed by the display unit 71 of the terminal device 7. As a result, each time a new examination image is captured, the medical support device 6 switches the input screen (first input screen) displayed by the display unit 71 of the terminal device 7 to the input screen P2 (second input screen), which includes qualitative diagnostic item information of the target organ according to the organ recognition information of the examination image and one or more attached candidate images, thereby reducing the burden of data input on the user during the examination.

[0166] Furthermore, according to one embodiment, the display control unit 665 switches the display of the attached candidate images by switching the input screen P4 (third input screen), which includes qualitative diagnostic item information of the target organ acquired by the setting unit 663 and one or more attached candidate images extracted by the extraction unit 664, from the input screen (first input screen) displayed by the display unit 71 of the terminal device 7. As a result, each time a new examination image is captured, the medical support device 6 switches the input screen displayed by the display unit 71 of the terminal device 7 to an input screen that includes qualitative diagnostic item information of the target organ according to the organ recognition information of the examination image and one or more attached candidate images, thereby reducing the burden of data input on the user during the examination. Moreover, if there is no organ recognition information of the examination image from the image diagnostic device 5, the medical support device 6 considers the organ recognition information of the immediately preceding examination image as the current organ recognition information and sets the qualitative diagnostic item information of the target organ according to this considered organ recognition information, thereby reducing the burden of data input on the user during the examination.

[0167] Furthermore, according to one embodiment, the medical support device 6, in accordance with the qualitative diagnostic information selected by the user, extracts images that satisfy each condition from the group of examination images recorded in the examination image DB 643, based on the AI ​​qualitative items, numerical conditions, and organ recognition information estimated by the estimation unit 666, as attached candidate images P51 and P52, and the display control unit 665 switches to the input screen P6 which has been switched to the attached candidate images P51 and P52 extracted by the extraction unit 664, thereby reducing the burden of data input on the user during the examination.

[0168] Furthermore, according to one embodiment, the display control unit 665 displays on the input screen P7 whether it is an already attached image or a candidate image for attachment when there are multiple lesions. This allows the medical support device 6 to reduce the burden of data input on the user during the examination.

[0169] In one embodiment, the diagnostic imaging device 5 recognized (estimated) organs, body parts, tumors, non-tumors, inflammatory activity, invasive cancer, and characteristic regions in the examination image (image data). However, the device is not limited to this, and the functions of the diagnostic imaging device 5 may be provided in the control unit 66 of the medical support device 6. Specifically, the control unit 66 may be provided with an acquisition unit 551, an organ / body part estimation unit 552, a tumor / non-tumor estimation unit 533, an inflammatory activity evaluation unit 554, and an invasive cancer estimation unit 555.

[0170] Various inventions can be formed by appropriately combining the multiple components disclosed in the medical support system according to one embodiment of the present disclosure described above. For example, some components may be removed from all the components described in the medical support system according to the embodiment of the present disclosure described above. Furthermore, the components described in the medical support system according to the embodiment of the present disclosure described above may be appropriately combined.

[0171] Furthermore, in the medical support system according to one embodiment of this disclosure, the "parts" described above can be replaced with "means" or "circuits," etc. For example, the control unit can be replaced with a control means or a control circuit.

[0172] Furthermore, the program to be executed by the medical support system according to one embodiment of this disclosure is provided as installable or executable file data recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), USB medium, or flash memory.

[0173] Furthermore, the program to be executed by the medical support system according to one embodiment of this disclosure may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0174] In this specification, while the flowchart descriptions use expressions such as "first," "then," and "followed by" to indicate the sequence of processes between steps, the order of processes necessary to carry out the present invention is not uniquely determined by these expressions. That is, the order of processes in the flowcharts described herein can be changed within a reasonable range. Furthermore, the program is not limited to simple branching processes; it may also branch by comprehensively evaluating a larger number of decision items.

[0175] Although some embodiments of this application have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the present invention.

[0176] 1 Medical support system 2 Endoscope 3 Display device 4 Control device 5 Diagnostic imaging device 6 Medical support device 7 Terminal device 11, 41, 51, 61, 74 Communication unit 16, 55, 66, 76 Control unit 52, 63, 72 Input unit 53, 75 Recording unit 54, 62, 71 Display unit 64 Database 65 Program recording unit 661 Acquisition unit 662 Judgment unit 663 Setting unit 664 Extraction unit 665 Display control unit 666 Estimation unit 641 Patient information DB 642 Findings information DB 643 Examination image DB 644 Image analysis assumption DB 645 Trained model DB

Claims

1. A control device for assisting in the input of findings information for a patient's examination report, comprising a processor, wherein the processor displays a first input screen on a display unit, the first input screen including one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the locations of organs included in the first examination images are recognized; determines whether a second examination image taken after the imaging time of the first examination image has been acquired; if the second examination image has been acquired, determines whether second organ recognition information in which the locations of organs included in the second examination image are recognized is present in the second examination image; if it is determined that the second organ recognition information is present, switches from the first input screen to display a second input screen on the display unit, the second input screen including the second examination image, one or more candidate images to be attached to the examination report according to the second organ recognition information, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

2. A control device according to claim 1, wherein, when the processor determines that there is no second organ recognition information, it switches from the first input screen to the display unit and displays a third input screen which includes a second examination image in a distinguishable display mode different from the display mode of the first examination image, which is determined to be an image of the same organ region as the first examination image, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

3. A control device according to claim 2, wherein the processor causes the display unit to display one of the text, figures, and icons corresponding to the first organ recognition information superimposed on the second examination image.

4. A control device according to claim 3, wherein the processor, upon receiving a selection signal to select one of the plurality of finding items, acquires diagnostic items and numerical conditions corresponding to the finding items corresponding to the selection signal based on a trained model recorded in a trained model database, image analysis assumption information for each qualitative diagnosis recorded in an image analysis assumption database, and the selection signal; extracts one or more candidate images to be attached according to the diagnostic items and numerical information from a group of examination images recorded in an examination image database; and switches from the first input screen to display a fourth input screen including the candidate images to display the display unit.

5. A control device according to claim 4, wherein the processor extracts candidate images from the examination image database that are not attached to other findings information.

6. A control device according to claim 5, wherein the processor extracts from the examination image database the candidate images to be attached to other findings information and which have multiple lesions, and switches from the first input screen to display the fifth input screen which includes the candidate images to be attached and which have multiple lesions.

7. A control device according to claim 6, wherein the processor extracts from the inspection image database at least one of inspection images taken within a predetermined time from the acquisition time of the second inspection image and a predetermined number of inspection images from the second inspection image as the attached candidate images.

8. A control device according to claim 7, wherein the control device is a medical support device.

9. A medical support system comprising: a control device equipped with a processor that assists in inputting findings information for a patient's examination report; and a terminal device usable by an assistant assisting the operator, wherein the processor causes the display unit of the terminal device to display a first input screen including one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the locations of organs included in the first examination images are recognized; determines whether a second examination image taken after the imaging time of the first examination image has been acquired; if the second examination image has been acquired, determines whether the second examination image contains second organ recognition information in which the locations of organs included in the second examination image are recognized; and if it is determined that the second organ recognition information exists, switches from the first input screen to display a second input screen on the display unit, which includes the second examination image, one or more candidate images to be attached to the examination report according to the second organ recognition information, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

10. A medical support system according to claim 9, wherein, if the processor determines that there is no second organ recognition information, it switches from the first input screen to the display unit and displays a third input screen which includes a second examination image in a distinguishable display mode different from the display mode of the first examination image, which is determined to be an image of the same organ part as the first examination image, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

11. A medical support system according to claim 10, wherein the processor causes the display unit to display one of the text, figures, and icons corresponding to the first organ recognition information superimposed on the second examination image.

12. A medical support system according to claim 11, wherein the processor, upon receiving a selection signal for selecting one of the plurality of finding items, acquires a diagnostic item and numerical conditions corresponding to the finding item corresponding to the selection signal based on a trained model recorded in a trained model database, image analysis assumption information for each qualitative diagnosis recorded in an image analysis assumption database, and the selection signal; extracts one or more candidate images for attachment corresponding to the diagnostic item and numerical information from a group of examination images recorded in an examination image database; and switches from the first input screen to display a fourth input screen including the candidate images for attachment on the display unit.

13. A medical support system according to claim 12, wherein the processor extracts candidate images from the examination image database that are not attached to other findings information.

14. A medical support system according to claim 13, wherein the processor extracts from the examination image database the candidate images to be attached to other findings information and which have multiple lesions, and switches from the first input screen to display the fifth input screen which includes the candidate images to be attached and which have multiple lesions.

15. A medical support system according to claim 14, wherein the processor extracts from the examination image database at least one of examination images taken within a predetermined time from the acquisition time of the second examination image and a predetermined number of examination images from the second examination image as attached candidate images.

16. A medical support method performed by a control device equipped with a processor and supporting the input of findings information for a patient's examination report, wherein the processor displays a first input screen on a display unit, which includes one or more first examination images and qualitative diagnostic item information relating to a plurality of qualitative diagnostic items corresponding to first organ recognition information in which the parts of organs included in the first examination images are recognized; determines whether a second examination image taken after the imaging time of the first examination image has been acquired; if the second examination image has been acquired, determines whether second organ recognition information in which the parts of organs included in the second examination image are recognized is present in the second examination image; and if it is determined that the second organ recognition information is present, switches from the first input screen to display a second input screen on the display unit, which includes the second examination image, one or more candidate images to be attached to the examination report according to the second organ recognition information, and qualitative diagnostic item information relating to a plurality of findings items corresponding to the second organ recognition information.

17. A medical support method according to claim 16, wherein if the processor determines that there is no second organ recognition information, it switches from the first input screen to the display unit and displays a third input screen which includes a second examination image in a distinguishable display mode different from the display mode of the first examination image, assuming that the same part of the organ as the first examination image has been imaged, one or more attached candidate images corresponding to the first organ recognition information, and qualitative diagnostic item information relating to a plurality of finding items corresponding to the first organ recognition information.

18. A medical support method according to claim 17, wherein the processor overlays one of the text, figures, and icons corresponding to the first organ recognition information onto the second examination image and displays it on the display unit.

19. A medical support method according to claim 18, wherein when the processor receives a selection signal for selecting one of the plurality of finding items, it obtains a diagnostic item and numerical conditions corresponding to the finding item corresponding to the selection signal based on a trained model recorded in a trained model database, image analysis assumption information for each qualitative diagnosis recorded in an image analysis assumption database, and the selection signal; extracts one or more candidate images to attach according to the diagnostic item and numerical information from a group of examination images recorded in an examination image database; and switches from the first input screen to the display unit to display a fourth input screen including the candidate images to attach.

20. A medical support method according to claim 19, wherein the processor extracts candidate images from the examination image database that are not attached to other findings information.

Images