Program, information processing method, and information processing apparatus

Abstract

A program for causing a computer to execute processing, in the computer, an image captured by an endoscope is acquired, in a case where an image captured by the endoscope is input, the acquired image is input to a learned model in which learning has been completed, so as to output a position of an attention region included in the image, a position of an attention region included in the acquired image is acquired from the learned model, and an enlarged image in which a portion of the image including the attention region is enlarged is output in accordance with the acquired position of the attention region.

Description

Technical Field

[0001] This technology relates to a program, an information processing method, and an information processing device.

[0002] This application claims priority based on Japanese Application No. 2021-034569, filed on March 4, 2021, and incorporates all disclosures set forth in that Japanese application. Background Technology

[0003] A computer-aided diagnostic technique has been developed that automatically detects lesions from medical images such as endoscopic images using a learning model. A method for generating a learning model is known using teacher-based machine learning that employs teacher data labeled with correct answers. This invention discloses a learning model that combines a first learning method using a set of images taken by a conventional endoscope as teacher data with a second learning method using a set of images taken by a capsule endoscope as teacher data (e.g., Patent Document 1).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1 International Publication No. 2017 / 175282 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, the computer-aided diagnostic technology described in Patent Document 1 has the following problem: when the input image contains a region of interest (ROI), it does not consider diagnostic support from the perspective of magnifying and displaying the region of interest.

[0009] In one aspect, the aim is to provide a program, etc., for effectively magnifying and displaying the region of interest in an endoscopic image.

[0010] Technical solutions for solving the problem

[0011] In one embodiment of this disclosure, the program causes a computer to perform the following processes: in the computer, acquiring an image taken by an endoscope; when the image taken by the endoscope is input, inputting the acquired image into a learned model that has been learned so as to output the position of the region of interest contained in the image; obtaining the position of the region of interest contained in the acquired image from the learned model; and outputting an enlarged image of the portion of the image containing the region of interest based on the obtained position of the region of interest.

[0012] One embodiment of the information processing method of this disclosure causes a computer to perform the following processing: acquiring an image taken by an endoscope; inputting the acquired image into a learned model after the image taken by the endoscope has been input, so as to output the position of the region of interest contained in the image; obtaining the position of the region of interest contained in the acquired image from the learned model; and outputting an enlarged image containing a portion of the image including the region of interest based on the obtained position of the region of interest.

[0013] An information processing apparatus according to one embodiment of this disclosure includes: an image acquisition unit that acquires an image captured by an endoscope; an input unit that, when the image captured by the endoscope is input, inputs the acquired image into a learned model so as to output the position of a region of interest contained in the image; a position acquisition unit that acquires the position of the region of interest contained in the acquired image from the learned model; and an output unit that, based on the acquired position of the region of interest, outputs a magnified image of a portion of the image including the region of interest.

[0014] Invention Effects

[0015] According to this disclosure, a program or similar tool may be provided for effectively magnifying and displaying the region of interest in an endoscopic image. Attached Figure Description

[0016] Figure 1 This is a schematic diagram showing an outline of the endoscope system according to Embodiment 1.

[0017] Figure 2 This is a block diagram illustrating a structural example of an endoscope device included in an endoscope system.

[0018] Figure 3 This is a block diagram illustrating a structural example of an information processing device included in an endoscope system.

[0019] Figure 4 This is a functional block diagram illustrating the functional units included in the control section of an information processing device.

[0020] Figure 5 This is an explanatory diagram about the learned model (the region-of-interest learning model).

[0021] Figure 6 This is an illustration showing one way of displaying a magnified image.

[0022] Figure 7 This is a flowchart illustrating an example of the processing steps performed by the control unit of an information processing device.

[0023] Figure 8 This is an explanatory diagram showing one way of displaying a magnified image related to Embodiment 2 (multiple regions of interest).

[0024] Figure 9 This is a flowchart illustrating an example of the processing steps performed by the control unit of an information processing device.

[0025] Figure 10 This is a functional block diagram of the functional units included in the control unit of the information processing device according to Embodiment 3 (Second Learned Model).

[0026] Figure 11 This is an explanatory diagram about the second learned model (diagnostic support learning model).

[0027] Figure 12 This is a flowchart illustrating an example of the processing steps performed by the control unit of an information processing device. Detailed Implementation

[0028] (Implementation Method 1)

[0029] The invention will be described in detail below with reference to the accompanying drawings illustrating its embodiments. Figure 1 This is a schematic diagram illustrating an outline of the diagnostic support system S according to Embodiment 1. The diagnostic support system S includes: an endoscope device 10, and an information processing device 6 communicatively connected to the endoscope device 10.

[0030] The endoscope device 10 transmits the images (captured images) captured by the camera element of the endoscope 40 to the endoscope processor 20. The endoscope processor 20 performs various image processing operations, such as gamma correction, white balance correction, and shadow correction, to generate an endoscope image that is easy for the operator to view. The endoscope device 10 outputs (sends) the generated endoscope image to the information processing device 6. The information processing device 6, having acquired the endoscope images sent from the endoscope device 10, performs various information processing operations on these endoscope images, extracts information related to the region of interest (ROI) contained in the endoscope image (ROI information), and generates a magnified image of the ROI based on the ROI information, and outputs it to the endoscope device 10 (endoscope processor 20). The ROI is the area of ​​interest for the doctor or other operator of the endoscope 40, such as the area where lesions, lesion candidates, medications, treatment instruments, and markers are located. A magnified image of the region of interest output from the information processing device 6 is displayed on the display device 50 connected to the endoscope device 10.

[0031] The endoscope device 10 includes an endoscope processor 20, an endoscope 40, and a display device 50. The display device 50 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0032] The display device 50 is located on the upper shelf of the wheeled storage rack 16. The endoscope processor 20 is stored in the middle shelf of the storage rack 16. The storage rack 16 is positioned near the endoscope examination bed (not shown). The storage rack 16 has a drawer-type shelf for holding the keyboard 15 connected to the endoscope processor 20.

[0033] The endoscope processor 20 is generally rectangular in shape and has a touch panel 25 on one side. A reading unit 28 is arranged below the touch panel 25. The reading unit 28 is an interface for reading and writing portable recording media, such as a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive.

[0034] The endoscope 40 has an insertion section 44, an operating section 43, a universal flexible cable 49, and an observer connector 48. The operating section 43 is equipped with a control button 431. The insertion section 44 is elongated, with one end connected to the operating section 43 via a bend-stop section 45. From the operating section 43 side, the insertion section 44 has a flexible section 441, a bending section 442, and a front end 443 in sequence. The bending section 442 bends in response to the operation of the bending knob 433. Physical detection devices such as a 3-axis accelerometer, a gyroscope sensor, a geomagnetic sensor, a magnetic coil sensor, or an endoscope insertion shape observation device (Colonavi) are installed in the insertion section 44, and the detection results of these physical detection devices can be obtained when the endoscope 40 is inserted into the body of the subject.

[0035] The general-purpose flexible cable 49 is elongated, with one end connected to the operating part 43 and the second end connected to the observer connector 48. The general-purpose flexible cable 49 is flexible. The observer connector 48 is approximately cuboid in shape. The observer connector 48 has air and water supply ports 36 for connecting air and water supply pipes (see reference). Figure 2 ).

[0036] Figure 2This is a block diagram illustrating a structural example of the endoscope device 10 included in the diagnostic support system S. The control unit 21 is an arithmetic control device for executing the program in this embodiment. The control unit 21 uses one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), or multi-core CPUs, etc. The control unit 21 is connected to the various hardware components constituting the endoscope processor 20 via a bus.

[0037] The main storage device 22 is, for example, an SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main storage device 22 temporarily stores information required for processing by the control unit 21, as well as the program currently being executed in the control unit 21. The auxiliary storage device 23 is, for example, an SRAM, flash memory, or hard disk, and is a storage device with a larger capacity than the main storage device 22. For example, acquired images and generated endoscopic images can be stored as intermediate data in the auxiliary storage device 23.

[0038] The communication unit 24 is a communication module or interface for communicating with the information processing device 6 via a network, either wired or wirelessly. Examples include narrowband wireless communication modules such as Wi-Fi and Bluetooth, or broadband wireless communication modules such as 4G, LTE, and 5G. The touch panel 25 includes a display unit such as a liquid crystal display panel and an input unit superimposed on the display unit. The communication unit 24 can communicate with CT devices, MRI devices (see reference...) Figure 5 Communication can be made using a storage device (not shown) to store data output from these devices.

[0039] Display device I / F26 is an interface for connecting the endoscope processor 20 and the display device 50. Input device I / F27 is an interface for connecting the endoscope processor 20 and input devices such as the keyboard 15.

[0040] The light source 33 includes high-brightness white light sources such as white LEDs and xenon lamps, as well as special light sources such as narrow-band LEDs that emit narrow-band light. The light source 33 is connected to a bus via a driver (not shown). The lighting, extinguishing, and brightness changes of the light source 33 are controlled by the control unit 21. Illumination light from the light source 33 is incident on the optical connector 312. The optical connector 312 engages with the observer connector 48 and provides illumination light to the endoscope 40.

[0041] Pump 34 generates pressure for supplying air and water to the endoscope 40. Pump 34 is connected to a bus via a driver (not shown). The on / off state of pump 34 and pressure changes are controlled by control unit 21. Pump 34 is connected to air and water supply ports 36 provided on the observer connector 48 via water supply tank 35.

[0042] The following is a summary of the functions of the endoscope 40 connected to the endoscope processor 20. Fiber optic cables, cable bundles, air supply tubes, and water supply tubes are inserted inside the observer connector 48, universal flexible cable 49, operation section 43, and insertion section 44. Illumination light emitted from the light source 33 is emitted through the illumination window provided on the front end portion 443 via the optical connector 312 and the fiber optic cable. The area illuminated by the illumination light is captured by an imaging element provided on the front end portion 443. The captured images are transmitted from the imaging element to the endoscope processor 20 via the cable bundle and electrical connector 311.

[0043] The control unit 21 of the endoscope processor 20 executes the program stored in the main storage device 22, thereby performing the function of the image processing unit 211. The image processing unit 211 performs various image processing on the image (captured image) output from the endoscope 40, such as gamma correction, white balance correction, and shadow correction, and outputs it as an endoscope image.

[0044] Figure 3 This is a block diagram illustrating an example structure of the information processing device 6 included in the diagnostic support system S. The information processing device 6 includes a control unit 62, a communication unit 61, a storage unit 63, and an input / output (I / O) unit 64. The information processing device 6 can be, for example, a server device, a personal computer, etc. The server device includes not only a single server device but also a cloud server device or a virtual server device composed of multiple computers. The information processing device 6 can also be configured as a cloud server, located on an external network accessible from the endoscope processor 20.

[0045] The control unit 62 has one or more arithmetic processing devices with timing functions, such as CPU (Central Processing Unit), MPU (Micro-Processing Unit), and GPU (Graphics Processing Unit). By reading and executing the program stored in the storage unit 63, it can perform various information processing and control processing involving the information processing device 6.

[0046] The storage unit 63 includes volatile storage areas such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory, as well as non-volatile storage areas such as EEPROM or hard disk. Programs and data to be referenced during processing are pre-stored in the storage unit 63. The programs stored in the storage unit 63 can also be programs read from a recording medium that can be read from the information processing device 6. Alternatively, programs can be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 63.

[0047] The storage unit 63 stores actual files (instance files of neural networks (NNs)) that constitute the learned model 631 (region of interest learning model) described later. These actual files can also be part of the program. The storage unit 63 also stores various preset values ​​(preset data) that are predetermined when generating and outputting (displaying) the magnified image. This preset data may include, for example, preset values ​​for determining the output form (main screen, sub-screen) of the magnified image, magnification for magnified display (outputting the magnified image), flag values ​​for whether to change to a special light viewing mode when magnifying the display, and predetermined values ​​(accuracy probability thresholds) for distinguishing the accuracy probability of the region of interest. Details of the preset data will be described later.

[0048] The communication unit 61 is a communication module or communication interface for communicating with the endoscope device 10 via wired or wireless means, such as a narrowband wireless communication module like Wi-Fi (registered trademark) or Bluetooth (registered trademark) or a broadband wireless communication module like 4G, LTE, or 5G.

[0049] The I / O input / output I / F64 conforms to communication standards such as USB or DSUB and is a communication interface for serial communication with external devices connected to the I / O input / output I / F64. The I / O input / output I / F64 is connected to a display unit 7 such as a monitor, and an input unit 8 such as a mouse and keyboard. The control unit 62 outputs the information processing results based on the execution commands or events input from the input unit 8 to the display unit 7.

[0050] Figure 4This is a functional block diagram illustrating the functional units included in the control unit 62 of the information processing device 6. The control unit 21 of the endoscope processor 20 (endoscope device 10) performs the function of the image processing unit 211 by executing programs stored in the main storage device 22. The image processing unit 211 of the endoscope processor 20 performs various image processing operations such as gamma correction, white balance correction, and shadow correction on the image (captured image) output from the endoscope, and outputs it as an endoscope image. The endoscope processor 20 also acquires the magnified image output from the information processing device 6 and the output format data when outputting the magnified image, and outputs the magnified image to the display device 50 based on the output format data.

[0051] The control unit 62 of the information processing device 6 performs the functions of the acquisition unit 621, the learned model 631, the magnified image generation unit 622, the output mode determination unit 623, and the output unit 624 by executing the program stored in the storage unit 63.

[0052] The acquisition unit 621 acquires the endoscopic image output by the endoscope processor 20. The acquisition unit 621 inputs the acquired endoscopic image into the learned model 631 (region of interest learning model). When the endoscopic image is output (sent) from the endoscope processor 20 to the information processing device 6 as a moving image, the acquisition unit 621 can also acquire the endoscopic image frame by frame in the moving image and input it into the learned model 631.

[0053] The learned model 631 acquires and inputs the endoscopic image output from the acquisition unit 621, and outputs information related to the region of interest contained in the input endoscopic image. The information related to the region of interest includes the location of a portion of the region of interest contained in the endoscopic image, and the accuracy probability when estimated as a region of interest. The location of the portion of the region of interest contained in the endoscopic image includes, for example, the coordinates of two points in the image coordinate system of the endoscopic image, and is determined by a rectangular frame (bounding box) formed by using these two points as diagonals. The accuracy when estimated as a region of interest is established, for example, by a class probability (estimation score) value representing the estimation accuracy of the region of interest extracted from the endoscopic image, represented by a value from 0 to 1. In this case, the closer to 1, the more likely the region of interest extracted from the endoscopic image is a true region of interest (high estimation accuracy). In this way, when the input endoscopic image contains a region of interest, the learned model 631 outputs the location and accuracy probability as information related to that region of interest; when the input endoscopic image does not contain a region of interest, it does not output any information related to that region of interest. Therefore, the learned model 631 functions as a region of interest presence / absence determination unit, which determines whether the input endoscopic image contains a region of interest. When the input endoscopic image contains a region of interest, the learned model 631 outputs information related to the region of interest (location, accuracy probability) to the magnified image generation unit 622 and the output shape determination unit 623.

[0054] Figure 5This is an explanatory diagram of the learned model 631 (Region of Interest Learning Model). The learned model 631 is a model used for object detection, such as RCNN (Regions with Convolutional Neural Network), FastRCNN, Faster RCNN, or SSD (Single Shot Multibook Detector), YOLO (You Only LookOnce), or a neural network (NN) with segmentation capabilities. In the case where the learned model 631 is composed of a neural network, such as RCNN, containing a CNN (Convolutional Neural Network) for extracting image features, the input layer of the learned model 631 has multiple neurons that accept pixel values ​​from the endoscope image as input and pass these pixel values ​​to an intermediate layer. The intermediate layer has multiple neurons that extract image features from the endoscope image and pass these extracted image features to an output layer. The output layer has one or more neurons that output information about the region of interest, including its location, and outputs the location and accuracy probability (estimated score) of the region of interest based on the image features output from the intermediate layer. Alternatively, the learned model 631 can also input the image features output from the intermediate layer into an SVM (support vector machine) for object recognition. The neural network (learned model 631) that learns using training data is assumed to be used as part of artificial intelligence software, i.e., a program module. The learned model 631 is used in an information processing device 6, which, as described above, includes a control unit 62 (CPU, etc.) and a storage unit 63. Thus, the information processing device 6, possessing computational processing capabilities, performs related operations, thereby constituting a neural network system. Specifically, the control unit 62 of the information processing device 6 performs calculations based on instructions from the learned model 631 stored in the storage unit 63 to extract features from the endoscopic image input to the input layer, and outputs the location and accuracy probability (estimated score) of the region of interest from the output layer.

[0055] The magnified image generation unit 622, having acquired information related to the region of interest from the learned model 631, extracts (cuts out) a portion of the endoscopic image containing the region of interest based on the location of the region of interest contained in the information. The magnified image generation unit 622 generates a magnified image of the endoscopic image containing the region of interest by magnifying the extracted (cut out) portion. For example, the magnified image generation unit 622 may also use electronic zoom (digital zoom) to generate the magnified image of the region of interest. This electronic zoom (digital zoom) allows for the cutting out and supplementary magnification of a portion of the endoscopic image (including the region of interest), and through software processing, a magnified image can be effectively generated. For example, the magnified image generation unit 622 generates the magnified image based on the magnification ratio contained in preset data stored in the storage unit 63. For example, the magnification ratio may be a fixed value such as 5x, or it may be determined based on the number of pixels in the portion of the endoscopic image containing the extracted (cut out) region of interest. When determining the magnification based on the number of pixels, the magnification can also be determined based on an inverse proportionality coefficient where the magnification decreases with a higher number of pixels. The magnified image generation unit 622 can also superimpose the accuracy probability contained in information related to the region of interest onto the magnified image to generate the magnified image. The magnified image generation unit 622 outputs the generated magnified image to the output unit 624.

[0056] When the output mode determination unit 623 has acquired information related to the region of interest from the learned model 631, it determines the output mode (display mode) for outputting (displaying) a magnified image based on the accuracy probability of the region of interest contained in that information. For example, this output mode (display mode) includes: a sub-screen display mode for displaying the magnified image on a screen (sub-screen) different from the screen displaying the endoscope image (main screen); and a main screen switching display mode for switching from the endoscope image to the magnified image in the screen displaying the endoscope image (main screen). If the accuracy probability of the region of interest output from the learned model 631 is less than a predetermined value, the output mode determination unit 623 determines the output mode (display mode) for outputting (displaying) the magnified image as a sub-screen display mode. If the accuracy probability of the region of interest output from the learned model 631 is above a predetermined value, the output mode determination unit 623 determines the output mode (display mode) when outputting (displaying) the magnified image as either a main screen switching display mode or a sub-screen display mode based on the preset values ​​(preset values ​​for determining the output mode) included in the preset data stored in the storage unit 63.

[0057] The preset data stored in the storage unit 63 includes a predetermined value for the accuracy probability used when determining the output mode (an accuracy probability threshold for distinguishing the area of ​​interest) and a setting value (preset value) for determining the output mode when the accuracy probability is above the predetermined value. Furthermore, the preset data includes a flag value indicating whether to switch to a special light viewing mode when displaying a magnified image. After acquiring information related to the area of ​​interest from the learned model 631, the output mode determination unit 623 determines whether to switch from a white light viewing mode to a special light viewing mode when displaying a magnified image, based on the flag value included in the preset data. The output mode determination unit 623 generates output mode data containing the output mode determined in this way and the flag value of the viewing mode (whether it needs to switch to a special light viewing mode), and outputs the generated output mode data to the output unit 624. The output mode data is data used to control the output mode when displaying a magnified image, and includes, for example, a setting value (preset value) indicating whether it is a main screen or a sub-screen, and a flag value indicating whether to switch to a special light viewing mode.

[0058] The output unit 624 outputs the magnified image acquired from the magnified image generation unit 622 and the output mode data acquired from the output mode determination unit 623 to the endoscope processor 20. The endoscope processor 20 displays the magnified image on the display device 50 in either the main screen or a sub-screen output mode, based on the magnified image and output mode data output from the information processing device 6 (output unit 624). When a special light observation mode is required, the endoscope processor 20 changes the magnified image display to a special light observation mode such as NBI (Narrow band imaging) based on the observation mode flag value included in the output mode data.

[0059] In this embodiment, the functional units in a series of processes are divided into functional units provided by the control unit 21 of the endoscope processor 20 and functional units provided by the control unit 62 of the information processing device 6, and described accordingly. However, the division of these functional units is merely an example and is not limited thereto. The control unit 21 of the endoscope processor 20 can function as all the functional units executed by the control unit 62 of the information processing device 6. That is, the endoscope processor 20 may substantially include the information processing device 6. Alternatively, the control unit 21 of the endoscope processor 20 may only output the captured image by the imaging element, and the control unit 62 of the information processing device 6 may function as all the functional units that perform subsequent processing. Alternatively, the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing device 6 may cooperate as functional units in a series of processes by, for example, performing inter-process communication.

[0060] Figure 6This is an explanatory diagram illustrating one method of displaying a magnified image. In the illustration of this embodiment, the left-hand screen is the main screen displaying the endoscopic image. The upper-right screen shows a state where a magnified image of the area of ​​interest is displayed in the main screen, representing a display mode based on switching display modes from the main screen. The lower-right screen shows a state where a magnified image of the area of ​​interest is displayed in a sub-screen, representing a display mode based on the sub-screen display mode.

[0061] The accuracy probability (estimated score) of the region of interest is superimposed on these magnified images. For example, if the accuracy probability is less than a predetermined value such as 0.9, the magnified image is displayed in a sub-screen. If the accuracy probability is higher than the predetermined value, the magnified image is displayed in the main screen or a sub-screen according to the setting value (preset value) contained in the preset data.

[0062] Figure 7 This is a flowchart illustrating an example of the processing steps performed by the control unit 62 of the information processing device 6. For example, the information processing device 6 begins processing according to the content input from the input unit 8 connected to this device.

[0063] The control unit 62 of the information processing device 6 acquires the endoscopic image output from the endoscope processor 20 (S101). The control unit 62 of the information processing device 6 can also acquire the endoscopic image from the endoscope processor 20 synchronously with the start of imaging inside the body cavity by the endoscope processor 20. The endoscopic image acquired by the control unit 62 of the information processing device 6 from the endoscope processor 20 can be a still image or a moving image.

[0064] The control unit 62 of the information processing device 6 inputs the endoscope image into the learned model 631 (S102). When the input endoscope image contains a region of interest, the learned model 631 outputs the position and accuracy probability as information related to that region of interest; when the input endoscope image does not contain a region of interest, it does not output information related to that region of interest.

[0065] The control unit 62 of the information processing device 6 determines whether information related to the region of interest has been obtained from the learned model 631 (S103). If no information related to the region of interest has been obtained from the learned model 631 (S103: NO), the control unit 62 of the information processing device 6 performs loop processing in order to execute the processing of S101 again.

[0066] Having obtained information related to the region of interest from the learned model 631 (S103: YES), the control unit 62 of the information processing device 6 generates a magnified image of the region of interest based on the location of the region of interest contained in the information related to the region of interest (S104). The control unit 62 of the information processing device 6 extracts (cuts out) a portion of the endoscopic image containing the region of interest based on the location of the region of interest, for example, using electronic zoom (digital zoom) to generate a magnified image of the region of interest. The control unit 62 of the information processing device 6 may also store the generated magnified image of the region of interest in association with the endoscopic image containing the region of interest in the storage unit 63.

[0067] The control unit 62 of the information processing device 6 determines whether the accuracy determination of the area of ​​interest contained in the information related to the area of ​​interest is above a set value (S105). The control unit 62 of the information processing device 6 determines whether the acquired accuracy determination is above the set value by referring to a predetermined value (an accuracy probability threshold used to distinguish the accuracy probability of the area of ​​interest) contained in the preset data stored in the storage unit 63. If the accuracy determination of the area of ​​interest is not above the set value (S105: NO), that is, if the accuracy determination of the area of ​​interest is less than the set value, the control unit 62 of the information processing device 6 determines to set the output mode to a sub-screen.

[0068] When the precision of the area of ​​interest is established to be above a set value (S105: YES), the control unit 62 of the information processing device 6 determines the output mode by referring to preset data (S106). The control unit 62 of the information processing device 6 determines whether to use the magnified image as the main screen or the sub-screen (output mode) based on the set value (preset value) included in the preset data. In this embodiment, the output mode is determined by referring to preset data when the precision of the area of ​​interest is established to be above a set value, but it is not limited to this. For example, the control unit 62 of the information processing device 6 may also display a selection screen for selecting the output mode on the display unit 7 connected to the information processing device 6, and determine the output mode based on the endoscope operator's selection operation input from the input unit 8.

[0069] The control unit 62 of the information processing device 6 acquires the viewing mode for magnified display by referring to preset data (S107). The control unit 62 of the information processing device 6 determines the viewing mode for magnified display based on the flag value of the viewing mode included in the preset data (whether it is necessary to change to a special light viewing mode).

[0070] The control unit 62 of the information processing device 6 outputs magnified image and output mode data (S108). The output mode data includes the output mode (display mode) when displaying the magnified image and the observation mode (changing to a special light observation mode). The control unit 62 of the information processing device 6 outputs the magnified image and output mode data to the endoscope processor 20. The endoscope processor 20 displays the magnified image on the display device 50 in either the main screen or a sub-screen output mode according to the magnified image and output mode data output from the information processing device 6 (output unit 624). When a change to a special light observation mode is required, the endoscope processor 20 changes the display of the magnified image to a special light observation mode such as NBI (Narrow band imaging) according to the flag value of the observation mode included in the output mode data.

[0071] The control unit 62 of the information processing device 6 acquires the endoscope images subsequently output from the endoscope processor 20 (S109). Endoscope images are sequentially output from the endoscope processor 20, and the control unit 62 of the information processing device 6 acquires these endoscope images sequentially.

[0072] The control unit 62 of the information processing device 6 inputs the endoscopic image into the learned model 631 (S110). The control unit 62 of the information processing device 6 determines whether information related to the region of interest has been obtained from the learned model 631 (S111). Similar to the processing in S102 and S103, the control unit 62 of the information processing device 6 performs the processing in S110 and S111.

[0073] If no information related to the region of interest is obtained from the learned model 631 (S111: NO), the control unit 62 of the information processing device 6 stops outputting the magnified image. By stopping the output of the magnified image, the image displayed on the main screen switches from the magnified image to the endoscopic image. If the magnified image is displayed in the generated sub-screen, the sub-screen is closed. After stopping the output of the magnified image, the control unit 62 of the information processing device 6 performs loop processing to execute the processing of S101 again. If information related to the region of interest is obtained from the learned model 631 (S111: YES), the control unit 62 of the information processing device 6 performs loop processing to execute the processing of S104 again.

[0074] In this embodiment, a series of processes are performed by the control unit 62 of the information processing device 6, but this is not a limitation. This series of processes may also be performed by the control unit 21 of the endoscope processor 20. Alternatively, this series of processes may be performed collaboratively by the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing device 6, for example, through inter-process communication. In this embodiment, the control unit 62 of the information processing device 6 outputs the generated magnified image to the endoscope processor 20, but this is not a limitation; the control unit 62 of the information processing device 6 may also output the magnified image to a display unit 7 connected to the information processing device 6 and display the magnified image on the display unit 7.

[0075] According to this embodiment, the information processing device 6 inputs an image (endoscope image) captured by an endoscope into a learned model 631. If the image (endoscope image) contains a region of interest (ROI), the device obtains the position of the ROI output from the learned model 631. Based on the obtained position of the ROI, the information processing device 6 extracts a portion of the endoscopic image including the ROI, magnifies and displays this portion, and outputs a magnified image of the portion of the endoscopic image containing the ROI. Therefore, when the endoscopic image contains a ROI, the process of magnifying and displaying the ROI can be effectively performed, and the magnified image generated by this magnification process can improve the visual recognition of the ROI by the endoscopic operator. Since the electronic zoom operation for magnification and display is automatically performed based on the presence or absence of the ROI, the operability of the endoscope can be improved without requiring zooming by the endoscopic operator. Since the magnified ROI and other ROIs can be effectively visualized, it is expected to facilitate the judgment of endoscopic operators such as physicians, and for example, reduce procedures such as biopsies.

[0076] According to this embodiment, the information processing device 6 determines the output format for outputting a magnified image based on the accuracy probability of the region of interest obtained from the learned model 631, and outputs the magnified image in this output format. This output format includes an output format in which the magnified image is displayed in a sub-screen different from the screen displaying the endoscope image, and an output format in which the magnified image is switched from the endoscope image to the main screen displaying the endoscope image. When the accuracy probability is less than a predetermined value, the information processing device 6 outputs the magnified image by displaying it in a sub-screen different from the screen displaying the endoscope image, thus maintaining the state of the screen displaying the endoscope image (main screen) while simultaneously displaying the magnified image of the region of interest in a different sub-screen. Therefore, when the accuracy probability estimated (output) as the region of interest according to the learned model 631 is less than a predetermined value, the endoscope operator can continue to view the endoscope image by maintaining the state of the screen displaying the endoscope image (main screen).

[0077] For areas of interest with an accuracy probability of ≥100%, switching the display screen (main screen) from the endoscope image to a magnified image of the area of ​​interest improves the visual recognition of the magnified image (area of ​​interest with an accuracy probability of ≥100%) by the operator. Even for areas of interest with an accuracy probability of ≥100%, similarly to areas of interest with an accuracy probability lower than ≥100%, the output can be controlled using preset values ​​while maintaining the display screen (main screen) of the endoscope image, resulting in a magnified image of the area of ​​interest displayed in a different screen (sub-screen). That is, the output modes for displaying magnified images of areas of interest with an accuracy probability of ≥100% include the output mode switching to the display screen (main screen) of the endoscope image, and the output mode displaying (showing) the image in a different screen (sub-screen) than the display screen (main screen), and these multiple output modes can be selectively used. The setting value (preset value) for determining the selective output mode is stored in a storage area such as the storage unit 63 of the information processing device 6, which can be accessed from the control unit 62 of the information processing device 6. The information processing device 6 determines the output mode when outputting a magnified image of the area of ​​interest with an accuracy probability of more than a predetermined value based on the setting value, thereby improving usability for endoscope operators.

[0078] According to this embodiment, when the information processing device 6 outputs a magnified image of the region of interest, it outputs a signal to the endoscope (endoscope processor) to change the endoscope's observation mode to a special light observation mode. Therefore, triggered by the presence of the region of interest in the endoscope image, the observation mode is automatically changed from the white light observation mode to the special light observation mode, effectively providing the endoscope operator with a magnified image of the region of interest illuminated by special light such as NBI (Narrow Band Imaging).

[0079] According to this embodiment, when the endoscopic image does not contain the region of interest, that is, when no information (position, accuracy probability) related to the region of interest is output from the learned model 631, the information processing device 6 stops outputting the magnified image. When the magnified image is displayed on a screen (sub-screen) different from the screen displaying the endoscopic image (main screen), the sub-screen is closed by stopping the output of the magnified image. Alternatively, the sub-screen may transition to a non-display state, an inactive state, or a minimized state. When the screen displaying the endoscopic image (main screen) is switched from the endoscopic image to the magnified image, the image displayed on the main screen is switched back to the original endoscopic image by stopping the output of the magnified image. When using a special light observation mode to display the magnified image, the observation mode can be switched from the special light observation mode to the white light observation mode when the endoscopic image does not contain the region of interest. If the endoscopic image acquired after switching to the output magnified image state in this way does not contain the region of interest, the output mode (display mode) when the magnified image is output (displayed) is automatically returned to the original output mode (display mode) when only the endoscopic image is output (displayed) as triggered by the absence of the region of interest. This can reduce the operation time of the endoscopic operator and improve operability.

[0080] (Implementation Method 2)

[0081] Figure 8 This is an explanatory diagram illustrating one way of displaying a magnified image related to Embodiment 2 (multiple regions of interest). The control unit 62 (learned model 631) of the information processing device 6, having acquired information (position, accuracy probability) related to multiple regions of interest from the endoscopic image, outputs each of the multiple magnified images in which these multiple regions of interest are magnified. In the illustration of this embodiment, the left-hand screen is the main screen displaying the endoscopic image, which includes multiple regions of interest.

[0082] The control unit 62 of the information processing device 6 may also generate a number of sub-screens (screens different from the screen displaying the endoscopic image) that are the same number as the number of areas of interest, and display each magnified image on each of the generated sub-screens. Alternatively, when outputting (displaying) multiple generated magnified images, the control unit 62 of the information processing device 6 may generate output pattern data in an output pattern (display pattern) that determines the magnified image with the highest accuracy among the multiple magnified images to be displayed on the main screen and outputs (displays) other magnified images on each sub-screen, and output the output pattern data to the endoscopic processor 20.

[0083] Figure 9 This is a flowchart illustrating an example of the processing steps performed by the control unit 62 of the information processing device 6. Similar to the processing in S101 to S106 and S1051 of Embodiment 1, the control unit 62 of the information processing device 6 performs the processing in S201 to S206 and S2061.

[0084] The control unit 62 of the information processing device 6 determines whether magnified images of all regions of interest have been generated (S207). If the endoscopic image contains multiple regions of interest, the learned model 631 outputs information including the location and accuracy probability of each of these multiple regions of interest (information related to the regions of interest). The control unit 62 of the information processing device 6 may also temporarily store the locations and accuracy probabilities of the multiple regions of interest output from the learned model 631 in an arranged manner in the storage unit 63, and process them according to the sequence number determined by the number of regions of interest contained in the endoscopic image. For example, if magnified images of the regions of interest corresponding to all sequence numbers have been generated, i.e., if there are no unprocessed regions of interest, the control unit 62 of the information processing device 6 determines that magnified images of all regions of interest have been generated. If magnified images of all regions of interest have not been generated (S207: NO), the control unit 62 of the information processing device 6 performs loop processing to re-execute the processing of S204.

[0085] When magnified images of all regions of interest have been generated (S207: YES), the control unit 62 of the information processing device 6 performs the processing in S208 and S209 in the same manner as in Embodiment 1 (S107 and S108). The control unit 62 of the information processing device 6 may also output output mode data to the endoscope processor 20 by outputting (displaying) the output mode (display mode) of each of the generated magnified images on each sub-screen. Alternatively, when outputting (displaying) the generated magnified images, the control unit 62 of the information processing device 6 may display the magnified image with the highest accuracy among the generated magnified images on the main screen, and output the output mode (display mode) of the other magnified images on each sub-screen. The control unit 62 of the information processing device 6 may also stop outputting the magnified image if the acquired endoscopic image does not contain the region of interest, similar to Embodiment 1.

[0086] According to this embodiment, when the information processing device 6 acquires information (position, accuracy probability) related to multiple regions of interest from a single endoscopic image, such as a frame in a moving image, it outputs each of a plurality of magnified images, each of these multiple regions of interest, to the endoscope processor 20. When outputting multiple magnified images separately, the information processing device 6 can also generate a number of sub-screens (screens different from the screen displaying the endoscopic image) equal to the number of regions of interest, and display each magnified image on each of the generated sub-screens. Even when the endoscopic image contains multiple regions of interest, by displaying magnified images of each of these multiple regions of interest on each sub-screen, information related to the regions of interest can be effectively provided to the endoscopic operator.

[0087] (Implementation Method 3)

[0088] Figure 10 This is a functional block diagram of the functional units included in the control unit 62 of the information processing device 6 according to the illustrative embodiment 3 (second learned model 632). Figure 11 This is an explanatory diagram regarding the second learned model 632 (diagnostic support learning model). The information processing apparatus 6 of Embodiment 3 also includes the second learned model 632. The storage unit 63 of the information processing apparatus 6 of Embodiment 3 stores actual files (instance files of neural networks (NN)) constituting the second learned model 632 (diagnostic support learning model). These actual files may also be incorporated into the program.

[0089] Similar to Embodiment 1, the control unit 62 of the information processing apparatus 6 in Embodiment 3 functions as the acquisition unit 621, the learned model 631, the magnified image generation unit 622, the output mode determination unit 623, and the output unit 624 by executing the program stored in the storage unit 63, and further functions as the second learned model 632. The acquisition unit 621, the learned model 631, the magnified image generation unit 622, and the output mode determination unit 623 perform the same processing as in the first embodiment. Furthermore, the magnified image generation unit 622 outputs the generated magnified image to the second learned model 632.

[0090] The second learned model 632, like the learned model 631, is composed of a neural network, such as a CNN (Convolutional Neural Network). When an enlarged image including the region of interest is input, the second learned model 632 learns by outputting diagnostic support information related to the region of interest. This diagnostic support information may include, for example, the type of region of interest such as lesion, lesion candidate, medication, treatment device, and markers, or the classification and staging of the lesion. The second learned model 632 models diagnostic support information including the type of the region of interest based on the input enlarged image (enlarged image including the region of interest), essentially functioning as a diagnostic support learning model. The second learned model 632 outputs (estimates) the diagnostic support information derived from the input enlarged image to the output unit 624.

[0091] Similar to Embodiment 1, the output unit 624 outputs the magnified image acquired from the magnified image generation unit 622, the output mode data acquired from the output mode determination unit 623, and the diagnostic support information acquired from the second learned model 632 (diagnostic support learning model) to the endoscope processor 20. Also similar to Embodiment 1, the endoscope processor 20 displays the magnified image and diagnostic support information on the display device 50 in either the main screen or a sub-screen output mode, based on the magnified image, diagnostic support information, and output mode data output from the information processing device 6 (output unit 624).

[0092] Figure 12 This is a flowchart illustrating an example of the processing steps performed by the control unit 62 of the information processing device 6. Similar to the processing in S101 to S107 of Embodiment 1, the control unit 62 of the information processing device 6 performs the processing in S301 to S307.

[0093] The control unit 62 of the information processing device 6 inputs the magnified image into the second learned model 632 (S308). The control unit 62 of the information processing device 6 obtains diagnostic support information from the second learned model 632 (S309). The second learned model 632, having input the magnified image, outputs diagnostic support information related to the region of interest. The diagnostic support information related to the region of interest includes, for example, the type of region of interest such as lesion, lesion candidate, medication, treatment device, and markings, or the classification and staging of the lesion.

[0094] The control unit 62 of the information processing device 6 outputs a magnified image, output format data, and diagnostic support information (S310). Similar to Embodiment 1, the control unit 62 of the information processing device 6 outputs the magnified image, output format data, and diagnostic support information to the endoscope processor 20. Also similar to Embodiment 1, the endoscope processor 20 displays the magnified image and diagnostic support information in either the main screen or a sub-screen output format on the display device 50 based on the magnified image, diagnostic support information, and output format data output from the information processing device 6 (output unit 624).

[0095] According to this embodiment, when the learned model 631 outputs information related to the region of interest (location, accuracy probability), the information processing device 6 inputs a magnified image of the region of interest into the second learned model 632, and obtains diagnostic support information related to the region of interest (type of region of interest, classification of lesions, and staging, etc.) from the second learned model 632. The image input to the second learned model 632 is a magnified image of the region of interest extracted from the endoscopic image; therefore, compared to the information ratio of the region of interest in the endoscopic image, the information ratio of the region of interest in the magnified image can be increased, and the estimation accuracy of the second learned model 632 can be improved. The information processing device 6 outputs the magnified image of the region of interest in association with the diagnostic support information output (estimated) by the second learned model 632 based on the magnified image, and displays it on a sub-screen or the like, thereby effectively providing the endoscopic operator with a magnified image of the region of interest and diagnostic support information.

[0096] It should be understood that the embodiments disclosed herein are illustrative in all respects and not restrictive. The technical features described in the various embodiments can be combined with each other, and all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within the scope of this invention.

[0097] Symbol Explanation

[0098] S Diagnostic Support System

[0099] 10 Endoscopic devices

[0100] 15-keyboard

[0101] 16 containment racks

[0102] 20 Endoscope Processor

[0103] 21 Control Department

[0104] 211 Image Processing Department

[0105] 22 main storage devices

[0106] 23 Auxiliary storage devices

[0107] 24 Ministry of Communications

[0108] 25 Touch Panel

[0109] 26 Display Devices I / F

[0110] 27 Input Devices I / F

[0111] 28 Reading Department

[0112] 31 Connector for Endoscopes

[0113] 311 electrical connector

[0114] 312 optical connector

[0115] 33 light sources

[0116] 34 pumps

[0117] 35 water supply tank

[0118] 36 gas and water supply outlets

[0119] 40 Endoscopes

[0120] 43 Operations Department

[0121] 431 control button

[0122] 433 Bend Knob

[0123] 44 Insertion section (flexible tube)

[0124] 441 Flexible Part

[0125] 442 Bending Section

[0126] 443 front end

[0127] 444 Filming Department

[0128] 45-degree bend stop

[0129] 48 Connectors for Observers

[0130] 49 General-purpose flexible wire

[0131] 50 display devices

[0132] 6 Information Processing Devices

[0133] 61 Ministry of Communications

[0134] 62 Control Department

[0135] 621 Acquisition Department

[0136] 622 Magnified Image Generation Unit

[0137] 623 Output Pattern Determination Unit

[0138] 624 Output Section

[0139] 63 Storage Division

[0140] 631 Learned Model (Region-Based Learning Model)

[0141] 632 Second Learned Model (Diagnostic Support Learning Model)

[0142] 64 Input / Output I / F

[0143] 7 Display Section

[0144] 8. Input section.

Claims

1. A program product comprising a program that causes a computer to perform processing, wherein, In computers, Acquire images captured by the endoscope. When an image captured by the endoscope is input, the acquired image is fed into a learned model to output the location of the region of interest contained in the image. The region of interest is the area of ​​interest to the endoscopic operator, which is the area containing the lesion, lesion candidate, medication, treatment instrument, or marker. The location of the region of interest contained in the acquired image is obtained from the learned model. Based on the location of the obtained region of interest, an enlarged image is output, which includes a portion of the image containing the region of interest. Given an image captured by the endoscope as input, the learned model outputs the location and accuracy probability of the region of interest contained in the image. If the accuracy probability is less than a predetermined value, the magnified image is output so that it can be displayed on a different screen than the one displaying the acquired image. When the accuracy probability is above a predetermined value, the magnified image is switched from the acquired image to the magnified image in the screen displaying the acquired image, or the magnified image is output in a screen display mode different from the screen displaying the acquired image.

2. The program product according to claim 1, wherein, Pre-store setting values ​​for determining the output format of the magnified image when the accuracy probability is above a predetermined value. The magnified image is output in an output format corresponding to the set value, when the accuracy probability is above the predetermined value.

3. The program product according to claim 1 or 2, wherein, When the magnified image is output, information is output to change the observation mode of the endoscope to a special light observation mode.

4. The program product according to claim 1 or 2, wherein, If the image obtained after outputting the magnified image does not contain the region of interest, the output of the magnified image is stopped.

5. The program product according to claim 1 or 2, wherein, Having obtained the positions of multiple regions of interest contained in the acquired image from the learned model, an enlarged image is output, which is a portion of the image containing each of the multiple regions of interest, based on the obtained positions of the multiple regions of interest.

6. The program product according to claim 1 or 2, wherein, When a magnified image including the region of interest is input, the magnified image is fed into the second learned model that has already been trained, so as to output diagnostic support information related to the region of interest. The diagnostic support information related to the region of interest is obtained from the second learned model. The acquired diagnostic support information is output in association with the magnified image.

7. An information processing method that enables a computer to perform processing, wherein, Acquire images captured by the endoscope. When an image captured by the endoscope is input, the acquired image is fed into a learned model to output the location of the region of interest contained in the image. The region of interest is the area of ​​interest to the endoscopic operator, which is the area containing the lesion, lesion candidate, medication, treatment instrument, or marker. The location of the region of interest contained in the acquired image is obtained from the learned model. Based on the location of the obtained region of interest, an enlarged image is output, which includes a portion of the image containing the region of interest. Given an image captured by the endoscope as input, the learned model outputs the location and accuracy probability of the region of interest contained in the image. If the accuracy probability is less than a predetermined value, the magnified image is output so that it can be displayed on a different screen than the one displaying the acquired image. When the accuracy probability is above a predetermined value, the magnified image is switched from the acquired image to the magnified image in the screen displaying the acquired image, or the magnified image is output in a screen display mode different from the screen displaying the acquired image.

8. An information processing apparatus comprising: The image acquisition unit acquires images captured by the endoscope; The input unit, when an image captured by the endoscope is input, inputs the acquired image into a learned model that has been learned, so as to output the location of the region of interest contained in the image. The region of interest is the area of ​​interest of the endoscope operator, which is the area where the lesion, lesion candidate, drug, treatment instrument or marker is located. A location acquisition unit acquires the location of a region of interest contained in the acquired image from the learned model; The output unit outputs a magnified image, which includes a portion of the image containing the region of interest, based on the acquired location of the region of interest. Given an image captured by the endoscope as input, the learned model outputs the location and accuracy probability of the region of interest contained in the image. If the accuracy probability is less than a predetermined value, the magnified image is output so that it can be displayed on a different screen than the one displaying the acquired image. When the accuracy probability is above a predetermined value, the magnified image is switched from the acquired image to the magnified image in the screen displaying the acquired image, or the magnified image is output in a screen display mode different from the screen displaying the acquired image.

Images