Diagnostic imaging assistance device, diagnostic imaging assistance system, and diagnostic imaging assistance method

Abstract

This diagnostic imaging assistance device 5 comprises: a captured image acquisition unit 514 that acquires captured images captured by an imaging device 2 that captures images of a subject; a motion determination unit 510 that determines the magnitude of relative motion between the imaging device 2 and the subject; an image processing unit 515 that executes image processing on the captured images and outputs processed images; an image generation unit 516 that generates, on the basis of the processed images, images to be diagnosed; and an estimation unit 517 that estimates diagnosis candidate regions serving as diagnosis candidates among the processed images by executing estimation processing on the processed images using a trained model. In the image generation unit 516, the input frame rate of the processed images for sequentially executing the high definition processing and the input frame rate of the processed images to be input sequentially to the estimation unit 517 are different according to the magnitude of the motion.

Description

Image diagnosis support device, image diagnosis support system, and image diagnosis support method

[0001] The present invention relates to an image diagnosis support device, an image diagnosis support system, and an image diagnosis support method.

[0002] In recent years, image recognition technologies based on AI (Artificial Intelligence) have been proposed in the medical field (see, for example, Patent Document 1). The technology described in Patent Document 1 estimates a diagnostic candidate region such as a lesion in an image captured by an endoscope by performing an estimation process using a trained model on the image.

[0003] International Publication No. 2022 / 004056

[0004] When using AI-based image recognition technology, doctors may consider increasing the speed of observation in less important observation areas by relying on the AI-based image recognition technology and removing the endoscope. However, if the endoscope is removed quickly, image blurring caused by high-speed movement reduces the accuracy of the AI-based image recognition technology's estimation of diagnostic candidate areas such as lesions. As a result, the endoscope cannot be removed quickly in less important observation areas, and the burden on the doctor cannot be reduced.

[0005] The present invention has been made in view of the above, and aims to provide an image diagnosis support device, an image diagnosis support system, and an image diagnosis support method that can realize diagnostic support that reduces the burden on doctors.

[0006] In order to solve the above-mentioned problems and achieve the object, the image diagnosis support device of the present invention comprises an image acquisition unit that acquires an image captured by an imaging device that captures an image of a subject, a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject, an image processing unit that performs image processing on the captured image and outputs a processed image, an image generation unit that generates an image to be diagnosed based on the processed image, and an estimation unit that estimates a diagnostic candidate region in the processed image that will be a diagnostic candidate by performing estimation processing on the processed image using a trained model, and the input frame rate of the processed images that are sequentially generated by the image generation unit to generate the image to be diagnosed and the input frame rate of the processed images that are sequentially input to the estimation unit differ depending on the magnitude of the motion.

[0007] The image diagnosis support device of the present invention includes an image acquisition unit that acquires an image captured by an imaging device that captures an image of a subject; a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject; an image processing unit that performs image processing on the captured image and outputs a processed image; an image generation unit that generates a high-definition image that is higher in definition than the processed image by performing high-definition processing on the processed image; an estimation unit that estimates a diagnostic candidate region in the processed image that becomes a diagnostic candidate by performing estimation processing on the processed image using a learned model; and a display control unit that generates a display image based on the high-definition image and the diagnostic candidate region, and the display control unit switches the form of the display image depending on the magnitude of the motion.

[0008] The image diagnosis support system according to the present invention comprises an imaging device that generates captured images by imaging a subject, and an image diagnosis support device that processes the captured images. The image diagnosis support device comprises an image acquisition unit that acquires the captured images, a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject, an image processing unit that performs image processing on the captured images and outputs a processed image, an image generation unit that performs high-definition processing on the processed images to generate high-definition images that are higher in resolution than the processed images, and an estimation unit that performs estimation processing on the processed images using a trained model to estimate diagnostic candidate regions in the processed images that become diagnostic candidates. The input frame rate of the processed images that are sequentially subjected to the high-definition processing in the image generation unit and the input frame rate of the processed images that are sequentially input to the estimation unit differ depending on the magnitude of the motion.

[0009] An image diagnosis support method according to the present invention is an image diagnosis support method executed by an image diagnosis support device, and includes the steps of: acquiring an image captured by an imaging device that captures an image of a subject; determining the magnitude of relative movement between the imaging device and the subject; performing image processing on the captured image to output a processed image; performing high-definition processing on the processed image to generate a high-definition image that is higher in resolution than the processed image; and performing estimation processing on the processed image using a trained model to estimate a diagnostic candidate region in the processed image that will become a diagnostic candidate, wherein the input frame rate of the processed image that is sequentially subjected to the high-definition processing in an image generation unit that performs the high-definition processing and the input frame rate of the processed image that is sequentially input to an estimation unit that performs the estimation processing differ depending on the magnitude of the movement.

[0010] According to the image diagnosis support device, image diagnosis support system, and image diagnosis support method of the present invention, it is possible to realize diagnostic support that reduces the burden on doctors.

[0011] FIG. 1 is a diagram illustrating the configuration of an endoscope system according to an embodiment. FIG. 2 is a diagram illustrating the configuration of an endoscope system according to an embodiment. FIG. 3 is a flowchart illustrating an image diagnosis support method. FIG. 4 is a diagram illustrating the image diagnosis support method. FIG. 5 is a diagram illustrating the image diagnosis support method. FIG. 6 is a diagram illustrating a specific example of a display image. FIG. 7 is a diagram illustrating a specific example of a display image. FIG. 8 is a diagram illustrating a specific example of a display image. FIG. 9 is a diagram illustrating a specific example of a display image. FIG. 10 is a diagram illustrating a first modification of the embodiment. FIG. 11 is a diagram illustrating a first modification of the embodiment. FIG. 12 is a diagram illustrating a second modification of the embodiment.

[0012] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0013] 1 and 2 are diagrams illustrating the configuration of an endoscope system 1 according to an embodiment. The endoscope system 1 corresponds to an image diagnosis support system according to the present invention. This endoscope system 1 is used in the medical field and is a system for observing the inside of the body (the large intestine in this embodiment) of a subject PA (FIG. 1) who is a patient on a bed BD (FIG. 1). As shown in FIGS. 1 and 2, this endoscope system 1 includes an endoscope 2 and a processing device 3.

[0014] The endoscope 2 corresponds to an imaging device according to the present invention. In this embodiment, the endoscope 2 is a so-called flexible endoscope. A portion of the endoscope 2 is inserted into the body of a subject PA, images the interior of the body, and outputs image signals generated by the images. As shown in FIGS. 1 and 2 , the endoscope 2 includes an insertion section 21, an operation section 22, a universal cord 23, and a connector section 24. Note that, for ease of explanation, the operation section 22, the universal cord 23, and the connector section 24 are not shown in FIG. 2 .

[0015] The insertion section 21 has at least a portion that is flexible and is inserted into the body of the subject PA. As shown in FIG. 2 , the insertion section 21 includes a light guide 25, an illumination lens 26, an imaging section 27, and first and second sensors 28 and 29.

[0016] The light guide 25 is routed from the insertion section 21, through the operation section 22 and the universal cord 23, to the connector section 24. One end of the light guide 25 is located at the tip portion within the insertion section 21. When the endoscope 2 is connected to the processing device 3, the other end of the light guide 25 is located within the processing device 3. The light guide 25 transmits observation light supplied from the light source device 4 within the processing device 3 from the other end to one end. The illumination lens 26 faces one end of the light guide 25 within the insertion section 21. The illumination lens 26 irradiates the observation light transmitted by the light guide 25 into the body of the subject PA.

[0017] The imaging unit 27 is provided at the tip portion of the insertion unit 21. The imaging unit 27 captures images of the inside of the subject PA and outputs image signals generated by the image capture. As shown in FIG. 2 , the imaging unit 27 includes a lens unit 271 and an imaging element 272.

[0018] The lens unit 271 receives the return light (subject image) of the observation light irradiated into the body of the subject PA from the illumination lens 26 , and forms the subject image on the light receiving surface of the image sensor 272 .

[0019] The image sensor 272 is configured with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that receives light from a subject image and converts it into an electrical signal, and generates an image signal by capturing the subject image. Note that, hereinafter, the image signal generated by the image capturing unit 27 will be referred to as a captured image.

[0020] The first sensor 28 is a sensor used to determine the magnitude of relative movement between the insertion portion 21 and the subject. In the present embodiment, the first sensor 28 is provided at the tip of the insertion portion 21 and is configured by an acceleration sensor or an angular velocity sensor.

[0021] The second sensor 29 is a sensor used to calculate the tip position of the insertion portion 21. In this embodiment, the second sensor 29 is configured by a magnetic coil that generates magnetism.

[0022] The operation section 22 is connected to the proximal end portion of the insertion section 21. The operation section 22 receives various operations on the endoscope 2.

[0023] The universal cord 23 extends from the operating section 22 in a direction different from the direction in which the insertion section 21 extends, and is a cord on which signal lines that electrically connect the imaging section 27 and the control device 5 in the processing device 3, a light guide 25, etc. are arranged.

[0024] The connector portion 24 is provided at the end of the universal cord 23 and is detachably connected to the processing device 3 .

[0025] As shown in FIG. 2 , the processing device 3 includes a light source device 4 and a control device 5 .

[0026] The light source device 4, under the control of the control device 5, supplies observation light to be irradiated onto the subject to the other end of the light guide 25. Examples of the observation light include white light, excitation light that excites a fluorescent agent such as indocyanine green, and narrow band light used in NBI (Narrow Band Imaging).

[0027] The control device 5 corresponds to the image diagnosis support device according to the present invention. As shown in FIG. 2 , the control device 5 includes a control unit 51, a display unit 52, an input unit 53, a storage unit 54, and a receiving unit 55.

[0028] The control unit 51 includes a controller such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), and controls the overall operation of the endoscope system 1. As shown in FIG. 2 , the control unit 51 includes a motion determination unit 510, a position calculation unit 511, an imaging control unit 512, a light source control unit 513, a captured image acquisition unit 514, an image processing unit 515, an image generation unit 516, an estimation unit 517, and a display control unit 518. Note that the functions of the control unit 51 as the motion determination unit 510, the position calculation unit 511, the imaging control unit 512, the light source control unit 513, the captured image acquisition unit 514, the image processing unit 515, the image generation unit 516, the estimation unit 517, and the display control unit 518 will be described in the “Image diagnosis support method” and “Specific example of a display image” sections described later.

[0029] The display unit 52 corresponds to a notification unit according to the present invention. The display unit 52 is an LCD (Liquid Crystal Display) or an EL (Electro Luminescence) display, and displays a display image generated by the control unit 51 under the control of the control unit 51.

[0030] The input unit 53 corresponds to an operation receiving unit according to the present invention. The input unit 53 is configured using a keyboard, a mouse, a switch, a touch panel, etc., and receives user operations by a user such as a surgeon. The input unit 53 then outputs an operation signal corresponding to the user operation to the control unit 51.

[0031] The storage unit 54 stores various programs executed by the control unit 51, information necessary for the processing of the control unit 51, and the like.

[0032] The receiving unit 55 is a receiving unit used together with the second sensor 29 in calculating the tip position of the insertion unit 21 , and receives the magnetism emitted from the second sensor 29 under the control of the control unit 51 .

[0033] [Image diagnosis support method] Next, an image diagnosis support method executed by the control device 5 described above will be described. Fig. 3 is a flowchart showing the image diagnosis support method. Figs. 4 and 5 are diagrams explaining the image diagnosis support method. First, the movement determination unit 510 determines the magnitude of relative movement between the insertion portion 21 and the subject based on the signal output from the first sensor 28 (step S1).

[0034] If the magnitude of the movement determined in step S1 is less than a predetermined threshold (if the magnitude of the movement is small) (step S2: Yes), the imaging control unit 512 and the light source control unit 513 switch the imaging mode to the first imaging mode and the lighting mode to the first lighting mode (step S3).

[0035] Here, when determining the magnitude of movement in step S1, the magnitude of movement is determined during the removal operation. During the insertion operation, the image changes in orientation significantly because the insertion is performed while searching for a path that is gentle on the living body. On the other hand, during the removal operation, the removal can be performed with a relatively simple operation, so the image changes in orientation only slightly, making observation and diagnosis relatively easy. However, regardless of the above, the magnitude of movement may be determined regardless of the direction of operation.

[0036] 4, the first imaging mode is a mode in which the imaging frame rate (FPS) of the imaging unit 27 is a normal imaging frame rate (normal FPS) such as 60 (FPS), and the resolution is a resolution that uses all pixels in the effective pixel area (normal resolution). That is, in step S3, the imaging control unit 512 controls the operation of the imaging unit 27 and switches the imaging mode to the first imaging mode.

[0037] 4, the first illumination mode is a mode in which the observation light output from the light source device 4 is normal. That is, in step S3, the light source control unit 513 controls the operation of the light source device 4 to switch the illumination mode to the first illumination mode.

[0038] After step S3, the captured image acquisition unit 514 sequentially acquires captured images generated by the imaging unit 27 capturing return light of the observation light from inside the subject PA while the light source device 4 is irradiating the inside of the subject PA with normal output observation light (step S4). Because the captured images are in the first imaging mode, they are images with normal resolution captured at normal FPS. Hereinafter, for convenience of explanation, the captured images will be referred to as first captured images.

[0039] After step S4, the image processing unit 515 performs image processing on the first captured images sequentially acquired in step S4 (step S5). Hereinafter, the first captured images after image processing by the image processing unit 515 in step S5 will be referred to as first processed images. The image processing unit 515 then outputs the first processed images to the image generation unit 516 and the estimation unit 517. Examples of the image processing include known image processing such as gain adjustment, white balance adjustment, gamma correction, edge emphasis correction, and zoom adjustment.

[0040] After step S5, the image generation unit 516 generates an image suitable for display from the first processed image and outputs it (step S6). In this embodiment, in step S6, the image generation unit 516 estimates the image quality of the input first processed image using a trained model for high-definition processing. If the first processed image is estimated to have low image quality, the image generation unit 516 performs high-definition processing to generate a high-definition image with image quality as if it had been generated by an endoscope that generates high-definition images (hereinafter referred to as a high-definition endoscope). On the other hand, if the input first processed image is estimated to have high image quality, the image generation unit 516 outputs an image with the image quality of the first processed image without performing high-definition processing. Hereinafter, the high-definition image generated in step S6 will be referred to as a first high-definition image. In other words, the first high-definition image and an image with the image quality of the first processed image described above correspond to the image to be diagnosed in the present invention. Furthermore, the high-definition processing corresponds to the process of generating an image to be diagnosed in the present invention. The first high-definition image is an image captured in the first imaging mode, and therefore in normal FPS.

[0041] Here, the trained model for high-definition processing is pre-stored in the storage unit 54. Specifically, the trained model for high-definition processing is a trained model obtained by repeatedly executing a training process on the trained model using multiple sets of training images and teacher data, each set consisting of a training image and teacher data. The training images are images obtained by reducing the image quality of captured images (hereinafter referred to as high-quality images) generated by a high-definition endoscope to a quality corresponding to the first processed image. The teacher data is the high-quality images. The trained model used in the training process is, for example, a convolutional neural network (CNN). The trained model for high-definition processing includes a weight file (learning parameters) having weight values ​​and bias values ​​for each layer of the CNN.

[0042] The neural network used in the learning process for generating the trained model for image quality improvement processing is not limited to a CNN, and other neural networks may be used. Furthermore, various well-known learning algorithms may be used as the machine learning algorithm for the neural network. For example, a supervised learning algorithm using backpropagation may be used.

[0043] After step S5, the estimation unit 517 performs an estimation process on the first processed image using the first trained model for estimation process, thereby estimating diagnostic candidate regions for each predetermined region in the first processed image (step S7). Note that although step S7 is shown to be performed after step S6 in Fig. 3, in reality, steps S6 and S7 are performed in parallel, approximately simultaneously.

[0044] Here, the first trained model for estimation processing corresponds to the trained model according to the present invention. This first trained model for estimation processing is pre-stored in the storage unit 54. Specifically, the first trained model for estimation processing is a trained model obtained by repeatedly performing a training process on the trained model using multiple sets of training images and teacher data, each set consisting of a training image and teacher data. The training images are captured images of an in-vivo image. The teacher data is data annotated with the classification class, correct position, and size of lesions and the like in the training images. The trained model used in the training process is, for example, a CNN. The trained model for estimation processing includes a weight file (learning parameters) having weight values ​​and bias values ​​for each layer of the CNN.

[0045] The neural network used in the learning process for generating the first trained model for estimation is not limited to a CNN, and other neural networks may be used. For example, neural networks such as a deep neural network (DNN), a transformer, or a generative adversarial network (GAN) may be used as appropriate. Furthermore, various well-known learning algorithms may be used as machine learning algorithms in the neural network. For example, a supervised learning algorithm using backpropagation may be used.

[0046] Furthermore, the estimation unit 517 executes the estimation process to output the reliability of the diagnostic candidate region for each predetermined region in the first processed image.

[0047] The reliability of a diagnostic candidate region is a value indicating the level of reliability. Specifically, the reliability is a value indicating the accuracy of image recognition in the diagnostic candidate region, and can also be considered an index indicating the probability that an object in the image is predicted to belong to a specific class. The reliability of a diagnostic candidate region can be used to determine whether an object has been accurately recognized within the region.

[0048] In step S7, the estimation unit 517 estimates, as a diagnostic candidate region, a region whose reliability is equal to or greater than the reliability threshold for a first number of consecutive frames among the reliability of each predetermined region in the first processed image output by the estimation process. If there is a region whose reliability is equal to or greater than the reliability threshold for only one frame in the time series, the estimated region may be a falsely detected region, and therefore is not treated as a diagnostic candidate region.

[0049] 5 illustrates an example in which the first number of frames is "4." That is, even if the reliability of the region Ar in the first processed image F1(n) of the nth frame is equal to or greater than the reliability threshold, the estimation unit 517 does not yet estimate the region Ar as a diagnostic candidate region. Then, when the reliability of the region Ar is equal to or greater than the reliability threshold for four consecutive frames, from the first processed image F1(n) of the nth frame to the first processed image F1(n+3) of the (n+3)th frame, the estimation unit 517 estimates the region Ar as a diagnostic candidate region.

[0050] As described above, when the magnitude of movement is small, the input frame rate of the first processed images that are sequentially subjected to high-definition processing in the image generation unit 516 and the input frame rate of the first processed images that are sequentially input to the estimation unit 517 are the same, at a normal FPS such as 60 (FPS).

[0051] On the other hand, if the magnitude of the movement determined in step S1 is greater than or equal to a predetermined threshold (if the magnitude of the movement is large) (step S2: No), the imaging control unit 512 and the light source control unit 513 switch the imaging mode to the second imaging mode and the lighting mode to the second lighting mode (step S8).

[0052] Here, as shown in FIG. 4 , the second imaging mode is a mode in which the imaging frame rate (FPS) of the imaging unit 27 is higher than the normal FPS, such as 120, 240, or 480 FPS (high FPS). Furthermore, in the second imaging mode, the number of pixels is reduced below the normal resolution (low resolution) by thinning out the image and adding pixels to enable good readout even at the high FPS. That is, in step S8, the imaging control unit 512 controls the operation of the imaging unit 27 and switches the imaging mode to the second imaging mode.

[0053] 4, the second illumination mode is a mode in which the observation light from the light source device 4 is pulsed light with a higher output than the normal output. That is, in step S8, the light source control unit 513 controls the operation of the light source device 4 to switch the illumination mode to the second illumination mode.

[0054] After step S8, the captured image acquisition unit 514 sequentially acquires captured images generated by the imaging unit 27 capturing return light of the observation light from inside the subject PA while the light source device 4 irradiates the inside of the subject PA with high-power pulsed light (step S9). The captured images are low-resolution images captured at a high FPS because they are captured in the second imaging mode. Hereinafter, for ease of explanation, the captured images will be referred to as second captured images.

[0055] After step S9, the image processing unit 515 sequentially performs image processing on the second captured images sequentially acquired in step S9, similar to step S5 (step S10). Hereinafter, the second captured images after image processing by the image processing unit 515 in step S10 will be referred to as second processed images. Then, the image processing unit 515 outputs the second processed images to the image generation unit 516 and the estimation unit 517, respectively.

[0056] After step S10, the image generation unit 516 thins out frames of the input second processed image to sequentially reduce the input frame rate at which high-definition processing is performed, in order to achieve a frame rate that can be displayed on the display unit 52. Similarly to step S6, the image generation unit 516 estimates the image quality of the input second processed image using a trained model for high-definition processing. If the second processed image is estimated to have low image quality, the image generation unit 516 performs high-definition processing to generate a high-definition image with high image quality (increased resolution) as if it were generated by a high-definition endoscope (step S11). On the other hand, if the input second processed image is estimated to have high image quality, the image generation unit 516 outputs an image with the image quality of the second processed image without performing high-definition processing. As described above, because the second imaging mode produces low-resolution images, the image generation unit 516 often performs high-definition processing on the second processed image. In steps S6 and S11, the execution of high-definition processing is determined based on the estimation result of whether the first and second processed images have high image quality, but this configuration is not limited to this. For example, the execution of high-definition processing may be switched depending on the first imaging mode and the second imaging mode. Hereinafter, the high-definition image generated in step S11 will be referred to as the second high-definition image. In other words, the second high-definition image and an image of the image quality of the second processed image described above correspond to the image to be diagnosed in the present invention. In this embodiment, the second high-definition image is generated in the second imaging mode, and frames are thinned by the image generation unit 516, so the input frame rate is a normal FPS, such as 60 FPS.

[0057] After step S10, the estimation unit 517 performs an estimation process on the second processed image using the second trained model for estimation process, thereby estimating diagnostic candidate regions for each predetermined region in the second processed image (step S12). Note that although step S12 is shown to be performed after step S11 in Fig. 3, in reality, steps S11 and S12 are performed in parallel substantially simultaneously.

[0058] Here, the second trained model for estimation processing corresponds to the trained model according to the present invention. This second trained model for estimation processing is pre-stored in the memory unit 54. Here, the resolution of the second processed image is lower than that of the first processed image, and therefore the size of the feature map output by each layer of the second trained model for estimation processing is different from that of the first trained model for estimation processing. Therefore, the second trained model for estimation processing is a trained model generated using the same training images and teacher data as the first trained model for estimation processing, but is a model that differs from the first trained model for estimation processing in terms of the layer structure and number of channels in the neural network network model. That is, the estimation unit 517 switches the trained model for estimation processing depending on the magnitude of the movement determined in step S1 ( FIG. 4 ).

[0059] In step S12, the estimation unit 517 estimates, as a diagnostic candidate region, a region in which the reliability of each predetermined region in the second processed image output by the estimation process is equal to or greater than the reliability threshold for a second number of consecutive frames that is smaller than the first number of frames. That is, the estimation unit 517 switches the number of frames between the first number of frames and the second number of frames depending on the magnitude of the movement determined in step S1.

[0060] As described above, when the magnitude of the movement is large, the input frame rate of the second processed images that are sequentially subjected to high-definition processing in the image generation unit 516 differs from the input frame rate of the second processed images that are sequentially input to the estimation unit 517. Specifically, the input frame rate of the second processed images that are sequentially subjected to high-definition processing in the image generation unit 516 is lower than the input frame rate of the second processed images that are sequentially input to the estimation unit 517.

[0061] After steps S6 and S7 or steps S11 and S12, the display control unit 518 generates a display image to be displayed on the display unit 52 (step S13). Specifically, after steps S6 and S7, the display control unit 518 generates a display image based on the first high-definition image generated in step S6 and the diagnostic candidate region estimated in step S7. Furthermore, after steps S11 and S12, the display control unit 518 generates a display image based on the second high-definition image generated in step S11 and the diagnostic candidate region estimated in step S12. Details of the display image will be described later in "Specific Examples of Display Images."

[0062] [Specific Examples of Display Images] Next, specific examples of display images displayed on the display unit 52 will be described. Figures 6 to 9 are diagrams showing specific examples of display images. For example, the display control unit 518 generates the display image F1 shown in Figure 6 in the image diagnosis support method described above. Then, the display control unit 518 causes the display image F1 to be displayed on the display unit 52.

[0063] As shown in FIG. 6, the display image F1 includes an observation position image F11 and a diagnostic image F12.

[0064] The observation position image F11 is an image in which the current observation position (tip position of the insertion section 21) OP is superimposed on an image showing the shape of the observation target (the large intestine in this embodiment). Specifically, the position calculation unit 511 calculates the tip position OP of the insertion section 21 by a known method based on the magnetism emitted from the second sensor 29 and received by the receiving unit 55. Then, the display control unit 518 generates the observation position image F11 in which the tip position OP of the insertion section 21 (current observation position) calculated by the position calculation unit 511 is superimposed on an image showing the shape of the observation target, the position of which has been specified in advance.

[0065] The diagnostic image F12 takes on different forms depending on the magnitude of the movement determined in step S1.

[0066] For example, if the magnitude of the movement is small (step S2: Yes), the diagnostic image F12 will be the image shown in FIG. 7. Note that (a) of FIG. 7 shows the sequentially generated diagnostic image F12, with the horizontal axis representing time. Note that (b) of FIG. 7 shows the sequentially generated frames of the first high-definition image, with the horizontal axis representing time. For ease of explanation, the frames are labeled "super-resolution." Note that, when the magnitude of the movement is small, the first processed image is often high-definition as described above, and the diagnostic image F12 has the image quality of the first processed image. Note that (c) of FIG. 7 shows the sequentially generated frames of the first processed image, with the horizontal axis representing time. For ease of explanation, the frames are labeled "CAD."

[0067] When the magnitude of the movement is small, as shown in (b) and (c) of Figures 7A and 7B, the input frame rate of the first processed images that are sequentially subjected to high-definition processing in the image generation unit 516 and the input frame rate of the first processed images that are sequentially input to the estimation unit 517 are the same, at a normal FPS such as 60 (FPS).

[0068] Then, when the magnitude of movement is small, the display control unit 518 generates a diagnostic image F12 in which the diagnostic candidate area Ar1 is superimposed on the first high-resolution image F121 based on the first high-resolution image F121 and the diagnostic candidate area Ar1 processed by the image generation unit 516 and the estimation unit 517 on the first processed image of the same frame (frame FL4 in the example of Figure 7), as shown in Figure 7.

[0069] Here, if the display control unit 518 is unable to estimate the diagnostic candidate area Ar1 as a result of the estimation process, it generates only the first high-resolution image F121 as the diagnostic image F12 (frames FL1 to FL3 in the example of Figure 7).

[0070] Furthermore, for example, if the magnitude of the movement is large (step S2: No), the diagnostic image F12 will be the image shown in FIG. 8. Note that (a) of FIG. 8 shows the diagnostic image F12 that is sequentially generated, with the horizontal axis representing time. (b) of FIG. 8 shows frames of the second high-definition image that are sequentially generated, with the horizontal axis representing time. For ease of explanation, the frames are labeled "super-resolution." (c) of FIG. 8 shows frames of the second processed image that are sequentially subjected to estimation processing, with the horizontal axis representing time. For ease of explanation, the frames are labeled "CAD."

[0071] When the magnitude of the movement is large, as shown in (b) and (c) of Figures 8, the input frame rate of the second processed images that are sequentially subjected to high-definition processing in the image generation unit 516 differs from the input frame rate of the second processed images that are sequentially input to the estimation unit 517. In the example of Figure 8, the input frame rate of the second processed images that are sequentially subjected to high-definition processing in the image generation unit 516 is 60 (FPS). On the other hand, the input frame rate of the second processed images that are sequentially input to the estimation unit 517 is 240 (FPS).

[0072] If the magnitude of the movement is large, the display control unit 518 generates a diagnostic image F12 shown below. As shown in Fig. 8 , the display control unit 518 generates a diagnostic image F12 by superimposing a diagnostic candidate region Ar2 on the second high-definition image F122 based on the second high-definition image F122 and the diagnostic candidate region Ar2 that have been processed by the image generation unit 516 and the estimation unit 517 on the second processed image of the same frame (frame FL1 in the example of Fig. 8 ).

[0073] 8 , the display control unit 518 generates a diagnostic image F12 including frame position information IF indicating the second high-definition image F122 and the frame in which the diagnostic candidate region Ar2 was estimated (frame FL11 in the example of FIG. 8 ) based on a second high-definition image F122 and a diagnostic candidate region Ar2 processed by the image generation unit 516 and the estimation unit 517 on a second processed image of a different frame (frames FL11 and FL13 in the example of FIG. 8 ). In this embodiment, the frame FL11 in which the diagnostic candidate region Ar2 was estimated was captured at a position further back in the second high-definition image F122 than frame FL13 of the second high-definition image F122 when the insertion unit 21 was removed. For this reason, information indicating the back side is used as the frame position information IF. As shown by the dashed line in Figure 8, the diagnostic candidate area Ar2 may be superimposed on a second high-resolution image F122 of the frame FL13 closest to the frame FL11 in which the diagnostic candidate area Ar2 was estimated, to form a diagnostic image F12.

[0074] Here, if the display control unit 518 is unable to estimate the diagnostic candidate area Ar2 as a result of the estimation process, it generates only the second high-resolution image F122 as the diagnostic image F12 (frames FL5 and FL9 in the example of Figure 8).

[0075] 9 . Then, the display control unit 518 causes the display unit 52 to display the display image F2. Here, the control unit 51 stores in the storage unit 54 a diagnostic image F12 in which diagnostic candidate regions Ar1 and Ar2 are superimposed on first and second high-definition images F121 and F122. Then, in response to a user operation on the input unit 53, the display control unit 518 generates a display image F2 that displays a list of thumbnail images FT1 to FT9 of the multiple diagnostic images F12 stored in the storage unit 54.

[0076] The above-described embodiment provides the following advantages. When the magnitude of movement is large, the control device 5 according to this embodiment switches the imaging mode to the second imaging mode and the illumination mode to the second illumination mode. Therefore, the control device 5 can perform estimation processing on the second processed image, which is free of image blur caused by high-speed movement of the insertion portion 21, and can accurately estimate the diagnostic candidate region. Furthermore, the control device 5 generates a second high-definition image, which has been enhanced in image quality (resolution) by high-definition processing of the second processed image, which has been reduced in resolution by the second imaging mode. This allows doctors and other personnel to confirm an appropriate diagnostic image F12. Therefore, the control device 5 according to this embodiment allows the insertion portion 21 to be quickly removed in observation regions of low importance, thereby realizing diagnostic support that reduces the burden on doctors.

[0077] Furthermore, the control device 5 according to this embodiment switches the trained model for estimation processing depending on the magnitude of the movement determined in step S1. Therefore, by using an appropriate trained model for estimation processing depending on the magnitude of the movement, it is possible to appropriately estimate the diagnostic candidate region.

[0078] Furthermore, the control device 5 according to this embodiment switches the number of frames used to estimate the diagnostic candidate region between the first number of frames and the second number of frames depending on the magnitude of the movement determined in step S1, thereby making it possible to prevent the diagnostic candidate region from being erroneously estimated.

[0079] Other Embodiments Up to this point, the embodiments for carrying out the present invention have been described, but the present invention should not be limited to only the above-described embodiments. In the above-described embodiments, the image diagnosis support device according to the present invention is mounted on an endoscope system 1 having an insertion section 21 configured by a flexible endoscope, but this is not limiting. For example, the image diagnosis support device according to the present invention may be mounted on an endoscope system having an insertion section 21 configured by a rigid endoscope. Furthermore, the image diagnosis support device according to the present invention may be mounted on a medical observation system such as a surgical microscope (see, for example, JP 2016-42981 A) that magnifies and observes a predetermined field of view inside a subject (inside a living body) or on the surface of a subject (surface of a living body).

[0080] In the above-described embodiment, the image generation unit 516 performs a high-definition process on the first and second processed images using a trained model for high-definition processing, thereby generating first and second high-definition images with high image quality that appear as if they were generated by a high-definition endoscope that generates high-definition captured images. However, the present invention is not limited to this. For example, the image generation unit 516 may generate, as the first and second high-definition images, an image after filter processing such as edge enhancement or image enhancement, a contrast-enhanced image, an image after filter processing such as structural color enhancement, an image after blur restoration (deconvolution image), or the like.

[0081] In the above-described embodiment, the motion determination unit 510 determines the magnitude of the relative motion between the insertion unit 21 and the subject based on the signal output from the first sensor 28 configured by an acceleration sensor or an angular velocity sensor, but this is not limited to this. For example, the motion determination unit 510 may determine the magnitude of the relative motion between the insertion unit 21 and the subject based on the captured image using a known method such as a block matching method or a gradient method.

[0082] In the above-described embodiment, the diagnostic image F12 may include discrimination information that enables discrimination between the insertion of the insertion portion 21 into the body and the removal of the insertion portion 21 from the body. Furthermore, when a diagnostic candidate region is estimated during insertion, the tip position of the insertion portion 21 at the time of estimation may be stored in the memory unit 54, and when the tip position of the insertion portion 21 approaches the tip position stored in the memory unit 54 during removal, a notification unit such as the display unit 52 may notify the user that the tip position has approached the tip position stored in the memory unit 54.

[0083] In addition, in the above-described embodiment, the following modified examples 1 and 2 may be adopted.

[0084] (Modification 1) Figures 10 and 11 are diagrams illustrating Modification 1 of the embodiment. Specifically, Figure 10 is a diagram corresponding to Figure 2. Figure 11 is a block diagram showing the functions of control unit 51. In the control unit 51 according to Modification 1, as shown in Figures 10 and 11, a frame selection unit 519 is added to the control unit 51 described in the above embodiment.

[0085] In the above-described embodiment, when the magnitude of motion is large, the image generation unit 516 thins out frames of the input second processed image, sequentially reducing the input frame rate at which high-definition processing is performed. That is, the image generation unit 516 itself performs the above-described thinning. In contrast, in the present modification example 1, the frame selection unit 519, rather than the image generation unit 516, performs the above-described thinning. That is, when the magnitude of motion is large, the frame selection unit 519 performs the above-described thinning, and sequentially inputs the second processed images having the input frame rate reduced by the thinning to the image generation unit 516. On the other hand, when the magnitude of motion is small, the frame selection unit 519 does not perform the above-described thinning. That is, the first processed images are sequentially input to the image generation unit 516 while the frame rate is maintained.

[0086] Furthermore, the image generation unit 516 according to this first modification estimates the image quality of the image input from the frame selection unit 519, and if it is estimated that the image quality is low, it performs high-definition processing to generate and output a high-definition image with improved image quality, and if it is estimated that the image quality is high, it outputs an image of the input image quality without performing high-definition processing. In other words, the high-definition image and the image of the input image quality described above correspond to the images to be diagnosed according to the present invention.

[0087] Even when the configuration of the present modified example 1 described above is adopted, the same effects as those of the above-described embodiment are achieved.

[0088] (Variation 2) FIG. 12 is a diagram illustrating Variation 2 of the embodiment. Specifically, FIG. 12 is a diagram corresponding to FIG. 4. In the above-described embodiment, the estimation unit 517 switches the trained model for estimation processing depending on the magnitude of the movement determined in step S1, but this is not limited to this. In Variation 2, the estimation unit 517 uses the same trained model for estimation processing both when the magnitude of the movement is small and when it is large. Furthermore, when the magnitude of the movement is small, the estimation unit 517 uses a first threshold stored in the storage unit 54 as the reliability threshold to be used in the estimation processing. On the other hand, when the magnitude of the movement is large, the estimation unit 517 uses a second threshold stored in the storage unit 54 as the reliability threshold to be used in the estimation processing. The first threshold and the second threshold are different thresholds.

[0089] The above-described second modification provides the same effects as the above-described embodiment, as well as the following effects. The control device 5 according to the second modification switches the reliability threshold used in the estimation process, i.e., the detection sensitivity of the diagnostic candidate region, depending on the magnitude of the movement determined in step S1. Therefore, by using an appropriate reliability threshold according to the magnitude of the movement, the diagnostic candidate region can be appropriately estimated.

[0090] REFERENCE SIGNS LIST 1 Endoscope system 2 Endoscope 3 Processing device 4 Light source device 5 Control device 21 Insertion section 22 Operation section 23 Universal cord 24 Connector section 25 Light guide 26 Illumination lens 27 Imaging section 28 First sensor 29 Second sensor 51 Control section 52 Display section 53 Input section 54 Memory section 55 Receiving section 271 Lens unit 272 Imaging element 510 Movement determination section 511 Position calculation section 512 Imaging control section 513 Light source control section 514 Captured image acquisition section 515 Image processing section 516 Image generation section 517 Estimation section 518 Display control section 519 Frame selection section Ar, Ar1, Ar2 Diagnostic candidate area BD Bed F1, F2 Display image F11 Observation position image F12 Diagnostic image F121 First high-resolution image F122 Second high-resolution image FT1 to FT9 Thumbnail image IF Frame position information OP Observation position PA Subject

Claims

1. a captured image acquisition unit that acquires a captured image captured by an imaging device that captures an image of a subject; a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject; an image processing unit that performs image processing on the captured image and outputs a processed image; an image generating unit that generates an image to be diagnosed based on the processed image; an estimation unit that estimates a diagnostic candidate region in the processed image by performing an estimation process on the processed image using a trained model, An image diagnosis support device in which, when the magnitude of the movement is equal to or greater than a predetermined threshold, the input frame rate of the processed images sequentially input to the estimation unit is higher than the input frame rate of the processed images sequentially executed in the image generation unit to generate the images to be diagnosed.

2. The image generation unit The image diagnosis support device according to claim 1 , wherein the image to be diagnosed is generated by performing high-definition processing according to the image quality of the processed image using a trained model.

3. The image generation unit The image diagnosis support device according to claim 1 , wherein a high-definition image having higher definition than the processed image is generated as the image to be diagnosed by performing a high-definition process on the processed image.

4. The image diagnosis support device according to claim 1 , further comprising a display control unit that generates a display image based on the image to be diagnosed and the diagnostic candidate region.

5. The image generation unit 5. The image diagnosis support device according to claim 4, wherein a high-definition image having a higher image quality than the processed image is generated as the image to be diagnosed by performing a high-definition process on the processed image.

6. 2. The image diagnosis support device according to claim 1, further comprising a frame selection unit that, depending on the magnitude of the movement, sets different input frame rates for the processed images used in the image generation unit to sequentially perform processing to generate the images to be diagnosed and the processed images sequentially input to the estimation unit.

7. The image diagnosis support device according to claim 1 , further comprising an imaging control unit that switches an imaging mode of the imaging device depending on the magnitude of the movement.

8. The imaging control unit The image diagnosis support device according to claim 7 , wherein the image capture frame rate of the image capture device is changed depending on the magnitude of the movement.

9. The image diagnosis support device according to claim 1 , further comprising a light source control unit that switches an illumination mode of a light source device that supplies illumination light to the subject in accordance with the magnitude of the movement.

10. The image generation unit 2. The image diagnosis support device according to claim 1, wherein, when the magnitude of the movement is equal to or greater than a predetermined threshold, frames of the processed image to be input are thinned out, and an input frame rate of the processed image, which is used to generate the image to be diagnosed in the image generation unit, is made lower than an input frame rate of the processed images to be sequentially input to the estimation unit.

11. a display control unit that generates a display image based on the image to be diagnosed and the diagnostic candidate region; The display control unit 2. The image diagnosis support device according to claim 1, wherein the display image is generated by superimposing the diagnostic candidate region on the image to be diagnosed, based on the image to be diagnosed and the diagnostic candidate region processed by the image generation unit and the estimation unit on the processed image of the same frame.

12. a display control unit that generates a display image based on the image to be diagnosed and the diagnostic candidate region; The display control unit 2. The image diagnosis support device according to claim 1, wherein the display image is generated based on the image to be diagnosed and the diagnostic candidate region processed by the image generation unit and the estimation unit on the processed images of different frames. The display image includes information indicating the image to be diagnosed and the frame in which the diagnostic candidate region is estimated.

13. a display control unit that generates a display image based on the image to be diagnosed and the diagnostic candidate region; The display control unit 2. The image diagnosis support device according to claim 1, wherein the display image is generated by superimposing the diagnostic candidate region on the image to be diagnosed that is generated from the processed image of a frame immediately preceding the processed image of the frame in which the diagnostic candidate region is estimated, based on the image to be diagnosed and the diagnostic candidate region that have been processed by the image generation unit and the estimation unit for the processed images of different frames.

14. The motion determination unit 2. The image diagnosis support device according to claim 1, wherein the magnitude of the movement is determined based on the output of at least one of an acceleration sensor and an angular velocity sensor provided in the imaging device.

15. The motion determination unit The image diagnosis support device according to claim 1 , wherein the magnitude of the movement is determined based on the processed image.

16. The estimation unit The image diagnosis support device according to claim 1 , wherein the trained model is switched depending on the magnitude of the movement.

17. The estimation unit 2. The image diagnosis support device according to claim 1, wherein, of the reliability of each predetermined region in the processed image output by the estimation process, a region having a reliability equal to or greater than a reliability threshold is estimated as the diagnostic candidate region, and the reliability threshold is switched depending on the magnitude of the movement.

18. The estimation unit 2. The image diagnosis support device according to claim 1, wherein, of the reliability of each predetermined region in the processed image output by the estimation process, a region having a reliability equal to or greater than a reliability threshold for a predetermined number of consecutive frames is estimated as the diagnostic candidate region, and the predetermined number of frames is switched depending on the magnitude of the movement.

19. a captured image acquisition unit that acquires a captured image captured by an imaging device that captures an image of a subject; a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject; an image processing unit that performs image processing on the captured image and outputs a processed image; an image generating unit that generates a high-definition image having higher definition than the processed image by performing a high-definition processing on the processed image; an estimation unit that estimates a diagnostic candidate region in the processed image by performing an estimation process on the processed image using a trained model; a display control unit that generates a display image based on the high-definition image and the diagnostic candidate region; The display control unit An image diagnosis support device that switches the form of the displayed image depending on the magnitude of the movement.

20. 20. The image diagnosis support device according to claim 19, wherein, when the magnitude of the movement is equal to or greater than a predetermined threshold, an input frame rate of the processed images sequentially input to the estimation unit is higher than an input frame rate of the processed images sequentially subjected to the high definition processing in the image generation unit.

21. The display control unit If the magnitude of the movement is less than the predetermined threshold, the display image is generated by superimposing the diagnostic candidate region on the high-definition image based on the high-definition image and the diagnostic candidate region processed by the image generation unit and the estimation unit on the processed image of the same frame, and 21. The image diagnosis support device according to claim 20, wherein, when the magnitude of the movement is equal to or greater than the predetermined threshold, the display image is generated by superimposing the diagnostic candidate region on the high-definition image based on the high-definition image and the diagnostic candidate region processed by the image generation unit and the estimation unit for the processed image of the same frame, and the display image is generated including the high-definition image and information indicating the frame in which the diagnostic candidate region was estimated based on the high-definition image and the diagnostic candidate region processed by the image generation unit and the estimation unit for the processed image of a different frame.

22. an imaging device that captures an image of a subject to generate a captured image; an image diagnosis support device for processing the captured image, The image diagnosis support device includes: a captured image acquisition unit that acquires the captured image; a motion determination unit that determines the magnitude of relative motion between the imaging device and the subject; an image processing unit that performs image processing on the captured image and outputs a processed image; an image generating unit that generates a high-definition image having higher definition than the processed image by performing a high-definition processing on the processed image; an estimation unit that estimates a diagnostic candidate region in the processed image by performing an estimation process on the processed image using a trained model, An image diagnosis support system in which, when the magnitude of the movement is equal to or greater than a predetermined threshold, the input frame rate of the processed images sequentially input to the estimation unit is greater than the input frame rate of the processed images sequentially performing the high-definition processing in the image generation unit.

23. An image diagnosis support method executed by an image diagnosis support device, acquiring a captured image captured by an imaging device that captures an image of a subject; determining a magnitude of relative motion between the imaging device and the object; performing image processing on the captured image and outputting a processed image; generating a high-definition image having higher resolution than the processed image by performing a high-definition processing on the processed image; and performing an estimation process on the processed image using the trained model to estimate a diagnostic candidate region in the processed image that is a diagnostic candidate; an image diagnostic support method, wherein, when the magnitude of the movement is equal to or greater than a predetermined threshold, an input frame rate of the processed images sequentially input to an estimation unit that performs the estimation process is greater than an input frame rate of the processed images sequentially subjected to the high-definition processing in an image generation unit that performs the high-definition processing.