Endoscopic examination assistance device, endoscopic examination assistance method, and recording medium

Abstract

This endoscopic examination assistance device comprises an image generation means and an inference means. The image generation means uses a parameter obtained on the basis of an endoscopic image including a subject in a body cavity to generate a first image group including a plurality of arbitrary viewpoint endoscopic images respectively corresponding to different viewpoint positions or generate a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from the viewpoint position corresponding to the endoscopic image. The inference means performs inference processing related to the subject on the basis of the first image group or on the basis of the endoscopic image and the second image group. The endoscopic examination assistance device can be used to assist a physician in making a decision related to diagnosis of a lesion.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

[0001] The present disclosure relates to techniques that can be used to present information to assist in endoscopic examinations.

[0002] Techniques have been proposed to assist endoscopy.

[0003] Specifically, for example, Patent Document 1 discloses a technology in which, in an endoscopic device equipped with a parent endoscope and a child endoscope, an endoscopic image obtained from the child endoscope is corrected, and the corrected endoscopic image is input into a diagnostic support learning model, thereby outputting diagnostic support information including the presence or absence of a lesion contained in the corrected endoscopic image.

[0004] Japanese Patent Application Laid-Open No. 2022-175037

[0005] However, Patent Document 1 does not specifically disclose a method for obtaining information related to a lesion from a plurality of endoscopic images corresponding to a plurality of different viewpoints, and therefore, the technique disclosed in Patent Document 1 poses a problem in that, for example, the accuracy of information related to a lesion obtained from endoscopic images may be reduced.

[0006] One object of the present disclosure is to provide an endoscopic examination support device that can improve the accuracy of information related to lesions obtained from endoscopic images.

[0007] In one aspect of the present disclosure, an endoscopic examination support device has an image generation means that uses parameters obtained based on an endoscopic image including a subject inside a body cavity to generate a first image group including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or generates a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic image, and an inference means that performs inference processing related to the subject based on the first image group or based on the endoscopic image and the second image group.

[0008] In another aspect of the present disclosure, an endoscopic examination support method uses parameters obtained based on an endoscopic image including a subject inside a body cavity to generate a first group of images including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or generates a second group of images including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic image, and performs inference related to the subject based on the first group of images or based on the endoscopic image and the second group of images.

[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of using parameters obtained based on an endoscopic image including a subject within a body cavity to generate a first group of images including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or to generate a second group of images including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic image, and to make an inference related to the subject based on the first group of images or based on the endoscopic image and the second group of images.

[0010] According to the present disclosure, it is possible to improve the accuracy of information relating to a lesion obtained from an endoscopic image.

[0011] 1 is a diagram showing an example of a schematic configuration of an endoscopic examination system according to the present disclosure. FIG. 2 is a block diagram showing an example of a hardware configuration of an endoscopic examination support device according to the present disclosure. FIG. 3 is a block diagram showing an example of a functional configuration of an endoscopic examination support device according to the present disclosure. FIG. 4 is a diagram for explaining an example of parameters used in processing of an endoscopic examination support device according to the present disclosure. FIG. 5 is a diagram for explaining an example of the functional configuration of an inference processing unit. FIG. 6 is a flowchart showing an example of processing performed in an endoscopic examination support device according to the present disclosure. FIG. 7 is a block diagram showing another example of the functional configuration of an endoscopic examination support device according to the present disclosure. FIG. 8 is a diagram for explaining another example of the functional configuration of an inference processing unit. FIG. 9 is a flowchart showing another example of processing performed in an endoscopic examination support device according to the present disclosure.

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0013] <First embodiment> [System configuration] Fig. 1 is a diagram showing an example of a schematic configuration of an endoscopic examination system according to the present disclosure. As shown in Fig. 1, the endoscopic examination system 100 includes an endoscopic examination support device 1, a display device 2, and an endoscope 3 connected to the endoscopic examination support device 1.

[0014] The endoscopic examination support device 1 acquires, from the endoscope 3, video including time-series images obtained by capturing images of a subject during an endoscopic examination (hereinafter also referred to as an endoscopic video), and displays a display image on the display device 2 for confirmation by an operator, such as a doctor, performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires, from the endoscope 3, video of the interior of the large intestine obtained during the endoscopic examination as an endoscopic video. The endoscopic examination support device 1 also extracts an endoscopic image from the endoscopic video and generates multiple arbitrary viewpoint endoscopic images corresponding to different viewpoints using parameters obtained based on the extracted endoscopic image. The endoscopic examination support device 1 also acquires information related to a lesion in the extracted endoscopic image based on the multiple arbitrary viewpoint endoscopic images. The endoscopic examination support device 1 can be used to support doctors in making decisions regarding the diagnosis of lesions.

[0015] The display device 2 has, for example, a liquid crystal monitor, etc. The display device 2 displays images, etc. output from the endoscopic examination support device 1.

[0016] The endoscope 3 mainly has an operation unit 36 ​​that allows the surgeon to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that incorporates an endoscopic camera such as an ultra-small imaging element, and a connection unit 39 for connecting to the endoscopic examination support device 1.

[0017] 2 is a block diagram showing an example of the hardware configuration of an endoscopic examination support device according to the present disclosure. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0018] The processor 11 executes predetermined processes by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer. The processor 11 also performs processes such as acquiring information related to a lesion from multiple arbitrary viewpoint endoscopic images corresponding to multiple different viewpoints.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and nonvolatile memories that store information necessary for processing by the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs for the endoscopic examination support device 1 to execute each process in this embodiment.

[0020] Furthermore, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images captured by the endoscope 3 during an endoscopic examination.

[0021] The interface 13 acts as an interface between the endoscopic examination assistance device 1 and an external device. For example, the interface 13 supplies a display image generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating an endoscopic video supplied from the endoscope 3 to the processor 11. The interface 13 also supplies an endoscopic image extracted from the endoscopic video to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal in response to an operation by the surgeon. The input unit 14 has at least one device selected from the group consisting of a button, a touch panel, a remote controller, a foot switch, and a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump for delivering water or air to the endoscope 3. The sound output unit 16 outputs sound under the control of the processor 11.

[0023] The DB 17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. The DB 17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be acquired from the server via communication.

[0024] The endoscopic examination support device 1 may also be provided with a sensor, such as a magnetic sensor, that can measure the rotation and translation of the endoscopic camera.

[0025] [Functional Configuration] Fig. 3 is a block diagram showing an example of the functional configuration of the endoscopic examination support device 1 according to the present disclosure. As shown in Fig. 3, the endoscopic examination support device 1 has a parameter acquisition unit 21, a parameter conversion unit 22, an image generation unit 23, five inference processing units 24A to 24E, and an inference result integration unit 25.

[0026] The parameter acquisition unit 21 functions as a parameter acquisition unit. The parameter acquisition unit 21 also acquires parameters PRA that represent the viewpoint position and line of sight direction corresponding to each pixel of the endoscopic image EG, and outputs the acquired parameters PRA to the parameter conversion unit 22.

[0027] The endoscopic image EG is preferably an image obtained at the time when the capture button on the input unit 14 is operated, for example. The endoscopic image EG may also be an image extracted from an endoscopic video at regular intervals. The endoscopic image EG may also be an image having the best image quality among the endoscopic images included in the endoscopic video during a predetermined period from before to after the capture button on the input unit 14 is operated. In the following description, unless otherwise specified, it is assumed that tissue corresponding to a lesion is included in the endoscopic image EG as a subject within a body cavity.

[0028] Here, a specific example of the processing performed by the parameter acquisition unit 21 will be described.

[0029] In response to the input of the endoscopic image EG, the parameter acquisition unit 21 sets, for example, a three-dimensional Cartesian coordinate system (hereinafter also referred to as a camera coordinate system) TCS as shown in Fig. 4. Fig. 4 is a diagram for explaining parameters used in processing by the endoscopic examination support device according to the present disclosure.

[0030] Specifically, the parameter acquisition unit 21 sets the position of the endoscopic camera when the endoscopic image EG is acquired as the origin O of the camera coordinate system TCS. The parameter acquisition unit 21 also sets the horizontal direction as seen from the origin O as the X-axis direction of the camera coordinate system TCS. The parameter acquisition unit 21 also sets the vertical direction as seen from the origin O as the Y-axis direction of the camera coordinate system TCS. The parameter acquisition unit 21 also sets the depth direction as seen from the origin O as the Z-axis direction of the camera coordinate system TCS.

[0031] The parameter acquisition unit 21 acquires coordinates P (x, y, z) in the camera coordinate system TCS as parameters representing the viewpoint position corresponding to each pixel of the endoscopic image EG (see FIG. 4 ). The parameter acquisition unit 21 also acquires an azimuth angle α and an elevation / depression angle β when viewing the coordinate P from the origin O as parameters representing the line-of-sight direction corresponding to each pixel of the endoscopic image EG (see FIG. 4 ). The parameter acquisition unit 21 also acquires five-dimensional parameters (x, y, z, α, β) corresponding to each pixel of the endoscopic image EG as parameters PRA, and outputs the acquired parameters PRA to the parameter conversion unit 22.

[0032] The parameter acquisition unit 21 can acquire the x and y values ​​of the parameter P as coordinate values ​​corresponding to the position of each pixel included in the endoscopic image EG. The parameter acquisition unit 21 can also acquire the z value of the parameter P as a coordinate value corresponding to a processing result obtained by performing a process such as depth estimation on the endoscopic image EG. The parameter acquisition unit 21 may also acquire the z value of the parameter P as the same coordinate value corresponding to the position of each pixel included in the endoscopic image EG, such as an average value, a representative value, or a predetermined value. When calculating the z value of the parameter P, the parameter acquisition unit 21 may use, for example, the method disclosed in Pierluigi Zama Ramirez, et al., "Unsupervised Novel View Synthesis from a Single Image" (hereinafter referred to as Non-Patent Document DH1). The azimuth angle α of the parameter PRA can be rephrased as the angle in the left-right direction or the angle in the horizontal direction when the coordinate P is viewed from the origin O. The elevation / depression angle β of the parameter PRA can be rephrased as the angle in the up-down direction or the angle in the vertical direction when the coordinate P is viewed from the origin O.

[0033] The parameter conversion unit 22 functions as a parameter conversion means. The parameter conversion unit 22 also has a trained conversion model 221. The conversion model 221 is configured as a machine learning model trained to output four-dimensional parameters representing color and volume density in Neural Radiance Fields (NeRF) in response to input of five-dimensional parameters having the same elements as the parameter PRA. In other words, the conversion model 221 is trained to convert five-dimensional parameters having the same elements as the parameter PRA into four-dimensional parameters representing color and volume density in NeRF. The conversion model 221 is also based on, for example, Non-Patent Document DH1 and Ben Mildenhall, et. al, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (hereinafter referred to as Non-Patent Document DH2). According to the present embodiment, the training data used for training the transformation model 221 is preferably data including a plurality of lesion model images obtained by photographing a lesion model that mimics the shape of an actual lesion at a plurality of different angles. According to the present embodiment, the training data used for training the transformation model 221 is preferably data including a plurality of simulation images obtained by changing the angle of an actual lesion included in an image to a plurality of different angles. According to the present embodiment, the training data used for training the transformation model 221 may include data including at least one lesion model image and at least one simulation image.

[0034] The parameter conversion unit 22 inputs the parameters PRA (x, y, z, α, β) obtained by the parameter acquisition unit 21 into the conversion model 221, thereby acquiring four-dimensional parameters (R, G, B, σ) representing the color and volume density in the NeRF as parameters PRB, and outputs the acquired parameters PRB to the image generation unit 23.

[0035] The parameter conversion unit 22 can acquire the R value of the parameter PRB as a pixel value representing the magnitude of the red component at the viewpoint position and line of sight direction corresponding to the parameter PRA. The parameter conversion unit 22 can also acquire the G value of the parameter PRB as a pixel value representing the magnitude of the green component at the viewpoint position and line of sight direction corresponding to the parameter PRA. The parameter conversion unit 22 can also acquire the B value of the parameter PRB as a pixel value representing the magnitude of the blue component at the viewpoint position and line of sight direction corresponding to the parameter PRA. The parameter conversion unit 22 can also acquire the σ value of the parameter PRB as a density value representing the volume density at the viewpoint position and line of sight direction corresponding to the parameter PRA. The volume density can also be referred to as opacity.

[0036] As described above, the parameter conversion unit 22 can convert the parameter PRA into the parameter PRB using the conversion model 221 that has been trained to output parameters representing color and volume density in response to input of a parameter having the same elements as the parameter PRA.

[0037] The image generation unit 23 functions as an image generation means. Furthermore, the image generation unit 23 uses the parameters PRB obtained by the parameter conversion unit 22 to perform volume rendering processing according to a combination of a preset azimuth angle αp and an elevation / depression angle βp, thereby generating five arbitrary viewpoint endoscopic images corresponding to five different viewpoint positions. Furthermore, the image generation unit 23 distributes the five arbitrary viewpoint endoscopic images generated as described above to the five inference processing units 24A-24E, one by one. In other words, the image generation unit 23 exclusively outputs the five arbitrary viewpoint endoscopic images generated as described above to the five inference processing units 24A-24E, one by one. Note that the image generation unit 23 may use the method disclosed in Non-Patent Document DH2 or another method when performing the volume rendering processing.

[0038] Here, a specific example of an arbitrary viewpoint endoscopic image generated by the processing of the image generating unit 23 will be described.

[0039] For example, the image generating unit 23 performs volume rendering processing according to the combination of (αp, βp) = (0°, 0°) using the parameters PRB obtained by the parameter converting unit 22. By performing such processing, the image generating unit 23 can generate an endoscopic image EGN in which the subject included in the endoscopic image EG is viewed from the same position as the viewpoint position of the endoscopic image EG.

[0040] For example, the image generator 23 performs volume rendering processing according to the combination of (αp, βp) = (+10°, 0°) using the parameters PRB obtained by the parameter converter 22. By performing such processing, the image generator 23 can generate an endoscopic image EGL in which the subject included in the endoscopic image EG is viewed from a position shifted to the left with respect to the viewpoint position of the endoscopic image EG.

[0041] For example, the image generating unit 23 performs volume rendering processing according to the combination of (αp, βp)=(−10°, 0°) using the parameters PRB obtained by the parameter converting unit 22. By performing such processing, the image generating unit 23 can generate an endoscopic image EGR in which the subject included in the endoscopic image EG is viewed from a position shifted to the right with respect to the viewpoint position of the endoscopic image EG.

[0042] For example, the image generating unit 23 performs volume rendering processing according to the combination of (αp, βp) = (0°, +10°) using the parameters PRB obtained by the parameter converting unit 22. By performing such processing, the image generating unit 23 can generate an endoscopic image EGD in which the subject included in the endoscopic image EG is viewed from a position shifted downward from the viewpoint position of the endoscopic image EG.

[0043] For example, the image generating unit 23 performs volume rendering processing according to the combination of (αp, βp)=(0°, −10°) using the parameters PRB obtained by the parameter converting unit 22. By performing such processing, the image generating unit 23 can generate an endoscopic image EGU in which the subject included in the endoscopic image EG is viewed from a position shifted upward relative to the viewpoint position of the endoscopic image EG.

[0044] As described above, the image generation unit 23 can generate a group of images including multiple arbitrary viewpoint endoscopic images corresponding to different viewpoint positions using the parameters PRB obtained based on the endoscopic image EG including the subject inside the body cavity.

[0045] In this embodiment, an example will be described in which the endoscopic image EGN is output to the inference processing unit 24A, the endoscopic image EGL is output to the inference processing unit 24B, the endoscopic image EGR is output to the inference processing unit 24C, the endoscopic image EGD is output to the inference processing unit 24D, and the endoscopic image EGU is output to the inference processing unit 24E. In this embodiment, as shown in FIG. 5 , the five inference processing units 24A to 24E have the same functions. Specifically, the five inference processing units 24A to 24E function as inference means that perform inference processing based on the endoscopic images EGN, EGL, EGR, EGD, and EGU. Therefore, hereinafter, the function of the inference processing unit 24A will be mainly described, while the functions of the inference processing units 24B to 24E will be omitted as appropriate. FIG. 5 is a diagram illustrating an example of the functional configuration of the inference processing units.

[0046] The inference processing unit 24A is configured to perform inference processing corresponding to the input signal NS generated by the input unit 14 as processing on the endoscopic image EGN obtained from the image generation unit 23, and output the inference result ERN obtained by the inference processing to the inference result integration unit 25. Specifically, for example, when the input unit 14 generates an input signal NS related to lesion detection, the inference processing unit 24A performs inference processing to detect the presence or absence of a lesion in the endoscopic image EGN. Furthermore, for example, when the input unit 14 generates an input signal NS related to lesion diagnosis, the inference processing unit 24A performs inference processing to evaluate whether a lesion included in the endoscopic image EGN is neoplastic or non-neoplastic. Furthermore, for example, when the input unit 14 generates an input signal NS related to lesion size measurement, the inference processing unit 24A performs inference processing to estimate the size of a lesion included in the endoscopic image EGN. As shown in FIG. 5 , the inference processing unit 24A includes a detection model 241, a classification model 242, and a size estimation model 243.

[0047] The detection model 241 is configured as a machine learning model trained to perform inference processing to detect the presence or absence of a lesion in an endoscopic image input from outside and to output the inference result obtained by the inference processing. With such a configuration of the detection model 241, it is possible to obtain an inference result that can identify, for example, the presence or absence of a lesion in an endoscopic image and the location of the lesion in the endoscopic image. Furthermore, with the above-described configuration of the detection model 241, it is possible to obtain an inference result in which a relatively high score is assigned when a lesion is included in an endoscopic image, and a relatively low score is assigned when a lesion is not included in the endoscopic image.

[0048] The classification model 242 is configured as a machine learning model trained to perform an inference process to evaluate the neoplastic or non-neoplastic nature of lesions included in endoscopic images input from an external source and to output the inference results obtained by the inference process. This configuration of the classification model 242 makes it possible to obtain an inference result that can identify whether a lesion included in an endoscopic image is neoplastic or non-neoplastic. Furthermore, the configuration of the classification model 242 described above makes it possible to obtain an inference result in which the sum of a score indicating the neoplastic nature of a lesion included in an endoscopic image and a score indicating the non-neoplastic nature of the lesion is 1.

[0049] The size estimation model 243 is configured as a machine learning model trained to perform an inference process to estimate the size of a lesion included in an endoscopic image input from an external device and to output the inference result obtained by the inference process. This configuration of the size estimation model 243 makes it possible to obtain an inference result that can identify, for example, whether a lesion included in an endoscopic image is larger than a predetermined size. Furthermore, the configuration of the size estimation model 243 described above makes it possible to obtain an inference result in which a relatively high score is assigned when the size of the lesion included in the endoscopic image is larger than a predetermined size, and a relatively low score is assigned when the size of the lesion is equal to or smaller than the predetermined size. It is preferable that the predetermined size be set to, for example, 5 mm.

[0050] The inference processing unit 24B is configured to perform inference processing corresponding to the input signal NS generated in the input unit 14 as processing on the endoscopic image EGL obtained from the image generation unit 23, and to output the inference result ERL obtained by the inference processing to the inference result integration unit 25. The inference processing unit 24B also has the same detection model 241, classification model 242, and size estimation model 243 as the inference processing unit 24A. Therefore, the inference processing unit 24B can perform the same inference processing as the inference processing unit 24A in accordance with the input signal NS generated in the input unit 14.

[0051] The inference processing unit 24C is configured to perform inference processing corresponding to the input signal NS generated in the input unit 14 as processing on the endoscopic image EGR obtained from the image generation unit 23, and to output the inference result ERR obtained by the inference processing to the inference result integration unit 25. The inference processing unit 24C also has the same detection model 241, classification model 242, and size estimation model 243 as the inference processing unit 24A. Therefore, the inference processing unit 24C can perform the same inference processing as the inference processing unit 24A in accordance with the input signal NS generated in the input unit 14.

[0052] The inference processing unit 24D is configured to perform inference processing corresponding to the input signal NS generated in the input unit 14 as processing on the endoscopic image EGD obtained from the image generation unit 23, and to output the inference result ERD obtained by the inference processing to the inference result integration unit 25. The inference processing unit 24D also has a detection model 241, a classification model 242, and a size estimation model 243 similar to those of the inference processing unit 24A. Therefore, the inference processing unit 24D can perform the same inference processing as the inference processing unit 24A in accordance with the input signal NS generated in the input unit 14.

[0053] The inference processing unit 24E is configured to perform inference processing corresponding to the input signal NS generated in the input unit 14 as processing on the endoscopic image EGU obtained from the image generation unit 23, and to output the inference result ERU obtained by the inference processing to the inference result integration unit 25. The inference processing unit 24E also has the same detection model 241, classification model 242, and size estimation model 243 as the inference processing unit 24A. Therefore, the inference processing unit 24E can perform the same inference processing as the inference processing unit 24A in accordance with the input signal NS generated in the input unit 14.

[0054] As described above, the inference processing units 24A to 24E can obtain five inference results by individually performing inference processing on each of the five arbitrary viewpoint endoscopic images.

[0055] The inference result integration unit 25 performs processing to integrate the inference result ERN output from the inference processing unit 24A, the inference result ERL output from the inference processing unit 24B, the inference result ERR output from the inference processing unit 24C, the inference result ERD output from the inference processing unit 24D, and the inference result ERU output from the inference processing unit 24E. Furthermore, the inference result integration unit 25 acquires the integrated inference result obtained by the above-mentioned processing as a final inference result ERF corresponding to the endoscopic image EG, and outputs the acquired final inference result ERF to the outside.

[0056] Here, a specific example of the processing performed in the inference result integration unit 25 will be described. For ease of explanation, the inference results ERN, ERL, ERR, ERD, and ERU will also be referred to as five inference results hereinafter.

[0057] The inference result integration unit 25 acquires the inference result that accounts for the majority of the five inference results as the integrated inference result. Such processing by the inference result integration unit 25 can be treated as processing corresponding to hard voting.

[0058] Alternatively, the inference result integration unit 25 acquires an integrated inference result based on the sum of the scores included in each of the five inference results. Specifically, for example, if five inference results are obtained by an inference process using the detection model 241, the inference result integration unit 25 acquires an integrated inference result based on whether the sum of the scores included in each of the five inference results is higher than a predetermined value. For example, if five inference results are obtained by an inference process using the classification model 242, the inference result integration unit 25 acquires an integrated inference result based on the results of comparing the sum of the scores indicating the neoplastic nature of the lesions included in each of the five inference results with the sum of the scores indicating the non-neoplastic nature of the lesions included in each of the five inference results. For example, if five inference results are obtained by an inference process using the size estimation model 243, the inference result integration unit 25 acquires an integrated inference result based on whether the sum of the scores included in each of the five inference results is higher than a predetermined value. These processes by the inference result integration unit 25 can be treated as processes corresponding to soft voting.

[0059] According to the processing described in the specific example above, for example, if five inference results are obtained by the inference processing using the detection model 241, the inference result integration unit 25 can obtain either an inference result that a lesion is present in the endoscopic image EG or an inference result that a lesion is not present in the endoscopic image EG as the final inference result ERF corresponding to the endoscopic image EG. Also, according to the processing described in the specific example above, for example, if five inference results are obtained by the inference processing using the classification model 242, the inference result integration unit 25 can obtain either an inference result that the lesion included in the endoscopic image EG is neoplastic or an inference result that the lesion included in the endoscopic image EG is non-neoplastic as the final inference result ERF corresponding to the endoscopic image EG. Also, according to the processing described in the specific example above, for example, if five inference results are obtained by the inference processing using the size estimation model 243, the inference result integration unit 25 can obtain either an inference result that the lesion included in the endoscopic image EG is larger than a predetermined size or an inference result that the lesion included in the endoscopic image EG is equal to or smaller than the predetermined size as the final inference result ERF corresponding to the endoscopic image EG.

[0060] As described above, the inference result integration unit 25 can acquire a final inference result corresponding to the endoscopic image EG by integrating multiple inference results obtained by performing inference processing individually on each of multiple arbitrary viewpoint endoscopic images. Also, as described above, the inference result integration unit 25 can integrate multiple inference results by performing processing corresponding to either hard voting or soft voting.

[0061] According to this embodiment, the endoscopic examination support device 1 may perform processing for presenting the final estimation result ERF to the surgeon. Specifically, for example, when the final inference result ERF is obtained through inference processing using the detection model 241, a frame indicating the position of the lesion in the endoscopic image EG may be displayed. Furthermore, for example, when the final inference result ERF is obtained through inference processing using the classification model 242, text information indicating the state of the lesion contained in the endoscopic image EG may be displayed. Furthermore, for example, when the final inference result ERF is obtained through inference processing using the size estimation model 243, text information indicating the size of the lesion contained in the endoscopic image EG may be displayed.

[0062] According to this embodiment, the image generation unit 23 may be configured to generate p arbitrary viewpoint endoscopic images corresponding to p (p≧2) different viewpoint positions, respectively. In such a case, the endoscopic examination support device 1 may be provided with p inference processing units having the same function as the inference processing unit 24A in order to perform inference processing individually for each of the p arbitrary viewpoint endoscopic images.

[0063] According to this embodiment, the endoscopic image EG may be input to the inference processing unit 24A instead of the endoscopic image EGN. In such a case, the image generation unit 23 may use the parameter PRB to generate an image group including (p−1) arbitrary viewpoint endoscopic images corresponding to (p−1) viewpoint positions different from the viewpoint position of the endoscopic image EG. In the above-described case, the endoscopic examination support device 1 may be provided with p inference processing units having the same function as the inference processing unit 24A to individually perform inference processing on the endoscopic image EG and the (p−1) arbitrary viewpoint endoscopic images. In the above-described case, the p inference processing units can perform inference processing based on the endoscopic image EG and the (p−1) arbitrary viewpoint endoscopic images. In the above-described case, the inference result integration unit 25 can integrate multiple inference results obtained by individually performing inference processing on the endoscopic image EG and the (p−1) arbitrary viewpoint endoscopic images, thereby obtaining a final inference result corresponding to the endoscopic image EG.

[0064] According to this embodiment, when the inference result integration unit 25 performs processing corresponding to hard voting, it is desirable that an odd number of inference results be input to the inference result integration unit 25 .

[0065] [Processing Flow] Next, a description will be given of the flow of processing performed in the endoscopic examination support device 1. Fig. 6 is a flowchart showing an example of processing performed in the endoscopic examination support device according to the present disclosure.

[0066] First, the endoscopic examination support device 1 acquires five-dimensional parameters that represent the viewpoint position and line of sight direction corresponding to each pixel of the endoscopic image (step S11).

[0067] Next, the endoscopic examination support device 1 performs processing using the conversion model 221 to convert the five-dimensional parameters obtained in step S11 into four-dimensional parameters representing color and volume density in NeRF (step S12).

[0068] Next, the endoscopic examination support device 1 performs volume rendering processing using the four-dimensional parameters obtained in step S12 to generate a plurality of endoscopic images corresponding to a plurality of different viewpoint positions (step S13). Note that in step S13, the endoscopic examination support device 1 may generate an endoscopic image having the same viewpoint position as the viewpoint position of the endoscopic image processed in step S11.

[0069] Next, the endoscopic examination support device 1 performs an inference process individually on each of the multiple endoscopic images generated in step S13, thereby obtaining multiple inference results corresponding to each of the multiple endoscopic images (step S14). According to this embodiment, the endoscopic examination support device 1 can perform the inference process in step S14 using any one of the detection model 241, the classification model 242, and the size estimation model 243.

[0070] Next, the endoscopic examination support device 1 performs a process for integrating the multiple inference results obtained by the inference process of step S14 (step S15). According to this embodiment, the endoscopic examination support device 1 can selectively perform a process corresponding to either hard voting or soft voting as the process of step S15.

[0071] Finally, the endoscopic examination support device 1 acquires the integrated estimation result obtained in step S15 as the final inference result corresponding to the endoscopic image processed in step S11, and outputs the acquired final estimation result (step S16).

[0072] As described above, according to this embodiment, multiple inference results can be obtained by individually performing inference processing on multiple endoscopic images corresponding to multiple different viewpoint positions, and a final inference result can be obtained by integrating the multiple inference results. Therefore, according to this embodiment, the accuracy of the inference results obtained by, for example, performing inference processing related to detecting the presence or absence of a lesion in an endoscopic image can be improved. Furthermore, according to this embodiment, the accuracy of the inference results obtained by, for example, performing inference processing related to diagnosing (classifying) a lesion included in an endoscopic image can be improved. Furthermore, the accuracy of the inference results obtained by, for example, performing inference processing related to estimating the size of a lesion included in an endoscopic image can be improved. Therefore, according to this embodiment, the accuracy of information related to a lesion obtained from an endoscopic image can be improved.

[0073] Second Embodiment Next, a second embodiment of the present disclosure will be described. Note that, for simplicity, specific descriptions of parts to which the above-described processes and the like can be applied will be omitted as appropriate.

[0074] Fig. 7 is a block diagram showing another example of the functional configuration of the endoscopic examination support device according to the present disclosure. The endoscopic examination support device 1A has the same hardware configuration as the endoscopic examination support device 1. As shown in Fig. 7, the endoscopic examination support device 1A also has a parameter acquisition unit 21, a parameter conversion unit 22, an image generation unit 23, and an inference processing unit 27. Note that, hereinafter, for convenience of explanation, the endoscopic images EGN, EGL, EGR, EGD, and EGU will also be referred to as five endoscopic images.

[0075] The image generating unit 23 generates five endoscopic images using the same method as that described in the first embodiment, and outputs the five generated endoscopic images to the inference processing unit 27 .

[0076] The inference processing unit 27 functions as an inference means. The inference processing unit 27 is configured to perform inference processing corresponding to the input signal NS generated by the input unit 14 as processing for the five endoscopic images obtained from the image generation unit 23, and to output a comprehensive inference result ERV obtained by the inference processing to the outside. Specifically, for example, when the input signal NS related to lesion detection is generated by the input unit 14, the inference processing unit 27 performs inference processing to detect the presence or absence of a lesion in each of the five endoscopic images. For example, when the input signal NS related to lesion diagnosis is generated by the input unit 14, the inference processing unit 27 performs inference processing to evaluate whether the lesion included in each of the five endoscopic images is neoplastic or non-neoplastic. For example, when the input signal NS related to lesion size measurement is generated by the input unit 14, the inference processing unit 27 performs inference processing to estimate the size of the lesion included in each of the five endoscopic images. As shown in FIG. 8 , the inference processing unit 27 includes a detection model 271, a classification model 272, and a size estimation model 273. FIG. 8 is a diagram illustrating another example of the functional configuration of the inference processing unit.

[0077] The detection model 271 is configured as a machine learning model that is trained to perform inference processing to detect the presence or absence of a lesion in each of multiple endoscopic images simultaneously input from the outside and to output a comprehensive inference result obtained by the inference processing. With such a configuration of the detection model 271, it is possible to obtain, as the comprehensive inference result, either an inference result that a lesion is present in the endoscopic image or an inference result that a lesion is not present in the endoscopic image.

[0078] The classification model 272 is configured as a machine learning model trained to perform inference processing to evaluate the neoplastic or non-neoplastic nature of lesions contained in each of multiple endoscopic images simultaneously input from the outside, and to output a comprehensive inference result obtained by the inference processing. With such a configuration of the classification model 272, it is possible to obtain, as the comprehensive inference result, either an inference result that the lesion contained in the endoscopic image is neoplastic, or an inference result that the lesion contained in the endoscopic image is non-neoplastic.

[0079] The size estimation model 273 is configured as a machine learning model trained to perform an inference process to estimate the size of a lesion contained in each of multiple endoscopic images simultaneously input from an external device and to output a comprehensive inference result obtained by the inference process. This configuration of the size estimation model 273 allows the comprehensive inference result to be obtained as either an inference result that the lesion contained in the endoscopic image is larger than a predetermined size, or an inference result that the lesion contained in the endoscopic image is equal to or smaller than the predetermined size. It is desirable that the predetermined size be set to, for example, 5 mm.

[0080] According to the processing described above, when the inference processing unit 27 performs inference processing using the detection model 271, for example, the inference processing unit 27 can obtain either an inference result that a lesion is present in the endoscopic image EG or an inference result that a lesion is not present in the endoscopic image EG as the overall inference result ERV corresponding to the endoscopic image EG. Furthermore, according to the processing described above, when the inference processing unit 27 performs inference processing using the classification model 272, for example, the inference processing unit 27 can obtain either an inference result that a lesion included in the endoscopic image EG is neoplastic or an inference result that a lesion included in the endoscopic image EG is non-neoplastic. Furthermore, according to the processing described above, when the inference processing unit 27 performs inference processing using the size estimation model 273, for example, the inference processing unit 27 can obtain either an inference result that a lesion included in the endoscopic image EG is larger than a predetermined size or an inference result that a lesion included in the endoscopic image EG is equal to or smaller than the predetermined size as the overall inference result ERV corresponding to the endoscopic image EG.

[0081] According to this embodiment, the endoscopic examination support device 1 may perform processing for presenting the comprehensive inference result ERV to the surgeon. Specifically, for example, when the comprehensive inference result ERV is obtained through inference processing using the detection model 271, a frame indicating the position of the lesion in the endoscopic image EG may be displayed. Furthermore, for example, when the comprehensive inference result ERV is obtained through inference processing using the classification model 272, text information indicating the state of the lesion included in the endoscopic image EG may be displayed. Furthermore, for example, when the comprehensive inference result ERV is obtained through inference processing using the size estimation model 273, text information indicating the size of the lesion included in the endoscopic image EG may be displayed.

[0082] According to this embodiment, the endoscopic image EG may be input to the inference processing unit 27 instead of the endoscopic image EGN. In such a case, the image generating unit 23 may use the parameter PRB to generate q arbitrary viewpoint endoscopic images corresponding to q (q≧1) viewpoint positions different from the viewpoint position of the endoscopic image EG. In the above-described case, the inference processing unit 27 can obtain a comprehensive inference result by performing inference processing based on the endoscopic image EG and the q arbitrary viewpoint endoscopic images.

[0083] [Processing Flow] Next, a description will be given of the flow of processing performed in the endoscopic examination support device 1 A. Fig. 9 is a flowchart showing another example of processing performed in the endoscopic examination support device according to the present disclosure.

[0084] The endoscopic examination support device 1A performs the same processes as steps S11 to S13 in the first embodiment as the processes in steps S31 to S33.

[0085] Next, the endoscopic examination support device 1A performs inference processing using the multiple endoscopic images generated in step S33 at once, thereby acquiring a comprehensive inference result corresponding to the endoscopic images processed in step S31 (step S34). According to this embodiment, the endoscopic examination support device 1A can perform processing using any one of the detection model 271, the classification model 272, and the size estimation model 273 as the inference processing in step S34.

[0086] Finally, the endoscopic examination support device 1A outputs the overall estimation result obtained in step S34 (step S35).

[0087] As described above, according to this embodiment, a comprehensive inference result can be obtained by simultaneously performing inference processing using multiple endoscopic images corresponding to multiple different viewpoint positions. Therefore, according to this embodiment, for example, the accuracy of the inference result obtained by performing inference processing related to detecting the presence or absence of a lesion in an endoscopic image can be improved. Furthermore, according to this embodiment, for example, the accuracy of the inference result obtained by performing inference processing related to diagnosing (classifying) a lesion included in an endoscopic image can be improved. Furthermore, for example, the accuracy of the inference result obtained by performing inference processing related to estimating the size of a lesion included in an endoscopic image can be improved. Therefore, according to this embodiment, the accuracy of information related to a lesion obtained from an endoscopic image can be improved.

[0088] Third Embodiment FIG. 10 is a block diagram showing another example of the functional configuration of the endoscopic examination support device according to the present disclosure.

[0089] The endoscopic examination support device 500 according to this embodiment has the same hardware configuration as the endoscopic examination support device 1. The endoscopic examination support device 500 also has an image generation unit 501 and an inference unit 502.

[0090] The image generating means 501 can be realized, for example, by using the function of the image generating unit 23 in the first embodiment. The inference means 502 can be realized, for example, by using the function of the inference processing units 24A to 24E in the first embodiment or the inference processing unit 27 in the second embodiment.

[0091] FIG. 11 is a flowchart showing another example of processing performed in the endoscopic examination support device according to the present disclosure.

[0092] The image generating means 501 generates a first image group including a plurality of arbitrary viewpoint endoscopic images corresponding to different viewpoint positions using parameters obtained based on an endoscopic image including an object inside a body cavity (step S51), or generates a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from the viewpoint position of the endoscopic image using parameters obtained based on an endoscopic image including an object inside a body cavity (step S51).

[0093] The inference means 502 performs inference processing related to the subject based on the first image group (step S52), or the inference means 502 performs inference processing related to the subject based on the endoscopic images and the second image group (step S52).

[0094] According to this embodiment, it is possible to improve the accuracy of information relating to a lesion obtained from an endoscopic image.

[0095] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0096] (Supplementary Note 1) An endoscopic examination support device having: an image generation means for generating a first image group including a plurality of arbitrary viewpoint endoscopic images corresponding to different viewpoint positions using parameters obtained based on endoscopic images including a subject inside a body cavity, or for generating a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic images; and an inference means for performing inference processing related to the subject based on the first image group, or based on the endoscopic images and the second image group.

[0097] (Appendix 2) The endoscopic examination support device of Appendix 1 further comprises an integration means for integrating multiple inference results obtained by performing the inference processing individually on each of the multiple arbitrary viewpoint endoscopic images included in the first image group, thereby obtaining a final inference result corresponding to the endoscopic image.

[0098] (Supplementary Note 3) The endoscopic examination support device according to Supplementary Note 2, wherein the integration means integrates the plurality of inference results by performing processing corresponding to either hard voting or soft voting.

[0099] (Appendix 4) The endoscopic examination support device of Appendix 1 further comprises an integration means for integrating multiple inference results obtained by individually performing the inference processing on the endoscopic image and the at least one arbitrary viewpoint endoscopic image included in the second image group, thereby obtaining a final inference result corresponding to the endoscopic image.

[0100] (Supplementary Note 5) The endoscopic examination support device according to Supplementary Note 4, wherein the integration means integrates the plurality of inference results by performing processing corresponding to either hard voting or soft voting.

[0101] (Supplementary Note 6) An endoscopic examination support device according to Supplementary Note 1, further comprising: a parameter acquisition means for acquiring first parameters representing a viewpoint position and a line of sight direction corresponding to each pixel of the endoscopic image; and a parameter conversion means for converting the first parameters into second parameters, wherein the image generation means generates the first group of images or the second group of images using the second parameters.

[0102] (Supplementary Note 7) The endoscopic examination support device according to Supplementary Note 6, wherein the parameter conversion means converts the first parameter into the second parameter using a machine learning model that has been trained to output parameters representing color and volume density in response to input of a parameter having the same elements as the first parameter.

[0103] (Supplementary Note 8) A method for supporting endoscopic examinations, which uses parameters obtained based on endoscopic images including a subject inside a body cavity to generate a first group of images including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position from each other, or generates a second group of images including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic images, and performs inference regarding the subject based on the first group of images or based on the endoscopic images and the second group of images.

[0104] (Supplementary Note 9) A recording medium having recorded thereon a program that causes a computer to execute a process of using parameters obtained based on an endoscopic image including a subject inside a body cavity to generate a first group of images including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or to generate a second group of images including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic image, and to make an inference regarding the subject based on the first group of images, or based on the endoscopic image and the second group of images.

[0105] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0106] REFERENCE SIGNS LIST 1, 1A Endoscopy support device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 21 Parameter acquisition unit 22 Parameter conversion unit 23 Image generation unit 24A, 24B, 24C, 24D, 24E, 27 Inference processing unit 25 Inference result integration unit 100 Endoscopy system

Claims

1. An endoscopic examination support device having: an image generation means for generating a first image group including a plurality of arbitrary viewpoint endoscopic images corresponding to different viewpoint positions using parameters obtained based on an endoscopic image including a subject within a body cavity, or for generating a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic images; and an inference means for performing inference processing related to the subject based on the first image group, or based on the endoscopic images and the second image group.

2. An endoscopic examination support device as described in claim 1, further comprising an integration means for integrating multiple inference results obtained by performing the inference processing individually on each of the multiple arbitrary viewpoint endoscopic images included in the first image group, thereby obtaining a final inference result corresponding to the endoscopic image.

3. The endoscopic examination support device according to claim 2, wherein the integration means integrates the multiple inference results by performing processing corresponding to either hard voting or soft voting.

4. An endoscopic examination support device as described in claim 1, further comprising an integration means for integrating multiple inference results obtained by performing the inference processing individually on the endoscopic image and at least one arbitrary viewpoint endoscopic image included in the second image group, thereby obtaining a final inference result corresponding to the endoscopic image.

5. The endoscopic examination support device according to claim 4, wherein the integration means integrates the multiple inference results by performing processing corresponding to either hard voting or soft voting.

6. An endoscopic examination support device as described in claim 1, further comprising: a parameter acquisition means for acquiring first parameters representing a viewpoint position and a line of sight direction corresponding to each pixel of the endoscopic image; and a parameter conversion means for converting the first parameters into second parameters, wherein the image generation means generates the first image group or the second image group using the second parameters.

7. The endoscopic examination support device according to claim 6, wherein the parameter conversion means converts the first parameter into the second parameter using a machine learning model trained to output parameters representing color and volume density in response to input of a parameter having the same elements as the first parameter.

8. A method for supporting endoscopic examination, which uses parameters obtained based on endoscopic images including a subject within a body cavity to generate a first image group including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or generates a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic images, and performs inference related to the subject based on the first image group or based on the endoscopic images and the second image group.

9. A recording medium having recorded thereon a program for causing a computer to execute a process of using parameters obtained based on an endoscopic image including a subject within a body cavity to generate a first image group including a plurality of arbitrary viewpoint endoscopic images each corresponding to a different viewpoint position, or to generate a second image group including at least one arbitrary viewpoint endoscopic image corresponding to a viewpoint position different from that of the endoscopic images, and to make an inference regarding the subject based on the first image group, or based on the endoscopic images and the second image group.

Images