Image processing device, image processing method and program

Abstract

An image processing device (1X) comprises an acquisition means (31X), a selection means (32X), and an assessment means (33X). The acquisition means (31X) acquires data obtained by Fourier-transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope. The selection means (32X) selects, from the data, partial data which is a portion of the data. On the basis of the partial data, the assessment means (33X) performs assessment of a region of interest which requires attention with respect to the subject. The invention can be used to assist a user in decision making or the like.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images acquired during an endoscopic examination.

[0002]

[0003] Conventionally, endoscopic examination systems that display images of the inside of organ lumens have been known. For example, Patent Literature 1 discloses an endoscopic examination system that detects a region of interest based on an endoscopic image and a region-of-interest detection threshold and determines whether the region of interest is a flat lesion or a raised lesion. Furthermore, Patent Literature 2 discloses an image processing device that generates a tomographic image by performing an inverse Fourier transform on k-space data obtained by an MRI device.

[0003] International Publication WO2019 / 146077 JP 2021-041065 A

[0004] The process of determining a site of interest, such as a lesion, from an image captured during an endoscopic examination requires real-time processing. Considering the possibility of performing complex processing, such as performing additional processing using the determination results, it is preferable to reduce the amount of calculation required for the above-mentioned determination process as much as possible.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of performing judgments regarding areas of interest while suppressing an increase in the amount of calculations during endoscopic examinations.

[0006] One aspect of the image processing device is an image processing device having: an acquisition means for acquiring data obtained by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope; a selection means for selecting partial data that is a part of the data; and a determination means for determining a location of interest that should be noted in the subject based on the partial data.

[0007] One aspect of the image processing method is an image processing method in which a computer acquires data by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope, selects partial data that is a part of the data, and determines a location of interest that should be noted in the subject based on the partial data.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of acquiring data by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope, selecting partial data that is a part of the data, and making a determination regarding a location of interest that should be noted in the subject based on the partial data.

[0009] As an example of an effect of the present disclosure, in an endoscopic examination, it becomes possible to perform a determination regarding a point of interest while suppressing an increase in the amount of calculations during the endoscopic examination.

[0010] 5A to 5C show examples of k-space data in which selected and non-selected regions are clearly indicated when selected as partial data in the k-space data. 5B show graphs illustrating the accuracy rate in an experiment using the partial data and k-space data shown in FIGS. 5A to 5C in a colonoscopy. 5C show examples of k-space data in which selected and non-selected regions are clearly indicated when non-selected regions are provided in both a partial range of the kx-axis and a partial range of the ky-axis. 5D show examples of display screens displayed by a display device during an endoscopic examination. 5E show an example of a flowchart illustrating an overview of processing executed by an image processing device during an endoscopic examination in the first embodiment. 5F shows a schematic configuration diagram of an endoscopic examination system in a modified example. 5G shows a block diagram of an image processing device in a second embodiment. 5H shows an example of a flowchart executed by an image processing device in the second embodiment.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First embodiment> (1) System configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. As shown in Fig. 1, the endoscopic examination system 100 is a system that presents information about a region of a subject suspected of having a lesion (also referred to as a "lesion region") to an examiner such as a doctor who performs an examination or treatment using an endoscope, and mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1. The lesion region is an example of a "region of interest."

[0013] The image processing device 1 acquires images (also referred to as "endoscopic images Ia") captured by the endoscope 3 in time series from the endoscope 3, and displays a screen based on the endoscopic images Ia on the display device 2. The endoscopic images Ia are images captured at a predetermined frame rate during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, the image processing device 1 analyzes the endoscopic images Ia to determine the presence or absence of a lesion site from the endoscopic images Ia, and displays information related to the determination result on the display device 2.

[0014] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the image processing device 1 .

[0015] The endoscope 3 mainly comprises an operation unit 36 ​​for the examiner to input predetermined information, a flexible shaft 37 that is inserted into the subject's organ to be photographed, a tip 38 that incorporates an imaging unit such as a micro-imaging element, and a connection unit 39 for connecting to the image processing device 1.

[0016] 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be configured from multiple devices.

[0017] Note that the subject of endoscopic examination in the present disclosure is not limited to the large intestine, but may be any organ that can be examined endoscopically, such as the esophagus, stomach, pancreas, etc. For example, endoscopes that are applicable in the present disclosure include pharyngoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes. Furthermore, examples of the pathological conditions of lesions that are the subject of endoscopic examination include the following (a) to (f).

[0018] (a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: esophageal cancer, esophagitis, Barred esophagus, hiatal hernia, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0019] (2) Hardware Configuration Fig. 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0020] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), etc. The processor 11 is an example of a computer.

[0021] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment.

[0022] The memory 12 also stores model information D1. The model information D1 is information about a lesion determination model that outputs a determination result regarding a lesion site in an endoscopic image. The model information D1 includes parameters necessary to configure the lesion determination model.

[0023] The lesion determination model is a model trained to output a determination result regarding a lesion site in an endoscopic image when input data based on the endoscopic image is input. In other words, the lesion determination model is a model that has learned the relationship between the input data input to the lesion determination model and the determination result regarding the lesion site in the endoscopic image used to generate the input data. Here, the lesion determination model may be a model that determines at least one of the presence or absence of a specific type of disease or the severity of the disease, or may be a model that determines at least the type of disease. For example, the lesion determination model may determine the degree of inflammation or bleeding in a site captured in an endoscopic image. Furthermore, the lesion determination model may output information indicating the position or area of ​​the lesion site in the input endoscopic image instead of or in addition to the above-mentioned determination result.

[0024] Here, a supplementary explanation will be given regarding the learning of the lesion determination model. The lesion determination model is trained in advance based on a pair of input data conforming to the input format of the lesion determination model and correct answer data indicating the correct determination result regarding the lesion site in the endoscopic image used to generate the input data. As will be described later, the data input to the lesion determination model is data in a partial frequency domain selected from k-space data obtained by Fourier transforming the endoscopic image. Here, the lesion determination model may be, for example, any machine learning model (including statistical models, the same applies hereinafter) such as a neural network or a support vector machine. Representative models of such neural networks include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. When the lesion determination model is configured by a neural network, the model information D1 includes various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter.

[0025] The lesion determination model is not limited to a machine learning model, but may be a model that determines the presence or absence of a lesion site due to a specific disease based on the RGB ratio of an input endoscopic image. For example, the lesion determination model may be a model that determines the presence of a lesion site due to a specific disease (e.g., inflammation) when the R ratio (e.g., the average ratio of all pixels) of the RGB of the input endoscopic image is equal to or greater than a predetermined threshold. In this case, a calculation formula and a threshold for calculating the R ratio are stored in advance in the memory 12 as model information D1. Furthermore, if there are multiple possible diseases, a lesion determination model may be provided for each possible disease. In this case, parameters for constructing a lesion determination model for each possible disease are stored in the model information D1.

[0026] The interface 13 acts as an interface between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0027] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0028] (4) Lesion Determination Processing The lesion determination processing, which is processing related to determining the lesion site, will be described.

[0029] (4-1) Overview FIG. 3 is a diagram showing an overview of the lesion determination process executed by the image processing device 1.

[0030] The image processing device 1 first generates data in k-space (also referred to as "k-space data") by performing a Fourier transform (more specifically, a two-dimensional Fourier transform in the vertical and horizontal directions of the image) on the endoscopic image Ia acquired during the endoscopic examination. Hereinafter, the position coordinates in real space corresponding to the horizontal and vertical axes of the endoscopic image Ia will be represented as "(x, y)," and the spatial frequency coordinates corresponding to the horizontal and vertical axes of the k-space data will be represented as "(kx, ky)."

[0031] For example, the k-space data is data in an image format (third-order tensor), and is data obtained by converting data representing complex numbers for each spatial frequency obtained by Fourier transforming the endoscopic image Ia into absolute values ​​for each spatial frequency. The k-space data may be the data itself obtained by Fourier transforming the endoscopic image Ia (i.e., data representing complex numbers for each spatial frequency), or may be data obtained by converting the data obtained by Fourier transforming the endoscopic image Ia into data representing the deflection angle (i.e., phase) for each spatial frequency. The k-space data shown in FIG. 3 may be data obtained by logarithmically transforming the value (complex number, absolute value, or phase) for each spatial frequency. The k-space data shown in FIG. 3 is set such that the higher the value for each spatial frequency, the higher the luminance at that spatial frequency.

[0032] Next, the image processing device 1 selects data (also referred to as "partial data") of a portion of the frequency domain in k-space where the k-space data exists from the k-space data. The partial data is data in an image format (third-order tensor) and is data that is consistent with the input format of the lesion detection model. In the example of FIG. 3, the image processing device 1 generates partial data by selecting the upper three-quarters range (value range) on the k-axis where the k-space data exists. Note that in FIG. 3, the frequency domain of k-space selected as partial data from the k-space data is clearly indicated as a "selected region," and the frequency domain of k-space not selected as partial data from the k-space data is clearly indicated as a "non-selected region." This allows the image processing device 1 to suitably reduce the amount of data used to determine the lesion site.

[0033] The image processing device 1 then inputs the partial data into a lesion determination model to which the trained parameters stored in the model information D1 have been applied, and obtains a determination result (lesion determination result) regarding the lesion site output by the lesion determination model in response to the input. The image processing device 1 then performs processing such as displaying information based on the lesion determination result output by the lesion determination model on the display device 2, and making further determinations (including automatic diagnosis) regarding the lesion site based on the lesion determination result.

[0034] By performing such processing, the image processing device 1 can preferably reduce the amount of calculation required for determining the lesion site while maintaining the accuracy of the lesion site determination. As a result, even when additional processing is performed based on the lesion determination result, it is possible to reduce the processing required for generating the lesion determination result and ensure real-time performance.

[0035] (4-2) Functional Blocks Figure 4 is a functional block diagram of the image processing device 1 related to lesion detection processing. Functionally, the processor 11 of the image processing device 1 has an endoscopic image acquisition unit 30, a Fourier transform unit 31, a selection unit 32, a lesion determination unit 33, an additional processing unit 34, and a display control unit 35. Note that in Figure 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to other functional block diagrams described below.

[0036] The endoscopic image acquisition unit 30 acquires the endoscopic images Ia captured by the endoscope 3 at predetermined intervals via the interface 13. Then, the endoscopic image acquisition unit 30 supplies the acquired endoscopic images Ia to the Fourier transform unit 31 and the display control unit 35, respectively.

[0037] The Fourier transform unit 31 generates k-space data by Fourier transforming the endoscopic image Ia supplied from the endoscopic image acquisition unit 30. Furthermore, the Fourier transform unit 31 may generate, as k-space data, data obtained by converting the complex value of each spatial frequency into an absolute value or a phase, and / or data obtained by logarithmically transforming the value of each spatial frequency, for the data obtained by Fourier transforming the endoscopic image Ia.

[0038] The selector 32 selects, as partial data, data in a partial frequency domain in the k-space in which the k-space data generated by the Fourier transformer 31 exists from the k-space data. The manner of selection by the selector 32 will be described later.

[0039] Based on the partial data generated by the selection unit 32, the lesion determination unit 33 performs a determination regarding the lesion site in the endoscopic image Ia that is the source of the partial data, and supplies information indicating the lesion determination result (also referred to as "lesion determination information") to the additional processing unit 34 and the display control unit 35. In this case, the lesion determination unit 33 inputs the partial data supplied from the selection unit 32 to a lesion determination model configured with reference to model information D1, and generates lesion determination information based on the lesion determination result output by the lesion determination model in response to the input of the partial data.

[0040] The additional processing unit 34 executes processing based on the lesion determination information generated by the lesion determination unit 33. For example, the additional processing unit 34 may execute an automatic diagnosis process based on the lesion determination information to diagnose specific conditions, such as the name of the disease and the severity of the disease, of the lesion site detected by the lesion determination unit 33. The additional processing unit 34 supplies information indicating the processing results based on the lesion determination information (also referred to as "additional processing information") to the display control unit 35.

[0041] The additional processing unit 34 may perform processing based on the lesion determination information, based on a model constructed with reference to parameters pre-stored in the memory 12. In this case, the above-mentioned model may be a model trained to output the above-mentioned diagnosis results, etc., when data including the endoscopic image Ia and the lesion determination information is used as input data. In this way, even when the additional processing unit 34 performs high-load processing using a model, real-time performance can be ensured by reducing the amount of calculation in the lesion determination unit 33.

[0042] The display control unit 35 generates display information Ib based on the latest endoscopic image Ia supplied from the endoscopic image acquisition unit 30, the lesion determination information supplied from the lesion determination unit 33, and the additional processing information supplied from the additional processing unit 34. The display control unit 35 then supplies the generated display information Ib to the display device 2, thereby causing the display device 2 to display the latest endoscopic image Ia, lesion detection results, etc. An example of display on the display device 2 by the display control unit 35 will be described later. Note that, when the display control unit 35 receives lesion determination information indicating that a lesion area has been detected, the display control unit 35 may control the sound output of the sound output unit 16 to output a warning sound, audio guidance, or the like to notify the user that a lesion area has been detected.

[0043] The components of the endoscopic image acquisition unit 30, the Fourier transform unit 31, the selection unit 32, the lesion determination unit 33, the additional processing unit 34, and the display control unit 35 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize the components. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0044] (4-3) Details of the Selector Next, a specific example of processing by the selector 32 will be described. The selector 32 selects partial data from the k-space data, which is data in a partial frequency domain in the k-space where the k-space data exists. In this case, preferably, the selector 32 regards the k-space data as an image, and generates partial data that is asymmetric with respect to at least one of the k-x axis and the ky axis when the center of the image is set as the origin of the k-x axis and the ky axis. The k-space is an example of a "frequency space," and the k-x axis and the ky axis are examples of a "first axis" and a "second axis," respectively.

[0045] 5(A) to 5(E) show specific examples clearly showing selected and non-selected regions in k-space data. In FIGS. 5(A) to 5(E), hatched regions indicate selected regions, and black regions indicate non-selected regions. In FIGS. 5(A) to 5(E), the kx-axis and ky-axis are clearly shown. Furthermore, FIG. 5(A) shows an example in which the upper three-quarters of the range on the ky-axis where k-space data exists is the selected region. FIG. 5(B) shows an example in which the upper half of the range on the ky-axis where k-space data exists is the selected region. FIG. 5(C) shows an example in which the upper quarter of the range on the ky-axis where k-space data exists is the selected region. FIG. 5(D) shows an example in which the central quarter of the range on the ky-axis where k-space data exists is the selected region. FIG. 5(E) shows an example in which the upper and lower half of the range on the ky-axis where k-space data exists are the non-selected regions, and the remaining quarter of the range is the selected region. Here, the selection regions shown in Figures 5(A) to 5(C) and 5(E) are asymmetric with respect to the kx axis, while the selection region shown in Figure 5(D) is symmetric (line symmetric) with respect to the kx axis. Also, all of the selection regions shown in Figures 5(A) to 5(E) are symmetric (line symmetric) with respect to the ky axis.

[0046] Fig. 6 is a graph showing the accuracy rate in an experiment using the partial data shown in Fig. 5(A) to Fig. 5(E) in a colonoscopy examination. In this experiment, k-space data and each of the partial data shown in Fig. 5(A) to Fig. 5(E) were input to a lesion determination model that had been trained to output the degree of inflammation, and the accuracy rate was calculated by comparing the degree of inflammation output by the lesion determination model with the correct degree of inflammation. In Fig. 6, "k-space data" indicates the accuracy rate when the entire k-space data was used as input data to the lesion determination model in both the learning stage and the inference stage, and "partial data (A)" indicates the accuracy rate when the partial data shown in Fig. 5(A) was used as input data to the lesion determination model in both the learning stage and the inference stage. Furthermore, "partial data (B)" indicates the accuracy rate when the partial data shown in Figure 5(B) is used as input data to the lesion determination model in both the learning stage and the inference stage, "partial data (C)" indicates the accuracy rate when the partial data shown in Figure 5(C) is used as input data to the lesion determination model in both the learning stage and the inference stage, "partial data (D)" indicates the accuracy rate when the partial data shown in Figure 5(D) is used as input data to the lesion determination model in both the learning stage and the inference stage, and "partial data (E)" indicates the accuracy rate when the partial data shown in Figure 5(E) is used as input data to the lesion determination model in both the learning stage and the inference stage.

[0047] 6, the accuracy rate is almost the same when using k-space data and when using partial data (A) with a 25% data reduction. Also, when using partial data (E) with a 75% data reduction, the accuracy rate is not significantly different from when using k-space data. Thus, the accuracy rate when using partial data is not significantly different from the accuracy rate when using k-space data.

[0048] Furthermore, the accuracy rate using partial data (A), partial data (B), partial data (C), and partial data (E), which are asymmetric with respect to the kx axis, is superior to the accuracy rate using partial data (D), which is symmetric with respect to the kx axis. In this way, by setting the selected region so that it is asymmetric with respect to at least one of the kx axis or the ky axis, it is possible to reduce the amount of data input to the lesion determination model and the amount of calculation, while suppressing a decrease in the accuracy rate.

[0049] 5(A) to 5(E), the selection unit 32 may generate partial data in which a partial range of the kx axis is set as a non-selected region. In yet another example, the selection unit 32 may generate partial data in which a non-selected region is set for both a partial range of the kx axis and a partial range of the ky axis.

[0050] 7(A) to 7(C) show examples of k-space data clearly showing selected and non-selected regions when non-selected regions are provided in both a partial range of the kx axis and a partial range of the ky axis. The partial data shown in FIG. 7(A) is asymmetric with respect to both the kx axis and the ky axis. The partial data shown in FIG. 7(B) is asymmetric with respect to the kx axis (and axisymmetric with respect to the ky axis), and the partial data shown in FIG. 7(C) is asymmetric with respect to the ky axis (and axisymmetric with respect to the kx axis). In these cases, the selected region is set so as to be asymmetric with respect to at least one of the kx axis or the ky axis. Therefore, even when these are used as partial data, it is possible to reduce the amount of data input to the lesion determination model while suppressing a decrease in the accuracy rate.

[0051] (4-4) Display Example Next, the display control of the display device 2 executed by the display control unit 35 will be described.

[0052] 8 shows an example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 35 of the image processing device 1 transmits display information Ib generated based on information supplied from the endoscopic image acquisition unit 30, the lesion determination unit 33, the additional processing unit 34, etc. to the display device 2, thereby causing the display screen shown in FIG.

[0053] The display control unit 35 of the image processing device 1 displays on the display screen the latest endoscopic image 70 representing a video based on the latest endoscopic image Ia acquired by the endoscopic image acquisition unit 30, a first display field 71 based on lesion determination information, and a second display field 72 based on additional processing information.

[0054] Here, the display control unit 35 displays content based on the lesion determination information in the first display field 71. Here, as an example, based on the lesion determination information indicating the determined degree of inflammation, the display control unit 35 displays in the first display field 71 that inflammation of level 3, out of levels 0 to 3, has occurred.

[0055] Furthermore, based on the additional processing information, the display control unit 35 displays a text statement indicating the possibility of the presence of a predetermined disease (here, "XX") and a score indicating the reliability of the presence of the disease (here, a value range of 0 to 100) in the second display field 72. Furthermore, based on the additional processing information, the display control unit 35 displays a frame 73 surrounding the area suspected of the disease on the latest endoscopic image 70.

[0056] In this way, the display control unit 35 can notify the examiner of lesion determination information and the like in real time.

[0057] (4-5) Processing Flow FIG. 9 is an example of a flowchart showing an outline of the processing executed by the image processing device 1 during endoscopic examination in the first embodiment.

[0058] First, the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13.

[0059] Next, the image processing device 1 converts the endoscopic image Ia acquired in step S11 into k-space data by Fourier transform (step S12). In this case, the Fourier transform unit 31 may generate k-space data by converting the complex values ​​of the data obtained by Fourier transforming the endoscopic image Ia into absolute values ​​or phases, and / or by logarithmically transforming the complex values.

[0060] Then, the image processing device 1 generates partial data that will be part of the k-space data (step S13). In this case, for example, the image processing device 1 sets a non-selected region based on at least one of the kx axis and the ky axis, and generates partial data excluding the frequency domain of the non-selected region.

[0061] Next, the image processing device 1 determines the lesion site in the endoscopic image Ia acquired in step S11 based on the partial data (step S14). The determination in step S14 may be, for example, a determination as to whether or not a lesion site is present in the endoscopic image Ia, or a determination as to the degree of a specific condition (including inflammation, etc.).

[0062] Then, the image processing device 1 displays the endoscopic image Ia acquired in step S11 and the lesion determination result in step S14 on the display device 2 (step S15).

[0063] The image processing device 1 then determines whether the endoscopic examination has ended (step S16). For example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36. If the image processing device 1 determines that the endoscopic examination has ended (step S16; Yes), it ends the processing of the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not ended (step S16; No), it returns the processing to step S11. The image processing device 1 then performs the processing of steps S11 to S15 on the endoscopic image Ia newly generated by the endoscope 3.

[0064] (5) Modifications Next, suitable modifications of the above-described embodiment will be described. The following modifications may be applied in combination to the above-described embodiment.

[0065] (Variation 1) Instead of performing a two-dimensional Fourier transform on the endoscopic image Ia, the Fourier transform unit 31 of the image processing device 1 may perform a one-dimensional Fourier transform on either the x-axis or the y-axis. In this case, the selector 32 sets a non-selected region in a partial range of the axis (kx-axis or ky-axis) on which the Fourier transform was performed, and generates partial data excluding the non-selected region. In this case, the data after the one-dimensional Fourier transform of the endoscopic image Ia is represented by a space having either a pair of axes, the kx-axis and the y-axis, or the x-axis and the ky-axis (hybrid space).

[0066] In this embodiment as well, the image processing device 1 can reduce the amount of data used to input the lesion determination model, and reduce the amount of calculation related to the lesion determination model.

[0067] (Modification 2) The image processing device 1 may process a video image composed of endoscopic images Ia generated during an endoscopic examination after the examination.

[0068] For example, at any timing after an examination, when an image to be processed is designated based on user input via the input unit 14, the image processing device 1 sequentially performs the processing of the flowchart in Fig. 9 on the time-series endoscopic images Ia that make up the designated image. Then, when it is determined in step S16 that the target image has ended, the image processing device 1 ends the processing of the flowchart, but when the target image has not ended, the image processing device 1 returns to step S11 and performs the processing of the flowchart on the next endoscopic image Ia in the time series.

[0069] (Modification 3) The model information D1 may be stored in a storage device separate from the image processing device 1.

[0070] 10 is a schematic configuration diagram of an endoscopic examination system 100A in Modification 3. For simplicity, the display device 2, the endoscope 3, etc. are not shown. The endoscopic examination system 100A includes a server device 4 that stores model information D1. The endoscopic examination system 100A also includes a plurality of image processing devices 1 (1A, 1B, ...) that are capable of data communication with the server device 4 via a network.

[0071] In this case, each image processing device 1 references the model information D1 via the network. In this case, the interface 13 of each image processing device 1 includes a communication interface such as a network adapter for communication. In this configuration, each image processing device 1 can reference the model information D1 and preferably execute processing related to lesion determination, as in the above-described embodiment.

[0072] (Variation 4) The image processing device 1 is not limited to making judgments about lesion sites, but may also make judgments about any desired site of interest that the examiner needs to pay attention to. Such sites of interest may be lesion sites, sites of inflammation, sites of surgical scars or other incisions, sites of folds or protrusions, sites on the wall surface of the lumen that are likely to come into contact with (become intrusive) with the tip 38 of the endoscope 3. In this case, the image processing device 1 uses a model that has been trained in advance to make judgments about the presence or absence of a site of interest, the degree of involvement of the site of interest, and the like, based on the above-described embodiment.

[0073] Furthermore, the image processing device 1 may determine a countermeasure based on, for example, a model generated by machine learning the correspondence between the determination results regarding the area of ​​interest (including determination results indicating the presence or absence of the area of ​​interest and determination results indicating the degree of the area of ​​interest) and countermeasures, and the determination results regarding the area of ​​interest of the subject. The above model is, for example, a model trained by machine learning so as to output an inference result regarding a countermeasure corresponding to the determination results when information regarding the determination results regarding the area of ​​interest is input, and model information including learned parameters is pre-stored in the memory 12 or the like. The "countermeasure" is a method of treatment that a user (examiner, etc.) should perform in accordance with the determination results regarding the area of ​​interest, and includes, for example, an instruction to collect tissue for biopsy.

[0074] Then, the image processing device 1 displays information indicating the determined countermeasure on the display device 2. Note that instead of or in addition to displaying the information on the display device 2, the image processing device 1 may output the information indicating the countermeasure by voice using a voice output device. The "information indicating the countermeasure" is, for example, any information for identifying the countermeasure (such as the name of the countermeasure, an identification number, a description of the content, or a combination thereof).

[0075] The method for determining the countermeasure is not limited to the above-described method. For example, the image processing device 1 may refer to table information indicating the correspondence between candidates for the determination result regarding the target area and the countermeasure corresponding to each candidate, and determine the countermeasure from the determination result regarding the target area. The table information is stored in advance in the memory 12 or the like.

[0076] This can be used to support the user in decision making, etc.

[0077] 11 is a block diagram of an image processing device 1X according to a second embodiment. The image processing device 1X includes an acquisition unit 31X, a selection unit 32X, and a determination unit 33X. The image processing device 1X may be composed of multiple devices.

[0078] The acquisition means 31X acquires data obtained by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in the endoscope. The acquisition means 31X may be the Fourier transform unit 31 in the first embodiment (including modified examples, the same applies below). The above-mentioned "data" may be k-space data. The Fourier transform is not limited to a two-dimensional Fourier transform, but may also be a one-dimensional Fourier transform. The acquisition means 31X may immediately acquire an endoscopic image generated by the imaging unit and acquire the Fourier-transformed data, or may acquire an endoscopic image generated in advance by the imaging unit and stored in a storage device at a predetermined timing, and then perform a Fourier transform to acquire the above-mentioned data.

[0079] The selection means 32X selects partial data that is a part of the data. The selection means 32X can be, for example, the selection unit 32 in the first embodiment.

[0080] The determining unit 33X determines a location of interest in the subject based on the partial data. The determining unit 33X can be, for example, the lesion determining unit 33 in the first embodiment.

[0081] 12 is an example of a flowchart showing a processing procedure in the second embodiment. The acquisition means 31X acquires data by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in the endoscope (step S21). Next, the selection means 32X selects partial data, which is a part of the data (step S22). Then, the determination means 33X determines a region of interest in the subject based on the partial data (step S23).

[0082] According to the second embodiment, the image processing device 1X can accurately detect a region of interest from an endoscopic image of a subject.

[0083] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0084] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0085] [Supplementary Note 1] An image processing device comprising: an acquisition means for acquiring data obtained by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope; a selection means for selecting partial data that is a part of the data; and a determination means for making a determination regarding a site of interest that should be noted in the subject based on the partial data. [Supplementary Note 2] The image processing device according to Supplementary Note 1, wherein the selection means selects the partial data such that the partial data is asymmetric with respect to at least one of the first axis and the second axis in a frequency space representing the data by a first axis and a second axis. [Supplementary Note 3] The image processing device according to Supplementary Note 1, wherein the determination means makes a determination regarding the site of interest based on the partial data and a model into which the partial data is input, and the model is a model that has learned the relationship between the partial data input to the model and a determination result regarding the site of interest in an endoscopic image used to generate the partial data. [Supplementary Note 4] The image processing device of Supplementary Note 1, wherein the acquisition means acquires the data obtained by two-dimensional Fourier transform of the endoscopic image, and the selection means generates the partial data by selecting a partial range on at least one axis along which the Fourier transform was performed. [Supplementary Note 5] The image processing device of Supplementary Note 1, wherein the acquisition means acquires the data obtained by one-dimensional Fourier transform of the endoscopic image, and the selection means generates the partial data by selecting a partial range on the axis along which the Fourier transform was performed. [Supplementary Note 6] The image processing device of Supplementary Note 1, wherein the acquisition means acquires the data by converting complex numbers for each frequency obtained by Fourier transform of the endoscopic image into values ​​representing absolute values ​​or phases. [Supplementary Note 7] The image processing device of Supplementary Note 1, wherein the acquisition means acquires the data by logarithmically transforming values ​​for each frequency obtained by Fourier transform of the endoscopic image. [Supplementary Note 8] The image processing device of Supplementary Note 1, further comprising display control means for displaying information about the result of the determination and the endoscopic image on a display device.[Supplementary Note 9] The image processing device according to Supplementary Note 1, further comprising a countermeasure determination means for determining a countermeasure based on information on the result of the determination and a model to which the information on the result of the determination is input, wherein the model is a model obtained by machine learning the relationship between the information on the result of the determination input to the model and the countermeasures according to the result of the determination. [Supplementary Note 10] An image processing method, in which a computer acquires data obtained by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope, selects partial data that is a part of the data, and makes a determination regarding a site of interest in the subject that should be noted based on the partial data. [Supplementary Note 11] A storage medium storing a program that causes a computer to execute processes of acquiring data obtained by Fourier transforming an endoscopic image of a subject captured by an imaging unit provided in an endoscope, selecting partial data that is a part of the data, and making a determination regarding a site of interest in the subject that should be noted based on the partial data.

[0086] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0087] REFERENCE SIGNS LIST 1, 1A, 1X Image processing device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 14 Input unit 15 Light source unit 16 Sound output unit 100, 100A Endoscopy system

Claims

1. an acquisition means for acquiring data obtained by subjecting an endoscopic image captured by an imaging unit provided in the endoscope to a Fourier transform; A selection means for selecting partial data that is a part of the data; A determination means for determining a site of interest in the subject based on the partial data; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the selection means selects the partial data such that the partial data is asymmetric with respect to at least one of the first axis or the second axis in a frequency space in which the data is represented by a first axis and a second axis.

3. The determination means performs a determination regarding the portion of interest based on the partial data and a model into which the partial data is input; The image processing device according to claim 1 , wherein the model is a model that is machine-learned to determine a relationship between the partial data input to the model and a determination result regarding the area of ​​interest in an endoscopic image used to generate the partial data.

4. the acquiring means acquires the data by performing a two-dimensional Fourier transform on the endoscopic image; The image processing apparatus according to claim 1 , wherein the selection means generates the partial data by selecting a partial range on at least one axis on which the Fourier transform has been performed.

5. the acquiring means acquires the data by performing a one-dimensional Fourier transform on the endoscopic image; The image processing apparatus according to claim 1 , wherein the selection means generates the partial data by selecting a part of a range on the axis on which the Fourier transform has been performed.

6. The image processing apparatus according to claim 1 , wherein the acquisition means acquires the data by converting complex numbers for each frequency obtained by Fourier transforming the endoscopic image into values ​​representing absolute values ​​or phases.

7. The image processing apparatus according to claim 1 , wherein the acquiring means acquires the data by performing a logarithmic transformation on values ​​for each frequency obtained by performing a Fourier transform on the endoscopic image.

8. 2. The image processing apparatus according to claim 1, further comprising a display control means for displaying information on the result of said determination and said endoscopic image on a display device.

9. The method further includes a countermeasure determination means for determining a countermeasure based on information about a result of the judgment and a model to which the information about the result of the judgment is input, The image processing device according to claim 1 , wherein the model is a model that is machine-learned to understand a relationship between information about the result of the determination input to the model and a countermeasure according to the result of the determination.

10. The computer Obtaining data by subjecting an endoscopic image captured by an imaging unit provided in the endoscope to a Fourier transform; Selecting partial data that is a part of the data; determining a location of interest in the subject based on the partial data; Image processing methods.

11. Obtaining data by subjecting an endoscopic image captured by an imaging unit provided in the endoscope to a Fourier transform; Selecting partial data that is a part of the data; A program that causes a computer to execute a process of determining a site of interest that should be noted in the subject based on the partial data.