Image processing device, operation method of image processing device, image processing program, and diagnosis support device

Abstract

To provide an image processing device, an operation method of an image processing device, an image processing program, and a diagnosis support device that allow an objective and accurate index on deglutition to be acquired without requiring much time and effort.SOLUTION: An image processing device acquires an examination moving image of an observation object in a deglutition endoscope examination, imparts stage information indicating at which stage an image is taken of the observation object of a plurality of preset deglutition stages to each of the frame images included in the examination moving image, and calculates a feature amount indicating features on the deglutition of a subject having the observation object on the basis of the stage information imparted to each of the plurality of frame images included in the examination moving image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a method of operating the image processing apparatus, an image processing program, and a diagnostic support apparatus.

Background Art

[0002] Since dysphagia occurs with aging and neurological diseases, the importance of examining swallowing function has been increasing in recent years in an aging society. The examination of swallowing function is performed to identify the pathological conditions of aspiration and to appropriately treat and prevent dysphagia. Video Endoscopic examination of swallowing (VE) or Fiberoptic Endoscopic Evaluation of Swallowing (FEES) has been established as a method for evaluating dysphagia (swallowing function evaluation examination).

[0003] In swallowing endoscopy, various devices for performing evaluations based on the acquired examination images are known (Patent Document 1 or Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In conventional swallowing endoscopy, a doctor observes an image of swallowing movement obtained through an endoscope to perform evaluations, diagnoses, etc. That is, the diagnosis has been mainly made based on the subjective evaluation of the doctor.

[0006] However, relying solely on diagnosis through subjective evaluation by a doctor's visual inspection has problems such as different diagnostic results depending on the doctor's skills, and it is difficult to achieve a higher-precision diagnosis.

[0007] An object of the present invention is to provide an image processing apparatus, an operation method of the image processing apparatus, an image processing program, and a diagnostic support apparatus that can easily obtain an objective and highly accurate index related to swallowing.

Means for Solving the Problems

[0008] The image processing apparatus of the present invention includes a control processor. The control processor acquires an inspection video obtained by photographing an observation target in a swallowing endoscopy examination, and performs recognition processing for each frame image of the inspection video, thereby adding stage information indicating which stage among a plurality of preset swallowing stages the image of the observation target was photographed in to each of the frame images. Based on the stage information added to each of the plurality of frame images included in the inspection video, a feature amount indicating a feature related to the swallowing of the subject having the observation target is calculated.

[0009] The stage information includes a mid-swallowing stage indicating that the observation target is in the process of swallowing and a non-mid-swallowing stage indicating that the observation target is not in the process of swallowing. The mid-swallowing stage preferably includes a mid-swallowing initial stage indicating that the observation target is in the initial stage of swallowing and a mid-swallowing later stage indicating that the observation target is in the later stage of swallowing.

[0010] The control processor detects a swallowing block that is a group of frame images obtained by photographing one swallowing operation by the observation target. The swallowing block preferably includes a plurality of consecutive frame images whose stage information is the mid-swallowing stage.

[0011] The feature amount is preferably the number of swallowing blocks.

[0012] The feature amount is preferably a basic statistic based on the number of frame images included in each of the swallowing blocks.

[0013] The feature amount is preferably the number of frame images in which the stage information is the initial stage during swallowing.

[0014] The feature amount is preferably a basic statistic based on the number of frame images in which the stage information is the initial stage during swallowing among the frame images included in the swallowing block.

[0015] The non-swallowing middle stage preferably includes the immediately after swallowing stage in which the stage information of the immediately preceding frame image is the late stage during swallowing and the open stage other than the immediately after swallowing stage.

[0016] The feature amount is preferably the number of frame images in which the stage information is the immediately after swallowing stage.

[0017] The control processor calculates the area of the halation region in the frame image in which the stage information is the initial stage during swallowing, and the feature amount is preferably the area of the halation region.

[0018] The control processor calculates the brightness value in the frame image in which the stage information is the immediately after swallowing stage, and the feature amount is preferably the brightness value.

[0019] The control processor assigns shooting time information, which is the time when the frame image was shot, to the frame image, and the feature amount is preferably calculated based on the respective brightness values in a plurality of consecutive frame images shot during a specific period among the plurality of time-series frame images included in the swallowing block or among the plurality of time-series frame images not included in the swallowing block.

[0020] The inspection video is preferably one that shoots an observation target when a certain amount of water is swallowed in a swallowing endoscopy examination.

[0021] The inspection video is preferably one that shoots an observation target when a certain amount of swallowed food is swallowed in a swallowing endoscopy examination.

[0022] The control processor preferably calculates different feature amounts according to the types of swallowed foods.

[0023] The control processor preferably accepts the specification of the start point and the end point for calculating the feature amount, and calculates the feature amount for the section defined based on the start point and the end point in the inspection video.

[0024] The control processor preferably performs control to display the feature amount on the display.

[0025] The operation method of the image processing apparatus of the present invention includes: a step of acquiring an inspection video obtained by photographing an observation target in a swallowing endoscopy examination; a step of performing recognition processing for each frame image of the inspection video to assign stage information indicating which stage of a plurality of preset swallowing stages the image of the observation target was photographed in to each frame image; and a step of calculating a feature amount indicating a feature related to the swallowing of the subject having the observation target based on the stage information assigned to each of the plurality of frame images included in the inspection video.

[0026] The image processing program of the present invention causes a computer to implement: a function of acquiring an inspection video obtained by photographing an observation target in a swallowing endoscopy examination; a function of performing recognition processing for each frame image of the inspection video to assign stage information indicating which stage of a plurality of preset swallowing stages the image of the observation target was photographed in to each frame image; and a function of calculating a feature amount indicating a feature related to the swallowing of the subject having the observation target based on the stage information assigned to the plurality of frame images included in the inspection video.

[0027] The diagnostic support apparatus of the present invention includes the image processing apparatus described above, and based on the feature amount calculated by the image processing apparatus, performs a determination regarding the swallowing function of the subject who has undergone a swallowing endoscopy examination, and performs control to display the determination result on the display.

Advantages of the Invention

[0028] According to the present invention, an objective and highly accurate index regarding swallowing can be obtained without trouble.

Brief Description of the Drawings

[0029]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0030] An example of an embodiment of an image processing device and the like of the present invention will be described. First, the background for obtaining the following embodiments will be described. In a swallowing endoscopy examination, a doctor observes an image of a swallowing motion obtained through an endoscope and evaluates its dynamics. However, the diagnosis is mainly based on the subjective evaluation of the doctor, and there has been no method for determining and evaluating a swallowing endoscopy image with objective numerical values or indices.

[0031] For example, heretofore, a system for determining whether it is during swallowing or non-swallowing from an examination image has been disclosed (Patent Document 1 or 2). Patent Document 1 describes automatically determining whether it is during swallowing or non-swallowing from an examination image. Further, Patent Document 2 describes determining whether it is during swallowing or non-swallowing for each frame of a video of a swallowing endoscope. However, generating diagnostic support information using the determination result has not been disclosed.

[0032] In the process of considering, for example, generating diagnostic support information using the determination result obtained by determining whether each frame of the inspection video by the swallowing endoscope is in the swallowing state or not during swallowing, as shown in FIG. 1, on the horizontal axis is the shooting time of each frame image in the time-series frame image, and on the vertical axis is the determination result determined by the doctor based on the frame image, that is, during swallowing (=0, identification tag "0") or not during swallowing (=1, identification tag "1"). When a graph was drawn, FIG. 1(A) is the graph of subject A, and FIG. 1(B) is the graph of subject B. It was found that different characteristics can be seen depending on the swallowing state of the subject. Therefore, when examining the determination of whether it is during swallowing or not during swallowing in more detail, it was found that different characteristics can be shown depending on the swallowing state of the subject.

[0033] Based on the above considerations, the present invention was able to provide an image processing apparatus, an operation method of the image processing apparatus, an image processing program, and a diagnostic support apparatus that can obtain an objective and highly accurate index related to swallowing without much effort.

[0034] The image processing apparatus of the present invention includes a control processor. The control processor acquires an inspection video obtained by photographing an observation target in a swallowing endoscope examination, and by performing recognition processing for each frame image of the inspection video, one of the stage information indicating which stage image among a plurality of preset swallowing stages each frame image is, is given to each of the frame images. Based on the stage information given to each of the plurality of frame images included in the inspection video, a feature amount indicating a feature related to the swallowing of the subject who has undergone the swallowing endoscope examination is calculated.

[0035] Dysphagia endoscopy is an examination conducted for the purposes of evaluating functional abnormalities in the pharyngeal stage during swallowing movements, evaluating organic abnormalities, confirming the effectiveness of compensatory methods and rehabilitation techniques, and providing educational guidance to patients, their families, and medical staff (quoted from "Procedures for Dysphagia Endoscopy 2012 Revision (Revised Edition)" by the Medical Examination Committee of the Japanese Society for Swallowing Rehabilitation; Journal of the Japanese Society for Swallowing Rehabilitation 2013;17:87-99). It involves inserting a relatively thin scope through either the left or right nasal cavity of the subject and observing endoscopic images taken of the pharynx, larynx, etc. during the subject's swallowing movements to make diagnoses related to the subject's swallowing, etc.

[0036] Specifically, the image processing device is a processing computer equipped with a control processor. The image processing device may be incorporated into the processor device provided in the endoscope system used for dysphagia endoscopy, or it may be a device separate from the processor device. Also, the image processing device may be placed remotely from the endoscope system by communicably connecting the endoscope system and the image processing device via a network, etc.

[0037] The image processing device acquires an examination video obtained by photographing the observation target with a dysphagia endoscope. That is, the images of the dysphagia endoscopy obtained by the endoscope system are input into the processing computer. The images are a set of frame images continuously acquired in time series, i.e., a so-called video.

[0038] For the examination video, which is a video acquired by the image processing device, still image data may be input sequentially, or video data may be input all at once and stored in the recording device in the processing computer before recognition processing, etc. is performed. The still image data can be treated in the same way as the frame images constituting the video data. Therefore, the frame images include the images for each frame constituting the examination video and the still image data.

[0039] The inspection video captures the observation target at a predetermined frame rate. The frame rate is the number of still images acquired by the image sensor included in the endoscope system per second. The frame rate in the swallowing endoscope can be set in advance. For example, 30 frames / second or 60 frames / second is common, and there are also cases where a setting such as 120 frames / second can be achieved using a faster image sensor. The image for each frame in the inspection video captured at such a frame rate is a frame image. Note that in a more high-performance endoscope system, a higher frame rate may be achieved, and in some cases, the observation target may be captured at a lower frame rate.

[0040] The inspection video includes a plurality of frame images in time series. By referring to the preset frame rate, the shooting time when a specific frame image was captured, or the shooting period when a plurality of specific frame images were captured, etc. can be calculated. For example, the shooting period when this frame image was captured can be calculated based on the number of consecutive frame images. Also, by using the shooting time of the first frame image in the inspection as the start time of the inspection, the shooting time when this frame image was captured can be obtained based on which number the specific frame image was captured.

[0041] The processing computer is equipped with a classifier that is a learning model previously learned by machine learning using deep learning. The classifier performs recognition processing for each frame image of the inspection video. Through the recognition processing by the classifier, stage information indicating which stage image among a plurality of preset swallowing stages is assigned to each of the frame images. Usually, one piece of stage information is assigned to each frame image.

[0042] The classifier is trained to classify or determine, for each frame image, whether the observation target captured is in any of the swallowing stages by inputting the inspection video. The stage information indicating the swallowing stage is set to a predetermined number of stages. Therefore, the classifier is a classifier that outputs a classification of which swallowing stage among the predetermined number of swallowing stages the frame image is an image of the observation target when a frame image with unknown stage information is input.

[0043] The classifier may be a classifier using any architecture as long as it can determine, for each frame image, whether the observation target captured is in any of the swallowing stages. In a classifier for achieving the purpose of classifying an image based on what is shown in the image, a convolutional neural network (CNN: Convolutional Neural Network) learned with teacher data consisting of frame images in which the observation target of each swallowing stage appears and the correct answer of each swallowing stage appearing in this frame image can preferably be used.

[0044] CNN is a type of deep learning and can extract local patterns and features of an image and learn hierarchically. Therefore, in an image classification task of outputting which swallowing stage the observation target was captured in by inputting a frame image with an unknown swallowing stage, it can exhibit high performance. The contents of the high performance in the image classification task of CNN include accuracy in providing accurate classification results, generalization ability to show good performance even for similar data and new data not included in the teacher data, robustness in exhibiting robust performance against various conversions and noises for the image, feature learning in capturing local features and hierarchical structures of the image, parameter efficiency in effectively reducing the number of model parameters, etc.

[0045] Through the recognition process by the classifier, a recognition result is obtained as to which stage among a plurality of preset swallowing stages the observation target shown in the frame image represents. Then, stage information indicating the swallowing stage, which is the recognition result, is assigned to each of the frame images.

[0046] The classification for training the classifier includes at least "during swallowing" and "not during swallowing". Therefore, the plurality of preset swallowing stages include at least "during swallowing" and "not during swallowing".

[0047] The image processing device calculates a feature amount indicating a feature related to the swallowing of the subject who has undergone the swallowing endoscopy examination based on the stage information assigned to each of the plurality of frame images included in the examination video in one swallowing endoscopy examination. When the stage information assigned to the frame image is "during swallowing" and "not during swallowing", the feature amount can be the number of frame images in which the stage information is "during swallowing" in the examination video. For example, by comparing with a subject having a normal swallowing function with the same number of swallows instructed during the examination, it can be objectively grasped that it takes a long time to swallow when the number of frame images having the stage information of "during swallowing" is large, and conversely, it can be objectively grasped that it does not take a long time to swallow when the number of frame images having the stage information of "during swallowing" is equal to or less than that.

[0048] The feature amount is displayed on a display or the like of the endoscope system. Also, since stage information such as "during swallowing" is assigned to each of the frames constituting the examination video, the doctor can make a more precise diagnosis while displaying the feature amount serving as a diagnosis index together with the image of the observation target at a specific stage on the display.

[0049] In addition to displaying the feature amount, the display may display an inspection video with the stage information superimposed thereon, or, according to an instruction from a user such as a doctor, each specific frame included in the inspection video with the stage information superimposed thereon may be arranged and displayed in time series. Each specific frame is extracted from the inspection video, and for example, a frame having specific stage information, a frame photographed at a specific time, or a frame photographed during a period from the specified time to the specified time may be extracted and displayed.

[0050] With the above configuration, the image processing apparatus can classify the swallowing state with an objective index by using the feature amount obtained by quantifying the features related to the swallowing of the subject. Specifically, the feature amount is calculated by inputting the inspection video into the image processing apparatus, and the feature amount is displayed on the display. The stage information used as the basis for calculating the feature amount is obtained by subjecting the frame image to recognition processing by machine learning. Therefore, according to the image processing apparatus, an objective and highly accurate index related to swallowing can be obtained without difficulty.

[0051] Hereinafter, embodiments will be further described with reference to the drawings. As shown in FIG. 2, the endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, an image processing device 15, a display 16, and a user interface 17. The endoscope 12 is an endoscope of a so-called electronic scope used for swallowing endoscopy. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14.

[0052] The endoscope 12 has an insertion portion 12a inserted into the body of the observation target, an operation portion 12b provided at the proximal end portion of the insertion portion 12a, and a bending portion 12c and a distal end portion 12d provided at the distal end side of the insertion portion 12a. The bending portion 12c bends by operating the angle knob 12e of the operation portion 12b. The distal end portion 12d is directed in a desired direction by the bending operation of the bending portion 12c. In swallowing endoscopy, the insertion portion 12a is inserted from either the left or right nasal cavity of the subject.

[0053] Inside the endoscope 12, an imaging optical system for forming an image of a subject and an illumination optical system for irradiating the subject with illumination light are provided. The subject to be observed is a structure within a living body related to the swallowing motion, specifically, the pharynx and the larynx. The illumination light passes through the insertion portion 12a of the endoscope 12 via a light guide and is emitted from the tip portion 12d toward the subject through the illumination lens of the illumination optical system. When the light source unit 20 is built into the tip portion 12d of the endoscope, it is emitted toward the subject through the illumination lens of the illumination optical system without passing through the light guide.

[0054] The imaging optical system has an objective lens and an imaging sensor. The light from the observation target due to the irradiation of the illumination light is incident on the imaging sensor via the objective lens and the zoom lens. Thereby, an image of the observation target is formed on the imaging sensor. The zoom lens is a lens for magnifying the observation target and moves between the tele end and the wide end by operating the zoom operation unit 12i. The imaging sensor may be disposed at the tip portion 12d of the endoscope 12, or may be a so-called fiber scope using a fiber bundle in the insertion portion of the endoscope 12 and may be at the operation unit side end of the insertion portion 12a.

[0055] The imaging sensor is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD sensor, or the like. An inspection image is generated based on the image signal detected by the imaging sensor. The imaging sensor may include a color imaging sensor provided with a color filter (such as a Bayer filter) for converting the sensed light into a color image signal, as well as a monochrome imaging sensor not provided with a color filter for converting the sensed light into a monochrome image signal. Note that the color imaging sensor may convert the sensed light into a CMY signal instead of an RBG signal.

[0056] When acquiring a color image, the image signal includes a B image signal output from the B pixels of the imaging sensor, a G image signal output from the G pixels, and an R image signal output from the R pixels. The image signal is input to the image acquisition unit 23 of the processor device 14 and acquired as an inspection image that is a monochrome image or a color image. The inspection image acquired by the image acquisition unit 23 is output to the image acquisition unit 25 of the image processing device 15. The inspection image output to the image acquisition unit 25 is output to the recognition processing unit 26. The inspection image is a still image captured during a swallowing endoscopy examination or an inspection video captured during an endoscopy examination.

[0057] In addition to the angle knob 12e, the operation unit 12b is provided with a still image acquisition instruction switch 12h used for acquiring an instruction to acquire a still image of the observation target and a zoom operation unit 12i used for operating the zoom lens.

[0058] The light source device 13 generates illumination light. The processor device 14 performs system control of the endoscope system 10 and image processing and the like for displaying an image on the display 16 with respect to the image signal output from the endoscope 12. The display 16 is a display unit that displays the image captured by the endoscope 12. The user interface 17 is an input device that performs setting input and the like to the processor device 14 and the like. Note that the display 16 and the user interface 17 also perform output of an image and the like or input of setting input and the like from the image processing device 15. When a touch panel is provided in the display 16, the touch panel is included in the user interface 17.

[0059] The light source device 13 includes a light source unit 20 that emits illumination light, and a light source control unit 21 that controls the operation of the light source unit 20. The light source unit 20 emits illumination light for illuminating a subject. The light source unit 20 includes, for example, a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp. The light source may include a plurality of light sources having different wavelengths. The light source control unit 21 controls the lighting or extinguishing of each light source constituting the light source unit 20, and the light emission amount and the like. Thereby, illumination light having a specific light amount or wavelength can be emitted. Usually, white illumination light that allows the observation target to be observed in natural colors is emitted.

[0060] Note that the light source unit 20 may be built into the endoscope 12. Also, the light source control unit 21 may be built into the endoscope 12, or may be built into the processor device 14. The white illumination light includes a so-called pseudo-white in which purple light V, blue light B, green light G, or red light R, which is substantially equivalent to white in imaging a subject using the endoscope 12, is mixed. A light source that irradiates ultraviolet light or infrared light for the purpose of special light observation may further be included. Also, the light source unit 20 includes an optical filter or the like that adjusts the wavelength band, spectrum, light amount, or the like of the illumination light as necessary.

[0061] In the light source unit 20, the wavelengths for high-speed irradiation, for example, blue light B, green light G, and red light R, may be sequentially switched for irradiation, and in the image sensor, images for each color of the illumination light may be acquired by a monochrome sensor or a color sensor, and these may be synthesized in the processor to generate a white image. As a mechanism for switching the wavelengths for high-speed irradiation in the light source unit 20, there are a method of mechanically switching a plurality of color filters of different colors with respect to a white light source such as a xenon lamp, and a method of electronically switching the ON / OFF of a plurality of LEDs (Light Emitting Diodes) that emit different colors.

[0062] The processor device 14 includes a control unit 22a, an image acquisition unit 23, and a display control unit 24. The processor device 14 is a computer and includes a CPU (Central Processing Unit), a memory, and the like. The CPU included in the computer is an example of a processor. In the processor device 14, the program in the program memory operates by the control unit 22a constituted by the processor, thereby realizing functions such as the control unit 22a, the image acquisition unit 23, and the display control unit 24.

[0063] The image processing device 15 includes a control unit 22b, an image acquisition unit 25, a recognition processing unit 26, a feature amount calculation unit 27, a storage unit 28, and a display control unit 29. The image processing device 15 is a computer and includes a CPU, a memory, and the like. The CPU included in the computer is an example of a control processor. In the image processing device 15, the program in the memory included in the computer operates by the control unit 22b constituted by the control processor, thereby realizing the functions of the control unit 22b, the image acquisition unit 25, the recognition processing unit 26, the feature amount calculation unit 27, and the storage unit 28. Note that the image processing device 15 and / or the light source control unit 21 may be included in the processor device 14.

[0064] As described above, the image acquisition unit 25 acquires an inspection video obtained by photographing an observation target in a swallowing endoscopy examination using the endoscope system 10. The inspection video may be acquired in real time, or the inspection video obtained in a previously performed inspection may be stored in the storage unit 28 and then the inspection video may be acquired from the storage unit 28 later.

[0065] The recognition processing unit 26 performs recognition processing for assigning stage information to each frame image of the inspection video. In the recognition processing, based on the image captured in each frame image, it is recognized which of a plurality of preset swallowing stages the image of the observation target was captured in, and the stage information as the recognition result is assigned to each frame image.

[0066] For the method of recognition processing, known technologies can be used, and it is preferable to use the classifier as described above. For example, a classifier using technologies such as semantic segmentation may be used. Also, not limited to supervised learning, as long as it can appropriately assign stage information to the frame image by image recognition processing, an architecture based on unsupervised learning may be used. For example, clustering, Generative Adversarial Networks (GANs), etc. may be used. Thus, in order to recognize the frame image and assign appropriate stage information, technologies used in machine learning such as the selection of the architecture or learning method of the learning model, using multiple CNNs, branching, etc. may be appropriately used.

[0067] As one of the technologies used in machine learning, a cascade classifier, which is a method of improving the detection efficiency and speed by applying multiple classifiers step by step, may be used. In deep learning, first, perform learning to detect the initial stage of swallowing (see Figure 6, whiteout) in the frame image, then perform learning to detect the later stage of swallowing, perform learning to detect the frame image when swallowing water, swallowing food, and other specific swallowing foods, and perform learning to detect the frame image at the stage immediately before and immediately after swallowing (see Figure 5, blackout), and perform step-by-step learning, recognition processing, etc. By performing the recognition processing of the frame image using the cascade classifier, effects such as high-speed detection, high detection accuracy, and resource savings can be achieved.

[0068] As one of the technologies used in machine learning, time-series filtering may be applied. Time-series filtering is a process of extracting or transforming specific patterns or trends in time-series data using a series of methods and algorithms in machine learning, and is used for noise removal, trend analysis, detection of outliers, etc. of time-series data. Since the frame image is time-series data, time-series filtering can be preferably used.

[0069] Note that the techniques used in such machine learning may be used in combination. When a cascade classifier and time-series filtering are used in combination, it is preferable because recognition processing can be performed quickly and with high accuracy.

[0070] Here, the swallowing stage will be described. Swallowing refers to a series of actions of putting food or drink into the mouth, chewing, swallowing, and sending it into the esophagus. As shown in FIG. 3, when the head 29 of a forward-facing person is viewed from the left of the person to represent the oral cavity, pharynx, etc., the swallowing movement includes an "oral stage" (FIG. 3(A)) in which the food F is transported from the oral cavity to the pharynx mainly by the movement of the tongue To, a "pharyngeal stage" (FIG. 3(B)) in which the food F is transported from the pharynx to the esophagus Es by a swallowing reflex, and an "esophageal stage" (FIG. 3(C)) in which the food F is transported from the esophagus Es to the stomach by peristaltic movement of the esophagus. During swallowing, in order to direct the food F toward the esophagus Es and prevent it from flowing into the trachea Tr, the epiglottis Eg, which plays a role of covering the trachea Tr, closes the entrance (glottis) of the trachea Tr by a reflex movement. Also, the soft palate Sp, which is the ceiling of the oral cavity, moves backward to close the passage between the oral cavity and the nasal cavity, preventing the food F from entering the nasal cavity.

[0071] The examination video obtained in the swallowing endoscopy is imaged by inserting the insertion portion 12a of the endoscope 12 from the nasal cavity into the pharynx so that the distal end portion 12d of the endoscope comes near the position R in the middle pharyngeal region as shown in FIG. 4(A). The frame images included in the examination video at this time preferably include anatomical structures such as the epiglottis Eg, the glottis fissure Rg, and the left and right pyriform sinuses Ps as shown in the frame image 31 of FIG. 4(B). The glottis fissure Rg is the space between the left and right folds that make up the vocal cords. The following describes the case where the distal end of the endoscope is placed in this middle pharyngeal region, but in addition to this, as long as the swallowing stage can be grasped, it may be placed in other parts, for example, the nasopharyngeal cavity, the upper pharynx, the lower pharynx, or the larynx to perform the determination of swallowing.

[0072] Since the inspection images taken during a swallowing endoscopy capture subjects with different shapes, locations, etc. depending on the stage of swallowing, the images have different features on different images. The plurality of preset swallowing stages or stage information based on the inspection images may be any information indicating the stage of the swallowing motion, and can be arbitrarily set according to the purpose of what kind of feature quantity is to be obtained, etc.

[0073] The stage information is information that shows the stages of a series of swallowing motions. In the swallowing motion, it starts from the mid non-swallowing stage, swallowing begins, passes through the mid swallowing stage, swallowing ends, and then it transitions back to the mid non-swallowing stage again. A series of swallowing motions from the start to the end of swallowing is regarded as one swallowing. When a swallowing motion occurs again starting from the mid non-swallowing stage, the series of motions of swallowing starting from the mid non-swallowing stage is repeated again. The inspection video includes frame images when the swallowing motion is performed multiple times.

[0074] As shown in FIG. 5, when the inspection images taken at a standard frame rate of 30 frames per second for the swallowing motion are arranged in chronological order, the inspection images are images with different features based on the swallowing stage. At the mid non-swallowing stage 33, which is the stage when no swallowing is being performed, the inspection image shown on the display 16 includes the frame image 31 and is an image in which the epiglottis Eg is shown. Note that in the figure, to avoid clutter, when there are multiple identical ones, not all of them may be labeled.

[0075] In the middle stage of swallowing, i.e., the middle stage of swallowing 32, first, pharyngeal contraction during swallowing begins, and movements such as the elevation of the epiglottis and the contraction of the pharyngeal mucosa are observed in a short period of time. Then, as the pharyngeal contraction intensifies, the mucosa comes into contact with the tip 12d, and the field of view of the endoscope disappears. Therefore, in the frame image 31, a wide area is covered with white halation or a slightly darker color tone. The white halation area of the frame image 31 is also called whiteout (see Fig. 6, the initial stage during swallowing). Then, the entire screen becomes blurred and is covered with a slightly darker color (see Fig. 6, the later stage during swallowing). In the non-swallowing middle stage 33, in the image immediately after the end of swallowing, the mucosa that was previously near the tip 12d disappears, and an overall dark image of the screen is obtained by the automatic exposure adjustment of the endoscope, etc. (see Fig. 5, the stage 33b immediately after swallowing). Then, the epiglottis Eg appears again.

[0076] As the stages of swallowing distinguished from the inspection images, it is preferable to set them as the middle stage of swallowing 32 and the non-swallowing middle stage 33. Therefore, the stage information can be the middle stage of swallowing 32 indicating that the observation target is in the process of swallowing, and the non-swallowing middle stage 33 indicating that the observation target is not in the process of swallowing.

[0077] The non-swallowing middle stage 33 preferably includes the pre-swallowing stage 33a which is the non-swallowing middle stage immediately before the middle stage of swallowing 32, the post-swallowing stage 33b which is the non-swallowing middle stage immediately after the middle stage of swallowing 32, and the open stage 33c which is the non-swallowing middle stage excluding the pre-swallowing stage 33a and the post-swallowing stage 33b.

[0078] In the stage information, the middle stage of swallowing 32 indicating that the observation target is in the process of swallowing preferably includes the initial stage during swallowing indicating that the observation target is in the initial stage of swallowing and the later stage during swallowing indicating that the observation target is in the later stage of swallowing.

[0079] As shown in FIG. 6, in the initial stage 32a during swallowing, an image called whiteout can be obtained in about two frames immediately after the stage 33a immediately before swallowing, where movement of the pharyngeal region and associated blurring can be seen. Thereafter, an image in which the entire screen is covered with a slightly blurred and dark color tone continues for several frames, and for these images, the late stage 32b during swallowing is assigned as stage information.

[0080] As described above, as a series of stage information for the swallowing motion, in chronological order, the stage information for the opening stage 33c, the stage 33a immediately before swallowing, the initial stage 32a during swallowing, the late stage 32b during swallowing, the stage 33b immediately after swallowing, and the opening stage 33c can be set. The number of frames for each stage varies depending on individual swallowing motions, and in some cases, some of these stage information may not be seen. The number of frames for each stage, unseen stage information, etc. can also be regarded as one of the feature quantities for grasping the swallowing state.

[0081] The feature quantity calculation unit 27 calculates a feature quantity indicating a feature related to the swallowing of the subject who has undergone the swallowing endoscopy based on the stage information assigned to each of the frame images 31. The display control unit 29 performs control to display the feature quantity on the display 16.

[0082] The feature quantity is preferably the number of frame images 31 included in the inspection video in one swallowing endoscopy, and the stage information of which is the stage 33b immediately after swallowing. In the stage 33b immediately after swallowing, since the mucosa that was near the distal end 12d until then disappears, often an image in which nothing is shown on the screen and the entire screen is dark is obtained. The image of this stage 33b immediately after swallowing can be regarded as blackout and used as an image of the stage when the swallowing motion has ended. Therefore, the number of frame images 31 whose stage information is the stage 33b immediately after swallowing can be used as a feature quantity representing the number of swallowing motions.

[0083] Note that the number of frame images 31 in which the stage information is the immediately - after - swallowing stage 33b may be the number of black - out frame images extracted for the frame images 31 during the period from the start to the end of the inspection, or may be the number of black - out frame images 31 extracted during a specific period. The specific period can be arbitrarily set.

[0084] The feature amount may be the brightness value in the frame image 31 in which the stage information is the immediately - after - swallowing stage 33b. In this case, the feature - amount calculation unit 27 calculates the brightness value for the frame image 31 to which the stage information of the black - out in the immediately - after - swallowing stage 33b is assigned. The brightness value may use the B - image signal, G - image signal, or R - image signal of the color image of the frame image 31, may use the luminance value, or may use both of them together. Statistical information such as the average value using the brightness values of the frame images 31 having the stage information of a plurality of immediately - after - swallowing stages 33b may be used as the feature amount, or the brightness value of the frame image 31 having the stage information of a specific immediately - after - swallowing stage 33b may be used as the feature amount.

[0085] By using the brightness value of the frame image 31 in the immediately - after - swallowing stage 33b as the feature amount, it can be used as an index such as the speed at which the pharynx that was contracted at the end of swallowing opens, the contraction pressure of the pharyngeal contraction, and the swallowing pressure. When the pharynx opens quickly, the brightness value tends to be relatively low, and when the opening of the pharynx is slow, the brightness value tends to be high. Also, when the contraction pressure, swallowing pressure, etc. of the pharynx are weak, the brightness value tends to be high, and when the contraction pressure, swallowing pressure, etc. are strong, the brightness value tends to be low.

[0086] Preferably, the feature amount is the number of frame images 31 in which the stage information is the initial stage 32a during swallowing. The initial stage 32a during swallowing becomes a white - out image, and this can be used as an image indicating the start stage of the swallowing movement. Therefore, the number of frame images 31 in which the stage information is the initial stage 32a during swallowing can be used as a feature amount representing the number of swallowing movements.

[0087] The feature amount may be the area of the halation region in the frame image 31 where the stage information is the initial stage 32a during swallowing. In this case, the feature amount calculation unit 27 extracts the frame image 31 where the stage information is the initial stage 32a during swallowing, and calculates the area of the halation region in this frame image 31. Regarding the calculation of the area of the halation region, the B image signal, G image signal, or R image signal of the color image of the frame image 31 may be used, the luminance value may be used, or both of these may be used together. For the whiteout frame image 31 that is the frame image 31 of the initial stage 32a during swallowing, it can be set that each pixel has a pixel value equal to or greater than a preset threshold value in terms of the lightness value or luminance value.

[0088] By using the area of the halation region of the frame image 31 in the initial stage 32a during swallowing as the feature amount, it can be used as an index for the contraction speed of the pharynx at the start of swallowing, the contraction pressure of the pharyngeal contraction, the contraction shape of the pharynx, etc. When the pharynx contracts strongly or quickly, the halation region becomes relatively high, and when the pharyngeal contraction is weak or slow, the halation region tends to become relatively low.

[0089] Note that, similar to the blackout image, in the whiteout image, the number of frame images 31 where the stage information is the stage 33b immediately after swallowing, or the area of the halation region, may be targeted at the whiteout frame images 31 extracted from the frame images 31 during the period from the start to the end of the inspection, or may be targeted at the whiteout frame images 31 extracted during a specific period. The specific period can be set arbitrarily.

[0090] When calculating the feature amount, it is preferable that the specific period for which the feature amount is calculated is one swallowing motion. The feature amount calculation unit 27 detects a swallowing block 34 that is a group of frame images obtained by photographing one swallowing motion of the observation target. The swallowing block 34 preferably includes a plurality of consecutive frame images 31 where the stage information is the middle stage 32 of swallowing (see FIG. 6).

[0091] As shown in FIG. 7, one swallowing block 34 consists of a plurality of consecutive frame images 31 of the middle swallowing stage 32 from the frame image 31 immediately after the stage 33a immediately before swallowing to the frame image 31 immediately before the stage 33b immediately after swallowing when the frame images 31 included in the inspection video are arranged in chronological order. Also, one swallowing block 34 consists of a plurality of frame images 31 in which the initial stage 32a during swallowing and the later stage 32b during swallowing are consecutive among the frame images 31 of the middle swallowing stage 32. With any of these detection methods, the swallowing block 34 can be detected. Therefore, by combining and using a plurality of stage information, it is possible to appropriately set and count one swallowing operation according to various swallowing states such as when the frame image 31 of the stage 33b immediately after swallowing cannot be seen.

[0092] It is preferable to set a feature amount regarding the appearance of the swallowing block 34 itself. Specifically, the feature amount is preferably the number of swallowing blocks 34. The number of swallowing blocks 34 can be considered to be the same as the number of swallows. The number of swallowing blocks 34 may be the number extracted for the frame images 31 during the period from the start to the end of the inspection, or may be the number extracted for the frame images 31 during a specific period. The specific period can be set arbitrarily. The number of swallows during a specific period can be one of the useful pieces of information for diagnosing the swallowing state.

[0093] Also, the feature amount is preferably a basic statistic based on the number of frame images 31 included in each of the swallowing blocks 34. When the frame rate does not vary at a predetermined rate, the number of frame images 31 included in each of the swallowing blocks 34 can be the time length of one swallowing operation. Since a plurality of swallowing operations are performed in one swallowing endoscopy examination, a plurality of the numbers of frame images 31 in each swallowing movement are obtained, and by calculating the basic statistic for these numbers, the characteristics of swallowing can be obtained.

[0094] As basic statistics, the average value (mean), which is an indicator representing the central tendency of data, the median, which is an indicator representing the central tendency of data that is less affected by outliers, the mode, which is an indicator representing the central tendency of discrete data, the variance, which is an indicator representing the degree of dispersion of data, the standard deviation, which is used to evaluate the variability and reliability of data in the same way as the variance, the range, which is an indicator representing the difference between the maximum and minimum values of a data set, the maximum value, or the minimum value, etc. can be arbitrarily used according to the purpose of indicating the characteristics of swallowing, etc.

[0095] As shown in Fig. 8(A), for subjects A to G who underwent swallowing endoscopy, regarding the video time (unit, minutes: seconds) from the start to the end of the examination, the total number of frames (unit, pieces), the detected swallowing block 34 within the video time, the detected number of swallows (unit, times), and as basic statistics for the number of frames included in each swallowing block 34, the average of the number of frames included in each swallowing block 34 is represented as swallowing length·average (unit, pieces), the median of the number of frames included in each swallowing block 34 is represented as swallowing length·median (unit, pieces), the variance of the number of frames included in each swallowing block 34 is represented as swallowing length·deviation (unit, pieces), and they are summarized in a table.

[0096] As shown in Table 41, in the swallowing block 34, when paying attention to swallowing length·average, it can be seen that there is a difference of more than twice in its value depending on the subject. Therefore, as shown in Fig. 8(B), for each swallowing block 34 of subject G with the least swallowing length·average, when representing the number of swallows at a predetermined swallowing length (the number of frames included in the swallowing block 34) by the histogram 42, it is clearly shown that there are many swallows with a relatively short swallowing length.

[0097] On the other hand, as shown in Fig. 8(C), for each swallowing block 34 of subject C with the most swallowing length·average, when representing the number of swallows at a predetermined swallowing length (the number of frames included in the swallowing block 34) by the histogram 43, it is clearly shown that the number of frames indicating the swallowing length ranges from few to many, and there are various situations of swallowing length, and there are many swallowing blocks 34, that is, the number of swallows is large.

[0098] Thus, it was found that in the entire inspection video, the number of frames included in one swallowing block 34 varies depending on the swallowing motion of the subject, and also has different distributions for each subject. Therefore, by using basic statistical quantities such as the average value, median value, mode value, variance, standard deviation, range, maximum value, and minimum value based on the number of frame images 31 included in each swallowing block 34 as feature quantities, these feature quantities are considered to be effective as feature quantities representing the characteristics of the subject's swallowing, and thus are preferable as feature quantities.

[0099] Also, it is preferable that the feature quantity is a basic statistical quantity based on the number of frame images 31 in which the stage information is the initial stage 32a during swallowing among the frame images 31 included in the swallowing block 34. The white-out frame images 31 in the initial stage 32a during swallowing are considered to be important among the images obtained in the swallowing endoscopy examination. Therefore, calculating and using these basic statistical quantities as feature quantities is considered to be useful as information for diagnosing the swallowing function.

[0100] As shown in FIG. 9(A), for subjects A to G who underwent swallowing endoscopy examination, regarding the video time (unit: minutes: seconds) from the start to the end of the examination, the total number of frames (unit: pieces), the number of detected swallows (unit: times) for the swallowing block 34 detected within the video time, and the basic statistical quantities for the number of frames in which the stage information included in each swallowing block 34 is the initial stage 32a during swallowing, the total number of frames (unit: pieces) of the number of frames in which the stage information included in each swallowing block 34 is the initial stage 32a during swallowing, and the average (unit: pieces) of the number of frames in which the stage information included in each swallowing block 34 is the initial stage 32a during swallowing are summarized in a table. In the table, the initial stage 32a during swallowing is described as WO (WhiteOut).

[0101] As shown in Table 44, in the swallowing block 34, paying attention to the average number of frames (the "average number of WO frames per swallowing") in which the stage information included in each swallowing block 34 is the initial stage 32a during swallowing, it can be seen that there is a difference of more than six times in this value among the subjects. Therefore, as shown in FIG. 9(B), for each swallowing block 34 of subject B with the least average number of WO frames per swallowing, that is, for one swallowing movement, when the number of frames in which the stage information is the initial stage 32a (whiteout) during swallowing is represented by the histogram 45, it is clearly shown that there are many swallowings with a relatively small number of whiteout frames in one swallowing movement.

[0102] On the other hand, as shown in FIG. 9(C), for one swallowing movement of subject C with the most average number of WO frames per swallowing, when the number of frames in which the stage information is the initial stage 32a during swallowing is represented by the histogram 46, it is clearly shown that swallowings with a relatively small number of whiteout frames, swallowings with a relatively medium number of whiteout frames, and swallowings with a relatively large number of whiteout frames are distributed, and that there are many swallowing blocks 34, that is, many swallowing times.

[0103] Thus, in the entire inspection video, it was found that the number of whiteout frames included in one swallowing block 34 takes different values according to the swallowing movement of the subject and has different distributions for each subject. Therefore, by using basic statistical quantities such as the average value, median, mode, variance, standard deviation, range, maximum value, and minimum value based on the number of whiteout frame images 31 included in each of the swallowing blocks 34 as feature quantities, these feature quantities are considered to be effective as feature quantities representing the characteristics of the subject's swallowing, and thus are preferable as feature quantities.

[0104] Since the feature amounts as described above are calculated as objective values that suitably represent the characteristics of swallowing, regardless of the subjectivity of the doctor or the like performing the examination, they are objective indicators regarding swallowing. Also, since these feature amounts are automatically calculated by the image processing apparatus 15, they can be generated without much effort. Further, these feature amounts can be used as one of the pieces of information for the doctor's diagnosis, and based on these feature amounts, it is possible to determine the classification of the swallowing function and the like.

[0105] Also, it is preferable that the feature amounts are calculated based on the brightness values of each of a plurality of consecutive frame images 31 taken during a specific period among the plurality of time-series frame images 31. Note that the image acquisition unit 23 of the processor device 14 assigns imaging time information, which is the time when the frame image was taken, to the frame image 31. Therefore, the feature amount calculation unit 27 can extract the frame images 31 of a specific period by using the imaging time information starting from the frame image 31 of the swallowing stage such as whiteout or blackout.

[0106] The specific period is preferably a plurality of consecutive frame images 31 taken during a specific period among the plurality of time-series frame images 31 included in the swallowing block 34. The specific period can be the first specific number of frame images 31 of the swallowing block 34, and for example, it is preferably the first three frame images 31 of the swallowing block 34.

[0107] As shown in FIG. 10, it is preferable that the feature amount calculation unit 27 is a basic statistic such as an average value calculated based on the brightness value of each frame image 31 for the image group 54 composed of the frame image 51, the frame image 52, and the frame image 53, which are the first three consecutive frame images 31 among the plurality of time-series frame images 31 included in the swallowing block 34.

[0108] Among the consecutive several frame images 31 of the swallowing block 34, the first several frame images 31 include white-out frame images 31 in which the stage information is the initial stage 32a during swallowing. As described above, by using the brightness value of the frame image 31 in the initial stage 32a during swallowing as a feature quantity, it is possible to use it as an index such as the contraction speed of the pharynx at the start of swallowing, the contraction pressure of the pharynx contraction, the contraction shape of the pharynx, etc. For example, by using the average value of the brightness values of the image group 54 which is the first three frame images 31 of the swallowing block 34 as a feature quantity, it is possible to use it as these indexes in consideration of time.

[0109] Also, it is preferable that the specific period is a plurality of frame images 31 taken during a specific period among a plurality of time-series frame images 31 not included in the swallowing block 34. The specific period can be a specific number of frame images 31 immediately after the swallowing block 34. For example, it is preferable that the first three frames immediately after the swallowing block 34 are used. The first three frames immediately after the swallowing block 34 include black-out frame images 31 in which the swallowing stage is the immediate post-swallowing stage 33b.

[0110] As shown in FIG. 11, it is preferable that the feature quantity calculation unit 27 is a basic statistic such as an average value calculated based on the brightness value of each frame image 31 for the image group 58 composed of the frame image 55, the frame image 56, and the frame image 57 which are the three frame images 31 immediately after the swallowing block 34.

[0111] The several frames immediately after the swallowing block 34 include black-out frame images 31 in which the stage information is the immediate post-swallowing stage 33b. As described above, by using the brightness value of the black-out frame image 31 as a feature quantity, it is possible to use it as an index such as the opening speed of the pharynx at the end of swallowing, the opening pressure of the pharynx opening, the opening shape of the pharynx, etc. For example, by using the average value of the brightness values of the image group 58 which are the three frames immediately after the swallowing block 34 as a feature quantity, it is possible to use it as these indexes in consideration of time.

[0112] As described above, as the feature quantity, it can be set according to what kind of information regarding swallowing is obtained. Summarizing what can be considered as the feature quantity based on the frame image 31 included in the inspection video, the following values can be used as the feature quantity. That is, as the feature quantity, the distribution of a plurality of swallowing blocks 34 themselves, the frequency or the number of frame images 31 in which the swallowing stage is the initial stage 32a during swallowing, the area itself or the ratio of the area of the halation region in the frame image 31 in which the swallowing stage is the initial stage 32a during swallowing, the brightness value of the frame image 31 itself in which the swallowing stage is the initial stage 32a during swallowing, the frequency or the number of frame images 31 in which the swallowing stage is the immediately after swallowing stage 33b, the number of frame images 31 in which the swallowing stage included in each of the plurality of swallowing blocks 34 is the initial stage 32a during swallowing, etc. can be mentioned.

[0113] Also, the brightness value in the above-described feature quantity may be replaced with the blur amount. That is, the image acquisition unit 23 of the processor device 14 calculates the blur amount for a specific plurality of frame images 31. Therefore, it is preferable that the feature quantity is calculated based on the blur amount in the plurality of frame images 31 taken during a specific period among the plurality of time-series frame images 31. As the method for calculating the blur amount, a method for calculating the blur amount of a subject shown in a conventional image can be used. The blur amount in the plurality of frame images 31 is considered to indicate the movement speed of the observation target which is the subject. The movement speed of the pharyngeal contraction observed in the stage immediately before swallowing shows a correlation with the blur amount. Therefore, the feature quantity using the blur amount is considered to be a value indicating the characteristics of swallowing.

[0114] As a feature quantity representing the speed of the pharyngeal contraction movement observed in the immediate pre-swallowing stage, instead of the amount of blur, the magnitude of the image difference or the amount of movement of the feature points may be used. Alternatively, the number of consecutive frames in which the amount of movement of the feature points observed in the immediate pre-swallowing stage becomes a certain threshold value may be used as the feature quantity. In the initial stage of swallowing, it transitions from the non-swallowing middle stage with a small amount of movement of the feature points, through the pharyngeal contraction stage with an increasing amount of movement of the feature points, to the whiteout with a small amount of movement of the feature points. The number of frames in the pharyngeal contraction stage with a large amount of movement of the feature points is considered as a feature quantity indicating the speed of pharyngeal contraction.

[0115] Similar to the brightness value, for the amount of blur, not only the amount of blur itself may be used as a feature quantity, but also basic statistical quantities using the amount of blur may be used as feature quantities. For example, the maximum amount of blur in the frame images of a specific period may be used as a feature quantity. In this case, similar to the brightness value, the amount of blur in the first three consecutive frame images included in the swallowing block 34, and the amount of blur in the three consecutive frame images immediately after the swallowing block 34, in this way, basic statistical quantities can also be used as feature quantities for the amount of blur. Note that a feature quantity combining both the brightness value and the amount of blur may be calculated and used as a new feature quantity.

[0116] Also, the inspection video can be one that captures the observation target during dry swallowing, one that captures the observation target when drinking a certain amount of water, one that captures the observation target when swallowing a certain amount of swallowing food, etc. in the swallowing endoscopy examination. Dry swallowing can be the swallowing when swallowing saliva. In the swallowing endoscopy examination, it is performed to capture the observation target when swallowing saliva, water, various swallowing foods, etc. In the calculation of the feature quantity in the image processing device 15, the feature quantity related to the swallowing operation in these various situations can also be obtained.

[0117] Examples of swallowing foods include, in addition to water for observing the state of swallowing and the mucosa, those added with dyes, etc. for more clearly visualizing the swallowing path and the above-mentioned parts, jelly-like foods for confirming the state of swallowing, foods with various hardnesses, foods in lumps with various sizes, etc.

[0118] Regarding the inspection video that captures the observation target when drinking water or various types of swallow foods, the recognition processing unit 26 may detect the water or various types of swallow foods shown in the frame image 31 included in the inspection video. Even after the swallowing operation is completed, it may detect how much, in which area, and for how long the water or swallow foods remain in the frame image 31, and use the calculated values regarding these as feature amounts.

[0119] Note that the feature amount calculation unit 27 may calculate different feature amounts according to the types of swallow foods to be swallowed. For example, for the inspection video when drinking water, it may calculate the feature amount for the entire inspection video, and for the inspection video when swallowing swallow foods, it may calculate the feature amount in the swallowing block 34, etc.

[0120] Also, the various feature amounts as described above may be used alone, or may be used as feature amounts calculated by combining a plurality of them. Also, calculate a plurality of feature amounts in parallel, and

[0121] The storage unit 28 stores the inspection videos used by the image processing device 15, or stores the calculated feature amounts, etc. The recognition processing unit 26, the feature amount calculation unit 27, etc. can use the data stored in the storage unit 28.

[0122] The display control unit 29 performs control to display the calculated feature amounts on the display 16, control to superimpose and display the calculated feature amounts on the frame image 31, and display control for performing various settings such as the processing and functions of the image processing device 15. The display control unit 29 controls the mode of displaying the feature amounts on the display 16 according to the user's settings, etc. from the user interface 17.

[0123] The feature quantity is used for the determination regarding the swallowing function of the subject. As shown in FIG. 12, the endoscope system 10 may include a diagnostic support device 61. The diagnostic support device 61 includes an image processing device 15 and a determination unit 62. Based on the feature quantity calculated by the determination unit 62 and the image processing device 15, a determination regarding the swallowing function of the subject who has undergone a swallowing endoscopy examination is made. The display control unit 24 also performs display control in the diagnostic support device 61. The display control unit 24 performs control to display the result of the determination on the display 16.

[0124] The determination regarding the swallowing function can be set in advance according to the purpose, such as the determination of swallowing characteristics and the determination of swallowing function. As the determination of swallowing characteristics, the determination of swallowing type can be mentioned. For the determination of swallowing type, the distribution of the feature quantity such as the distribution of the swallowing block 34 can be used. As the swallowing type, it can be classified into a type with a fast swallowing movement in both water and swallowing food, a type with a slow swallowing movement in both water and swallowing food, etc.

[0125] For the determination of swallowing function, the feature quantity using the inspection video when drinking water or various swallowing foods can be used. As the determination of swallowing function, based on the above feature quantity, it can be classified into good swallowing function where the swallowing movement is considered to be appropriately performed according to the swallowing food, etc., possible swallowing function where the swallowing movement is not considered to be appropriately performed due to the size, hardness, etc. of the swallowing food, impossible swallowing function where the possibility of aspiration is high due to the size, hardness, etc. of the swallowing food, etc.

[0126] Also, the determination of swallowing function may be performed according to the swallowing food. Calculate the necessary feature quantity using the inspection videos when swallowing three types of swallowing foods, namely swallowing food A, swallowing food B, and swallowing food C, with different hardnesses. Note that swallowing food A is the softest, swallowing food B has medium softness, and swallowing food C is the hardest swallowing food. Based on the determination of swallowing function when swallowing food B as a reference, when swallowing food A, the determination of swallowing function based on the feature quantity can be adjusted in a strict direction, and when swallowing food C, the determination of swallowing function based on the feature quantity can be adjusted in a lenient direction.

[0127] Further, the determination unit 62 may determine whether to perform the determination of the swallowing function according to the type of the swallowed food. In this case, the recognition processing unit 26 recognizes the type of the swallowed food based on the frame image 31 included in the inspection video. This recognition can be performed automatically. When it is recognized that the swallowed food is the above-described swallowed food A, the determination by the determination unit 62 may not be performed, and when it is recognized that the swallowed food is the swallowed food C, the determination by the determination unit 62 may be performed.

[0128] By performing the determination of the swallowing function according to the swallowed food, or automatically recognizing whether to perform the determination of the swallowing function according to the type of the swallowed food, and performing the determination of the swallowing function in the case of a preset specific swallowed food, a doctor or the like can obtain a certain degree of determination result in the swallowing endoscopy examination without much effort. These determination results are useful in various scenarios. For example, the efficiency of diagnosis can be improved, such as by examining the inspection video for a specific period without examining the entire inspection video. Also, based on the determination result automatically displayed during the swallowing endoscopy examination, the type of the swallowed food can be determined and the examination can be continued, and the examination content in one swallowing endoscopy examination can be performed more appropriately. Therefore, according to the diagnostic support device 61, useful information for supporting the diagnosis can be obtained without much effort.

[0129] Note that as the inspection video, an inspection video that captures from the start to the end of the inspection is used, but the user may specify the start point and the end point for calculating the feature amount. The feature amount may be calculated for a section defined based on the start point and the end point in the inspection video.

[0130] In one swallowing endoscopy examination, when it is desired to perform the processing by the image processing device 15 for a plurality of parts in time series, a plurality of parts may be set. In this case, the inspection video includes a pair of a plurality of start points and end points. The inspection video without the swallowed food may be set by the start point 1 and the end point 1, the inspection video when drinking water may be set by the start point 2 and the end point 2, and the inspection video when swallowing the swallowed food may be set by the start point 3 and the end point 3.

[0131] Examples of the specified method include a method using various switches of the endoscope 12, a method of specifying via the user interface 17, a method of specifying by voice recognition, etc. When using the switch of the endoscope 12, for example, the still image acquisition instruction switch 12h may be used after setting it as a trigger for specifying the start point and the end point. During the examination, when the end point is specified and the start point is not specified, the time when the still image acquisition instruction switch 12h is clicked may be set as the start point, and the time when it is clicked again may be set as the examination end time.

[0132] As another method of specifying, via the user interface 17, the recognition processing unit 26 may receive the specification of the start point and the end point, and the feature amount calculation unit 27 may calculate the feature amount for the examination video that starts from the start point and ends at the first end point after this start point. The frame image 31 included in the examination video is provided with information on the shooting time. Therefore, while displaying the examination video on the display 16, the user can also select the part for which the processing by the image processing apparatus 15 is to be performed by specifying the start point and the end point of the swallowing endoscopy examination.

[0133] Also, the start point and the end point may be automatically set. The recognition processing unit 26 may use, as the start point, the frame image 31 for which a preset recognition result is obtained by recognizing the frame image 31, and similarly, use, as the end point, up to the frame image 31 for which a preset recognition result is obtained. As the preset recognition result, for example, based on the stage information, the start of the swallowing block 34 may be used as the start point, and similarly, the frame image 31 of the whiteout, which is the stage 33b immediately after swallowing, may be used as the end point. Also, for the continuous time-series frame images 31 when the recognition processing unit 26 recognizes a specific swallowed food, the time when the frame image 31 with the earliest shooting time is obtained may be used as the start point, and the time when the frame image with the latest shooting time is obtained may be used as the end point.

[0134] In calculating the feature amount, for the inspection video of each subject, pharyngeal blocks, frame images of specific pharyngeal stages or periods, etc. are extracted. However, it is also possible to analyze a plurality of inspection videos obtained at different times for the same subject. When calculating the feature amount for inspection videos respectively obtained at a plurality of times before and after swallowing function treatment for the same subject, and performing analysis, diagnosis, etc. of the swallowing function based on the feature amount, it is preferable because it can be used as an objective index indicating changes in swallowing characteristics, the effect of swallowing function treatment, the degree of progression of swallowing function decline, etc.

[0135] Next, an example of the flow of swallowing endoscopy examination and diagnosis, and the flow of processing in the image processing apparatus 15 and the diagnosis support apparatus 61 will be described using a flowchart. As shown in FIG. 13, in the swallowing endoscopy examination, the insertion portion 12a of the endoscope 12 is inserted into the subject, and video shooting is started (step ST100). The distal end portion 12d of the endoscope is arranged at an appropriate position, and four examinations of dry swallowing, storing water in the mouth and swallowing it, putting water into the pharynx and swallowing it, and eating swallowing food are sequentially performed.

[0136] In each examination, at the start and end of the examination, by clicking the still image acquisition instruction switch 12h of the endoscope 12, the start point and end point of feature point calculation are indicated. The still image acquisition instruction switch 12h has a function like an alternate switch that repeatedly performs the instruction of the start point of feature point calculation and the instruction of the end point every time it is clicked. By first clicking the still image acquisition instruction switch 12h of the endoscope 12, a mark serving as a mark of the start point of feature amount calculation is displayed on the display 16 together with the inspection image. Next, by clicking the still image acquisition instruction switch 12h, the mark displayed on the display 16 disappears. Thereby, a user such as a doctor can set the period for calculating the feature amount and recognize this period.

[0137] By clicking the still image acquisition instruction switch 12h, the start point of the feature amount is set (step ST110), and a signal is given to the subject to perform multiple dry swallows (step ST120). After that, after setting the end point of the feature amount, the start point of the feature amount is set again (step ST130). A signal is given to the subject to take in a certain amount (for example, 20 cc) of water in the mouth and swallow it all at once (step ST140). After that, after setting the end point of the feature amount, the start point of the feature amount is set again (step ST150).

[0138] A signal is given to the subject to put a certain amount (for example, 20 cc) of water up to the pharynx and swallow it (step ST160). After that, after setting the end point of the feature amount, the start point of the feature amount is set again (step ST170). An instruction is given to the subject to ingest a specific swallowing food and observed (step ST180). After the observation is completed, the shooting is terminated, and the obtained examination video is analyzed (calculation and determination of feature amounts) by the image processing device 15 and the diagnostic support device 61 (step ST190). The determination result is displayed on the display 16 (step ST200). The doctor makes a diagnosis such as optimizing the rehabilitation program of the subject based on the determination result obtained by the diagnostic support device 61 and displayed on the display 16 (step ST210).

[0139] As shown in FIG. 14, as the processing by the image processing device 15 and the diagnostic support device 61 (step ST190), first, the image acquisition unit 25 acquires an examination video obtained in the swallowing endoscopy examination, and the examination video to which the start point and the end point of the feature amount are added (step ST300). The recognition processing unit 26 performs recognition processing for each frame image in the examination video of the section for calculating the feature amount based on the start point and the end point of the feature amount, and assigns stage information (step ST310).

[0140] The feature quantity calculation unit 27 calculates a specified feature quantity in advance for the frame image with stage information added (step ST320). The determination unit 62 makes a determination regarding the swallowing function based on the calculated feature quantity (step ST330). Thereafter, it is the same as after step ST200. The display control unit 24 displays the result of the determination on the display 16, and the doctor makes a diagnosis such as optimizing the rehabilitation program for the subject based on the determination result displayed on the display 16.

[0141] Regarding the image processing device 15 or the diagnostic support device 61, the processor device 14 and the image processing device 15 or the diagnostic support device 61 may be configured by separate computers, or the processor device 14 may be configured to also exhibit the functions of the image processing device 15 and the diagnostic support device 61. Further, the image processing device 15 or the diagnostic support device 61 may be arranged at a position close to the processor device 14, etc., or by communicating via a network, the image processing device 15 or the diagnostic support device 61 may be arranged at a location remote from the processor device 14.

[0142] In the present embodiment, the hardware structures (processors for the processor device, control processors, etc.) of the processing units (processing unit) that execute various processes such as the control unit 22a, the control unit 22b, the image acquisition unit 23, the display control unit 24, the image acquisition unit 25, the recognition processing unit 26, the feature quantity calculation unit 27, and the determination unit 62 are various processors as shown below. The various processors include a CPU, which is a general-purpose processor that executes software (program) and functions as various processing units, a GPU (Graphics Processing Unit) that performs high-speed image processing, a programmable logic device (Programmable Logic Device: PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0143] One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). Also, a plurality of processing units may be composed of one processor. As an example of configuring a plurality of processing units with one processor, firstly, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Secondly, as represented by a System On Chip (SoC), etc., there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. Thus, various processing units are configured using one or more of the above various processors as a hardware structure.

[0144] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit (circuitry) in the form of a combination of circuit elements such as semiconductor elements. Also, the hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

Explanation of Reference Numerals

[0145] 10 Endoscope system 12 Endoscope 12a Insertion portion 12b Operation portion 12c Bending portion 12d Tip portion 12e Angle knob 12h Still image acquisition instruction switch 12i Zoom operation portion 13 Light source device 14 Processor device 15 Image processing device 16 Display 17 User interface 20 Light source portion 21 Light source control unit 22a, 22b Control units 23, 25 Image acquisition units 24 Display control unit 26 Recognition processing unit 27 Feature quantity calculation unit 28 Memory unit 29 Head 31 Frame image 32 Middle stage of swallowing 32a Initial stage of middle stage of swallowing 32b Later stage of middle stage of swallowing 33 Non - swallowing middle stage 33a Stage immediately before swallowing 33b Stage immediately after swallowing 33c Opening stage 34 Swallowing block 41, 44 Tables 42, 43, 45, 46 Histograms 51, 52, 53, 55, 56, 57 Frame images 54, 58 Image groups 61 Diagnostic support device 62 Judgment unit Es Esophagus Eg Epiglottis F Food Rg Glottis fissure Ps Piriform sinus Sp Soft palate To Tongue Tr Trachea ST100 - 210 Steps ST300 - 330 Steps

Claims

1. An image processing apparatus comprising a control processor, wherein the control processor acquires an inspection video obtained by photographing an observation target during a swallowing endoscopy examination, performs recognition processing on each frame image of the inspection video, and thereby assigns stage information indicating an image of the observation target taken at any one of a plurality of preset swallowing stages to each of the frame images, and calculates a feature amount indicating a feature related to the swallowing of a subject having the observation target based on the stage information assigned to each of the plurality of frame images included in the inspection video.

2. The stage information includes a mid-swallowing stage indicating that the observation target is in the process of swallowing and a non-mid-swallowing stage indicating that the observation target is not in the process of swallowing, and the mid-swallowing stage includes an early mid-swallowing stage indicating that the observation target is in an initial stage of swallowing and a late mid-swallowing stage indicating that the observation target is in a later stage of swallowing. The image processing apparatus according to claim 1.

3. The control processor detects a swallowing block, which is a group of frame images obtained by photographing one swallowing motion of the observation target, and the swallowing block includes a plurality of consecutive frame images in which the stage information is the mid-swallowing stage. The image processing apparatus according to claim 2.

4. The feature amount is the number of the swallowing blocks. The image processing apparatus according to claim 3.

5. The feature amount is a basic statistic based on the number of frame images included in each of the swallowing blocks. The image processing apparatus according to claim 3.

6. The feature amount is the number of frame images in which the stage information is the early mid-swallowing stage. The image processing apparatus according to claim 2.

7. The feature amount is a basic statistic based on the number of frame images in which the stage information is the early mid-swallowing stage among the frame images included in the swallowing block. The image processing apparatus according to claim 3.

8. The non-mid-swallowing stage includes an immediately-after-swallowing stage in which the stage information of the immediately preceding frame image is the late mid-swallowing stage and an open stage other than the immediately-after-swallowing stage. The image processing apparatus according to claim 2.

9. The feature amount is the number of frame images in which the stage information is the immediately-after-swallowing stage. The image processing apparatus according to claim 8.

10. The control processor calculates the area of a halation region in the frame image in which the stage information is the early mid-swallowing stage, The image processing apparatus according to claim 2, wherein the feature amount is the area of the halation region.

11. The control processor calculates a brightness value in the frame image in which the stage information is the stage immediately after swallowing, The image processing apparatus according to claim 8, wherein the feature amount is the brightness value.

12. The control processor assigns shooting time information, which is the time when the frame image was shot, to the frame image, The image processing apparatus according to claim 3, wherein the feature amount is calculated based on the respective brightness values in a plurality of consecutive frame images taken during a specific period among the plurality of time-series frame images included in the swallowing block or among the plurality of time-series frame images not included in the swallowing block.

13. The inspection video according to claim 1, wherein the inspection video is a video obtained by shooting the observation target when a certain amount of water is swallowed in the swallowing endoscopy.

14. The inspection video according to claim 1, wherein the inspection video is a video obtained by shooting the observation target when a certain amount of swallowing food is swallowed in the swallowing endoscopy.

15. The image processing apparatus according to claim 14, wherein the control processor calculates different feature amounts according to the types of the swallowing food to be swallowed.

16. The control processor accepts designation of a start point and an end point for calculating the feature amount, The image processing apparatus according to claim 1, wherein the feature amount is calculated for an interval defined based on the start point and the end point in the inspection video.

17. The image processing apparatus according to claim 1, wherein the control processor performs control to display the feature amount on a display.

18. A step of acquiring an inspection video obtained by shooting an observation target in a swallowing endoscopy; A step of performing recognition processing for each frame image of the inspection video to assign stage information indicating an image of the observation target taken at which stage among a plurality of preset swallowing stages to each of the frame images; A method of operating an image processing apparatus, including a step of calculating a feature amount indicating a feature related to swallowing of a subject having the observation target based on the stage information assigned to each of the plurality of frame images included in the inspection video.

19. To a computer, A function of acquiring an inspection video obtained by shooting an observation target in a swallowing endoscopy, By performing recognition processing for each frame image of the inspection video, a function of attaching stage information indicating which of a plurality of preset swallowing stages the image capturing the observation target at each stage is, to each of the frame images is provided; An image processing program for implementing a function of calculating a feature amount indicating a feature related to swallowing of a subject having the observation target based on the stage information attached to the plurality of frame images included in the inspection video.

20. Equipped with the image processing apparatus according to any one of claims 1 to 17, Based on the feature amount calculated by the image processing apparatus, a determination regarding the swallowing function of the subject who has undergone the swallowing endoscopy is made, A diagnostic support apparatus that performs control to display the result of the determination on a display.

Images