Image processing device and operation method thereof

Abstract

To provide an image processing device capable of preventing overlooking during an examination and reducing a burden on a user in a deglutition endoscope examination, and to provide an operation method thereof.SOLUTION: An image acquisition unit 33 acquires an examination image. A first frame classification unit 40 classifies each frame of the examination image into a frame SF during deglutition in which deglutition is performed, or into a frame NSF during non-deglutition in which deglutition is not performed. When a time interval between the frames SF during deglutition temporally adjacent to each other is a first threshold or less, a deglutition block generation unit 41 integrates at least the adjacent frames SF during deglutition as deglutition blocks BL1 and BL2.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for analyzing an image obtained by a swallowing endoscopy examination and an operation method thereof.

Background Art

[0002] Since swallowing disorders occur along with aging and nervous system diseases, the importance of examining swallowing function has been increasing in recent years in an aging society. The examination of swallowing function is desired to identify the pathological conditions of aspiration and to perform appropriate treatment and prevention of swallowing disorders. Therefore, video endoscopic examination of swallowing (VE) has been established as a method for evaluating swallowing disorders (swallowing function evaluation examination) (for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a swallowing endoscopy examination, a doctor, who is a user, observes an image of the swallowing movement obtained through the endoscope and evaluates the movement state. Therefore, since it is necessary to observe a large number of acquired images, there is a possibility of overlooking during the examination. In addition, it is a burden on the user to review a long video after the examination. Prevention of overlooking during such an examination or reduction of the user's burden has been demanded.

[0005] Note that Patent Document 1 discloses a system for determining whether a subject is in the process of swallowing or not for each inspection image. Patent Document 1 also discloses the use of Deep Learning to determine whether a subject is in the process of swallowing or not. However, in the determination based on only one image, there may be an image similar to the swallowing state even in a time period when the subject is actually not swallowing, or there may be an image similar to the non-swallowing state even in a time period when the subject is actually swallowing. Considering such similar images, an accurate swallowing determination has been demanded.

[0006] In addition, Patent Document 2 describes a method for detecting the amount of blur, the magnitude of image difference, and the amount of movement of feature points. However, there may be a value similar to the swallowing state even in a time period when the subject is actually not swallowing, or there may be a value similar to the non-swallowing state even in a time period when the subject is actually swallowing. Considering such similar values, an accurate swallowing determination has been demanded.

[0007] In addition, in Patent Document 1, the frames acquired in a certain period from the inspection image determined to be in the process of swallowing are determined to be in the process of swallowing, and in Patent Document 2, the initial stage (start time) and the final stage (end time) of swallowing in which movement occurs are detected. However, with the methods described in Patent Documents 1 and 2 as such, it has been difficult to detect the entire swallowing motion.

[0008] An object of the present invention is to provide an image processing apparatus and an operating method thereof that can prevent overlooking during an inspection or reduce the burden on a user in a swallowing endoscopy examination.

Means for Solving the Problems

[0009] The image processing apparatus of the present invention includes a control processor, which acquires inspection images and classifies each frame of the inspection images into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which swallowing is not being performed. When the time interval between temporally adjacent swallowing frames is equal to or less than a first threshold, at least the adjacent swallowing frames are integrated as a swallowing block.

[0010] The control processor preferably determines that a swallowing frame having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing frame, or determines that a swallowing block having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block.

[0011] The image processing apparatus of the present invention includes a control processor, which acquires inspection images and classifies each frame of the inspection images into either a plurality of frames for each swallowing operation corresponding to the swallowing operation or a non-swallowing frame in which swallowing is not being performed. When the time interval between temporally adjacent frames for each swallowing operation is equal to or less than a first threshold, at least the adjacent frames for each swallowing operation are integrated as a swallowing block for each operation.

[0012] The control processor preferably determines that a frame for each swallowing operation having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a frame for each swallowing operation, or determines that a swallowing block for each operation having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block for each operation. The control processor preferably determines that the expression pattern is either in the middle of swallowing or not in the middle of swallowing based on the expression pattern including the frame for each swallowing operation or the swallowing block for each operation.

[0013] The frame for each swallowing action preferably includes a frame immediately before swallowing, a frame during swallowing, or a frame immediately after swallowing. The frame immediately before swallowing preferably includes a pre-swallowing breve frame, an epiglottis inversion frame, or a food inflow frame. The frame during swallowing preferably includes a spitting frame, a shielding frame, a flow frame, a colored water inflow frame, or a swallowed food frame. The frame immediately after swallowing preferably includes a post-swallowing breve frame or a blackout frame.

[0014] When the control processor classifies the inspection image into a plurality of brightness information frames according to brightness, the control processor preferably determines that the expression pattern is either during swallowing or not during swallowing based on an expression pattern including a brightness information frame, a frame for each swallowing action, or a swallowing block for each action. The brightness information frame preferably includes a high brightness frame, a medium brightness frame, or a low brightness frame.

[0015] When the control processor classifies a frame in which the blur of the inspection image is equal to or less than the blur threshold as a pre-swallowing breve frame, the control processor preferably determines that the expression pattern is either during swallowing or not during swallowing based on an expression pattern including a pre-swallowing breve frame, a frame for each swallowing action, or a swallowing block for each action. The control processor preferably calculates the value of a weighting function for evaluating whether it is during swallowing or not for each frame, and changes the type of the frame to a frame during swallowing or a frame not during swallowing according to the value of the weighting function.

[0016] The operation method of the image processing apparatus of the present invention includes a step of the control processor acquiring an inspection image, a step of classifying each frame of the inspection image into either a frame during swallowing in which swallowing is being performed or a frame not during swallowing in which swallowing is not being performed, and a step of integrating at least adjacent frames during swallowing as a swallowing block when the time interval between adjacent frames during swallowing is equal to or less than a first threshold.

[0017] It is preferable that the control processor determines that a swallowing frame with a time width less than or equal to a second threshold or greater than or equal to a third threshold is not a swallowing frame, or determines that a swallowing block with a time width less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing block.

Advantages of the Invention

[0018] According to the present invention, in a swallowing endoscopy examination, it is possible to prevent overlooking during the examination or reduce the burden on the user.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

[0020] [First Embodiment] As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, a computer 16, a recording device 17, a display 18, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15. The endoscope 12 has an insertion portion 12a that is inserted into the body of the observation target, an operation portion 12b provided at the proximal end portion of the insertion portion 12a, and a bending portion 12c and a distal end portion 12d provided at the distal end side of the insertion portion 12a. The bending portion 12c bends by operating the angle knob 12e of the operation portion 12b. The distal end portion 12d is directed in a desired direction by the bending operation of the bending portion 12c. The endoscope 12 is an endoscope used for a swallowing endoscopy examination.

[0021] Inside the endoscope 12, an imaging optical system for forming an image of the subject and an illumination optical system for irradiating the subject with illumination light are provided. The subject is a biological structure related to swallowing movement. Specifically, it is the pharynx and larynx. The illumination light passes through the insertion portion 12a of the endoscope 12 via a light guide and is emitted from the distal end portion 12d toward the subject through the illumination lens of the illumination optical system. When the light source unit 20 is built in the distal end portion 12d of the endoscope, it is emitted toward the subject through the illumination lens of the illumination optical system without passing through the light guide.

[0022] The imaging optical system has an objective lens and an imaging sensor. The light from the observation target due to the irradiation of the illumination light enters the imaging sensor through the objective lens and the zoom lens. Thereby, an image of the observation target is formed on the imaging sensor. The zoom lens is a lens for magnifying the observation target, and moves between the tele end and the wide end by operating the zoom operation unit 12i. The imaging sensor may be disposed at the distal end portion 12d of the endoscope, or may be a so-called fiber scope using a fiber bundle in the insertion portion of the endoscope 12 and may be at the side end on the operation portion side of the insertion portion 12a.

[0023] The imaging sensor is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, or the like. An inspection image is generated based on the image signal detected by the imaging sensor.

[0024] The imaging sensor may include a color imaging sensor provided with a color filter (such as a Bayer filter) that converts the detected light into a color image signal, as well as a monochrome imaging sensor not provided with a color filter that converts the detected light into a monochrome image signal. Note that the color imaging sensor may convert the detected light into CMY signals instead of RBG signals.

[0025] When acquiring a color image, the image signal includes a B image signal output from B pixels, a G image signal output from G pixels, and an R image signal output from R pixels. The image signal is output to the image acquisition unit 31 of the processor device 15 and acquired as an inspection image that is a monochrome image or a color image. The inspection image acquired by the image acquisition unit 31 is output to the image acquisition unit 33 of the computer 16. The inspection image output to the image acquisition unit 33 is output to the swallowing determination unit 34. The inspection image is a still image captured during the endoscopic examination or a series of consecutive moving images captured in time series during the endoscopic examination.

[0026] In addition to the angle knob 12e, the operation unit 12b is provided with a still image acquisition instruction switch 12h used for acquiring an instruction for a still image of the observation target and a zoom operation unit 12i used for operating the zoom lens.

[0027] The light source device 14 generates illumination light. The processor device 15 performs system control of the endoscopic system 10 and image processing and the like on the image signal output from the endoscope 12. The display 18 is a display unit that displays the image captured by the endoscope 12. The user interface 19 is an input device that performs setting input and the like to the processor device 15 and the like.

[0028] The light source device 14 includes a light source unit 20 that emits illumination light, and a light source control unit 22 that controls the operation of the light source unit 20. The light source unit 20 emits illumination light for illuminating a subject. The light source unit 20 includes, for example, a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp. The light source control unit 22 controls the lighting or extinguishing of each light source constituting the light source unit 20, and the light emission amount and the like.

[0029] Note that the light source unit 20 may be built into the endoscope 12. Further, the light source control unit 22 may be built into the endoscope 12, or may be built into the processor device 15. White includes a so-called pseudo-white obtained by mixing purple light V, blue light B, green light G, or red light R, which is substantially equivalent to white in imaging a subject using the endoscope 12. A light source that irradiates ultraviolet light or infrared light for the purpose of special light observation may be further included. Further, the light source unit 20 includes an optical filter or the like that adjusts the wavelength band, spectrum, light amount, etc. of the illumination light as necessary. In the light source unit 20, for example, blue light B, green light G, and red light R are sequentially switched and irradiated at high speed (high-speed switching), and in the imaging sensor, images for each color of the illumination light are acquired by a monochrome sensor or a color sensor, and these are synthesized in the processor to generate a white image. As a mechanism for performing high-speed switching of illumination light of a plurality of wavelengths in the light source unit 20, there are a method of mechanically switching a plurality of color filters of different colors with respect to a white light source such as a xenon lamp, and a method of electronically switching the ON / OFF of a plurality of LEDs that emit different colors.

[0030] The processor device 15 includes a control unit 30, an image acquisition unit 31, and a display control unit 32. In the processor device 15, the functions of the image acquisition unit 31 and the display control unit 32 are realized by the operation of a program in the program memory by the control unit 30 constituted by a processor.

[0031] The computer 16 (image processing device) includes an image acquisition unit 33, a swallowing determination unit 34, a result recording unit 35, a control processor 62, and a program memory 63. In the computer 16, the functions of the image acquisition unit 33, the swallowing determination unit 34, and the result recording unit 35 are realized by the operation of the program in the program memory 63 by the control processor 62. Note that the computer 16 and / or the light source control unit 22 may be included in the processor device 15. The result recording unit 35 generates and edits videos of images to be displayed on the display 18 and images to be output to the recording device 17.

[0032] Hereinafter, the function of the swallowing determination unit 34 will be described. The swallowing determination unit 34 determines whether the acquired inspection image is during swallowing or not during swallowing. Swallowing refers to a series of actions of putting food or drink into the mouth, chewing, swallowing, and sending it into the esophagus. Fig. 2 is an explanatory diagram of normal swallowing, and Fig. 3 is an explanatory diagram of abnormal swallowing (aspiration). As shown in Fig. 2, the swallowing motion is divided into an "oral phase" in which the food F is transported from the oral cavity to the pharynx mainly by the movement of the tongue To, a "pharyngeal phase" in which the food F is transported from the pharynx to the esophagus Es by the swallowing reflex, and an "esophageal phase" in which the food F is transported from the esophagus Es to the stomach by the peristaltic motion of the esophagus.

[0033] During swallowing, in order to direct the food F toward the esophagus Es and prevent it from flowing into the trachea Tr, the epiglottis Eg, which plays a role of covering the trachea Tr, closes the entrance (glottis) of the trachea Tr by a reflex motion. Also, the soft palate Sp, which is the ceiling of the oral cavity, moves backward to close the passage between the oral cavity and the nasal cavity, preventing the food F from entering the nasal cavity. If some kind of dysfunction occurs at any timing of the oral phase, pharyngeal phase, or esophageal phase, as shown in Fig. 4, the food F that should be transported to the esophagus Es in a normal state flows into the trachea Tr, which is called aspiration.

[0034] Example 1 of aspiration in FIG. 3 is an example of aspiration in which food F flows into the trachea Tr before the swallowing reflex occurs from the oral phase to the pharyngeal phase. Example 2 of aspiration in FIG. 3 is an example of aspiration in which food F flows into the trachea Tr due to incomplete closure of the glottis (the entrance of the trachea Tr) by the epiglottis Eg during the middle of the swallowing reflex from the pharyngeal phase to the esophageal phase. Example 3 of aspiration in FIG. 3 is an example of aspiration in which the food F remaining in the vallecula epiglottica Ev or the piriform recess (refer to the examination image PT in FIG. 4) which is a depression existing on the left and right of the entrance of the esophagus flows into the trachea Tr after the swallowing reflex.

[0035] The examination image obtained in the present embodiment is captured by inserting the insertion portion 12a of the endoscope 12 from the nasal cavity into the pharynx so that the distal end portion 12d of the endoscope comes near the position R of the middle pharyngeal portion shown in FIG. 4. It is preferable that the examination image includes anatomical structures such as the epiglottis Eg, the rima glottidis Rg, and the left and right piriform recesses Ps as shown in the examination image PT of FIG. 4. The rima glottidis Rg is the space between the left and right folds that make up the vocal cords. The following describes the case where the distal end portion 12d of the endoscope is disposed in the middle pharyngeal portion, but it may be disposed in the nasopharyngeal cavity, the upper pharyngeal portion, the lower pharyngeal portion, or the laryngeal portion to perform the determination of swallowing.

[0036] As shown in FIG. 5, the swallowing determination unit 34 includes a first frame classification unit 40, a swallowing block generation unit 41, and a time width determination unit 42. The first frame classification unit 40 classifies each frame of the examination image into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which swallowing is not being performed. Specifically, when the first frame classification unit 40 classifies the examination image acquired in frame units as a swallowing frame, an identification tag "1" representing the swallowing frame is attached to the examination image. On the other hand, when the first frame classification unit 40 classifies the examination image as a non-swallowing frame, an identification tag "0" representing the non-swallowing frame is attached to the examination image. When these identification tags "1" and "0" are represented in time series, for example, they are represented as shown in FIG. 6. The examination image to which the identification tag "1" is attached represents that it is a swallowing frame SF, and the examination image to which the identification tag "0" is attached represents that it is a non-swallowing frame NSF.

[0037] Incidentally, the first frame classification unit 40 is preferably a learning model learned by Deep Learning (the same applies to the following second frame classification unit 50). Specifically, the first frame classification unit 40 preferably performs machine learning in advance on inspection images of swallowing frames and non-swallowing frames. The machine learning may use unsupervised learning or semi-supervised learning that automatically clusters images of swallowing frames or non-swallowing frames. Also, it is known that one swallowing usually takes about several hundred milliseconds to 2 seconds. In the conventional swallowing detection method, images correctly determined to be during swallowing and images determined not to be during swallowing were mixed, seemingly showing multiple swallows.

[0038] Deep Learning is known as a method excellent in image recognition, and is an excellent technique for extracting features from a single image, detecting a target pattern, performing segmentation, or classification. However, the field of action recognition, which recognizes what kind of movement is taking place between images arranged in multiple in time sequence, is known to be technically more difficult than simple image recognition. In the present invention, as a process for supplementing the recognition of the swallowing action extending between a plurality of frames, the following swallowing block generation unit 41 and time width determination unit 42 are used.

[0039] When the time interval between adjacent swallowing frames is equal to or less than the first threshold, the swallowing block generation unit 41 blocks at least the adjacent swallowing frames as a swallowing block. Specifically, as shown in FIG. 7, the swallowing block generation unit 41 calculates the time intervals P1 to P10 between adjacent swallowing frames. The adjacent swallowing frames CF1 and CF2 having the time intervals P3 and P8 that are equal to or less than the first threshold among these time intervals P1 to P10 are integrated as swallowing blocks BL1 and BL2.

[0040] Incidentally, as a method of integration, for example, it is preferable to perform a process of changing all non-swallowing frames between adjacent swallowing frames CF1 to swallowing frames.

[0041] Note that the first threshold value is preferably 0.6 seconds or the number of frames equivalent to 0.6 seconds. The number of frames equivalent to 0.6 seconds is, for example, 18 frames when the frame rate of a video is 30 frames (flame per second), and 36 frames when the frame rate of the video is 60 frames. The reason why the first threshold value is 0.6 seconds or the like will be described later. Also, the time interval may be an interval represented in units such as seconds, or may be an interval represented in other units such as the number of frames determined from the frame rate or the number of pixels converted in a computer for executing image processing.

[0042] The time width determination unit 42 determines that a swallowing block whose time width is less than or equal to the second threshold value or greater than or equal to the third threshold value is not a swallowing block, or a swallowing frame whose time width is less than or equal to the second threshold value or greater than or equal to the third threshold value is not a swallowing frame. Specifically, as shown in FIG. 8, the time width determination unit 42 calculates the time widths W3 and W7 of the swallowing blocks, and also calculates the time widths W1, W2, and W3 to W9 of the swallowing frames. Since the time widths W3 and W7 of the swallowing blocks BL1 and BL2 are not less than the second threshold value, the swallowing blocks BL1 and BL2 are determined as swallowing blocks as they are. Along with this determination, the swallowing blocks BL1 and BL2 are assigned the identification tag "1". Among the time widths W1, W2, and W3 to W9, the swallowing frames having the time widths W1, W2, W5, W6, W8, and W9 that are less than or equal to the second threshold value are determined not to be swallowing frames. Along with this determination, the identification tag of the swallowing frames having the time widths W1, W2, W5, W6, W8, and W9 is changed from "1" to "0". On the other hand, the swallowing frame having the time width W4 that exceeds the second threshold value is determined to be a swallowing frame. Therefore, the identification tag "1" of the swallowing frame having the time width W4 is maintained as it is.

[0043] Furthermore, the time width determination unit 42 determines that a swallowing block with a time width equal to or greater than the third threshold is not a swallowing block, or a frame during swallowing with a time width equal to or greater than the third threshold is not a frame during swallowing. The third threshold is preferably 1.8 seconds, or 54 frames corresponding to 1.8 seconds (when the frame rate of the video is 30 frames (flame per second)).

[0044] Note that the second threshold is preferably 0.2 seconds, or 6 frames corresponding to 0.2 seconds (when the frame rate is 30 fps (flame per second)). The reason why the second threshold is 0.2 seconds or the like will be described later. Also, the time width may be, in addition to the time width expressed in seconds, etc., a width expressed in other units such as frames determined from the frame rate.

[0045] As described above, through the processing in the swallowing determination unit 34, by attaching the identification tag "1" to the swallowing block or the frame during swallowing, during the diagnosis or when playing back the video after diagnosis, the user can easily search for the inspection images during swallowing.

[0046] The reason for setting the first threshold to 0.6 seconds is as follows. FIG. 9 shows the histogram HGF of the time intervals of the frames during swallowing that were diagnosed as being during swallowing or not during swallowing by a user who is actually proficient in swallowing endoscopy examination from the swallowing endoscopy video. The histogram HGF has a median value of 8 seconds, a minimum value of 0.5 seconds, and a maximum value of 60 seconds. Also, there are almost no frames during swallowing with a time interval of 0.5 seconds or less. From the consideration of the above histogram HGF, the first threshold representing the boundary value for determining whether to integrate into the swallowing block is preferably set to 0.6 seconds.

[0047] The reason for setting the second threshold value to 0.2 seconds is as follows. FIG. 10 shows a histogram HGW of the time width of swallowing frames diagnosed as being in the process of swallowing or not in the process of swallowing by a user who is actually proficient in swallowing endoscopy from a swallowing endoscopy video. The histogram HGW has a median value of 0.6 seconds, a minimum value of 0.1 seconds, and a maximum value of 0.9 seconds. Further, FIG. 11 shows a histogram HGWx of the time width of swallowing frames correctly classified (TP (true positive)) and a histogram HGWy of the time width of swallowing frames misclassified (FP (false positive)) among the swallowing frames classified by the first frame classification unit 40, compared with the user's diagnosis. The histogram HGWx and the histogram HGWy are separated. In the histogram HGWx, the frequency is high at 0.1 second (3 frames) or less. In the histogram HGWy, the frequency is high at 0.5 second (15 frames) or more. From the consideration of the above histograms HGW, HGWx, and HGWy, it is preferable that the second threshold value representing the boundary value for determining whether or not it is in the process of swallowing is 0.2 seconds, which is between 0.2 and 0.5 seconds.

[0048] The reason for setting the third threshold value to 1.8 seconds is as follows. As shown in the histogram HGW, even considering those with a low occurrence probability, a time width of a swallowing frame that exceeds twice the maximum value, that is, 1.8 seconds, can be regarded as an incorrect classification. Therefore, it is preferable that the third threshold value is 1.8 seconds.

[0049] Next, a series of processes of the present invention will be described with reference to the flowchart of FIG. 12. The image acquisition unit 33 acquires inspection images. The first frame classification unit 40 classifies each frame of the inspection image into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which swallowing is not being performed. When the time interval between temporally adjacent swallowing frames is equal to or less than the first threshold value, the swallowing block generation unit 41 integrates at least the adjacent swallowing frames as a swallowing block. For example, adjacent swallowing frames CF1 and CF2 having time intervals P3 and P8 that are equal to or less than the first threshold value are integrated as swallowing blocks BL1 and BL2 (see FIG. 7).

[0050] The time width determination unit 42 determines that a swallowing frame with a time width less than or equal to the second threshold value or greater than or equal to the third threshold value is not a swallowing frame, or determines that the swallowing block with a time width less than or equal to the second threshold value or greater than or equal to the third threshold value is not a swallowing block. An identification tag "1" indicating that it is in the process of swallowing is assigned to the swallowing block and the swallowing frame. The above series of processes is repeated as long as the swallowing examination is pending.

[0051] [Second Embodiment] In the second embodiment, in the swallowing determination unit 34, the inspection images are classified according to the actions during swallowing. As shown in FIG. 13, the swallowing determination unit 34 includes a second frame classification unit 50, a swallowing block generation unit 51 according to actions, a time width determination unit 52, and an expression pattern determination unit 53. Note that since the parts other than the swallowing determination unit 34 are the same as those in the first embodiment, the description thereof is omitted.

[0052] The second frame classification unit 50 classifies each frame of the inspection image into either a plurality of frames according to swallowing actions corresponding to the swallowing action or a non-swallowing frame in which no swallowing is performed. Specifically, when the second frame classification unit 50 classifies the inspection image acquired in frame units as a frame according to swallowing actions, it is classified as a frame according to swallowing actions into either a pre-swallowing frame, a swallowing frame, or a post-swallowing frame.

[0053] The non-swallowing frame, the pre-swallowing frame, the swallowing frame, or the post-swallowing frame is obtained as follows according to the swallowing action. FIG. 14 shows inspection images of non-swallowing, pre-swallowing, swallowing, post-swallowing, and non-swallowing along with the swallowing action arranged in chronological order. In the non-swallowing state, a laryngeal lid opening frame in which the laryngeal lid is opened is obtained as the non-swallowing frame. The laryngeal lid opening frame is about one frame.

[0054] Just before swallowing, as a pre-swallowing frame, a pre-swallowing blur frame in which the epiglottis moves at high speed and blurring occurs can be obtained. The pre-swallowing blur frame is about 1 frame. The blur of the pre-swallowing blur frame is characteristically represented by movements such as the lifting of the epiglottis and the approach of structures. In addition, as other pre-swallowing frames, an epiglottis inversion frame or a food inflow frame can be obtained.

[0055] During swallowing, as a swallowing frame, a white-out frame in which a wide part of the screen is covered with white halation or a shielding frame in which the entire screen is out of focus and covered with a slightly dark color tone can be obtained. The shielding frame appears after the white-out frame. The white-out frame is about 2 frames, which is more than the pre-swallowing blur frame. The shielding frame is about 7 frames, which is more than the white-out frame. In addition, as other swallowing frames, a flow frame in the process of swallowing colored swallowing water, a colored water inflow frame, or a swallowed food frame can be obtained. The colored water includes liquids such as milk and water mixed with food coloring, such as green, yellow, red, etc. Also, the colored water inflow frame may be classified into a plurality of frames according to the type of colored water. Similarly, the swallowed food frame may be classified into a plurality of frames according to the type of swallowed food. The swallowed food includes, for example, liquid foods such as pudding, solids such as rice and meat.

[0056] Just after swallowing, as a post-swallowing frame, a blackout frame in which the entire screen is dark can be obtained. The blackout frame is about 1 frame. In addition, as other post-swallowing frames, a post-swallowing blur frame characteristically represented by the movement of the epiglottis can be obtained.

[0057] In addition, when the second frame classification unit 50 classifies the inspection image as a frame according to the swallowing action, an identification tag "1" indicating that the swallowing action is being performed is attached to the inspection image. On the other hand, when the inspection image is classified as a non-swallowing frame, an identification tag "0" representing the non-swallowing frame is attached to the inspection image.

[0058] When the time interval between swallowing-action-specific frames that are temporally adjacent is less than or equal to the first threshold, the swallowing-block generation unit 51 for each action integrates at least the adjacent swallowing-action-specific frames as a swallowing block for each action. Specifically, as shown in FIG. 15, when the second frame classification unit 50 classifies the frames into blank frames, shielded frames, and flow frames, the swallowing-block generation unit 51 for each action calculates the time intervals P1 to P7 between adjacent swallowing-action-specific frames. Among these time intervals P1 to P7, the adjacent shielded frames SD1 and SD2 having a time interval P6 that is less than or equal to the first threshold are integrated as a shielded block SDBL as a swallowing-action-specific frame. Note that, similar to the first embodiment, the first threshold is preferably 1 second or a time corresponding to 30 frames (when the frame rate is 30 fps (frames per second)).

[0059] The time-width determination unit 52 determines that a swallowing-action-specific block whose time width is less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing-action-specific block, or a swallowing-action-specific frame whose time width is less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing-action-specific frame. Specifically, as shown in FIG. 16, the time-width determination unit 52 calculates the time width W6 of a shielded block SDBL that is a swallowing-action-specific block, and also calculates the time widths W1 to W5 of shielded frames, blank frames, and flow frames. Since none of the time widths W6 of the shielded blocks SDBL are less than or equal to the second threshold, the shielded blocks SDBL are determined as swallowing blocks as they are. Along with this determination, an identification tag "1" is assigned to the shielded block SDBL1. Note that, similar to the first embodiment, the second threshold is preferably 0.2 second or a time corresponding to 6 frames (when the frame rate is 30 fps (frames per second)).

[0060] A shielding frame having time widths W1 and W5 exceeding the second threshold, or a white-out frame having a time width W2 exceeding the second threshold is determined to be a swallowing frame. In this case, the identification tag "1" is maintained as it is. On the other hand, a white-out frame having a time width W3 equal to or less than the second threshold, or a shielding frame having a time width W4 equal to or less than the second threshold is determined not to be a swallowing frame. Along with this determination, the identification tag is changed from "1" to "0".

[0061] The expression pattern determination unit 53 determines whether the expression pattern is in the swallowing state or the non-swallowing state based on the expression pattern including the swallowing-specific motion frame or the motion-specific swallowing block. The expression pattern represents a classification pattern in time series. Specifically, since the expression patterns in FIGS. 17(A) to (E) do not correspond to any of the non-swallowing expression patterns, these expression patterns are determined to be in the swallowing state.

[0062] FIG. 17(A) is an expression pattern that occurs in the order of a non-swallowing frame, a white-out frame, a shielding frame, and a non-swallowing frame. FIG. 17(B) is an expression pattern that occurs in the order of a non-swallowing frame, a pre-swallowing breve frame, a white-out frame, a shielding frame, a blackout frame, and a non-swallowing frame. FIG. 17(C) is an expression pattern that occurs in the order of a non-swallowing frame, a pre-swallowing breve frame, a shielding frame, a blackout frame, and a non-swallowing frame. FIG. 17(D) is an expression pattern that occurs in the order of a non-swallowing frame, a flow frame, a blackout frame, and a non-swallowing frame. FIG. 17(E) is an expression pattern that occurs in the order of a non-swallowing frame, a food inflow frame, a swallowing food frame, a shielding frame, and a non-swallowing frame.

[0063] Note that, as shown in FIGS. 17(D) and (E), there are also swallowing operations in which a white-out frame does not occur. As a result, even in patients with weakened swallowing function where no high pixel value region appears, accurate swallowing determination can be made. Also, as shown in FIG. 17(D), when colored swallowing water is swallowed, there are no pre-swallowing blur frames or white-out frames, and by determining the flow frame, it can be determined that swallowing is in progress. Further, as shown in FIG. 17(E), when swallowing food is swallowed, a pattern characteristic of the swallowing food can also be a point for determining swallowing.

[0064] On the other hand, since the expression patterns in FIGS. 17(F) and (G) both correspond to non-swallowing expression patterns, these expression patterns are determined to be during non-swallowing. FIG. 17(F) is an expression pattern that occurs in the order of a non-swallowing frame, a white-out frame, and a non-swallowing frame. In this case, a white-out frame occurs, for example, when the tip 12d of the endoscope approaches the mucosal side wall, but since no pre-swallowing blur frame or shielding frame occurs, it can be determined that the series of operations is during non-swallowing. FIG. 17(G) is an expression pattern that occurs in the order of a non-swallowing frame, a shielding frame, a blackout frame, and a non-swallowing frame. In this case, since no pre-swallowing blur frame and white-out frame exist due to, for example, dirt adhering to the imaging surface of the tip 12 of the endoscope, it can be determined that the series of operations is during non-swallowing.

[0065] [Third Embodiment] In the third embodiment, in the swallowing determination unit 34, in addition to classifying the inspection image according to the operation during swallowing as in the second embodiment, the inspection image is classified according to brightness, and the inspection image is classified according to blur. As shown in FIG. 18, the swallowing determination unit 34 includes a second frame classification unit 50, a third frame classification unit 55, a fourth frame classification unit 56, a swallowing block generation unit 51 for each operation, a time width determination unit 52, and an expression pattern determination unit 53. Note that the second frame classification unit 50, the swallowing block generation unit 51 for each operation, and the time width determination unit 52 are the same as those in the second embodiment, and thus the description thereof is omitted.

[0066] The third frame classification unit 55 classifies the inspection image into a plurality of brightness information frames according to brightness. The brightness information frames include a high-brightness frame, a medium-brightness frame, or a low-brightness frame according to the magnitude of brightness. The magnitude of the brightness of a certain frame is, for example, the average value of the luminance in the HLS color space (minimum 0, maximum 255) for all the pixels in that frame, with 130 or more being high brightness, 50 or more and less than 130 being medium brightness, and less than 50 being low brightness. More preferably, within a certain period of time, for example, within 10 seconds, the average value and standard deviation of the brightness per frame are calculated, the threshold values for high brightness and medium brightness are set as (average value + standard deviation), the threshold values for medium brightness and low brightness are set as (average value - standard deviation), and the threshold values may be sequentially calculated using the moving average method.

[0067] The expression pattern determination unit 53 determines that the expression pattern is either during swallowing or not during swallowing based on the expression pattern including the brightness information frame, the frame for each swallowing motion, or the swallowing block for each motion. Specifically, as shown in FIG. 19, in the case of an expression pattern where a high-brightness frame exists before the frame during swallowing or overlaps with the frame during swallowing, and a low-brightness frame occurs after the frame during swallowing, the expression pattern determination unit 53 determines that from the high-brightness frame to the end of the frame during swallowing is during swallowing. Note that the expression pattern determination unit 53 preferably also determines that it is during swallowing when the frames during swallowing are continuous within 0.6 seconds behind the high-brightness frame in terms of time.

[0068] On the one hand, as shown in FIG. 20, even if the second frame classification unit 50 determines that it is a swallowing frame, in the case of an expression pattern in which a medium brightness frame continues through the third frame classification unit 55 before and after that, the expression pattern determination unit 53 determines that it is not during swallowing. As another example, as a result of the third frame classification unit 55, even if a high brightness frame, a medium brightness frame, and a low brightness frame appear in this order, if there is no swallowing frame in the second frame classification unit 50, it is determined that it is not during swallowing. Note that the expression pattern determination unit 53 preferably determines non-swallowing when the high brightness frame is alone, or when there is no high brightness frame and only the swallowing food frame is present. When a low brightness frame appears during the appearance of the swallowing frame, the expression pattern determination unit 53 preferably determines non-swallowing after the appearance of the low brightness frame.

[0069] Note that when classifying inspection images according to brightness as in the third frame classification unit 55, the time width may be calculated based on the appearance pattern of the pixel values of the images. Further, the third frame classification unit 55 may separate a high brightness frame when the area of the halation pixels whose pixel values exceed a certain value is equal to or larger than a certain area.

[0070] The fourth frame classification unit 56 classifies a frame in which the blur of the inspection image is equal to or greater than the blur threshold as a blur frame. The expression pattern determination unit 53 determines that the expression pattern is either during swallowing or not during swallowing based on an expression pattern including a blur frame, a frame for each swallowing motion, or a swallowing block for each motion. Further, the fourth frame classification unit 56 classifies alone the blur frame, or in addition to or instead of the blur frame, after detecting any one of the blur amount, the magnitude of the image difference, and the movement amount of the feature points, based on the classification or detection result and the frame classified by at least any one of the first frame classification unit 40, the second frame classification unit 50, the third frame classification unit 55, or the fourth frame classification unit 56, the expression pattern determination unit 53 may determine whether the expression pattern based on the swallowing motion is during swallowing or not during swallowing.

[0071] For example, as shown in FIG. 21, when the fourth frame classification unit 56 classifies the frame immediately before the shielding frame as a blur frame and classifies the frame immediately before the low-brightness frame as a blur frame, the expression pattern determination unit 53 determines the blur frame immediately before the shielding frame as the first frame during swallowing, and determines the blur frame immediately before the low-brightness frame as the last frame during swallowing. On the other hand, as shown in FIG. 22, in the case of an expression pattern in which there is no blur frame immediately before the shielding frame and the shielding frame and blackout occur in this order, the expression pattern determination unit 53 determines that it is not during swallowing.

[0072] Note that the best embodiment is as follows. The video obtained by the endoscope 12 is processed by the processor frame by frame. In the second frame classification unit 50, the images are classified into "not during swallowing", "spitting", "shielding", "colored water inflow", and "swallowed food". The second frame classification unit 50 is a classifier that has previously learned the above-classified images by machine learning. Among these, "spitting", "shielding", "colored water inflow", and "swallowed food" are all treated as frames during swallowing, and "not during swallowing" is treated as a frame not during swallowing. In the swallowing block generation unit 41, the first threshold is set to 0.6 seconds, and integration into the swallowing block is performed.

[0073] At the same time, in the third frame classification unit 55, first, the average value of the luminance in the HLS color space is calculated as the brightness for all the pixels in the frame. To classify a certain frame, the average and standard deviation of the brightness in the 30 seconds before in terms of time are calculated. If the brightness of the frame is equal to or higher than the average + standard deviation, it is classified as "high brightness", if it is less than the average - standard deviation, it is classified as "low brightness", and if it is in between, it is classified as "medium brightness".

[0074] The expression pattern determination unit 53 determines whether it is during swallowing or not during swallowing according to the expression patterns in the second frame classification unit 50, the third frame classification unit 55, and the fourth frame classification unit 56. For example, if there is a "high brightness" frame and a "during swallowing frame" (swallowing block) follows within 0.6 seconds after the "high brightness" frame in terms of time, it is considered "during swallowing". However, in the case of only the "high brightness" frame or when there is no "high brightness" frame and only the "swallowed food" frame, it is determined as not during swallowing. Also, a swallowing frame with a time width of 0.2 seconds or less or 1.8 seconds or more is determined as not during swallowing. Note that the patterns determined by the expression pattern determination unit 53 are not limited to these, and a plurality of swallowing patterns and non-swallowing patterns are determined in advance for the determination.

[0075] In the above embodiment, the swallowing block generation unit 41 and the time width determination unit 42 are used to determine the swallowing frame or the swallowing block. However, the swallowing frame and the like may be determined by other methods. For example, corresponding to the time axis, according to the appearance frequencies of the identification tags "0" and "1" between adjacent swallowing frames, a weighting function may be used to change the frame to a swallowing frame or a non-swallowing frame. In this case, as shown in FIG. 23, the swallowing determination unit 40 is provided with a frame type change unit 60 instead of the swallowing block generation unit 41 and the time width determination unit 42.

[0076] The frame type change unit 60 calculates the value of the weighting function for evaluating whether it is during swallowing or not during swallowing for each frame, and changes the type of the frame to a swallowing frame or a non-swallowing frame according to the value of the weighting function. The weighting function is a function that evaluates whether a specific frame is during swallowing or not during swallowing according to the appearance frequencies and appearance patterns of swallowing frames and non-swallowing frames in the time before and after the specific frame.

[0077] Specifically, when a swallowing frame appears in the nth frame, the weighting function of that frame is calculated as f(n) = 1. When a non-swallowing frame appears in the next (n + 1)th frame, using a coefficient k (for example, k = 0.1), f(n + 1) is calculated as f(n + 1) = f(n) - k. When non-swallowing frames appear continuously for two frames in the next (n + 2)th frame, f(n + 2) = f(n + 1) - k × 2. When non-swallowing frames continue up to the (n + i)th frame, f(n + i) = f(n + i - 1) - k × i. The function is such that the value of the weighting function becomes smaller as non-swallowing frames continue. When the value of the weighting function is greater than or equal to a type change threshold (for example, 0.5), the frame type change unit 60 maintains the swallowing frame as it is and changes the non-swallowing frame to a swallowing frame. On the other hand, when the value of the weighting function is less than the type change threshold, the frame type change unit 60 maintains the non-swallowing frame as it is and changes the swallowing frame to a non-swallowing frame.

[0078] Also, as another form of the weighting function, a moving average process corresponding to the time axis may be performed on the identification tags "0" and "1" between adjacent swallowing frames. For example, the weighting function f(n) of the nth frame is subjected to a simple moving average process for the 20 frames before and after. That is, it is calculated as f(n) = (f(n - 9) + f(n - 8) + ··· + f(n + 10)) / 20. Regarding changing the frame type to a swallowing frame or a non-swallowing frame based on the value of this weighting function, similar to the above frame type change unit 60, a type change threshold (for example, 0.5) is used. The range of the moving average and the type change threshold are not limited to these values and may be other values. The calculation formula of the moving average is not limited to the simple moving average shown in the example, and generally known moving average calculation methods such as the central moving average, backward moving average, forward moving average, or weighted moving average can also be used.

[0079] In this embodiment, the hardware structure (such as the control processor 62) of the processing unit that executes various processes such as the control unit 30, the image acquisition unit 31, the display control unit 32, the image acquisition unit 33, the swallowing determination unit 34, the result recording unit 35, the first frame classification unit, the swallowing block generation unit 41, the temporal width determination unit 42, the second frame classification unit 50, the movement-specific swallowing block generation unit 51, the temporal width determination unit 52, the expression pattern determination unit 53, the third frame classification unit 55, and the fourth frame classification unit 56 (not shown) is various processors as follows. The various processors include the CPU (Central Processing Unit), which is a general-purpose processor that executes software (program) and functions as various processing units, the GPU (Graphics Processing Unit) that performs high-speed image processing, the Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0080] One processing unit may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). Also, a plurality of processing units may be constituted by one processor. As an example of constituting a plurality of processing units by one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is constituted by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Second, as represented by a System On Chip (SoC) or the like, there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. Thus, the various processing units are constituted by using one or more of the above various processors as a hardware structure.

[0081] Furthermore, the hardware structure of each of these various processors is, more specifically, an electrical circuitry in the form of a combination of circuit elements such as semiconductor elements. Also, the hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

Explanation of Signs

[0082] 10 Endoscope system 12 Endoscope 12a Insertion section 12b Operation section 12c Bending section 12d Tip section 12e Angle knob 12h Still image acquisition instruction switch 12i Zoom operation section 14 Light source device 15 Processor device 16 Computer 17 Recording device 18 Display 19 User interface 20 Light source section 22 Light source control section 30 Control section 31 Image acquisition section 32 Display control section 33 Image acquisition section 34 Swallowing determination section 35 Result recording section 40 First frame classification section 41 Swallowing block generation section 42 Temporal width determination section 50 Second frame classification section 51 Swallowing block generation section by action 52 Temporal width determination section 53 Expression pattern determination section 55 Third frame classification section 56 Fourth frame classification section 60 Frame type change section 62 Control processor Memory for 63 programs Es Esophagus Eg Epiglottis Ev Vallecula epiglottica F Food Rg Rima glottidis Ps Piriform recess Sp Soft palate To Tongue Tr Trachea

Claims

1. An image processing apparatus comprising a control processor, wherein the control processor acquires inspection images, and for each frame of the inspection images, classifies them into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which swallowing is not being performed, and when the time interval between adjacent swallowing frames is equal to or less than a first threshold, integrates at least the adjacent swallowing frames as a swallowing block.

2. The image processing apparatus according to claim 1, wherein the control processor determines that the swallowing frame having a time width equal to or less than a second threshold or equal to or more than a third threshold is not a swallowing frame, or determines that the swallowing block having a time width equal to or less than a second threshold is not a swallowing block.

3. An image processing apparatus comprising a control processor, wherein the control processor acquires inspection images, and for each frame of the inspection images, classifies them into either a plurality of frames for each swallowing motion corresponding to the swallowing motion or a non-swallowing frame in which swallowing is not being performed, and when the time interval between adjacent frames for each swallowing motion is equal to or less than a first threshold, integrates at least the adjacent frames for each swallowing motion as a swallowing block for each motion.

4. The image processing apparatus according to claim 3, wherein the control processor determines that the frame for each swallowing motion having a time width equal to or less than a second threshold or equal to or more than a third threshold is not a frame for each swallowing motion, or determines that the swallowing block for each motion having a time width equal to or less than a second threshold is not a swallowing block for each motion.

5. The image processing apparatus according to claim 3, wherein the control processor determines that the expression pattern including the frame for each swallowing motion or the swallowing block for each motion is either in the state of swallowing or non-swallowing based on the expression pattern.

6. The image processing apparatus according to claim 3, wherein the frame for each swallowing motion includes a pre-swallowing frame, a swallowing frame, or a post-swallowing frame.

7. The image processing apparatus according to claim 6, wherein the pre-swallowing frame includes a pre-swallowing breve frame, an epiglottis inversion frame, or a food inflow frame.

8. The image processing apparatus according to claim 6, wherein the swallowing frame includes a spitting frame, a shielding frame, a flow frame, a colored water inflow frame, or a swallowed food frame.

9. The image processing apparatus according to claim 6, wherein the immediately after swallowing frame includes an immediately after swallowing breath frame or a blackout frame.

10. When the control processor classifies the inspection image into a plurality of brightness information frames according to brightness, The control processor determines that the expression pattern is either during swallowing or not during swallowing based on the expression pattern including the brightness information frame, the frame for each swallowing motion, or the swallowing block for each motion, according to the image processing apparatus of claim 3.

11. The image processing apparatus according to claim 10, wherein the brightness information frame includes a high brightness frame, a medium brightness frame, or a low brightness frame.

12. When the control processor classifies a frame in which the blur of the inspection image is equal to or less than a blur threshold as an immediately before swallowing blur frame, The control processor determines that the expression pattern is either during swallowing or not during swallowing based on the expression pattern including the immediately before swallowing blur frame, the frame for each swallowing motion, or the swallowing block for each motion, according to the image processing apparatus of claim 3.

13. The control processor calculates a value of a weighting function for evaluating whether it is during swallowing or not for each frame, and changes the type of the frame to a frame during swallowing or a frame not during swallowing according to the value of the weighting function, according to the image processing apparatus of claim 1.

14. A control processor, A step of acquiring an inspection image; A step of classifying each frame of the inspection image into either a frame during swallowing in which swallowing is being performed or a frame not during swallowing in which swallowing is not being performed; A method of operating an image processing apparatus, comprising a step of integrating at least the adjacent frames during swallowing as a swallowing block when a time interval between adjacent frames during swallowing is equal to or less than a first threshold.

15. The control processor determines that the frame during swallowing having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a frame during swallowing, or determines that the swallowing block having a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block, according to the method of operating an image processing apparatus of claim 14.

Images