Image processing apparatus and operating method thereof

Abstract

The invention provides an image processing device and a method for operating the same, which can prevent omission during an examination or reduce the burden of a user in an endoscope swallowing examination. The image acquisition unit (33) acquires an inspection image. A first frame classification unit (40) classifies, for each frame of an examination image, either a swallowing frame (SF) in which swallowing is performed or a non-swallowing frame (NSF) in which no swallowing is performed. When the time interval between temporally adjacent swallowing intermediate frames (SF) is equal to or less than a first threshold value, the swallowing mass generation unit (41) integrates at least adjacent swallowing intermediate frames (SF) into swallowing masses (BL1, BL2).

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for analyzing an image obtained in an endoscopic swallowing examination and a method of operating the same. Background Art

[0002] Dysphagia occurs along with aging or neurological diseases. Therefore, in recent years in an aging society, the importance of examining swallowing function has become increasingly high. It is desired that the examination of swallowing function can determine the condition of aspiration and perform appropriate treatment and prevention of dysphagia. Therefore, video endoscopic examination of swallowing (VE) has been established as a method for evaluating dysphagia (swallowing function evaluation examination) (for example, Patent Documents 1 and 2).

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-179222

[0004] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2022-179218

[0005] In an endoscopic swallowing examination, a doctor as a user observes an image of a swallowing motion obtained through an endoscope and evaluates it dynamically. Therefore, since it is necessary to observe a large number of acquired images, there is a possibility of omission during the examination. Also, it is a burden on the user to review the dynamic image for a long time after the examination. It is necessary to prevent such omissions during the examination or reduce the burden on the user.

[0006] In addition, Patent Document 1 discloses a system for determining whether it is during swallowing or not for each image based on an examination image. Similarly, Patent Document 1 discloses a technique using deep learning for determining whether it is during swallowing or not. However, in the case of determining only one image, even during a period when it is actually not during swallowing, an image similar to that during swallowing may sometimes be obtained, or even during a period when it is actually during swallowing, an image similar to that not during swallowing may sometimes be obtained. Considering such similar images, highly accurate swallowing determination is required.

[0007] In addition, Patent Document 2 describes a method of detecting the amount of jitter, the magnitude of image difference, and the amount of movement of feature points. However, even during a period when it is actually not during swallowing, a value similar to that during swallowing may sometimes be obtained, or even during a period when it is actually during swallowing, a value similar to that not during swallowing may sometimes be obtained. Considering such similar values, highly accurate swallowing determination is required.

[0008] Further, in Patent Document 1, frames obtained from inspection images determined to be in the process of swallowing during a certain period are determined to be in the process of swallowing, and in Patent Document 2, the initial stage (start time) and the final stage (end time) of the swallowing action that occurs are detected. However, in the methods described in these Patent Documents 1 and 2, it is difficult to detect the entire swallowing action. Summary of the Invention

[0009] An object of the present invention is to provide an image processing apparatus and a method of operating the same that can prevent omissions during inspection or reduce the burden on the user in endoscopic swallowing inspection.

[0010] The image processing apparatus of the present invention includes a control processor that performs the following processing: acquiring inspection images, classifying each frame of the inspection images into either a swallowing frame in the process of swallowing or a non-swallowing frame not in the process of swallowing, and when the time interval between temporally adjacent swallowing frames is equal to or less than a first threshold, integrating at least the adjacent swallowing frames into a swallowing block.

[0011] Preferably, the control processor determines that a swallowing frame with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing frame, or determines that a swallowing block with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block.

[0012] The image processing apparatus of the present invention includes a control processor that performs the following processing: acquiring inspection images, classifying each frame of the inspection images into either a plurality of frames classified according to the swallowing action corresponding to the swallowing action or a non-swallowing frame not in the process of swallowing, and when the time interval between temporally adjacent frames classified according to the swallowing action is equal to or less than a first threshold, integrating at least the adjacent frames classified according to the swallowing action into a swallowing block classified according to the action.

[0013] Preferably, the control processor determines that a frame classified according to the swallowing action with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a frame classified according to the swallowing action, or determines that a swallowing block classified according to the action with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block classified according to the action. Preferably, the control processor determines either that the expression pattern including the frame classified according to the swallowing action or the swallowing block classified according to the action is in the process of swallowing or not in the process of swallowing according to the expression pattern.

[0014] Preferably, the frames classified according to the swallowing action include a frame immediately before swallowing, a frame in the process of swallowing, or a frame immediately after swallowing. That is, preferably, the frame immediately before swallowing includes a jitter frame immediately before swallowing, an epiglottis inversion frame, or a food inflow frame. Preferably, the frame in the process of swallowing includes a whitening frame, a shielding frame, a flowing frame, a colored water inflow frame, or a frame of swallowing food. Preferably, the frame immediately after swallowing includes a jitter frame immediately after swallowing or a black screen frame.

[0015] Preferably, when the control processor classifies the inspection image into a plurality of brightness information frames corresponding to brightness, the control processor determines whether the expression pattern is in the process of swallowing or not based on the expression pattern including the brightness information frame, the frame classified by the swallowing action, or the swallowing block classified by the action. Preferably, the brightness information frame includes a high brightness frame, a medium brightness frame, or a low brightness frame.

[0016] Preferably, when the control processor classifies a frame with jitter of the inspection image below the jitter threshold as a pre-swallowing jitter frame, the control processor determines whether the expression pattern is in the process of swallowing or not based on the expression pattern including the pre-swallowing jitter frame, the frame classified by the swallowing action, or the swallowing block classified by the action. Preferably, the control processor calculates the value of the weighting function for evaluating whether it is in the process of swallowing or not for each frame, and changes the type of the frame to a swallowing frame or a non-swallowing frame according to the value of the weighting function.

[0017] A method of operating an image processing apparatus according to the present invention includes the following steps performed by a control processor: acquiring an inspection image; classifying each frame of the inspection image into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which swallowing is not being performed; and when the time interval between adjacent swallowing frames in time is below a first threshold, integrating at least the adjacent swallowing frames into a swallowing block.

[0018] Preferably, the control processor determines that a swallowing frame with a time width below a second threshold or above a third threshold is not a swallowing frame, or determines that a swallowing block with a time width below a second threshold or above a third threshold is not a swallowing block.

[0019] Advantageous Effects of the Invention

[0020] According to the present invention, it is possible to prevent omissions during the inspection or reduce the burden on the user in endoscopic swallowing examinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of an endoscopic system.

[0022] Figure 2 is an explanatory diagram showing swallowing.

[0023] Figure 3 is an explanatory diagram showing aspiration.

[0024] Figure 4 is an explanatory diagram and an image diagram showing a method of photographing an endoscopic swallowing examination.

[0025] Figure 5 is a block diagram showing the function of a swallowing determination unit in the first embodiment.

[0026] Figure 6It is an explanatory diagram showing frames during swallowing and frames not during swallowing.

[0027] Figure 7 It is an explanatory diagram showing the integration of frames during swallowing in the first embodiment.

[0028] Figure 8 It is an explanatory diagram showing the determination of the time width in the first embodiment.

[0029] Figure 9 It is a curve graph showing the histogram of the time interval of frames during swallowing.

[0030] Figure 10 It is a curve graph showing the histogram of the time width of frames during swallowing.

[0031] Figure 11 It is a curve graph showing the histogram of the time width of frames during swallowing diagnosed as correct by the user and the histogram of the time width of frames during swallowing diagnosed as incorrect by the user.

[0032] Figure 12 It is a flowchart showing a series of processes of the present invention.

[0033] Figure 13 It is a block diagram showing the function of the swallowing determination unit in the second embodiment.

[0034] Figure 14 It is an explanatory diagram showing the inspection images during non-swallowing, just before swallowing, during swallowing, just after swallowing, and non-swallowing following the swallowing action arranged in chronological order.

[0035] Figure 15 It is an explanatory diagram showing the integration of the masked frames in the second embodiment.

[0036] Figure 16 It is an explanatory diagram showing the determination of the time width in the second embodiment.

[0037] Figure 17 Among them, (A) to (E) are explanatory diagrams showing expression patterns that do not conform to the expression pattern during non-swallowing, and (F) and (G) are explanatory diagrams showing expression patterns that conform to the expression pattern during non-swallowing.

[0038] Figure 18 It is a block diagram showing the function of the swallowing determination unit in the third embodiment.

[0039] Figure 19 It is an explanatory diagram of the case where the expression pattern determination unit determines that it is during swallowing based on the classification in the second frame classification unit and the third frame classification unit.

[0040] Figure 20It is an explanatory diagram of a case where the expression pattern determination unit determines that it is not in the process of swallowing according to the classification in the second frame classification unit and the third frame classification unit.

[0041] Figure 21 It is an explanatory diagram of a case where the expression pattern determination unit determines that it is in the process of swallowing according to the classification in the second frame classification unit, the third frame classification unit, or the fourth frame classification unit.

[0042] Figure 22 It is an explanatory diagram of a case where the expression pattern determination unit determines that it is not in the process of swallowing according to the classification in the second frame classification unit, the third frame classification unit, or the fourth frame classification unit.

[0043] Figure 23 It is a block diagram showing the function of the swallowing determination unit in a case different from the first to third embodiments.

[0044] Symbol Explanation

[0045] 10 - Endoscope system, 12 - Endoscope, 12a - Insertion portion, 12b - Operation portion, 12c - Bending portion, 12d - Tip portion, 12e - Angle control knob, 12h - Still image acquisition instruction switch, 12i - Zoom operation portion, 14 - Light source device, 15 - Processor device, 16 - Computer, 17 - Recording device, 18 - Display, 19 - User interface, 20 - Light source portion, 22 - Light source control portion, 30 - Control portion, 31 - Image acquisition portion, 32 - Display control portion, 33 - Image acquisition portion, 34 - Swallowing determination portion, 35 - Result recording portion, 40 - First frame classification portion, 41 - Swallowing block generation portion, 42 - Time width determination portion, 50 - Second frame classification portion, 51 - Swallowing block generation portion classified by action, 52 - Time width determination portion, 53 - Expression pattern determination portion, 55 - Third frame classification portion, 56 - Fourth frame classification portion, 60 - Frame type change portion, 62 - Control processor, 63 - Program memory, Es - Esophagus, Eg - Epiglottis, Ev - Vallecula epiglottica, F - Food, Rg - Glottis, Ps - Piriform sinus, Sp - Soft palate, To - Tongue, Tr - Trachea. Detailed Embodiments

[0046] [First Embodiment]

[0047] As Figure 1As shown, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, a computer 16, a recording device 17, a display 18, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15. The endoscope 12 includes an insertion portion 12a inserted into the body of the observed object, an operating portion 12b provided at the base end portion of the insertion portion 12a, and a bending portion 12c and a front end portion 12d provided at the front end side of the insertion portion 12a. The bending portion 12c is bent by operating the angle button 12e of the operating portion 12b. The front end portion 12d is directed to a desired direction by the bending action of the bending portion 12c. The endoscope 12 is an endoscope used in endoscopic swallowing examination.

[0048] An imaging optical system for imaging a subject and an illumination optical system for irradiating illumination light to the subject are provided inside the endoscope 12. The subject is a structure in a living body related to swallowing movement. Specifically, it is the pharynx and laryngeal part. The illumination light passes through the insertion part 12a of the endoscope 12 via a light guide, and is emitted toward the subject from the front end 12d via an illumination lens of the illumination optical system. In addition, when the light source unit 20 is built into the front end 12d of the endoscope, the illumination light is emitted toward the subject via the illumination lens of the illumination optical system instead of the light guide.

[0049] The imaging optical system has an objective lens and an imaging sensor. Light from the object of observation based on the irradiated illumination light is incident on the imaging sensor via the objective lens and the zoom lens. As a result, an image of the object of observation is formed in the imaging sensor. The zoom lens is a lens for magnifying the object of observation, and moves between the telephoto end and the wide-angle end by operating the zoom operating part 12i. The imaging sensor can be arranged at the front end 12d of the endoscope, or it can be a so-called fiberscope using an optical fiber bundle in the insertion part of the endoscope 12, and can be located at the operating part side end of the insertion part 12a.

[0050] The image sensor is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, etc. An inspection image is generated based on an image signal detected by the image sensor.

[0051] In addition to color image sensors provided with color filters (such as Bayer filters) for converting sensed light into color image signals, image sensors may also include monochrome image sensors that are not provided with color filters for converting sensed light into monochrome image signals. In addition, color image sensors may convert sensed light into CMY signals instead of RBG signals.

[0052] When acquiring a color image, the image signal includes a B image signal output from B pixels, a G image signal output from G pixels, and an R image signal output from R pixels. The image signal is output to the image acquisition unit 31 of the processor device 15 and acquired as a monochrome image or a color image, i.e., an inspection image. The inspection image acquired by the image acquisition unit 31 is output to the image acquisition unit 33 of the computer 16. The inspection image output to the image acquisition unit 33 is output to the swallowing determination unit 34. The inspection image is a static image taken during an endoscopic examination or a series of dynamic images that are continuous in the time series taken during an endoscopic examination.

[0053] At the operation unit 12b, in addition to the bending angle knob 12e, a still image acquisition instruction switch 12h used for acquiring an instruction of a still image of an observation object and a zoom operation unit 12i used for operating a zoom lens are provided.

[0054] The light source device 14 generates illumination light. The processor device 15 performs system control of the endoscopic system 10 and performs image processing on the image signal output from the endoscope 12, etc. The display 18 is a display unit that displays the image taken by the endoscope 12. The user interface 19 is an input device for performing setting input, etc. on the processor device 15, etc.

[0055] The light source device 14 includes a light source unit 20 that emits illumination light and a light source control unit 22 that controls the operation of the light source unit 20. The light source unit 20 emits illumination light for illuminating the subject. The light source unit 20 includes, for example, a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp. The light source control unit 22 controls the lighting or extinguishing and the light emission amount, etc. of each light source constituting the light source unit 20.

[0056] In addition, the light source unit 20 can be built into the endoscope 12. Also, the light source control unit 22 can be built into the endoscope 12, or can also be built into the processor device 15. The white color includes a so-called pseudo-white that is substantially equivalent to white when using the endoscope 12 to photograph a subject, and that is a mixture of purple light V, blue light B, green light G, or red light R. It may also include a light source that irradiates ultraviolet light or infrared light for the purpose of special light observation. And the light source unit 20 includes, as needed, a filter or the like that adjusts the wavelength band, spectrum, or light quantity of the illumination light. For example, in the light source unit 20, blue light B, green light G, and red light R can be sequentially switched and irradiated at high speed (high-speed switching), and in the imaging sensor, images of each color of the respective illumination lights are acquired by a monochromatic sensor or a color sensor, and they are synthesized in the processor to generate a white image. As a mechanism for high-speed switching of illumination lights of multiple wavelengths in the light source unit 20, there are a method of mechanically switching multiple color filters of different colors for a white light source such as a xenon lamp, a method of electronically switching the ON / OFF of multiple LEDs that emit different colors, and the like.

[0057] The processor device 15 includes a control unit 30, an image acquisition unit 31, and a display control unit 32. In the processor device 15, the control unit 30 composed of a processor operates according to a program in the program memory, thereby implementing the functions of the image acquisition unit 31 and the display control unit 32.

[0058] The computer 16 (image processing device) includes an image acquisition unit 33, a swallowing determination unit 34, a result recording unit 35, a control processor 62, and a program memory 63. In the computer 16, the functions of the image acquisition unit 33, the swallowing determination unit 34, and the result recording unit 35 are implemented by causing the program in the program memory 63 to operate by the control processor 62. Additionally, the computer 16 and / or the light source control unit 22 can be included in the processor device 15. The result recording unit 35 generates and performs dynamic image editing on the images displayed on the display 18 or output to the recording device 17.

[0059] Hereinafter, the function of the swallowing determination unit 34 will be described. The swallowing determination unit 34 determines whether the acquired examination image is during swallowing or not during swallowing. Swallowing refers to a series of actions of putting food or drink into the mouth, chewing, swallowing, and sending it into the esophagus. Figure 2 This is an explanatory diagram of normal swallowing, Figure 3 This is an explanatory diagram of abnormal swallowing (aspiration). As Figure 2 shown, the swallowing movement is divided into an "oral phase" in which food F is transported from the oral cavity to the pharynx mainly by the movement of the tongue To, a "pharyngeal phase" in which food F is transported from the pharynx to the esophagus Es through a swallowing reflex, and an "esophageal phase" in which food F is transported from the esophagus Es to the stomach through the peristaltic movement of the esophagus.

[0060] When swallowing, in order to prevent the food F from flowing into the trachea Tr towards the esophagus Es, the epiglottis Eg, which functions to cover the trachea Tr, closes the entrance (glottis) of the trachea Tr through a reflex movement. Also, the soft palate Sp, which is the roof of the oral cavity, moves backward to close the passage between the oral cavity and the nasal cavity, preventing the food F from entering the nasal cavity. When a certain dysfunction occurs at any timing during the oral phase, pharyngeal phase, or esophageal phase, as Figure 4 shown, the situation where the food F that should normally be transported to the esophagus Es flows into the trachea Tr is called aspiration.

[0061] Figure 3 Example 1 of aspiration is an example of aspiration where the food F flows into the trachea Tr before the swallowing reflex appears, from the oral phase through the pharyngeal phase. Figure 3 Example 2 of aspiration is an example of aspiration where the food F flows into the trachea Tr during the middle of the swallowing reflex from the pharyngeal phase through the esophageal phase due to incomplete closure of the glottis (the entrance of the trachea Tr) by the epiglottis Eg. Figure 3 Example 3 of aspiration is an example of aspiration where the food F remaining in the vallecula epiglottica Ev or the recesses on the left and right of the esophageal entrance, i.e., the piriform fossae (refer to Figure 4 the inspection image PT), flows into the trachea Tr after the swallowing reflex.

[0062] In the present embodiment, the inspection image is acquired by inserting the insertion portion 12a of the endoscope 12 from the nasal cavity into the pharynx and bringing the distal end portion 12d of the endoscope to a position Figure 4 near the position R of the mid-pharyngeal region as shown. As shown in Figure 4 the inspection image PT, the inspection image preferably includes anatomical structures such as the epiglottis Eg, the rima glottidis Rg, and the left and right piriform fossae Ps. The rima glottidis Rg refers to the space between the left and right folds that form the vocal cords. Hereinafter, the case where the distal end portion 12d of the endoscope is disposed in the mid-pharyngeal region will be described, but in addition to this, it may also be disposed in the nasopharyngeal cavity, the upper pharyngeal region, the lower pharyngeal region, or the laryngeal region to determine swallowing.

[0063] As shown in Figure 5 the swallowing determination unit 34 includes a first frame classification unit 40, a swallowing block generation unit 41, and a time width determination unit 42. The first frame classification unit 40 classifies each frame of the inspection image into either a swallowing-in-progress frame in which swallowing is being performed or a non-swallowing-in-progress frame in which swallowing is not being performed. Specifically, when the first frame classification unit 40 classifies the inspection image acquired in frame units as a swallowing-in-progress frame, an identification label "1" indicating the swallowing-in-progress frame is assigned to the inspection image. On the other hand, when the first frame classification unit 40 classifies the inspection image as a non-swallowing-in-progress frame, an identification label "0" indicating the non-swallowing-in-progress frame is assigned to the inspection image. If these identification labels "1" and "0" are represented in time series, then for example, as shown in Figure 6As shown, the inspection image assigned the identification label "1" represents the swallowing frame SF, and the inspection image assigned the identification label "0" represents the non-swallowing frame NSF.

[0064] In addition, the first frame classification unit 40 is preferably a learning model that has undergone deep learning (the same applies to the following second frame classification unit 50). Specifically, it is preferred that the first frame classification unit 40 performs pre-machine learning on the inspection images of the swallowing frames and the inspection images of the non-swallowing frames. The machine learning can also use unsupervised learning or semi-supervised learning that automatically clusters the images of the swallowing frames or non-swallowing frames. And it is known that usually several hundred ms to about 2 s are required for one swallowing. In the existing swallowing detection methods, the images determined to be correctly swallowing and the images determined to be non-swallowing are mixed, and there seemingly are multiple swallows on the surface.

[0065] Deep learning is considered an excellent method for image recognition, and it is an excellent method for extracting features from a single image to detect target patterns or perform segmentation or classification. However, it is known that it is technically more difficult in the field of action recognition, which involves determining what actions are taken between multiple images arranged in chronological order, compared to simple image recognition. In the present invention, as a supplement to the processing of action recognition during swallowing involving multiple frames, the following swallowing block generation unit 41 and time width determination unit 42 are used.

[0066] When the time interval between adjacent swallowing frames in time is equal to or less than the first threshold, the swallowing block generation unit 41 blocks at least the adjacent swallowing frames into swallowing blocks. Specifically, as Figure 7 shown, the swallowing block generation unit 41 calculates the time intervals P1 to P10 between adjacent swallowing frames. Among these time intervals P1 to P10, the adjacent swallowing frames CF1 and CF2 with time intervals P3 and P8 equal to or less than the first threshold are integrated as swallowing blocks BL1 and BL2.

[0067] In addition, as an integration method, for example, it is preferred to set it as a process of changing all the non-swallowing frames between adjacent swallowing frames CF1 to swallowing frames.

[0068] In addition, the first threshold is preferably set to 0.6 seconds or the number of frames equivalent to 0.6 seconds. The number of frames equivalent to 0.6 seconds is, for example, 18 frames when the frame rate of the moving image is 30 frames (frames per second), and 36 frames when the frame rate of the moving image is 60 frames. The reason for the first threshold being 0.6 seconds etc. will be described later. And the time interval can be set as an interval represented by other units such as the number of frames determined according to the frame rate and the number of pixels converted in the computer for image processing in addition to the time interval represented in seconds etc.

[0069] The time width determination unit 42 determines that a swallowing block with a time width less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing block, or a frame during swallowing with a time width less than or equal to the second threshold or greater than or equal to the third threshold is not a frame during swallowing. Specifically, as Figure 8 shown, the time width determination unit 42 calculates the time widths W3 and W7 of the swallowing blocks, and calculates the time widths W1, W2, and W3 to W9 of the frames during swallowing. Since the time widths W3 and W7 of the swallowing blocks BL1 and BL2 are not less than the second threshold, the swallowing blocks BL1 and BL2 are directly determined to be swallowing blocks. Along with this determination, the identification label "1" is assigned to the swallowing blocks BL1 and BL2. Among the time widths W1, W2, and W3 to W9, the frames during swallowing with the time widths W1, W2, W5, W6, W8, and W9 less than or equal to the second threshold are determined not to be frames during swallowing. Along with this determination, the identification labels of the frames during swallowing with the time widths W1, W2, W5, W6, W8, and W9 are changed from "1" to "0". On the other hand, the frame during swallowing with the time width W4 exceeding the second threshold is determined to be a frame during swallowing. Therefore, the frame during swallowing with the time width W4 maintains the identification label "1" unchanged.

[0070] Moreover, the time width determination unit 42 determines that a swallowing block with a time width greater than or equal to the third threshold is not a swallowing block, or a frame during swallowing with a time width greater than or equal to the third threshold is not a frame during swallowing. The third threshold is preferably 1.8 seconds or the number of frames equivalent to 1.8 seconds, i.e., 54 frames (when the frame rate of the moving image is 30 frames per second).

[0071] In addition, the second threshold is preferably set to 0.2 seconds or the number of frames equivalent to 0.2 seconds, i.e., 6 frames (when the frame rate is 30 fps). The reason for the second threshold being 0.2 seconds etc. will be described later. And the time width can be expressed in other units such as frames determined according to the frame rate in addition to the time width expressed in seconds etc.

[0072] As described above, through the processing of the swallowing determination unit 34, the identification label "1" is assigned to the swallowing block or the frame during swallowing, so that the user can easily search for the examination images during swallowing when playing the moving image during or after diagnosis.

[0073] In addition, the reason for setting the first threshold to 0.6 seconds is as follows. Figure 9Histogram HGF representing the time intervals of frames during swallowing that are diagnosed as frames during swallowing or frames during non-swallowing by a user proficient in endoscopic swallowing examination based on endoscopic swallowing dynamic images. The median of histogram HGF is 8 seconds, the minimum value is 0.5 seconds, and the maximum value is 60 seconds. Also, there are almost no frames during swallowing with a time interval of 0.5 seconds or less. Based on the consideration of the above histogram HGF, for the first threshold representing the boundary value for determining whether to integrate with the swallowing block, it is preferably set to 0.6 seconds.

[0074] In addition, the reason for setting the second threshold to 0.2 seconds is as follows. Figure 10 Histogram HGW representing the time widths of frames during swallowing that are diagnosed as frames during swallowing or frames during non-swallowing by a user proficient in endoscopic swallowing examination based on endoscopic swallowing dynamic images. The median of histogram HGW is 0.6 seconds, the minimum value is 0.1 seconds, and the maximum value is 0.9 seconds. And, Figure 11 Histogram HGWx representing the time widths of frames during swallowing that are correctly classified (TP (true positive)) compared to the user's diagnosis among the frames during swallowing classified by the first frame classification unit 40, and histogram HGWy representing the time widths of frames during swallowing that are misclassified (FP (false positive)). Histogram HGWx and histogram HGWy are separate. In histogram HGWx, the frequency is higher for 0.1 seconds (3 frames) or less. In histogram HGWy, the frequency is higher for 0.5 seconds (15 frames) or more. Based on the consideration of the above histograms HGW, HGWx, and HGWy, for the second threshold representing the boundary value for determining whether it is during swallowing, it is preferably set to 0.2 seconds between 0.1 and 0.5 seconds.

[0075] The reason for setting the third threshold to 1.8 seconds is as follows. As shown in histogram HGW, even considering the case with a low occurrence probability, a case where the time width of the frame during swallowing exceeds twice the maximum value, i.e., 1.8 seconds, can be regarded as misclassification. Therefore, the third threshold is preferably set to 1.8 seconds.

[0076] Next, a series of processes of the present invention will be described according to the Figure 12 flowchart. The image acquisition unit 33 acquires inspection images. The first frame classification unit 40 classifies each frame of the inspection image into either a frame during swallowing or a frame during non-swallowing where swallowing is not performed. When the time interval between adjacent frames during swallowing is less than or equal to the first threshold, the swallowing block generation unit 41 integrates at least the adjacent frames during swallowing into a swallowing block. For example, adjacent frames during swallowing CF1, CF2 with a time interval P3, P8 less than or equal to the first threshold are integrated into swallowing blocks BL1, BL2 (refer to Figure 7 ).

[0077] The time width determination unit 42 determines that a swallowing-in progress frame with a time width less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing-in progress frame, or determines that a swallowing block with a time width less than or equal to the second threshold or greater than or equal to the third threshold is not a swallowing block. The swallowing block and the swallowing-in progress frame are assigned an identification label "1" indicating that they are in the process of swallowing. As long as the swallowing examination is in progress, the above series of processes are repeated.

[0078] [Second Embodiment]

[0079] In the second embodiment, the swallowing determination unit 34 classifies the inspection images according to the actions during swallowing. As Figure 13 shown, the swallowing determination unit 34 includes a second frame classification unit 50, a swallowing block generation unit 51 that classifies by action, a time width determination unit 52, and an expression pattern determination unit 53. In addition, regarding the content other than the swallowing determination unit 34, it is the same as that in the first embodiment, so the description is omitted.

[0080] The second frame classification unit 50 classifies each frame of the inspection image into either a plurality of swallowing-action-classified frames corresponding to the swallowing action or a non-swallowing-in progress frame in which swallowing has not occurred. Specifically, when the inspection image obtained in frame units is classified as a swallowing-action-classified frame, the second frame classification unit 50 classifies it as a swallowing-action-classified frame and further classifies it into any one of a pre-swallowing frame, a swallowing-in progress frame, or a post-swallowing frame.

[0081] According to the swallowing action, the non-swallowing-in progress frame, the pre-swallowing frame, the swallowing-in progress frame, or the post-swallowing frame is obtained as follows. Figure 14 is a diagram showing the inspection images of non-swallowing, pre-swallowing, swallowing-in progress, post-swallowing, and non-swallowing in chronological order during the swallowing action. During non-swallowing, as a non-swallowing-in progress frame, an epiglottis-opening frame in which the epiglottis is open is obtained. The epiglottis-opening frame is about 1 frame.

[0082] Just before swallowing, as a pre-swallowing frame, a pre-swallowing jitter frame in which the epiglottis moves at high speed and jitters is obtained. The pre-swallowing jitter frame is about 1 frame. The jitter of the pre-swallowing jitter frame is characteristically represented by actions such as the raising of the epiglottis or the approach of a structure. In addition, as other pre-swallowing frames, an epiglottis-inverting frame or a food-inflow frame can be obtained.

[0083] During swallowing, as frames during swallowing, a whitened frame in which a wide part of the screen is covered with a white halo or an obscured frame in which the entire screen is blurred and covered with a slightly darker tone can be obtained. The obscured frame appears after the whitened frame. The whitened frame is about 2 frames more than the jitter frame just before swallowing. The obscured frame is about 7 frames more than the whitened frame. In addition, as other frames during swallowing, a flow frame, a colored water inflow frame, or a food swallowing frame during the process of swallowing colored swallowing water can be obtained. The colored water includes milk, and liquids such as green, yellow, red, etc. in which a food coloring agent is mixed in water. And the colored water inflow frame can be classified into a plurality of color-classified frames corresponding to the type of the colored water. Similarly, the food swallowing frame can be classified into a plurality of food-swallowing-classified frames corresponding to the type of the swallowed food. The swallowed food includes, for example, liquid foods such as pudding, and solid substances such as rice and meat.

[0084] Just after swallowing, as a frame just after swallowing, a black screen frame in which the entire screen is darker can be obtained. The black screen frame is about 1 frame. In addition, as other frames just after swallowing, a jitter frame just after swallowing that is characteristically represented by the movement of the epiglottis can be obtained.

[0085] In addition, when the inspection image is classified as a frame classified by swallowing action, the second frame classification unit 50 assigns the recognition label "1" indicating that a swallowing action is performed to the inspection image. On the other hand, when the inspection image is classified as a non-swallowing frame, the recognition label "0" indicating a non-swallowing frame is assigned to the inspection image.

[0086] When the time interval between temporally adjacent frames classified by swallowing action is equal to or less than the first threshold, the swallowing block generation unit 51 integrated by action at least integrates adjacent frames classified by swallowing action into a swallowing block integrated by action. Specifically, as Figure 15 shown, when the second frame classification unit 50 classifies a frame as a whitened frame, an obscured frame, or a flow frame, the swallowing block generation unit 51 integrated by action calculates the time intervals P1 to P7 between adjacent frames classified by swallowing action. The adjacent occlusion frames SD1 and SD2 having a time interval P6 equal to or less than the first threshold among these time intervals P1 to P7 are integrated into an obscured block SDBL as a frame classified by swallowing action. In addition, similar to the first embodiment, the first threshold is preferably set to 1 second or a time equivalent to 30 frames (when the frame rate is 30 fps (frames per second)).

[0087] The time width determination unit 52 determines that a swallowing block classified by swallowing action with a time width equal to or less than the second threshold or equal to or greater than the third threshold is not a swallowing block classified by swallowing action, or a frame classified by swallowing action with a time width equal to or less than the second threshold or equal to or greater than the third threshold is not a frame classified by swallowing action. Specifically, as Figure 16As shown, the time width determination unit 52 calculates the time width W6 of the occlusion block SDBL for the swallowing action classification block, and calculates the time widths W1 to W5 of the occlusion frame, the whitening frame, and the flowing frame. Since the time width W6 of the occlusion block SDBL is not below the second threshold, the occlusion block SDBL is directly determined as the swallowing block. Along with this determination, the recognition label "1" is assigned to the occlusion block SDBL. In addition, similar to the first embodiment, the second threshold is preferably set to 0.2 seconds or the time corresponding to 6 frames (when the frame rate is 30 fps (frames per second)).

[0088] The occlusion frame having a time width W1 exceeding the second threshold or the whitening frames having time widths W2 and W5 exceeding the second threshold are determined as the frames during swallowing. In this case, the recognition label "1" remains unchanged. On the other hand, the whitening frame having a time width W3 below the second threshold or the flowing frame having a time width W4 below the second threshold are determined as not being frames during swallowing. Along with this determination, the recognition label is changed from "1" to "0".

[0089] The expression pattern determination unit 53 determines whether the expression pattern is either during swallowing or not during swallowing based on the expression pattern including the frames classified by the swallowing action or the swallowing blocks classified by the action. The expression pattern represents the pattern of classification in the time series. Specifically, Figure 17 The expression patterns of (A) to (E) do not belong to the expression pattern of not during swallowing, so these expression patterns are determined as during swallowing.

[0090] Figure 17 (A) is the expression pattern generated in the order of the frames not during swallowing, the whitening frame, the occlusion frame, and the frames not during swallowing. Figure 17 (B) is the expression pattern generated in the order of the frames not during swallowing, the jitter frame before swallowing, the whitening frame, the occlusion frame, the black screen frame, and the frames not during swallowing. Figure 17 (C) is the expression pattern generated in the order of the frames not during swallowing, the jitter frame before swallowing, the occlusion frame, the black screen frame, and the frames not during swallowing. Figure 17 (D) is the expression pattern generated in the order of the frames not during swallowing, the flowing frame, the black screen frame, and the frames not during swallowing. Figure 17 (E) is the expression pattern generated in the order of the frames not during swallowing, the food inflow frame, the food swallowing frame, the occlusion frame, and the frames not during swallowing.

[0091] In addition, as shown in Figure 17 (D) and (E), there are also swallowing actions that do not generate whitening frames. Thus, even in the case where no high pixel value region appears in patients with weak swallowing function, the swallowing determination can be accurately performed. And, as shown in Figure 17As shown in (D), in the case of swallowing colored water, there are no jitter frames or whitening frames just before swallowing, and it can be determined as being in the process of swallowing through the determination of the flow frames. And, as Figure 17 shown in (E), in the case of swallowing food, the characteristic pattern of swallowing food can also be the key point for determining swallowing.

[0092] In contrast, Figure 17 The expression patterns of (F) and (G) both conform to the expression pattern of non-swallowing, so these expression patterns are determined to be in the state of non-swallowing. Figure 17 The expression pattern of (F) is generated in the order of non-swallowing frames, whitening frames, and non-swallowing frames. In this case, the front end portion 12d of the endoscope approaches the mucosal side wall or the like to generate whitening frames, but no jitter frames or occlusion frames just before swallowing are generated, so a series of actions can be determined to be in the state of non-swallowing. Figure 17 The expression pattern of (G) is generated in the order of non-swallowing frames, occlusion frames, black screen frames, and non-swallowing frames. In this case, since dirt or the like adheres to the imaging surface of the front end portion 12d of the endoscope and there are no jitter frames and whitening frames just before swallowing, a series of actions can be determined to be in the state of non-swallowing.

[0093] [Third Embodiment]

[0094] In the third embodiment, as in the second embodiment, in the swallowing determination unit 34, in addition to classifying the inspection images according to the actions during swallowing, the inspection images are also classified according to brightness, and the inspection images are classified according to jitter. As Figure 18 shown, the swallowing determination unit 34 includes a second frame classification unit 50, a third frame classification unit 55, a fourth frame classification unit 56, a swallowing block generation unit 51 for classifying by action, a time width determination unit 52, and an expression pattern determination unit 53. In addition, the second frame classification unit 50, the swallowing block generation unit 51 for classifying by action, and the time width determination unit 52 are the same as those in the second embodiment, so the description thereof is omitted.

[0095] The third frame classification unit 55 classifies the inspection images into a plurality of brightness information frames corresponding to brightness. Among the brightness information frames, there are high-brightness frames, medium-brightness frames, or low-brightness frames according to the magnitude of brightness. Regarding the magnitude of brightness of a certain frame, for example, for all pixels in the frame, the average value of the brightness in the HLS color space (minimum 0, maximum 255) is set, 130 or more is set as high brightness, 50 or more and less than 130 is set as medium brightness, and less than 50 is set as low brightness. More preferably, the following method is used: calculate the average value and standard deviation of the brightness of each frame within a certain specified time, for example, within 10 seconds, set the thresholds for high brightness and medium brightness as (average value + standard deviation), set the thresholds for medium brightness and low brightness as (average value - standard deviation), and use the moving average method to calculate the thresholds sequentially.

[0096] The expression pattern determination unit 53 determines whether the expression pattern is either during swallowing or not during swallowing, based on the expression pattern including the brightness information frame, the frame classified by swallowing action, or the swallowing block classified by action. Specifically, as Figure 19 shown, in the case of an expression pattern in which a high-brightness frame exists in a manner that repeats before or during the swallowing frame and a low-brightness frame is generated after the swallowing frame, the expression pattern determination unit 53 determines that it is during swallowing from the high-brightness frame to the end of the swallowing frame. In addition, the expression pattern determination unit 53 preferably determines that it is during swallowing also when the swallowing frames are continuous within 0.6 seconds after the time of the high-brightness frame.

[0097] On the other hand, as Figure 20 shown, even if the second frame classification unit 50 determines it as a swallowing frame, if it is an expression pattern in which medium-brightness frames continue in the third frame classification unit 55 before and after it, the expression pattern determination unit 53 also determines that it is not during swallowing. And, as another example, even if the result of the third frame classification unit 55 appears in the order of high-brightness frame, medium-brightness frame, and low-brightness frame, if there is no swallowing frame in the second frame classification unit 50, it is also determined that it is not during swallowing. In addition, when a high-brightness frame exists alone or when there is no high-brightness frame and a food-swallowing frame exists alone, the expression pattern determination unit 53 preferably determines that it is not during swallowing. When a low-brightness frame appears during the swallowing frame, the expression pattern determination unit 53 preferably determines that it is not during swallowing after the low-brightness frame appears.

[0098] In addition, as in the case of the third frame classification unit 55 where the inspection image is classified according to brightness, the time width can also be calculated based on the appearance pattern of the pixel values of the image. And, the third frame classification unit 55 can also separate from the high-brightness frame when the area of the halo pixels whose pixel values exceed the specified value is equal to or greater than the specified area.

[0099] The fourth frame classification unit 56 classifies the frames of the inspection image whose jitter is above the jitter threshold as jitter frames. The expression pattern determination unit 53 determines whether the expression pattern is either during swallowing or not during swallowing, based on the expression pattern including the jitter frames, the frame classified by swallowing action, or the swallowing block classified by action. And, the fourth frame classification unit 56 can also classify the jitter frames alone, or after detecting any one of the jitter amount, the magnitude of the image difference, and the movement amount of the feature points in addition to or instead of the jitter frames, determine whether the expression pattern based on the swallowing action is during swallowing or not during swallowing through the expression pattern determination unit 53 according to the classification or detection result and the frames classified by at least any one of the first frame classification unit 40, the second frame classification unit 50, the third frame classification unit 55, or the fourth frame classification unit 56.

[0100] For example, as Figure 21As shown, when the 4th frame classification unit 56 classifies the frame immediately in front of the occlusion frame as a jitter frame and classifies the frame immediately in front of the low-brightness frame as a jitter frame, the expression mode determination unit 53 determines the jitter frame immediately in front of the occlusion frame as the first frame during swallowing and determines the jitter frame immediately in front of the low-brightness frame as the last frame during swallowing. On the other hand, as Figure 22 shown, in the case of an expression mode in which there is no jitter frame immediately in front of the occlusion frame and the occlusion frame and the black screen frame are generated in this order, the expression mode determination unit 53 determines that it is not during swallowing.

[0101] In addition, the best embodiment is as follows. The moving image obtained by the endoscope 12 is processed frame by frame by the processor. In the 2nd frame classification unit 50, the image is classified into "not during swallowing", "whitening", "occlusion", "colored water inflow", and "swallowing food". The 2nd frame classification unit 50 is a classifier that has learned the above-classified images in advance by machine learning. Among them, "whitening", "occlusion", "colored water inflow", and "swallowing food" are all processed as frames during swallowing, and "not during swallowing" is processed as a frame not during swallowing. In the swallowing block generation unit 41, the first threshold is set to 0.6 seconds and integrated with the swallowing block.

[0102] At the same time, in the 3rd frame classification unit 55, first, the average value of the brightness in the HLS color space is calculated for all the pixels in the frame as the brightness. In order to classify a certain frame, the average value and the standard deviation of the brightness within the previous 30 seconds in time are calculated. If the brightness of this frame is greater than or equal to the average value + the standard deviation, it is classified as "high brightness", if it is less than the average value - the standard deviation, it is classified as "low brightness", and if it is in between, it is classified as "medium brightness".

[0103] The expression mode determination unit 53 determines whether it is during swallowing or not during swallowing based on the expression modes in the 2nd frame classification unit 50, the 3rd frame classification unit 55, and the 4th frame classification unit 56. For example, when there is a "high brightness" frame and "frames during swallowing" (swallowing blocks) are continuous within 0.6 seconds after the time of the "high brightness" frame, it is set as "during swallowing", but when there is only a "high brightness" frame or there is no "high brightness" frame and there is only a "swallowing food" frame alone, it is determined as not during swallowing. Also, frames during swallowing with a time width of 0.2 seconds or less or 1.8 seconds or more are determined as not during swallowing. In addition, the mode determined by the expression mode determination unit 53 is not limited to these, and multiple swallowing modes and non-swallowing modes are determined in advance.

[0104] In addition, in the above-described embodiment, the swallowing block generation unit 41 and the time width determination unit 42 are used to determine the swallowing middle frames or swallowing blocks. However, the determination of the swallowing middle frames or the like can also be performed by other methods. For example, corresponding to the time axis, according to the appearance frequencies of the identification tags "0" and "1" between adjacent swallowing middle frames, a weighting function can be used to perform the change to the swallowing middle frames or frames. In this case, as Figure 23 shown, a frame type change unit 60 is provided in the swallowing determination unit 34 instead of the swallowing block generation unit 41 and the time width determination unit 42.

[0105] The frame type change unit 60 calculates the value of the weighting function for evaluating swallowing or non-swallowing for each frame, and changes the type of the frame to a swallowing middle frame or a non-swallowing middle frame according to the value of the weighting function. The weighting function is a function that evaluates whether a specific frame is in the middle of swallowing or not according to the appearance frequencies or appearance patterns of the swallowing middle frames and non-swallowing middle frames in the front and back times for the specific frame.

[0106] Specifically, when a swallowing middle frame appears in the nth frame, the weighting function of this frame is calculated as f(n) = 1. When a non-swallowing middle frame appears in the next (n + 1)th frame, using a coefficient k (for example, k = 0.1), it is calculated as f(n + 1) = f(n) - k. When two consecutive non-swallowing middle frames also appear in the next (n + 2)th frame, it is set as f(n + 2) = f(n + 1) - k × 2. Until the next (n + i)th frame, when the non-swallowing middle frames are consecutive, it is set as f(n + i) = f(n + i - 1) - k × i, and it is set as a function where the value of the weighting function becomes smaller as the non-swallowing middle frames are more consecutive. When the value of the weighting function is equal to or greater than the type change threshold (for example, 0.5), the frame type change unit 60 keeps the swallowing middle frames unchanged and changes the non-swallowing middle frames to swallowing middle frames. On the other hand, when the value of the weighting function is less than the type change threshold, the frame type change unit 60 keeps the non-swallowing middle frames unchanged and changes the swallowing middle frames to non-swallowing middle frames.

[0107] Further, as another method of the weighting function, a moving average process corresponding to the time axis may be performed on the identification tags "0" and "1" between adjacent in-swallowing frames. For example, the weighting function f(n) of the n-th frame performs a simple moving average process on the 20 frames before and after. That is, it is calculated by f(n) = (f(n - 9) + f(n - 8) +... + f(n + 10)) / 20. Regarding the case of changing the frame type to an in-swallowing frame or a non-in-swallowing frame according to the value of this weighting function, similar to the above-described frame type changing unit 60, a type-changing threshold value (for example, 0.5) is used. The range of the moving average or the type-changing threshold value is not limited to this value and may be other values. The calculation formula of the moving average is not limited to the illustrated simple moving average, and general known moving average calculation methods such as a central moving average, a backward moving average, a forward moving average, or a weighted moving average may be used.

[0108] In the present embodiment, the hardware structure (such as the control processor 62) of the processing unit (processing unit) for performing various processes by the control unit 30, the image acquisition unit 31, the display control unit 32, the image acquisition unit 33, the swallowing determination unit 34, the result recording unit 35, the first frame classification unit 40, the swallowing block generation unit 41, the time width determination unit 42, the second frame classification unit 50, the swallowing block generation unit 51 classified by action, the time width determination unit 52, the expression mode determination unit 53, the third frame classification unit 55, the fourth frame classification unit 56 (not shown), etc. is various processors as follows. The various processors include a general-purpose processor that executes software (program) and functions as various processing units, namely, a CPU (Central Processing Unit), a GPU (graphics processing unit) that performs high-speed image processing, a processor that can change the circuit structure after manufacturing, such as an FPGA (Field Programmable Gate Array), i.e., a programmable logic device (Programmable Logic Device), and a processor with a circuit structure specifically designed to execute various processes, i.e., an application-specific circuit, etc.

[0109] A processing unit may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Also, a single processor may constitute multiple processing units. As an example of a single processor constituting multiple processing units, first, as typified by computers such as clients and servers, there is a way in which a processor is constituted by a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as typified by a system on chip (SoC), there is a way in which a processor that uses one IC (integrated circuit) chip to implement the functions of an entire system including multiple processing units is used. Thus, as a hardware structure, various processing units are constituted by using one or more of the above various processors.

[0110] Moreover, the hardware structure of these various processors is more specifically a circuitry that is a way of combining circuit elements such as semiconductor elements. And the hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

Claims

1. An image processing apparatus, which includes a control processor, The control processor performs the following processing: Obtain a check image, For each frame of the check image, classify it into either a swallowing frame in which swallowing is being performed or a non-swallowing frame in which the swallowing is not being performed, When the time interval between adjacent swallowing frames in time is equal to or less than a first threshold, at least integrate the adjacent swallowing frames into a swallowing block.

2. The image processing apparatus according to claim 1, wherein The control processor determines that a swallowing frame with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing frame, or determines that a swallowing block with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block.

3. The image processing apparatus according to claim 1, wherein The control processor calculates the value of a weighting function for evaluating swallowing or non-swallowing for each frame, and changes the type of the frame to a swallowing frame or a non-swallowing frame according to the value of the weighting function.

4. An image processing apparatus, which includes a control processor, The control processor performs the following processing: Obtain a check image, For each frame of the check image, classify it into either a plurality of frames classified by swallowing action corresponding to the swallowing action or a non-swallowing frame in which the swallowing is not being performed, When the time interval between adjacent frames classified by swallowing action in time is equal to or less than a first threshold, at least integrate the adjacent frames classified by swallowing action into a swallowing block classified by action.

5. The image processing apparatus according to claim 4, wherein The control processor determines that a frame classified by swallowing action with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a frame classified by swallowing action, or determines that a swallowing block classified by action with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block classified by action.

6. The image processing apparatus according to claim 4, wherein The control processor determines whether the expression pattern including the frame classified by swallowing action or the swallowing block classified by action is either in the middle of swallowing or not swallowing according to the expression pattern.

7. The image processing apparatus according to claim 4, wherein The frames classified by swallowing action include frames before swallowing, frames during swallowing, or frames just after swallowing.

8. The image processing apparatus according to claim 7, wherein The frames before swallowing include frames with jitter before swallowing, epiglottis inversion frames, or food inflow frames.

9. The image processing apparatus according to claim 7, wherein The frames during swallowing include whitening frames, masking frames, flowing frames, colored water inflow frames, or food swallowing frames.

10. The image processing apparatus according to claim 7, wherein The frames just after swallowing include frames with jitter just after swallowing or black screen frames.

11. The image processing apparatus according to claim 4, wherein When the control processor classifies the check image into a plurality of brightness information frames corresponding to brightness, The control processor determines whether the expression pattern is in the process of swallowing or not based on the expression pattern including the brightness information frame, the frame classified by swallowing action, or the swallowing block classified by action.

12. The image processing apparatus according to claim 11, wherein The brightness information frame includes a high brightness frame, a medium brightness frame, or a low brightness frame.

13. The image processing apparatus according to claim 4, wherein When the control processor classifies a frame of the inspection image with jitter below the jitter threshold as a pre-swallowing jitter frame, The control processor determines whether the expression pattern is in the process of swallowing or not based on the expression pattern including the pre-swallowing jitter frame, the frame classified by swallowing action, or the swallowing block classified by action.

14. A working method of an image processing apparatus, which includes the following steps performed by a control processor: Obtain an inspection image; Classify each frame of the inspection image as either a swallowing frame in the process of swallowing or a non-swallowing frame not in the process of swallowing; and When the time interval between temporally adjacent swallowing frames is equal to or less than a first threshold, at least integrate the adjacent swallowing frames into a swallowing block.

15. The working method of the image processing apparatus according to claim 14, wherein The control processor determines that a swallowing frame with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing frame, or determines that a swallowing block with a time width equal to or less than a second threshold or equal to or greater than a third threshold is not a swallowing block.

Images