Deglutition determination system
Patent Information
- Application Number
- HK62026125437
- Authority / Receiving Office
- HK · HK
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-02-22
- Filing Date
- 2026-06-29
- Publication Date
- 2026-09-11
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202580006043.7 (22) Application Date 2025.01.22 (30) Priority Data 2024-025130 2024.02.22 JP (85) PCT International Application Entering National Phase Date 2026.05.15 (86) PCT International Application Application Data PCT / JP2025 / 001819 2025.01.22 (87) PCT International Application Publication Data WO2025 / 177753 JA 2025.08.28 (71) Applicant Dentsu Inc. Address Tokyo, Japan (72) Inventor Izumi Koki (74) Patent Agency Beijing Huisicheng Intellectual Property Agency Co., Ltd. 11444 Patent Attorney Zhang Li Wang Gang (51) Int.Cl. A61B 5 / 11 (2006.01) G06V 40 / 20 (2022.01) G06V 40 / 16 (2022.01) G06N 20 / 00 (2019.01) (54) Invention Title Swallowing Determination System (57) Abstract The swallowing determination system (1) comprises: a camera image input unit (2) which inputs a camera image obtained by shooting a person to be determined; a person to be determined area determination unit (3) which determines a person to be determined area that includes the throat of the person to be determined in the entire area of the camera image; an image analysis unit (4) which performs image analysis processing on the person to be determined area of the camera image and determines whether the person to be determined reflected in the camera image is swallowing; and a determination result output unit (5) which outputs the determination result of the image analysis unit. Claims (2 pages), Description (9 pages), Drawings (7 pages), CN 122228056 A, 2026.06.16, CN 1 22 22 80 56 A 1. A swallowing determination system, comprising: a camera image input unit that inputs a camera image obtained by capturing a subject; a subject area determination unit that determines a subject area in the entire area of the camera image that includes the throat of the subject; an image analysis unit that performs image analysis processing on the subject area of the camera image to determine whether the subject shown in the camera image is performing a swallowing action; and a determination result output unit that outputs the determination result of the image analysis unit.2. The swallowing determination system according to claim 1, wherein the image parsing unit comprises: an inter-frame difference image generation unit, which performs inter-frame difference processing on the determination object region of the camera image to generate an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before a given frame; a ratio calculation unit, which calculates the ratio R of the number of difference pixels in the inter-frame difference image relative to the total number of pixels in the determination object region; and a swallowing action determination unit, which determines whether the ratio R is higher than a given threshold Rt, and if the ratio R is higher than the threshold Rt, determines that the determination object reflected in the camera image is performing a swallowing action. 3. The swallowing determination system according to claim 2, wherein the image analysis unit comprises: a machine learning unit that uses given learning data to analyze the relationship between inter-frame difference images and whether a person being judged is performing a swallowing action through machine learning; and an estimation unit that, based on the relationship generated by the machine learning unit, uses the inter-frame difference image generated from the judgment object region of the camera image as input to estimate and output whether the person being judged reflected in the camera image is performing a swallowing action. 4. The swallowing determination system according to claim 1, wherein the image analysis unit comprises: a feature point extraction unit that extracts feature points of the pharynx contained in the judgment object region of the camera image; a movement calculation unit that calculates the movement amount D of the feature points of the pharynx; and a swallowing action determination unit that determines whether the movement amount D is higher than a given threshold Dt, and if the movement amount D is determined to be higher than the given threshold Dt, determines that the person being judged reflected in the camera image is performing a swallowing action. 5. The swallowing determination system according to claim 3, wherein the image analysis unit comprises: a machine learning unit that uses given learning data to analyze the relationship between the movement of feature points of the pharynx contained in the determination object region of the camera image and whether the person being determined is performing a swallowing action; and an estimation unit that, based on the relationship generated by the machine learning unit, uses the movement of feature points of the pharynx extracted from the determination object region of the camera image as input to estimate and output whether the person being determined, as reflected in the camera image, is performing a swallowing action. 6. The swallowing determination system according to claim 1, wherein the swallowing determination system comprises a performance device that assigns a given performance effect to the person being determined, and when it is determined that the person being determined is performing a swallowing action, the determination result output unit outputs a trigger signal to the performance device for executing the performance effect.7. The swallowing determination system according to claim 1, wherein the determination object region determining unit determines the determination object region, which includes the throat of the determination object, within the entire region of the camera image, based on the facial contour, eye position, nose position, and mouth position of the determination object detected from the camera image. 8. A method performed by a swallowing determination system, comprising the following steps: inputting a camera image obtained by photographing a determination object; determining a determination object region, which includes the throat of the determination object, within the entire region of the camera image; performing image analysis processing on the determination object region of the camera image to determine whether the determination object reflected in the camera image is performing a swallowing action; and outputting the result of the determination. 9. A program executed by a swallowing determination system, the program causing the computer of the swallowing determination system to perform the following processing: inputting a camera image obtained by photographing a person to be determined; determining a determination object region in the entire area of the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined, as reflected in the camera image, is performing a swallowing action; and outputting the result of the determination. Claims 2 / 2 Page 3 CN 122228056 A Swallowing Determination System Technical Field
[0001] The present invention relates to a swallowing determination system for determining whether a person to be determined is performing a swallowing action. Background Art
[0002] Conventionally, swallowing sensors for detecting a person's swallowing action have been developed. For example, conventional swallowing sensors include a piezoelectric thin film sensor having multiple sensing portions in the length direction of the neck, the piezoelectric thin film sensor being located within the range of movement of the thyroid cartilage that accompanies swallowing and being attached to the skin of the anterior neck, and outputting analog signals that accompanies the deformation of the multiple sensing portions respectively. The main body of the swallowing sensor determines whether swallowing has occurred based on the low-frequency component of the analog signal, i.e., the displacement signal (e.g., Japanese Patent Application Publication No. 2022-33367).
[0003] However, conventional swallowing sensors are attached to the throat (skin of the front of the neck) and are therefore used, which can cause discomfort and psychological burden depending on the person with swallowing difficulties. Therefore, it is desirable to develop a swallowing determination system that can reduce the psychological burden on the person being determined (the person with swallowing difficulties). Summary of the Invention
[0004] Technical Problem to be Solved by the Invention The present invention was made in the above context. The object of the present invention is to provide a swallowing determination system that can reduce the psychological burden on the person being determined.
[0005] Technical Solution for Solving the Technical Problem One aspect of the present invention is a swallowing determination system, which includes: a camera image input unit that inputs a camera image obtained by photographing a person to be determined; a determination object region determination unit that determines a determination object region in the entire region of the camera image that includes the throat of the person to be determined; an image analysis unit that performs image analysis processing on the determination object region of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and a determination result output unit that outputs the determination result of the image analysis unit.
[0006] Another aspect of the present invention is a method performed by the swallowing determination system, comprising the following steps: inputting a camera image obtained by photographing a person to be determined; determining a determination object region in the entire region of the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and outputting a determination result.
[0007] Another aspect of the present invention is a program executed by a swallowing determination system, which causes the computer of the swallowing determination system to perform the following processes: processing inputting a camera image obtained by photographing the person to be determined; processing to determine a determination object region in the entire area of the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and processing to output the determination result.
[0008] As will be explained below, there are other aspects of the present invention. Therefore, the disclosure of the present invention is intended to provide some aspects of the invention and is not intended to limit the scope of the claimed invention described herein. Specification 1 / 9 pages 4 CN 122228056 A Brief Description of the Drawings
[0009] FIG1 is a block diagram showing the configuration of the swallowing determination system in the first embodiment.
[0010] FIG2 is a diagram showing an example of the determination of the determination object region.
[0011] FIG3 is a diagram showing an example of swallowing determination based on the ratio of differential pixel count.
[0012] FIG4 is a diagram showing an example of swallowing determination considering the estimation results based on machine learning.
[0013] FIG5 is a timing diagram illustrating the operation of the swallowing determination system in the first embodiment.
[0014] FIG6 is a block diagram showing the configuration of the swallowing determination system in the second embodiment.
[0015] FIG7 is a diagram showing an example of swallowing determination based on the amount of movement of feature points in the pharynx.
[0016] FIG8 is a diagram showing an example of swallowing determination considering the estimation results based on machine learning.
[0017] FIG9 is a timing diagram illustrating the operation of the swallowing determination system in the second embodiment.Detailed Description of Embodiments
[0018] The present invention will now be described in detail. However, the following detailed description and accompanying drawings are not intended to limit the present invention.
[0019] The swallowing determination system of the present invention includes: a camera image input unit that inputs a camera image obtained by photographing a person to be determined; a determination object area determination unit that determines a determination object area in the entire area of the camera image that includes the throat of the person to be determined; an image analysis unit that performs image analysis processing on the determination object area of the camera image to determine whether the person to be determined reflected in the camera image is swallowing; and a determination result output unit that outputs the determination result of the image analysis unit.
[0020] According to this configuration, image analysis processing is performed on the determination object area in the camera image of the person to be determined that includes the throat of the person to be determined, and it is determined whether the person to be determined reflected in the camera image is swallowing. In this way, swallowing determination can be performed based on a camera image obtained by photographing the throat of the person to be determined, and therefore, compared with conventional swallowing sensors (used by attaching to the throat), the psychological burden on the person to be determined can be reduced.
[0021] It should be noted that the camera image input from the camera image input unit only needs to include the throat of the person being judged. There is no limitation on the angle or direction of shooting the person being judged. However, by shooting from the front, it is possible to more appropriately determine whether a swallowing action is being performed. Therefore, it is preferable to obtain a camera image by shooting the person being judged from the front.
[0022] Furthermore, in the swallowing judgment system of the present invention, the image analysis unit may include: an inter-frame difference image generation unit, which performs inter-frame difference processing on the judgment object area of the camera image to generate an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame; a pixel count calculation unit, which calculates the ratio R of the difference pixel number of the inter-frame difference image relative to the total number of pixels in the judgment object area; and a swallowing action judgment unit, which determines whether the ratio R is higher than a given threshold Rt. If the ratio R is higher than the threshold Rt, it is determined that the person being judged reflected in the camera image is performing a swallowing action.
[0023] According to this configuration, inter-frame differential processing is performed on the target area of the camera image to generate an inter-frame differential image. Based on whether the ratio R (=Nd / N) of the differential pixel number Nd of the inter-frame differential image to the total number of pixels N of the target area is higher than a given threshold Rt, it is determined whether the person being judged, as reflected in the camera image, is performing a swallowing action. In this way, swallowing determination can be appropriately performed based on the camera image obtained by shooting the throat of the person being judged.
[0024] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a machine learning unit that uses learning data specified in the specification (page 2 / 9, CN 122228056 A) to analyze the relationship between inter-frame difference images and whether the subject is swallowing; and an estimation unit that, based on the relationship generated by the machine learning unit, uses an inter-frame difference image generated from the subject area of the camera image as input to estimate and output whether the subject reflected in the camera image is swallowing.
[0025] According to this configuration, based on the relationship generated by the machine learning unit, and based on the inter-frame difference image generated from the subject area of the camera image, it is estimated whether the subject reflected in the camera image is swallowing. In this way, by using estimation based on machine learning, the determination accuracy of swallowing determination based on camera images obtained by shooting the throat of the subject can be improved.
[0026] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a feature point extraction unit that extracts feature points of the pharynx included in the determination object region of the camera image; a movement amount calculation unit that calculates the movement amount D of the feature points of the pharynx; and a swallowing action determination unit that determines whether the movement amount D is higher than a given threshold Dt, and if it is determined that the movement amount D is higher than the given threshold Dt, it determines that the determination object shown in the camera image is performing a swallowing action.
[0027] According to this configuration, feature points of the pharynx are extracted from the determination object region of the camera image, and it is determined whether the determination object shown in the camera image is performing a swallowing action based on whether the movement amount D of the feature points of the pharynx is higher than a given threshold Dt. In this way, swallowing determination can be appropriately performed based on the camera image obtained by shooting the pharynx of the determination object.
[0028] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a machine learning unit that uses given learning data to analyze the relationship between the movement of feature points of the pharynx included in the determination object region of the camera image and whether the person being determined is swallowing; and an estimation unit that, based on the relationship generated by the machine learning unit, uses the movement of feature points of the pharynx extracted from the determination object region of the camera image as input to estimate and output whether the person being determined, as reflected in the camera image, is swallowing.
[0029] According to this configuration, based on the relationship generated by the machine learning unit, and based on the movement of feature points of the pharynx extracted from the determination object region of the camera image, it is estimated whether the person being determined, as reflected in the camera image, is swallowing. In this way, by using estimation based on machine learning, the determination accuracy of swallowing determination based on camera images obtained by photographing the pharynx of the person being determined can be improved.
[0030] Furthermore, the swallowing determination system of the present invention can include a performance device that assigns a given performance effect to the person being determined. When it is determined that the person being determined is swallowing, the determination result output unit can output a trigger signal to the performance device for executing the performance effect.
[0031] According to this configuration, if it is determined that the person being determined is swallowing, a trigger signal for executing the performance effect is output to the performance device. Thus, at the moment the swallowing determination is completed, the performance generated by the performance device can be triggered (e.g., vibration generated by bone conduction headphones, sound played by a speaker, video image displayed on a monitor, wind pressure generated by an electric fan, image projection from a projector onto a screen or human body, light show achieved by moving a light source such as an LED, etc.).
[0032] In addition, in the swallowing determination system of the present invention, the determination object area determination unit can determine a determination object area that includes the throat of the person being determined in the entire area of the camera image, based on the facial contour, eye position, nose position, and mouth position of the person being determined detected from the camera image.
[0033] According to this configuration, based on the facial contour, eye position, nose position, and mouth position of the person to be judged detected from the camera image, a judgment object region containing the throat of the person to be judged is appropriately determined in the entire area of the camera image. Thus, swallowing judgment can be appropriately performed based on the camera image obtained by photographing the throat of the person to be judged (see specification page 3 / 9, CN 122228056 A).
[0034] The method of the present invention is a method executed by a swallowing judgment system, comprising the following steps: inputting a camera image obtained by photographing the person to be judged; determining a judgment object region containing the throat of the person to be judged in the entire area of the camera image; performing image analysis processing on the judgment object region of the camera image to determine whether the person to be judged reflected in the camera image is performing a swallowing action; and outputting the judgment result.
[0035] According to this method, similarly to the system described above, image analysis processing is performed on the target area, which includes the throat of the target person, in the camera image captured by the target person, to determine whether the target person shown in the camera image is swallowing. In this way, swallowing determination can be performed based on the camera image obtained by capturing the throat of the target person, thus reducing the psychological burden on the target person compared to conventional swallowing sensors (which are attached to the throat).
[0036] The program of the present invention is executed by a swallowing determination system, which causes the computer of the swallowing determination system to perform the following processes: inputting a camera image obtained by photographing the person to be determined; determining a determination object area in the entire area of the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object area of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and outputting the determination result.
[0037] According to this program, similarly to the system described above, image analysis processing is performed on the determination object area in the camera image of the person to be determined that includes the throat of the person to be determined, and it is determined whether the person to be determined reflected in the camera image is performing a swallowing action. In this way, swallowing determination can be performed based on a camera image obtained by photographing the throat of the person to be determined, and therefore, compared with conventional swallowing sensors (used by attaching to the throat), the psychological burden on the person to be determined can be reduced.
[0038] According to the present invention, the psychological burden on the person to be determined can be reduced.
[0039] (Embodiment) Hereinafter, a swallowing determination system according to an embodiment of the present invention will be described using the accompanying drawings. In this embodiment, a swallowing determination system used in a rehabilitation system for people with swallowing disorders is illustrated. The swallowing determination system of this embodiment has the function of performing swallowing determination based on a camera image obtained by photographing the throat of the person being determined from the front. These functions are implemented by a program stored in the memory area of the swallowing determination system, etc.
[0040] (First Embodiment) The configuration of the swallowing determination system of the first embodiment of the present invention will be described with reference to the accompanying drawings. FIG1 is a block diagram showing the configuration of the swallowing determination system of this embodiment. As shown in FIG1, the swallowing determination system 1, as an external device, includes a camera 100 and a performance device 101. The camera 100 is, for example, a camera, configured to photograph the person being determined from the front. The performance device 101 is, for example, a bone conduction headphone, a speaker, a display, a fan, a projector, an LED, or other light-emitting body, and has the function of imparting various performance effects to the person being determined.
[0041] Furthermore, as shown in FIG1, the swallowing determination system 1, as a functional module, includes a camera image input unit 2, a determination object area determination unit 3, an image analysis unit 4, and a determination result output unit 5. The camera image input unit 2 receives a camera image captured by the shooting device 100 (a camera image obtained by shooting the determination object from the front).
[0042] As shown in FIG2, the determination object area determination unit 3 has the function of determining a determination object area that includes the throat of the determination object within the entire area of the camera image.For example, the object region determination unit 3 determines the object region containing the throat of the person being judged within the entire area of the camera image, based on the facial contour, eye position, nose position, and mouth position of the person being judged detected from the camera image. More specifically, based on the relative positional relationship between the facial contour, eye position, nose position, mouth position and the object region containing the throat, the object region containing the throat of the person being judged is determined according to the facial contour, eye position, nose position, and mouth position of the person being judged. Alternatively, the color of the facial surface can be set as the reference color, and the part of the color similar to it can be determined as the object region containing the throat of the person being judged. In this case, the object region containing the throat of the person being judged and the clothes and collar worn by the person being judged can be identified based on the color similar to the skin color.
[0043] The image analysis unit 4 has the function of performing image analysis processing on the object region of the camera image to determine whether the person being judged reflected in the camera image is performing a swallowing action. As shown in Figure 1, in this embodiment, the image parsing unit 4 includes an inter-frame difference image generation unit 6, a ratio calculation unit 7, a swallowing action determination unit 8, a machine learning unit 9, and an estimation unit 10 as functional modules.
[0044] The inter-frame difference image generation unit 6 has the function of performing inter-frame difference processing on the determination object area of the camera image and generating an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame. Known techniques can be used in the generation of the inter-frame difference image. Figure 3 is a diagram showing an example of an inter-frame difference image. As shown in Figure 3, when the determination object does not perform a swallowing action, the number of difference pixels (shown as black dots in Figure 3) in the inter-frame difference image is small (see Figure 3(a)), and when the determination object performs a swallowing action, the number of difference pixels in the inter-frame difference image increases (see Figure 3(b)).
[0045] The ratio calculation unit 7 has the function of calculating the total number of pixels N of the target area and the number of differential pixels Nd of the inter-frame difference image, and calculating the ratio R of the number of differential pixels of the inter-frame difference image to the total number of pixels of the target area. Then, the swallowing action determination unit 8 determines whether the ratio R is higher than a given threshold Rt. If the ratio R is higher than the threshold Rt (e.g., 50%), it determines that the target person reflected in the camera image is swallowing (see FIG4). However, as will be explained below, the swallowing action determination unit 8 has the function of improving the accuracy of swallowing determination by using machine learning-based estimation.
[0046] The machine learning unit 9 has the function of using given learning data to analyze the relationship between the inter-frame difference image and whether the target person is swallowing by machine learning.In this machine learning, any method such as deep learning based on neural networks is used. For example, if it is a neural network, it is configured to input the inter-frame difference image of the learning data into the input layer and output information related to whether the person being judged is swallowing from the output layer. Moreover, by using supervised learning of the learning data that establishes a correlation between the data input to the input layer and the data output from the output layer, the weighting coefficients between the neurons of the neural network are optimized.
[0047] The estimation unit 10 has the following function: based on the relationship generated by the machine learning unit 9, it uses the inter-frame difference image generated from the judgment object region of the camera image as input, estimates whether the person being judged reflected in the camera image is swallowing and outputs the result. For example, if it is the above-mentioned neural network, it inputs the inter-frame difference image generated by the inter-frame difference image generation unit 6 into the input layer and outputs information related to whether the person being judged is swallowing from the output layer, thereby making an estimation. Furthermore, even when the proportion R is higher than the threshold Rt, the swallowing action determination unit 8 determines that the person being judged in the camera image has not performed a swallowing action when the estimation unit 10 estimates that the person being judged has not performed a swallowing action (see Figure 4).
[0048] The determination result output unit 5 has the function of outputting the determination result of the image analysis unit 4. For example, the determination result output unit 5 has the function of outputting a trigger signal for performing a performance effect to the performance equipment 101 when it is determined that the person being judged has performed a swallowing action.
[0049] Regarding the swallowing determination system 1 configured as described above, its operation will be explained with reference to the timing diagram of Figure 5. Specification 5 / 9 pages 8 CN 122228056 A
[0050] When performing swallowing determination using the swallowing determination system 1 of this embodiment, firstly, the camera device 100 takes a picture of the person being judged from the front (S10). The camera image obtained by taking a picture of the person being judged from the front is sent from the camera device 100 to the swallowing determination system 1 (S11). In the swallowing determination system 1, if a camera image obtained by shooting the subject from the front is input from the shooting device 100 (S12), as shown in FIG2, a determination object region containing the throat of the subject is determined in the entire area of the camera image (S13), and inter-frame difference processing is performed on the determination object region of the camera image to generate an inter-frame difference image (see FIG3) (S14).
[0051] Next, in the swallowing determination system 1, the ratio R (=Nd / N) of the difference pixel number Nd of the inter-frame difference image to the total number of pixels N of the determination object region is calculated (S15), and it is determined whether the ratio R is higher than a given threshold Rt (S16).Furthermore, in the swallowing determination system 1, using the relationship generated by the machine learning unit 9, based on the inter-frame difference image generated from the determination object region of the camera image, it is estimated whether the determination object reflected in the camera image is performing a swallowing action (S17).
[0052] Then, as shown in FIG4, based on the comparison result of the ratio R and the threshold Rt and the estimation result of the machine learning, it is determined whether the determination object reflected in the camera image is performing a swallowing action (S18), and the determination result is output (S19). In this embodiment, if the determination result is output, a trigger signal is sent from the swallowing determination system 1 to the performance device 101 (S20), and various performance effects are given to the determination object by the performance device 101 (S21).
[0053] According to the swallowing determination system 1 of such a first embodiment, image analysis processing is performed on the determination object region containing the throat of the determination object in the camera image obtained by shooting the determination object from the front, and it is determined whether the determination object reflected in the camera image is performing a swallowing action. In this way, swallowing determination can be performed based on camera images obtained by shooting the throat of the person being judged from the front, thus reducing the psychological burden on the person being judged compared to conventional swallowing sensors (which are attached to the throat).
[0054] In this embodiment, inter-frame difference processing is performed on the judgment area of the camera image to generate an inter-frame difference image. Based on whether the ratio R (=Nd / N) of the difference pixel number Nd of the inter-frame difference image to the total number of pixels N of the judgment area is higher than a given threshold Rt, it is determined whether the person being judged reflected in the camera image is performing a swallowing action. In this way, swallowing determination can be performed appropriately based on camera images obtained by shooting the throat of the person being judged from the front.
[0055] Furthermore, in this embodiment, based on the relationship generated by the machine learning unit 9, it is estimated whether the person being judged reflected in the camera image is performing a swallowing action based on the inter-frame difference image generated from the judgment area of the camera image. Thus, by utilizing machine learning-based inference, the accuracy of swallowing determination based on camera images obtained from frontal shots of the subject's throat can be improved.
[0056] Furthermore, in this embodiment, if it is determined that the subject is swallowing, a trigger signal for executing a performance effect is output to the performance device 101. Therefore, at the moment the swallowing determination is completed, the performance generated by the performance device 101 can be triggered (e.g., vibrations generated by bone conduction headphones, sound played by speakers, video images displayed on a monitor, wind pressure generated by a fan, image projection from a projector onto a screen or human body, light shows achieved by moving light-emitting bodies such as LEDs, etc.).
[0057] Furthermore, in this embodiment, based on the facial contour, eye position, nose position, and mouth position of the person to be judged detected from the camera image, a judgment object area including the throat of the person to be judged is appropriately determined in the entire area of the camera image. Therefore, swallowing determination can be appropriately performed based on the camera image obtained by shooting the throat of the person to be judged from the front.
[0058] (Second Embodiment) Specification Pages 6 / 9 CN 122228056 A Next, the swallowing determination system of the second embodiment of the present invention will be described. Here, the description will focus on the differences between the swallowing determination system of the second embodiment and the first embodiment. Here, unless otherwise specifically mentioned, the configuration and operation of this embodiment are the same as those of the first embodiment.
[0059] FIG6 is a block diagram showing the configuration of the swallowing determination system of this embodiment. As shown in FIG6, in the swallowing determination system 1 of this embodiment, the image analysis unit 4 includes a feature point extraction unit 11, a movement calculation unit 12, a swallowing action determination unit 13, a machine learning unit 14, and an estimation unit 15 as functional modules.
[0060] The feature point extraction unit 11 has the function of extracting feature points of the pharynx included in the determination object area of the camera image. The feature point extraction can use known techniques. FIG7 is a diagram showing an example of feature points of the pharynx. As shown in FIG7, when the determination object does not perform a swallowing action, the feature points of the pharynx do not move much (see FIG7(a)), while when the determination object performs a swallowing action, the feature points of the pharynx move (see FIG7(b)).
[0061] The movement calculation unit 12 has the function of calculating the movement amount D of the feature points of the pharynx. The calculation of the movement amount of the feature points can use known techniques. Then, the swallowing action determination unit 13 determines whether the movement amount D is higher than a given threshold Dt (e.g., a change rate of 5% based on the distance to the nearest feature point when stationary). If the movement amount D is determined to be higher than the given threshold Dt, it is determined that the person being judged, as shown in the camera image, is swallowing (see Figure 8). However, in this embodiment, the swallowing action determination unit 13 also has the function of improving the accuracy of swallowing determination by using machine learning-based estimation, as described below.
[0062] The machine learning unit 14 has the function of using given learning data to analyze the relationship between the movement of feature points of the throat and whether the person being judged is swallowing. In this machine learning, any method such as deep learning based on neural networks is used. For example, if it is a neural network, it is configured to input the movement of feature points of the throat in the learning data into the input layer and output information related to whether the person being judged is swallowing from the output layer.Furthermore, by using supervised learning data that establishes a correlation between the data input to the input layer and the data output from the output layer, the weighting coefficients between neurons in the neural network are optimized.
[0063] The estimation unit 15 has the following function: based on the relationship generated by the machine learning unit 14, it uses the motion of the feature points of the throat extracted from the judgment object region of the camera image as input, and estimates and outputs whether the judgment object reflected in the camera image is performing a swallowing action. For example, if it is the above-mentioned neural network, the motion of the feature points of the throat extracted by the feature point extraction unit 11 is input to the input layer, and information related to whether the judgment object is performing a swallowing action is output from the output layer, thereby making an estimation. Moreover, even if the movement amount D is higher than the given threshold Dt, the swallowing action determination unit 13 determines that the judgment object reflected in the camera image is not performing a swallowing action if the estimation unit 15 estimates that the judgment object is not performing a swallowing action (see FIG8).
[0064] Regarding the swallowing determination system 1 configured as described above, its operation will be explained with reference to the timing diagram of FIG9.
[0065] When performing swallowing determination using the swallowing determination system 1 of this embodiment, similarly to the first embodiment, firstly, the imaging device 100 captures a frontal image of the person to be determined (S10). The camera image of the person to be determined captured from the front is sent from the imaging device 100 to the swallowing determination system 1 (S11). In the swallowing determination system 1, if the camera image of the person to be determined captured from the front is input from the imaging device 100 (S12), then as shown in FIG2, a determination object region containing the throat of the person to be determined is determined within the entire area of the camera image (S13).
[0066] In the swallowing determination system 1 of this embodiment, feature points of the throat contained in the determination object region of the camera image are extracted (S22), the movement amount D of the feature points is calculated (S23), and it is determined whether the movement amount D is higher than a given threshold Dt (S24). Furthermore, in the swallowing determination system 1, using the relationship generated by the machine learning unit 14, based on the movement of feature points of the pharynx extracted from the determination object area of the camera image, it is estimated whether the determination object reflected in the camera image is performing a swallowing action (S25).
[0067] Then, as shown in FIG8, based on the comparison result of the movement amount D and the threshold Dt and the estimation result of the machine learning, it is determined whether the determination object reflected in the camera image is performing a swallowing action (S26), and the determination result is output (S27). In this embodiment, if the determination result is output, a trigger signal is sent from the swallowing determination system 1 to the performance device 101 (S20), and various performance effects are given to the determination object by the performance device 101 (S21).
[0068] The swallowing determination system 1 according to this second embodiment also has the same effect as the first embodiment.
[0069] In this embodiment, feature points of the pharynx are extracted from the determination object area of the camera image, and it is determined whether the determination object reflected in the camera image is swallowing based on whether the movement amount D of the feature points of the pharynx is higher than a given threshold Dt. In this way, swallowing determination can be performed appropriately based on the camera image obtained by shooting the pharynx of the determination object from the front.
[0070] Furthermore, in this embodiment, based on the relationship generated by the machine learning unit 14, it is estimated whether the determination object reflected in the camera image is swallowing based on the movement of the feature points of the pharynx extracted from the determination object area of the camera image. In this way, by using machine learning-based estimation, the determination accuracy of swallowing determination based on the camera image obtained by shooting the pharynx of the determination object from the front can be improved.
[0071] The embodiments of the present invention have been described above by way of example, but the scope of the present invention is not limited thereto, and modifications and variations can be made according to the purpose within the scope of the claims.
[0072] For example, in the above example, the camera image input from the camera image input unit is an example of a camera image obtained by shooting the subject from the front, but the scope of the present invention is not limited to this. The camera image input from the camera image input unit only needs to include an image of the subject's throat, and it does not necessarily have to be a camera image obtained by shooting the subject from the front (the angle and direction of shooting the subject are not limited).
[0073] As described above, the swallowing determination system of the present invention has the effect of reducing the psychological burden on the subject, and is applicable to rehabilitation systems for people with swallowing disorders, etc., and is practical.
[0074] Explanation of reference numerals 1 Swallowing determination system 2 Camera image input unit 3 Determination of determination object area unit 4 Image analysis unit 5 Determination result output unit 6 Inter-frame difference image generation unit 7 Proportion calculation unit 8 Swallowing action determination unit 9 Machine learning unit 10 Estimation unit 11 Feature point extraction unit 12 Movement calculation unit 13 Swallowing action determination unit 14 Machine learning unit instruction manual 8 / 9 pages 11 CN 122228056 A 15 Estimation unit 100 Shooting device 101 Performance equipment.Instruction manual page 9 / 9, 12 CN 122228056 A, Figure 1; Instruction manual figure 1 / 7, 13 CN 122228056 A, Figure 2; Instruction manual figure 2 / 7, 14 CN 122228056 A, Figure 3; Figure 4; Instruction manual figure 3 / 7, 15 CN 122228056 A, Figure 5; Instruction manual figure 4 / 7, 16 CN 122228056 A, Figure 6; Instruction manual figure 5 / 7, 17 CN 122228056 A, Figure 7; Instruction manual figure 6 / 7, 18 CN 122228056 A, Figure 8; Figure 9; Instruction manual figure 7 / 7, 19 CN 122228056 A.
Claims
1. A swallowing determination system, comprising: The camera image input unit receives camera images obtained by capturing images of the subject being judged. The object region determination unit determines the object region, which includes the throat of the person being determined, within the entire area of the camera image. An image analysis unit performs image analysis processing on the target area of the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; and The determination result output unit outputs the determination result of the image analysis unit.
2. The swallowing determination system according to claim 1, wherein, The image parsing unit includes: The inter-frame difference image generation unit performs inter-frame difference processing on the target area of the camera image to generate an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame. The ratio calculation unit calculates the ratio R of the number of difference pixels in the inter-frame difference image relative to the total number of pixels in the determination object region; as well as The swallowing action determination unit determines whether the ratio R is higher than a given threshold Rt. If the ratio R is higher than the threshold Rt, it determines that the person being determined and reflected in the camera image is performing a swallowing action.
3. The swallowing determination system according to claim 2, wherein, The image parsing unit includes: The Machine Learning Department uses given learning data to analyze the relationship between inter-frame difference images and whether the subject is swallowing through machine learning. as well as The estimation unit, based on the relationship generated by the machine learning unit, takes the inter-frame difference image generated from the determination object region of the camera image as input, and estimates and outputs whether the determination object reflected in the camera image is performing a swallowing action.
4. The swallowing determination system according to claim 1, wherein, The image parsing unit includes: The feature point extraction unit extracts feature points of the throat area contained in the target area of the camera image; The movement calculation unit calculates the movement D of the feature points of the pharynx; as well as The swallowing action determination unit determines whether the movement amount D is higher than a given threshold Dt. If the movement amount D is higher than the given threshold Dt, the unit determines that the person being judged, as reflected in the camera image, is performing a swallowing action.
5. The swallowing determination system according to claim 3, wherein, The image parsing unit includes: The Machine Learning Department uses given learning data to analyze the relationship between the movement of feature points of the throat contained in the target area of the camera image and whether the target is swallowing. as well as The estimation unit, based on the relationship generated by the machine learning unit, takes the motion of feature points of the throat extracted from the target area of the camera image as input to estimate and output whether the target person reflected in the camera image is swallowing.
6. The swallowing determination system according to claim 1, wherein, The swallowing determination system is equipped with performance equipment that can assign a given performance effect to the person being determined. When the person identified as the subject of the judgment is performing a swallowing action, the judgment result output unit outputs a trigger signal to the performance equipment to execute the performance effect.
7. The swallowing determination system according to claim 1, wherein, The determination object region unit uses the facial contour, eye position, nose position, and mouth position of the person being determined detected from the camera image as a reference to determine the determination object region, which includes the throat of the person being determined, within the entire area of the camera image.
8. A method performed by a swallowing determination system, comprising the following steps: The steps for inputting camera images obtained by capturing images of the target being identified; The step of determining the target area, which includes the throat of the person being judged, within the entire area of the camera image; The step of performing image analysis processing on the target area of the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; as well as The step of outputting the result of the determination.
9. A program executed by a swallowing determination system, the program causing the computer of the swallowing determination system to perform the following processes: Input: Processing of camera images obtained by capturing images of the target; Processing to determine the target area that includes the throat of the person being judged, within the entire area of the camera image; Image analysis processing is performed on the target area of the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; as well as The processing of outputting the results of the determination.
