Deglutition determination system
A camera-based swallowing assessment system addresses the discomfort of throat-attached sensors by accurately determining swallowing through image analysis and machine learning, reducing psychological burden.
Patent Information
- Application Number
- JP2024025130
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-02-22
- Publication Date
- 2025-09-03
AI Technical Summary
Conventional swallowing sensors attached to the throat cause discomfort and psychological burden for individuals with swallowing disorders.
A swallowing assessment system that utilizes a camera to capture images of the throat, performs image analysis to determine swallowing actions, and outputs assessment results, reducing the need for direct contact with the throat.
Reduces psychological burden by determining swallowing through camera-based image analysis, improving accuracy with machine learning, and providing performance effects when swallowing is detected.
Smart Images
Figure 2025128475000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a swallowing determination system that determines whether a subject is swallowing or not. [Background technology]
[0002] Swallowing sensors that detect human swallowing behavior have been developed. For example, a conventional swallowing sensor includes a piezoelectric film sensor with multiple sensing portions along the length of the neck. The piezoelectric film sensors are attached to the skin of the anterior neck at positions within the range of thyroid cartilage movement that occurs with swallowing, and individually output analog signals corresponding to the deformation of the multiple sensing portions. The main body of the swallowing sensor determines whether or not swallowing has occurred based on a displacement signal, which is the low-frequency component of the analog signal (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Publication No. 2022-33367 Summary of the Invention [Problem to be solved by the invention]
[0004] However, because conventional swallowing sensors are attached to the throat (the skin in the anterior neck), some people with swallowing disorders find them uncomfortable and place a psychological burden on them. Therefore, there has been a demand for the development of a swallowing assessment system that can reduce the psychological burden on those being assessed (those with swallowing disorders).
[0005] The present invention has been made in view of the above-mentioned problems, and has an object to provide a swallowing assessment system that can reduce the psychological burden on the person to be assessed. [Means for solving the problem]
[0006] The swallowing assessment system of the present invention comprises a camera image input unit to which camera images of the person to be assessed are input, a assessment target area determination unit that determines the assessment target area that includes the throat of the person to be assessed from the entire area of the camera image, an image analysis unit that performs image analysis processing on the assessment target area of the camera image to determine whether the person to be assessed who is shown in the camera image is swallowing, and a assessment result output unit that outputs the assessment result of the image analysis unit.
[0007] According to this configuration, an image analysis process is performed on a region to be determined that includes the throat of the person to be determined in the camera image of the person to be determined, and it is determined whether the person to be determined in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person to be determined, which reduces the psychological burden on the person to be determined compared to conventional swallowing sensors (which are attached to the throat).
[0008] In addition, the camera image input from the camera image input unit may be shot from any angle or direction as long as it includes the throat of the person being evaluated. However, camera images shot from the front of the person being evaluated are preferred, as shooting from the front makes it possible to more appropriately determine whether the person is swallowing.
[0009] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a frame difference image generation unit that performs frame difference processing on the assessment target area of the camera image to generate a frame difference image consisting of difference pixels between the current frame image and the frame image a predetermined number of frames before; a pixel number calculation unit that calculates the ratio R of the number of difference pixels in the frame difference image to the total number of pixels in the assessment target area; and a swallowing action assessment unit that determines whether the ratio R is higher than a predetermined threshold value Rt, and if the ratio R is higher than the threshold value Rt, determines that the person to be assessed shown in the camera image is swallowing.
[0010] According to this configuration, frame difference processing is performed on the determination target area of the camera image to generate a frame difference image, and whether the subject appearing in the camera image is swallowing or not is determined based on whether the ratio R (=Nd / N) of the number of difference pixels Nd in the frame difference image to the total number of pixels N in the determination target area is higher than a predetermined threshold Rt. In this way, swallowing can be appropriately determined based on the camera image capturing the throat of the subject.
[0011] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the frame difference image and whether or not the person being assessed is performing a swallowing action, and an estimation unit that uses, as input, the frame difference image generated from the assessment target area of the camera image based on the relationship generated by the machine learning unit, and estimates and outputs whether or not the person being assessed who is shown in the camera image is performing a swallowing action.
[0012] According to this configuration, based on the relationship generated by the machine learning unit, it is estimated from the frame difference image generated from the determination target area of the camera image whether the subject of the determination in the camera image is swallowing. In this way, by utilizing estimation by machine learning, it is possible to improve the accuracy of swallowing determination based on the camera image capturing the throat of the subject of the determination.
[0013] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a feature point extraction unit that extracts feature points of the throat included in the assessment target area of the camera image, a movement amount calculation unit that calculates a movement amount D of the feature points of the throat, and a swallowing action assessment unit that determines whether the movement amount D is higher than a predetermined threshold Dt and, if it is determined that the movement amount D is higher than the predetermined threshold Dt, determines that the subject of assessment shown in the camera image is performing a swallowing action.
[0014] According to this configuration, feature points of the throat are extracted from the region to be determined in the camera image, and whether the subject of the determination shown in the camera image is swallowing or not is determined based on whether the amount of movement D of the feature points of the throat is higher than a predetermined threshold Dt. In this way, swallowing can be appropriately determined based on the camera image capturing the throat of the subject of the determination.
[0015] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the movement of throat feature points included in the assessment target area of the camera image and whether or not the person being assessed is performing a swallowing action, and an estimation unit that uses the movement of the throat feature points extracted from the assessment target area of the camera image as input, based on the relationship generated by the machine learning unit, and estimates and outputs whether or not the person being assessed shown in the camera image is performing a swallowing action.
[0016] According to this configuration, based on the relationship generated by the machine learning unit, it is estimated from the movement of the throat feature points extracted from the target area of the camera image whether the subject of the assessment in the camera image is swallowing. In this way, by utilizing estimations by machine learning, it is possible to improve the accuracy of swallowing assessment based on the camera image capturing the throat of the subject of the assessment.
[0017] In addition, the swallowing assessment system of the present invention may include a performance device that provides a predetermined performance effect to the person being assessed, and the assessment result output unit may output a trigger signal to the performance device to provide the performance effect when it is determined that the person being assessed is performing a swallowing action.
[0018] According to this configuration, when it is determined that the subject is swallowing, a trigger signal for producing a performance effect is output to the performance device. As a result, at the timing when swallowing is determined, a performance effect (for example, vibration from bone conduction earphones, sound from a speaker, video image from a display, wind pressure from an electric fan, image projection onto a screen or human body by a projector, light-up effect by moving light sources such as LEDs, etc.) can be generated by the performance device.
[0019] Furthermore, in the swallowing assessment system of the present invention, the assessment target area determination unit may determine the assessment target area that includes the throat of the person being assessed from the entire area of the camera image based on the facial contour, eye position, nose position, and mouth position of the person being assessed that are detected from the camera image.
[0020] With this configuration, the target area of the camera image that includes the throat of the person being assessed is appropriately determined based on the facial contour, eye position, nose position, and mouth position of the person being assessed that are detected from the camera image, thereby making it possible to appropriately assess swallowing based on the camera image of the throat of the person being assessed.
[0021] The method of the present invention is a method executed by a swallowing assessment system, and includes the steps of inputting camera footage of a person to be assessed, determining a target area of the camera footage that includes the throat of the person to be assessed, applying image analysis processing to the target area of the camera footage to determine whether the person to be assessed in the camera footage is swallowing, and outputting the assessment result.
[0022] With this method, as with the above-described system, an image analysis process is performed on a region to be determined that includes the throat of the person being assessed in a camera image of the person being assessed, and it is determined whether the person being assessed in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person being assessed, which reduces the psychological burden on the person being assessed compared to conventional swallowing sensors (which are attached to the throat).
[0023] The program of the present invention is a program executed in a swallowing assessment system, and the program causes the computer of the swallowing assessment system to execute the following processes: inputting camera footage of the person to be assessed; determining a target area of the camera footage that includes the throat of the person to be assessed; performing image analysis on the target area of the camera footage to determine whether the person to be assessed in the camera footage is swallowing; and outputting the assessment result.
[0024] Like the above-mentioned system, this program also performs image analysis processing on a region to be determined that includes the throat of the person being assessed in a camera image of the person being assessed, and determines whether the person being assessed in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person being assessed, which reduces the psychological burden on the person being assessed compared to conventional swallowing sensors (which are attached to the throat). [Effects of the Invention]
[0025] According to the present invention, it is possible to reduce the psychological burden on the person being assessed. [Brief explanation of the drawings]
[0026] [Figure 1] 1 is a block diagram showing a configuration of a swallowing determination system according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of determining a determination target region. [Figure 3] FIG. 10 is a diagram illustrating an example of swallowing determination based on the ratio of the number of difference pixels. [Figure 4] FIG. 10 is a diagram showing an example of swallowing determination that takes into account estimation results obtained by machine learning. [Figure 5] FIG. 3 is a sequence diagram illustrating the operation of the swallowing determination system according to the first embodiment. [Figure 6] FIG. 10 is a block diagram showing the configuration of a swallowing determination system according to a second embodiment. [Figure 7] FIG. 10 is a diagram showing an example of swallowing determination based on the amount of movement of a feature point of the throat. [Figure 8] FIG. 10 is a diagram showing an example of swallowing determination that takes into account estimation results obtained by machine learning. [Figure 9] FIG. 10 is a sequence diagram illustrating the operation of the swallowing determination system according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0027] A swallowing assessment system according to an embodiment of the present invention will be described below with reference to the drawings. In this embodiment, a swallowing assessment system used in a rehabilitation system for people with swallowing disorders will be illustrated. The swallowing assessment system according to this embodiment has a function of assessing swallowing based on a camera image of the throat of a subject taken from the front. These functions are realized by a program stored in a memory area of the swallowing assessment system.
[0028] (First embodiment) The configuration of a swallowing assessment system according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the swallowing assessment system according to this embodiment. As shown in FIG. 1, the swallowing assessment system 1 includes an imaging device 100 and a performance device 101 as external devices. The imaging device 100 is, for example, a camera, and is positioned so that it can capture an image of the person being assessed from the front. The performance device 101 is, for example, a bone conduction earphone, a speaker, a display, a fan, a projector, an LED or other light source, and has the function of providing various performance effects to the person being assessed.
[0029] 1, the swallowing determination system 1 includes, as functional blocks, a camera image input unit 2, a determination target area determination unit 3, an image analysis unit 4, and a determination result output unit 5. Camera image input unit 2 receives as input a camera image captured by an imaging device 100 (camera image of a person to be determined taken from the front).
[0030] As shown in FIG. 2, the determination target area determination unit 3 has a function of determining a determination target area including the throat of the person being determined from the entire area of the camera image. For example, the determination target area determination unit 3 determines a determination target area including the throat of the person being determined from the entire area of the camera image based on the facial contour, eye positions, nose positions, and mouth positions of the person being determined from the camera image. More specifically, the determination target area including the throat of the person being determined is determined from the facial contour, eye positions, nose positions, and mouth positions of the person being determined based on the relative positional relationship between the facial contour, eye positions, nose positions, and mouth positions and the determination target area including the throat. Alternatively, the color of the facial surface may be used as a reference color, and an area of a similar color may be determined as the determination target area including the throat of the person being determined. In this case, the determination target area including the throat of the person being determined can be distinguished from the clothing and collar worn by the person being determined using a color similar to the skin color as a reference.
[0031] The image analysis unit 4 has a function of performing image analysis processing on a determination target area of the camera image to determine whether or not the subject of determination shown in the camera image is swallowing. As shown in Fig. 1, in this embodiment, the image analysis unit 4 has, as functional blocks, a frame difference image generation unit 6, a ratio calculation unit 7, a swallowing action determination unit 8, a machine learning unit 9, and an estimation unit 10.
[0032] The frame difference image generation unit 6 has a function of performing frame difference processing on the determination target area of the camera image to generate a frame difference image composed of difference pixels between the current frame image and the frame image a predetermined number of frames before. A known technique can be used to generate the frame difference image. FIG. 3 is a diagram showing an example of a frame difference image. As shown in FIG. 3, when the subject is not swallowing, the number of difference pixels (shown by black dots in FIG. 3) in the frame difference image is small (see FIG. 3(a)), whereas when the subject is swallowing, the number of difference pixels in the frame difference image is large (see FIG. 3(b)).
[0033] The proportion calculation unit 7 has a function of calculating the total number of pixels N in the determination target region and the number of difference pixels Nd in the frame difference image, and calculating the proportion R of the number of difference pixels in the frame difference image to the total number of pixels in the determination target region. The swallowing movement determination unit 8 then determines whether the proportion R is higher than a predetermined threshold Rt, and if the proportion R is higher than the threshold Rt (e.g., 50%), determines that the subject of determination shown in the camera video is swallowing (see FIG. 4). However, as will be described below, this swallowing movement determination unit 8 also has a function of improving the accuracy of swallowing determination using estimation by machine learning.
[0034] The machine learning unit 9 has a function of analyzing, by machine learning, the relationship between the frame difference image and whether or not the subject is swallowing using predetermined training data. Any method, such as deep learning using a neural network, can be used for this machine learning. For example, in the case of a neural network, the frame difference image of the training data is input to an input layer, and information regarding whether or not the subject is swallowing is output from an output layer. Then, the weighting coefficients between the neurons of the neural network are optimized by supervised learning using training data in which the data input to the input layer and the data output from the output layer are linked.
[0035] The estimation unit 10 has a function of estimating whether or not the subject of judgment shown in the camera video is swallowing, based on the relationship generated by the machine learning unit 9, using as input a frame difference image generated from the judgment target region of the camera video. For example, in the case of the above-mentioned neural network, estimation is performed by inputting the frame difference image generated by the frame difference image generation unit 6 to the input layer and outputting information regarding whether or not the subject of judgment is swallowing from the output layer. Then, even if the ratio R is higher than the threshold value Rt, if the estimation unit 10 estimates that the subject of judgment is not swallowing, the swallowing action determination unit 8 determines that the subject of judgment shown in the camera video is not swallowing (see FIG. 4).
[0036] The determination result output unit 5 has a function of outputting the determination result of the image analysis unit 4. For example, the determination result output unit 5 has a function of outputting a trigger signal to the performance device 101 to perform a performance effect when it is determined that the person to be determined is swallowing.
[0037] The operation of the swallowing determination system 1 configured as above will be described with reference to the sequence diagram of FIG.
[0038] When swallowing assessment is performed using the swallowing assessment system 1 of this embodiment, first, the person to be assessed is photographed from the front using the imaging device 100 (S10). The camera image of the person to be assessed taken from the front is transmitted from the imaging device 100 to the swallowing assessment system 1 (S11). In the swallowing assessment system 1, when the camera image of the person to be assessed taken from the front is input from the imaging device 100 (S12), a target area for assessment that includes the throat of the person to be assessed is determined from the entire area of the camera image, as shown in FIG. 2 (S13), and a frame difference process is performed on the target area of the camera image to generate a frame difference image (see FIG. 3) (S14).
[0039] Next, the swallowing determination system 1 calculates the ratio R (=Nd / N) of the number of difference pixels Nd of the frame difference image to the total number of pixels N of the determination target region (S15), and determines whether the ratio R is higher than a predetermined threshold Rt (S16). Furthermore, the swallowing determination system 1 uses the relationship generated by the machine learning unit 9 to estimate whether the subject of determination shown in the camera video is swallowing or not, based on the frame difference image generated from the determination target region of the camera video (S17).
[0040] 4, based on the comparison result between the ratio R and the threshold value Rt and the estimation result by machine learning, it is determined whether the subject of judgment shown in the camera image is swallowing (S18), and the determination result is output (S19). In this embodiment, when the determination result is output, a trigger signal is transmitted from the swallowing judgment system 1 to the performance device 101 (S20), and various performance effects are given to the subject of judgment by the performance device 101 (S21).
[0041] According to the swallowing assessment system 1 of the first embodiment, an image analysis process is performed on a region to be assessed that includes the throat of the person to be assessed in a camera image taken from the front of the person to be assessed, and it is determined whether the person to be assessed in the camera image is swallowing. In this way, swallowing assessment can be performed based on a camera image taken from the front of the throat of the person to be assessed, which reduces the psychological burden on the person to be assessed compared to conventional swallowing sensors (which are attached to the throat and used).
[0042] In this embodiment, frame difference processing is performed on the determination target area of the camera image to generate a frame difference image, and whether the subject appearing in the camera image is swallowing is determined based on whether the ratio R (=Nd / N) of the number of difference pixels Nd in the frame difference image to the total number of pixels N in the determination target area is higher than a predetermined threshold Rt. In this way, swallowing can be appropriately determined based on the camera image capturing the subject's throat from the front.
[0043] Furthermore, in this embodiment, whether or not the subject of judgment shown in the camera video is swallowing is estimated from a frame difference image generated from the judgment target region of the camera video, based on the relationship generated by the machine learning unit 9. In this way, estimation by machine learning can be used to improve the accuracy of swallowing judgment based on camera video capturing the throat of the subject of judgment from the front.
[0044] Furthermore, in this embodiment, when it is determined that the person being assessed is swallowing, a trigger signal for producing a performance effect is output to performance device 101. As a result, at the timing when swallowing is assessed, performance (for example, vibrations from bone conduction earphones, sound from a speaker, video images from a display, wind pressure from an electric fan, image projection onto a screen or a human body by a projector, light-up performance by moving light-emitting bodies such as LEDs, etc.) can be generated by performance device 101.
[0045] Furthermore, in this embodiment, the region to be determined that includes the throat of the person to be determined is appropriately determined from the entire region of the camera image based on the facial contour, eye position, nose position, and mouth position of the person to be determined that are detected from the camera image. This makes it possible to appropriately determine swallowing based on the camera image that captures the throat of the person to be determined from the front.
[0046] (Second embodiment) Next, a swallowing determination system according to a second embodiment of the present invention will be described. Here, the swallowing determination system according to the second embodiment will be described, focusing on the differences between the system according to the first embodiment. Unless otherwise specified, the configuration and operation of this embodiment are the same as those of the first embodiment.
[0047] Fig. 6 is a block diagram showing the configuration of the swallowing determination system of this embodiment. As shown in Fig. 6, in the swallowing determination system 1 of this embodiment, the image analysis unit 4 includes, as functional blocks, a feature point extraction unit 11, a movement amount calculation unit 12, a swallowing movement determination unit 13, a machine learning unit 14, and an estimation unit 15.
[0048] The feature point extraction unit 11 has a function of extracting throat feature points included in the region to be determined in the camera image. Publicly known techniques can be used to extract the feature points. FIG. 7 is a diagram showing an example of throat feature points. As shown in FIG. 7, when the subject is not swallowing, the throat feature points do not move much (see FIG. 7(a)), whereas when the subject is swallowing, the throat feature points move (see FIG. 7(b)).
[0049] The movement amount calculation unit 12 has a function of calculating the movement amount D of the throat feature point. A known technique can be used to calculate the movement amount of the feature point. Then, the swallowing action determination unit 13 determines whether or not the movement amount D is higher than a predetermined threshold Dt (for example, a change rate of 5% based on the distance to a nearby feature point when stationary), and if it is determined that the movement amount D is higher than the predetermined threshold Dt, it determines that the subject of determination shown in the camera video is swallowing (see FIG. 8). However, even in this embodiment, the swallowing action determination unit 13 has a function of improving the accuracy of swallowing determination by utilizing estimation by machine learning, as described below.
[0050] The machine learning unit 14 has a function of analyzing, by machine learning, the relationship between the movement of throat feature points and whether or not the subject of assessment is performing a swallowing action using predetermined training data. Any method, such as deep learning using a neural network, can be used for this machine learning. For example, in the case of a neural network, the movement of throat feature points in the training data is input to an input layer, and information regarding whether or not the subject of assessment is performing a swallowing action is output from an output layer. Then, the weighting coefficients between the neurons of the neural network are optimized by supervised learning using training data in which the data input to the input layer and the data output from the output layer are linked.
[0051] The estimation unit 15 has a function of estimating whether or not the subject of judgment shown in the camera video is swallowing, based on the relationship generated by the machine learning unit 14, using as input the movement of the throat feature points extracted from the judgment target area of the camera video. For example, in the case of the above-mentioned neural network, the movement of the throat feature points extracted by the feature point extraction unit 11 is input to the input layer, and information regarding whether or not the subject of judgment is swallowing is output from the output layer, thereby making the estimation. Then, even if the amount of movement D is higher than a predetermined threshold Dt, if the estimation unit 15 estimates that the subject of judgment is not swallowing, the swallowing action determination unit 13 determines that the subject of judgment shown in the camera video is not swallowing (see FIG. 8).
[0052] The operation of the swallowing determination system 1 configured as above will be described with reference to the sequence diagram of FIG.
[0053] When swallowing assessment is performed using the swallowing assessment system 1 of this embodiment, as in the first embodiment, first, the person to be assessed is photographed from the front using the imaging device 100 (S10). The camera image of the person to be assessed taken from the front is transmitted from the imaging device 100 to the swallowing assessment system 1 (S11). In the swallowing assessment system 1, when the camera image of the person to be assessed taken from the front is input from the imaging device 100 (S12), a assessment target region that includes the throat of the person to be assessed is determined from the entire region of the camera image (S13), as shown in FIG.
[0054] In the swallowing determination system 1 of this embodiment, throat feature points included in the determination target area of the camera video are extracted (S22), the amount of movement D of the throat feature points is calculated (S23), and it is determined whether or not the amount of movement D is higher than a predetermined threshold Dt (S24). Furthermore, in the swallowing determination system 1, using the relationship generated by the machine learning unit 14, it is estimated whether or not the person to be determined shown in the camera video is swallowing, based on the movement of the throat feature points extracted from the determination target area of the camera video (S25).
[0055] 8, based on the comparison result between the movement amount D and the threshold value Dt and the estimation result by machine learning, it is determined whether the subject of the judgment shown in the camera image is swallowing or not (S26), and the judgment result is output (S27). Also in this embodiment, when the judgment result is output, a trigger signal is transmitted from the swallowing judgment system 1 to the performance device 101 (S20), and various performance effects are given to the subject of the judgment by the performance device 101 (S21).
[0056] The swallowing determination system 1 of the second embodiment also provides the same effects as those of the first embodiment.
[0057] In this embodiment, feature points of the throat are extracted from the region to be determined in the camera image, and whether the subject of the determination shown in the camera image is swallowing or not is determined based on whether the amount of movement D of the feature points of the throat is higher than a predetermined threshold Dt. In this way, swallowing can be appropriately determined based on the camera image capturing the throat of the subject of the determination from the front.
[0058] Furthermore, in this embodiment, whether or not the subject of judgment shown in the camera video is swallowing is estimated from the movement of feature points of the throat extracted from the judgment target area of the camera video, based on the relationship generated by the machine learning unit 14. In this way, estimation by machine learning can be used to improve the accuracy of swallowing judgment based on camera video capturing the throat of the subject of judgment from the front.
[0059] Although the embodiments of the present invention have been described above by way of example, the scope of the present invention is not limited to these, and can be modified and changed according to the purpose within the scope of the claims.
[0060] For example, in the above example, the camera image input from the camera image input unit is a camera image taken from the front of the person to be judged, but the scope of the present invention is not limited to this. The camera image input from the camera image input unit does not necessarily have to be a camera image taken from the front of the person to be judged, as long as it includes the throat of the person to be judged (the angle or direction at which the person to be judged is photographed is not limited). [Industrial Applicability]
[0061] As described above, the swallowing assessment system according to the present invention has the effect of reducing the psychological burden on the person being assessed, and is useful when applied to rehabilitation systems for people with swallowing disorders, etc. [Explanation of symbols]
[0062] 1 Swallowing assessment system 2 Camera video input section 3. Judgment target area determination unit 4. Image analysis section 5. Judgment result output section 6 Frame difference image generation unit 7 Percentage calculation section 8 Swallowing motion determination section 9. Machine Learning Department 10 Estimation part 11 Feature point extraction unit 12. Movement amount calculation section 13 Swallowing motion determination section 14 Machine Learning Department 15 Estimation part 100 Imaging device 101 Production Devices
Claims
1. a camera image input unit to which a camera image of a person to be determined is input; a determination target area determination unit that determines a determination target area including the throat of the person to be determined from the entire area of the camera image; an image analysis unit that performs image analysis processing on a determination target area of the camera image to determine whether the person being determined in the camera image is swallowing; a determination result output unit that outputs the determination result of the image analysis unit; A swallowing determination system comprising:
2. The image analysis unit a frame difference image generation unit that performs frame difference processing on the determination target region of the camera image to generate a frame difference image composed of difference pixels between a current frame image and a frame image a predetermined number of frames before; a ratio calculation unit that calculates a ratio R of the number of difference pixels of the frame difference image to the total number of pixels in the determination target region; a swallowing action determination unit that determines whether the ratio R is higher than a predetermined threshold Rt, and determines that the subject of determination shown in the camera image is performing a swallowing action when the ratio R is higher than the threshold Rt; The swallowing determination system according to claim 1 , comprising:
3. The image analysis unit a machine learning unit that uses predetermined learning data to analyze, by machine learning, the relationship between the frame difference image and whether or not the subject is swallowing; an estimation unit that receives the frame difference image generated from the determination target area of the camera image based on the relationship generated by the machine learning unit, estimates whether the subject of determination captured in the camera image is performing a swallowing action, and outputs the result; The swallowing determination system according to claim 2 , comprising:
4. The image analysis unit a feature point extraction unit that extracts feature points of the throat included in the determination target area of the camera image; a movement amount calculation unit that calculates a movement amount D of the throat feature point; a swallowing action determination unit that determines whether the amount of movement D is higher than a predetermined threshold Dt, and determines that the subject of determination shown in the camera image is swallowing when it is determined that the amount of movement D is higher than the predetermined threshold Dt; The swallowing determination system according to claim 1 , comprising:
5. The image analysis unit a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the movement of feature points of the throat included in the determination target area of the camera image and whether or not the person being determined is swallowing; an estimation unit that estimates whether the subject of judgment shown in the camera image is performing a swallowing action based on the relationship generated by the machine learning unit and outputs the result using the movement of the throat feature points extracted from the judgment target area of the camera image as an input; The swallowing determination system according to claim 3 , comprising:
6. a performance device that provides a predetermined performance effect to the person to be judged, The determination result output unit The swallowing determination system according to claim 1 , further comprising: a trigger signal for producing the performance effect output to the performance device when it is determined that the subject is performing a swallowing action.
7. The judgment target region determination unit The swallowing determination system of claim 1, wherein the determination target area including the throat of the person being determined is determined from the entire area of the camera image based on the facial contour, eye position, nose position, and mouth position of the person being determined detected from the camera image.
8. A method performed in a swallowing determination system, comprising: The method comprises: A step of inputting a camera image of a person to be judged; determining a determination target area including the throat of the person to be determined from the entire area of the camera image; a step of performing an image analysis process on a determination target area of the camera image to determine whether the subject of determination shown in the camera image is swallowing; outputting the determination result; A method comprising:
9. A program executed in a swallowing determination system, The program is installed in a computer of the swallowing determination system. A process of inputting a camera image of a person to be judged; A process of determining a determination target area including the throat of the person to be determined from the entire area of the camera image; A process of performing an image analysis process on a determination target area of the camera image to determine whether the person to be determined in the camera image is swallowing; a process of outputting the determination result; A program that executes.
Citation Information
Patent Citations
Swallowing sensor, swallowing analysis system, and swallowing analysis method
JP2022033367A