Deglutition determination system

Abstract

A deglutition determination system (1) comprises: a camera video input unit (2) to which a camera video resulting from imaging a determination subject is inputted; a determination target region determination unit (3) that determines, from among the entire region of the camera video, a determination target region including the throat of the determination subject; an image analysis unit (4) that subjects the determination target region of the camera video to image analysis processing, and determines whether the determination subject appearing in the camera video is performing a deglutition action; and a determination result output unit (5) that outputs the determination result from the image analysis unit.

Description

Swallowing assessment system

[0001] The present invention relates to a swallowing determination system that determines whether a subject is swallowing or not.

[0002] Swallowing sensors that detect swallowing movements have been developed. For example, a conventional swallowing sensor includes a piezoelectric film sensor with multiple sensing portions along the length of the neck. The piezoelectric film sensor is attached to the skin of the anterior neck at a position within the range of thyroid cartilage movement that occurs with swallowing, and outputs analog signals corresponding to the deformation of the multiple sensing portions. The main body of the swallowing sensor determines whether or not a person has swallowed based on a displacement signal, which is a low-frequency component of the analog signal (see, for example, Japanese Patent Application Laid-Open No. 2022-33367).

[0003] However, because conventional swallowing sensors are attached to the throat (the skin in the anterior neck), some people with swallowing disorders may find them uncomfortable and place a psychological burden on them. Therefore, there has been a demand for the development of a swallowing assessment system that can reduce the psychological burden on those being assessed (those with swallowing disorders).

[0004] The present invention has been made under the above-mentioned circumstances, and an object of the present invention is to provide a swallowing assessment system that can reduce the psychological burden on a subject to be assessed.

[0005] One aspect of the present invention is a swallowing assessment system, which includes a camera image input unit that receives camera footage of the person being assessed, a target area determination unit that determines the target area of ​​the camera image that includes the throat of the person being assessed, an image analysis unit that performs image analysis processing on the target area of ​​the camera image to determine whether the person being assessed in the camera image is swallowing, and a assessment result output unit that outputs the assessment result of the image analysis unit.

[0006] Another aspect of the present invention is a method executed in a swallowing assessment system, which includes the steps of inputting camera footage of a person to be assessed, determining a target area of ​​the camera footage that includes the throat of the person to be assessed, performing image analysis processing on the target area of ​​the camera footage to determine whether the person to be assessed shown in the camera footage is swallowing, and outputting the assessment result.

[0007] Another aspect of the present invention is a program executed in a swallowing assessment system, which causes the computer of the swallowing assessment system to perform the following processes: inputting camera footage of the person being assessed; determining a target area of ​​the camera footage that includes the throat of the person being assessed; performing image analysis on the target area of ​​the camera footage to determine whether the person being assessed in the camera footage is swallowing; and outputting the assessment result.

[0008] As will be described below, there are other aspects of the present invention, and therefore, this disclosure is intended to provide some aspects of the present invention and is not intended to limit the scope of the invention as described and claimed herein.

[0009] FIG. 1 is a block diagram showing the configuration of a swallowing determination system according to a first embodiment. FIG. 2 is a diagram showing an example of determining a determination target region. FIG. 3 is a diagram showing an example of swallowing determination based on the ratio of the number of difference pixels. FIG. 4 is a diagram showing an example of swallowing determination taking into account an estimation result by machine learning. FIG. 5 is a sequence diagram for explaining the operation of the swallowing determination system according to the first embodiment. FIG. 6 is a block diagram showing the configuration of a swallowing determination system according to a second embodiment. FIG. 7 is a diagram showing an example of swallowing determination based on the amount of movement of feature points in the throat. FIG. 8 is a diagram showing an example of swallowing determination taking into account an estimation result by machine learning. FIG. 9 is a sequence diagram for explaining the operation of the swallowing determination system according to the second embodiment.

[0010] DETAILED DESCRIPTION OF THE INVENTION The present invention is described in detail below. However, the following detailed description and the accompanying drawings do not limit the invention.

[0011] The swallowing assessment system of the present invention comprises a camera image input unit to which camera images of the person to be assessed are input, a target area determination unit that determines the target area of ​​the camera image that includes the throat of the person to be assessed, an image analysis unit that performs image analysis processing on the target area of ​​the camera image to determine whether the person to be assessed shown in the camera image is swallowing, and a assessment result output unit that outputs the assessment result of the image analysis unit.

[0012] According to this configuration, an image analysis process is performed on a region of the camera image of the person being evaluated that includes the throat of the person being evaluated, and it is determined whether the person being evaluated in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person being evaluated, which reduces the psychological burden on the person being evaluated compared to conventional swallowing sensors (which are attached to the throat).

[0013] In addition, the camera image input from the camera image input unit may be shot from any angle or direction as long as it includes the throat of the person being evaluated. However, camera images shot from the front of the person being evaluated are preferred, as shooting from the front makes it possible to more appropriately determine whether the person is swallowing.

[0014] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a frame difference image generation unit that performs frame difference processing on the assessment target area of ​​the camera image to generate a frame difference image composed of difference pixels between the current frame image and the frame image a predetermined number of frames before; a pixel number calculation unit that calculates the ratio R of the number of difference pixels in the frame difference image to the total number of pixels in the assessment target area; and a swallowing action assessment unit that determines whether the ratio R is higher than a predetermined threshold value Rt, and if the ratio R is higher than the threshold value Rt, determines that the person being assessed shown in the camera image is swallowing.

[0015] According to this configuration, frame difference processing is performed on the determination target area of ​​the camera image to generate a frame difference image, and whether the subject appearing in the camera image is swallowing or not is determined based on whether the ratio R (=Nd / N) of the number of difference pixels Nd in the frame difference image to the total number of pixels N in the determination target area is higher than a predetermined threshold Rt. In this way, swallowing can be appropriately determined based on the camera image capturing the throat of the subject.

[0016] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the frame difference image and whether or not the person being assessed is performing a swallowing action, and an estimation unit that uses, as input, a frame difference image generated from the area to be assessed in the camera image based on the relationship generated by the machine learning unit, and estimates and outputs whether or not the person being assessed shown in the camera image is performing a swallowing action.

[0017] According to this configuration, based on the relationship generated by the machine learning unit, it is estimated from the frame difference image generated from the determination target area of ​​the camera image whether the subject of the determination in the camera image is swallowing. In this way, by utilizing estimation by machine learning, it is possible to improve the accuracy of swallowing determination based on the camera image capturing the throat of the subject of the determination.

[0018] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a feature point extraction unit that extracts feature points of the throat included in the assessment target area of ​​the camera image, a movement amount calculation unit that calculates the movement amount D of the feature points of the throat, and a swallowing action assessment unit that determines whether the movement amount D is higher than a predetermined threshold Dt and, if it is determined that the movement amount D is higher than the predetermined threshold Dt, determines that the person to be assessed shown in the camera image is performing a swallowing action.

[0019] According to this configuration, feature points of the throat are extracted from the region to be determined in the camera image, and whether the subject of the determination shown in the camera image is swallowing or not is determined based on whether the amount of movement D of the feature points of the throat is higher than a predetermined threshold Dt. In this way, swallowing can be appropriately determined based on the camera image capturing the throat of the subject of the determination.

[0020] Furthermore, in the swallowing assessment system of the present invention, the image analysis unit may include a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the movement of throat feature points included in the assessment target area of ​​the camera image and whether or not the person being assessed is performing a swallowing action, and an estimation unit that uses the movement of the throat feature points extracted from the assessment target area of ​​the camera image as input, based on the relationship generated by the machine learning unit, and estimates and outputs whether or not the person being assessed shown in the camera image is performing a swallowing action.

[0021] According to this configuration, based on the relationship generated by the machine learning unit, it is estimated from the movement of the throat feature points extracted from the target area of ​​the camera image whether the subject of the assessment in the camera image is swallowing. In this way, by utilizing estimations by machine learning, it is possible to improve the accuracy of swallowing assessment based on the camera image capturing the throat of the subject of the assessment.

[0022] In addition, the swallowing assessment system of the present invention may include a performance device that provides a predetermined performance effect to the person being assessed, and the assessment result output unit may output a trigger signal to the performance device to provide the performance effect when it is determined that the person being assessed is performing a swallowing action.

[0023] According to this configuration, when it is determined that the subject is swallowing, a trigger signal for producing a performance effect is output to the performance device, which allows the performance device to generate a performance effect (for example, vibrations from bone conduction earphones, sound from a speaker, video images from a display, wind pressure from an electric fan, image projection onto a screen or the human body by a projector, light-up effects by moving light sources such as LEDs, etc.) at the timing when swallowing is determined.

[0024] In addition, in the swallowing assessment system of the present invention, the assessment target area determination unit may determine the assessment target area that includes the throat of the person being assessed from the entire area of ​​the camera image based on the facial contour, eye position, nose position, and mouth position of the person being assessed that are detected from the camera image.

[0025] With this configuration, the target area of ​​the camera image that includes the throat of the person being assessed is appropriately determined based on the facial contour, eye position, nose position, and mouth position of the person being assessed that are detected from the camera image, thereby making it possible to appropriately assess swallowing based on the camera image of the throat of the person being assessed.

[0026] The method of the present invention is a method executed by a swallowing assessment system, and includes the steps of inputting camera footage of the person to be assessed, determining a target area of ​​the camera footage that includes the throat of the person to be assessed, performing image analysis processing on the target area of ​​the camera footage to determine whether the person to be assessed shown in the camera footage is swallowing, and outputting the assessment result.

[0027] With this method, as with the above-described system, an image analysis process is performed on a region of the camera image of the person being evaluated that includes the throat of the person being evaluated, and it is determined whether the person being evaluated in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person being evaluated, which reduces the psychological burden on the person being evaluated compared to conventional swallowing sensors (which are attached to the throat).

[0028] The program of the present invention is a program executed in a swallowing assessment system, and causes the computer of the swallowing assessment system to perform the following processes: inputting camera footage of the person being assessed; determining a target area of ​​the camera footage that includes the throat of the person being assessed; performing image analysis on the target area of ​​the camera footage to determine whether the person being assessed in the camera footage is swallowing; and outputting the assessment result.

[0029] Like the above-mentioned system, this program also performs image analysis on a region of the camera image of the person being evaluated that includes the throat of the person being evaluated, and determines whether the person being evaluated in the camera image is swallowing. In this way, swallowing can be determined based on the camera image of the throat of the person being evaluated, which reduces the psychological burden on the person being evaluated compared to conventional swallowing sensors (which are attached to the throat).

[0030] According to the present invention, it is possible to reduce the psychological burden on the person being assessed.

[0031] (Embodiments) A swallowing assessment system according to an embodiment of the present invention will be described below with reference to the drawings. In this embodiment, a swallowing assessment system used in a rehabilitation system for persons with swallowing disorders will be illustrated. The swallowing assessment system according to this embodiment has a function of assessing swallowing based on a camera image of the throat of a subject taken from the front. These functions are realized by a program stored in a memory area of ​​the swallowing assessment system.

[0032] (First embodiment) The configuration of a swallowing assessment system according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the swallowing assessment system according to this embodiment. As shown in FIG. 1, the swallowing assessment system 1 includes an imaging device 100 and a performance device 101 as external devices. The imaging device 100 is, for example, a camera, and is positioned so as to be able to capture an image of the person being assessed from the front. The performance device 101 is, for example, a bone conduction earphone, a speaker, a display, a fan, a projector, an LED or other light source, and has the function of providing various performance effects to the person being assessed.

[0033] 1 , the swallowing assessment system 1 includes, as functional blocks, a camera image input unit 2, a assessment target region determination unit 3, an image analysis unit 4, and a assessment result output unit 5. Camera images captured by an imaging device 100 (camera images of a person to be assessed taken from the front) are input to the camera image input unit 2.

[0034] As shown in FIG. 2 , the determination target area determination unit 3 has a function of determining a determination target area including the throat of the subject of determination from the entire area of ​​the camera image. For example, the determination target area determination unit 3 determines a determination target area including the throat of the subject of determination from the entire area of ​​the camera image based on the facial contour, eye positions, nose positions, and mouth positions of the subject of determination detected from the camera image. More specifically, the determination target area including the throat of the subject of determination is determined from the facial contour, eye positions, nose positions, and mouth positions of the subject of determination based on the relative positional relationship between the facial contour, eye positions, nose positions, and mouth positions and the determination target area including the throat. Alternatively, the color of the facial surface may be used as a reference color, and a portion of a similar color may be determined as the determination target area including the throat of the subject of determination. In this case, the determination target area including the throat of the subject of determination can be distinguished from the clothing and collar worn by the subject of determination using a color similar to the skin color as a reference.

[0035] The image analysis unit 4 has a function of performing image analysis processing on a determination target area of ​​the camera image to determine whether or not the subject of determination shown in the camera image is swallowing. As shown in Fig. 1 , in this embodiment, the image analysis unit 4 has, as functional blocks, a frame difference image generation unit 6, a ratio calculation unit 7, a swallowing action determination unit 8, a machine learning unit 9, and an estimation unit 10.

[0036] The frame difference image generation unit 6 has a function of performing frame difference processing on the determination target area of ​​the camera image to generate a frame difference image composed of difference pixels between the current frame image and the frame image a predetermined number of frames before. A known technique can be used to generate the frame difference image. FIG. 3 is a diagram showing an example of a frame difference image. As shown in FIG. 3, when the subject is not swallowing, the number of difference pixels (shown by black dots in FIG. 3) in the frame difference image is small (see FIG. 3(a)), whereas when the subject is swallowing, the number of difference pixels in the frame difference image is large (see FIG. 3(b)).

[0037] The proportion calculation unit 7 has a function of calculating the total number of pixels N in the determination target region and the number of difference pixels Nd in the frame difference image, and calculating the proportion R of the number of difference pixels in the frame difference image to the total number of pixels in the determination target region. The swallowing movement determination unit 8 then determines whether the proportion R is higher than a predetermined threshold Rt, and if the proportion R is higher than the threshold Rt (e.g., 50%), determines that the subject appearing in the camera image is swallowing (see FIG. 4). However, as described below, the swallowing movement determination unit 8 also has a function of improving the accuracy of swallowing determination by utilizing estimation through machine learning.

[0038] The machine learning unit 9 has a function of analyzing, by machine learning, the relationship between the frame difference image and whether or not the subject is swallowing using predetermined training data. Any method, such as deep learning using a neural network, can be used for this machine learning. For example, in the case of a neural network, the frame difference image of the training data is input to an input layer, and information regarding whether or not the subject is swallowing is output from an output layer. Then, the weighting coefficients between the neurons of the neural network are optimized by supervised learning using training data in which the data input to the input layer and the data output from the output layer are linked.

[0039] The estimation unit 10 has a function of estimating and outputting whether or not the subject of judgment shown in the camera video is performing a swallowing action, using a frame difference image generated from the judgment target area of ​​the camera video as input, based on the relationship generated by the machine learning unit 9. For example, in the case of the above-mentioned neural network, the frame difference image generated by the frame difference image generation unit 6 is input to the input layer, and information regarding whether or not the subject of judgment is performing a swallowing action is output from the output layer, thereby performing estimation. Then, even if the ratio R is higher than the threshold value Rt, if the estimation unit 10 estimates that the subject of judgment is not performing a swallowing action, the swallowing action determination unit 8 determines that the subject of judgment shown in the camera video is not performing a swallowing action (see FIG. 4 ).

[0040] The determination result output unit 5 has a function of outputting the determination result of the image analysis unit 4. For example, the determination result output unit 5 has a function of outputting a trigger signal to the performance device 101 to perform a performance effect when it is determined that the person to be determined is swallowing.

[0041] The operation of the swallowing determination system 1 configured as above will be described with reference to the sequence diagram of FIG.

[0042] When swallowing assessment is performed using the swallowing assessment system 1 of this embodiment, first, the person to be assessed is photographed from the front using the imaging device 100 (S10). The camera image of the person to be assessed taken from the front is transmitted from the imaging device 100 to the swallowing assessment system 1 (S11). In the swallowing assessment system 1, when the camera image of the person to be assessed taken from the front is input from the imaging device 100 (S12), a target area for assessment that includes the throat of the person to be assessed is determined from the entire area of ​​the camera image, as shown in Fig. 2 (S13), and a frame difference process is performed on the target area of ​​the camera image to generate a frame difference image (see Fig. 3) (S14).

[0043] Next, the swallowing assessment system 1 calculates the ratio R (=Nd / N) of the number of difference pixels Nd of the frame difference image to the total number of pixels N of the assessment target region (S15), and determines whether the ratio R is higher than a predetermined threshold Rt (S16). Furthermore, the swallowing assessment system 1 uses the relationship generated by the machine learning unit 9 to estimate whether the subject of assessment shown in the camera video is swallowing, based on the frame difference image generated from the assessment target region of the camera video (S17).

[0044] 4, based on the comparison result between the ratio R and the threshold value Rt and the estimation result by machine learning, it is determined whether the subject of the evaluation in the camera image is swallowing (S18), and the determination result is output (S19). In this embodiment, when the determination result is output, a trigger signal is transmitted from the swallowing evaluation system 1 to the performance device 101 (S20), and various performance effects are applied to the subject of the evaluation by the performance device 101 (S21).

[0045] According to the swallowing assessment system 1 of the first embodiment, an image analysis process is performed on a region to be assessed that includes the throat of the person to be assessed in a camera image taken from the front of the person to be assessed, and it is determined whether the person to be assessed in the camera image is swallowing. In this way, swallowing assessment can be performed based on a camera image taken from the front of the throat of the person to be assessed, which reduces the psychological burden on the person to be assessed compared to conventional swallowing sensors (which are attached to the throat).

[0046] In this embodiment, frame difference processing is performed on the determination target area of ​​the camera image to generate a frame difference image, and whether the subject appearing in the camera image is swallowing is determined based on whether the ratio R (=Nd / N) of the number of difference pixels Nd in the frame difference image to the total number of pixels N in the determination target area is higher than a predetermined threshold Rt. In this way, swallowing can be appropriately determined based on the camera image capturing the subject's throat from the front.

[0047] Furthermore, in this embodiment, whether or not the subject of judgment shown in the camera video is swallowing is estimated from a frame difference image generated from the judgment target region of the camera video, based on the relationship generated by the machine learning unit 9. In this way, estimation by machine learning can be used to improve the accuracy of swallowing judgment based on camera video capturing the throat of the subject of judgment from the front.

[0048] Furthermore, in this embodiment, when it is determined that the person being assessed is swallowing, a trigger signal for producing a performance effect is output to the performance device 101. As a result, at the timing when swallowing is assessed, a performance (for example, vibrations from bone conduction earphones, sound from a speaker, video images from a display, wind pressure from an electric fan, image projection onto a screen or the human body by a projector, light-up effects by moving light sources such as LEDs, etc.) can be generated by the performance device 101.

[0049] Furthermore, in this embodiment, the region to be determined that includes the throat of the person to be determined is appropriately determined from the entire region of the camera image based on the facial contour, eye position, nose position, and mouth position of the person to be determined that are detected from the camera image. This makes it possible to appropriately determine swallowing based on the camera image that captures the throat of the person to be determined from the front.

[0050] Second Embodiment Next, a swallowing determination system according to a second embodiment of the present invention will be described. Here, the swallowing determination system according to the second embodiment will be described, focusing on differences between the system according to the first embodiment. Unless otherwise specified, the configuration and operation of this embodiment are the same as those of the first embodiment.

[0051] Fig. 6 is a block diagram showing the configuration of the swallowing assessment system of this embodiment. As shown in Fig. 6, in the swallowing assessment system 1 of this embodiment, the image analysis unit 4 includes, as functional blocks, a feature point extraction unit 11, a movement amount calculation unit 12, a swallowing movement assessment unit 13, a machine learning unit 14, and an estimation unit 15.

[0052] The feature point extraction unit 11 has a function of extracting throat feature points included in the region to be determined in the camera image. Publicly known techniques can be used to extract the feature points. FIG. 7 is a diagram showing an example of throat feature points. As shown in FIG. 7, when the subject is not swallowing, the throat feature points do not move much (see FIG. 7(a)), whereas when the subject is swallowing, the throat feature points move (see FIG. 7(b)).

[0053] The movement amount calculation unit 12 has a function of calculating the movement amount D of the throat feature point. A known technique can be used to calculate the movement amount of the feature point. The swallowing action determination unit 13 then determines whether the movement amount D is higher than a predetermined threshold Dt (e.g., a 5% change rate based on the distance to a nearby feature point when stationary), and if it determines that the movement amount D is higher than the predetermined threshold Dt, determines that the subject appearing in the camera video is swallowing (see FIG. 8 ). However, even in this embodiment, the swallowing action determination unit 13 has a function of improving the accuracy of swallowing determination by utilizing estimation through machine learning, as described below.

[0054] The machine learning unit 14 has a function of analyzing, by machine learning, the relationship between the movement of throat feature points and whether or not the subject of assessment is performing a swallowing action using predetermined training data. Any method, such as deep learning using a neural network, can be used for this machine learning. For example, in the case of a neural network, the movement of throat feature points in the training data is input to an input layer, and information regarding whether or not the subject of assessment is performing a swallowing action is output from an output layer. Then, the weighting coefficients between the neurons of the neural network are optimized by supervised learning using training data in which the data input to the input layer and the data output from the output layer are linked.

[0055] The estimation unit 15 has a function of estimating and outputting whether the subject of judgment shown in the camera video is performing a swallowing action, using as input the movement of throat feature points extracted from the judgment target area of ​​the camera video, based on the relationship generated by the machine learning unit 14. For example, in the case of the above-mentioned neural network, the movement of the throat feature points extracted by the feature point extraction unit 11 is input to the input layer, and information regarding whether the subject of judgment is performing a swallowing action is output from the output layer, thereby performing estimation. Then, even if the movement amount D is higher than a predetermined threshold Dt, if the estimation unit 15 estimates that the subject of judgment is not performing a swallowing action, the swallowing action determination unit 13 determines that the subject of judgment shown in the camera video is not performing a swallowing action (see FIG. 8 ).

[0056] The operation of the swallowing determination system 1 configured as above will be described with reference to the sequence diagram of FIG.

[0057] When swallowing assessment is performed using the swallowing assessment system 1 of this embodiment, as in the first embodiment, first, the person to be assessed is photographed from the front using the imaging device 100 (S10). The camera image of the person to be assessed taken from the front is transmitted from the imaging device 100 to the swallowing assessment system 1 (S11). In the swallowing assessment system 1, when the camera image of the person to be assessed taken from the front is input from the imaging device 100 (S12), a assessment target region that includes the throat of the person to be assessed is determined from the entire region of the camera image (S13), as shown in FIG.

[0058] In the swallowing determination system 1 of the present embodiment, throat feature points included in the determination target area of ​​the camera video are extracted (S22), the amount of movement D of the throat feature points is calculated (S23), and it is determined whether the amount of movement D is higher than a predetermined threshold Dt (S24). Furthermore, in the swallowing determination system 1, using the relationship generated by the machine learning unit 14, it is estimated whether the subject person shown in the camera video is swallowing, based on the movement of the throat feature points extracted from the determination target area of ​​the camera video (S25).

[0059] 8, based on the comparison result between the movement amount D and the threshold value Dt and the estimation result by machine learning, it is determined whether the subject of the evaluation in the camera image is swallowing (S26), and the determination result is output (S27). In the present embodiment, when the determination result is output, a trigger signal is transmitted from the swallowing evaluation system 1 to the performance device 101 (S20), and various performance effects are applied to the subject of the evaluation by the performance device 101 (S21).

[0060] The swallowing determination system 1 according to the second embodiment also provides the same effects as those of the first embodiment.

[0061] In this embodiment, feature points of the throat are extracted from the region to be determined in the camera image, and whether the subject of the determination shown in the camera image is swallowing or not is determined based on whether the amount of movement D of the feature points of the throat is higher than a predetermined threshold Dt. In this way, swallowing can be appropriately determined based on the camera image of the subject's throat taken from the front.

[0062] Furthermore, in this embodiment, whether or not the subject of assessment shown in the camera video is swallowing is estimated from the movement of feature points of the throat extracted from the assessment target area of ​​the camera video, based on the relationship generated by the machine learning unit 14. In this way, estimation by machine learning can be used to improve the accuracy of swallowing assessment based on camera video capturing the throat of the subject of assessment from the front.

[0063] The above describes the embodiments of the present invention by way of example, but the scope of the present invention is not limited to these, and can be changed or modified according to the purpose within the scope of the claims.

[0064] For example, in the above example, the camera image input from the camera image input unit is a camera image taken from the front of the person to be judged, but the scope of the present invention is not limited to this. The camera image input from the camera image input unit does not necessarily have to be a camera image taken from the front of the person to be judged, as long as it includes the throat of the person to be judged (the angle or direction at which the person to be judged is photographed is not limited).

[0065] As described above, the swallowing assessment system according to the present invention has the effect of reducing the psychological burden on the person being assessed, and is useful when applied to rehabilitation systems for people with swallowing disorders, etc.

[0066] REFERENCE SIGNS LIST 1 Swallowing determination system 2 Camera image input unit 3 Determination target area determination unit 4 Image analysis unit 5 Determination result output unit 6 Frame difference image generation unit 7 Ratio calculation unit 8 Swallowing action determination unit 9 Machine learning unit 10 Estimation unit 11 Feature point extraction unit 12 Movement amount calculation unit 13 Swallowing action determination unit 14 Machine learning unit 15 Estimation unit 100 Imaging device 101 Performance device

Claims

1. A swallowing assessment system comprising: a camera image input unit to which camera images of a person to be assessed are input; a target area determination unit that determines a target area for assessment that includes the throat of the person to be assessed from the entire area of ​​the camera image; an image analysis unit that performs image analysis processing on the target area of ​​the camera image to determine whether the person to be assessed who is shown in the camera image is swallowing; and a assessment result output unit that outputs the assessment result of the image analysis unit.

2. The swallowing determination system of claim 1, wherein the image analysis unit comprises: a frame difference image generation unit that performs frame difference processing on the determination target area of ​​the camera image to generate a frame difference image consisting of difference pixels between the current frame image and the frame image a predetermined number of frames before; a ratio calculation unit that calculates the ratio R of the number of difference pixels of the frame difference image to the total number of pixels in the determination target area; and a swallowing action determination unit that determines whether the ratio R is higher than a predetermined threshold value Rt, and if the ratio R is higher than the threshold value Rt, determines that the person to be determined in the determination target area shown in the camera image is swallowing.

3. The swallowing assessment system of claim 2, wherein the image analysis unit comprises: a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the frame difference image and whether or not the person being assessed is performing a swallowing action; and an estimation unit that, based on the relationship generated by the machine learning unit, inputs the frame difference image generated from the area to be assessed in the camera image and estimates and outputs whether or not the person being assessed who appears in the camera image is performing a swallowing action.

4. The swallowing determination system of claim 1, wherein the image analysis unit comprises: a feature point extraction unit that extracts feature points of the throat included in the determination target area of ​​the camera image; a movement amount calculation unit that calculates a movement amount D of the feature points of the throat; and a swallowing action determination unit that determines whether the movement amount D is higher than a predetermined threshold Dt, and if it is determined that the movement amount D is higher than the predetermined threshold Dt, determines that the subject of determination shown in the camera image is performing a swallowing action.

5. The swallowing determination system of claim 3, wherein the image analysis unit comprises: a machine learning unit that uses predetermined learning data to analyze, through machine learning, the relationship between the movement of throat feature points included in the determination target area of ​​the camera image and whether the person being determined is performing a swallowing action; and an estimation unit that uses the movement of the throat feature points extracted from the determination target area of ​​the camera image as input, based on the relationship generated by the machine learning unit, and estimates and outputs whether the person being determined in the camera image is performing a swallowing action.

6. A swallowing assessment system as described in claim 1, comprising a performance device that provides a predetermined performance effect to the person being assessed, and wherein the assessment result output unit outputs a trigger signal to the performance device to provide the performance effect when it is determined that the person being assessed is performing a swallowing action.

7. The swallowing determination system of claim 1, wherein the determination target area determination unit determines the determination target area that includes the throat of the person being determined from the entire area of ​​the camera image based on the facial contour, eye position, nose position, and mouth position of the person being determined detected from the camera image.

8. A method executed by a swallowing assessment system, the method including the steps of: inputting camera footage of a person to be assessed; determining a target area of ​​the camera footage that includes the throat of the person to be assessed; performing image analysis processing on the target area of ​​the camera footage to determine whether the person to be assessed in the camera footage is swallowing; and outputting the assessment result.

9. A program executed in a swallowing assessment system, the program causing a computer of the swallowing assessment system to execute the following processes: inputting camera footage of a person to be assessed; determining a target area of ​​the camera footage that includes the throat of the person to be assessed; performing image analysis on the target area of ​​the camera footage to determine whether the person to be assessed in the camera footage is swallowing; and outputting the assessment results.

Images