Swallowing determination system

Abstract

A swallowing determination system (1) includes a camera image input unit (2) that inputs a camera image obtained by photographing a determination target person, a determination target region determination unit (3) that determines a determination target region including a throat of the determination target person in an entire region of the camera image, an image analysis unit (4) that performs image analysis processing on the determination target region of the camera image, determines whether or not the determination target person is performing a swallowing action in the camera image, and a determination result output unit (5) that outputs a determination result of the image analysis unit.

Description

Technical Field

[0001] This invention relates to a swallowing determination system for determining whether a subject is swallowing. Background Technology

[0002] Previously, swallowing sensors for detecting human swallowing movements have been developed. For example, conventional swallowing sensors include piezoelectric thin-film sensors with multiple sensing portions along the length of the neck. These sensors are located within the range of movement of the thyroid cartilage that accompanies swallowing and are attached to the skin of the anterior neck, outputting analog signals that accompanies the deformation of the multiple sensing portions. The main body of the swallowing sensor determines whether swallowing has occurred based on the low-frequency component of the analog signal, i.e., the displacement signal (e.g., Japanese Patent Application Laid-Open No. 2022-33367).

[0003] However, conventional swallowing sensors are attached to the throat (the skin of the front of the neck), which can be uncomfortable and cause psychological burden depending on the individual with swallowing difficulties. Therefore, there is a desire to develop a swallowing assessment system that can reduce the psychological burden on the person being assessed (the individual with swallowing difficulties). Summary of the Invention

[0004] The technical problem that the invention aims to solve This invention was made against the above background. The object of this invention is to provide a swallowing determination system that can reduce the psychological burden on the person making the determination.

[0005] Technical solutions for solving technical problems One aspect of the present invention is a swallowing determination system, comprising: a camera image input unit that inputs a camera image obtained by capturing a subject being determined; a subject area determination unit that determines a subject area in the entire area of ​​the camera image that includes the throat of the subject being determined; an image analysis unit that performs image analysis processing on the subject area of ​​the camera image to determine whether the subject being determined shown in the camera image is performing a swallowing action; and a determination result output unit that outputs the determination result of the image analysis unit.

[0006] Another aspect of the present invention is a method performed by a swallowing determination system, comprising the following steps: inputting a camera image obtained by photographing a person to be determined; determining a determination object region in the entire area of ​​the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and outputting the determination result.

[0007] Another aspect of the present invention is a program executed by a swallowing determination system, which causes the computer of the swallowing determination system to perform the following processes: inputting a camera image obtained by photographing the person to be determined; determining a determination object region in the entire area of ​​the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined shown in the camera image is performing a swallowing action; and outputting the determination result.

[0008] As explained below, other embodiments of the invention exist. Therefore, this disclosure is intended to provide some aspects of the invention and not to limit the scope of the claimed invention described herein. Attached Figure Description

[0009] Figure 1 This is a block diagram showing the configuration of the swallowing determination system in the first embodiment.

[0010] Figure 2 This is a diagram illustrating an example of determining the defined region of an object.

[0011] Figure 3 This is a diagram illustrating an example of swallowing determination based on the proportion of differential pixel counts.

[0012] Figure 4 This is a diagram illustrating an example of a swallowing decision that takes into account machine learning-based inferences.

[0013] Figure 5 This is a timing diagram used to illustrate the operation of the swallowing determination system in the first embodiment.

[0014] Figure 6 This is a block diagram illustrating the configuration of the swallowing determination system in the second embodiment.

[0015] Figure 7 This is a diagram illustrating an example of swallowing determination based on the amount of movement of feature points in the pharynx.

[0016] Figure 8 This is a diagram illustrating an example of a swallowing decision that takes into account inferences based on machine learning.

[0017] Figure 9 This is a timing diagram used to illustrate the operation of the swallowing determination system in the second embodiment. Detailed Implementation

[0018] The present invention will now be described in detail. However, the following detailed description and accompanying drawings are not intended to limit the scope of the invention.

[0019] The swallowing determination system of the present invention comprises: a camera image input unit that inputs a camera image obtained by capturing a person to be determined; a person to be determined area determination unit that determines a person to be determined area that includes the throat of the person to be determined in the entire area of ​​the camera image; an image analysis unit that performs image analysis processing on the person to be determined area of ​​the camera image to determine whether the person to be determined shown in the camera image is performing a swallowing action; and a determination result output unit that outputs the determination result of the image analysis unit.

[0020] Based on this configuration, image analysis processing is performed on the target area, which includes the throat of the target person, in the camera image captured by the camera, to determine whether the target person shown in the camera image is swallowing. In this way, swallowing determination can be made based on camera images obtained by capturing the throat of the target person, thus reducing the psychological burden on the target person compared to conventional swallowing sensors (which are attached to the throat).

[0021] It should be noted that the camera image input from the camera image input unit only needs to include the throat of the person being judged. There are no restrictions on the angle or direction of shooting the person being judged, but by shooting from the front, it is possible to more appropriately determine whether a swallowing action is being performed. Therefore, it is preferable to shoot the person being judged from the front.

[0022] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: an inter-frame difference image generation unit that performs inter-frame difference processing on the determination object region of the camera image to generate an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame; a pixel count calculation unit that calculates the ratio R of the difference pixel count of the inter-frame difference image to the total number of pixels in the determination object region; and a swallowing action determination unit that determines whether the ratio R is higher than a given threshold Rt, and if the ratio R is higher than the threshold Rt, determines that the determination object reflected in the camera image is performing a swallowing action.

[0023] Based on this configuration, inter-frame differential processing is performed on the target area of ​​the camera image to generate an inter-frame differential image. The determination of whether the subject appearing in the camera image is swallowing is based on whether the ratio R (=Nd / N) of the differential pixel count Nd in the inter-frame differential image to the total pixel count N of the target area is higher than a given threshold Rt. In this way, swallowing can be appropriately determined based on camera images obtained by capturing the throat area of ​​the subject.

[0024] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a machine learning unit that uses given learning data to analyze the relationship between the inter-frame difference image and whether the subject is swallowing; and an estimation unit that, based on the relationship generated by the machine learning unit, uses the inter-frame difference image generated from the subject area of ​​the camera image as input to estimate and output whether the subject reflected in the camera image is swallowing.

[0025] Based on this configuration, and using the relationship generated by the machine learning unit, it is inferred whether the person being judged, as shown in the camera image, is swallowing, based on the inter-frame difference image generated from the judgment object region of the camera image. Thus, by utilizing machine learning-based inference, the accuracy of swallowing judgment based on camera images obtained by filming the throat of the judgment object can be improved.

[0026] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a feature point extraction unit that extracts feature points of the pharynx included in the determination object area of ​​the camera image; a movement calculation unit that calculates the movement amount D of the feature points of the pharynx; and a swallowing action determination unit that determines whether the movement amount D is higher than a given threshold Dt, and if it is determined that the movement amount D is higher than the given threshold Dt, it determines that the determination object reflected in the camera image is performing a swallowing action.

[0027] Based on this configuration, feature points of the pharynx are extracted from the target area of ​​the camera image. Whether the movement D of these feature points exceeds a given threshold Dt is used to determine if the person being judged, as shown in the camera image, is swallowing. In this way, swallowing can be appropriately determined based on camera images obtained by capturing the pharynx of the person being judged.

[0028] Furthermore, in the swallowing determination system of the present invention, the image analysis unit may include: a machine learning unit that uses given learning data to analyze the relationship between the movement of feature points of the pharynx contained in the determination object region of the camera image and whether the person being determined is swallowing; and an estimation unit that, based on the relationship generated by the machine learning unit, uses the movement of feature points of the pharynx extracted from the determination object region of the camera image as input to estimate and output whether the person being determined, as reflected in the camera image, is swallowing.

[0029] Based on this configuration, and using the relationships generated by the machine learning unit, the system infers whether the person being judged, as shown in the camera image, is swallowing based on the movement of feature points of the throat extracted from the judgment area of ​​the camera image. Thus, by utilizing machine learning-based inference, the accuracy of swallowing judgments based on camera images obtained by capturing images of the judgment subject's throat can be improved.

[0030] Furthermore, the swallowing determination system of the present invention can be equipped with a performance device that assigns a given performance effect to the person being determined. When the person being determined is performing a swallowing action, the determination result output unit can output a trigger signal to the performance device for performing the performance effect.

[0031] According to this configuration, if the person being judged performs a swallowing action, a trigger signal is output to the performance equipment to execute the performance effect. Thus, at the moment the swallowing judgment is completed, the performance generated by the performance equipment can be triggered (e.g., vibration generated by bone conduction headphones, sound played by speakers, video images displayed on a monitor, wind pressure generated by an electric fan, image projection from a projector onto a screen or human body, light show achieved by moving light sources such as LEDs, etc.).

[0032] Furthermore, in the swallowing determination system of the present invention, the determination object area determination unit can determine the determination object area that includes the throat of the determination object in the entire area of ​​the camera image, based on the facial contour, eye position, nose position and mouth position of the determination object detected from the camera image.

[0033] Based on this configuration, using the facial contours, eye positions, nose positions, and mouth positions of the person being judged as detected from the camera image as a reference, a judgment object region encompassing the throat of the person being judged is appropriately determined within the entire area of ​​the camera image. Therefore, swallowing judgment can be appropriately performed based on the camera image obtained by capturing the throat of the person being judged.

[0034] The method of the present invention is a method executed by a swallowing determination system, comprising the following steps: inputting a camera image obtained by photographing a person to be determined; determining a determination object region in the entire area of ​​the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object region of the camera image to determine whether the person to be determined reflected in the camera image is performing a swallowing action; and outputting the determination result.

[0035] According to this method, similarly to the system described above, image analysis processing is performed on the target area, which includes the throat of the target person, in the camera image captured by the system, to determine whether the target person shown in the camera image is swallowing. In this way, swallowing determination can be made based on camera images obtained by capturing the throat of the target person, thus reducing the psychological burden on the target person compared to conventional swallowing sensors (which are attached to the throat).

[0036] The program of the present invention is executed by a swallowing determination system, which causes the computer of the swallowing determination system to perform the following processes: processing of inputting a camera image obtained by photographing the person to be determined; processing of determining a determination object area in the entire area of ​​the camera image that includes the throat of the person to be determined; performing image analysis processing on the determination object area of ​​the camera image to determine whether the person to be determined shown in the camera image is performing a swallowing action; and processing of outputting the determination result.

[0037] According to this procedure, similar to the system described above, image analysis processing is performed on the target area, which includes the throat of the target person, in the camera image captured by the system, to determine whether the target person shown in the camera image is swallowing. In this way, swallowing determination can be made based on camera images obtained by capturing the throat of the target person, thus reducing the psychological burden on the target person compared to conventional swallowing sensors (which are attached to the throat).

[0038] According to the present invention, the psychological burden on the person making the judgment can be reduced.

[0039] (Implementation Method) Hereinafter, a swallowing determination system according to an embodiment of the present invention will be described using the accompanying drawings. In this embodiment, a swallowing determination system used in a rehabilitation system for people with swallowing disorders is illustrated. The swallowing determination system of this embodiment has the function of determining swallowing based on a camera image obtained by taking a frontal view of the throat of the person being determined. These functions are implemented by a program stored in the memory area of ​​the swallowing determination system or the like.

[0040] (First Implementation) The configuration of the swallowing determination system according to the first embodiment of the present invention will be described with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating the configuration of the swallowing determination system in this embodiment. For example... Figure 1As shown, the swallowing determination system 1, as an external device, includes a camera 100 and a performance device 101. The camera 100 is, for example, a camera configured to capture images of the person being determined from the front. The performance device 101 is, for example, a bone conduction headset, a speaker, a monitor, a fan, a projector, an LED light source, etc., and has the function of imparting various performance effects to the person being determined.

[0041] In addition, such as Figure 1 As shown, the swallowing determination system 1, as a functional module, includes a camera image input unit 2, a determination object area determination unit 3, an image analysis unit 4, and a determination result output unit 5. The camera image input unit 2 is input with a camera image captured by the shooting device 100 (a camera image obtained by shooting the determination object from the front).

[0042] like Figure 2 As shown, the object region determination unit 3 has the function of determining an object region that includes the throat of the person being judged, within the entire area of ​​the camera image. For example, the object region determination unit 3 determines the object region that includes the throat of the person being judged, within the entire area of ​​the camera image, based on the facial contour, eye position, nose position, and mouth position of the person being judged detected from the camera image. More specifically, based on the relative positional relationship between the facial contour, eye position, nose position, mouth position, and the object region including the throat, the object region including the throat of the person being judged is determined according to the facial contour, eye position, nose position, and mouth position of the person being judged. Alternatively, the color of the facial surface can be set as a reference color, and the portion of the color similar to it can be determined as the object region including the throat of the person being judged. In this case, the object region including the throat of the person being judged and the clothing and collar worn by the person being judged can be identified based on a color similar to the skin tone.

[0043] The image analysis unit 4 has the function of performing image analysis processing on the target area of ​​the camera image to determine whether the target person reflected in the camera image is performing a swallowing action. For example... Figure 1 As shown, in this embodiment, the image parsing unit 4 is a functional module comprising an inter-frame difference image generation unit 6, a ratio calculation unit 7, a swallowing action determination unit 8, a machine learning unit 9, and an estimation unit 10.

[0044] The inter-frame difference image generation unit 6 has the function of performing inter-frame difference processing on the target region of the camera image and generating an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame. Known techniques can be used in the generation of the inter-frame difference image. Figure 3 This is a diagram representing an example of inter-frame difference images. For example... Figure 3As shown, when the subject is determined not to be swallowing, the difference pixels in the inter-frame difference image (in) Figure 3 The number of (using black dots as an illustration) is small (refer to) Figure 3 (a) indicates that when the subject is determined to be swallowing, the number of difference pixels in the inter-frame difference image increases (see reference). Figure 3 (b)

[0045] The ratio calculation unit 7 has the function of calculating the total number of pixels N of the target area and the number of differential pixels Nd of the inter-frame difference image, and calculating the ratio R of the number of differential pixels of the inter-frame difference image to the total number of pixels of the target area. Then, the swallowing action determination unit 8 determines whether the ratio R is higher than a given threshold Rt. If the ratio R is higher than the threshold Rt (e.g., 50%), it determines that the target person reflected in the camera image has performed a swallowing action (see reference). Figure 4 However, as described below, the swallowing action determination unit 8 has the function of improving the accuracy of swallowing determination by using machine learning-based estimation.

[0046] The Machine Learning Unit 9 has the capability to analyze the relationship between inter-frame difference images and whether a subject is swallowing using machine learning with given learning data. This machine learning employs any method, such as deep learning based on neural networks. For example, if a neural network is used, it is configured to input the inter-frame difference images of the learning data into the input layer and output information related to whether the subject is swallowing from the output layer. Furthermore, through supervised learning using learning data that establishes a correlation between the data input to the input layer and the data output from the output layer, the weighting coefficients between neurons in the neural network are optimized.

[0047] The estimation unit 10 has the following function: based on the relationship generated by the machine learning unit 9, it takes the inter-frame difference image generated from the target area of ​​the camera image as input, estimates whether the target person reflected in the camera image is swallowing, and outputs the result. For example, if it is the aforementioned neural network, the inter-frame difference image generated by the inter-frame difference image generation unit 6 is input to the input layer, and information related to whether the target person is swallowing is output from the output layer, thereby making an estimation. Moreover, even if the proportion R is higher than the threshold Rt, the swallowing action determination unit 8 determines that the target person reflected in the camera image is not swallowing when the estimation unit 10 estimates that the target person is not swallowing (see reference). Figure 4 ).

[0048] The judgment result output unit 5 has the function of outputting the judgment result of the image analysis unit 4. For example, the judgment result output unit 5 has the function of outputting a trigger signal to the performance equipment 101 for performing a performance effect when the person judged as the judgment object performs a swallowing action.

[0049] Regarding the swallowing determination system 1 constructed as described above, refer to... Figure 5 The timing diagram illustrates its actions.

[0050] When performing a swallowing determination using the swallowing determination system 1 of this embodiment, firstly, the imaging device 100 captures a frontal image of the person being determined (S10). The camera image obtained from capturing the frontal image of the person being determined is sent from the imaging device 100 to the swallowing determination system 1 (S11). In the swallowing determination system 1, if the camera image obtained from capturing the frontal image of the person being determined is input from the imaging device 100 (S12), then... Figure 2 As shown, a target region (S13) containing the throat of the person being judged is determined within the entire area of ​​the camera image. Inter-frame difference processing is then performed on the target region of the camera image to generate an inter-frame difference image (see reference). Figure 3 (S14).

[0051] Next, in the swallowing determination system 1, the ratio R (=Nd / N) of the difference pixel number Nd of the inter-frame difference image to the total number of pixels N of the determination object region is calculated (S15), and it is determined whether this ratio R is higher than a given threshold Rt (S16). Furthermore, in the swallowing determination system 1, using the relationship generated by the machine learning unit 9, based on the inter-frame difference image generated from the determination object region of the camera image, it is estimated whether the determination object reflected in the camera image is performing a swallowing action (S17).

[0052] Then, as Figure 4 As shown, based on the comparison result of the ratio R and the threshold Rt and the inference result of machine learning, it is determined whether the person being judged in the camera image is performing a swallowing action (S18), and the judgment result is output (S19). In this embodiment, if the judgment result is output, a trigger signal is sent from the swallowing judgment system 1 to the performance device 101 (S20), and various performance effects are given to the person being judged by the performance device 101 (S21).

[0053] According to this first embodiment of the swallowing determination system 1, image analysis processing is performed on the determination area containing the throat of the determination subject in a camera image obtained from a frontal view of the determination subject, to determine whether the determination subject reflected in the camera image is performing a swallowing action. In this way, swallowing determination can be performed based on a camera image obtained from a frontal view of the determination subject's throat, thus reducing the psychological burden on the determination subject compared to conventional swallowing sensors (used by attaching to the throat).

[0054] In this embodiment, inter-frame differential processing is performed on the target area of ​​the camera image to generate an inter-frame differential image. Based on whether the ratio R (=Nd / N) of the differential pixel count Nd in the inter-frame differential image to the total pixel count N of the target area is higher than a given threshold Rt, it is determined whether the person being judged, as shown in the camera image, is performing a swallowing action. In this way, swallowing can be appropriately determined based on a camera image obtained by capturing the throat of the person being judged from the front.

[0055] Furthermore, in this embodiment, based on the relationship generated by the machine learning unit 9, it is inferred whether the person being judged in the camera image is performing a swallowing action based on the inter-frame difference image generated from the judgment object region of the camera image. In this way, by using machine learning-based inference, the judgment accuracy of swallowing judgment based on camera images obtained from frontal shots of the judgment object's throat can be improved.

[0056] Furthermore, in this embodiment, if the person being judged performs a swallowing action, a trigger signal for executing a performance effect is output to the performance device 101. Thus, at the moment the swallowing judgment is completed, the performance generated by the performance device 101 can be triggered (e.g., vibration generated by bone conduction headphones, sound played by speakers, video images displayed on a monitor, wind pressure generated by a fan, image projection from a projector onto a screen or human body, light performance achieved by moving light-emitting bodies such as LEDs, etc.).

[0057] Furthermore, in this embodiment, based on the facial contours, eye positions, nose positions, and mouth positions of the person to be judged detected from the camera image, a judgment object region containing the throat of the person to be judged is appropriately determined within the entire area of ​​the camera image. Therefore, swallowing determination can be appropriately performed based on the camera image obtained by capturing the throat of the person to be judged from the front.

[0058] (Second Implementation) Next, the swallowing determination system according to the second embodiment of the present invention will be described. Here, the description will focus on the differences between the swallowing determination system of the second embodiment and the first embodiment. Unless otherwise specified, the structure and operation of this embodiment are the same as those of the first embodiment.

[0059] Figure 6 This is a block diagram illustrating the configuration of the swallowing determination system in this embodiment. For example... Figure 6 As shown, in the swallowing determination system 1 of this embodiment, the image analysis unit 4 is a functional module comprising a feature point extraction unit 11, a movement calculation unit 12, a swallowing action determination unit 13, a machine learning unit 14, and an estimation unit 15.

[0060] The feature point extraction unit 11 has the function of extracting feature points of the throat area contained in the target region of the camera image. The feature point extraction can be performed using known techniques. Figure 7 This is an example diagram showing characteristic points of the pharynx. For example... Figure 7 As shown, when the subject is not swallowing, the characteristic points of the pharynx do not move much (refer to...). Figure 7 (a) In contrast, when the subject performs a swallowing action, the characteristic points of the pharynx move (see reference). Figure 7 (b)

[0061] The movement calculation unit 12 has the function of calculating the movement amount D of feature points in the pharynx. The calculation of the movement amount of feature points can be performed using known techniques. Then, the swallowing action determination unit 13 determines whether the movement amount D is higher than a given threshold Dt (e.g., a rate of change of 5% based on the distance to a nearby feature point when stationary). If the movement amount D is determined to be higher than the given threshold Dt, it is determined that the person being judged, as reflected in the camera image, has performed a swallowing action (see reference). Figure 8 However, in this embodiment, the swallowing action determination unit 13 also has the function of improving the accuracy of swallowing determination by using machine learning-based estimation, as will be described below.

[0062] The machine learning unit 14 has the function of analyzing the relationship between the movement of feature points in the throat and whether a person is swallowing using machine learning with given learning data. In this machine learning, any method such as deep learning based on neural networks is used. For example, if it is a neural network, it is configured to input the movement of feature points in the throat of the learning data into the input layer and output information related to whether the person is swallowing from the output layer. Furthermore, through supervised learning of the learning data that establishes a correlation between the data input to the input layer and the data output from the output layer, the weighting coefficients between neurons in the neural network are optimized.

[0063] The estimation unit 15 has the following function: based on the relationship generated by the machine learning unit 14, it takes the motion of the feature points of the throat extracted from the target area of ​​the camera image as input, and estimates and outputs whether the target person reflected in the camera image is swallowing. For example, if it is the aforementioned neural network, the motion of the feature points of the throat extracted by the feature point extraction unit 11 is input to the input layer, and information related to whether the target person is swallowing is output from the output layer, thereby making an estimation. Moreover, even if the swallowing action determination unit 13 is higher than a given threshold Dt, and the estimation unit 15 estimates that the target person is not swallowing, it also determines that the target person reflected in the camera image is not swallowing (see reference). Figure 8 ).

[0064] Regarding the swallowing determination system 1 constructed as described above, refer to... Figure 9 The timing diagram illustrates its actions.

[0065] When performing a swallowing determination using the swallowing determination system 1 of this embodiment, similarly to the first embodiment, firstly, the imaging device 100 captures a frontal image of the person being determined (S10). The camera image of the person being determined from the front is then transmitted from the imaging device 100 to the swallowing determination system 1 (S11). In the swallowing determination system 1, if the camera image of the person being determined from the front is input from the imaging device 100 (S12), then... Figure 2 As shown, the determination object region (S13) is determined within the entire area of ​​the camera image, which includes the throat of the person being determined.

[0066] In the swallowing determination system 1 of this embodiment, feature points of the pharynx included in the determination object region of the camera image are extracted (S22), the movement amount D of the feature points is calculated (S23), and it is determined whether the movement amount D is higher than a given threshold Dt (S24). Furthermore, in the swallowing determination system 1, based on the movement of the feature points of the pharynx extracted from the determination object region of the camera image, a relationship generated by the machine learning unit 14 is used to infer whether the person being determined and reflected in the camera image is performing a swallowing action (S25).

[0067] Then, as Figure 8 As shown, based on the comparison result of the movement amount D and the threshold Dt and the inference result of machine learning, it is determined whether the person being judged, as reflected in the camera image, is performing a swallowing action (S26), and the judgment result is output (S27). In this embodiment, if the judgment result is output, a trigger signal is sent from the swallowing judgment system 1 to the performance device 101 (S20), and various performance effects are given to the person being judged by the performance device 101 (S21).

[0068] The swallowing determination system 1 according to this second embodiment also has the same effect as the first embodiment.

[0069] In this embodiment, feature points of the throat are extracted from the target area of ​​the camera image. Based on whether the movement D of the feature points of the throat is higher than a given threshold Dt, it is determined whether the person being judged, as shown in the camera image, is swallowing. In this way, swallowing determination can be appropriately performed based on the camera image obtained by shooting the throat of the person being judged from the front.

[0070] Furthermore, in this embodiment, based on the relationship generated by the machine learning unit 14, the movement of feature points of the throat extracted from the target area of ​​the camera image is used to infer whether the person being judged in the camera image is swallowing. Thus, by utilizing machine learning-based inference, the accuracy of swallowing determination based on camera images obtained from frontal shots of the person being judged's throat can be improved.

[0071] The embodiments of the present invention have been illustrated above by way of example, but the scope of the present invention is not limited thereto, and modifications and variations can be made according to the purpose within the scope of the claims.

[0072] For example, in the above example, the camera image input from the camera image input unit is an image obtained by shooting the subject from the front, but the scope of the present invention is not limited to this. The camera image input from the camera image input unit can be any image that includes the throat of the subject, and it does not necessarily have to be a camera image obtained by shooting the subject from the front (the angle and direction of shooting the subject are not limited).

[0073] Industrial availability As described above, the swallowing determination system of the present invention has the effect of reducing the psychological burden on the person being determined, and is applicable to rehabilitation systems for people with swallowing disorders, etc., and is practical.

[0074] Explanation of reference numerals in the attached figures 1 Swallowing determination system 2. Camera image input section 3. Determination of the target area 4 Image Analysis Unit 5. Judgment Result Output Section 6. Inter-frame differential image generation unit 7. Proportion Calculation Department 8. Swallowing Action Judgment Section 9. Machine Learning Department 10. Presumption Department 11 Feature Point Extraction Unit 12. Mobility Calculation Department 13 Swallowing Action Judgment Section 14 Machine Learning Department 15. Presumption Department 100 filming devices 101 Performance Equipment.

Claims

1. A swallowing determination system, comprising: The camera image input unit receives camera images obtained by capturing images of the subject being judged. The object region determination unit determines the object region, which includes the throat of the person being determined, within the entire area of ​​the camera image. An image analysis unit performs image analysis processing on the target area of ​​the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; and The determination result output unit outputs the determination result of the image analysis unit.

2. The swallowing determination system according to claim 1, wherein, The image parsing unit includes: The inter-frame difference image generation unit performs inter-frame difference processing on the target area of ​​the camera image to generate an inter-frame difference image composed of the difference pixels between the current frame image and the frame image before the given frame. The ratio calculation unit calculates the ratio R of the number of difference pixels in the inter-frame difference image relative to the total number of pixels in the determination object region; as well as The swallowing action determination unit determines whether the ratio R is higher than a given threshold Rt. If the ratio R is higher than the threshold Rt, it determines that the person being determined and reflected in the camera image is performing a swallowing action.

3. The swallowing determination system according to claim 2, wherein, The image parsing unit includes: The Machine Learning Department uses given learning data to analyze the relationship between inter-frame difference images and whether the subject is swallowing through machine learning. as well as The estimation unit, based on the relationship generated by the machine learning unit, takes the inter-frame difference image generated from the determination object region of the camera image as input, and estimates and outputs whether the determination object reflected in the camera image is performing a swallowing action.

4. The swallowing determination system according to claim 1, wherein, The image parsing unit includes: The feature point extraction unit extracts feature points of the throat area contained in the target area of ​​the camera image; The movement calculation unit calculates the movement D of the feature points of the pharynx; as well as The swallowing action determination unit determines whether the movement amount D is higher than a given threshold Dt. If the movement amount D is higher than the given threshold Dt, the unit determines that the person being judged, as reflected in the camera image, is performing a swallowing action.

5. The swallowing determination system according to claim 3, wherein, The image parsing unit includes: The Machine Learning Department uses given learning data to analyze the relationship between the movement of feature points of the throat contained in the target area of ​​the camera image and whether the target is swallowing. as well as The estimation unit, based on the relationship generated by the machine learning unit, takes the motion of feature points of the throat extracted from the target area of ​​the camera image as input to estimate and output whether the target person reflected in the camera image is swallowing.

6. The swallowing determination system according to claim 1, wherein, The swallowing determination system is equipped with performance equipment that can assign a given performance effect to the person being determined. When the person identified as the subject of the judgment is performing a swallowing action, the judgment result output unit outputs a trigger signal to the performance equipment to execute the performance effect.

7. The swallowing determination system according to claim 1, wherein, The determination object region unit uses the facial contour, eye position, nose position, and mouth position of the person being determined detected from the camera image as a reference to determine the determination object region, which includes the throat of the person being determined, within the entire area of ​​the camera image.

8. A method performed by a swallowing determination system, comprising the following steps: The steps for inputting camera images obtained by capturing images of the target being identified; The step of determining the target area, which includes the throat of the person being judged, within the entire area of ​​the camera image; The step of performing image analysis processing on the target area of ​​the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; as well as The step of outputting the result of the determination.

9. A program executed by a swallowing determination system, the program causing the computer of the swallowing determination system to perform the following processes: Input: Processing of camera images obtained by capturing images of the target; Processing to determine the target area that includes the throat of the person being judged, within the entire area of ​​the camera image; Image analysis processing is performed on the target area of ​​the camera image to determine whether the target person reflected in the camera image is performing a swallowing action; as well as The processing of outputting the results of the determination.

Images