Chewing state determination device, chewing state determination system, chewing state determination method, chewing state determination program, control device, control method, and control determination program

Abstract

The present invention performs control in conjunction with a face action, especially in conjunction with a chewing and opening state. Provided is a control system provided with: a chewing state determination device that determines the chewing state of a subject; and a control device that controls the motor or the image output device in conjunction with the determined chewing state.

Description

Technical Field

[0001] The present invention relates to a chewing state determination device, a chewing state determination system, a chewing state determination method, a chewing state determination program, a control state determination device, a control state determination method, and a control state determination program. Background Art

[0002] Patent Documents 1 and 2 disclose techniques for determining chewing movements.

[0003] Prior art literature Patent Literature Patent Document 1: Japanese Patent Application Laid-Open No. 2021-105878 Patent Document 2: Japanese Patent Application Laid-Open No. 2022-074671 Non-Patent Literature 3: https: / / www.jstage.jst.go.jp / article / soshaku1991 / 2 / 1 / 2_1_55 / _pdf / -char / en Non-Patent Document 4: https: / / google.github.io / mediapipe / Summary of the Invention Problems to be solved by the invention One of the problems of the present invention is to perform chewing judgment with high accuracy. The problem of the present invention is to perform control linked to facial movements, especially chewing and opening state.

[0004] Technical solutions to problems [1] According to one embodiment of the present invention, there is provided a chewing state determination device comprising: a feature point extraction section that extracts, from an image including a face of a subject, an upper feature point located above a mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth; a calculation unit that calculates a difference between a longitudinal distance between the upper feature point and the lower feature point and a lateral distance between the left feature point and the right feature point; and The determination unit determines the chewing state based on the difference.

[0005] [2] In the chewing state determination device described in [1], it can be, The determination unit determines whether the jaw is open based on the difference.

[0006] [3] In the chewing state determination device described in [2], it can be, The determination unit determines whether the jaw is open by comparing the difference with a first threshold value. [4] In the chewing state determination device described in [3], it can be, The first threshold is determined based on the difference in a state in which the subject closes his jaw. [5] In the chewing state determination device described in any one of [1] to [4], the determination unit determines the degree of jaw opening based on comparison between the difference and a reference value.

[0009] [6] In the chewing state determination device described in any one of [1] to [5], it may be that The determination unit may determine whether the bite has started, is in the middle of, or has ended during one chewing session based on the difference.

[0010] [7] In the chewing state determination device described in [1] to [6], it can be, The feature point extraction unit extracts upper lip feature points corresponding to the upper lip and lower lip feature points corresponding to the lower lip, The calculation unit calculates the interlip distance between the upper lip feature point and the lower lip feature point, The determination unit determines a chewing state based on the difference and the inter-lip distance.

[0011] [8] In the chewing state determination device described in [7], it can be, The determination unit Based on the difference, it is determined whether the jaw is open, And based on the distance between the lips, it is determined whether the lips are open, The chewing state is determined based on whether the jaw is open and whether the lips are open.

[0012] [9] In the chewing state determination device described in [8], it can be, When the jaw is open and the lips are open, the determination unit determines that the chewing state is the first bite, When the jaw is open and the lips are closed, the determination unit determines that the chewing state is after the second bite.

[0013]

[10] In the chewing state determination device described in [7], it can be, The determination unit determines whether the chewing state is the first bite or the second bite or later based on a comparison between the difference and a first threshold value and a comparison between the inter-lip distance and a second threshold value.

[0014]

[11] In the chewing state determination device described in any one of [1] to

[10] , An output control unit may be provided that controls the output unit so as to output at least one of an image, text, sound, vibration, and flavor according to the determined chewing state.

[0015]

[12] In the chewing state determination device described in [5], An output control unit may be provided for outputting at least one of an image, text, sound, and vibration according to the degree of jaw opening.

[0016]

[13] In the chewing state determination device described in any one of [1] to

[12] , it may be, The determination unit estimates calorie consumption based on the determined chewing state.

[0017]

[14] According to one embodiment of the present invention, a chewing state determination system is provided, comprising: a camera that captures the subject's face; and The chewing state determination device described in any one of [1] to

[13] .

[0018]

[15] According to one embodiment of the present invention, a method for determining a chewing state is provided, comprising: The feature point extraction unit extracts, from the image including the face of the subject, an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth; a step of calculating, by a calculation unit, a difference between a longitudinal distance between the upper feature point and the lower feature point and a lateral distance between the left feature point and the right feature point; and The step of: the determining unit determining the chewing state based on the difference.

[0019]

[16] According to one embodiment of the present invention, there is provided a chewing state determination program, which enables a computer to function as a feature point extraction unit, a calculation unit, and a determination unit. The feature point extraction unit extracts an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth from an image including the face of the subject. The calculation unit calculates a difference between a longitudinal distance between the upper feature point and the lower feature point and a lateral distance between the left feature point and the right feature point. The determination unit determines the chewing state based on the difference.

[0020]

[17] A control system comprising: a chewing state determination device for determining the chewing state of a subject; and The control device controls the motor or the image output device in conjunction with the determined chewing state.

[0021]

[18] In the control system described in

[17] , the motor moves the jaw in the facial model.

[0022]

[19] In the control system described in

[17] , the image output device outputs an image corresponding to the determined chewing state.

[0023]

[20] In the control system described in any one of

[17] to

[19] , the chewing state determination device determines the chewing state of the subject based on the pressure detected by a pressure sensor placed in the subject's ear.

[0024]

[21] In the control system described in any one of

[17] to

[20] , the chewing state determination device determines the chewing state of the subject based on an image obtained by photographing the subject's face.

[0025]

[22] In the control system described in

[21] , the chewing state determination device extracting a first feature point corresponding to the nose and a second feature point corresponding to the lower jaw from the image, The chewing state of the subject is determined based on the distance between the first feature point and the second feature point.

[0026]

[23] In the control system described in

[21] , the chewing state determination device extracting from the image an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth, and calculating the difference between the longitudinal distance between the upper feature point and the lower feature point and the transverse distance between the left feature point and the right feature point, Based on the difference, the chewing state of the subject is determined.

[0027]

[24] In the control system described in any one of

[17] to

[23] , the chewing state determination device determines the chewing state of the subject based on the movement of the muscle detected by the myoelectric sensor that detects the movement of the subject's jaw muscles.

[0028]

[25] A control method for determining the chewing state of a subject, The motor or image output device is controlled in conjunction with the determined chewing state.

[0029]

[26] A control program that causes a computer to execute the following steps: a step of determining the chewing state of the subject; and A step of controlling a motor or an image output device in conjunction with the determined chewing state.

[0030]

[27] A control system comprising: an opening detection device for detecting whether the subject has opened his mouth; and The control device controls the spray device to spray flavors or seasonings toward the mouth of the subject in response to detecting that the subject has opened his mouth.

[0031]

[28] In the control system described in

[27] , the opening detection device detects whether the subject has opened his mouth based on an image obtained by photographing the subject's face.

[0032]

[29] In the control system described in

[28] , the opening detection device extracting a first feature point corresponding to the upper lip and a second feature point corresponding to the lower lip from the image, And based on the distance between the first feature point and the second feature point, it is detected whether the subject has opened his mouth.

[0033]

[30] In the control system described in any one of

[27] to

[29] , the spray device is composed of a fan or an air gun that blows gas containing spices or seasonings.

[0034]

[31] In the control system described in

[30] , the opening detection device detects the position of the subject's mouth based on an image obtained by photographing the subject's face, The control device controls the blowing direction of the fan based on the detected position of the nozzle.

[0035]

[32] In the control system described in any one of

[27] to

[30] , the control device generates at least one of an image, a sound, a vibration, and a light in response to detecting that the subject has opened his mouth.

[0036]

[33] A control method for detecting when a subject opens his mouth. In response to detecting that the subject has opened his mouth, the spray device is controlled to spray fragrance or seasoning toward the subject's mouth.

[0037]

[34] A control program that causes a computer to execute the following steps: The step of detecting that the subject has opened his mouth; and In response to detecting that the subject has opened his mouth, the step of controlling a spray device to spray a flavor or seasoning toward the subject's mouth.

[0038] Effects of the Invention It can be controlled in conjunction with the chewing and opening states. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a block diagram showing a schematic configuration of a chewing state determination system according to the first embodiment.

[0040] Figure 2A FIG. 1 is a diagram illustrating feature points extracted from an image including a face.

[0041] Figure 2B It is a diagram explaining the upper feature point, the lower feature point, the left feature point, and the right feature point.

[0042] Figure 3A It is a diagram schematically showing upper feature points and lower feature points.

[0043] Figure 3B It is a diagram schematically showing left feature points and right feature points.

[0044] Figure 4 This is a diagram illustrating an example of a method for determining whether the jaw is open.

[0045] Figure 5 is a diagram schematically showing the longitudinal distance Y when the subject chews. Figure 5 (a), lateral distance X ( Figure 5 (b)) and the difference D ( Figure 5 (c) is a diagram showing an example of the change.

[0046] Figure 6 Schematically shows an example of how the difference D changes when a subject chews.

[0047] Figure 7 Schematically shows an example of how the difference D changes when a subject chews.

[0048] Figure 8 3 is a diagram illustrating the upper lip feature points and the lower lip feature points.

[0049] Figure 9 This is a diagram illustrating an example of a method for determining whether the lips are open.

[0050] Figure 10 This is a diagram illustrating an example of a method for determining whether it is the first occlusion or the second occlusion or later.

[0051] Figure 11 This is a block diagram showing a schematic configuration of a chewing state determination system according to a third embodiment.

[0052] Figure 12A This is a block diagram showing a schematic configuration of a system according to the fourth embodiment.

[0053] Figure 12B It shows Figure 12A A block diagram showing a schematic configuration of a system according to a modification of .

[0054] Figure 13This is a block diagram showing a schematic configuration of a system according to the fifth embodiment. DETAILED DESCRIPTION

[0055] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0056] (First embodiment) Figure 1 This is a block diagram showing a schematic configuration of a chewing state determination system according to the first embodiment. The chewing state determination system includes a camera 1 and a chewing state determination device 2. The chewing state determination device 2 may be a general-purpose computer or a dedicated device.

[0057] The camera 1 captures an image of the subject's face, and the captured image is input to the chewing state determination device 2 .

[0058] The chewing state determination device 2 includes a feature point extraction unit 21, a calculation unit 22, and a determination unit 23. Some or all of these units may be implemented as hardware or software. In the latter case, each function can be implemented by a processor included in the chewing state determination device 2 executing a predetermined program.

[0059] The feature point extraction unit 21 extracts, from an image of the subject's face, an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth. The lower and upper feature points are preferably located on or near a vertical line. Furthermore, the right and left feature points are preferably located on or near a horizontal line.

[0060] As an example of a specific extraction method, the feature point extraction unit 21 first applies a known method (eg, non-patent document 3) to extract Figure 2A As shown in FIG, multiple facial feature points. Figure 2B As shown, the feature point extraction unit 21 extracts a predetermined feature point located above the mouth as an upper feature point. The same applies to the lower feature point, the right feature point, and the left feature point.

[0061] return Figure 1 The calculation unit 22 calculates the distance Y between the upper and lower feature points (hereinafter referred to as the "longitudinal distance") and the distance X between the right and left feature points (hereinafter referred to as the "lateral distance"). Furthermore, the calculation unit 22 calculates the difference D = Y - X between the longitudinal distance Y and the lateral distance X.

[0062] The determination unit 23 determines the subject's chewing state based on the difference D. The chewing state determination in this embodiment can include, for example, determining whether the jaw is open, the degree of jaw opening, and at least one of the following: whether the jaw has begun, is in the middle of, or has completed a single chew. This will be described in detail below.

[0063] First, the determination of whether the jaw is open or not will be described.

[0064] Figure 3A This diagram schematically illustrates the upper and lower characteristic points. (a) shows the jaw closed, and (b) shows the jaw open. As indicated by symbol A, the upper characteristic point moves upward when the jaw opens. On the other hand, as indicated by symbol B, the lower characteristic point moves downward when the jaw opens. Therefore, as the jaw opens, the longitudinal distance Y increases.

[0065] Figure 3B This diagram schematically illustrates the left and right characteristic points. (a) shows the jaw closed, and (b) shows the jaw open. As the jaw opens, the left characteristic point shifts rightward, as indicated by symbol C. On the other hand, as the jaw opens, the right characteristic point shifts leftward, as indicated by symbol D. Therefore, as the jaw opens, the lateral distance X decreases.

[0066] Therefore, the difference D (=longitudinal distance Y - transverse distance X) becomes larger when the jaw is open and smaller when the jaw is closed. Here, the position of each feature point always fluctuates, but the fluctuation is offset by taking the difference.

[0067] Figure 4 This diagram illustrates an example method for determining whether the jaw is open. Determination unit 23 determines whether the jaw is open by comparing difference D with threshold value TH1. Specifically, if difference D is less than threshold value TH1, determination unit 23 determines that the jaw is closed. On the other hand, if difference D is greater than threshold value TH1, determination unit 23 determines that the jaw is open.

[0068] Figure 5 is a diagram schematically showing the longitudinal distance Y when the subject chews. Figure 5 (a), lateral distance X ( Figure 5 (b)) and the difference D ( Figure 5 (c) is a diagram showing an example of the change.

[0069] Assume that the mouth and jaw are closed before time t1, and the longitudinal distance Y and the lateral distance X are initial values ​​(hereinafter referred to as 0). Assume that the subject puts food into the mouth at time t1 and starts the first bite, and ends the first bite at time t4. In the first bite, in order to put food into the mouth, the mouth must be opened (in addition, since it is not necessarily clear which part of the face is referred to by "mouth", and in order to clearly distinguish it from opening the jaw, it is expressed as "opening the lips" below). Therefore, if Figure 5 As shown in (a), the longitudinal distance Y increases significantly, and then decreases after reaching the maximum value Y1. On the other hand, the lateral distance X shrinks when the lips are opened, so Figure 5As shown in (b), the lateral distance X decreases significantly and then increases after reaching the minimum value X1.

[0070] After that, it is assumed that the subject starts the second bite at time t4 and ends the second bite at time t7. During the second bite, the food is already in the mouth, so there is no need to open the lips again. Normally, chewing is done by opening and closing the jaw with the lips closed. Figure 5 As shown in (a), although the longitudinal distance Y increases, the increase is smaller than that of the first bite, and the maximum value Y2 is smaller than Y1. Figure 5 As shown in (b), although the lateral distance X is reduced, the amount of reduction is smaller than that in the first occlusion, and the minimum value X2 (the absolute value) is smaller than X1.

[0071] After the third occlusion, the distances are approximately the same as those of the second occlusion, or the increase in the longitudinal distance Y is smaller, and the decrease in the transverse distance X is also smaller.

[0072] The difference D between the longitudinal distance Y and the transverse distance X is as follows: Figure 5 As shown in (c), the jaw is judged to be open at times t2 to t3 and t4 to t5 when the difference D becomes equal to or greater than the threshold value TH1. The jaw is otherwise judged to be closed.

[0073] Furthermore, threshold TH1 can be set as appropriate, and may be a fixed value, for example. Preferably, it is set based on the difference D when the subject's jaw is closed. As a specific example, threshold TH1 may be set to a value that is slightly greater than a given value of the difference D when the subject's jaw is closed. This setting may be performed externally or automatically by the determination unit 23 based on the image captured by the camera 1.

[0074] Here, one of the characteristics of this embodiment is that the determination of whether the jaw is open is made based not only on the longitudinal distance Y or the lateral distance X but also on the difference D between them. Figure 5 Explain its advantages.

[0075] For example, it is also considered to be based only on the longitudinal distance Y ( Figure 5 (a) is used to determine whether the jaw is open.

[0076] Indeed, when the lips are open (times t1 to t4), the longitudinal distance Y increases significantly, reaching a maximum value Y1. Specifically, when the lips are open, the difference between the longitudinal distances when the jaws are closed and when they are open is large, indicating that the jaws are open.

[0077] However, when the lips are closed (times t4 to t7), the increase in the longitudinal distance Y is small, reaching only a maximum value Y2. Specifically, when the lips are closed, the difference between the longitudinal distances when the jaws are closed and when they are open is not significant. Consequently, setting an appropriate threshold value TH1 can be difficult, and fluctuations in the position of feature points can lead to erroneous judgments.

[0078] The same applies to the case where only the lateral distance X ( Figure 5 B) to determine whether the jaw is open.

[0079] As described above, the inventors of the present application have noticed that when the lips are closed, even if only the longitudinal distance Y or only the lateral distance X is used, it is not possible to accurately determine whether the jaw is open or closed.

[0080] Therefore, in this embodiment, the difference D between the longitudinal distance Y and the lateral distance X is used. Regardless of whether the lips are open or closed, when the jaw is open, the longitudinal distance Y increases and the lateral distance X decreases. Therefore, by taking these differences, the difference between the values ​​of difference D when the jaw is open and when it is closed is amplified. Furthermore, by calculating the difference, fluctuations in the positions of the feature points are offset.

[0081] Thus, even when the lips are closed, the opening and closing of the jaw can be determined with high accuracy by using the difference D. This is one feature of the present embodiment.

[0082] Next, an example of determining the degree of jaw opening as a chewing state will be described.

[0083] Figure 6 This diagram schematically illustrates an example of how difference D changes while a subject is chewing. Assume that chewing begins at time t11 and ends at time t14. The determination unit 23 determines the degree of jaw opening based on a comparison (e.g., ratio) of difference D at each time point with a reference value A0. For example, if the value of difference D at time t12 is D1, the degree of jaw opening is determined as D1 / A0. If the value of difference D at time t13 is D2, the degree of jaw opening is determined as D2 / A0.

[0084] The reference value A0 can be set as appropriate, and can be, for example, a fixed value. Preferably, the reference value A0 is set based on the difference D in the subject's jaw-opening state. As a specific example, the difference D in the widely opened jaw state can be measured for multiple subjects based on chewing data, and the average of these differences or a slightly smaller value can be used as the reference value A0.

[0085] Next, an example will be described in which the determination unit 23 determines whether the chewing state is the start of chewing, the middle of chewing, or the end of chewing during one chewing.

[0086] Figure 7This diagram schematically illustrates an example of how the difference D changes while the subject chews. Assume that chewing begins at time t21 and ends at time t24. The determination unit 23 can determine that occlusion has begun at time t22, when the difference D increases and reaches threshold TH2. Furthermore, the determination unit 23 can determine that occlusion is in progress until time t23, when the difference D decreases to threshold TH3 after reaching threshold TH2. Furthermore, the determination unit 23 can determine that occlusion has ended at time t23, when the difference D decreases to threshold TH3.

[0087] In addition, whether the difference D increases or decreases can be determined by, for example, comparing the difference D at a certain moment with the difference D at the next moment, or by comparing the average movement of multiple moments including a certain moment with the average movement of multiple moments including the next moment.

[0088] The threshold values ​​TH2 and TH3 may be set appropriately. The relationship between them is also arbitrary, and may be TH2>TH3, TH2=TH3, or TH2<TH3.

[0089] As described above, in the first embodiment, since the difference D between the longitudinal distance Y and the lateral distance X is used, the chewing state can be determined with high accuracy.

[0090] (Second embodiment) The second embodiment described below uses the characteristic points of the upper lip and the lower lip in addition to the difference D. The matters described below can be performed separately from the first embodiment or based on the first embodiment. The general structure of the chewing state determination system involved in the second embodiment is the same as Figure 1 The structure shown is the same, and the following description will focus on the differences from the first embodiment.

[0091] The feature point extraction unit 21 of this embodiment extracts feature points corresponding to the upper lip (hereinafter referred to as "upper lip feature points") and feature points corresponding to the lower lip (hereinafter referred to as "lower lip feature points") from an image including the face of the subject. Figure 8 ). The upper lip feature points may be feature points on the upper lip, and the lower lip feature points may be feature points on the lower lip.

[0092] The calculation unit 22 calculates the distance L between the upper lip feature point and the lower lip feature point (hereinafter referred to as “inter-lip distance”).

[0093] The determination unit 23 determines the chewing state based on the difference D and the inter-lip distance L. In the present embodiment, the chewing state may include, for example, whether the lips are open, and whether the chewing state is the first bite or the second bite or later.

[0094] Figure 9This figure illustrates an example method for determining whether the lips are open. Determination unit 23 determines whether the lips are open by comparing the inter-lip distance L with threshold value TH4. Specifically, if the inter-lip distance L is less than threshold value TH4, determination unit 23 determines that the lips are closed. On the other hand, if the inter-lip distance L is greater than threshold value TH4, determination unit 23 determines that the lips are open.

[0095] Figure 10 This figure illustrates an example of a method for determining whether the first bite or the second bite is in progress. In the first bite, the lips are widely opened to put food into the mouth. In contrast, in the second bite and after, the food is already in the mouth, so the lips do not need to be opened.

[0096] As shown in the figure, when the jaws are open and the lips are open, the determination unit 23 determines that it is the first occlusion. In other words, when the difference D is greater than or equal to the threshold value TH1 and the inter-lip distance L is greater than or equal to the threshold value TH4, the determination unit 23 determines that it is the first occlusion.

[0097] On the other hand, when the jaw is open but the lips are closed, the determination unit 23 determines that the second occlusion is in progress. In other words, when the difference D is greater than or equal to the threshold TH1 and the inter-lip distance L is less than the threshold TH4, the determination unit 23 determines that the second occlusion is in progress.

[0098] As described above, in the second embodiment, since the inter-lip distance L is considered in addition to the difference D for determination, it is possible to determine the open / closed state of the lips and whether the state is the first occlusion or the second occlusion or later.

[0099] (Third embodiment) The third embodiment described below performs an output according to the determined chewing state, and in particular, can perform an output according to the chewing state even when nothing has actually been eaten. The following description is simplified for the common parts of the first and second embodiments.

[0100] Figure 11 This is a block diagram showing the schematic structure of a chewing state determination system according to a third embodiment. The chewing state determination system includes an output unit 3 in addition to a camera 1 and a chewing state determination device 2. The output unit 3 can include one or more of a speaker (earphone) 31, a vibrator 32, a display 33, and an aroma diffuser 34. Furthermore, the output unit 3 outputs at least one of an image, text, sound, vibration, and fragrance. Furthermore, the chewing state determination device 2 includes an output control unit 24. The output control unit 24 controls the output unit 3 based on the chewing state so that the output unit 3 outputs an output corresponding to the determined chewing state. Furthermore, the chewing state determination method may be the above-described method or another method.

[0101] For example, the output unit 3 includes a speaker 31 (preferably a directional speaker) that outputs a sound linked to the chewing state. The output unit 3 may also output a sound of a size corresponding to the degree of jaw opening. In addition, the output unit 3 may also output a sound corresponding to whether it is the first bite or the second bite. In addition, a pre-prepared sound file may be output at a volume and reproduction order corresponding to the chewing state, or the output control unit 24 may generate and output a sound corresponding to the chewing state. The output sound may be a sound that matches chewing or a sound that has little relevance to chewing and is unexpected. Through the sound output, the subject can experience the pleasure of the sound reproduced in conjunction with the movement of the jaw by chewing even if he or she has not eaten anything. By setting the output unit 3 as a speaker, the chewing sound can be enjoyed without wearing a device.

[0102] In addition, the output unit 3 may also include a vibrator 32 that outputs vibrations linked to the chewing state. The output unit 3 may also output vibrations of a size corresponding to the degree of jaw opening. In order to output both sound and vibration, the output unit 31 is preferably a bone conduction earphone that serves as both the earphone 31 and the vibrator 32. In this case, an experience such as the sound and vibration affecting the subject's mind is provided. Therefore, even if the subject does not eat anything, by chewing, he or she can experience the feeling of actually chewing food.

[0103] In addition, the output unit 3 may also include a display 33 that displays an image that is linked to the chewing state. The output unit 3 may also output an image corresponding to the degree of jaw opening. In addition, the image may be an image of eating food such as snacks, rice crackers, and cookies in synchronization with chewing. Alternatively, the image may be an image of eating something that is inedible in reality, such as a rock or a building, in synchronization with chewing. In addition, the image may also include a time series change such as an object being eaten in conjunction with chewing being cut or disintegrated. Moreover, the feeling of chewing may be enhanced by displaying an abstract image in a graphic that is linked to the movement of the jaw.

[0104] The output unit 3 may also include a display 33 that displays text linked to the chewing status. The text may also indicate the degree of jaw opening. Alternatively, the text may be any string of characters, such as onomatopoeia or poetry, that corresponds to the chewing status. Furthermore, the determination unit 23 may estimate calorie consumption based on the chewing status and display the calorie consumption as text. This estimation method may employ known techniques (e.g., Non-Patent Document 4); for example, a single chew may be set to 0.014675 kcal.

[0105] The display 33 may be a head-mounted display. Alternatively, the output unit 3 may be a projector that projects an image onto a wall, a plate, or the like.

[0106] Alternatively, the output unit 3 may include an aroma diffuser 34 that sprays fragrance in sync with chewing to deliver aroma. Alternatively, the output unit 3 may include a sheet with a food aroma that the subject can smell. Alternatively, the output unit 3 may include a pendant with aroma that can be hung from the neck, or the subject can inhale the aroma through a straw.

[0107] (Fourth embodiment) Figure 12A This is a block diagram schematically illustrating the configuration of a system according to a fourth embodiment. The system includes a chewing state determination device 2, a control device 11, and a motor 12. The control device 11 may be included in the chewing state determination device 2 or may be another device. Some or all of the functions of the chewing state determination device 2 and the control device 11 may be implemented by a processor executing a program.

[0108] The chewing state determination device 2 determines the chewing state of the subject. The control device 11 controls the motor 12 in conjunction with the chewing state determined by the chewing state determination device 2. For example, the motor 12 opens and closes the driven object in conjunction with the determined opening and closing of the jaw.

[0109] For example, motor 12 is built into a facial (bone) model to move the jaw. Alternatively, motor 12 may be part of an actuator used for operation. More specifically, motor 12 controls the movement of a nutshell, a camera aperture, or barbecue shears. The movement is not limited to up and down movements; it can also be opening and closing in any direction.

[0110] Here, the chewing state determination method of the chewing state determination device 2 may be the above-mentioned method or another method.

[0111] As another example of a method, the chewing state determination device 2 can also determine the subject's chewing state based on the pressure detected by a pressure sensor placed in the subject's ear. An earbud-type pressure sensor, whose resistance value changes with bending (strain), can be used as the pressure sensor. By comparing the bending (resistance value) output from the pressure sensor with a threshold value, the chewing state determination device 2 can determine whether the subject's jaw is open or closed.

[0112] As another example, the chewing state determination device 2 may also determine the chewing state of the subject based on an image obtained by photographing the subject's face. Figure 2ASimilarly to the description, feature points corresponding to the nose and the lower jaw are extracted from the image of the subject's face. Furthermore, the chewing state determination device 2 can determine the subject's chewing state based on the distance between these feature points. For example, the chewing state determination device 2 can determine whether the subject's jaw is open or closed by comparing the distance with a threshold. Alternatively, the chewing state determination device 2 can determine that the jaw is in the process of opening if the distance increases, and that the jaw is in the process of closing if the distance decreases.

[0113] As another example, the chewing state determination device 2 may determine the subject's chewing state based on muscle movement detected by a myoelectric sensor that detects movement of the subject's jaw muscles. Specifically, if jaw muscle movement is detected by the myoelectric sensor, the chewing state determination unit 2 may determine that the subject is chewing.

[0114] The method of determining the chewing state by the chewing state determination device 2 is not limited to the above-described method, and a plurality of determination methods may be combined.

[0115] Figure 12B It shows Figure 12A The system includes an image output device 13 instead of Figure 12A The control device 11 controls the image output device 13 in conjunction with the chewing state determined by the chewing state determination device 2. As a result, the image output device 13 outputs an image corresponding to the chewing state. The image output device 13 is, for example, any display (including a head-mounted display) or a projector.

[0116] As described above, according to the fourth embodiment, the device can be controlled in conjunction with the movement of the jaw of the subject.

[0117] (Fifth embodiment) Figure 13 This is a block diagram schematically illustrating the configuration of a system according to a fifth embodiment. The system includes an opening state determination device 21, a control device 22, and a spray device 23. The control device 22 may be included in the opening state determination device 21 or may be another device. Some or all of the functions of the opening state determination device 21 and the control device 22 may be implemented by a processor executing a program.

[0118] The mouth opening state determining device 21 detects whether the subject has opened his mouth. The detection method is not particularly limited, and the mouth opening state can be determined based on an image obtained by photographing the subject's face. As a specific example, the mouth opening state determining device 21 is used in conjunction with a Figure 2ASimilarly to the description, feature points corresponding to the upper lip and feature points corresponding to the lower lip are extracted from the image of the subject's face. And based on the distance between these feature points, the opening state determination device 21 can determine the opening state of the subject. For example, the opening state determination device 21 can compare the distance with a threshold to determine whether the subject has his or her mouth open or closed. Alternatively, the opening state determination device 21 can determine that the mouth is halfway open if the distance becomes larger, and determine that the mouth is halfway closed if the distance becomes smaller.

[0119] If the subject opens their mouth, the control device 22 controls the spray device 23 to spray fragrance (aromatherapy) or seasoning. Alternatively, the control device 22 may control the spray device 23 in response to a predetermined operation by the subject (e.g., pressing a button on the control device 22).

[0120] The spray device 23 sprays flavorings or seasonings (along with air as needed) into the mouth of the subject under the control of the control device 22 .

[0121] The spray device 23 can also be configured as a fan that blows gas containing fragrances or seasonings. In this case, the control device 22 can turn on the fan when it detects that the subject has opened their mouth, and then turn it off after a predetermined time. Alternatively, the open-mouth determination device 21 can detect the position of the subject's mouth based on an image of the subject's face, and the control device 22 can control the fan so that it blows in a direction corresponding to the mouth position. This on / off control and blowing direction control are performed by controlling a motor built into the fan.

[0122] Alternatively, the spray device 23 may be configured as an air gun that blows gas containing flavorings or seasonings. For example, an air gun has a hole in a barrel covered with rubber. The rubber is repelled, forcing air out of the hole and forming a mass of air. In this case, if the subject's mouth is detected to be open, the control device 22 pulls a trigger that repel the rubber.

[0123] Alternatively, a mechanism for cooling or heating the air to be blown may be built into the spray device 23 .

[0124] Furthermore, if the control device 22 detects that the subject has opened their mouth (in other words, in sync with the control of the spray device 23), it can also generate images, sounds, vibrations, and / or lights. For example, the control device 22 can cause the display to display a picture of food disappearing or produce chewing sounds in response to the subject's mouth opening. This creates a performance that stimulates the visual, auditory, and tactile senses in addition to the olfactory (spices) and taste (seasonings), emphasizing the sensation of eating.

[0125] As described above, according to the fifth embodiment, the device can be controlled in conjunction with the movement of the subject's mouth.

[0126] According to this embodiment, even without actually eating anything, chewing generates output linked to chewing. Therefore, the subject can consume calories by chewing without consuming calories, while also experiencing the simulated chewing texture and tactile sensation of eating.

[0127] Any part or all of the functional units described in this specification may be implemented by a program. The program referred to in this specification may be distributed by non-transitory recording on a computer-readable recording medium, distributed via a communication line such as the Internet (including wireless communication), or distributed in a state where it is installed on any terminal.

[0128] Based on the above description, those skilled in the art may be able to conceive of additional effects and various modifications of the present invention, but the present invention is not limited to the above-mentioned embodiments. Various additions, changes, and partial deletions are possible without departing from the scope of the claims and the conceptual ideas and gist of the present invention derived from their equivalents.

[0129] For example, a device described in this specification as a single device (or component, the same applies hereinafter) (including a device depicted as a single device in the accompanying drawings) may be implemented by multiple devices. Conversely, a device described in this specification as multiple devices (including a device depicted as multiple devices in the accompanying drawings) may be implemented by a single device. Alternatively, a unit or function included in a certain device (e.g., a server) may be partially or entirely included in another device (e.g., a user terminal). Furthermore, a "system" may consist of a single device or two or more devices.

[0130] Furthermore, not all matters described in this specification are essential requirements. In particular, matters described in this specification but not described in the claims may be regarded as optional additional matters.

[0131] In addition, it should be noted that the applicant is only aware of the publicly known inventions described in the documents listed in the "Prior Art Documents" column of this specification, and the present invention does not necessarily aim to solve the problems in the publicly known inventions. The problems to be solved by the present invention should be determined by considering the entire specification. For example, in this specification, when there is a description that a given effect is achieved through a specific structure, problems opposite to the given effect can also be solved. However, this does not necessarily mean that such a specific structure must be a necessary requirement.

[0132] Explanation of symbols 1. Camera 2 Chewing state determination device 21 Feature point extraction unit 22 Computing Department 23 Judgment Department 24 Output control unit 3 Output section 31 Speakers (Headphones) 32 Vibrator 33 Display 34 Aromatherapy Diffuser 11 Control device 12 Electric Motor 13 Image output device 21 Opening state determination device 22 Control Device 23 Spray device.

Claims

1. A chewing state determination device comprising: a feature point extraction section that extracts, from an image including a face of a subject, an upper feature point located above a mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth; a calculation unit that calculates a difference between a longitudinal distance between the upper feature point and the lower feature point and a lateral distance between the left feature point and the right feature point; as well as The determination unit determines the chewing state based on the difference.

2. The chewing state determination device according to claim 1, wherein: The determination unit determines whether the jaw is open based on the difference.

3. The chewing state determination device according to claim 2, wherein: The determination unit determines whether the jaw is open by comparing the difference with a first threshold value.

4. The chewing state determination device according to claim 3, wherein: The first threshold value is determined based on the difference in a state in which the subject closes his jaw.

5. The chewing state determination device according to claim 1, wherein: The determination unit determines the degree of jaw opening based on comparison between the difference and a reference value.

6. The chewing state determination device according to claim 1, wherein: The determination unit determines whether the bite has started, is in the middle of, or has ended during one chewing session based on the difference.

7. The chewing state determination device according to claim 1, wherein: The feature point extraction unit extracts upper lip feature points corresponding to the upper lip and lower lip feature points corresponding to the lower lip, The calculation unit calculates the interlip distance between the upper lip feature point and the lower lip feature point, The determination unit determines a chewing state based on the difference and the inter-lip distance.

8. The chewing state determination device according to claim 7, wherein: The determination unit determines whether the jaw is open based on the difference, The determination unit determines whether the lips are open based on the inter-lip distance. The determination unit determines the chewing state based on whether the jaw is open and whether the lips are open.

9. The chewing state determination device according to claim 8, wherein: When the jaw is open and the lips are open, the determination unit determines that the chewing state is the first bite, When the jaw is open and the lips are closed, the determination unit determines that the chewing state is after the second bite.

10. The chewing state determination device according to claim 7, wherein: The determination unit determines whether the chewing state is the first bite or the second bite or later based on a comparison between the difference and a first threshold value and a comparison between the inter-lip distance and a second threshold value.

11. The chewing state determination device according to claim 1, wherein: The chewing state determination device includes an output control unit configured to control the output unit so as to output at least one of an image, text, sound, vibration, and fragrance corresponding to the determined chewing state.

12. The chewing state determination device according to claim 5, wherein: The chewing state determination device includes an output control unit configured to output at least one of an image, text, sound, and vibration according to the degree of jaw opening.

13. The chewing state determination device according to claim 1, wherein: The determination unit estimates calorie consumption based on the determined chewing state.

14. A chewing state determination system comprising: a camera that captures the subject's face; and The chewing state determination method according to any one of claims 1 to 13.

15. A method for determining a chewing state, comprising: The feature point extraction unit extracts, from the image including the face of the subject, an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth; a step of calculating, by a calculation unit, a difference between a longitudinal distance between the upper feature point and the lower feature point and a transverse distance between the left feature point and the right feature point; as well as The step of: the determining unit determining the chewing state based on the difference.

16. A chewing state determination program, which causes a computer to function as a feature point extraction unit, a calculation unit, and a determination unit. The feature point extraction unit extracts an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth from an image including the face of the subject. The calculation unit calculates a difference between a longitudinal distance between the upper feature point and the lower feature point and a lateral distance between the left feature point and the right feature point. The determination unit determines the chewing state based on the difference.

17. A control system comprising: an opening detection device for detecting whether the subject has opened his mouth; and The control device controls the spray device to spray flavors or seasonings toward the mouth of the subject in response to detecting that the subject has opened his mouth.

18. The control system according to claim 17, wherein: The mouth opening detection device detects whether the subject has opened his mouth based on an image obtained by capturing the subject's face.

19. The control system according to claim 18, wherein: The opening detection device extracts a first feature point corresponding to the upper lip and a second feature point corresponding to the lower lip from the image, The mouth opening detection device detects whether the subject has opened his mouth based on the distance between the first feature point and the second feature point.

20. A control system according to any one of claims 17 to 19, wherein: The spray device is constructed using a fan or an air gun that blows gas including spices or seasonings.

21. The control system according to claim 20, wherein: The mouth detection device detects the position of the subject's mouth based on an image obtained by photographing the subject's face. The control device controls the blowing direction of the fan based on the detected position of the nozzle.

22. A control system according to any one of claims 17 to 19, wherein: The control device generates at least one of an image, a sound, a vibration, and a light in response to detecting that the subject has opened his mouth.

23. A control method comprising the following steps: Detecting when the subject's mouth is open; and In response to detecting that the subject has opened his mouth, a spray device is controlled to spray flavoring or seasoning toward the subject's mouth.

24. A control program causing a computer to execute the following steps: The step of detecting whether the subject has opened his mouth; and In response to detecting that the subject has opened his mouth, the step of controlling a spray device to spray a flavor or seasoning toward the subject's mouth.

25. A control system comprising: a chewing state determining device for determining the chewing state of the subject; and The control device controls the motor or the image output device in conjunction with the determined chewing state.

26. The control system according to claim 25, wherein: The motors move the jaws in the facial model.

27. The control system according to claim 25, wherein: The image output device outputs an image corresponding to the determined chewing state.

28. A control system according to any one of claims 25 to 27, wherein: The chewing state determination device determines the chewing state of the subject based on pressure detected by a pressure sensor placed in the subject's ear.

29. A control system according to any one of claims 25 to 27, wherein: The chewing state determination device determines the chewing state of the subject based on an image obtained by capturing the subject's face.

30. The control system according to claim 29, wherein: The chewing state determination device extracts a first feature point corresponding to the nose and a second feature point corresponding to the lower part of the jaw from the image, The chewing state determination device determines the chewing state of the subject based on the distance between the first feature point and the second feature point.

31. The control system of claim 29, wherein: The chewing state determination device extracts an upper feature point located above the mouth, a lower feature point located below the mouth, a right feature point located to the right of the mouth, and a left feature point located to the left of the mouth from the image. The chewing state determination device calculates the difference between the longitudinal distance between the upper feature point and the lower feature point and the transverse distance between the left feature point and the right feature point, The chewing state determination device determines the chewing state of the subject based on the difference.

32. A control system according to any one of claims 25 to 27, wherein: The chewing state determination device determines the chewing state of the subject based on the movement of the muscle detected by a myoelectric sensor that detects the movement of the muscle of the subject's jaw.

33. A control method comprising the following steps: Determining the subject's chewing status; and The motor or the image output device is controlled in conjunction with the determined chewing state.

34. A control program causing a computer to execute the following steps: a step of determining the chewing state of the subject; and A step of controlling a motor or an image output device in conjunction with the determined chewing state.

Images