Control device, control method, and control determination program

Abstract

To perform control in conjunction with a facial movement, such as mastication or a mouth-opening state among others.SOLUTION: Provided is a control system comprising: a mastication state determination device that determines a mastication state of a subject; and a control device that controls a motor or a video output device in conjunction with the determined mastication state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control state determination device, a control state determination method, and a control state determination program. [Background technology]

[0002] Patent Documents 1 and 2 disclose techniques for determining chewing movements. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-105878 [Patent Document 2] Japanese Patent Publication No. 2022-074671 [Non-patent document 3] https: / / www.jstage.jst.go.jp / article / soshaku1991 / 2 / 1 / 2_1_55 / _pdf / -char / en [Non-patent document 4] https: / / google.github.io / mediapipe / Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to perform control in conjunction with facial movements, particularly chewing and mouth opening. [Means for solving the problem]

[0005] [1] According to one aspect of the present invention, a feature point extraction unit that extracts upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from an image including the face of the subject; a calculation unit that calculates the difference between the vertical distance between the top feature point and the bottom feature point and the horizontal distance between the left feature point and the right feature point; and a determination unit that determines the mastication state based on the difference.

[0006] [2] In the chewing condition determination device according to [1], The determination unit may determine whether the jaws are open or not based on the difference.

[0007] [3] In the chewing condition determination device according to [2], The determination unit may determine whether the jaws are open by comparing the difference with a first threshold value.

[0008] [4] In the chewing condition determination device according to [3], The first threshold may be determined according to the difference when the subject has his / her jaw closed.

[0009] [5] 5. The chewing condition determining device according to claim 1, wherein the determining unit determines the degree of jaw opening based on a comparison between the difference and a reference value.

[0010] [6] In the chewing condition determination device according to any one of [1] to [5], The determination unit may determine, based on the difference, whether a single chewing has started, is in progress, or has ended.

[0011] [7] In the chewing condition determination device according to any one of [1] to [6], the feature point extraction unit extracts upper lip feature points corresponding to the upper lip and lower lip feature points corresponding to the lower lip; the calculation unit calculates an inter-lip distance between the upper lip feature point and the lower lip feature point, The determination unit may determine the chewing state based on the difference and the interlip distance.

[0012] [8] In the chewing condition determination device according to [7], The determination unit determining whether the jaws are open or not based on the difference; determining whether the lips are open or not based on the inter-lip distance; The chewing state may be determined based on whether the jaw is open or not and whether the lips are open or not.

[0013] [9] In the chewing condition determination device according to [8], The determination unit If the jaw is open and the lips are open, the chewing state is determined to be the first bite. If the jaw is open and the lips are closed, the chewing state may be determined to be the second or subsequent bite.

[0014]

[10] In the chewing condition determination device according to [7], The determination unit may determine whether the chewing state is the first bite or a second or subsequent bite based on a comparison between the difference and a first threshold value and a comparison between the interlip distance and a second threshold value.

[0015]

[11] In the chewing condition determination device according to any one of [1] to

[10] , An output control unit may be provided that controls the output unit so that at least one of video, text, audio, vibration, and scent is output according to the determined chewing state.

[0016]

[12] In the chewing condition determination device according to [5], The device may include an output control unit that outputs at least one of video, text, audio, and vibration according to the degree of jaw opening.

[0017]

[13] In the chewing condition determination device according to any one of [1] to

[12] , The determination unit may estimate calories burned based on the determined chewing state.

[0018]

[14] According to one aspect of the present invention, A camera that photographs the subject's face, [1] to

[13] . A mastication condition determination system comprising the mastication condition determination device according to any one of [1] to

[13] .

[0019]

[15] According to one aspect of the present invention, a step in which a feature point extraction unit extracts upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from an image including the face of the subject; a calculation unit calculating a difference between a vertical distance between the top feature point and the bottom feature point and a horizontal distance between the left feature point and the right feature point; and a step in which the determining unit determines the mastication state based on the difference.

[0020]

[16] According to one aspect of the present invention, Computer, a feature point extraction unit that extracts upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from an image including the face of the subject; a calculation unit that calculates the difference between the vertical distance between the top feature point and the bottom feature point and the horizontal distance between the left feature point and the right feature point; A mastication state determination program is provided that functions as a determination unit that determines the mastication state based on the difference.

[0021]

[17] a mastication state determination device for determining the mastication state of a subject; a control device that controls a motor or a video output device in accordance with the determined chewing state.

[0022]

[18] The control system described in

[17] , wherein the motor operates the jaw of a facial model.

[0023]

[19] The control system according to

[17] , wherein the video output device outputs a video according to the determined chewing state.

[0024]

[20] The control system according to any one of

[17] to

[19] , wherein the chewing state determination device determines the chewing state of the subject based on pressure detected by a pressure sensor inserted into the ear of the subject.

[0025] [twenty one] The control system according to any one of

[17] to

[20] , wherein the mastication state determination device determines the mastication state of the subject based on an image of the subject's face.

[0026] [twenty two] The mastication state determination device includes: extracting a first feature point corresponding to a nose and a second feature point corresponding to a lower chin from the image; The control system described in

[21] determines the subject's chewing state based on the distance between the first feature point and the second feature point.

[0027] [twenty three] The mastication state determination device includes: extracting upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from the image; calculating a difference between a vertical distance between the top feature point and the bottom feature point and a horizontal distance between the left feature point and the right feature point; The control system described in

[21] determines the subject's chewing state based on the difference.

[0028] [twenty four] The mastication state determination device determines the mastication state of the subject based on muscle movement detected by an electromyography sensor that detects muscle movement in the jaw of the subject. A control system according to any one of

[17] to

[23] .

[0029] [twenty five] The chewing state of the subject is determined. A control method for controlling a motor or a video output device in conjunction with the determined chewing state.

[0030]

[26] On the computer, determining the mastication state of the subject; and a step of controlling a motor or a video output device in accordance with the determined chewing state.

[0031]

[27] an opening detection device that detects when the subject opens their mouth; A control system comprising: a control device that controls a spray device that sprays a fragrance or seasoning toward the subject's mouth in response to detection that the subject has opened their mouth.

[0032]

[28] The control system described in

[27] , wherein the mouth opening detection device detects that the subject has opened their mouth based on an image of the subject's face.

[0033]

[29] The opening detection device extracting a first feature point corresponding to the upper lip and a second feature point corresponding to the lower lip from the image; The control system described in

[28] detects that the subject has opened their mouth based on the distance between the first feature point and the second feature point.

[0034]

[30] The control system according to any one of

[27] to

[29] , wherein the spray device is configured using a fan or an air cannon that blows gas containing a fragrance or seasoning.

[0035]

[31] the mouth opening detection device detects the position of the subject's mouth based on an image of the subject's face; The control system described in

[30] , wherein the control device controls the direction of air blown by the fan based on the detected position of the mouth.

[0036]

[32] A control system described in any of

[27] to

[30] , wherein the control device generates at least one of an image, a sound, a vibration, and a light in response to detecting that the subject has opened their mouth.

[0037]

[33] It detects when the subject opens their mouth, A control method for controlling a spray device that sprays a flavoring or seasoning toward a subject's mouth in response to detecting that the subject has opened their mouth.

[0038]

[34] On the computer, detecting when the subject opens their mouth; a control program for executing a step of controlling a spray device that sprays a fragrance or seasoning toward the subject's mouth in response to detection that the subject has opened their mouth; [Effects of the Invention]

[0039] It is possible to control the mouth opening in conjunction with chewing and mouth opening. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a mastication state determination system according to a first embodiment. [Figure 2A] FIG. 1 is a diagram illustrating feature points extracted from an image including a face. [Figure 2B] 4A and 4B are diagrams illustrating upper feature points, lower feature points, left feature points, and right feature points. [Figure 3A] FIG. 2 is a diagram schematically showing upper and lower feature points; [Figure 3B] FIG. 3 is a diagram schematically showing left and right feature points. [Figure 4] 10A and 10B are diagrams illustrating an example of a method for determining whether the jaw is open or not. [Figure 5] Schematic diagram showing an example of the change in vertical distance Y (Fig. 5(a)), horizontal distance X (Fig. 5(b)), and difference D (Fig. 5(c)) when a subject chews. [Figure 6] FIG. 10 is a diagram schematically showing an example of the change in the difference D when a subject chews. [Figure 7] FIG. 10 is a diagram schematically showing an example of the change in the difference D when a subject chews. [Figure 8] FIG. 4 is a diagram illustrating upper lip feature points and lower lip feature points. [Figure 9] 10A and 10B are diagrams illustrating an example of a method for determining whether lips are open. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for determining whether it is the first bite or the second or subsequent bite. [Figure 11] FIG. 10 is a block diagram showing a schematic configuration of a mastication state determination system according to a third embodiment. [Figure 12A] FIG. 10 is a block diagram showing a schematic configuration of a system according to a fourth embodiment. [Figure 12B] FIG. 12B is a block diagram showing a schematic configuration of a system according to a modification of FIG. 12A. [Figure 13] FIG. 11 is a block diagram showing a schematic configuration of a system according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0041] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0042] (First embodiment) 1 is a block diagram showing a schematic configuration of a mastication state determination system according to the first embodiment. The mastication state determination system includes a camera 1 and a mastication state determination device 2. The mastication state determination device 2 may be a general-purpose computer or a dedicated device.

[0043] The camera 1 captures an image of the subject's face. The captured image is input to the chewing condition determination device 2.

[0044] The mastication state determination device 2 includes a feature point extraction unit 21, a calculation unit 22, and a determination unit 23. Some or all of these may be implemented in hardware or software. In the latter case, each function can be realized by a processor included in the mastication state determination device 2 executing a predetermined program.

[0045] The feature point extraction unit 21 extracts upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from an image containing the subject's face. The lower and upper feature points are preferably located on or near a vertical line. The right and left feature points are preferably located on or near a horizontal line.

[0046] As a specific example of the extraction method, the feature point extraction unit 21 first applies a known method (for example, Non-Patent Document 3) to extract a large number of facial feature points as shown in Fig. 2A. Then, the feature point extraction unit 21 extracts predetermined feature points above the mouth as upper feature points, as shown in Fig. 2B. The same applies to lower feature points, right feature points, and left feature points.

[0047] 1, the calculation unit 22 calculates the distance Y between the top feature point and the bottom feature point (hereinafter referred to as the "vertical distance") and the distance X between the right feature point and the left feature point (hereinafter referred to as the "horizontal distance"). Furthermore, the calculation unit 22 calculates the difference D=YX between the vertical distance Y and the horizontal distance X.

[0048] The determination unit 23 determines the mastication state of the subject based on the difference D. The determination of the mastication state in this embodiment may include, for example, determining at least one of whether the jaw is open or not, the degree to which the jaw is open, and whether a single chew is at the beginning, middle, or end of chewing. This will be explained in detail below.

[0049] First, the determination of whether the jaw is open or not will be described. FIG. 3A is a diagram showing the upper and lower characteristic points, with (a) in the jaw closed state and (b) in the jaw open state. As indicated by symbol A, the upper characteristic point moves upward when the jaw opens. On the other hand, as indicated by symbol B, the lower characteristic point moves downward when the jaw opens. Therefore, when the jaw opens, the vertical distance Y increases.

[0050] FIG. 3B is a diagram showing the left and right feature points, with (a) in the jaw closed state and (b) in the jaw open state. As indicated by symbol C, the left feature point moves to the right when the jaw opens. On the other hand, as indicated by symbol D, the right feature point moves to the left when the jaw opens. Therefore, when the jaw opens, the lateral distance X becomes smaller.

[0051] Therefore, the difference D (= vertical distance Y - horizontal distance X) increases when the jaw opens and decreases when the jaw closes. Here, the position of each feature point is constantly fluctuating, but by taking the difference, the fluctuation is canceled out.

[0052] 4 is a diagram illustrating an example of a method for determining whether the jaws are open or not. The determination unit 23 determines whether the jaws are open or not by comparing the difference D with a threshold TH1. Specifically, if the difference D is less than the threshold TH1, the determination unit 23 determines that the jaws are closed. On the other hand, if the difference D is equal to or greater than the threshold TH1, the determination unit 23 determines that the jaws are open.

[0053] FIG. 5 is a diagram showing an example of changes in vertical distance Y (FIG. 5(a)), horizontal distance X (FIG. 5(b)), and difference D (FIG. 5(c)) when a subject chews.

[0054] Before time t1, the mouth and jaw are closed, and the vertical distance Y and horizontal distance X are initial values ​​(hereinafter, 0). Assume that the subject puts food in their mouth and starts the first bite at time t1, and finishes the first bite at time t4. In the first bite, the mouth is opened to put food into the mouth (note that because it is not always clear which part of the face "mouth" refers to and to clearly distinguish it from opening the jaw, hereinafter, we will refer to this as "opening the lips"). Therefore, as shown in Figure 5(a), the vertical distance Y increases significantly, reaches a maximum value Y1, and then decreases. On the other hand, the horizontal distance X shrinks when the lips are opened, so as shown in Figure 5(b), the horizontal distance X decreases significantly, reaches a minimum value X1, and then increases.

[0055] Then, suppose the subject begins the second bite at time t4 and finishes it at time t7. Because food is already in the mouth during the second bite, there is no need to open the lips again; chewing is usually done by opening and closing the jaw with the lips closed. Therefore, as shown in Figure 5(a), although the vertical distance Y increases, the increase is smaller than in the first bite, and the maximum value Y2 is smaller than Y1. On the other hand, as shown in Figure 5(b), although the horizontal distance X decreases, the decrease is smaller than in the first bite, and the minimum value X2 (absolute value) is smaller than X1.

[0056] From the third bite onwards, the results are similar to the second bite, or the increase in vertical distance Y becomes smaller and the decrease in horizontal distance X becomes smaller.

[0057] Such a difference D between the vertical distance Y and the horizontal distance X is shown in Figure 5(c). When the difference D is equal to or greater than the threshold value TH1, it is determined that the jaws are open from time t2 to t3 and from t4 to t5. At other times, it is determined that the jaws are closed.

[0058] The threshold value TH1 may be set appropriately, for example, as a fixed value. Preferably, it is set according to the difference D when the subject has his / her jaw closed. As a specific example, the threshold value TH1 may be set to a value that is a predetermined value greater than the difference D when the subject has his / her jaw closed. Such setting may be performed externally, or may be automatically set by the determination unit 23 based on the image captured by the camera 1.

[0059] One of the features of this embodiment is that the determination of whether the jaws are open is based not only on the vertical distance Y or the horizontal distance X, but also on the difference D between them. The advantage of this will be explained with reference to FIG. 5.

[0060] For example, it may be possible to determine whether the jaws are open or not based only on the vertical distance Y (FIG. 5(a)).

[0061] Indeed, if the lips are open (times t1 to t4), the vertical distance Y increases significantly and reaches a maximum value Y1. In other words, if the lips are open, the difference between the vertical distance when the jaw is closed and the vertical distance when the jaw is open is large, and it can be determined that the jaw is open.

[0062] However, when the lips are closed (times t4 to t7), the increase in vertical distance Y is small and only reaches a maximum value Y2. In other words, when the lips are closed, the difference between the vertical distance when the jaw is closed and the vertical distance when the jaw is open is not very large. This makes it difficult to set an appropriate threshold TH1, and there is a risk of erroneous determination due to the influence of fluctuations in the positions of feature points, etc.

[0063] It is also possible to determine whether the jaws are open or not based solely on the lateral distance X (FIG. 5(b)).

[0064] Thus, the inventors of the present application have noticed that when the lips are closed, it is not possible to accurately determine whether the jaw is open or closed by using only the vertical distance Y or only the horizontal distance X.

[0065] Therefore, in this embodiment, the difference D between the vertical distance Y and the horizontal distance X is used. Regardless of whether the lips are open or closed, when the jaw opens, the vertical distance Y increases and the horizontal distance X decreases. Therefore, by taking these differences, the difference between the value of the difference D when the jaw is open and the value of the difference D when the jaw is closed is amplified. Moreover, by calculating the difference, fluctuations in the position of the feature points are offset.

[0066] As a result, even when the lips are closed, the opening and closing of the jaw can be determined with high accuracy by using the difference D. This is one feature of this embodiment. Next, an example will be shown in which the degree of jaw opening is determined as the chewing state.

[0067] FIG. 6 is a diagram showing an example of changes in the difference D when a subject chews. Assume that chewing begins at time t11 and one chewing session ends at time t14. The determination unit 23 determines the degree of jaw opening based on a comparison (e.g., ratio) of the difference D at each time with a reference value A0. For example, the value of the difference D at time t12 is D1, and the degree of jaw opening is determined to be D1 / A0. The value of the difference D at time t13 is D2, and the degree of jaw opening is determined to be D2 / A0.

[0068] The reference value A0 may be set appropriately, and may be, for example, a fixed value. Preferably, the reference value A0 is set according to the difference D when the subject opens their jaw. As a specific example, the difference D when the jaw is widely opened may be measured from the chewing data of multiple subjects, and the average value or a value slightly smaller than that may be used as the reference value A0.

[0069] Next, an example will be shown in which the determination unit 23 determines whether the mastication state is the beginning of mastication, the middle of mastication, or the end of mastication in one mastication.

[0070] FIG. 7 is a diagram schematically showing an example of changes in difference D when a subject chews. Assume that chewing begins at time t21 and one chewing session ends at time t24. The judgment unit 23 can determine that time t22, when difference D increases and reaches threshold value TH2, is the start of chewing. The judgment unit 23 can then determine that the period from when difference D reaches threshold value TH2 until time t23, when difference D decreases to threshold value TH3, is the middle of chewing. The judgment unit 23 can then determine that time t23, when difference D decreases to threshold value TH3, is the end of chewing.

[0071] Whether the difference D is increasing or decreasing may be determined, for example, by comparing the difference D at a certain point in time with the difference D at the next point in time, or by comparing a moving average of multiple points in time including a certain point in time with a moving average of multiple points in time including the next point in time.

[0072] The thresholds TH2 and TH3 may be set as appropriate. The magnitude relationship between them may be arbitrary, and may be TH2 > TH3, TH2 = TH3, or TH2 <TH3でもよい。

[0073] As described above, in the first embodiment, the difference D between the vertical distance Y and the horizontal distance X is used, so that the mastication state can be determined with high accuracy.

[0074] (Second embodiment) The second embodiment described next uses feature points of the upper and lower lips in addition to the difference D. The matters described below may be performed separately from the first embodiment, or may be performed in addition to the first embodiment. The schematic configuration of the chewing state determination system according to the second embodiment is the same as that shown in Fig. 1, and the following description will focus on the differences from the first embodiment.

[0075] The feature point extraction unit 21 of this embodiment extracts feature points corresponding to the upper lip (hereinafter referred to as "upper lip feature points") and lower lip feature points corresponding to the lower lip (hereinafter referred to as "lower lip feature points") from an image including the face of the subject (FIG. 8). The upper lip feature points may be feature points on the upper lip, and the lower lip feature points may be feature points on the lower lip.

[0076] The calculation unit 22 calculates the distance L between the upper lip feature point and the lower lip feature point (hereinafter referred to as the "inter-lip distance").

[0077] The determination unit 23 determines the chewing state based on the difference D and the interlip distance L. In this embodiment, the chewing state may include, for example, determining whether the lips are open or not, and whether it is the first bite or the second bite or later.

[0078] 9 is a diagram illustrating an example of a method for determining whether or not the lips are open. The determination unit 23 determines whether or not the lips are open by comparing the inter-lip distance L with a threshold value TH4. Specifically, if the inter-lip distance L is less than the threshold value TH4, the determination unit 23 determines that the lips are closed. On the other hand, if the inter-lip distance L is equal to or greater than the threshold value TH4, the determination unit 23 determines that the lips are open.

[0079] 10 is a diagram illustrating an example of a method for determining whether it is the first bite or the second or subsequent bites. This utilizes the fact that in the first bite, the lips are opened wide to put food into the mouth, whereas in the second or subsequent bites, there is already food in the mouth and so there is no need to open the lips.

[0080] As shown in the figure, when the jaw is open and the lips are open, the determination unit 23 determines that it is the first bite. In other words, when the difference D is equal to or greater than the threshold value TH1 and the inter-lip distance L is equal to or greater than the threshold value TH4, the determination unit 23 determines that it is the first bite.

[0081] On the other hand, if the jaw is open but the lips are closed, the determination unit 23 determines that the bite is the second or subsequent bite. In other words, if the difference D is equal to or greater than the threshold TH1 and the inter-lip distance L is less than the threshold TH4, the determination unit 23 determines that the bite is the second or subsequent bite.

[0082] In this way, in the second embodiment, the determination is made taking into consideration the inter-lip distance L in addition to the difference D, so it is also possible to determine whether the lips are open or closed, and whether it is the first bite or the second bite or later.

[0083] (Third embodiment) The third embodiment described below provides an output according to the determined chewing state, and in particular, enables output according to the chewing state even when nothing has actually been eaten. Below, the explanation common to the first and second embodiments will be simplified.

[0084] 11 is a block diagram showing a schematic configuration of a mastication state determination system according to the third embodiment. This mastication state determination system further includes an output unit 3 in addition to a camera 1 and a mastication state determination device 2. The output unit 3 may include one or more of a speaker (earphone) 31, a vibrator 32, a display 33, and an aroma diffuser 34. The output unit 3 outputs at least one of video, text, sound, vibration, and scent. The mastication state determination device 2 also includes an output control unit 24. The output control unit 24 controls the output unit 3 based on the mastication state so that the output unit 3 performs output according to the determined mastication state. The method for determining the mastication state may be the one described above or another method.

[0085] For example, the output unit 3 includes a speaker 31 (preferably a directional speaker) and outputs sound linked to the chewing state. The output unit 3 may output a sound at a volume corresponding to the degree of jaw opening. The output unit 3 may also output a sound corresponding to whether it is the first bite or the second bite or later. Pre-prepared audio files may be output at a volume or playback order corresponding to the chewing state, or the output control unit 24 may generate and output sounds corresponding to the chewing state. The output sound may be appropriate for chewing, or may be unexpected and unrelated to chewing. By outputting audio, the subject can enjoy the pleasure of hearing sounds played in conjunction with the movement of the jaw by chewing, even if they are not eating anything. By using a speaker as the output unit 3, the subject can enjoy the sound of chewing without wearing a device.

[0086] The output unit 3 may also include a vibrator 32 and output vibrations linked to the chewing state. The output unit 3 may output vibrations of a magnitude corresponding to the degree to which the jaw is open. In order to output both sound and vibration, the output unit 31 is preferably a bone conduction earphone that also functions as the earphone 31 and the vibrator 32. In this case, the subject is provided with an experience in which the sound and vibration resonate inside his or her head. Therefore, even if the subject is not eating anything, by chewing, the subject can experience the sensation of actually chewing food.

[0087] The output unit 3 may also include a display 33 and display an image linked to the state of chewing. The output unit 3 may output an image according to the degree of jaw opening. The image may also be an image of eating food such as snacks, rice crackers, or cookies in synchronization with chewing. Alternatively, the image may be an image of eating something that cannot be eaten in reality, such as a rock or a building, in synchronization with chewing. The image may also include a time-series change in which the food being eaten is chipped or crumbled in synchronization with chewing. Furthermore, the sensation of chewing may be enhanced by displaying a graphical and abstract image linked to jaw movement.

[0088] The output unit 3 may also include a display 33, which may display text linked to the chewing state. The text may indicate the degree to which the jaw is open. The text may also be any character string, such as onomatopoeia or poetry, corresponding to the chewing state. The determination unit 23 may also estimate calories burned based on the chewing state and display this calorie burned in text. The estimation method may be a known technique (for example, Non-Patent Document 4), and for example, one chewing session may be set to 0.014675 kcal.

[0089] The display 33 may be a head-mounted display. Alternatively, the output unit 3 may be a projector that projects an image onto a wall, a plate, or the like.

[0090] The output unit 3 may also include an aroma diffuser 34 that outputs a scent by spraying a fragrance in synchronization with the chewing state. The output unit 3 may also include a food-scented sheet that the subject sniffs. Alternatively, the output unit 3 may be an aroma-infused pendant worn around the neck, or the aroma may be inhaled through a straw.

[0091] (Fourth embodiment) 12A is a block diagram showing a schematic configuration of a system according to the fourth embodiment. This system includes a mastication state determination device 2, a control device 11, and a motor 12. The control device 11 may be a device included in the mastication state determination device 2, or may be a separate device. Some or all of the functions of the mastication state determination device 2 and the control device 11 may be realized by a processor executing a program.

[0092] The mastication state determination device 2 determines the mastication state of the subject. The control device 11 controls the motor 12 in conjunction with the mastication state determined by the mastication state determination device 2. As an example, the motor 12 opens and closes a driven object in conjunction with the determined opening and closing of the jaw.

[0093] The motor 12 is built into, for example, a model face (skeleton) and moves the jaw. Alternatively, the motor 12 may be part of an actuator for a task. As a more specific example, the motor 12 controls the movement of a nutcracker, a camera aperture, or a pair of scissors for barbecuing. The movement here is not limited to up and down movement, but may be an opening and closing movement in any direction.

[0094] Here, the method for determining the mastication state by the mastication state determining device 2 may be the one described above or another method.

[0095] As another example of a method, the mastication state determination device 2 may determine the mastication state of the subject based on the pressure detected by a pressure sensor inserted in the ear of the subject. An earplug-type pressure sensor whose resistance value changes depending on bending (strain) can be applied as the pressure sensor. By comparing the bending (resistance value) output from the pressure sensor with a threshold value, the mastication state determination device 2 can determine whether the subject's jaw is open or closed.

[0096] As another example, the chewing state determination device 2 may determine the chewing state of the subject based on an image of the subject's face. As a specific example, the chewing state determination device 2 extracts a feature point corresponding to the nose and a feature point corresponding to the lower part of the chin from the image of the subject's face in the same manner as described using FIG. 2A. Then, based on the distance between these feature points, the chewing state determination device 2 can determine the chewing state of the subject. For example, the chewing state determination device 2 can determine whether the subject's jaw is open or closed by comparing the distance with a threshold. Alternatively, the chewing state determination device 2 can determine that the jaw is in the process of opening if the distance is large, and that the jaw is in the process of closing if the distance is small.

[0097] As another example, the mastication state determination device 2 may determine the mastication state of the subject based on muscle movement detected by an electromyographic sensor that detects movement of the jaw muscles of the subject. As a specific example, when the electromyographic sensor detects movement of the jaw muscles, the mastication state determination unit 2 can determine that mastication is being performed.

[0098] The method for determining the mastication state by the mastication state determining device 2 is not limited to the above. In addition, a plurality of determination methods may be combined.

[0099] Fig. 12B is a block diagram showing a schematic configuration of a system according to a modified example of Fig. 12A. This system includes a video output device 13 instead of the motor 12 of Fig. 12A. The control device 11 controls the video output device 13 in conjunction with the mastication state determined by the mastication state determination device 2. This causes the video output device 13 to output an image according to the mastication state. The video output device 13 is, for example, any display (including a head-mounted display) or a projector.

[0100] As described above, according to the fourth embodiment, the device can be controlled in conjunction with the movement of the subject's jaw.

[0101] (Fifth embodiment) 13 is a block diagram showing a schematic configuration of a system according to a fifth embodiment. This system includes an opening state determination device 21, a control device 22, and a spray device 23. The control device 22 may be a device included in the opening state determination device 21, or may be a separate device. Some or all of the functions of the opening state determination device 21 and the control device 22 may be realized by a processor executing a program.

[0102] The mouth-opening state determination device 21 detects that the subject has opened their mouth. There are no particular limitations on the detection method, and for example, the mouth-opening state may be determined based on an image of the subject's face. As a specific example, the mouth-opening state determination device 21 extracts feature points corresponding to the upper lip and feature points corresponding to the lower lip from an image of the subject's face, in the same manner as described with reference to FIG. 2A. Then, based on the distance between these feature points, the mouth-opening state determination device 21 can determine the mouth-opening state of the subject. For example, the mouth-opening state determination device 21 can compare the distance with a threshold and determine whether the subject has their mouth open or closed. Alternatively, the mouth-opening state determination device 21 can determine that the mouth is in the process of opening if the distance is increasing, and that the mouth is in the process of closing if the distance is decreasing.

[0103] When it is detected that the subject has opened his / her mouth, the control device 22 controls the spray device 23 to spray a flavor (aroma) or a seasoning. The control device 22 may also control the spray device 23 in response to a predetermined operation by the subject (for example, pressing a button provided on the control device 22).

[0104] The spray device 23, under the control of the control device 22, sprays a flavoring or seasoning (and optionally air) into the subject's mouth.

[0105] The spray device 23 may be configured using a fan that blows gas containing a fragrance or seasoning. In this case, when it is detected that the subject has opened their mouth, the control device 22 may turn on the fan and turn it off after a predetermined time has passed. Alternatively, the mouth-opening state determination device 21 may detect the position of the mouth from an image of the subject's face, and the control device 22 may control the fan so that the air is blown in a direction corresponding to the position of the mouth. Such on / off control and control of the air blowing direction are performed by controlling a motor built into the fan.

[0106] Alternatively, the spray device 23 may be configured using an air cannon that blows gas containing a fragrance or seasoning. The air cannon works by, for example, covering one side of a tube hole with rubber, and by snapping the rubber, air is pushed out through the open hole, forming a mass of air. In this case, when it is detected that the subject has opened their mouth, the control device 22 may pull a trigger to snap the rubber.

[0107] The air being blown may be at room temperature, or the spray device 23 may include a mechanism for cooling or heating the air being blown.

[0108] Furthermore, when it is detected that the subject has opened their mouth (in other words, in synchronization with controlling the spray device 23), the control device 22 may generate images, sounds, vibrations, and / or lights. As an example, the control device 22 may display a screen on the display showing food disappearing or generate chewing sounds in response to the subject opening their mouth. This provides stimulation to the senses of sight, hearing, and touch in addition to the sense of smell (fragrances) and taste (seasonings), thereby emphasizing the feeling of having eaten.

[0109] As described above, according to the fifth embodiment, the device can be controlled in conjunction with the movement of the subject's mouth.

[0110] According to this embodiment, even if the subject is not actually eating anything, chewing produces an output linked to the chewing action, allowing the subject to burn calories through chewing without actually ingesting any calories, while experiencing the simulated sensation of chewing, such as the texture and texture of the food.

[0111] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed by being non-temporarily recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on any terminal.

[0112] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents.

[0113] For example, what is described in this specification as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by a single device. Alternatively, some or all of the means or functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal). Furthermore, a "system" may consist of a single device, or may consist of two or more devices.

[0114] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0115] It should be noted that the applicant is merely aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this specification, and the present invention does not necessarily aim to solve the problems of the disclosed inventions. The problem that the present invention aims to solve should be determined by taking into consideration the entire specification. For example, if this specification states that a specific configuration achieves a certain effect, it can also be said that the present invention solves a problem that is the reverse of that effect. However, it is not necessarily intended that such a specific configuration be an essential requirement. [Explanation of symbols]

[0116] 1 camera 2. Chewing condition determination device 21 Feature point extraction unit 22 Calculation section 23 Judgment section 24 Output control section 3 Output section 31 Speaker (earphone) 32 Vibrator 33 Display 34 Aroma Diffuser 11 Control device 12 motors 13 Video output device 21 Opening condition determination device 22 Control device 23 Spray equipment

Claims

1. an opening detection device that detects when the subject opens their mouth; A control system comprising: a control device that controls a spray device that sprays a fragrance or seasoning toward the subject's mouth in response to detection that the subject has opened their mouth.

2. The control system according to claim 1 , wherein the mouth opening detection device detects whether the subject has opened their mouth based on an image of the subject's face.

3. The opening detection device extracting a first feature point corresponding to an upper lip and a second feature point corresponding to a lower lip from the image; The control system according to claim 2 , wherein the control system detects that the subject has opened their mouth based on the distance between the first feature point and the second feature point.

4. 4. The control system according to claim 1, wherein the spraying device is configured using a fan or an air cannon that blows gas containing a fragrance or seasoning.

5. the mouth opening detection device detects the position of the subject's mouth based on an image of the subject's face; The control system according to claim 4 , wherein the control device controls the direction of airflow from the electric fan based on the detected position of the mouth.

6. 4. The control system according to claim 1, wherein the control device generates at least one of an image, a sound, a vibration, and a light in response to detection that the subject has opened their mouth.

7. It detects when the subject opens their mouth, A control method for controlling a spray device that sprays a flavoring or seasoning toward a subject's mouth in response to detecting that the subject has opened their mouth.

8. On the computer, detecting when the subject opens their mouth; a control program for executing a step of controlling a spray device that sprays a fragrance or seasoning toward the subject's mouth in response to detection that the subject has opened their mouth;

9. a mastication state determination device for determining the mastication state of a subject; a control device that controls a motor or a video output device in accordance with the determined chewing state.

10. 10. The control system of claim 9, wherein the motor operates a jaw of a facial model.

11. The control system according to claim 9 , wherein the video output device outputs a video according to the determined chewing state.

12. 12. The control system according to claim 9, wherein the mastication state determination device determines the mastication state of the subject based on pressure detected by a pressure sensor inserted in the ear of the subject.

13. The control system according to claim 9 , wherein the mastication state determination device determines the mastication state of the subject based on an image of the face of the subject.

14. The mastication state determination device includes: extracting a first feature point corresponding to a nose and a second feature point corresponding to a lower chin from the image; The control system according to claim 13 , wherein the mastication state of the subject is determined based on the distance between the first feature point and the second feature point.

15. The mastication state determination device includes: extracting upper feature points above the mouth, lower feature points below the mouth, right feature points to the right of the mouth, and left feature points to the left of the mouth from the image; calculating a difference between a vertical distance between the top feature point and the bottom feature point and a horizontal distance between the left feature point and the right feature point; The control system according to claim 13 , wherein the mastication state of the subject is determined based on the difference.

16. 12. The control system according to claim 9, wherein the mastication state determination device determines the mastication state of the subject based on muscle movement detected by an electromyographic sensor that detects muscle movement of the jaw of the subject.

17. The chewing state of the subject is determined. A control method for controlling a motor or a video output device in conjunction with the determined chewing state.

18. On the computer, determining the mastication state of the subject; and a step of controlling a motor or a video output device in accordance with the determined chewing state.

Images