Chewing number of times measuring device, chewing number of times measuring method, and program

Abstract

To provide a chewing frequency measuring device, a chewing frequency measuring method, and a program for accurately measuring the chewing frequency during a meal.SOLUTION: The mastication count measurement device measures a face image of a user, and acquires motion specification information that is information output on the basis of the measured face image and specifies a motion of a specific part included in the face. Then, the motion of the user is specified based on at least one of the plurality of variables included in the motion specification information, and the number of chews is measured based on the fact that the motion in the opening and closing direction of the jaw based on chewing is specified as the motion of the user. The user's motion is specified using a learning model that has learned, as teacher data, data in which the temporal displacement of the value of the variable included in the motion specification information is associated with the user's motion.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a chewing frequency measuring device, a chewing frequency measuring method, and a program. [Background technology]

[0002] Various techniques have been proposed for accurately determining a person's mastication state. Patent Document 1 discloses a mastication state determination device that determines the mastication state based on the difference between the vertical distance between the upper and lower feature points and the horizontal distance between the left and right feature points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7436720 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described mastication state determining device is still insufficient in terms of accurately measuring the number of chews during a meal.

[0005] The present invention is devised to solve the above-mentioned problems, and has an object to provide a chewing frequency measuring device, a chewing frequency measuring method, and a program that can accurately measure the number of chews during a meal. [Means for solving the problem]

[0006] The chewing frequency measuring device according to the first aspect of the present invention comprises: a specific information acquisition unit that measures a face image of a user and acquires motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information that is output based on the measured face image; a motion determination unit that determines a user's motion based on at least one of a plurality of variables included in the motion specification information, The operation determination unit counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. It is characterized by:

[0007] Further, in the chewing frequency measuring device according to the above aspect, the operation determination unit is configured to count the number of times of chewing as one when a jaw opening operation based on chewing is identified once and then a jaw closing operation based on chewing is identified once as the operation of the user, The device is characterized by further comprising:

[0008] Further, in the chewing frequency measuring device according to the above aspect, the motion determination unit determines the motion of the user based on a time change in the value of the variable included in the motion specification information continuously acquired for a certain period by the specification information acquisition unit. It is characterized by:

[0009] Further, in the chewing frequency measuring device according to the above aspect, a front determining unit that determines whether the user is facing forward, the identification information acquisition unit acquires the motion identification information when the front determination unit determines that the user is facing forward. It is characterized by:

[0010] Further, in the chewing frequency measuring device according to the above aspect, the front-facing determination unit determines whether the user is facing forward by estimating a three-dimensional vector of the user's face direction from the user's face image and three-dimensional configuration information of a face mesh generated from the face image; It is characterized by:

[0011] Further, in the chewing frequency measuring device according to the above aspect, a front instruction unit that instructs the user to face forward when the front determination unit determines that the user is not facing forward, The device is characterized by further comprising:

[0012] Further, in the chewing frequency measuring device according to the above aspect, The operation determination unit a predation determination unit that determines that the user's action is an opening and closing action for predation; a swallowing determination unit that determines that the user's motion is a swallowing motion; an other determination unit that determines that the user's motion is a motion in the opening and closing direction of the jaw due to chewing, an opening and closing motion for preying, or another motion that does not correspond to a swallowing motion; The device further comprises:

[0013] Further, in the chewing frequency measuring device according to the above aspect, a bite interval specifying unit that specifies a period from when the predation determination unit determines that the user's action is the opening and closing action for predation to when the other determination unit determines that the user's action is the other action as a bite interval corresponding to a bite of the user's meal; The device is characterized by further comprising:

[0014] Further, in the chewing frequency measuring device according to the above aspect, The one-bite section identification unit As a first pattern, after the swallowing determination unit determines that the user's motion is the swallowing motion, the other determination unit determines that the user's motion is the other motion, and further the predation determination unit determines that the user's motion is the opening and closing motion for predation, As a second pattern which is a pattern following the first pattern, when the swallowing determination unit determines that the user's motion is the swallowing motion, the other determination unit determines that the user's motion is the other motion, and the predation determination unit further determines that the user's motion is the opening and closing motion for predation, The section from when the predation determination unit determines that the opening and closing action for predation is in the first pattern to when the predation determination unit determines that the other action is in the second pattern is identified as a one-bite section corresponding to one bite of the user's meal. It is characterized by:

[0015] Further, in the chewing frequency measuring device according to the above aspect, a chewing frequency calculation unit that calculates the number of chews of the user in the one-bite section identified by the one-bite section identification unit, The device is characterized by further comprising:

[0016] Further, in the chewing frequency measuring device according to the above aspect, a mastication speed calculation unit that calculates the mastication speed of the user by determining, by the mastication determination unit, the time from the timing at which it is determined that the jaw is moving in the opening direction based on mastication to the timing at which it is determined that the jaw is moving in the closing direction based on mastication as a time for one mastication; The device is characterized by further comprising:

[0017] Further, in the chewing frequency measuring device according to the above aspect, Estimating the user's exercise-induced calorie expenditure or diet-induced thermogenesis based on the time spent measuring the meal and the number of chews of the user. It is characterized by:

[0018] A method for measuring the number of chewing strokes according to a second aspect of the present invention comprises: A method for measuring the number of chewing times using a chewing frequency measuring device, a specific information acquisition step of measuring a face image of the user and acquiring motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information output based on the measured face image; a motion determination step of determining a user's motion based on at least one of a plurality of variables included in the motion specification information, In the operation determination step, counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. It is characterized by:

[0019] A program according to a third aspect of the present invention comprises: A chewing frequency measuring device was used. a specific information acquisition unit that measures a face image of the user and acquires motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information that is output based on the measured face image; a motion determination unit that determines a user's motion based on at least one of a plurality of variables included in the motion specification information; The operation determination unit counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. It is characterized by:

[0020] The program may be recorded on a non-transitory recording medium. The non-transitory recording medium can be distributed or sold independently of the computer. Here, a non-transitory recording medium refers to a tangible recording medium. Examples of non-transitory recording media include compact discs, flexible disks, hard disks, magneto-optical disks, digital video disks, magnetic tapes, and semiconductor memories. A transitory recording medium refers to the transmission medium (propagation signal) itself. Examples of transitory recording media include electrical signals, optical signals, and electromagnetic waves. A temporary storage area is an area for temporarily storing data and programs, such as volatile memory such as RAM (Random Access Memory). [Effects of the Invention]

[0021] According to the present invention, it is possible to provide a chewing frequency measuring device, a chewing frequency measuring method, and a program that accurately measure the number of chews during a meal. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram showing an example of a chewing frequency measuring device. [Figure 2] FIG. 10 is an explanatory diagram illustrating an example of a learning model. [Figure 3] FIG. 10 is an explanatory diagram illustrating an example of bite pattern information. [Figure 4] 10 is a flowchart showing an example of a process for measuring mastication and the like. [Figure 5] 10 is a flowchart illustrating an example of a front determination process. [Figure 6] 10 is a flowchart illustrating an example of a motion determination process. [Figure 7] 10 is a flowchart illustrating an example of a process for determining a bite section or the like. [Figure 8] FIG. 10 is an explanatory diagram showing a display example at the start of the chewing and other measurement process. [Figure 9]FIG. 10 is an explanatory diagram showing an example of display of measurement results of the chewing and other measurement process. DETAILED DESCRIPTION OF THE INVENTION

[0023] (Overall composition) A chewing frequency measurement device, a chewing frequency measurement method, and a program according to embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, identical or corresponding parts are designated by the same reference numerals. The chewing frequency measurement device 100 is an information terminal (so-called computer) used by a user, such as a smartphone, tablet, or PC (Personal Computer). The chewing frequency measurement device 100 is equipped with an application that measures the number of chews, etc., of the user, and measures the number of chews, etc., of the user based on the user's operation of the application. The chewing frequency measurement device 100 measures the number of chews, etc., of the user by propping up the chewing frequency measurement device 100 owned by the user and capturing an image of the user's face while eating with an internal camera (input / output unit 130) built into the chewing frequency measurement device 100.

[0024] As shown in FIG. 1, a chewing frequency measuring device 100 according to an embodiment of the present invention includes a memory unit 110, a control unit 120, an input / output unit 130, a communication unit 140, and a system bus (not shown) interconnecting these units.

[0025] The storage unit 110 includes a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The ROM stores a program 111 to be executed by the control unit 120, various data (not shown) required in advance for executing the program 111, frontal direction determination information 112, a learning model 113, and bite pattern information 114.

[0026] The program 111 is a program for executing a process for measuring mastication and the like, which will be described later, and is stored in the storage unit 110 in advance.

[0027] The front-facing determination information 112 is information used to determine whether or not the user is facing forward in the front-facing determination process described below. Specifically, the front-facing determination information 112 includes reference values ​​for determining whether the user is facing forward. The front-facing determination information 112 includes the ranges of Roll, Pitch, and Yaw, which are three-dimensional vectors of the user's facial orientation, as reference values ​​for determining whether the user is facing forward.

[0028] The learning model 113 is a learning model that has been machine-learned based on facial images of people eating in order to determine which of a plurality of actions the user's action while eating is.

[0029] Specifically, as shown in FIG. 2, learning model 113 is a learning model that learns, as training data, human motions corresponding to the amount of change over time of multiple parameters obtained from facial images of a person eating. In this embodiment, learning model 113 uses, for example, blendShapes (https: / / developer.apple.com / documentation / arkit / arfaceanchor / 2928251-blendshapes), which are coefficient values ​​that represent the movement of specific parts of facial expressions from multiple amounts of change of each feature point in a normal state, as specific information, associates combinations of the amount of change of each parameter in the specific information (blendShapes) with human motions, and trains this correspondence as training data. The content of the first line in the illustrated example shows an example of supervised learning in which the "amount of change over time of parameters X, Y, and Z" when the jaw is opened in a chewing motion is associated with a motion such as "jaw opening in a chewing motion." In the illustrated example, the displacement amounts of multiple parameters are associated with the content of human motion, and this association is learned as training data, but the number of associated parameters may be single. Also, more than the number of parameters illustrated may be associated. Furthermore, multiple learning models 113 may be prepared according to attributes such as age, for example, adult males, adult females, and children. The identification information corresponds to motion identification information.

[0030] Returning to FIG. 1, the bite pattern information 114 is list information indicating the patterns of the user's time-series actions in a bite section from the first time the user takes a bite to the time when the user swallows the food. Specifically, as shown in FIG. 3, the bite pattern 114 has a plurality of patterns, Pattern 1 to Pattern 4, preset for each pattern of the user's time-series actions. In this embodiment, if the time-series actions correspond to any of Patterns 1 to 4, the section is determined to be a bite section. More specifically, in this embodiment, if the time-series actions of Patterns 1 to 4 are present in the section from the first time-series action of "swallowing → no action → eating" is detected to the next time-series action of "swallowing → no action → eating," the time-series action indicated by that pattern is determined to be a bite. Note that, in the illustrated example, four patterns, Pattern 1 to Pattern 4, are set, but five or more patterns may be present. Furthermore, in this embodiment, an example is shown in which bite pattern information 114 is stored in advance in memory unit 110, but bite pattern information 114 may not be stored, and instead, for example, a bite may be defined as the period from "predation" to "no movement" within the interval from when the time series movement of "swallowing → no movement → predation" is first detected until the time series movement of "swallowing → no movement → predation" is next detected.

[0031] 1 is configured with a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), etc. The control unit 120 operates in accordance with a program 111 stored in the storage unit 110, and executes processing in accordance with the program 111. The control unit 120 includes, as main functional units provided by the program 111 stored in the storage unit 110, a front determination unit 121, a specific information acquisition unit 122, a motion determination unit 123, a bite interval etc. measurement unit 128, and a calorie consumption etc. estimation unit 129.

[0032] The front-facing determination unit 121 is a functional unit that determines whether the user is facing forward. Specifically, the front-facing determination unit 121 is a functional unit that estimates Roll, Pitch, and Yaw, which are three-dimensional vectors of the user's face orientation, from three-dimensional configuration information of a face mesh generated from a face image of the user who is the subject of chewing count measurement, and determines that the user is facing forward when the Roll, Pitch, and Yaw are within the ranges of Roll, Pitch, and Yaw indicated by the front-facing determination information 112. The angles of Roll, Pitch, and Yaw are indicated as angles relative to the in-camera (input / output unit 130). The front-facing determination unit 121 also has a function of changing the color of the frame shown in FIG. 8 to green when it determines that the user is facing forward, and a function of displaying a message such as "Please face forward" when it determines that the user is not facing forward.

[0033] The specific information acquisition unit 122 is a functional unit that acquires time-series data of each parameter of specific information based on the user's facial image. Specifically, the specific information acquisition unit 122 is a functional unit that acquires information on each parameter in blendShapes, which are coefficient values ​​that represent the movement of specific parts of facial expressions. The parameters include, for example, a parameter indicating the degree of opening of the lower jaw, a parameter indicating the degree of forward movement of the lower jaw, a parameter indicating the degree of left-right rotation of the lower jaw, a parameter indicating the degree of lip closure that is independent of the position of the lower jaw, a parameter indicating the degree of contraction when both lips are open, a parameter indicating the degree of contraction when both lips are closed, a parameter indicating the degree of up-down, left-right, or backward movement of the corners of the mouth, a parameter indicating the degree of inward or outward lip curl, and a parameter indicating the degree of lip flattening. In this embodiment, the specific information acquisition unit 122 acquires eight or more frames of data for each parameter in time-series data of each parameter included in the specific information derived from time-series images with a frame rate of 60 FPS (frames per second). For example, when acquiring eight frames of time-series data, the time-series data includes the current frame of data and the past seven frames of data. The identification information acquisition unit 122 also acquires the time at which the time-series data was acquired along with the time-series data. In other words, in this embodiment, data that continues for a certain period of time, such as eight or more frames of data for each parameter, is acquired as time-series data. The step of acquiring the time-series data of each parameter included in the identification information by the identification information acquisition unit 122 corresponds to the identification information acquisition step.

[0034] The movement determination unit 123 is a functional unit that determines the movement of the user, who is the subject of the chewing frequency measurement, based on the amount of time change of each parameter of the specific information acquired by the specific information acquisition unit 122. Specifically, the movement determination unit 123 is a functional unit that inputs the amount of time change of each parameter of the specific information acquired by the specific information acquisition unit 122 (i.e., time-series data of 8 frames or more) into the learning model 113 and determines the movement of the user using the learning model 113. The movement determination unit 123 includes a chewing determination unit 124, a eating determination unit 125, a swallowing determination unit 126, and an other determination unit 127 as functional units that determine various movements of the user. Note that the step of determining the movement of the user by the movement determination unit 123 corresponds to the movement determination step.

[0035] The mastication determination unit 124 is a part that, when the amount of change over time of each parameter acquired by the specific information acquisition unit 122 is input to the learning model 113, determines that the user's movement determined by the learning model 113 is a "jaw opening movement based on mastication" or a "jaw closing movement based on mastication." That is, in this embodiment, a functional unit that determines the user's movement as a "jaw opening movement based on mastication" or a "jaw closing movement based on mastication" by the learning model 113 is referred to as the mastication determination unit 124. Note that determining both the "jaw opening movement based on mastication" and the "jaw closing movement based on mastication" by the mastication determination unit 124 corresponds to specifying the jaw opening and closing movement based on mastication.

[0036] The predation determination unit 125 is a part that determines that the user's movement determined by the learning model 113 is "mouth opening and closing movement associated with predatory movement" when the time displacement amount of each parameter acquired by the specific information acquisition unit 122 is input to the learning model 113. That is, in this embodiment, the functional unit that determines the user's movement as "mouth opening and closing movement associated with predatory movement" (predatory movement) by the learning model 113 is called the predation determination unit 125.

[0037] The swallowing determination unit 126 is a part that determines that the user's behavior determined by the learning model 113 is a "swallowing behavior" when the amount of time change of each parameter acquired by the specific information acquisition unit 122 is input to the learning model 113. That is, in this embodiment, the functional unit that determines the user's behavior as a "swallowing behavior" by the learning model 113 is called the swallowing determination unit 126.

[0038] The other determination unit 127 is a part that determines that the user's behavior determined by the learning model 113 is "other behavior" when the time variation amount of each parameter acquired by the specific information acquisition unit 122 is input to the learning model 113. That is, in this embodiment, the functional unit that determines the user's behavior as "other behavior" by the learning model 113 is called the other determination unit 127. Note that in this embodiment, when the other determination unit 127 determines that the user's behavior is "other behavior," it determines that the user's meal is interrupted, such as "no behavior."

[0039] The bite interval measurement unit 128 is a functional unit that measures a bite interval from the first bite to the swallowing of a user who is a subject for measuring the number of chews. Specifically, the bite interval measurement unit 128 is a functional unit that, when any of the time-series actions of patterns 1 to 4 indicated by the bite pattern information 114 is present in the interval from the first detection of the time-series action of "swallowing → no action → eating" to the next detection of the time-series action of "swallowing → no action → eating" among the patterns of the user's time-series action determined by the action determination unit 123, determines the time-series action indicated by that pattern as a bite. Note that, when the bite pattern information 114 is not stored, the bite interval measurement unit 128 may, for example, determine the period from the first detection of the time-series action of "swallowing → no action → eating" (first pattern) to the next detection of the time-series action of "swallowing → no action → eating" (second pattern) to the period from the eating of the first pattern to the no action of the second pattern within the interval. Furthermore, bite interval etc. measuring unit 128 has a function of calculating the number of chews per mouthful, the time per mouthful, and the time for one chew. Note that the functional unit that specifies the bite interval in bite interval etc. measuring unit 128 corresponds to the bite interval specifying unit.

[0040] The calorie consumption etc. estimation unit 129 is a functional unit that estimates exercise-induced calorie consumption and diet-induced thermogenesis based on the meal eaten by the user, who is the subject of measuring the number of chews. Specifically, the calorie consumption etc. estimation unit 129 is a functional unit that estimates exercise-induced calorie consumption and diet-induced thermogenesis due to the meal by performing a predetermined calculation based on the time spent measuring the user's meal and the number of chews.

[0041] The input / output unit 130 is a device that is configured with a keyboard, a mouse, a camera, a microphone, a liquid crystal display, an organic EL (Electro-Luminescence) display, and the like, and is used to input and output various types of data.

[0042] The communication unit 140 is a device that enables the chewing frequency measurement device 100 to communicate with other information terminals via a computer communication network. The configuration of the chewing frequency measurement device 100 has been described above.

[0043] (operation) Next, the operation of the chewing frequency measuring device 100 will be described. Fig. 4 is a flowchart showing an example of the chewing, etc. measuring process in the chewing frequency measuring device 100. The chewing, etc. measuring process in this embodiment starts to be executed when a user operates an application that has been started. Specifically, the chewing, etc. measuring process starts to be executed when a user presses a "measurement start button" shown in Fig. 8. Note that the chewing, etc. measuring process may start even if the user does not press the "measurement start button" shown in Fig. 8, but simply by starting an application, for example.

[0044] When the chewing etc. measurement process shown in Fig. 4 is started, the control unit 120 executes a front-facing determination process using the function of the front-facing determination unit 121 to determine whether or not the user is facing forward (step S11). Fig. 5 is a flowchart showing an example of the front-facing determination process executed in step S11 of Fig. 4. In the front-facing determination process shown in Fig. 5, the control unit 120 acquires angle information using the function of the front-facing determination unit 121 (step S21). Specifically, in the process of step S21, the front-facing determination unit 121 acquires estimated values ​​of Roll, Pitch, and Yaw, which are three-dimensional vectors of the user's facial orientation, as angle information, based on three-dimensional configuration information of a face mesh generated from a facial image of the user who is the subject of mastication count measurement.

[0045] After executing the process of step S21, the control unit 120 uses the function of the front-facing determination unit 121 to determine whether or not the user is facing forward based on the front-facing determination information 112 (step S22). Specifically, in the process of step S22, the front-facing determination unit 121 determines that the user is facing forward if the angle information acquired in the process of step S21 is within the ranges of Roll, Pitch, and Yaw indicated by the front-facing determination information 112, and determines that the user is not facing forward if any one or more of the angle information is outside the ranges.

[0046] If it is determined in the process of step S22 that the user is not facing forward (step S22; No), the control unit 120 uses the function of the front-facing determination unit 121 to output an adjustment instruction to the input / output unit 130, which is an instruction to adjust the angle of the face so that the user faces forward (step S23), and returns to the process of step S21. Specifically, in the process of step S23, the front-facing determination unit 121 displays a message such as "Face forward and adjust the angle of the face so that the frame color becomes green" on a display screen included in the input / output unit 130, as shown in FIG. 8. Note that, although the illustrated example shows an example in which the message is displayed on the display screen, the message may also be a sound, a vibration, a video, or a combination of these. Note that the front-facing determination unit 121 that executes the process of step S23 corresponds to a front-facing instruction unit.

[0047] On the other hand, if it is determined in the process of step S22 that the user is facing forward (step S22; Yes), the control unit 120 uses the function of the front-facing determination unit 121 to perform normal output, which is a display indicating that the user is facing forward (step S24), and ends the front-facing determination process. Specifically, in the process of step S24, the front-facing determination unit 121 performs a display indicating that the user is facing forward, i.e., a normal output, by changing the color of the frame shown in Fig. 8 to green.

[0048] Returning to Fig. 4, after executing the front determination process in step S11, the control unit 120 executes a movement determination process using the functions of the specification information acquisition unit 122 and the movement determination unit 123. Fig. 6 is a flowchart showing an example of the movement determination process executed in step S12 of Fig. 4. In the movement determination process shown in Fig. 6, the control unit 120 first acquires specification information using the function of the specification information acquisition unit 122 (step S31). Specifically, in the process of step S31, the specification information acquisition unit 122 acquires, as time-series data, data of 8 frames or more for each of the time-series data of each parameter included in the specification information derived from time-series images at a frame rate of 60 FPS.

[0049] After executing the process of step S31, the control unit 120 determines whether or not the jaw opening movement is based on mastication using the function of the movement determination unit 123 (step S32). Specifically, in the process of step S32, when eight or more frames of time-series data for each parameter acquired in the process of step S31 are input to the learning model 113, the movement determination unit 123 determines whether or not the mastication determination unit 124 has determined that the user's movement determined by the learning model 113 is "a jaw opening movement based on mastication."

[0050] If the mastication determination unit 124 determines in the processing of step S32 that the user's movement is a "jaw opening direction movement based on mastication" (step S32; Yes), the control unit 120 turns on an opening direction flag that is provided in advance in the storage unit 110 (step S33), and ends the movement determination processing. Note that the opening direction flag is set to off as an initial state. On the other hand, if the mastication determination unit 124 does not determine in the processing of step S32 that the user's movement is a "jaw opening direction movement based on mastication" (step S32; No), the control unit 120 determines whether or not the movement is a jaw closing direction movement based on mastication, using the function of the movement determination unit 123 (step S34). Specifically, in the processing of step S34, when eight or more frames of time series data for each parameter acquired in the processing of step S31 are input to the learning model 113, the movement determination unit 123 determines whether or not the user movement determined by the learning model 113 has been determined by the chewing determination unit 124 to be a "jaw closing movement based on chewing."

[0051] If the chewing determination unit 124 determines in the processing of step S34 that the user's movement is a "jaw closing movement based on chewing" (step S34; Yes), the control unit 120 determines whether the opening direction flag is on or not using the function of the movement determination unit 123 (step S35).

[0052] If the opening direction flag is on (step S35; Yes), the control unit 120 increments the value of a chewing counter that is provided in advance in the storage unit 110 by +1 (step S36). The chewing counter is initially set to "0", and is cleared to the initial value after the user performs an operation to end measurement and the measurement result is output. The chewing counter is a counter that counts the number of chews the user makes while eating. On the other hand, if the opening direction flag is not on (step S35; No), the control unit 120 ends the operation determination process.

[0053] After executing the process of step S36, the control unit 120 sets the opening direction flag to OFF, which is the initial state (step S37). After executing the process of step S37, the control unit 120 uses the function of the movement determination unit 123 to identify the user's movement as a chewing movement (step S38). That is, the movement determination unit 123 determines that a chewing movement has been performed once when it is determined by the mastication determination unit 124 that a "jaw opening direction movement based on mastication" has been performed and then a "jaw closing direction movement based on mastication" has been performed. That is, it determines that one chewing has been performed when a "jaw opening direction movement based on mastication" has been identified once and then a "jaw closing direction movement based on mastication" has been identified once.

[0054] On the other hand, if the chewing determination unit 124 does not determine in the process of step S34 that the user's movement is a "jaw closing movement due to chewing" (step S34; No), the control unit 120, using the function of the movement determination unit 123, determines that the movement identified by the learning model 113 based on the identification information acquired in the process of step S31 is the user's movement (step S39). Specifically, in the process of step S39, the movement determination unit 123 determines that the determination results of the respective functional units of the predation determination unit 125, the swallowing determination unit 126, and the other determination unit 127 are the user's movement. For example, when eight or more frames of time-series data for each parameter acquired in the process of step S31 are input to the learning model 113, if the predation determination unit 125 determines that the user's movement determined by the learning model 113 is a "mouth opening and closing movement associated with a predatory movement," in the process of step S39, the movement determination unit 123 determines that the user's movement is a "predatory movement." Furthermore, when eight or more frames of time-series data for each parameter acquired in the process of step S31 are input to the learning model 113, if the swallowing determination unit 126 determines that the user's movement determined by the learning model 113 is a "swallowing movement," the movement determination unit 123 identifies the user's movement as a "swallowing movement" in the process of step S39. Furthermore, when eight or more frames of time-series data for each parameter acquired in the process of step S31 are input to the learning model 113, if the predation determination unit 125 determines that the user's movement determined by the learning model 113 is an "other movement," the movement determination unit 123 identifies the user's movement as "no movement" in the process of step S39. After executing the process of step S39, the control unit 120 sets the opening direction flag to its initial state, OFF (step S39A). Note that the process of step S39A need only be performed when the opening direction flag is ON.

[0055] After executing the process of step S38 or step S39, the control unit 120 records the user's actions identified in the process of step S38 or step S39 in the storage unit 110 as time-series actions (step S40), and ends the action determination process. In this embodiment, the action determination process is repeatedly executed until the user performs an end operation. Therefore, by repeatedly executing the process of step S40, time-series data such as "swallowing → no action → eating → chewing → ... → chewing → swallowing → no action → eating" is recorded in the storage unit 110.

[0056] 4, after executing the process of step S12, the control unit 120 determines whether or not the user has performed an end operation to end the chewing, etc. measurement process in the application (step S13). Specifically, the process of step S13 determines whether or not the user has performed an end operation in the input / output unit 130. If the end operation has not been performed (step S13; No), the control unit 120 returns to the process of step S11 and repeats the processes of step S11 and step S12 until the end operation is performed.

[0057] On the other hand, if an end operation has been performed (step S13; Yes), control unit 120 executes a bite interval etc. identification process using the function of bite interval etc. measurement unit 128 (step S14). FIG. 7 is a flowchart showing an example of the bite interval etc. identification process executed in step S14 of FIG. 4. In the bite interval etc. identification process shown in FIG. 7, control unit 120 divides a time-series action into bite intervals using the function of bite interval etc. measurement unit 128 (step S51). Specifically, in the process of step S51, bite interval etc. measurement unit 128 divides, as bite intervals, intervals that fit into any of the time-series actions of pattern 1 to pattern 4 shown in bite pattern information 114 for the action patterns included in the time-series action recorded in storage unit 110 in the process of step S40 of FIG. 6.

[0058] After executing the process of step S51, the control unit 120 calculates the number of chews per mouthful using the function of the bite interval etc. measuring unit 128 (step S52). Specifically, in the process of step S52, the bite interval etc. measuring unit 128 calculates the number of chews per mouthful by dividing the value of the chewing counter by the number divided in the process of step S51. Alternatively, the number of chews in each bite interval may be calculated by counting the number of "chewing actions" included in the time-series actions of the bite interval divided in the process of step S51. The function of the bite interval etc. measuring unit 128 that calculates the number of chews in the process of step S52 corresponds to the number of chews calculation unit.

[0059] After executing the process of step S52, the control unit 120 calculates the time per mouthful using the function of the mouthful interval etc. measuring unit 128 (step S53). Specifically, in the process of step S53, the mouthful interval etc. measuring unit 128 calculates the time per mouthful by dividing the value of the chewing counter by the number divided in the process of step S51, that is, by dividing the time from the start of the chewing etc. measurement process to the operation to end the measurement by the number divided in the process of step S51. Alternatively, the time per mouthful may be calculated from the time when the time-series data marking the start of the mouthful interval was acquired to the time-series data marking the end of the mouthful interval, that is, the start and end times in patterns 1 to 4.

[0060] After executing the process of step S53, the control unit 120 calculates the time and the like of one chewing session using the function of the bite interval measurement unit 128 (step S54), and ends the bite interval determination process. Specifically, in the process of step S54, the bite interval measurement unit 128 calculates the time and the speed of one chewing session. The time of one chewing session may be calculated from the time (timing) at which the user's movement is determined to be a "jaw opening movement based on chewing" in the process of step S32 of FIG. 6 and the time (timing) at which the chewing determination unit 124 determines that the user's movement is a "jaw closing movement based on chewing" in the process of step S34. The chewing speed may be calculated based on the time of one chewing session. For example, if the time of one chewing session is 0.5 seconds, the chewing speed is 2 times / second. The bite interval etc. measuring unit 128 that executes the process of step S54 corresponds to the chewing speed calculation unit.

[0061] 4, after executing the process of step S14, the control unit 120 estimates the exercise-induced calorie expenditure and diet-induced thermogenesis based on the meal the user has eaten, using the function of the calorie expenditure etc. estimation unit 129 (step S15). Specifically, in the process of step S15, the calorie expenditure etc. estimation unit 129 estimates the exercise-induced calorie expenditure and diet-induced thermogenesis due to the meal by performing a predetermined calculation based on the time spent measuring the user's meal and the number of chews. In this embodiment, a calculation formula is set in advance to calculate an average energy expenditure of 1.375 kcal / min when chewing at a rate of 80 times / min (https: / / www.jstage.jst.go.jp / article / soshaku1991 / 2 / 1 / 2_1_55 / _pdf / -char / en), and the calorie expenditure estimation unit 129 calculates the user's exercise-induced calorie expenditure based on the value of the chewing counter counted by the process of step S36 in Fig. 6, i.e., the total number of chews during a meal. Note that, because the average energy expenditure at rest is 1.174 kcal / min, the user's exercise-induced calorie expenditure, excluding the energy expenditure at rest, may be calculated as (1.375 kcal / min - 1.174 kcal / min) / 80 times / min = 0.0025125 kcal / time ≒ 0.0025 kcal / time. In addition, diet-induced thermogenesis (DIT) can be calculated based on the experimental report (https: / / www.nature.com / articles / s41598-021-03109-x) and the Japanese commentary by the author (https: / / www.waseda.jp / top / news / 77113). Since DIT due to pure meal time can be considered the DIT when no chewing occurs at all, the average DIT per experimental period of 30 seconds x 10 times (= 5 minutes) is set to 3.4 kcal, giving 3.4 [kcal] / 5 [min] = 0.68 [kcal / min].DIT due to the number of chews is 7.4 kcal per 300 chews (30 chews x 10 times), and since the DIT not including chewing is 3.4 kcal, we use (7.4 [kcal] - 3.4 [kcal]) / 300 [times] = 0.01333 [kcal / time], which gives 0.68 [kcal / min] and 0.01333 [kcal / time].

[0062] After executing the process of step S15, the control unit 120 outputs the measurement result to the input / output unit 130 (step S16), and ends the mastication and other measurement process. Specifically, in the process of step S16, the control unit 120 outputs the measurement result as shown in Fig. 9 to the input / output unit 130. After the measurement result is output in the process of step S16, the value of the mastication counter may be cleared to "0".

[0063] The above is the operation of the chewing frequency measuring device 100. As described above, according to the chewing frequency measuring device 100 of this embodiment, the user's behavior is identified by the learning model 113 based on the time-series data of a plurality of parameters included in the identification information, and the chewing situation is measured. Therefore, the number of chews during a meal can be measured with high accuracy.

[0064] (Variation) It should be noted that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible. For example, the chewing frequency measuring device 100 according to the above-described embodiment does not need to have all of the technical features described above, and may have some of the configurations described in the above-described embodiment so as to solve at least one problem in the prior art. Furthermore, at least a portion of each of the following modifications may be combined.

[0065] In the above embodiment, when it is determined that the user is not facing forward in the frontal determination process shown in Fig. 5, in the process of step S23, a message such as "Face forward and adjust the angle of your face so that the frame color turns green" is displayed on the input / output unit 130 as shown in Fig. 8, but this is just one example. In addition, for example, which of the roll, pitch, and yaw is not within the range indicated by the frontal determination information 112 may be displayed so that the user can visually recognize it. For example, the angles of roll, pitch, and yaw may be displayed, and those outside the range indicated by the frontal determination information 112 may be displayed in green.

[0066] Furthermore, in the above embodiment, an example has been shown in which the user's attributes are not taken into consideration for ease of understanding. However, for example, the chewing and other measurement process may be performed according to the attributes. In this case, the user may select the attribute by himself / herself before starting the chewing and other measurement process. For example, an attribute according to age, such as adult male, adult female, or child, may be selected. Then, the user's actions according to the attribute may be determined by the action determination process shown in FIG. 6 using a learning model 113 according to the attribute. This can further improve the accuracy of measuring the number of chews during a meal.

[0067] It should be noted that the chewing frequency measuring device 100 according to the above embodiment can be realized using a normal computer, not a dedicated device. For example, the chewing frequency measuring device 100 that executes the above-described processes may be configured by installing a program for executing any of the above-described processes into a computer from a recording medium storing the program. Furthermore, one chewing frequency measuring device 100 may be configured by multiple computers operating in cooperation with each other.

[0068] Furthermore, when the above-mentioned functions are realized by sharing the functions between an OS (Operating System) and an application, or by cooperation between the OS and the application, only the parts other than the OS may be stored on the medium.

[0069] It is also possible to superimpose the program on a carrier wave and distribute it via a communication network. For example, the program may be posted on a bulletin board system (BBS) on the communication network and distributed via the network. These programs may then be started and run under the control of an operating system in the same way as other application programs, thereby enabling the above-mentioned processing to be performed.

[0070] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and are not intended to limit the scope of the present invention. [Industrial Applicability]

[0071] According to the present invention, it is possible to provide a chewing frequency measuring device, a chewing frequency measuring method, and a program that can accurately measure the number of chews during a meal. [Explanation of symbols]

[0072] 100 Chewing frequency measuring device 110 Storage section 111 Program 112 Information for frontal judgment 113 Learning Model 114 Bite Pattern Information 120 control section 121 Front judgment section 122 Specific information acquisition department 123 Operation judgment section 124 Chewing judgment section 125 Predation Judgment Department 126 Swallowing determination unit 127 Other Judgment Section 128 One-way section measurement section 129 Calorie Consumption Estimation Section 130 Input / output section 140 Communications Department

Claims

1. a specific information acquisition unit that measures a face image of a user and acquires motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information that is output based on the measured face image; a motion determination unit that determines a user's motion based on at least one of a plurality of variables included in the motion specification information, The operation determination unit counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. A chewing frequency measuring device characterized by the above.

2. a mastication determination unit that counts mastication as one time when a jaw opening movement based on mastication is identified once and then a jaw closing movement based on mastication is identified once as the user's movement; The chewing frequency measuring device according to claim 1, further comprising:

3. the motion determination unit determines the motion of the user based on a time change in the value of the variable included in the motion specification information continuously acquired for a certain period by the specification information acquisition unit.

3. The chewing frequency measuring device according to claim 1 or 2.

4. a front determining unit that determines whether the user is facing forward, the identification information acquisition unit acquires the motion identification information when the front determination unit determines that the user is facing forward.

3. The chewing frequency measuring device according to claim 1 or 2.

5. the front-facing determination unit determines whether the user is facing forward by estimating a three-dimensional vector of the user's face direction from the user's face image and three-dimensional configuration information of a face mesh generated from the face image; The chewing frequency measuring device according to claim 4 .

6. a front instruction unit that instructs the user to face forward when the front determination unit determines that the user is not facing forward, The chewing frequency measuring device according to claim 5, further comprising:

7. The operation determination unit a predation determination unit that determines that the user's action is an opening and closing action for predation; a swallowing determination unit that determines that the user's motion is a swallowing motion; an other determination unit that determines that the user's motion is a motion in the opening and closing direction of the jaw due to chewing, an opening and closing motion for preying, or another motion that does not correspond to a swallowing motion; The chewing frequency measuring device according to claim 2, further comprising:

8. a bite interval specifying unit that specifies a period from when the predation determination unit determines that the user's action is the opening and closing action for predation to when the other determination unit determines that the user's action is the other action as a bite interval corresponding to a bite of the user's meal; The chewing frequency measuring device according to claim 7, further comprising:

9. The one-bite section identification unit As a first pattern, after the swallowing determination unit determines that the user's motion is the swallowing motion, the other determination unit determines that the user's motion is the other motion, and further the predation determination unit determines that the user's motion is the opening and closing motion for predation, As a second pattern which is a pattern following the first pattern, when the swallowing determination unit determines that the user's motion is the swallowing motion, the other determination unit determines that the user's motion is the other motion, and the predation determination unit further determines that the user's motion is the opening and closing motion for predation, The section from when the predation determination unit determines that the opening and closing action for predation is in the first pattern to when the predation determination unit determines that the other action is in the second pattern is identified as a one-bite section corresponding to one bite of the user's meal. The chewing frequency measuring device according to claim 8 .

10. a chewing frequency calculation unit that calculates the number of chews of the user in the one-bite section identified by the one-bite section identification unit, The chewing frequency measuring device according to claim 8 or 9, further comprising:

11. a mastication speed calculation unit that calculates a mastication speed of the user by determining, by the mastication determination unit, a time period from a time when it is determined that the jaw is moving in an opening direction based on mastication to a time when it is determined that the jaw is moving in a closing direction based on mastication as a time period for one mastication; The chewing frequency measuring device according to claim 2, further comprising:

12. Estimating the user's exercise-induced calorie expenditure or diet-induced thermogenesis based on the time spent measuring the meal and the number of chews of the user. The chewing frequency measuring device according to claim 11 .

13. A method for measuring the number of chewing times using a chewing frequency measuring device, a specific information acquisition step of measuring a face image of the user and acquiring motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information output based on the measured face image; a motion determination step of determining a user's motion based on at least one of a plurality of variables included in the motion specification information, In the operation determination step, counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. A method for measuring the number of chews.

14. A chewing frequency measuring device was used. a specific information acquisition unit that measures a face image of the user and acquires motion specific information that identifies a motion of a specific part included in the face, the motion specific information being information that is output based on the measured face image; a motion determination unit that determines a user's motion based on at least one of a plurality of variables included in the motion specification information; The operation determination unit counting the number of chews based on the identified jaw opening and closing movement as the user's movement; The determination of the user's motion is performed using a learning model that has been trained using data that associates the time change of the value of the variable included in the motion identification information with the user's motion as training data. A program characterized by:

Images