Talking determination device and program

Abstract

To provide a talking determination device capable of determining suitable conversation timing for a person who is taking a meal.SOLUTION: A talking determination device executes: first processing (talking influence degree evaluation part 22) to evaluate the degree of influence when talking to a person who is taking a meal about conversation details (conversation detail management part 21) to be talked about at the present time; second processing (meal-taking person situation evaluation part 23) to evaluate the situation at the current time of the meal-taking person as a meal-taking-person situation; and third processing (talking determination part 25) to determine whether the current time is proper timing of talking to the meal-taking person about the conversation details based upon the degree of influence and the meal-taking-person situation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus and program for determining whether or not a person is being spoken to while eating a meal. [Background technology]

[0002] Regarding conversations with diners using a dialogue agent or the like, technologies related to or potentially related to conversations at appropriate times are given in, for example, Patent Documents 1 and 2.

[0003] Patent Document 1 discloses an automatic dialogue technology with customers who enter a restaurant, in which the content of the dialogue matches the situation, such as before ordering, after ordering and before food and drink are served, during the meal, etc. Patent Document 2 discloses a technology that performs image analysis of the progress of eating and drinking at each table in a restaurant and creates a customer service schedule. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-163504 A [Patent Document 2] Patent No. 7155295 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, it was not possible to determine the appropriate timing for talking to a target person who is eating. For example, if the timing of talking is not appropriately controlled, problems such as asking a person to answer while they have food in their mouth, and forcing the person to answer may cause the person to choke, or interrupting a person watching other media such as television or a smartphone by talking to them may interrupt the person, but determining the appropriate timing for talking to deal with such problems was not taken into consideration in the conventional technology.

[0006] In Patent Document 1, a dialogue scenario is used according to the scenario of a customer's stay at a restaurant (before ordering, after ordering but before food and drink are served, during the meal, etc.), but the purpose is to have an appropriate dialogue according to the distinction of the customer's stay situation and to grasp the stay situation, and no consideration was given to the specific timing during the meal when to talk to the customer. Patent Document 2 also uses the image to determine the progress of the meal in a customer service schedule, but does not consider determining the appropriate timing to talk to the customer.

[0007] In view of the above-mentioned problems with the conventional technology, an object of the present invention is to provide a conversation determination device and program capable of determining an appropriate conversation timing for a diner. [Means for solving the problem]

[0008] In order to achieve the above object, the present invention is a speech judgment device that executes a first process of evaluating the impact of speaking to a diner about the content of a conversation at the current time, a second process of evaluating the diner's current situation as a diner situation, and a third process of judging whether the current time is appropriate for speaking to the diner about the content of the conversation based on the impact and the diner situation.The present invention is also characterized by a program that causes a computer to function as the speech judgment device. Effect of the Invention

[0009] According to the present invention, it is possible to determine whether the current time is an appropriate timing to speak to a diner about the content of a conversation, based on the degree of influence and the diner's situation. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional block diagram of a conversation determination system according to an embodiment; [Diagram 2] FIG. 2 is a functional block diagram of a diner status evaluation unit according to one embodiment. [Diagram 3] 1 is a flowchart illustrating an operation of the conversation determination system according to an embodiment. [Figure 4] 13 is a flowchart showing the process contents of a conversation content management unit and a speaking influence evaluation unit as details of step S22 according to an embodiment. [Diagram 5] 13 is a flowchart showing the process content of a visual occupancy state estimation unit as details of step S231 according to an embodiment. [Figure 6] 13 is a flowchart showing the process of a hearing occupancy state estimation unit as details of step S232 according to an embodiment. [Figure 7] 13 is a flowchart showing the processing contents of a mouth occupancy state estimation unit as details of step S233 according to an embodiment. [Figure 8] 11 is a diagram showing a method of determination by an address determination unit according to an embodiment; FIG. [Figure 9] FIG. 11 is a functional block diagram of a conversation determination system according to a modified example. [Figure 10] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] 1 is a functional block diagram of an interaction determination system 100 according to an embodiment. The interaction determination system 100 includes a monitoring device 10 and an interaction determination device 20.

[0012] The monitoring device 10 is configured as a device to be placed near the diner or as a device worn by the diner, and comprises, as hardware, an imaging unit 11 configured as a camera to obtain video footage of the diner, a recording unit 12 configured as hardware by a microphone to record sounds around the diner (the diner's own voice and, if there is sound from a television or the like nearby, that sound), and a meal state monitoring unit 13 configured as hardware including a biosensor (see non-patent document 3 described below) worn by the diner around the ear to determine chewing from movement behind the ear, a biosensor worn around the throat (for example, a throat microphone capable of detecting chewing and swallowing sounds in addition to speaking) and a biosensor worn around the neck (capable of detecting swallowing as seen in non-patent document 4 described below) to monitor the diner's swallowing state, etc.

[0013] In FIG. 1, the monitoring device 10 and the speech determination device 20 are depicted separately, but all or a part of the components of the monitoring device 10 may be included in the speech determination device 20.

[0014] The speech determination device 20 includes a conversation content management unit 21 , a speech influence degree evaluation unit 22 , a diner state evaluation unit 23 , a speech determination unit 24 , and a speech processing unit 25 .

[0015] The conversation content management unit 21 manages the conversation content to be spoken to the diners as a predetermined scenario prepared in accordance with the progress of the meal, or as an automatic conversation by an arbitrary dialogue agent, and outputs the conversation content to be spoken to at a given point during the meal to the conversation influence evaluation unit 22 and the conversation processing unit 25.

[0016] The conversation influence evaluation unit 22 evaluates the influence of the conversation content obtained from the conversation content management unit 21 and outputs the evaluation to the conversation determination unit 24. The diner status evaluation unit 23 evaluates the diner status in terms of vision, hearing, and mouth based on the output from the monitoring device 10, and outputs the evaluated diner status to the conversation determination unit 24.

[0017] The significance of the speech influence evaluation unit 22 is as follows: The speech influence evaluation unit 22 judges how much of a burden the contents of a speech from an agent will place on the diner, for example, whether it is of a level that will not cause a problem if the diner can simply hear it, or whether it is of a level that is inappropriate to simply ignore and requires the diner to give a clear reply.

[0018] The speaking judgment unit 24 judges whether the current time is appropriate as the timing to actually speak to the diners about the conversation content managed by the conversation content management unit 21, based on the influence degree obtained from the speaking influence degree evaluation unit 22 and the diners' status obtained from the diners' status evaluation unit 23, and outputs the judgment result to the speaking processing unit 25. When the speaking processing unit 25 judges that the timing is appropriate at the current time, the speaking processing unit 25 performs processing to actually speak to the diners about the conversation content managed by the conversation content management unit 21 at the current time.

[0019] The hardware of the conversation processing unit 25 is composed of a speaker placed near the diner, and outputs the conversation content to the diner as voice (in the case of a dialogue agent, mechanical voice (synthetic voice) output), thereby realizing conversation with the diner at the appropriate time.

[0020] 1, the conversation determination system 100 (excluding the meal state monitoring unit 13 when configured as a dedicated biosensor device worn by the diner for swallowing detection) can be realized in the form of a mobile terminal such as a tablet terminal or a smartphone terminal, which is a general-purpose computer device equipped with a camera, a microphone, a speaker, etc. The conversation determination system 100 configured as a mobile terminal can be placed near the diner, and the conversation content can be output to the diner from the speaker at appropriate timing while monitoring the diner's condition with a camera, etc.

[0021] 2 is a functional block diagram of a diner state assessment unit 23 according to an embodiment. The diner state assessment unit 23 includes a visual occupancy state estimation unit 231, an auditory occupancy state estimation unit 232, and a mouth occupancy state estimation unit 233.

[0022] These units 231, 232, and 233 are used to grasp the visual, auditory, and oral conditions of the diner. That is, (1) it is determined whether the diner's visual sense is already occupied, for example, whether the diner is staring at something such as a television or smartphone. Also, (2) it is determined whether the diner's auditory sense is already occupied, for example, whether the diner is listening to the audio of a television or a smartphone. Furthermore, (3) it is determined whether there is food remaining in the diner's mouth, for example, whether the diner has food in his or her mouth and therefore has difficulty in responding verbally.

[0023] 3 is a flowchart of the operation of the speech determination system according to one embodiment. In step S1, speech determination system 100 is installed near a diner, monitoring of the diner is started, the diner starts eating, and the process proceeds to step S22. Note that only the monitoring device 10 and speech processing unit 25 as a speaker need to be placed near the diner as hardware components constituting speech determination system 100, and all or part of the other components may be realized by a program executed on a separate server device accessible via a network.

[0024] In step S22, the conversation content at the current time t (t=1, 2, 3, . . . ) is acquired in the conversation content management section 21, and the influence of this conversation content is evaluated in the speaking influence evaluation section 22, and then the process proceeds to step S23.

[0025] In step S23, the diner status evaluation unit 23 evaluates the diner status at the current time t, and then the process proceeds to step S24. Here, step S23 includes steps S231, S232, and S233, and each unit 231, 232, and 233 performs the following processes as shown in FIG.

[0026] In step S231, the visual occupancy estimation unit 231 estimates the visual occupancy status of the diner at the current time t based on the video obtained from the imaging unit 11 at the current time t. In step S232, the auditory occupancy estimation unit 232 estimates the auditory occupancy status of the diner at the current time t based on the sound obtained from the recording unit 12 at the current time t. In step S233, the mouth occupancy estimation unit 233 estimates the mouth occupancy status of the diner at the current time t based on the video obtained from the imaging unit 11 at the current time t and the eating state related to swallowing obtained from the eating state monitoring unit 13. The mouth occupancy status is defined as a situation in which the mouth is occupied by drink and / or food (food and drink), and specifically, the status can be estimated by a method described later with reference to FIG. 7.

[0027] In step S24, based on the degree of influence evaluated for the current time t in step S22 and the diner status estimated for the current time t in step S23, the conversation judgment unit 24 judges whether the current time t is an appropriate timing to speak to the conversation content obtained by the conversation content management unit 21, and proceeds to step S241.

[0028] In step S241, it is determined whether the result of the determination in step S24 is positive (the current time t is an appropriate timing for speaking) or negative (the current time t is not an appropriate timing for speaking). If the result of the determination in step S24 is positive, the process proceeds from step S241 to step S25, whereby the conversation content obtained from the conversation content management unit 21 is output as voice from the conversation processing unit 25 to the diner, thereby realizing a conversation at the current time t, and then proceeds to step S26. If the determination in step S24 is negative, step S25 is skipped and the process proceeds from step S241 to step S26, whereby no conversation processing is performed to the diner at the current time t.

[0029] In step S26, the current time t is updated to the next time t+1, and the process returns to step S22, where the process of talking to the diner is continued after determining the timing. Step S26 determines the processing rate of the talking determination system 100 that performs processing in real time, and the time t=1, 2, 3, ... may be, for example, every 0.1 seconds or every second in real time. In addition, until the talking process performed in step S25 is completed, the process may remain in step S25 during the passage of real time. For example, when the time t=1, 2, 3, ... updated in step S26 is every second in real time, if it takes 5 seconds to complete the talking in the conversation content in step S25, the process may remain in step S25 for the 5 seconds and proceed to step S26 5 seconds after the talking is completed.

[0030] The general operation of the conversation determination system 100 has been described above with reference to Figures 1 to 3. The details will be described below.

[0031] FIG. 4 is a flowchart showing the process contents of the conversation content management section 21 and the speaking influence evaluation section 22 as details of step S22 according to one embodiment.

[0032] In step S2201, the conversation content management unit 21 acquires the conversation content at the current time t, and then the process proceeds to step S2202. The conversation content may be prepared in advance as text information of one or more sentences according to the purpose of speaking to the diner. For example, text prepared in advance on a rule basis may be used as the conversation content in response to all or part of (a) the diner's progress in eating, (b) the diner's condition, and (c) the type of food the diner is eating at the current time t.

[0033] Here, item (a) the degree of progress of the diner's meal can be estimated by photographing the diner's food with the photographing unit 11 and applying object recognition using deep learning or the like to the images at each time to estimate the volume of the food currently remaining and the change in area on the image over time (for example, it can be estimated by assuming that at the current time, half of the entire meal has been eaten), or it can be determined based on the percentage of the time elapsed since the start of the meal relative to the meal time by inputting in advance the standard meal time expected for the target diner as advance information (for example, it can be determined that at the current time, half of the entire meal time has passed).

[0034] In this way, for example, predetermined text corresponding to each of 11 stages of meal progress from 0%, 10%, to 100% may be preset as conversation content.

[0035] Item (b) The state of the diner may be estimated as the current state of the diner, which is acquired by photographing the diner himself with the photographing unit 11 and applying behavioral recognition by deep learning or the like to the obtained image. As a result, for example, if the diner is estimated to be choking as a result of behavioral recognition, a text to call attention to the diner may be set in advance as the conversation content. In order to estimate the diner's state and have the conversation content management unit 21 set the conversation content at the current time t based on the estimation result, all or part of the information evaluated by the diner evaluation unit 23 described later may be used. (In this case, since the conversation content management unit 21 sets the conversation content based on the evaluation result of the eating situation evaluation unit 23, there is a flow of information output from the diner situation evaluation unit 23 and input to the conversation content management unit 21, but the arrows representing this flow are omitted from FIG. 1 and the like as they are optional embodiments.)

[0036] Item (c) the type of food the diner is eating can be estimated by taking a picture of the diner and the food with the imaging unit 11, applying object recognition by deep learning or the like to the obtained image, and estimating the type of food at the time the food is brought to the diner's mouth. This allows, for example, predetermined text according to the type of food the diner is eating (for example, information about the nutritional content of the food or information giving advice on how to eat the food) to be set in advance as the content of the conversation.

[0037] The conversation content prepared by the conversation content management unit 21 is not limited to the above-mentioned method, but may be prepared by using as input all or part of the information obtained by analyzing information obtained from the monitoring device 10 including the imaging unit 11, or by using any existing method of conversation agent (a method for automatically generating conversation content text using a program) without using any of the information as input.

[0038] The steps S2202 to S2210 after the step S2201 are steps in which the conversation content obtained in step S2201 is automatically classified by the speech influence evaluation unit 22 using text analysis based on a rule base and / or any existing method, and the like, thereby evaluating the influence of the conversation content. (The subject of the operations in the steps S2202 to S2210 is the speech influence evaluation unit 22, so a description of the subject of the operations will be omitted.)

[0039] In step S2202, it is determined whether the conversation requires a response from the diner, and if the answer is yes, proceed to step S2203, and if the answer is no, proceed to step S2204. In step S2203, it is determined whether a response to the conversation would be impossible if there was food in the mouth, and if the answer is yes, proceed to step S2205, and if the answer is no, proceed to step S2208. In step S2204, it is determined whether the conversation requires an immediate response regardless of the diner's situation, and if the answer is yes, proceed to step S2206, and if the answer is no, proceed to step S2207. In step S2205, it is determined whether the conversation requires thought before the response, and if the answer is yes, proceed to step S2210, and if the answer is no, proceed to step S2209.

[0040] As shown in the figure, the final evaluation result of the impact of the conversation content based on the case classification in steps S2202 to S2205 is obtained in steps S2206 to S2210, and in this embodiment, the impact is evaluated on a five-level scale of L0 to L4.

[0041] In step S2206, regardless of the state of the person eating, an immediate call is required, and the impact is evaluated as level L0 "immediate call", which is an exceptional / urgent situation. In step S2207, if only the ears are open, a call is possible, and the impact is evaluated as level L1 "call without response". In step S2208, if the ears are open and the mouth is somewhat open, a call is possible, and the impact is evaluated as level L2 "call requiring a simple response" (call requiring a simple response). In step S2209, if the mouth is empty, a call is possible, and the impact is evaluated as level L3 "call that cannot be handled if there is food in the mouth". In step S2210, if the eyes, ears, and mouth are open, a call is possible, and the impact is evaluated as level L4 "call that cannot be handled unless the hearing, vision, and mouth are open".

[0042] As described above, level L0 is an exceptional case that requires urgent intervention, while levels L1 to L4 are classified in stages from cases where the user only needs to ignore the conversation content to cases where a reply is required and the reply requires a lot of effort. In other words, the levels are set according to the amount of effort required for the user to respond to the conversation content (including cases where the user only ignores the conversation without replying).

[0043] In addition, the conversation judgment unit 24 described later judges whether the diner actually has the effort required to respond to the conversation content quantified as the influence level L1 to L4 based on the diner's situation, and judges it to be an appropriate timing when the diner's situation is sufficient for the effort (when the diner has the effort to respond), and judges it to be an inappropriate timing when the diner's situation is insufficient for the effort (when the diner does not have the effort to respond). However, the level L0 is an exception, and the timing is always judged to be appropriate regardless of the diner's situation.

[0044] In this embodiment, the degree to which the diner is making an effort to respond to the conversation content can be determined based on the condition of the eyes (hearing), ears (vision) and mouth evaluated by each section 231, 232, 233 of the diner status evaluation section 23, and the flow in Figure 4 can be used to determine whether the conditions of the eyes, ears and mouth are appropriate for the conversation content by dividing them into levels, and the conversation judgment section 24 can determine whether the current time t is an appropriate timing for speaking about the conversation content.

[0045] 5 is a flowchart showing the process of the visual occupancy state estimation unit 231 as details of step S231 according to one embodiment. (Since the subject of each step in FIG. 5 is the visual occupancy state estimation unit 231, a description of the subject of the operation will be omitted.)

[0046] In step S2311, the line of sight of the person eating the meal is detected, and the process proceeds to step S2312. In step S2312, it is determined whether the line of sight detected in step S2311 is in a state of continuously gazing at a specific position for a certain period of time until the current time t, and if the result is positive, the process proceeds to step S2314, and if the result is negative, the process proceeds to step S2313.

[0047] In step S2313, an inference is obtained that the diner's vision is not occupied by other media (a viewing medium other than food, such as a television or a smartphone) or by another person present in the vicinity of the diner, i.e., the visual occupancy situation is unoccupied (eyes are open). In step S2314, an inference is obtained that the diner's vision is occupied by other media or by another person, i.e., the visual occupancy situation is occupied (eyes are not open).

[0048] In addition, the visual occupancy state estimation unit 231 can determine whether the direction of the gaze of the person eating changes along with the time progression of the video and is not staring at a specific target (another media or another person) or whether it is constant and staring at a specific target by applying gaze detection of any existing method using deep learning or the like to the video obtained by the imaging unit 11. Whether the direction of the gaze is changing or constant can be determined as constant if the range of change in the gaze direction within a certain period of time falls within a predetermined threshold.

[0049] The gaze may be detected by detecting the center of the eyeball and the center of the iris using face tracking called OpenFace, which is an existing gaze detection method, or various gaze detection methods disclosed in the following non-patent document 1 may be used. [Non-Patent Document 1] Takashi Nagamatsu; Yusuke Kanno; Kentaro Takemura. Research trends in calibration-free gaze measurement methods. Journal of Human Interface Society, 2021, 23.1: 73-88.

[0050] 6 is a flowchart showing the process of the auditory occupation state estimation unit 232 as details of step S232 according to one embodiment. (The auditory occupation state estimation unit 232 is the subject of each step in FIG. 6, so a description of the subject of the operation will be omitted.)

[0051] In step S2321, a recording for a certain period up to the current time t is obtained from the recording unit 12 as a voice around the diner, and the process proceeds to step S2322. In step S2322, voice analysis is applied to the voice obtained in step S2321 to determine whether or not the diner's speech is included, and if the result is positive, the process proceeds to step S2323, and if the result is negative, the process proceeds to step S2324. In step S2323, the process determines whether or not the speech of another person (a stranger) or other media (television, etc.) present near the diner is detected in the result of the voice analysis (not only the speech of the diner but also other people), and if the result is positive, the process proceeds to step S2325, and if the result is negative, the process proceeds to step S2326. In step S2324, the process determines whether or not the speech of another person or other media (television, etc.) present near the diner is detected in the result of the voice analysis, and if the result is positive, the process proceeds to step S2327, and if the result is negative, the process proceeds to step S2328. When a diner who is the subject of judgment by conversation judgment device 20 is eating alone, it is assumed that what is detected in steps S2324 and S2323 is speech from other media, and when there are other people in the vicinity of the diner, it is assumed that what is detected in steps S2324 and S2323 is speech from other people and / or other media. (Similar situations are assumed for situations in which the vision is occupied by visual occupancy situation estimation unit 231 in Fig. 5 described above.) This embodiment can handle either of these cases.

[0052] As shown in the figure, steps S2325 to S2328 obtain an estimate of the diner's auditory occupancy state by judging it in four stages St.1 to St.4 from the loud side to the quiet side (from the side where the ear is not available due to concentrating on others or the television, etc. to the side where the ear is available).

[0053] Step S2325 determines stage St.1 "The diner's hearing is occupied by another media, etc." (The diner is concentrating on listening to the television, etc., while interjecting his / her own response, or is concentrating on a conversation with another person). Step S2326 determines stage St.2 "The diner's hearing may no longer be occupied by another media or another person." Step S2327 determines stage St.3 "The diner's hearing may be occupied by another media or another person." Step S2328 determines stage St.4 "The diner's hearing is not occupied by another media or another person."

[0054] The auditory occupancy state estimation unit 232 can use any existing voice analysis method to determine whether the recording includes the diner's speech, the speech of another media such as television, or the speech of another person. For example, the voiceprint of the diner may be acquired in advance, and it may be determined whether only the voiceprint of the diner is detected from the recording, whether the voiceprints of both the diner and another person (a speaker of another media or another person actually present near the diner) are detected, or whether only the voiceprint of another person is detected. In addition, a speaker identification technology from a voice such as that disclosed in the following non-patent document 2 may be used. For detecting the speech of another media or another person present near the diner, the detected voice may be converted into text and its content may be analyzed, and a distinction may be made as to whether the speech of a character from a drama or the like from television or the like corresponds to a state of concentrating on watching television or the like, whether the speech of a character from a drama or the like from television or the like is not detected, which indicates a state of not concentrating on the speech, or whether the speech of another person present near the diner is detected and the speech of the other person is detected and the person is concentrating on the speech. [Non-Patent Document 2] Voice recognition API service "AmiVoice Cloud Platform" can distinguish speakers [online], February 7, 2022 (Monday) Keizo Hikawa (IT Leaders Editorial Department), [Searched August 28, 2023], Internet<URL:https: / / it.impress.co.jp / articles / - / 22679>

[0055] 7 is a flowchart showing the processing contents of the oral occupancy state estimation unit 233 as details of step S233 according to one embodiment. (Since the oral occupancy state estimation unit 233 is the subject of operation in each step in FIG. 7, a description of the subject of operation will be omitted.)

[0056] In step S2331, video and eating state (swallowing state) data over a certain period up to the current time t is obtained from the imaging unit 11 and eating state monitoring unit 13, the data is analyzed to obtain an eating behavior analysis result, and then the process proceeds to step S2332. In step S2332, it is determined whether the eating behavior analysis result corresponds to chewing or not, and if it is positive, the process proceeds to step S2324, and if it is negative, the process proceeds to step S2333. In step S2333, it is determined whether the eating behavior analysis result corresponds to there being residue in the mouth, and if it is positive, the process proceeds to step S2335, and if it is negative, the process proceeds to step S2336.

[0057] As shown in the figure, steps S2334-S2336 obtain an estimation result by determining the occupancy state of the eater's mouth from the larger side to the smaller side into three stages 1, 2, and 3. Step S2334 determines stage 1 as "eater is in the middle of chewing". Step S2335 determines stage 2 as "eater's mouth is occupied by food (not chewing)". Step S2336 determines stage 3 as "no food in eater's mouth".

[0058] The mouth occupancy state estimation unit 233 can use any existing method to determine whether the eater is chewing from the video. For example, the movement of the lower jaw may be detected by the aforementioned OpenFace technology (a technology that can detect each element of the face, not just the line of sight), and when it is determined that the lower jaw is moving, it may be determined that the eater is chewing. Alternatively, the eating state monitoring unit 13 may be configured to include a dedicated biosensor that detects chewing, and the determination may be made from the output of the chewing sensor. For example, the chewing sensor disclosed in the following Non-Patent Document 3 may be used as the chewing sensor. It may also be possible to detect chewing sounds from sounds acquired by a throat microphone attached to the eater's throat using a model such as deep learning, and to determine whether the eater is chewing. [Non-patent literature 3] Chewing meter "bitescan" [Searched on August 28, 2023], Internet<URL:https: / / jp.sharp / business / bitescan / >

[0059] Furthermore, the mouth occupancy state estimation unit 233 can make the determination in step S2333 by confirming whether or not the food in the mouth has been swallowed after chewing using any existing method. For example, the eating state monitoring unit 13 can be configured to include a swallowing sensor, which is a dedicated neck-worn biosensor as described in Non-Patent Document 4, and swallowing can be determined by detecting swallowing within a certain period of time up to the current time t. The presence or absence of swallowing may be determined by detecting swallowing sounds using a model such as deep learning for sounds acquired by a throat microphone worn by the eater on the throat. [Non-patent literature 4] GOKURI AI-based swallowing measurement "bitescan" [Retrieved August 28, 2023], Internet<URL:https: / / www.plimes.com / >

[0060] 8 is a diagram showing a method of judgment by the speech judgment unit 24 according to an embodiment. As shown in the figure, the speech judgment unit 24 judges whether it is okay to actually have a conversation based on the influence (levels L0 to L4) of the conversation contents obtained by the speech influence evaluation unit 22 for 2*4*3=24 different diners' situations, which are combinations of the following: two stages St.1 and St.2 of occupancy and no occupancy for vision; four stages St.1, St.2, St.3, and St.4 of occupancy from the larger to the smaller side for hearing; and three stages 1, 2, and 3 of occupancy from the larger to the smaller side for the inside of the mouth. Note that when the influence of the conversation contents is level L0, this is an exception to the emergency situation, and the diners are always spoken to regardless of the situation.

[0061] In FIG. 8, "◯" indicates a judgment to speak (i.e., a judgment that the current time t is an appropriate timing to speak the conversation content), "△" indicates a judgment to speak (although speaking is not actively recommended), and "-" indicates a judgment not to speak. For example, in the seventh line of FIG. 8, when the eater's situation corresponds to visual occupancy (stage St.1), auditory occupancy stage St.3 (third stage out of four stages from the high occupancy side), and mouth occupancy stage 1 (first stage, the highest stage out of three stages from the high occupancy side), and the influence level of the conversation content is L1, "if only the ears are free, you can speak", and the auditory occupancy in the seventh line is stage St.3 (third stage out of four stages from the high occupancy side), and the ears are just barely free, so it is "△". Interactive speaking may be realized by using the judgment as shown in FIG. 8. For example, in the case of "◯", except in the case of level L0, it is assumed that the eater will respond to the speaking, and if there is no response, the eater may be spoken to again. In addition, by assuming that there is a possibility that no reply will be received in the case of "△", repeated attempts to speak to the subject may not be made even if no reply is received.

[0062] As described above, according to the embodiment of the present invention, when an agent speaks to a diner, it is possible to avoid problems such as interrupting the diner or speaking at a time when the diner is unable to respond.

[0063] Various supplementary examples, modified examples, additional examples, etc. will be described below.

[0064] (1) According to the conversation determination system 100 of this embodiment, for example, the conversation content can be set to prevent the user from swallowing food or to promote health through food. This makes it possible to contribute to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs), which is to “Ensure healthy lives and promote well-being for all at all ages.”

[0065] (2) The present invention is not limited to a program that automatically converses with a single subject using a mechanical voice (a dialogue agent), but can also be applied to cases where an expert remotely monitors multiple diners and determines the appropriate timing to converse with a diner over a network when the expert determines that it is an appropriate time to converse.

[0066] In other words, in the speech determination system 100, the process of preparing the conversation content in the conversation content management unit 21 and the process of speaking to the diners about this conversation content in the speech processing unit 25 (at an appropriate timing determined by the speech determination unit 24) have been described as being automatically processed by a computer executing a program as a dialogue agent, but as a variation of the present invention, as shown in Figure 9, an advisor (such as an expert providing dietary advice, etc.) using a terminal device 30 remotely monitors the diners and transmits advice about diet, etc. remotely as conversation content, so that the process of preparing the conversation content in the conversation content management unit 21 and the process of speaking to the diners about the conversation content in the speech processing unit 25 may be left to manual processing by the remote advisor.

[0067] FIG. 9 is a functional block diagram of an interaction judgment system 100 relating to a modified example. The changes to the configuration of FIG. 1 include the addition of a terminal device 30, which is a general computer used by an advisor who conducts conversations remotely, and, as described above, part of the processing of the conversation content management unit 21 and the processing of the interaction processing unit 25 are changed from automatic processing to one that accepts manual input from the advisor.

[0068] In this case, the remote advisor prepares conversation contents for advice in line with the diners' meal progress as a scenario and manually inputs them into the conversation content management unit 21, which acquires conversation contents in line with the current meal progress stage by analyzing the video output from the monitoring device 10, and when the conversation determination unit 24 determines that the current time t is appropriate as the timing for speaking to the relevant conversation content, it notifies the advisor, and the advisor who receives the notification conveys the relevant conversation content by voice input via a microphone from a remote location, and the conversation content is conveyed to the diners by voice output from the conversation processing unit 25. At this time, the conversation from the advisor to the diners (and vice versa) may be realized by a two-way video conference method using not only voice but also video between the advisor and the diners. As a method similar to the method in which the conversation content management unit 21 sets the conversation content based on the evaluation results of the eating situation evaluation unit 23 described above, the conversation content management unit 21 may determine, based on the evaluation results of the eating situation evaluation unit 23, not the conversation content itself, but an impact level (L0 to L4) of the conversation content that is suitable as the timing to speak to the current time t, and notify the advisor, enabling the advisor to speak to the diner about the conversation content of the determined impact level.

[0069] (3) The diner state evaluation unit 23 evaluates three types of conditions, namely, vision, hearing, and the inside of the mouth, but it may evaluate only a part of these conditions. In other words, any one or two of the three units 231, 232, and 233 constituting the diner state evaluation unit 23 may be omitted.

[0070] (4) Fig. 10 is a diagram showing an example of a hardware configuration of a general computer. Each part of the speech determination system 100 (including the speech determination device 20 and the terminal device 30) can be realized as one or more computer devices 70 having such a configuration. When each part is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device (memory) that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device (storage), a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 81, a speaker 82, a sensor 83, a microphone 84, and a bus BS for transmitting and receiving data between them.

[0071] Each unit of the conversation determination system 100 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and the GPU 72.

[0072] The photographing unit 11 may be realized by a camera 81, the recording unit 12 may be realized by a microphone 84, and various dedicated biosensors constituting the dietary state monitoring unit 13 may be realized by a sensor 83. Audio output from the conversation processing unit 25 may be realized by a speaker .

[0073] (5) In cases where privacy protection of diners is required (for example, when used in a conversation agent in a restaurant), all or part of the information acquired by the monitoring device 10 may be used only temporarily for processing by the speech determination device 20, and the records may be deleted thereafter. Alternatively, the video and audio may be subjected to concealment processing using any privacy removal processing of existing technology so that the state of the diners can be grasped but the details of their privacy cannot be grasped, and these may be acquired by the monitoring device 10. On the other hand, in cases where an advisor uses the speech determination device 20, for example, the state of the conversation with the diners may be recorded, and the recorded data may be used to improve the timing of the speech according to the contents of the speech to be more appropriate for the diners. For example, the recorded data may be used as a record to improve a situation where a keyword in the conversation touches a heartstring of the diners and the diners cannot stop talking, causing the conversation to be interrupted for a long time. [Explanation of symbols]

[0074] 100... speaking judgment system, 10... monitoring device, 20... speaking judgment device, 30... terminal device, 11... imaging unit, 12... recording unit, 13... eating state monitoring unit, 21... conversation content management unit, 22... speaking influence evaluation unit, 23... eating person state evaluation unit, 24... speaking judgment unit, 25... speaking processing unit, 231... visual occupancy state estimation unit, 232... auditory occupancy state estimation unit, 233... oral occupancy state estimation unit

Claims

1. a first process for evaluating an influence of a conversation to be made with the diner at a current time; A second process of evaluating the current state of the diner as a diner state; and a third process of determining whether the current time is an appropriate timing to speak to the diner about the conversation content based on the influence degree and the diner's status.

2. 2. The conversation determination device according to claim 1, wherein in the second process, all or some of the visual occupancy state, the auditory occupancy state, and the mouth occupancy state of the diner are evaluated as the diner state.

3. The conversation judgment device described in claim 1, characterized in that in the second process, by analyzing the video of the diner, at least the visual occupancy state of the diner is estimated as the diner's status, regarding whether the diner is looking at a viewing medium other than the meal or at other people.

4. The conversation judgment device described in claim 1, characterized in that the second process estimates at least the diner's auditory occupancy status as the diner's situation, by analyzing recordings made around the diner, regarding whether the diner's attention is directed to audio-visual media other than the meal or to other people.

5. The conversation judgment device described in claim 1, characterized in that in the second process, by analyzing video of the diner and / or using the output of a biosensor worn by the diner, at least the occupancy status of the diner's mouth, as the diner's status, regarding whether or not chewing is occurring, and, if not, regarding the presence or absence of residue in the mouth, is estimated.

6. In the first process, the influence degree is evaluated as a degree of effort of the diner's response to the conversation content, The conversation judgment device described in claim 1, characterized in that in the third process, it is determined whether the conversation content is appropriate as a timing to speak to the diner based on whether the diner in the diner situation has the effort.

7. The conversation judgment device described in claim 1, characterized in that if the conversation content corresponds to an emergency, the third process always determines that the current time is an appropriate timing to speak to the diner about the conversation content, regardless of the diner's situation.

8. When it is determined in the third process that the timing is appropriate, The content of the conversation is output to the diner as a machine voice, or The conversation determination device according to claim 1, further comprising a fourth process of outputting to the diner the content of the conversation uttered by the subject who has been notified that it is the appropriate time.

9. The conversation determination device according to claim 1 , further comprising a process of setting the conversation content based on the diner situation evaluated in the second process.

10. 10. A program for causing a computer to function as the conversation determination device according to claim 1.

Images