Speech control device, speech control method, and speech control program

The speech control device enhances robot interaction by identifying target users and optimizing speech timing through user detection, sound localization, and behavior estimation, ensuring non-intrusive and timely engagement.

JP7586746B2Active Publication Date: 2024-11-19NIPPON HOSO KYOKAI
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021049491
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-24
Publication Date
2024-11-19
Estimated Expiration
2041-03-24

AI Technical Summary

Technical Problem

Conventional methods fail to determine the target audience and control the timing of speech when multiple people are viewing content with a robot, leading to potential interference with user interactions.

Method used

A speech control device that includes user detection, television detection, sound source localization, behavior estimation, and utterance determination units to identify the target user and optimize speech timing, using a panoramic image generation and direction information conversion to enhance precision.

Benefits of technology

The device allows a robot to selectively engage in conversations with users without interrupting their activities, adapting speech motions based on user behavior and reactions, ensuring timely and appropriate interactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586746000001
    Figure 0007586746000001
  • Figure 0007586746000002
    Figure 0007586746000002
  • Figure 0007586746000003
    Figure 0007586746000003
Patent Text Reader

Abstract

To provide an utterance control device that enables a robot to determine an object person to speak among a plurality of users and timing thereof.SOLUTION: An utterance control device 1 comprises: a user detection part 11 which detects a user in an image; a television set detection part 13 which detects a television set in the image; a direction information conversion part 14 which converts position information on the user and television set into direction information; a speech analysis part 15 which attenuates a speech input from the direction of the television set and then acquires speech information classified into directions by users through sound source localization; an action estimation part 16 which acquires actions of the users with a learning model based upon a plurality of image frames in time series; an utterance determination part 17 which calculates utterance frequencies based upon the speech information classified by the users and determines the user whose utterance frequency is equal to or less than a threshold as the object user to whom the robot utters; and an utterance execution part 18 which lets a robot execute a predetermined utterance operation according to values of listening coefficients defined by actions of the users in advance.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an apparatus, a method, and a program for controlling the speech behavior of a robot. [Background technology]

[0002] Conventionally, a method has been devised for controlling the timing at which a robot that watches video content such as television programs together with a user at home speaks without interfering with the user's actions. For example, Patent Document 1 proposes a method of using information indicating at least one of a user's speech, behavior, and emotions to determine whether or not to speak based on an utterance decision table. Moreover, Patent Document 2 proposes a method for determining whether or not a target person is in a state that allows conversation based on the results of estimating the behavior and state of the target person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-140477 A [Patent Document 1] International Publication No. 2019 / 187256 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional methods, the person to whom the robot would speak was fixed, and when multiple people were watching content, it was not possible to determine the person to speak to and control the timing of the speech.

[0005] The present invention aims to provide a speech control device, a speech control method, and a speech control program that enable a robot that watches television together with humans to decide who to speak to from among multiple users and the timing to do so. [Means for solving the problem]

[0006] The speech control device of the present invention includes a user detection unit that detects a user from an image captured around the robot, a television detection unit that detects a television from the image, a direction information conversion unit that converts position information of the user and the television into direction information, a voice analysis unit that attenuates voice input from the direction of the television among sounds collected around the robot and acquires voice information classified into directions for each user by sound source localization, a behavior estimation unit that acquires the behavior of the user using a learning model based on a plurality of image frames in a time series capturing the direction of each user, a speech determination unit that calculates a speech frequency based on the voice information classified for each user and determines users whose speech frequency is below a threshold as target users to whom the robot will speak, and a speech execution unit that causes the robot to perform a predetermined speech action depending on the value of a listening coefficient that is predefined for each behavior of the user.

[0007] The speech control device may include a panoramic image generating unit that generates a panoramic image by superimposing feature points from the plurality of images, and the directional information converting unit may convert coordinates in the panoramic image into directional information.

[0008] The sound analysis unit may correct the direction of the user with the direction obtained by the sound source localization.

[0009] The speech execution unit may emit light for a predetermined period of time before utterance as an initial operation of the speech action.

[0010] The speech execution unit may output a notification sound before speaking as an initial operation of the speech action.

[0011] The speech execution unit may perform speech only when there is a voice input from the target user after the initial operation.

[0012] The speech control method of the present invention includes a user detection step of detecting a user from an image captured around the robot, a television detection step of detecting a television from the image, a direction information conversion step of converting position information of the user and the television into direction information, a voice analysis step of attenuating voice input from the direction of the television among sounds collected around the robot and acquiring voice information classified into directions for each user by sound source localization, a behavior estimation step of acquiring the behavior of the user by a learning model based on a plurality of image frames in a time series capturing the direction of each user, a speech determination step of calculating a speech frequency based on the voice information classified for each user and determining users whose speech frequency is below a threshold as target users to whom the robot will speak, and a speech execution step of causing the robot to execute a predetermined speech operation according to a value of a listening coefficient predefined for each behavior of the user.

[0013] The speech control program according to the present invention causes a computer to function as the speech control device. Effect of the Invention

[0014] According to the present invention, a robot that watches television together with humans can determine a target user to speak to from among a plurality of users. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing a usage scene of a robot equipped with a speech control device in an embodiment. FIG. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of the speech control device according to the embodiment. [Diagram 3] 10 is a flowchart showing a flow of a speech control process in the embodiment. [Figure 4] 10 is a flowchart showing details of a user detection process in the embodiment. [Diagram 5] 5A to 5C are diagrams illustrating details of a panoramic image generating process in the embodiment. [Figure 6] 10 is a flowchart showing details of a television detection process in the embodiment. [Figure 7] 10 is a flowchart showing details of a direction information conversion process in the embodiment. [Figure 8] 10A and 10B are diagrams illustrating examples of relationships between coordinate information and direction information in the embodiment. [Figure 9] 5 is a flowchart showing details of a voice analysis process in the embodiment. [Figure 10] 10 is a flowchart showing details of a behavior inference process in the embodiment. [Figure 11] 10 is a flowchart showing details of an utterance determination process in the embodiment. [Figure 12] FIG. 10 is a diagram illustrating a user information table according to the embodiment. [Figure 13] FIG. 10 is a diagram illustrating a robot operation table in the embodiment. [Figure 14] 10 is a flowchart showing details of an utterance execution process in the embodiment. [Figure 15] FIG. 2 is a diagram illustrating an example of an utterance sentence database in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An example of an embodiment of the present invention will now be described. The speech control device of this embodiment is installed in a robot that watches video content such as television programs together with a user, and estimates the behavior of each user, particularly when there are multiple users, and determines the timing and target of speech. As a result, the robot makes speech that will start a conversation between users without interrupting their viewing behavior.

[0017] FIG. 1 is a diagram showing a usage scene of a robot equipped with a speech control device 1 according to the present embodiment. In a situation where users (eg, U1 and U2) are watching video content such as a television program, a robot R that watches the video content together with the users is provided on a table or the like.

[0018] By the processing of the speech control device 1 described later, for example, when users are enjoying a conversation triggered by the contents they are watching, the robot R can avoid interrupting the conversation by speaking on its own. Also, the speech control device 1 prevents the user from getting tired of the speech action of the robot R by changing the speech action of the robot R according to the user's actions and deciding whether to speak or not while checking the user's reaction. Furthermore, for example, in a situation where one user loses interest in viewing the content and starts operating an electronic device such as a smartphone, the robot R can select such a user and become a conversation partner.

[0019] FIG. 2 is a diagram showing a functional configuration of the speech control device 1 in this embodiment. The speech control device 1 includes a control unit 10, an imaging unit 20, a voice input unit 30, a driving unit 40, a voice output unit 50, a storage unit 60, a time unit 70, and a lighting unit 80.

[0020] The control unit 10 may be a computer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The CPU reads out an operation program (utterance control program) stored in the ROM and executes it on the RAM, whereby the control unit 10 controls the operation of the entire utterance control device 1.

[0021] The control unit 10 detects the direction of the user and the direction of the television from the image of the robot surroundings acquired by the imaging unit 20, attenuates the sound from the direction of the television input from the audio input unit 30, and inputs only the user's voice. After that, the control unit 10 estimates the behavior of each user, such as whether the user is having a conversation, operating a smartphone, or eating, and determines whether the robot will speak and to whom it will speak from the user's behavior and speech frequency, and then executes a speaking action to output a voice from the audio output unit 50. The detailed functions of the control unit 10 will be described later.

[0022] The imaging unit 20 is a camera equipped with a lens and an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures images of the robot's surroundings. As shown in FIG. 2, the imaging unit 20 is configured as a camera array in which multiple cameras are mounted on the circumference of the robot. The imaging unit 20 may be configured such that one camera periodically captures images of the surroundings while rotating around a circle. In this case, the orientation of the imaging unit 20 (camera) may be controlled to match the orientation of the robot's housing (face orientation).

[0023] The voice input unit 30 is composed of a microphone, an A / D converter, etc., and collects sounds around the robot. The voice input unit 30 amplifies the collected sounds, and outputs digital signal data that has been subjected to A / D conversion, encoding, etc. to the control unit 10. The voice input unit 30 is installed on the side of the robot as shown in FIG. 1, and is configured as a microphone array in which multiple microphones are mounted on the circumference of the robot.

[0024] The driving unit 40 is a motor attached to a stand or a housing on which the audio output unit 50 is mounted, and outputs the voice of the robot to the user through the speaker of the audio output unit 50. The driving unit 40 may be controlled by the rotation angle, but a potentiometer that acquires the rotation angle as a resistance value may also be attached to the shaft on which the motor is attached, and this resistance value may be used as the control value.

[0025] The audio output unit 50 is composed of a speaker, a D / A converter, etc., and outputs an analog audio signal obtained by decoding, D / A converting, amplifying, etc., the audio data transferred from the control unit 10 from a speaker placed on the robot's face, for example, as shown in Figure 1.

[0026] The storage unit 60 is a storage medium that stores data required by the control unit 10, and may be, for example, a non-volatile flash memory, a solid state drive (SSD), a hard disk, etc. The storage unit 60 stores information about the user, such as face information, voice information, and a spoken sentence database, which will be described later.

[0027] The time unit 70 compares the time when the user is detected with the time when the user's voice is acquired, and measures the time when multiple image frames are acquired when estimating the user's behavior, which will be described later.

[0028] The lighting unit 80 lights up when the robot speaks or when checking the user's reaction to speaking. The lighting unit 80 is composed of a light source element such as an LED (Light Emitting Diode), and is arranged so that the entire circumference of the robot emits light, for example, as shown in FIG.

[0029] The control unit 10 includes a user detection unit 11, a panoramic image generation unit 12, a television detection unit 13, a directional information conversion unit 14, a voice analysis unit 15, a behavior estimation unit 16, an utterance determination unit 17, and an utterance execution unit 18.

[0030] FIG. 3 is a flowchart showing the flow of the speech control process by the control unit 10 of the speech control device 1 in this embodiment. Through the speech control process, the control unit 10 grasps the direction of the television and the user around the robot, and determines the user to whom the robot is to be spoken and whether or not to actually speak to this user.

[0031] In step S101, the control unit 10 acquires an image of the surroundings of the robot captured by the imaging unit 20.

[0032] In step S102, the control unit 10 (user detection unit 11) detects users around the robot (user detection process). The details of the user detection process will be described later.

[0033] In step S103, the control unit 10 judges whether or not a user has been detected in step S102. If the judgment is YES, the process proceeds to step S104, and if the judgment is NO, the process returns to step S101. The processes of steps S101 to S102 are repeatedly executed until a user is detected.

[0034] In step S104, the control unit 10 (panoramic image generating unit 12) generates a panoramic image from the image of the surroundings of the robot (panoramic image generating process). The panoramic image generating process will be described in detail later.

[0035] In step S105, the control unit 10 acquires pixel information indicating the position of the user in the image from the panoramic image generated in step S104, and stores the pixel information in the storage unit 60.

[0036] In step S106, the control unit 10 (television detection unit 13) detects the position of the television around the robot, extracts pixel information indicating the position of the television, and stores the pixel information in the storage unit 60 (television detection process). The television detection process will be described in detail later.

[0037] In step S107, the control unit 10 (directional information conversion unit 14) converts the pixel information indicating the positions of the user and the television into directional information about the surroundings of the robot (directional information conversion process). The directional information conversion process will be described in detail later.

[0038] In step S108, the control unit 10 acquires voice information from the voice input unit 30.

[0039] In step S109, the control unit 10 (audio analysis unit 15) attenuates the audio input from the television to prevent the television audio and the user's audio from being input simultaneously. When multiple users are present, the audio analysis unit 15 localizes the sound source and identifies the user from the voice direction (audio analysis process). The details of the audio analysis process will be described later.

[0040] In step S110, the control unit 10 (behavior inference unit 16) infers the behavior of each identified user, and stores the behavior in the storage unit 60 (behavior inference process). The behavior inference process will be described in detail later.

[0041] In step S111, the control unit 10 (utterance determination unit 17) determines a user (target user) to whom the robot will make an utterance based on the user's utterance frequency and the estimated behavior (utterance determination process). The utterance determination process will be described in detail later.

[0042] In step S112, the control unit 10 judges whether or not it has been decided to speak to the user. If the judgment is YES, the process proceeds to step S113, and if the judgment is NO, the process ends.

[0043] In step S113, the control unit 10 (the speech execution unit 18) performs a speech action on the target user (speech execution process). The details of the speech execution process will be described later.

[0044] FIG. 4 is a flowchart showing details of the user detection process (step S102 in FIG. 3) in this embodiment.

[0045] In step S201, the user detection unit 11 performs face detection processing from an image acquired by the imaging unit 20. For face detection, for example, Faster R-CNN, which is an open source library, is used, but the present invention is not limited to this, and various methods such as OpenCV can be applied.

[0046] In step S202, the user detection unit 11 extracts features of the detected face and stores the extracted features in the storage unit 60. As the feature, for example, a Haar-like feature that handles brightness information of a local region in an image is used, but it is not limited to this. Various features can be applied, such as a Histogram of Oriented Gradients (HOG) feature that represents the strength of the image gradient as a directional histogram.

[0047] In step S203, the user detection unit 11 determines whether or not a plurality of faces have been detected in step S201. If the determination is YES, the process proceeds to step S204, and if the determination is NO, the process ends.

[0048] In step S204, the user detection unit 11 classifies the detected users based on the feature amounts. For example, when feature amounts of face 1 and face 2 are extracted at time T1, the user detection unit 11 assigns different user IDs to each feature amount, such as face 1: user 1 and face 2: user 2, and stores them in the storage unit 60. Furthermore, when feature amounts of face 3 are extracted at time T2, the user detection unit 11 compares the feature amounts of face 3 with the feature amounts of face 1 and face 2 stored in the storage unit 60, and classifies face 3 as user 2 if the difference with the feature amount of face 2 is smaller than a predetermined criterion, for example.

[0049] FIG. 5 is a diagram for explaining details of the panoramic image generating process (step S104 in FIG. 3) in this embodiment.

[0050] When the panoramic image generator 12 acquires image 1 from a camera in direction 1 in the camera array (imaging unit 20) and image 2 from a camera in direction 2, it searches for a portion where feature points in image 1 and image 2 match, and overlays and synthesizes image 1 and image 2 at this matching portion. The panoramic image generator 12 generates a panoramic image by performing this process on all images of the robot's surroundings.

[0051] To extract feature points, for example, Speeded Up Robust Features (SURF), which expresses local features on an image as feature vectors, may be used. However, the present invention is not limited to this, and various other methods such as Scale Invariant Feature Transform (SIFT) can also be used.

[0052] FIG. 6 is a flowchart showing the details of the television detection process (step S106 in FIG. 3) in this embodiment.

[0053] In step S301, the television detection unit 13 detects the position of the television from the image acquired from the imaging unit 20. For example, the open source library Faster R-CNN is used for the television detection.

[0054] In step S302, the television detection unit 13 determines whether or not a television is present, that is, whether or not a television was detected in step S301. If the determination is YES, the process proceeds to step S303, and if the determination is NO, the process ends.

[0055] In step S303, the television detection unit 13 acquires pixel information (coordinates on the panoramic image) of the center position where the television is located in the image, and stores it in the storage unit 60.

[0056] FIG. 7 is a flowchart showing details of the direction information conversion process (step S107 in FIG. 3) in this embodiment.

[0057] In step S401, the direction information conversion unit 14 acquires pixel information of the position of the user extracted by the user detection unit 11.

[0058] In step S 402 , the direction information conversion unit 14 obtains pixel information on the position of the television extracted by the television detection unit 13 .

[0059] In step S403, the direction information conversion unit 14 converts the pixel information of the user and the television into direction information about the surroundings of the robot.

[0060] FIG. 8 is a diagram illustrating an example of the relationship between coordinate information and direction information in this embodiment. In this example, the directional information conversion unit 14 converts pixel information (x1, y1) of a user U1, pixel information (x2, y2) of a user U2, and pixel information (x TV ,y TV ) has been obtained. For example, if the size of the panoramic image is (1800, 360), the pixel information of user U1 is (500, 200), and the position where x is 0 is 0 degrees in front of the robot, the angular resolution is 1800 / 360 = 5 (pixels / degree), and the angle θ1 to the direction of user U1 can be calculated as 500 / 5 = 100 degrees. Similarly, the pixel information of the positions of user U2 and the television can be calculated as directional information θ2, θ TV is converted to

[0061] In this example, the directional information is expressed as an angle from 0 degrees to 360 degrees around the robot, but this is not limited to this. For example, the resistance value obtained from the potentiometer of the drive unit 40 may be used as the directional information. In this case, the resistance value and the rotation angle of the driving unit 40 are associated in advance, and the direction information conversion unit 14 converts the position on the image into an angle, and then converts the angle into the resistance value.

[0062] FIG. 9 is a flowchart showing details of the voice analysis process (step S109 in FIG. 3) in this embodiment.

[0063] In step S501, the audio analysis unit 15 attenuates the audio input from the direction of the television in order to prevent the user's voice and the television audio from being input simultaneously.

[0064] In step S502, the audio analysis unit 15 acquires audio only from the direction of the user with the audio from the television direction attenuated.

[0065] In step S503, the voice analysis unit 15 judges whether or not a plurality of users are detected. If the judgment is YES, the process proceeds to step S504, and if the judgment is NO, the process ends.

[0066] In step S504, the voice analysis unit 15 performs sound source localization to acquire voices for each user. For example, the open source software HARK (https: / / www.hark.jp / document / 2.4.0 / hark-document-ja / index.html) or the MUSIC method (Futoshi Asano, Separating Sounds, Measurement and Control, 2004, Vol. 43, No. 4, pp. 325-330) can be used as a method for sound source localization.

[0067] In step S505, the voice analysis unit 15 associates the direction of the sound source with each detected user. For example, if the direction of the sound source and the direction of the user acquired from the image are within a predetermined range, the user and the voice are associated. This allows the voice of each user to be acquired.

[0068] FIG. 10 is a flowchart showing details of the behavior estimation process (step S110 in FIG. 3) in this embodiment.

[0069] In step S601, the activity estimation unit 16 acquires a plurality of image frames in the user direction for each user.

[0070] In step S602, the activity estimation unit 16 estimates the activity for each user, and obtains results such as, for example, User 1 is "conversing," User 2 is "operating a smartphone," and User 3 is "eating."

[0071] Here, to estimate the actions, a learning model trained on a large-scale video dataset with labeled actions is used. For example, Kinetics or ActivityNet can be used as the learning dataset. The user's behavior is estimated using a Convolution Neural Network (CNN), although the estimation method is not limited to this.

[0072] In step S603, the behavior estimation unit 16 stores the estimation results in the storage unit 60 as user behavior for each user.

[0073] FIG. 11 is a flowchart showing details of the speech determination process (step S111 in FIG. 3) in this embodiment.

[0074] In step S701, the speech determination unit 17 counts the number of voice inputs from each user direction. The number of voice inputs is counted when the acquired voice level exceeds a predetermined threshold. At this time, if there are multiple users, the speech determination unit 17 counts the number of voice inputs for each user.

[0075] In step S702, the utterance determination unit 17 stores the number of times the voice has been input in the storage unit 60 as the number of times the user has spoken.

[0076] In step S703, the utterance determination unit 17 calculates the frequency of utterances by the user within a predetermined period of time. The speech frequency can be set as an appropriate time period, such as the number of utterances within 1 minute, the number of utterances within 5 minutes, the number of utterances within 10 minutes, etc., and a predetermined threshold value for the number of utterances is set according to the set time period.

[0077] In step S704, the utterance determination unit 17 determines whether or not the user's utterance frequency is equal to or lower than a predetermined threshold value If the determination is YES, the process proceeds to step S705, and if the determination is NO, the process ends.

[0078] In step S705, the utterance determination unit 17 determines that the utterance frequency is low and that the possibility that the user is engaged in conversation is low, and selects the user as a target user to whom the robot will utter. In other words, if the frequency of an utterance exceeds the threshold, the utterance determination unit 17 determines that there is a high possibility that the user is engaged in conversation, and does not select the target person for the robot to speak to.

[0079] For example, when the speech frequency of user 1 within one minute is 5 and the threshold is 20, the speech frequency is below the threshold, so the speech determination unit 17 determines user 1 as the target user to speak. Also, when there are multiple users, and the speech frequency of user 1 within one minute is 30, the speech frequency of user 2 within one minute is 5, and the threshold is 20, the speech determination unit 17 determines user 2 as the target user to speak. When there are a plurality of users who speak infrequently, the utterance determination unit 17 may determine, for example, the user who speaks the least frequently among them, or a user selected randomly, as the target user.

[0080] The speech execution unit 18 executes a speech action corresponding to the action of the user selected by the speech determination unit 17 to make an utterance, based on the robot action table stored in the storage unit 60. Here, for each of the estimated user actions, a listening coefficient fu indicating the degree of ease of listening to the robot's speech is set in advance, and the robot's behavior is defined according to this value.

[0081] FIG. 12 is a diagram illustrating a user information table in this embodiment. The user information table stores, for each detected user, the image position (coordinates) on the panoramic image, the direction (angle), the speech frequency, the estimated user behavior, and the listening coefficient fu corresponding to the behavior.

[0082] FIG. 13 is a diagram illustrating a robot operation table in this embodiment. For example, when the user is talking (fu=0.1), the utterance execution unit 18 turns on the LED as an initial operation, and allows the user to speak if there is a reaction from the user. When the user is watching television (fu=0.5), the utterance execution unit 18 turns on the LED along with the notification sound "Um..." as an initial operation, and allows the user to speak if there is a reaction from the user. When the user is operating an electronic device such as a smartphone (fu=0.9), the utterance execution unit 18 allows the user to speak without prior operation such as turning on the LED.

[0083] In this way, the utterance execution unit 18 changes the robot's behavior to make it speak according to the listening coefficient fu preset for the user's behavior. Also, the utterance execution unit 18 performs an operation to prevent the user from getting bored by continuing to speak, such as not making the robot speak when the user has no reaction and is not interested in the robot.

[0084] FIG. 14 is a flowchart showing details of the speech execution process (step S113 in FIG. 3) in this embodiment.

[0085] In step S801, the speech execution unit 18 acquires a listening coefficient fu based on the behavior of the target user from the storage unit 60. The listening coefficient fu is set in advance for each behavior, and here, for example, a value from 0 to 1.0 is set in increments of 0.1.

[0086] In step S802, the utterance executing unit 18 judges whether the listening coefficient fu is large (for example, 0.7 to 1.0) If the judgment is YES, the process proceeds to step S803, and if the judgment is NO, the process proceeds to step S804.

[0087] In step S803, the utterance execution unit 18 executes the operation 3 defined in the robot operation table, i.e., the utterance, and the process ends. The content of the utterance may be, for example, a randomly selected open sentence such as "I want to eat cake" or a question such as "Have you ever been to the Kamo River in Kyoto?" registered in the utterance sentence database.

[0088] In step S804, the utterance executing unit 18 judges whether the listening coefficient fu is medium (for example, 0.3 to 0.6) If the judgment is YES, the process proceeds to step S805, and if the judgment is NO, the process proceeds to step S807.

[0089] In step S805, the speech execution unit 18 executes the operation 2 defined in the robot operation table, that is, outputting a notification sound and turning on an LED. The content of the notification sound may be, for example, a word such as "Um..." or "By the way," which is registered in the utterance sentence database. The LED lighting may be appropriately set so that it is easily recognized by the user, such as blinking five times in five seconds or staying lit for ten seconds.

[0090] In step S806, the speech execution unit 18 judges whether or not there is a reaction from the user to the action 2. For example, if there is no voice input from the user within a certain period of time, it may be judged that there is no reaction. If this judgment is YES, the process proceeds to step S803, and if the judgment is NO, the process proceeds to step S807.

[0091] In step S807, the utterance executing unit 18 judges whether or not the listening coefficient fu is small (for example, 0 to 0.2). If the judgment is YES, the process proceeds to step S808, and if the judgment is NO, the process ends.

[0092] In step S808, the utterance execution unit 18 executes the operation 1 defined in the robot operation table, that is, turning on the LED. Note that the turning on operation may be the same as that in step S805.

[0093] In step S809, the speech execution unit 18 judges whether or not the user has responded to the action 1. The judgment method is the same as that in step S806. If the judgment is YES, the process proceeds to step S803, and if the judgment is NO, the process ends.

[0094] FIG. 15 is a diagram illustrating an example of an utterance sentence database in this embodiment. In the utterance sentence database, for example, a notification sound to be output in motion 2 in the robot motion table, and a disclosure sentence and a question sentence as candidates for an utterance sentence are registered.

[0095] In this example, the notification sound is registered with words such as "Um..." or "By the way..." to attract the user's attention before speaking, but it is not limited to these and could also be a unique robot sound such as "beep" or "pururu" to notify the user of the speech.

[0096] The disclosure sentences are words that express the robot's own feelings and emotions, and include sentences containing words such as "I want to eat" and "I want to go." The questions registered include whether or not the user has been to a particular place or eaten a particular food. In addition, the words for food, place, etc. used in these spoken sentences may be dynamically updated according to the content being viewed, such as subtitle information or image information.

[0097] According to this embodiment, the speech control device 1 targets users who have a low speech frequency among the detected users and causes the robot to execute a speech action according to the listening coefficient corresponding to the behavior. Therefore, a robot equipped with the speech control device 1 and watching television together with a user can determine which of multiple users to talk to without interrupting the conversation between the users, and can flexibly determine the timing of whether or not to talk to the user based on the user's reaction.

[0098] The speech control device 1 generates a panoramic image by extracting feature points of a plurality of images and superimposing them, and can easily grasp the direction in which the user is located based on the coordinates on this panoramic image.

[0099] The speech control device 1 can specify the direction from which the voice was input, that is, the direction in which the user is located, by localizing the sound source of the collected voice, and can appropriately correct the user information.

[0100] The speech control device 1 can make the robot speak at an appropriate time by emitting light for a short time or outputting a notification sound as an initial operation of the speech operation to attract the attention of the target user. Furthermore, the speech control device 1 allows the robot to speak only when there is a response from the user to these initial actions, thereby making it possible to suppress speech at unnecessary times that the user does not want.

[0101] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0102] In this embodiment, the configuration and operation of the speech control device 1 have been mainly described, but the present invention is not limited to this and may be configured as a method or program having each component and for causing a robot to speak.

[0103] Furthermore, the functions of the speech control device 1 may be realized by recording a program for realizing the functions of the speech control device 1 on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.

[0104] The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.

[0105] Furthermore, the term "computer-readable recording medium" may include a recording medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that stores a program for a certain period of time, such as a volatile memory in a computer system that serves as a server or client in such a case. The above program may be a program for realizing part of the above-mentioned functions, or may be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0106] 1 Speech control device 10 Control section 11 User detection section 12 Panorama image generator 13 TV detector 14 Directional information conversion unit 15 Audio Analysis Section 16 Behavior estimation section 17 Speech determination unit 18 Speech execution unit 20 Imaging unit 30 Audio input section 40 Drive unit 50 Audio output section 60 Storage section 70 Time section 80 Lighting section

Claims

1. a user detection unit that detects a user from an image captured around the robot; a television detection unit for detecting a television from the image; a direction information conversion unit that converts position information of the user and the television into direction information; a voice analysis unit that attenuates a voice input from the direction of the television among the voices collected from around the robot, and acquires voice information classified into directions for each user by sound source localization; A behavior estimation unit that acquires a behavior of the user by a learning model based on a plurality of time-series image frames captured in the respective directions of the user; an utterance determination unit that calculates an utterance frequency based on the voice information classified for each user, and determines a user whose utterance frequency is equal to or lower than a threshold as a target user to which the robot will make an utterance; A speech control device comprising: a speech execution unit that causes the robot to execute a predetermined speech action according to a value of a listening coefficient predefined for each action of the user.

2. a panoramic image generating unit that generates a panoramic image by superimposing feature points from the plurality of images; The speech control device according to claim 1 , wherein the direction information conversion unit converts coordinates in the panoramic image into direction information.

3. The speech control device according to claim 1 , wherein the voice analysis unit corrects the direction of the user based on the direction obtained by the sound source localization.

4. The speech control device according to claim 1 , wherein the speech execution unit emits light for a predetermined period of time before utterance as an initial operation of the speech action.

5. The speech control device according to claim 1 , wherein the speech execution unit outputs a notification sound before utterance as an initial operation of the speech action.

6. The speech control device according to claim 4 , wherein the speech execution unit performs speech only when there is a voice input from the target user after the initial operation.

7. a user detection step of detecting a user from an image captured around the robot; a television detection step of detecting a television from the image; a direction information conversion step of converting the position information of the user and the television into direction information; a sound analysis step of attenuating a sound input from the direction of the television among sounds collected from around the robot, and acquiring sound information classified into directions for each user by sound source localization; a behavior estimation step of acquiring a behavior of the user by a learning model based on a plurality of time-series image frames captured in the respective directions of the user; an utterance determination step of calculating an utterance frequency based on the voice information classified for each user, and determining a user whose utterance frequency is equal to or lower than a threshold as a target user to which the robot will make an utterance; a speech execution step of making the robot execute a predetermined speech action according to a value of a listening coefficient predefined for each action of the user, the speech control method being executed by a computer.

8. 7. A speech control program for causing a computer to function as the speech control device according to claim 1.

Citation Information

Patent Citations

  • Terminal device, method for providing information, and program

    JP2013237124A

  • Utterance control device, electronic apparatus, control method for utterance control device, and control program

    JP2018140477A

  • Utterance timing determination device, robot, utterance timing determination method and program

    JP2019113696A

  • Speech extracting device and speech extracting method

    JP2019176332A

  • Personal assistant control system

    JP2021019966A