Information processing device and program

Abstract

To provide an information processing device and program, which allow for providing conversational sentences tailored to circumstances of each user.SOLUTION: An information processing device provided herein is configured to acquire event identification information for identifying a user event associated with date / time information, and generate associated conversational sentence related to the event identification information in speech generating processing at date and time after the date and time indicated by the date / time information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] In recent years, various tools for managing user schedules have been developed. Among these tools, there is a known technique for registering and managing schedules based on predetermined keywords included in conversations with users (see, for example, Non-Patent Document 1, etc.).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above conventional tools, while interacting with the user in a conversational sentence, they only manage future schedules and cannot be utilized, such as providing the conversational sentences related to the schedules, and the information provided according to the circumstances of each user is not sufficient.

[0005] In view of the above situation, the present invention is made, and one of its objects is to provide an information processing apparatus and a program that enable the provision of conversational sentences according to the circumstances of each user.

Means for Solving the Problems

[0006] One aspect of the present invention, which solves the problems of the above-mentioned conventional example, is an information processing device comprising: acquisition means for acquiring event identification information that identifies a user event associated with date and time information; and conversational text generation means for generating related conversational text related to the event identification information in a process of speaking at a date and time later than the date and time represented by the date and time information. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to provide conversational text tailored to the individual circumstances of each user. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of an information processing system according to an embodiment of the present invention. [Figure 2] This is a block diagram showing an example configuration of a terminal device according to an embodiment of the present invention. [Figure 3] This is a functional block diagram showing an example of a server according to an embodiment of the present invention. [Figure 4] This is an explanatory diagram showing an example of a conversational text queue used in an information processing system according to an embodiment of the present invention. [Figure 5] This is an explanatory diagram showing an example of the contents of an action database used in an information processing system according to an embodiment of the present invention. [Figure 6] This is a functional block diagram showing an example of a terminal device according to an embodiment of the present invention. [Figure 7] This is an explanatory diagram showing an example of configuration information used in an information processing system according to an embodiment of the present invention. [Figure 8] This flowchart illustrates an example of the operation of an information processing system according to an embodiment of the present invention. [Figure 9] This is another flowchart illustrating an example of the operation of an information processing system according to an embodiment of the present invention. [Figure 10] This is a flowchart illustrating an example of the process for selecting conversational text in an information processing system according to an embodiment of the present invention. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described with reference to the drawings. As illustrated in Figure 1, the information processing system 1 according to an embodiment of the present invention is configured to include a server 10 as an information processing device and a terminal device 20 that is connected to the server 10 in a manner that allows communication via a communication means such as a network.

[0010] Here, the server 10, as shown in Figure 1, is composed of a control unit 11, a storage unit 12, and a communication unit 13. The terminal device 20 is a robot, and as illustrated in Figure 2, it includes at least a leg unit 21 and a main body unit 22. The main body unit 22 houses a control unit 31, a storage unit 32, a sensor unit 33, a display unit 34, an audio output unit 35, a communication unit 36, and a drive unit 37. The leg unit 21 and the main body unit 22 are connected via actuators that can rotate around at least one axis, allowing the orientation of the main body unit 22 to be rotated relative to the leg unit 21.

[0011] The control unit 11 of the server 10 is a program control device such as a CPU, and operates according to a program stored in the memory unit 12. In this embodiment, the control unit 11 receives request information from the terminal device 20. The control unit 11 also executes processing based on the received request information. For example, as one of the processes, the control unit 11 sends action information to the terminal device 20, which is the source of the request information, including an action instruction that instructs the terminal device 20 to perform an action, and string information representing the content of the voice spoken by the terminal device 20.

[0012] In another example of this embodiment, the control unit 11 acquires information representing a user's event, such as a "date" (hereinafter referred to as event-specific information), associated with date and time information, and generates related conversational text related to the acquired event-specific information during the process of sending voice to the terminal device 20 at a date and time later than the date and time represented by the date and time information. The details of the processing of this control unit 11 will be explained later.

[0013] The storage unit 12 is a disk device or memory device that holds programs executed by the control unit 11. This storage unit 12 also functions as the work memory for the control unit 11. In one example of this embodiment, the storage unit 12 may store information for generating instructions to the terminal device 20. For example, the storage unit 12 holds a conversation queue that stores candidate conversation sentences to be spoken by the terminal device 20. The contents of this conversation queue will be described later.

[0014] The communication unit 13 is a network interface, etc., and sends various information to the terminal device 20 via the network according to instructions input from the control unit 11. The communication unit 13 also outputs the information received via the network to the control unit 11.

[0015] The control unit 31 of the terminal device 20 is a program control device such as a CPU, and operates according to a program stored in the memory unit 32. In this embodiment, the control unit 31 sends request information to the server 10 at a predetermined timing, along with device identification information specific to the terminal device 20, which will be described later.

[0016] In one example of this embodiment, when the control unit 31 of the terminal device 20 receives voice input from the user via the sensor unit 33 (described later), it converts the input voice into string information. This process can use a widely known voice recognition process, and the control unit 31 may perform this process by, for example, sending the voice information input to a voice recognition server that performs voice recognition processing and receiving the recognized string information.

[0017] Also, when the voice is input by the user, the control unit 31 uses this as a trigger and sends request information to the server 10. This request information includes information for identifying the trigger (for example, information indicating that voice has been input by the user) and information necessary for processing in the server 10, for example, in this case, the character string information which is the recognition result of the voice input by the user.

[0018] That is, when the control unit 31 determines that a predetermined trigger has occurred, it collects the information necessary for processing in the server 10, and sends the request information including the collected information together with the information for identifying the trigger to the server 10. This trigger can be arbitrarily determined, such as, in addition to the case where voice is input by the user as in the previous example, when a predetermined time has come. The detailed operation content of this control unit 31 will also be described later.

[0019] The storage unit 32 is a memory device or the like, and holds the program executed by the control unit 31. This storage unit 32 also operates as a work memory of the control unit 31.

[0020] In the present embodiment, the storage unit 32 may store setting information in which the above trigger and information for identifying the information to be sent to the server 10 are associated. Specific examples of this setting information will be described later. Also, the storage unit 32 stores device identification information uniquely set for the terminal device 20.

[0021] Furthermore, in an example of the present embodiment, this storage unit 32 stores the image data of the animation to be displayed on the display unit 34. Specifically, the storage unit 32 may store the image data of the eyes of the smiling face, the image data of the eyes of the animation with tears flowing... etc. the image data of the animation of the eyes, the image data of the closed mouth in the smiling state, the image data of the closed mouth in the crying state, the image data of the animation of the mouth during vocalization... etc., that is, a plurality of image data of each part representing the expression of the character.

[0022] The sensor unit 33 includes at least a microphone, which is an audio sensor. The sensor unit 33 may also include a touch sensor, an acceleration sensor, etc. The sensor unit 33 outputs audio signals detected by each sensor, information indicating the location touched by the user, acceleration information, etc., to the control unit 31.

[0023] The display unit 34 is a liquid crystal display or the like, and displays image data according to instructions input from the control unit 31. In one example of this embodiment, the display unit 34 displays the character's facial expression using image data of eyes and mouth. The sound output unit 35 is a speaker or the like, and plays sound according to the sound signal input from the control unit 31.

[0024] The communication unit 36 ​​includes a network interface. This communication unit 36 ​​sends and receives information to and from the server 10 via the network, either wirelessly or via a wired connection. Specifically, the communication unit 36 ​​sends request information and the like to the server 10 according to instructions input from the control unit 31. The communication unit 36 ​​also outputs information received from the server 10 to the control unit 31.

[0025] The drive unit 37 drives the actuator to rotate the main body 22 relative to the leg portion 21, in accordance with instructions input from the control unit 31.

[0026] Next, the operation of the control unit 11 of the server 10 in this embodiment will be described. In this embodiment, the control unit 11 of the server 10 is configured to include a receiving unit 41, an event management unit 42, an event information acquisition unit 43, a conversation text generation unit 44, a conversation history management unit 45, an action information generation unit 46, and an instruction transmission unit 47, as illustrated in Figure 3.

[0027] The receiving unit 41 receives request information and device identification information from the terminal device 20. This request information includes information for specifying the cause (trigger) of the request for the action to be executed by the terminal device 20. Although the types of triggers will be described later, for example, voice input by the user, etc. is an example thereof. The above request information based on the trigger that there has been voice input by the user may include, together with the information for specifying the trigger, information representing the content of the voice input by the user. Here, the information representing the content of the voice input by the user may be character string information that is the result of recognizing the voice.

[0028] The receiving unit 41 outputs information for specifying the trigger, character string information representing the content of the voice input by the user, etc. included in the request information received here to the event information acquisition unit 43 and the action information generation unit 46.

[0029] The event management unit 42 stores and manages event information for each user in the storage unit 12. Here, the event information is the user's schedule, which is associated with date and time information and event specifying information for specifying the event.

[0030] The event information acquisition unit 43 accepts the input of character string information representing the content of the voice input by the user from the receiving unit 41. Then, the event information acquisition unit 43 determines whether or not the accepted character string information is information related to an event. Specifically, the event information acquisition unit 43 in the present embodiment compares the character string information with a pattern of information related to a predetermined event.

[0031] Here, the pattern includes a part that matches words (date and time related words) representing predetermined date and time such as "tomorrow", "the day after tomorrow", "next week", etc., and a part that matches words (event related words) related to an event such as "date", "work", "beauty salon", etc. This pattern can be represented by, for example, a character string of a regular expression. For example, it is set as "[tomorrow|the day after tomorrow|weekend|next week][D+ days][W+ days of the week] [day][is|at]W+[has|what]*", etc.

[0032] In the example above, "*" is a regular expression that matches any character, including spaces, "+" is a regular expression that matches a character of the type specified by one or more preceding characters, and [X|Y|…] is a regular expression that matches any of X, Y,…. Also, [Z] means a pattern that may or may not exist, "D" matches a number, and "W" matches a letter. Therefore, the above string matches string information such as "I have a date tomorrow" or "I have an exam next Monday". In the above pattern, the part "[tomorrow|the day after tomorrow|weekend|next week][D+day][W+day][day]" corresponds to date and time related words, and the first "W+" part of "W+[is][there is|is]" (which matches any character) corresponds to event related words.

[0033] Furthermore, this pattern is not limited to one; there may be multiple patterns. For example, in addition to the above pattern, patterns such as "W+[ga][aru|nan]*[tomorrow|the day after tomorrow|weekend|next week][D+day][W+day][Sunday]" may also be set. This example matches an example that does not match the previously exemplified pattern, such as "There's a live performance tomorrow." The event information acquisition unit 43 determines that the accepted string information is event-related information when the string information representing the content of the voice input by the user matches a pattern of event-related information (or any of the multiple patterns) as shown in the above example.

[0034] In other words, if the event information acquisition unit 43 determines that the string information representing the content of the voice input by the user matches a pattern of event-related information, as in the example above, and is therefore event-related information, it extracts date and time information and event identification information that identifies the event from the string information. Specifically, if the string information is "I have a date tomorrow," the event information acquisition unit 43 obtains the date and time information for the day following the date and time the processing is being performed.

[0035] Here, the date and time information is estimated based on the date and time-related words found using the above pattern, representing the date and time when the event occurs. Since widely known techniques can be used for estimating date and time information from date and time-related words, a detailed explanation is omitted here.

[0036] For example, if the process is performed on December 15th, the event information acquisition unit 43 will acquire date and time information of December 16th, which is "tomorrow" from that day. In addition, as event identification information to identify the event, the event information acquisition unit 43 will extract "date" from the above string information. A widely known method can also be used for extracting this event identification information.

[0037] The event information acquisition unit 43 outputs the date and time information and event identification information that identifies the event, which it has extracted, to the conversation text generation unit 44. The event information acquisition unit 43 also interrupts processing if the string information representing the content of the voice input by the user does not match any of the event-related information patterns (if there are multiple patterns).

[0038] The conversation generation unit 44 receives the date and time information and event identification information output by the event information acquisition unit 43 and generates conversation text. Specifically, the conversation generation unit 44 determines a date and time (utterance period) that is later than the date and time output by the event information acquisition unit 43.

[0039] The determination of this utterance period is performed, for example, as follows. In one example of this embodiment, the server 10 stores a day data table in the storage unit 12 as candidate day information, which associates predetermined candidate days such as "1 day," "2 days," and "1 week," the number of days from the first to the last day of the utterance period defined for each candidate day (utterance period length), and a word that identifies the date of the event (called a date and time indicator).

[0040] The conversation generation unit 44 determines the start day of the utterance period by randomly selecting one of the candidate days included in the above-mentioned day data table and adding it to the date and time output by the event information acquisition unit 43. The conversation generation unit 44 also determines the end day of the utterance period by adding the length of the utterance period associated with the selected candidate day to this first day of the utterance period.

[0041] For example, if the event information acquisition unit 43 outputs "December 15th," and selects "2 days" from the list of possible days (with the associated utterance period length set to "0 days"), the conversation generation unit 44 will set the first day of the utterance period to "December 17th" and the last day to "December 17th."

[0042] The conversation generation unit 44 also generates conversation text using the date and time indicator associated with the selected day candidate and the event identification information output by the event information acquisition unit 43. For example, the conversation generation unit 44 randomly selects one of the predetermined conversation pattern candidates, such as "By the way, how was the <event identification information> for <date and time indicator>?" "Oh. <date and time indicator> was <event identification information>, wasn't it?", and inserts the date and time indicator and event identification information into the selected conversation pattern.

[0043] As a result, the conversation generation unit 44 generates conversational sentences such as, "By the way, how was your date the day before yesterday?"

[0044] The conversation generation unit 44 associates the conversation generated here with the determined utterance period as an utterance condition and stores it in the conversation queue stored in the storage unit 12. In other words, the conversation queue stored in the storage unit 12 of the server 10 is, in this example, as illustrated in Figure 4, the utterance condition This stores (C) and the conversation text (D) in association.

[0045] Furthermore, the utterance conditions may include not only the utterance period mentioned above, but also conditions related to the time of utterance and conditions such as the temperature at the time of utterance. Moreover, utterance conditions are not necessarily required and do not need to be defined. If no utterance conditions are defined (no utterance conditions), the conversational text without defined utterance conditions will be considered a conversational text that can be uttered at any time.

[0046] In this embodiment, in addition to the conversations generated by the conversation generation unit 44, conversations may also be pre-registered in the conversation queue. Examples of pre-registered conversations include: - "It was hot today, wasn't it?" associated with the utterance conditions that the maximum temperature was 35 degrees or higher and the time was after 6 PM; and - "Shall we go to bed soon?" associated with the utterance conditions that the time was between 2 AM and 4 AM.

[0047] The conversation history management unit 45 sequentially records, for each user, the content of the voice input by the user received by the receiving unit 41, and the content of the conversation text spoken by the user's terminal device 20 as instructed by the action information generation unit 46, which will be described later. In other words, the conversation history management unit 45 records the history of conversations between the user and the terminal device 20.

[0048] The action information generation unit 46 determines the action that the terminal device 20 that sent the request should perform based on the information input from the receiving unit 41, and generates action information that includes information instructing the action (action instruction) and information necessary for the execution of the action (hereinafter referred to as parameter information), and outputs it to the instruction transmission unit 47.

[0049] In one example of this embodiment, the storage unit 12 of the server 10 stores an action database as information for generating instructions to the terminal device 20, as illustrated in Figure 5, which includes at least one record (R) that associates information for identifying a trigger (T), information for comparing with information representing the content of voice input by the user (V, hereinafter referred to as comparison string information; however, this comparison string information may be omitted depending on the type of trigger), and information (S) representing the process that the server 10 executes to generate action information.

[0050] The action information generation unit 46 obtains comparison string information (V, if any) associated with the trigger identification information (T) input from the receiving unit 41, and information representing the processing that the server 10 should perform in order to generate the action information.

[0051] Then, if comparison string information is obtained (if comparison string information is associated with information that identifies a trigger), the action information generation unit 46 compares the string information output by the receiving unit 41 with the comparison string information. If the action information generation unit 46 determines that the string information output by the receiving unit 41 matches the comparison string information, it executes the process represented by the obtained information and generates action information.

[0052] Furthermore, if comparison string information has not been obtained, the action information generation unit 46 executes the process represented by the obtained information to generate action information.

[0053] As a concrete example, let's assume that the action database contains a record that associates information identifying a trigger indicating "user voice input has been received" with comparison string information that should be compared with the content of the voice input by the user, such as "*news[is]*[tell me|read it aloud]*", and information that instructs the system to "obtain the news string information from a designated web server on the internet and read that string information aloud."

[0054] Furthermore, this comparison string information is also assumed to be represented by a regular expression. Therefore, the above string will match string information such as "Tell me today's news" or "Is there any news?".

[0055] The action information generation unit 46, for example, when it receives input from the receiving unit 41, such as information identifying a trigger that indicates "there has been voice input from the user," and string information representing the content of the voice input by the user, such as "Any news?", searches the action database for a record that contains the information identifying the trigger.

[0056] Here, the action information generation unit 46 finds the above record from the action database and compares the comparison string information contained in the record with the received string information. In the above example, it is determined that the received string information "Are there any news?" matches the comparison string information "*news[is]*[not|tell|read it aloud]*", so the action information generation unit 46 generates information in the above record found by the search that represents the process that the server 10 should execute, for example, "(Step 1) Obtain the string information of the news from a predetermined web server on the internet." (Step 2) Generate instructions to read aloud the string information. (Step 3) Generate instructions to display animation information to be played when the text is read aloud. Obtain this information and execute the process according to it.

[0057] In other words, the action information generation unit 46 obtains news string information from a predetermined web server on the internet according to the information it has read. The action information generation unit 46 also generates animation information display instructions that include information (which may be the file names of each image data) that identifies the animation image data to be displayed at the start of execution of the parallel action processing, the animation image data to be displayed during execution, and the animation image data to be displayed at the end of execution.

[0058] In this example, the action information generation unit 46 generates action information including an action instruction and parameter information and outputs it to the instruction transmission unit 47. Here, the action instruction includes an instruction to read out the string information and an instruction to display the animation information. The parameter information includes the string information obtained above and information that identifies the animation image data.

[0059] Furthermore, the information representing the process that the server 10 should execute may include an instruction to "select a conversation sentence." If such an instruction is included, the action information generation unit 46 will select a conversation sentence in accordance with the instruction, for example, in the following manner.

[0060] The action information generation unit 46 extracts conversational sentences from the conversational sentence queue that satisfy the utterance conditions associated with those conversational sentences. Various information necessary to determine whether or not the utterance conditions are satisfied, such as the current date and time (the date and time the processing is being performed) and weather information, can be obtained via the network from an NTP (Network Time Protocol) server or a designated web server.

[0061] The action information generation unit 46 selects one of the extracted conversation sentences that satisfy the utterance conditions, for example, randomly. The action information generation unit 46 also reads the conversation history related to the user of the terminal device 20 that sent the request, which is recorded by the conversation history management unit 45. The action information generation unit 46 then determines whether the selected conversation sentence would result in a natural conversation if it were spoken following the currently recorded conversation history.

[0062] This judgment can be achieved, for example, by using a neural network trained on the text of conversations between humans to determine whether a given sentence is a valid continuation of the currently recorded conversation history, that is, whether the conversation has continuity. This type of processing is known as Next Sentence Prediction (NSP). As a neural network for performing next sentence prediction, a model known as BERT (https: / / arxiv.org / pdf / 1706.03762.pdf) can be used. For machine learning training data for such next sentence prediction, widely known training data and machine learning processing methods can be employed, such as using a pair of conversational sentences (let's call them the first and second conversational sentences) and information representing the continuity between the first and second conversational sentences, and associating them with each other.

[0063] When a neural network is used in this way to determine whether a selected conversational sentence is appropriate as a continuation of the currently recorded conversation history, that is, to judge the continuity of the conversation, the output of the neural network will be a numerical representation of that appropriateness. The action information generation unit 46 then determines that the conversation is natural (determined to have continuity) when the value exceeds a predetermined threshold.

[0064] When the action information generation unit 46 determines that there is continuity as described above, it generates an action instruction that includes an instruction to read out the string information and an instruction to display the animation information. The action information generation unit 46 also generates parameter information that includes the string information of the selected conversation text and information that identifies the image data of the animation, and instructs the instruction transmission unit 47 to send the generated action instruction and parameter information as action information to the terminal device 20.

[0065] The instruction transmission unit 47 sends the action information generated by the action information generation unit 46 to the terminal device 20, which is the source of the request information received by the receiving unit 41.

[0066] Next, the operation of the control unit 31 of the terminal device 20 will be described. In this embodiment, the control unit 31 is functionally configured to include a request sending unit 51, an action information receiving unit 52, a speech synthesis unit 53, and an action processing execution unit 54, as illustrated in Figure 6.

[0067] When the request sending unit 51 determines that a predetermined trigger has occurred, it collects the information necessary for processing at the server 10 and sends request information containing the collected information, along with information identifying the trigger, to the server 10. Specifically, triggers such as voice input by the user or the arrival of a predetermined time are listed in advance, included in the configuration information, and stored in the storage unit 32.

[0068] As an example, the configuration information is recorded for each trigger, as illustrated in Figure 7, with its name (trigger name: N) associated with the conditions for occurrence (C), information that identifies the information necessary for processing on server 10 in relation to the trigger (P), interval time (T), and other information related to the trigger's processing.

[0069] The request sending unit 51 refers to this configuration information and, if it determines that the occurrence conditions have been met, it considers that a trigger has occurred in which the occurrence conditions have been met and refers to the information (P) necessary for processing at the server 10 in relation to that trigger.

[0070] The request sending unit 51 then collects the information necessary for processing at the server 10, which is identified by the referenced information, and sends request information to the server 10, which includes the collected information and information identifying the trigger that occurred (which may be the trigger name).

[0071] The action information receiving unit 52 receives action information from the server 10 and outputs the received action information to the action processing execution unit 54.

[0072] The speech synthesis unit 53 synthesizes speech data based on the string information input from the action processing execution unit 54, which will be described later. The speech synthesis unit 53 outputs the synthesized speech data to the action processing execution unit 54.

[0073] The action processing execution unit 54 extracts action instructions and parameter information from the action information sent by the server 10 and executes processing according to the action instructions. Specifically, the case in which the action information receiving unit 52 receives action instructions that include an instruction to read aloud the acquired string information and an instruction to display the animation information, as well as parameter information that includes the acquired string information and animation image data, as in the example above, will be described.

[0074] In this example, the action processing execution unit 54 outputs the acquired string information to the speech synthesis unit 53 to obtain audio data. The action processing execution unit 54 also reads the animation image data identified by the information contained in the action information from the storage unit 22. The action processing execution unit 54 then outputs the audio data output by the speech synthesis unit 53 to the audio output unit 35 to play the sound, and also outputs the read animation image data to the display unit 34 to play the animation image data.

[0075] [Operation] The information processing system 1 of this embodiment has the above configuration and operates as shown in the following example. In the following example, it is assumed that the storage unit 12 of the server 10 stores information representing the process that the server 10 executes to generate action information for each trigger that is the cause of the action request, as an action database.

[0076] In the following example, examples of information included in this action database include: • Information identifying the trigger (T): The user is having a conversation • Action to perform: (Step 1) Select the text information of the conversation. (Step 2) Generate instructions to read aloud the string information. (Step 3) The information to be included is 0, which is used to generate an instruction to display animation information to be played when the text is read aloud.

[0077] Furthermore, the storage unit 32 of the terminal device 20 stores setting information, as illustrated in Figure 7, associated with each trigger, such as the conditions for occurrence (C) and information (P) that identifies the information necessary for processing on the server 10 in relation to the trigger.

[0078] In the following example, this configuration information is assumed to include information such as: • Information identifying the trigger (trigger name N): There is voice input from the user • Occurrence condition (C): The user has uttered a predetermined wake word • Information identifying the information necessary for processing on server 10 (P): String information of what the user has uttered... Here, the wake word is a word such as "Hey, listen to me" or "Wake up" that should be recognized as the start of voice input when the user utters it, and is assumed to be predetermined. Terminal device 20 may remove the string portion corresponding to this wake word from the string information of what the user has uttered that is necessary for processing on server 10.

[0079] The operation of the server 10, which holds such configuration information, and the terminal device 20 will be explained below with reference to Figures 8 and 9.

[0080] When a user speaks to the terminal device 20, for example, "Hey, listen. I have a date tomorrow..." (S11 in Figure 8), the terminal device 20 performs a process to recognize the user's voice (S12) and obtains string information corresponding to the voice spoken by the user. As already mentioned, the voice recognition process does not have to be performed by the terminal device 20 itself, but can also be performed by accessing a voice recognition service via the network.

[0081] The terminal device 20 checks whether any trigger conditions have been met by referring to the configuration information (S13). In this case, since the user has uttered the wake word "Hey, listen to me," it is determined that a trigger indicating "voice input from the user" has occurred (S13: Yes), and the terminal device 20 collects string information of what the user said according to the configuration information. If it is determined in step S13 that none of the trigger conditions have been met (S13: No), the terminal device 20 terminates processing.

[0082] Here, since the string information of what the user uttered has already been obtained in step S12, the terminal device 20 sends request information to the server 10 that includes the string information and information identifying the trigger that occurred (trigger name "Voice input from user") (S14).

[0083] Server 10 receives request information from terminal device 20. Server 10 then determines whether the string information contained in the request information is related to an event by comparing it with a predetermined pattern of event-related information (S15).

[0084] If the pattern here is "*[tomorrow|the day after tomorrow|weekend|next week][D+day][W+day][day][is|to]W+[there|is]*", then the received string information above is "Hey, listen. I have a date tomorrow...", so server 10 determines that it matches the pattern. In other words, this string information is determined to be information related to an event (S15:Yes).

[0085] 1 If it is determined that the information is not related to the event (S15: No), server 10 will perform other processing, namely processing obtained by referring to the action database.

[0086] Server 10 then extracts date and time information and event identification information from this string information. (S16). Here it says "tomorrow," so server 10 is executing this process on the date and time. The server obtains date and time information such as "December 16th," which is the day after (for example, December 15th). Additionally, server 10 extracts "date" from the above string information as event identification information.

[0087] Next, server 10 generates a conversation sentence. Server 10 first randomly selects one day from the number of day candidates predetermined in the number of days data table as the utterance period of the conversation sentence (S17: Determination of utterance period). Here, let's assume that "7 days" was selected as the number of day candidate. Also, in the processing of this step S16, server 10 associates the selected number of day candidate with the length of the utterance period recorded in the number of days data table (here, "7 days") and the date and time indicator. The time period (referred to here as "during this period") is obtained, and using the date and time information (December 16th) extracted in step S16, the obtained candidate number of days, and the length of the utterance period, the utterance period is determined to be "from December 23rd to December 30th".

[0088] In this example of determining the utterance period, one number of days is randomly selected from a list of candidate days predetermined in a days data table. However, this embodiment is not limited to this, and the server 10 may also obtain a predetermined number of days as the utterance period, associated with the event identification information obtained in step S16.

[0089] In this example, for instance, if the event identification information is "date" (an event that is likely to occur relatively frequently and is considered to have a relatively short topic lifetime), a data table is predetermined and stored in the storage unit 12. For example, if the event identification information is "date" (an event that is likely to occur relatively frequently and is considered to have a relatively short topic lifetime), a relatively short number of days is set to "4 days." If the event is "travel" (an event that is considered to have a relatively long topic lifetime), a relatively long number of days is set to "14 days." The server 10 then refers to this data table to set the utterance period.

[0090] In another example, the utterance period may be determined by referring to a data table that stores multiple candidate days associated with each of the event-specific pieces of information obtained in step S16.

[0091] In this example, if the event identification information is "date" (an event that can occur relatively frequently and is considered to have a relatively short lifespan), a data table is predetermined and stored in the storage unit 12, with possible dates being "1 day, 2 days, 3 days, 4 days" for relatively short durations, and "travel" (an event that is considered to have a relatively long lifespan) for relatively long durations, such as "1 day, 2 days, ..., 13 days, 14 days".

[0092] The server 10 then randomly selects one number of days from a predetermined list of candidate days in the data table, in association with the event identification information obtained in step S16, and sets the selected number of days as the utterance period.

[0093] In this example, if the event identified by the event identification information obtained in step S16 is "date," a speech duration of 1 to 4 days will be set. If the event identified by the event identification information obtained in step S16 is "travel," a speech duration of 1 to 14 days will be set.

[0094] Server 10 generates a conversation sentence using the date and time indicator associated with the selected day number candidate, the event identification information output by the event information acquisition unit 43, and the information of the predetermined conversation sentence pattern candidates (S18). Here, Server 10 selects one of several predetermined conversation sentence pattern candidates and inserts the date and time indicator ("the other day") and the event identification information ("date") to generate a conversation sentence, for example, "By the way, how was your date the other day?"

[0095] The server 10 associates the speech duration determined in step S17 with the speech text generated in step S18 as a speech condition and stores it in the speech text queue stored in the memory unit 12 (S19).

[0096] This process adds pre-configured conversational phrases to the conversation queue. (1) Speech conditions: The maximum temperature was 35 degrees Celsius or higher, and the time was after 6 PM. Conversation: "It was hot today, wasn't it?" (2) Speech condition: The time is between 2:00 AM and 4:00 AM. Along with conversational phrases like "Shall we go to bed now?", (n) Speech condition: The speech must take place between December 23rd and December 30th. The conversation will accumulate with the phrase, "By the way, how was our date the other day?"

[0097] During this operation, the server 10 sequentially records the content of the voice input by the user and the content of the conversation spoken by the user's terminal device 20, thereby generating conversation history information. For example, if the following conversation takes place between the user and the server, the conversation history information will be retained.

[0098] In other words, server 10 will record conversation history information such as: User: "Hey, listen. I have a date tomorrow." Terminal device 20's utterance: "Oh, really?" User: "What's the weather going to be like?" Terminal device 20's utterance: "It looks like it's going to be sunny tomorrow."

[0099] In addition, the server 10 may perform other processes, namely processes obtained by referring to the action database, in parallel with the processing in steps S16 to S19.

[0100] Subsequently, for example, on December 24, if the user speaks to the terminal device 20 saying, "Wake up. Any news?" (S21 in Figure 9), the terminal device 20 performs a process to recognize the user's voice (S22) and obtains string information corresponding to the voice spoken by the user.

[0101] The terminal device 20 also checks whether any trigger conditions have been met by referring to the configuration information (S23). In this case, since the user has spoken the wake word, it is assumed that the trigger indicating "user voice input has been received" has occurred (S23: Yes), and according to the configuration information, it collects the string information of what the user said, "Wake up. Any news?". Then the terminal device 20 sends a request to the server 10 containing this string information and information identifying the trigger that occurred (trigger name "user voice input has been received") (S24).

[0102] Server 10 compares the string information "*news[is]*[not|tell|readit]*" associated with a trigger indicating that "there has been voice input from the user" with the string information "Wake up. Is there any news?" sent by terminal device 20 to determine if they match (S25).

[0103] Here, the string information "Wake up. Any news?" received by server 10 from terminal device 20 matches the comparison string information. Therefore, in step S25, server 10 determines that the comparison string information and the received string information match and starts a trigger indicating "Voice input from the user" and the process identified by the information associated with the comparison string information.

[0104] For example, server 10 obtains news string information from a designated web server on the internet and sends it to terminal device 20 as action information along with an instruction to read the string information aloud (S26).

[0105] Terminal device 20 follows this instruction and performs the process of reading out news information (S27). Here, news such as "The warm weather continues from last week, but a cold wave is expected to hit at the end of the year" is read aloud.

[0106] Server 10 records the string information sent and received with terminal device 20 up to this point as a conversation history. In the example above, the conversation history is recorded as follows: User: "Wake up. Any news?" Terminal device 20's utterance: "'The warm weather continues from last week, but a cold wave is expected to hit at the end of the year,' it says."

[0107] In the processing up to this point, server 10 determines whether the string information contained in the request information is event-related information by comparing it with a predetermined pattern of event-related information. However, the string information "Wake up. Any news?" does not match the pattern "*[Tomorrow|The day after tomorrow|Weekend|Next week][D+Sun][W+Day of the week][Sun][is|to]W+[there][is|something]*", so it is not considered event-related information, and the corresponding processing is not performed.

[0108] If the user then speaks to the terminal device 20, for example, "Ah, it was certainly warm last week," (S28), the terminal device 20 acquires string information corresponding to the voice spoken by the user. The terminal device 20 then, assuming that a trigger has occurred, for example, that "the user is having a conversation," sends a request to the server 10, according to the configuration information, which includes the string information of what the user said, "Ah, it was certainly warm last week," and information identifying the trigger that occurred (S29).

[0109] Server 10 determines whether the string information contained in this request information is related to an event by comparing it with a predetermined pattern of event-related information. However, since this string information does not match the above pattern, it is not considered to be related to an event, and no corresponding processing is performed.

[0110] Meanwhile, server 10 executes a process identified by information associated with the information that identifies the trigger "the user is having a conversation," selects the string information of the conversation text, and sends an instruction to terminal device 20 to speak the string information (S30).

[0111] In other words, as illustrated in Figure 10, in the process of step S30, the server 10 extracts a conversation sentence from the conversation sentence queue that satisfies the four utterance conditions associated with that conversation sentence (S41). The server 10 also randomly selects one of the unselected conversation sentences from among the conversation sentences (S42).

[0112] On the other hand, the server 10 refers to the conversation history of the user of the terminal device 20 that sent the request, and determines whether the conversation selected in step S42 will be a natural conversation when spoken following the currently recorded conversation history (S43).

[0113] Specifically, Server 10 uses a BERT model, which has been trained on the text of conversations between humans, to predict the next sentence. This process obtains a numerical value representing the validity of the selected conversation sentence as a continuation of the currently recorded conversation history (the larger the value, the more valid the sentence). Server 10 then determines that the conversation is natural (continuity exists) if the obtained value exceeds a predetermined threshold, and that it is not natural (no continuity exists) otherwise.

[0114] In step S43, if the server 10 determines that the selected conversational sentence will result in a natural conversation when spoken (S43: Yes), it generates action information that includes an instruction to read aloud the string information of the selected conversational sentence. The server 10 then sends this action information to the terminal device 20 (S44).

[0115] If, in step S43, the server determines that the selected conversational text would not result in a natural conversation when spoken (S43: No), the server 10 returns to process S42 and continues processing. If there are no unselected conversational texts in process S42, the server 10 generates action information including an instruction to read aloud the predefined conversational text string information and sends it to the terminal device 20, or performs other predetermined processing (S45: Execution of default processing).

[0116] Returning to Figure 9, the terminal device 20 executes the process of reading out the string information in accordance with the instructions in the action information (S31).

[0117] As a specific example, in the process of step S43 above, following a conversation history such as User: "Wake up. Any news?" Terminal device 20's utterance: "'It's been warm again this week, but a cold wave is expected at the end of the year,' they say," User: "Yeah. It was certainly warm last week," it is determined whether the following conversation sentence, "By the way, how was your date the other day?" naturally follows as one of the conversation sentences that satisfies the utterance condition.

[0118] If the server determines that there is continuity in the conversation, it instructs the terminal device 20 to speak the conversational sentences, so the conversation as a whole will look like this: User: "Wake up. Any news?" Terminal device 20's utterance: "'The warm weather continues from last week, but a cold wave is expected to hit at the end of the year,' it says." User: "Yeah. It was certainly warm last week." Terminal device 20's utterance: "By the way, how was your date the other day?"

[0119] In this embodiment, the appearance of conversations about past plans enables conversations tailored to the user, thereby fostering a sense of familiarity.

[0120] [In the case of a standalone terminal device] In the explanation so far, the terminal device 20 recognized the voice spoken by the user and sent the resulting string information to the server 10. The server 10 then generated and provided the string information (conversation text) that would serve as the basis for the voice data to be spoken by the terminal device 20.

[0121] However, in one embodiment of the present invention, the terminal device 20 may function as an information processing device. In this case, the server 10 is not necessarily required.

[0122] In this example, the control unit 31 of the terminal device 20 functions as a request sending unit 51, an action information receiving unit 52, a speech synthesis unit 53, and an action processing execution unit 54, while also operating as a receiving unit 41, an event management unit 42, an event information acquisition unit 43, a conversation text generation unit 44, a conversation history management unit 45, and an action information generation unit 46.

[0123] In this case, the data stored in the memory unit 12 as described above, such as the action database and conversation queue, are all stored in the memory unit 32 of the terminal device 20.

[0124] In this example, the request sending unit 51 outputs the request information to the receiving unit 41, which is implemented by the control unit 31 itself. The receiving unit 41 then receives and processes the outputted request information. Furthermore, the action information generation unit 46 outputs the generated action information to the action information receiving unit 52, which is also implemented by the control unit 31 itself.

[0125] The output and reception of this information may be performed via a local loopback through the network (communication unit 36), or it may be performed by storing and reading it from the storage unit 32 without going through the network.

[0126] [Stopping speech] Furthermore, the terminal device 20 may interrupt the playback of audio data (including audio data synthesized based on string information received from the server 10) when the user performs a predetermined operation on the terminal device 20 during playback of the audio data.

[0127] This predetermined operation may be performed, for example, by touching the touch sensor if the sensor unit 33 is equipped with a touch sensor. In this example, when the terminal device 20 detects that the user has touched the touch sensor while audio data is being played back, it interrupts the playback of the audio data.

[0128] In another example, this predetermined operation may be triggered by the user uttering a predetermined word. For instance, if the terminal device 20 determines that the user has uttered the word "shhh," a word requesting silence, while audio data is being played, the terminal device 20 will interrupt the playback of the audio data.

[0129] In yet another example, the information processing system 1 may be controlled so as not to utter predetermined conversational phrases even when the user is not performing any operations.

[0130] Specifically, the sensor unit 33 of the terminal device 20 may be equipped with a human presence sensor capable of detecting the number of people (a widely known device, so a detailed explanation will be omitted). The information on the number of people detected by this human presence sensor around the terminal device 20 may then be used for selecting conversational text.

[0131] In this example, when the server 10 or terminal device 20, which performs the function of a conversation text generation unit 44, generates a conversation text based on the information output by the event information acquisition unit 43 and registers it in the conversation text queue, it includes a condition related to the number of people detected by the terminal device 20, in addition to the information on the utterance period, in the utterance conditions. As an example, this condition related to the number of people is set to the condition that the number of people is "1" (singular).

[0132] When terminal device 20 selects a conversation text from server 10, it sends request information to server 10 that includes information indicating the number of people detected in the surrounding area.

[0133] The server 10 or terminal device 20, which performs the function of an action information generation unit 46, selects a conversation sentence that satisfies the utterance conditions associated with that conversation sentence from among the conversation sentences stored in the conversation sentence queue, and also selects a conversation sentence that satisfies the conditions using this person information.

[0134] As in the example above, if the condition that the number of people is "1" is included as a condition for uttering conversational text related to past event information, then if there are multiple people around the terminal device 20, conversational text related to past event information will not be selected, thereby reducing the chance of the user's private information being leaked.

[0135] [Expression of emotions] In one example of this embodiment, the action information may further include information related to the emotions of the character displayed on the display unit 34 of the terminal device 20. This emotion information may include, for example, joy. This could include information that identifies emotions such as (Joy) or (Sad). This emotional information may also include information that represents a lack of emotion (flat).

[0136] The terminal device 20 refers to the emotion information contained in the received action information and selects the eye image data to display on the display unit 34. For example, if the emotion information is "Sad," the terminal device 20 selects and displays the image data of an animated eye shedding tears from the eye image data.

[0137] Furthermore, in one example of this embodiment, when determining the emotional information to be included in the action information, the server 10 or terminal device 20 that generates the string information to be spoken may determine the emotion based on the words contained in the string information. Such processing can be carried out, for example, as a process of determining the polarity of the sentence represented by the string information based on the words contained in the string information. Specifically, when speaking string information of news, if it contains words that represent a tragic event, the string information of the news is included in the action information along with information that identifies the emotion of sadness.

[0138] [Delete information] Furthermore, in this embodiment, the server 10 or terminal device 20 that stores the conversation queue may delete conversations associated with utterance periods that have already elapsed (utterance periods that have ended) from the conversation queue at predetermined intervals.

[0139] Furthermore, among the information stored in the conversation queue, conversations stored by the conversation generation unit 44 may be deleted from the conversation queue when the conversation is spoken. This prevents the same conversation from being played back multiple times.

[0140] [Retrieved from schedule information] Furthermore, in the description of this embodiment so far, the server 10 or terminal device 20, which functions as an event information acquisition unit 43, extracts date and time information and event identification information from string information that represents the content of voice input by the user when the string information matches a pattern of information related to the event. However, this embodiment is not limited to this.

[0141] In one example of this embodiment, the event information acquisition unit 43 may acquire predetermined schedule information and extract date and time information and event 7 specific information in place of, or in conjunction with, the above processing. Specifically, the server 10 or terminal device 20, which functions as the event information acquisition unit 43, receives in advance permission settings from the user for access to the web service where the user registers schedule information. The terminal device 20 then acquires schedule information from the web service according to these settings.

[0142] The event information acquisition unit 43, when the schedule information acquired here includes date and time information and event identification information that identifies an event the user is involved in at that date and time (for example, "date" or "work"), extracts this date and time information and event identification information and outputs it to the conversation text generation unit 44.

[0143] This example shows that even if a conversation doesn't explicitly involve the user, it's possible to provide information based on the user's past schedule.

[0144] [When allowing for a longer speaking period] Furthermore, in the explanation so far, the dialogue generation unit 44 was assumed to generate the entire dialogue in advance and store it in the dialogue queue. Therefore, depending on the period of speaking, the dialogue generation unit 44 pre-included relative demonstrative pronouns (such as "yesterday" and "the day before yesterday") that refer to the past event that is the subject of the conversation up to the day it occurred.

[0145] However, this embodiment is not limited to this, and the dialogue generation unit 44 may generate a dialogue template that represents only the position of the word indicating the date on which a past event occurred, and store it in the dialogue queue. In this case, when the action information generation unit 46 generates an instruction to speak the selected dialogue after the dialogue has been selected, it may generate the word indicating the date on which the target past event occurred from that point onward.

[0146] In this example, the event information acquisition unit 43 causes the event management unit 42 to record the acquired event information and issue an event identifier unique to the recorded event.

[0147] The conversation generation unit 44 then generates a conversation template that represents only the position of the word indicating the date on which the past event occurred, when generating a conversation related to the event information (including date and time information and event identification information) acquired by the event information acquisition unit 43, and stores it in the conversation queue in association with the corresponding event identifier. Here, the conversation template will include a code that should be replaced with a word that specifies the date and time, such as "How was your date on <date and time>?".

[0148] Subsequently, when the action information generation unit 46 selects a conversation sentence associated with this event identifier as the target of an utterance, the action information generation unit 46 refers to the event identifier associated with the selected conversation sentence and obtains event information (including at least its date and time information) corresponding to that event identifier from the event management unit 42.

[0149] Then, based on the current date and time, a word that identifies the relevant date and time is generated, and the corresponding code in the selected conversation text is replaced with the appropriate code. This word generation can be done based on the difference (interval in days) between the current date and time and the date and time when the event in question occurred, by pre-determining based on intervals of days, such as "yesterday" for 1 day, "the day before yesterday" for 2 days, "recently" for 3 to 6 days, "last week" for 7 to 13 days, etc.

[0150] The action information generation unit 46 then generates action information that includes an instruction to utter a conversational sentence that includes a word specifying the date and time.

[0151] [Features of the Embodiment] This embodiment is also characterized by the following: One aspect of this embodiment is an information processing device comprising: acquisition means for acquiring event identification information, such as "date," which identifies a user event associated with date and time information; and conversational text generation means for generating related conversational text, such as "How was last week's date?", which is conversational text related to the event identification information, in a process that speaks at a date and time later than the date and time represented by the date and time information.

[0152] This information processing device may be implemented as a terminal device provided to each user, or it may be implemented as a server that is communicatively connected to the terminal device and implements each of the above-mentioned means.

[0153] In this example, the information processing device provides conversational text about past events, enabling conversations about past events. This allows for more private conversations for the user, fostering a sense of familiarity.

[0154] Furthermore, when the conversational text input by the user includes event-related words such as "date" related to a predetermined event, and date-related words that specify a date and time such as "next weekend," the system includes storage means for associating and storing the event-specific information related to the event-related words with the date and time information represented by the date and time-related words.

[0155] In this example, information about an event can be accumulated simply by the user entering it as conversational text, without the user having to register it as schedule information or anything like that.

[0156] Furthermore, the conversation generation means may include an execution means that records the related conversation sentences, such as "How was your date last week?", in association with information representing the period during which the related conversation sentences are spoken, and determines whether the conversation sentences input by the user and the related conversation sentences are continuous if the current date and time are included in the period, and executes the speech processing of the related conversation sentences when it is determined that they are continuous.

[0157] In this example, conversations about past plans appear at times when it is judged that they should naturally follow, making the conversation sound natural.

[0158] Here, the execution means may use a neural network trained with machine learning using training data that associates a first conversation sentence, a second conversation sentence, and information representing the continuity between the first and second conversation sentences, to determine whether the received conversation sentence and the associated conversation sentence have continuity.

[0159] Furthermore, while the aforementioned related conversational sentence is being spoken, the system may be configured to stop the speech if the user performs a predetermined input, such as an action to forcibly stop the speech. This allows the speech to be stopped if it is inappropriate.

[0160] Furthermore, the system may further include a playback device that speaks the generated related conversation sentences, and means for detecting a person located near the playback device. The playback device may only speak the related conversation sentences when there is a single person located near the playback device. This prevents speech from being spoken in inappropriate situations. [Explanation of symbols]

[0161] 1 Information processing system, 10 Server, 11 Control unit, 12 Storage unit, 13 Communication unit, 20 Terminal device, 21 Leg unit, 22 Main unit, 31 Control unit, 32 Storage unit, 33 Sensor unit, 34 Display unit, 35 Audio output unit, 36 Communication unit, 37 Drive unit, 41 Receiving unit, 42 Event management unit, 43 Event information acquisition unit, 44 Conversation text generation unit, 45 Conversation history management unit, 46 Action information generation unit, 47 Instruction transmission unit, 51 Request transmission unit, 52 Action information reception unit, 53 Speech synthesis unit, 54 Action processing execution unit.

Claims

1. Equipped with a processor, The aforementioned processor, Based on event identification information and date / time information obtained by recognizing the user's voice, a related conversation sentence related to the event identification information is generated, which is spoken at a date and time after the event. Based on the aforementioned event identification information, the speech duration is determined. After the aforementioned event and within the aforementioned speaking period, the following is spoken to the user: the conversation sentences prepared in advance before the generation of the related conversation sentences and the related conversation sentences. When uttering the aforementioned conversation sentence and the aforementioned related conversation sentence, the utterance continues if the numerical value representing the validity of the sentence as a continuation of the conversation history exceeds a predetermined threshold. Information processing device.

2. The processor is instructed to generate related conversational sentences related to the event identification information, which are spoken at a date and time after the event, based on event identification information and date and time information obtained by recognizing the user's speech. The processor is instructed to determine the speech duration based on the event identification information. The processor causes the user to utter a pre-prepared conversation sentence and the related conversation sentence, after the event and within the utterance period, prior to the generation of the related conversation sentence. When uttering the aforementioned conversation sentence and the aforementioned related conversation sentence, the utterance continues if the numerical value representing the validity of the sentence as a continuation of the conversation history exceeds a predetermined threshold. program.

3. The processor recognizes the user's speech and, from the event identification information and date / time information obtained, generates related conversational sentences related to the event identification information, which are spoken at a date and time after the event. The processor determines the speech duration based on the event identification information, The processor, after the event and within the utterance period, speaks to the user a pre-prepared conversation sentence and the related conversation sentence that were generated before the related conversation sentence was generated. When uttering the aforementioned conversation sentence and the aforementioned related conversation sentence, the utterance continues if the numerical value representing the validity of the sentence as a continuation of the conversation history exceeds a predetermined threshold. method.

4. Equipped with a server and terminals, The aforementioned server, Based on event identification information and date / time information obtained by recognizing the user's voice, a related conversation sentence related to the event identification information is generated, which is spoken at a date and time after the event. Based on the aforementioned event identification information, the speech duration is determined. After the aforementioned event and within the aforementioned speaking period, the terminal is instructed to utter a conversation sentence prepared in advance before the generation of the related conversation sentence and the related conversation sentence. When uttering the aforementioned conversation sentence and the aforementioned related conversation sentence, the utterance continues if the numerical value representing the validity of the sentence as a continuation of the conversation history exceeds a predetermined threshold. system.

Images