Processing apparatus, processing method, and program

Abstract

To provide a processing device capable of appropriate audio processing. [Solution] A processing device comprising: a voice input unit that receives voice input from the user; a recognition unit that recognizes an input sentence from the voice input; a prompt generation unit that generates a prompt from the input sentence; an output data acquisition unit that inputs the prompt to a language model and acquires output data; and an output unit that generates and outputs a response sentence from the output data. The prompt generation unit generates a prompt that returns a response sentence in response to a candidate sentence when there is one candidate sentence generated from the input sentence, and generates a prompt that returns a response sentence that allows the user to select the correct candidate sentence from among the multiple candidate sentences when there are multiple candidate sentences generated from the input sentence.

Description

Technical Field

[0001] This disclosure relates to a processing device and the like.

Background Art

[0002] For example, as disclosed in Patent Document 1, an Internet television system capable of command input by voice input is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by this disclosure is to provide a processing device and the like capable of appropriate voice processing.

Means for Solving the Problems

[0005] This disclosure provides a processing device including a voice input unit that receives a user's voice input, a recognition unit that recognizes an input sentence from the voice input, a prompt generation unit that generates a prompt from the input sentence, an output data acquisition unit that inputs the prompt into a language model and acquires output data, and an output unit that generates and outputs a response sentence from the output data. When there is one candidate sentence generated from the input sentence, the prompt generation unit generates a prompt that returns a response sentence that responds to the candidate sentence. When there are multiple candidate sentences generated from the input sentence, the prompt generation unit generates a prompt that returns a response sentence that causes the correct candidate sentence to be selected from the multiple candidate sentences.

[0006] Furthermore, this disclosure provides a control method for a processing device that receives voice input from a user, recognizes an input sentence from the voice input, generates a prompt from the input sentence, inputs the prompt to a language model, obtains output data, generates a response sentence from the output data, and outputs it, wherein when there is one candidate sentence generated from the input sentence, a prompt is generated that returns a response sentence in response to the candidate sentence, and when there are multiple candidate sentences generated from the input sentence, a prompt is generated that returns a response sentence that causes the user to select the correct candidate sentence from among the multiple candidate sentences.

[0007] Furthermore, this disclosure provides a program for a computer that implements the following functions: a function to receive voice input from a user; a function to recognize an input sentence from the voice input; a function to generate a prompt from the input sentence; a function to input the prompt to a language model and obtain output data; a function to generate and output a response sentence from the output data; a function to generate a prompt that returns a response sentence in response to a candidate sentence when there is one candidate sentence generated from the input sentence; and a function that returns a prompt that returns a response sentence to select the correct candidate sentence from among the multiple candidate sentences when there are multiple candidate sentences generated from the input sentence. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide a processing device, etc., capable of appropriate speech processing. [Brief explanation of the drawing]

[0009] [Figure 1] This is a conceptual schematic diagram of the system according to the first embodiment of this disclosure. [Figure 2] This is a block diagram illustrating the hardware configuration of a display device according to the first embodiment of this disclosure. [Figure 3] This is a block diagram illustrating the software configuration of a display device according to the first embodiment of this disclosure. [Figure 4] This is a block diagram illustrating the configuration of the storage unit of a display device according to the first embodiment of this disclosure. [Figure 5] This is a flowchart illustrating the basic operation of the display device according to the first embodiment of this disclosure. [Figure 6] Figure 6(a) is a diagram illustrating an example of the basic operation of the display device according to the first embodiment of this disclosure, where Figure 6(a) is an example of a display screen, Figure 6(b) is an example of a prompt, and Figure 6(c) is an example of output data. [Figure 7] This is a conceptual schematic diagram illustrating the comparison of the difficulty of speech recognition between conversations between humans and machines, and between conversations between humans. [Figure 8] This is a conceptual schematic diagram illustrating conversations between humans and avatars (AI). [Figure 9] These are conceptual schematic diagrams to explain speech recognition models. Figure 9(a) shows a conventional model, and Figure 9(b) shows an end-to-end speech recognition model. [Figure 10] This is a flowchart illustrating the operation of the display device according to the first embodiment of this disclosure when it outputs a response sentence in response to an utterance by user U. [Figure 11] This is a flowchart illustrating the operation of the display device according to the third embodiment of this disclosure when it outputs a response sentence in response to an utterance by user U. [Figure 12] This is a block diagram illustrating the configuration of the storage unit of a display device according to the fourth embodiment of this disclosure. [Figure 13] This is a flowchart illustrating the operation of the display device according to the fourth embodiment of this disclosure when it outputs a response sentence in response to an utterance by user U. [Figure 14] This is a flowchart illustrating the operation of a display device according to the fifth embodiment of this disclosure in response to a wake word. [Modes for carrying out the invention]

[0010] Generally, technologies for displaying an avatar on a display screen such as a TV screen are known. For example, a voice conversation is conducted between a user and a system using a voice assistant, and a character avatar is displayed on the TV screen in联动 with the conversation, making it appear as if a person inside the TV screen is conversing with the user.

[0011] However, in this type of technology, errors may occur in input voice processing when the user's utterance is unclear. On the other hand, even when the user's utterance is clear, errors may occur in output voice processing from the response text. In such cases, the system cannot provide a proper response to the user's voice input.

[0012] Regarding the processing device and the like of the present disclosure that solve one or more of the above-described problems, the following embodiments will be described while referring to the drawings. Note that the following embodiments describe the invention recited in the claims as an example, and the technical scope of the present invention is not limited to the description of the following embodiments. Further, in the following embodiments, the case where the processing device of the present disclosure is applied to a display device will be described, but the processing device is not limited to the display device. For example, the processing device may be a single device of a set-top box (STB) having a tuner function, or may be applied to a recording device using a hard disk or the like, a display control device for playing a recording medium, a projector, or the like. Further, the processing device may be an information processing device such as a smartphone, a tablet, or a computer that a user uses as a device capable of displaying content, or may be an in-vehicle device such as a car navigation system.

[0013] [1. First Embodiment] [1.1 About the System] [1.1.1 System Overview] FIG. 1 is a diagram for explaining the overview of system 1. System 1 has a display device 10 and is connected to a network NW. Also, the display device 10 may be able to recognize a user U who is around the device. The display device 10 incorporates a camera 12 and a microphone 14 and may recognize the user U in front of the display device 10.

[0014] Also, system 1 may include a server device 20. The server device 20 is connected to the network NW, for example. Also, the server device 20 may store a large language model 22 as a language model.

[0015] The large language model 22 is one of the language models and is generally called LLM (Large Language Models). The large language model 22 is stored in the server device 20, but may also be stored in the display device 10. The display device 10 may store a small language model (SLM) as a language model different from the large language model 22, for example, an edge LLM. That is, as a language model, a so-called large language model (LLM) with a large amount of learning data may be stored in the server device 20, and a language model (SLM) with less learning data than the large language model 22 may be stored in the display device 10 or other devices. Also, in the following specification, the language model stored in the display device 10 may be referred to as an edge language model.

[0016] The large language model 22 may be prepared by the service provider constructing system 1 or may use an external service. For example, large language models such as GPT (Generative Pre-trained Transformers, GPT-3, GPT-4, GPT-4o), PaLM (Pathways Language Model), LLaMA (Large Language Model Meta AI), and tsuzumi may be used.

[0017] Furthermore, the display device 10 can acquire and display content. The display device 10 may acquire content from the broadcasting station 30, from the distribution device 40, or from a recording medium. The broadcasting station 30 may also transmit content to the display device 10 via a network NW. For example, the broadcasting station 30 may transmit content to the display device 10 using terrestrial digital broadcasting, BS broadcasting, or CS broadcasting. The broadcasting station 30 may also transmit content using IP multicast broadcasting via a network NW.

[0018] In this embodiment, an "avatar" is used to output the response sentence, but for example, the response sentence may be displayed on the display screen of the display device 10 using a language model without using an avatar, or the display device 10 may read the response sentence aloud. Furthermore, the display device 10 may be a display system that includes a device that simply displays an external video signal and a device that outputs a video signal. For example, the display device 10 may include a system that includes a display and a recording and / or playback device connected via HDMI®.

[0019] [1.1.2 Terminology] Hereafter, terms used in this specification will be used in a manner that can be understood by those skilled in the art, for example as follows:

[0020] An "avatar" is an object represented based on a character. Users can primarily select or create them. An avatar only needs to correspond to a given character; for example, it could represent a virtual friend, secretary, or advisor—someone to converse with. Although called an avatar, it doesn't necessarily have to be a representation of the user themselves.

[0021] "Conversation" refers to the exchange of sentences, and in this embodiment in particular, it refers to the exchange and combination of input sentences and response sentences. Here, the input sentence is a sentence based on the voice spoken by the user. The response sentence is a sentence output or spoken by the avatar. These sentences are sometimes also called messages. When the avatar speaks a sentence (response sentence), it means, for example, that the control unit displays the sentence (message) or outputs the sentence (message) as sound. In this case, by linking with the display of the avatar, the user can interact with the avatar as if it were actually speaking.

[0022] A "prompt" is information used to instruct a language model (large-scale language model, small-scale language model) to generate output data that includes a response sentence.

[0023] "User state" refers to the user's state, which may, for example, be the user's state when they are in front of the display device 10, but preferably it may be the user's state when they are viewing content. For example, the user state may include the user's facial expression (laughing, surprised, crying, etc.), gaze direction, and face orientation that can be recognized from the image. The user state may also include the user's emotions (wow, amazing, I want it, etc.) that can be recognized from their voice. Furthermore, the user state may include the user's age, gender, and the number of users.

[0024] "Content" includes program content broadcast from broadcasting station 30 and distributed content distributed from distribution device 40. Content may also be content recorded on a recording medium (e.g., Blu-ray). Content includes video containing one or more images and audio, and may also include explanatory information, attributes, and subtitle information related to the content. While this explanation primarily uses video content as an example, it may also include still images, audio, text, and web pages as needed.

[0025] [1.2 Hardware Configuration] Next, the hardware configuration of the device in this embodiment will be described. The display device 10, the server device 20, and the distribution device 40 each include at least a control unit, a storage unit, and a communication unit. In addition, each device has general functions as needed. Figure 2 illustrates the hardware configuration of the display device 10.

[0026] The control unit 100 controls the entire display device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the memory unit 110 (e.g., ROM 110A, storage 110C). The control unit 100 may be implemented by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). Alternatively, the control unit 100 may be composed of a control circuit.

[0027] The memory unit 110 is one or more storage devices that store necessary data or programs. The memory unit 110 temporarily or permanently stores data and programs. For example, the memory unit 110 includes a ROM 110A, a RAM 110B, and a storage device 110C.

[0028] ROM110A is a non-volatile memory that can retain programs and data even when the power is turned off.

[0029] RAM110B is the main memory primarily used by the control unit 100 during processing. RAM110B is a rewritable memory that temporarily holds data including programs read from ROM110A and storage 110C, as well as execution results.

[0030] Storage 110C is a non-volatile storage device capable of storing programs and data. For example, it may consist of storage devices such as HDDs (Hard Disk Drives) or SSDs (Solid State Drives). Alternatively, Storage 110C may be configured as an externally connectable USB memory stick. Furthermore, Storage 110C may be, for example, a storage area located in the cloud.

[0031] The broadcast control unit 120 receives broadcast waves transmitted by the broadcasting station selected by the user, decodes video data from the broadcast waves and outputs it to the display unit 140, and decodes audio data from the broadcast waves and outputs it to the audio output unit 160. The broadcast control unit 120 may have components such as a digital tuner unit (terrestrial / BS / CS, etc.), an OFDM demodulation unit, a DEMUX unit, and an MPEG2 decoding unit. Furthermore, there may be multiple broadcast control units 120.

[0032] The operation unit 130 receives operations from the user, issues operation instructions to each function unit, and notifies the control unit 100 of operation signals corresponding to the received operations. For example, the operation unit 130 receives operation signals from a remote control or the like and operates the display device 10 according to the received operation signals. Alternatively, the operation unit 130 may perform control based on operations received using, for example, operation buttons on the display device 10 or software keys using a touch panel.

[0033] The display unit 140 is a display device capable of displaying the video of the received program and various information. The display unit 140 may be, for example, a liquid crystal display (LCD) or an organic electroluminescent (OLED) display, or other device capable of displaying video. Alternatively, the display unit 140 may be a projection device such as a projector.

[0034] The audio input unit 150 is an input device capable of receiving sounds from the surrounding area where the display device 10 is installed, such as a microphone. The audio input unit 150 may also consist of multiple input devices (for example, a microphone array). The audio input unit 150 is primarily used for inputting the voice of the user listening, but it can also receive general sounds such as ambient noise. The display device 10 can recognize the user's state from the audio input from the audio input unit 150.

[0035] The audio output unit 160 outputs the audio contained in the content. The audio output unit 160 may be a device such as a speaker or headphones. The audio output unit 160 only needs to output sound, and can output general sounds such as music or ambient sounds. There may be multiple audio output units 160.

[0036] The imaging unit 170 is an imaging device that captures images of the area surrounding the display device 10, and is, for example, a camera. The imaging unit 170 may consist of one or more imaging devices. The imaging unit 170 outputs the captured images as image signals. The imaging unit 170 may also output one or more images as a continuous video. From the images captured by the imaging unit 170, the display device 10 can recognize the user's state.

[0037] The communication unit 180 is a communication interface for communicating with other devices. For example, the communication unit 180 may be a network interface that can connect to a wireless LAN, or a network interface that can connect to Ethernet (registered trademark) via a wired connection. Alternatively, the communication unit 180 may be a communication device that can connect to a mobile communication network such as LTE / 4G / 5G / 6G.

[0038] Furthermore, the configuration shown in Figure 2 only needs to include the necessary components in the embodiment. For example, if the user's status is not acquired from the camera, the imaging unit 170 may not be necessary.

[0039] Furthermore, the display device 10 may connect to an external device via an interface unit, and the above-mentioned functions may be implemented by the external device. For example, the imaging unit 170 may be implemented by a camera device connected via a USB interface.

[0040] [1.3 Software Configuration] [1.3.1 Configuration of the Control Unit] Figure 3 is a diagram illustrating the configuration of the control unit within the software configuration. For example, each configuration in Figure 3 is realized when the control unit 100 executes a program stored in the memory unit 110. The control unit 100 also functions as a display control unit that displays content, avatars, etc., on the display unit 140. The display control unit may also include functions such as a content image output unit 1006 and an avatar image output unit 1046.

[0041] (content) The content acquisition unit 1002 acquires content from the broadcast control unit 120 or the communication unit 180. Alternatively, the content acquisition unit 1002 may acquire content stored in the storage unit 110.

[0042] The content playback unit 1004 plays (displays) the content acquired by the content acquisition unit 1002. The content playback unit 1004 separates the multiplexed image data from the content and outputs it to the content image output unit 1006. The content playback unit 1004 also separates the multiplexed audio data from the content and outputs it to the content audio output unit 1008. Any known method can be used to separate the image data (image signal) and audio data (audio signal) from the content.

[0043] The content image output unit 1006 outputs the image data contained in the content to the display unit 140 as an image (or a video as a series of images). Here, the display unit 140 displays the image of the content as the first image.

[0044] Furthermore, the content audio output unit 1008 outputs the audio data contained in the content as audio to the audio output unit 165. Here, the audio output unit 165 outputs the audio of the content as the first audio.

[0045] Furthermore, the content recognition unit 1010 recognizes image data, audio data, and content information contained in the content (for example, program information and subtitle information in the case of broadcast waves) and outputs them to the prompt generation unit 1032. The information about the content recognized by this content recognition unit 1010 is called content information.

[0046] (User perception) The user recognition unit 1022 recognizes a user who is a viewer of the content based on the image captured by the shooting unit 155, and recognizes information about that user. Information about the user is called user information, and the user recognition unit 1022 can recognize the following as user information.

[0047] (1) The number of users. For example, the user recognition unit 1022 recognizes how many people are watching the content. Alternatively, the user recognition unit 1022 may recognize the number of people watching the content by using the direction of the face and the direction of the gaze.

[0048] (2) Identifying the user. For example, the user recognition unit 1022 recognizes who the viewer of the content is. The user recognition unit 1022 can identify the user by acquiring an image of the user's face (face image) and matching it with a face image of a user that has been registered in advance.

[0049] (3) User's facial expression. For example, the user recognition unit 1022 can recognize the user's facial expression.

[0050] The speech recognition unit 1024 recognizes the conversation (input text) spoken by the user based on the voice input unit 150. The speech recognition unit 1024 outputs the input text to the conversation processing unit 1030.

[0051] Furthermore, the speech recognition unit 1024 may recognize the user's emotions (for example, interjections or clapping sounds) as user information.

[0052] In other words, the user recognition unit 1022 can recognize user information based on an image, and the voice recognition unit 1024 can recognize user information based on voice.

[0053] The conversation processing unit 1030 outputs the input sentence, the past conversation history, and any other necessary supplementary information to the large-scale language model 22, and retrieves output data including the response sentence from the large-scale language model 22. This allows the conversation processing unit 1030 to perform a conversation using the input sentence entered by the user and the response sentence output by the avatar.

[0054] To explain in more detail, the conversation processing unit 1030 includes a prompt generation unit 1032 and an output data acquisition unit 1034.

[0055] The prompt generation unit 1032 generates prompts to be fed into the language model (e.g., the large-scale language model 22) based on the input information. The prompts may be in natural language format or in a predetermined format corresponding to the language model 1102 using tags, etc. The prompt generation unit 1032 generates prompts based on the necessary information from the input sentence, conversation history, and supplementary information. Hereinafter, information other than conversation-related information may be referred to as supplementary information.

[0056] Here, supplementary information other than the input text and conversation history may include, for example, the following:

[0057] • Content information. This may include, for example, the name of the content, the content analyzed from the images of the content, the content recognized from the audio of the content, etc.

[0058] • User information. For example, in addition to basic user information (age, gender, number of people, etc.), it may also include information such as the user's facial expressions, movements, and interests.

[0059] • Environmental information. This may include information about the user's environment, such as the date and time, room environment, and weather information.

[0060] • Product information. For example, this may include one or more pieces of information about a product on a designated shopping site. In addition to basic information such as product name and price, the product information may also include information such as product features, weight, size, and product images that can be obtained from the product sales site.

[0061] The prompt generation unit 1032 may, for example, generate a prompt to input to the large-scale language model 22 by inputting the above-mentioned information to the edge LLM once.

[0062] The conversation processing unit 1030 (prompt generation unit 1032) inputs a prompt containing the input sentence as an example of a language model to the large-scale language model 22 of the server device 20. The large-scale language model 22 outputs output data to the conversation processing unit 1030 according to the input prompt. The large-scale language model 22 is based on the one in the server device 20, but if it is stored in the storage unit 110, the control unit 100 can also use the one stored in the storage unit 110. Furthermore, the control unit 100 may choose to use the large-scale language model 22 or the language model stored in the storage unit 110 depending on the content of the conversation and the type of output data requested.

[0063] The conversation processing unit 1030 (output data acquisition unit 1034) acquires output data from the large-scale language model 22. Here, the output data may include the following information.

[0064] • Response to the input statement • Product recommendation information The conversation processing unit 1030 outputs, for example, the input sentence included in the prompt and the response sentence included in the output data as a conversation to the conversation history storage processing unit 1050. The conversation history storage processing unit 1050 stores the conversation, including the input sentence and response sentence, and information related to the conversation as conversation history information in the conversation history information storage area 1110.

[0065] Furthermore, the conversation processing unit 1030 outputs the conversation to the avatar output control unit 1040. The conversation processing unit 1030 mainly outputs the response sentences contained in the output data and information about the avatar contained in the output data (for example, the avatar's facial expressions, etc.) to the avatar output control unit 1040.

[0066] The avatar output control unit 1040 functions as an avatar generation unit that generates avatars. The avatar output control unit 1040 controls, for example, the avatar's facial expressions, movements, and appearance. The avatar output control unit 1040 also determines what the avatar will say. The avatar output control unit 1040 then outputs a signal to the avatar image generation unit 1042 for generating an image of the avatar. Here, the avatar image may include not only an image representing the avatar but also a message spoken by the avatar. The avatar image generated by the avatar image generation unit 1042 is then output to the display unit 140 via the avatar image output unit 1046.

[0067] Furthermore, the avatar output control unit 1040 outputs audio signals and text data for generating the voice spoken by the avatar to the avatar voice generation unit 1044. The avatar voice generation unit 1044 generates the avatar's voice in accordance with the response sentence spoken by the avatar. The avatar voice output unit 1048 outputs the generated avatar voice via the voice output unit 160.

[0068] Here, the avatar image generation unit 1042 may acquire information about the voice emitted by the avatar from the avatar voice generation unit 1044. Alternatively, the avatar image generation unit 1042 may acquire information about the voice emitted by the avatar from the avatar output control unit 1040.

[0069] Here, the avatar image may change its facial expression or mouth shape in response to the response text. For example, the avatar can receive a response text from the avatar image generation unit 1042 or the avatar voice generation unit 1044 and change the display of the avatar in accordance with the response text.

[0070] Furthermore, the avatar image may be displayed superimposed on the content image, or it may be displayed separately in a different area.

[0071] [1.3.2 Memory Unit Configuration] The configuration of the memory unit 110 will be explained with reference to Figure 4.

[0072] Language model 1102 is a language model available on the display device 10. Language model 1102 may be, for example, an SLM or an LLM. Language model 1102 may function as an edge LLM when using the large-scale language model 22. Alternatively, language model 1102 may be used instead of the large-scale language model 22.

[0073] The user information storage area 1104 stores user information, which is information about the user. User information is also called user attributes. User information may, for example, pre-store users who use the display device 10, or it may be stored as a user each time a user is recognized. User information may include, for example, user identification information such as the user's name, full name, an icon representing the user, or an image representing the user. User information may also store the user's age, age group, and gender.

[0074] The content information storage area 1106 stores content information, which is information about the content. The content information may store information about the content being played, or it may store content information corresponding to content stored in the storage unit 110. The content information may include information such as the content name, the length (time) of the content, and the genre (attributes) of the content. The content information may also include the results of recognizing the image and sound of the content. For example, the content information may include scenes of the content recognized by the content recognition unit 1010 from the image of the content (information recognized from the image, information about people, objects, etc. included in the image, location information), words included in the audio of the content recognized from the audio of the content, and information about products introduced in the content. The content information may also be stored in chronological order.

[0075] The avatar information memory area 1108 stores avatar information, which is information about the avatar. The avatar information may include, for example, gender (male, female), age, face shape, arrangement of facial features, type of clothing, and type of accessories. The avatar information may also include attributes such as whether the avatar is human, cat, dog, or anime character. Furthermore, the avatar information may include information about the avatar's emotions and the sounds the avatar makes.

[0076] The conversation history information storage area 1110 stores conversation history information, which is information about the history of a conversation. The conversation history information includes input sentences and response sentences as conversations. In addition to conversations, the conversation history information may also store one or more pieces of related information, such as user information, content information, and environment information, as needed. Furthermore, the conversation history information may store the conversation history as a single unit from the start of viewing the content to the end of viewing the content, or it may store the conversation history chronologically by date.

[0077] The Similar Word Database (DB) 1112 is a database that stores similar word relationships, associating words that sound alike with each other. The Similar Word Database 1112 is also called the Similar Word Memory Unit. Words that sound alike refer to words that sound similar, such as "Kanazawa" and "Canada," "wide" and "white," and "1 o'clock" and "7 o'clock." As can be seen from the example of "Kanazawa" and "Canada," even if the number of characters is different, if they sound similar, they are considered similar words.

[0078] [1.4 Overall Explanation] The following describes the overall flow of the conversation between the user and the avatar in this embodiment.

[0079] [1.4.1 Conversation Processing Flow] Figure 5 illustrates the general flow of conversation processing. While it is preferable that the following processes be performed by any of the configurations described in Figure 3, for the sake of explanation, these processes will be described as being performed by the control unit 100.

[0080] The control unit 100 receives the content (S10). The control unit 100 recognizes the content and acquires the content information.

[0081] When the control unit 100 detects that the user has spoken (S14; Yes), it recognizes the speech and creates an input sentence (S16). The control unit 100 also obtains necessary information from user information and conversation history information (S18). Here, the control unit 100 only needs to obtain at least the information necessary for the processing described later.

[0082] The control unit 100 selects one or more necessary pieces of information from the input sentence, content information, user information, conversation history information, and other supplementary information to generate a prompt (S18). Then, the control unit 100 inputs the generated prompt into the language model (for example, the large-scale language model 22) (S20).

[0083] When the control unit 100 obtains output data corresponding to a prompt from a language model (e.g., a large-scale language model 22) (S22; Yes), it generates a response sentence from the output data (S24).

[0084] Then, the control unit 100 outputs a response sentence from the avatar (S26). The control unit 100 repeatedly executes the process until the conversation processing is completed (S28; No → S10).

[0085] Figure 5 illustrates an example of conversation processing between a user and an avatar, including input and response sentences. The necessary processing will be executed as appropriate depending on the situation. For example, in conversation processing where content is not used, such as when using a shopping site, it is not necessary to receive content or obtain information about the content. For instance, steps S10 and S12 in Figure 5 do not need to be executed.

[0086] Furthermore, when processing conversations in scenes such as menu screens and settings screens, it is not necessary to receive content or obtain content information. In this case as well, steps S10 and S12 in Figure 5 do not need to be executed.

[0087] Furthermore, although Figure 5 illustrates the example of the user initiating the speech, the avatar may also initiate the speech. For example, the control unit 100 may execute processing from the prompt generation process (S20) in S20. In this case, the avatar will output a response before the user initiates the speech.

[0088] [1.4.2 Interaction between the display area and the large language model] Figure 6(a) shows an example of a display screen W10. The display screen W10 has a first area R10 for displaying content, a second area R12 for displaying an avatar, and a third area R13 for displaying dialogue. The second area R12 and the third area R14 may be configured as a single unit.

[0089] Furthermore, it is preferable to display the conversation in a way that distinguishes between the user's utterance (input) and the avatar's utterance (response). For example, Figure 6(a) displays the user's utterance and the avatar's utterance in a way that distinguishes between them based on the direction of the speech bubble. Alternatively, the user's utterance and the avatar's utterance may be distinguished by, for example, background color, text color, font, etc.

[0090] Furthermore, in the display screen W10 of Figure 6(a), the first area R10, the second area R12, and the third area R14 are displayed, but the display of these areas can be switched. For example, the user can configure the display screen W10 to display only the avatar by enabling only the second area R12. In this case, the avatar may be adjusted to a size appropriate for the screen size of the display screen W10. Similarly, the user can configure the display screen W10 to display only the dialogue by enabling only the third area R14. In this case, the dialogue may be adjusted to a size appropriate for the screen size of the display screen W10, the font may be enlarged, or more dialogue may be displayed than usual. It is also possible to select and display multiple areas, such as the first area R10 and the third area R14, or the second area R12 and the third area R14. In the following display screens, the avatar, content, and dialogue can each be selectively displayed.

[0091] Furthermore, for the sake of explanation, the avatar, content, and dialogue were described using the example of switching display areas, but the control unit 100 may also switch the display content itself on and off while keeping the display area the same. Also, the display area includes the display screen. In addition, the control unit 100 may output the content displayed in each display area (display screen) to different displays. For example, the content displayed in the first area R10 may be displayed on the display unit 140 of the display device 10, and the avatar and dialogue displayed in the second area R12 and the third area R14 may be output to an external display or to another terminal device (e.g., a smartphone) connected via the communication unit 180.

[0092] Figure 6(b) schematically shows an example of a prompt for input to a language model (e.g., large-scale language model 22) as natural language. Figure 6(c) schematically shows an example of output data output from the language model (e.g., large-scale language model 22) as natural language.

[0093] The prompt preferably includes the input sentence. For example, Figure 6(b) includes the input sentence, "Where is Tenri?". The prompt may also include one or more pieces of information, such as conversation history information, content information, user information, interest information, and product information. Furthermore, the prompt may include this information that has been optimized once by the edge LLM.

[0094] The output data preferably includes a response statement. For example, Figure 6(c) includes the response statement, "Northern Nara Prefecture, are there many tourists?" The output data may also include one or more pieces of information, such as avatar information (avatar's facial expressions, etc.), recommendation information, site information, and supplementary information.

[0095] [1.4.3 Voice Assistant] Voice assistants are technologies that combine speech recognition technology with artificial intelligence (AI). Examples of voice assistants include Google Assistant from Google, Alexa from Amazon, and Siri from Apple.

[0096] Devices equipped with voice assistants include, for example, in-car infotainment systems, smart speakers, and smart TVs. In-car infotainment systems include functions such as navigation, location services, voice communication, multimedia playback, various search functions, and air conditioning. Smart speakers include functions such as streaming services, music playback, home appliance control, reading out information such as news and weather forecasts, creating and sending emails, playing videos, reading books aloud, shopping, and creating notes. Smart TVs include functions such as voice control of the TV, broadcast reception, multimedia playback, home appliance control, and smart home hub functionality.

[0097] Voice recognition and voice control offer several advantages, such as the following: • It can be operated hands-free. Therefore, it can be operated without any problems even when your hands are full, unable to use, or dirty. • It enables intuitive operation and direct access to desired functions. In other words, it avoids the cumbersome operation of hierarchical menus. • It eliminates the need to focus on the object being operated, reducing the amount of eye movement required. • Because it does not require displays, buttons, etc., for presenting information, the user interface can be simplified and streamlined.

[0098] On the other hand, voice recognition and voice control have the following disadvantages, for example: • There is uncertainty in the operation. Due to variations and inconsistencies in wording and other expressions, the user's intentions may not be accurately conveyed, which may reduce the reproducibility of the operation. • Correcting and adjusting operations is difficult, resulting in low tolerance for errors. While voice control allows for intuitive execution of multi-step operations, it is difficult to correct operations that the user did not intend. In other words, if the voice input contains a mix of operations that align with the user's intentions and operations that do not, it is difficult to execute only the former or correct only the latter.

[0099] [1.4.4 Challenges and Ideals of Speech Recognition] Figure 7 is a conceptual schematic diagram illustrating the comparison of the difficulty of speech recognition between conversations between people and devices, and between people and people. When using voice commands to operate smartphones, smart speakers, robots, etc., that implement general speech recognition technology, many people tend to utter the following phrases. Speak a short sentence. • Speak sentences with the correct word order. • Speak sentences that are complete and accurate in content. Speak slowly. • Speak clearly and articulately. Pronounce it accurately. To speak without hesitation or rephrasing. These types of utterances are easy to recognize through speech recognition.

[0100] In contrast to conversations with devices, people tend to make the following types of statements in face-to-face conversations: They may speak in fragments. • Sometimes the speaker may utter sentences with inappropriate word order. They may speak in long sentences. • Sometimes, two or more sentences are spoken consecutively. • Sometimes speaks in redundant sentences. • They may utter sentences that are incomplete (for example, sentences where the subject, object, etc., are omitted). Speak quickly. To pronounce something vaguely. • To speak while hesitating or correcting oneself. • Speaks with poor articulation.

[0101] While such utterances are difficult for machines to recognize, they are less of a problem in person-to-person conversations because the listener can fill in the blanks themselves.

[0102] On the other hand, when implementing a voice assistant in the form of a conversation with an avatar on a display device such as a television, the speaker may not necessarily perceive the avatar solely as a device. Because the avatar is presented as a human figure, the speaker is likely to perceive the avatar as something closer to a person than a device, and thus speak in a manner similar to a face-to-face conversation. However, problems can arise when attempting to recognize the utterances of a face-to-face conversation using speech recognition technology.

[0103] [1.4.5 Conversation between a person and an avatar (AI)] Figure 8 is a conceptual schematic diagram illustrating conversation between a human and an avatar (AI, Artificial Intelligence). As mentioned above, when conversing with an avatar, the speaker (user) may make the following utterances. • Speaks in fragments. • Speak a sentence with the word order reversed from the original sentence. To hesitate, to rephrase. • Poor articulation.

[0104] Even if the speech recognition unit accurately converts the user's spoken audio into text word for word, the resulting text may not necessarily make sense as a whole. Therefore, even if the speech recognition unit generates a prompt string based on that text, the AI ​​conversation service may not be able to respond in a way that aligns with the user's intent.

[0105] Another problem unique to the Japanese language is the error in reading kanji characters by their on'yomi and kun'yomi pronunciations. Even when a generative AI conversation service generates an appropriate response string in response to a user's utterance, the speech synthesis unit may misread the kanji characters included in the response string. For example, if a generative AI conversation service outputs the response string "We cannot start a new recording because we are recording another program," and the speech synthesis unit outputs speech based on this response string, it may misread "itte" as "itte" instead of "okonatte."

[0106] [1.4.6 Speech Recognition Methods] Figure 9 is a conceptual schematic diagram illustrating a speech recognition model. Figure 9(a) shows a conventional model. Figure 9(b) shows an end-to-end speech recognition model.

[0107] Conventional models are also called hybrids of DNN (Deep Neural Network) and HMM (Hidden Markov Model). As shown in Figure 9(a), conventional speech recognition converts speech to text by combining multiple statistical models (acoustic model, pronunciation model, and language model). The acoustic model decomposes speech into phonemes, the pronunciation model maps speech to words using a pronunciation dictionary, and the language model formats the word set into a document based on clock processing data.

[0108] In contrast, End-to-End speech recognition directly converts speech to text using a single neural network, as shown in Figure 9(b).

[0109] [1.5 Processing Flow] Figure 10 is a flowchart illustrating the operation of the display device according to the first embodiment of this disclosure when it outputs a response sentence to a user U's utterance. The following processes are preferably performed by any of the configurations described in Figure 3, but for the sake of explanation, they will be described as being performed by the control unit 100.

[0110] When voice input is received via the voice input unit 150 (step S41), the control unit 100 performs voice recognition processing to generate an input sentence (step S43). The control unit 100 stores the generated input sentence in the conversation history information storage area 1110 (step S45).

[0111] The control unit 100 divides the input sentence into multiple phrases 1, 2, ... and determines whether a similar phrase corresponding to each phrase exists in the similar phrase DB 1112 (step S47). For example, if a phrase of the same part of speech as phrase 1 and with a similar pronunciation model is registered in the similar phrase DB 1112, it is determined that a similar phrase exists for phrase 1. The input sentence can be divided using a morphological analysis engine such as MeCab, ChaSen, JUMAN++, or Sudachi.

[0112] If a similar phrase exists for any of phrases 1, 2, ... (step S49, yes), the control unit 100 generates a candidate sentence based on the phrase for which a similar phrase exists and that similar phrase (step S51). If no similar phrase exists for any of phrases 1, 2, ... (step S49, no), the process proceeds to step S61, which will be described later.

[0113] Following step S51, the control unit 100 excludes candidate sentences that are deemed invalid from candidate sentence 1, candidate sentence 2, ... based on the conversation history stored in the conversation history information storage area 1110 (step S53). Next, the control unit 100 excludes candidate sentences that are deemed invalid from candidate sentence 1, candidate sentence 2, ... based on the user attributes stored in the user information storage area 1104 (step S55).

[0114] If, after step S55, the number of remaining candidate sentences is 0, that is, if there are no candidate sentences remaining (step S57, "0"), the control unit 100 generates a prompt requesting the generation of a response sentence as follows (step S59). That is, a response sentence requesting the user to re-enter voice input.

[0115] If, after step S55, the number of remaining candidate sentences is 1 (step S57, "1"), the control unit 100 generates a prompt based on that candidate sentence (step S61).

[0116] If, after step S55, the number of remaining candidate sentences is two or more (step S57, "two or more"), the control unit 100 generates a prompt requesting the generation of a response sentence as follows (step S63). That is, a response sentence that inquires which candidate sentence is in line with the user's intent.

[0117] After step S59, S61, or 63, the control unit 100 uses the communication unit 180 to send the generated prompt to the large-scale language model 22 of the server device 20 (step S67). Next, the control unit 100 uses the communication unit 180 to receive output data from the large-scale language model 22 of the server device 20 (step S69). The control unit 100 extracts a response sentence from the received output data, generates voice data for an avatar to read the response sentence aloud, and outputs it as voice from the voice output unit 160 (step S71). At this time, the text of the response sentence may be added to the avatar image, for example, in the form of a speech bubble, and displayed on the display unit 140. The control unit 100 also adds the text of the response sentence to the conversation history in the conversation history information storage area 1110 (step S73).

[0118] In the above description, the control unit 100 executes step S55 after step S53, but the order of these two steps does not matter. Furthermore, the control unit 100 may be configured to execute either of these two steps.

[0119] [1.6 Example of Operation] In step S41, suppose user U says, "How do I get to Kanazawa from here at 7 o'clock tomorrow?" In step S47, if the similar phrase database finds that the phrase "7 o'clock" has a similar phrase "1 o'clock" and the phrase "Kanazawa" has a similar phrase "Canada", then in step S51, the control unit 100 generates 2 x 2 = 4 candidate sentences, which are two combinations of the phrases "7 o'clock" and "1 o'clock", and two combinations of the phrases "Kanazawa" and "Canada". Specifically, these are candidate sentences 1 to 4 below.

[0120] Candidate sentence 1 "How can I get from here to Kanazawa by 7 AM tomorrow?" Candidate sentence 2 "How can I get from here to Kanazawa by 1 AM tomorrow?" Candidate sentence 3 "How can I get to Canada from here by 7 AM tomorrow?" Candidate sentence 4 "How can I get to Canada from here at 1 AM tomorrow?"

[0121] For example, if the conversation history in the conversation history information storage area 1110 contains many conversations related to "Canada" but no topics related to "Kanazawa," in step S53, the control unit 100 excludes candidate sentences 1 and 2, which include "Kanazawa," as invalid candidates. Alternatively, the control unit 100 may compare the frequency of "Kanazawa" and "Canada," select the one that appears more frequently, and generate a response sentence. Furthermore, if the frequency of each candidate is similar, the control unit 100 may determine that the user U's intention is unclear and execute the processing in step S59 or step S63, which will be described later.

[0122] Furthermore, for example, if Tokyo is registered as the user's location (the location where the display device 10 is installed) in the user information (user attributes) of the user information storage area 1104, it is unlikely that the user would go to Canada tomorrow at "1 o'clock" or "7 o'clock," so candidate sentences 3 and 4, which contain "Canada," are excluded as invalid candidates.

[0123] If there are no remaining candidate sentences in step S57, the control unit 100 generates a prompt requesting the user to generate a response sentence that asks them to re-enter the voice input (step S59). The prompt at this time may be, for example, "Please create a response sentence that asks for the previous inquiry again."

[0124] If there is only one candidate sentence remaining in step S57, the control unit 100 generates a prompt based on that candidate sentence (step S61). For example, if only candidate sentence 1 remains, the control unit 100 sends candidate sentence 1, "How can I get from here to Kanazawa at 7 o'clock tomorrow?" as a prompt to the large-scale language model 22.

[0125] If there are two or more candidate sentences remaining in step S57, the control unit 100 generates a prompt requesting the generation of a response sentence inquiring which candidate sentence best reflects the user's intent (step S63). For example, if candidate sentence 1 and candidate sentence 2 remain, the control unit 100 generates a prompt requesting the large-scale language model 22 to generate a response sentence inquiring which of candidate sentence 1 or candidate sentence 2 best reflects the user U's intent. This prompt may look something like this:

[0126] "Please generate a response that asks the user which of the following inquiry options best reflects their intent." 1) How can I get from here to Kanazawa by 7 AM tomorrow? 2) How can I get from here to Kanazawa at 1 PM tomorrow?

[0127] In response to this prompt, the large-scale language model 22 generates a response sentence like the following, for example: "I didn't hear you very well. Which of the following did you say?" 1) How can I get from here to Kanazawa by 7 AM tomorrow? 2) How can I get from here to Kanazawa at 1 PM tomorrow? In this way, if the input sentence is unclear, the display device 10 generates possible candidate sentences from the input sentence, excludes invalid candidate sentences based on the conversation history and user attributes, and if multiple candidate sentences still remain, it lists the remaining candidate sentences and displays a response asking which one best reflects the user's intention. This allows the device to perform an action equivalent to asking for clarification in a conversation between humans.

[0128] [1.7 Variation 1] Modification 1 of the first embodiment will now be described. In the first embodiment, step S63 generated a prompt requesting the large-scale language model 22 to generate a response sentence for the user to select a candidate sentence. In Modification 1, however, the candidate sentence is selected based on feature data that shows the characteristics of the speech uttered by user U.

[0129] In step S41, the control unit 100 stores the audio data of user U's speech input from the audio input unit 150 in the storage unit 110, associating it with user U. Based on this audio data, it generates feature data indicating the characteristics of the user's pronunciation and stores it in the storage unit 110, associating it with user U. Instead of step S63, the control unit 100 selects one of the candidate sentences based on the feature data and generates a prompt based on that candidate sentence.

[0130] [1.8 Variation 2] Modification 2 of the first embodiment will now be described. Modification 2, like Modification 1, replaces step S63.

[0131] In Modification 1, candidate sentences were selected based on the characteristics of the voice spoken by user U, but in Modification 2, candidate sentences are selected based on the flow of the conversation. That is, the control unit 100 selects one of the two or more remaining candidate sentences in step S57 based on the conversation history stored in the conversation history information storage area 1110, and generates a prompt based on that candidate sentence.

[0132] [1.9 Summary] According to the first embodiment, it is possible to provide a processing device capable of handling unclear voice input, and a display device to which the processing device is applied.

[0133] [2. Second Embodiment] A second embodiment will now be described. In the second embodiment, the input sentence is divided into words and phrases based on silent intervals in the voice input. Here, the differences from the first embodiment will be explained, and the same points will be omitted from the explanation.

[0134] In step S41, when the voice input unit 150 receives voice input and detects a silent interval, the control unit 100 inserts a space (for example, a character represented by Unicode U+0020) into that interval. Regardless of the length of the silent interval, one space may be inserted for each silent interval, or a number of spaces corresponding to the length of the silent interval may be inserted. In step S47, when dividing the input sentence into phrase 1, phrase 2, ..., a space is used as the boundary between phrases to identify each phrase.

[0135] Furthermore, before generating a prompt in step S61 or S63, the control unit 100 may display a candidate sentence that will serve as the basis for the prompt on the display unit 140 and request the user U to make an input via the operation unit 130 to confirm whether or not the candidate sentence is in line with the user's intent.

[0136] According to this embodiment, it is possible to prevent misunderstandings of input sentences due to errors in word division. For example, a user U may list multiple search words and ask the avatar to perform a search. Suppose user U utters "clothes, storage, furniture" as search words. In this case, according to the second embodiment, spaces are inserted between "clothes," "storage," and "furniture," and it is identified as "fuku shūnō kagu," and then identified as a search word such as "clothes storage furniture" through kana-kanji conversion. On the other hand, if no spaces are inserted, user U's utterance "clothes, storage, furniture" is identified as "fuku shūnō kagu." Therefore, there is a risk that it may be misconverted to, for example, "furniture of revenge," and user U's intention may not be correctly understood.

[0137] [3. Third Embodiment] A third embodiment will now be described. People sometimes omit words or phrases unintentionally when speaking. Therefore, if the input sentence is interpreted only in terms of the omitted words or phrases, it may not be possible to output a response sentence that reflects the speaker's intent. In the third embodiment, the omitted (missing) words or phrases are added to the input sentence to output a response sentence that reflects the speaker's intent. Here, the differences from the first embodiment will be explained, and the same points will be omitted from the explanation.

[0138] Figure 11 is a flowchart illustrating the operation of the display device according to the third embodiment of this disclosure when it outputs a response sentence to a user U's utterance. Compared to the flowchart of the first embodiment in Figure 10, it differs in that step S81 has been added.

[0139] In step S81, natural language processing such as morphological analysis, syntactic analysis, semantic analysis, and contextual analysis is performed on each candidate sentence generated in step S51. If the meaning of the candidate sentence cannot be uniquely determined as a result of natural language processing, the control unit 100 estimates the part of speech and sentence components of the omitted words. For example, verbs and nouns may be omitted as parts of speech. Also, subjects and objects may be omitted as sentence components. Based on the estimation results of the parts of speech and sentence components and the conversation history in the conversation history information storage area 1110, the control unit 100 estimates the omitted words and modifies the candidate sentence based on the estimation results.

[0140] If multiple candidates for the missing word or phrase are estimated, additional candidate sentences may be added depending on the number of candidates. For example, if it is detected that the subject is missing from candidate sentence 1, and there are two possible subject candidates, person P1 and person P2, then two candidate sentences will be generated from candidate sentence 1: one with person P1 added as the subject, and another with person P2 added as the subject.

[0141] As an example, consider the input sentence "Can I watch movie M on video service A?". This input sentence can be interpreted in the following ways. (Interpretation 1-1) The question asks whether video service A is currently streaming movie M. (Interpretation 1-2) This question asks whether viewing is possible under user U's viewing restrictions (restrictions based on user rights contracted with video service A, age restrictions based on user U's age, etc.). (Interpretation 1-3) This question asks whether watching movie M to the end from now will affect user U's schedule. (Interpretation 1-4) This question asks whether video service A has any plans to distribute movie M in the future.

[0142] In this case, in step S81, the control unit 100 generates a total of four candidate sentences from the input sentence "Can I watch movie M on video service A?" based on interpretations 1-1 to 1-4.

[0143] As another example, let's consider the case where the input phrase is "Turn up the volume." This input phrase can be interpreted in the following ways. (Interpretation 2-1) This requests that the volume of the avatar's voice displayed in the second area R12 be increased. (Interpretation 2-2) This requests that the volume of the content displayed in the first area R10 be increased.

[0144] In this case, in step S81, the control unit 100 generates a total of two candidate sentences from the input sentence "Turn up the volume" based on interpretation 2-1 and interpretation 2-2.

[0145] In the above description, the control unit 100 executes steps S53 and S55 after step S81, but the order of these three steps does not matter. Furthermore, the control unit 100 may be configured to omit at least one of these three steps, namely steps S53 and S55.

[0146] According to the third embodiment, even if user U omits words or phrases when speaking, the display device can output a response sentence that is in line with user U's intent.

[0147] [4. Fourth Embodiment] A fourth embodiment will now be described. In the fourth embodiment, errors in the on-yomi and kun-yomi readings of kanji characters are avoided when synthesizing speech data from response text received from the large-scale language model 22. The words that make up the response sentence are highly likely to be included in the input sentence, especially the most recent input sentence. Taking this into consideration, in this embodiment, the correspondence between hiragana and words containing kanji when the input sentence is converted to kana-kanji is stored, and this correspondence is referred to when reading the response sentence aloud. Here, the differences from the first embodiment will be explained, and the same points will be omitted from the explanation.

[0148] Figure 12 is a block diagram illustrating the configuration of the storage unit 110 of the display device 10 according to the fourth embodiment of this disclosure. Compared to the storage unit 110 of the first embodiment shown in Figure 4, it differs in that it includes a reading database (DB, Database) 1114. The reading DB 1114 stores the correspondence between hiragana and kanji conversion when inputting text.

[0149] Figure 13 is a flowchart illustrating the operation of the display device 10 according to the fourth embodiment of this disclosure when it outputs a response sentence to a user U's utterance. Compared to the flowchart of the first embodiment in Figure 10, it differs in that steps S131 and S133 have been added.

[0150] In step S43, the voice input audio data is recognized by speech recognition to generate hiragana, and then kana-kanji conversion is performed to generate the text of the input sentence. In the next step S131, the control unit 100 associates the hiragana that were the target of kana-kanji conversion with the words containing the kanji obtained by converting those hiragana, and stores them in the reading DB 1114.

[0151] In step S69, when the control unit 100 receives the text of the response sentence as part of the output data from the large language model 22, the control unit 100 extracts a phrase containing Chinese characters from the response sentence text. If the extracted phrase is registered in the reading DB 1114, the corresponding hiragana is read out from the reading DB 1114. The read hiragana is added to the response sentence as the reading data for the phrase. For example, it is conceivable to describe the hiragana enclosed in parentheses () immediately after the phrase in the response sentence.

[0152] For example, the Chinese character-containing phrase "お札" has readings of "おふだ" and "おさつ". Also, for the Chinese character-containing phrase "最中", readings such as "もなか", "さなか", and "さいちゅう" are conceivable. According to this embodiment, the correspondence between the phrases and hiragana used in the kana-Chinese character conversion when creating the input sentence is also applied when reading aloud the response sentence. Thereby, in this embodiment, it is possible to avoid misreading the on-yomi or kun-yomi of Chinese characters when synthesizing voice data from the response sentence text.

[0153] In the above description, the control unit 100 executes steps S53 and S55, but the order of these two steps does not matter. Also, the control unit 100 may be configured not to execute at least one of steps S53 and S55 out of these two steps.

[0154] Thus, according to the fourth embodiment, it is possible to suppress the occurrence of an error in the output voice processing from the response sentence text.

[0155] [5. Fifth Embodiment] The fifth embodiment will be described. In the first embodiment, an operation of re-listening to unclear voice input was performed. In the fifth embodiment, the wake word is changed according to whether the user U is in front of the display device 10 or not. Here, the points different from the first embodiment will be described, and the description of the same points will be omitted.

[0156] Figure 14 is a flowchart illustrating the operation of a display device according to the fifth embodiment of this disclosure in response to a wake word. Before the start of the flowchart, the power to the main functions of the display device 10 is assumed to be off. In addition, the display device 10 is equipped with a power supply that is always on, separate from the power supplies for these main functions, and the operation of the flowchart in Figure 14 is performed by this power supply.

[0157] The control unit 100 uses the imaging unit 170 to photograph the area around where the display device 10 is installed and attempts to recognize the user based on the captured image (step S101). If no user is recognized in the vicinity of the display device 10 (step S103, no), the control unit 100 selects a full wake word. The full wake word is a general wake word, preferably a word chosen so that user U does not utter it independently of activating the display device 10. On the other hand, if a user is recognized in the vicinity of the display device 10 (step S103, yes), the control unit 100 selects a simplified wake word (step S107). The simplified wake word is a wake word that is only valid when the user is in the vicinity of the display device 10. Instead of imposing the condition that the user is in the vicinity of the display device 10 to be valid, the simplified wake word can use a word that is easier to pronounce than the full wake word. Typically, the number of characters in the simplified wake word is preferably fewer than the number of characters in the full wake word. After a simple wake word or a complete wake word is selected, the control unit 100 performs speech recognition on the voice input from the voice input unit 150. If it detects that the selected wake word has been input by voice (step S109, yes), the control unit 100 activates the main function of the display device 10 (step S111). If it does not detect that the selected wake word has been input by voice (step S109, no), the control unit 100 returns to step S101.

[0158] In the above description, the display device 10 was described as detecting only one of either a simple wake word or a complete wake word as a wake word. However, the simple wake word may be detected only when a user is near the display device 10, while the complete wake word may always be detected regardless of whether a user is near the display device 10 or not.

[0159] Furthermore, in the above explanation, the simple wake word was enabled on the condition that an unspecified user was in the vicinity of the display device 10, but this condition may be changed. For example, the following are possible conditions for enabling the simple wake word. (Condition 1) When a specific user is near the display device 10. (Condition 2) When a specific or unspecified user is facing the imaging unit 170 of the display device 10 and visually inspecting the lens of the imaging unit 170. (Condition 3) When a specific or unspecified user is pointing towards the imaging unit 170 of the display device 10.

[0160] If the activation conditions for a simple wake word are stricter, the simple wake word can be made shorter. For example, under condition 3, words like "Hey," "Excuse me," or "Come on" could be used as simple wake words.

[0161] According to the fifth embodiment, when a user is near the display device 10, the user can activate the display device 10 with a simpler wake word that is easier to pronounce.

[0162] [6. Variant] This disclosure is not limited to the embodiments described above, and various modifications are possible. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of this disclosure are also included in the technical scope.

[0163] Furthermore, although the embodiments described above are explained separately for the sake of explanation, they can be combined and implemented to the extent possible. In addition, we intend to obtain rights to any of the technologies described in this specification through amendments or divisional applications.

[0164] Furthermore, although the embodiments described above used HDMI as an example of the device connection format, other connection formats may also be used. For example, at the time of filing, DVI, VGA, DisplayPort, etc., may be used as connection formats that can utilize EDID, which indicates display capability. Also, although the embodiments described above use EDID as information indicating display capability, other information may be used as long as it indicates display capability. In addition, although two HDMI standards, the first standard and the second standard, have been used as examples in the explanation, three or more standards may be mixed. Even in this case, for example, the device that outputs video will be able to use the highest resolution among the resolutions that the device that displays video can output.

[0165] Furthermore, in each embodiment, the program that operates in each device is a program that controls the CPU and other components (a program that makes the computer function) in order to realize the functions of the embodiments described above. The information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) during processing, and then stored in various ROMs or HDDs, and read, modified, and written by the CPU as needed.

[0166] Here, the recording medium for storing the program may be any of the following: semiconductor media (e.g., ROM or non-volatile memory card), optical recording medium or magneto-optical recording medium (e.g., DVD (Digital Versatile Disc), CD (Compact Disc), BD (Blu-ray® Disc)), magnetic recording medium (e.g., magnetic tape, flexible disk), etc.

[0167] Furthermore, when distributing the program to the market, it can be stored on a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server device is, of course, also included in this disclosure.

[0168] Furthermore, the data mentioned above may not be stored within the device itself, but rather stored on an external device and retrieved as needed. For example, the data may be stored on a NAS (Network Attached Storage) or on the cloud.

[0169] Furthermore, the scope of this disclosure is not limited to the configurations explicitly described in the specification, but also includes combinations of the technologies disclosed herein. While the configurations for which patent protection is sought are described in the attached claims, there is no intention to exclude them from the technical scope simply because they are not described in the claims.

[0170] Furthermore, the phrases "in the case of..." and "when..." in the above-mentioned specification are explained as examples only, and do not represent a configuration limited to those described. Even for configurations other than those described, we disclose information that would be obvious to a person skilled in the art, and we intend to acquire rights to such information.

[0171] Furthermore, the descriptions of the processes and data flows described in the specification are not limited to the order in which they are described. For example, configurations in which parts of the process are deleted or the order is rearranged are also disclosed, and the company intends to acquire rights to them.

[0172] Furthermore, although the functions described in the embodiments are described as being performed by each device, they may also be implemented by a single device or by utilizing an external server. For example, they may be implemented as a standalone device, such as an electronic device including a set-top box. In this case, the electronic device may have at least some of the functions shown in Figures 1 to 4 or Figure 12, with the other functions provided elsewhere. Alternatively, the electronic device may have all of these functions. In particular, if there are multiple language models, all of the multiple language models may be provided by the electronic device, or at least some may be provided by the electronic device, with the other language models provided elsewhere. Furthermore, the display of messages in the embodiments described above may be selectable by the user. Similarly, the display of avatar speech may be selectable by the user.

[0173] Furthermore, the display screen described in the embodiments and drawings above is merely an example, and does not limit the displayed items, content, or arrangement. For example, the items displayed on the display screen can be rearranged, or the number of items can be changed. Also, the example display screen only shows a portion of the functions that can actually be implemented for illustrative purposes, and additional necessary information and items may be displayed, or conversely, items may be omitted as needed.

[0174] Furthermore, each functional block or feature of the apparatus used in the embodiments described above may be implemented or executed by an electrical circuit, such as an integrated circuit or a plurality of integrated circuits. An electrical circuit designed to perform the functions described herein may include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gates or transistor logic, discrete hardware components, or a combination thereof. The general-purpose processor may be a microprocessor, a conventional processor, controller, microcontroller, or state machine. The aforementioned electrical circuit may consist of digital circuits or analog circuits. Also, if advances in semiconductor technology lead to the emergence of integrated circuit technologies that replace current integrated circuits, one or more aspects of this disclosure may use new integrated circuits based on such technologies. [Explanation of Symbols]

[0175] 1 System 10 Display device 12 cameras 14 Mike 20 Server Devices 22 Large-scale language models 30 broadcasting stations 40 Distribution device 100 Control Unit 110 Storage section 110A ROM (Read Only Memory) 110B RAM (Random Access Memory) 110C Storage 120 Broadcast Control Unit 130 Operation section 140 Display section 150 Voice input section 160 Audio output section 170 Photography Department 180 Communications Department 1002 Content Acquisition Unit 1004 Content Playback Unit 1006 Content Image Output Unit 1008 Content audio output section 1010 Content Recognition Unit 1022 User Recognition Unit 1024 Voice Recognition Unit 1030 Conversation Processing Unit 1032 Prompt generation unit 1034 Output data acquisition unit 1040 Avatar Output Control Unit 1042 Avatar Image Generation Unit 1044 Avatar Voice Generation Unit 1046 Avatar Image Output Unit 1048 Avatar Voice Output Unit 1050 Conversation history storage processing unit 1102 Language Models 1104 User information storage area 1106 Content information storage area 1108 Avatar Information Storage Area 1110 Conversation history information storage area 1112 Similar Word Database (DB) 1114 Pronunciation Database (DB, Database) NW Network U User

Claims

1. A voice input unit that accepts voice input from the user, A recognition unit that recognizes an input sentence from the aforementioned voice input, A prompt generation unit that generates a prompt from the aforementioned input sentence, An output data acquisition unit inputs the aforementioned prompt to the language model and acquires output data, An output unit that generates and outputs a response statement from the aforementioned output data, Equipped with, The prompt generation unit, If there is only one candidate sentence generated from the input sentence, a prompt is generated that returns a response sentence in response to that candidate sentence. If multiple candidate sentences are generated from the input sentence, a prompt is generated that returns a response sentence instructing the user to select the correct candidate sentence from among the multiple candidate sentences. Processing device.

2. If no candidate sentence is generated, the prompt generation unit generates a prompt that returns a response sentence prompting the user to enter an input sentence again. The apparatus according to claim 1.

3. It further includes a similar word memory unit that stores similar word relationships, which associate auditory similar words with each other. Based on the similar word relationships stored in the similar word memory unit, similar words corresponding to the words extracted from the input sentence are obtained. Based on the extracted words and the obtained similar words, one or more candidate sentences are generated. The apparatus according to claim 1.

4. The system further includes a conversation history information storage unit that stores the input sentence and the response sentence as a conversation history. The prompt generation unit generates a prompt by excluding candidate sentences that are deemed invalid based on the conversation history stored in the conversation history information storage unit. The apparatus according to claim 1.

5. The prompt generation unit generates a prompt by excluding candidate sentences that are deemed invalid from among the multiple candidate sentences, based on the user's attributes. The apparatus according to claim 1.

6. The prompt generation unit generates a prompt by performing natural language processing on the candidate sentences, thereby modifying or adding to the candidate sentences. The apparatus according to claim 1.

7. The processing apparatus according to claim 1, wherein the recognition unit generates the candidate sentence from the input sentence.

8. The processing apparatus according to claim 1, wherein the prompt generation unit generates the candidate sentence from the input sentence.

9. The processing apparatus according to claim 1, wherein the output unit adjusts the reading of the response text as output data.

10. The system further includes a display control unit that controls the display of a display screen having a display area capable of displaying an avatar corresponding to a character, The output unit outputs the response sentence from the avatar. The apparatus according to claim 1.

11. Accepts user voice input, The input text is recognized from the aforementioned voice input. A prompt is generated from the aforementioned input statement, The aforementioned prompt is input to the language model, and the output data is obtained. From the aforementioned output data, a response statement is generated and output. In a control method for a processing device, If there is only one candidate sentence generated from the input sentence, a prompt is generated that returns a response sentence in response to that candidate sentence. If multiple candidate sentences are generated from the input sentence, a prompt is generated that returns a response sentence instructing the user to select the correct candidate sentence from among the multiple candidate sentences. Processing method.

12. A function that accepts user voice input, The function of recognizing input text from the aforementioned voice input, A function to generate a prompt from the aforementioned input sentence, The function inputs the aforementioned prompt into the language model and retrieves the output data, The function generates and outputs a response statement from the aforementioned output data, When there is only one candidate sentence generated from the input sentence, the function generates a prompt that returns a response sentence in response to that candidate sentence, When multiple candidate sentences are generated from the input sentence, the function generates a prompt that returns a response sentence allowing the user to select the correct candidate sentence from among the multiple candidate sentences. A program that implements this on a computer.

Images