Processing device, processing method, and program

Abstract

The present invention is provided with: a display control unit that controls display of a display screen having a display region in which an avatar corresponding to a character can be displayed; a voice input unit that receives voice input from a user; a recognition unit that recognizes an input sentence from content or the voice input; a prompt generation unit that generates a prompt from the input sentence; an output data acquisition unit that inputs the prompt to a first language model and acquires output data; an output unit that generates a response sentence from the output data and outputs the response sentence from the avatar; a storage unit that stores a combination of the input sentence and the response sentence as conversation history information; and a summary information acquisition unit that inputs the conversation history information to a second language model and acquires summary information. The prompt generation unit generates the prompt according to the input sentence and the summary information. Thus, it is possible to provide a processing device or the like capable of achieving a response suitable for a request from the user.

Description

Processing device, processing method, and program

[0001] The present disclosure relates to a processing device, etc. This application is based on Japanese Patent Application No. 2024-152312 filed in Japan on September 4, 2024, and the contents of that application are incorporated herein by reference.

[0002] For example, as disclosed in Patent Document 1, a metaverse is known in which users can converse with each other.

[0003] JP 2024-035786 A

[0004] An object of the present disclosure is to provide, for example, a processing device or the like that can realize a response suited to a user's request.

[0005] The processing device of the present disclosure includes a display control unit that controls the display of a display screen having a display area capable of displaying an avatar corresponding to a character; a voice input unit that accepts a user's voice input; a recognition unit that recognizes an input sentence from content or the voice input; a prompt generation unit that generates a prompt from the input sentence; an output data acquisition unit that inputs the prompt into a first language model and acquires output data; an output unit that generates a response sentence from the output data and outputs it from the avatar; a memory unit that stores a combination of the input sentence and the response sentence as conversation history information; and a summary information acquisition unit that inputs the conversation history information into a second language model and acquires summary information, wherein the prompt generation unit generates the prompt corresponding to the input sentence and the summary information.

[0006] The processing method in the processing device of the present disclosure is a processing method in a processing device capable of controlling the display of a display screen having a display area in which an avatar corresponding to a character can be displayed, and includes a voice input step of accepting voice input from a user, a recognition step of recognizing an input sentence from content or the voice input, a prompt generation step of generating a prompt from the input sentence, an output data acquisition step of inputting the prompt into a first language model and acquiring output data, an output step of generating a response sentence from the output data and outputting it from the avatar, a storage step of storing a combination of the input sentence and the response sentence as conversation history information, and a summary information acquisition step of inputting the conversation history information into a second language model and acquiring summary information, wherein the prompt generation step includes a step of generating the prompt corresponding to the input sentence and the summary information.

[0007] The program of the present disclosure provides a computer capable of controlling the display of a display screen having a display area capable of displaying an avatar corresponding to a character, the computer having a voice input function for accepting voice input from a user, a recognition function for recognizing an input sentence from content or the voice input, a prompt generation function for generating a prompt from the input sentence, an output data acquisition function for inputting the prompt into a first language model and acquiring output data, an output function for generating a response sentence from the output data and outputting it from the avatar, a memory function for storing a combination of the input sentence and the response sentence as conversation history information, and a summary information acquisition function for inputting the conversation history information into a second language model and acquiring summary information, and the prompt generation function realizes the function of generating the prompt corresponding to the input sentence and the summary information.

[0008] According to the present disclosure, for example, it is possible to provide a processing device or the like that can realize a response suited to a user's request.

[0009] FIG. 1 is a diagram for explaining an overview of a system in a first embodiment. FIG. 2 is a diagram for explaining the hardware configuration of a display device in the first embodiment. FIG. 3 is a diagram for explaining the software configuration in the first embodiment. FIG. 4 is a diagram for explaining the configuration of a storage unit in the first embodiment. FIG. 5 is a diagram for explaining a basic processing flow in the first embodiment. FIG. 6 is a diagram for explaining (a) an example of a display screen, (b) an example of a prompt, and (c) an example of output data in the first embodiment. FIG. 7 is a diagram for explaining (a) an example of conversation history information and (b) an example of summary information in the first embodiment. FIG. 8 is a diagram for explaining an example of a display screen as an operation example in the first embodiment. FIG. 9 is a diagram for explaining the software configuration in a second embodiment. FIG. 10 is a diagram for explaining the processing flow in the second embodiment. FIG. 11 is a diagram for explaining an example of a display screen as an operation example in the second embodiment. FIG. 12 is a diagram for explaining the software configuration in a third embodiment. FIG. 13 is a diagram for explaining the processing flow in the third embodiment. FIG. 14 is a diagram for explaining an example of a display screen as an operation example in the third embodiment. FIG. 15 is a diagram for explaining the processing flow in the third embodiment. FIG. 16 is a diagram for explaining an example of a display screen as an operation example in the third embodiment. FIG. 17 is a diagram for explaining the processing flow in a fourth embodiment. FIG. 18 is a diagram for explaining an example of a display screen as an operation example in the fourth embodiment.

[0010] Generally, in a metaverse where users can converse with each other, a process for estimating the relationship between the users may involve, for example, extracting users whose intimacy with the estimated user is above a certain level.

[0011] Here, the intimacy level of a user is determined by, for example, a conversation history learned using LLMs (Large Language Models; hereinafter, sometimes referred to as large-scale language models), which learns the frequency and volume of conversations, etc. Such large-scale language models are trained by machine learning using training data in which, for example, behavioral histories of multiple users up to a certain period and attribute information for that period are input, and behavioral histories for a subsequent period and attribute information for that period are output.

[0012] However, when a large-scale language model is generally used to estimate relationships between users, there are problems such as the impracticality of storing all past conversations in the large-scale language model from the viewpoint of resources, etc. On the other hand, even if only past conversations are stored, there are also problems such as (1) results that do not necessarily cover all past conversations, (2) simply storing conversation history does not provide a sufficient function for collecting information on the speaker's preferences, and (3) external information cannot be reflected in the conversation.

[0013] As described above, the processing device and the like of the present disclosure that solve one or more of the above-described problems will be described in the following embodiments with reference to the drawings. Note that the following embodiments describe the invention set forth in the claims as an example, and the technical scope of the present invention is not limited to the description of the following embodiments. Furthermore, in the following embodiments, a case where the processing device of the present disclosure is applied to a display device will be described, but the processing device is not limited to a display device. For example, the processing device may be a standalone set-top box (STB) with a tuner function, or may be applied to a recording device using a hard disk, a display control device that plays back a recording medium, a projector, etc. Furthermore, the processing device may be an information processing device such as a smartphone, tablet, or computer used by a user as a device capable of displaying content, or may be an in-vehicle device such as a car navigation system.

[0014] [1. First Embodiment] [1.1 About the System] [1.1.1 Overview of the System] Fig. 1 is a diagram illustrating an overview of a system 1. The system 1 has a display device 10 and is connected to a network NW. The display device 10 may also be capable of recognizing a user U who is in the vicinity of the device, for example. The display device 10 may have a built-in camera 12 and a microphone 14, and may recognize a user U who is in front of the display device 10.

[0015] The system 1 may also include a server device 20. The server device 20 is connected to, for example, a network NW. The server device 20 may also store a large-scale language model 22 as a language model.

[0016] The large-scale language model 22 is one type of language model and is generally referred to as an LLM. The large-scale language model 22 is stored in the server device 20 as a first language model, but may also be stored in the display device 10. The display device 10 may store a small-scale language model (SLM), for example, as an edge LLM, different from the large-scale language model 22. That is, as a language model, a language model (LLM) with a large amount of training data may be stored in the server device 20, and a language model (SLM) with less training data than the large-scale language model 22 may be stored in the display device 10 or another device as a second language model. In the following specification, a language model stored in the display device 10 or the like may be referred to as an edge language model.

[0017] The large-scale language model 22 may be prepared by the service provider that builds the system 1, or an external service may be used. For example, large-scale language models such as GPT (Generative Pre-trained Transformers, GPT-3, GPT-4, GPT-4o), PaLM (Pathways Language Model), LLaMA (Large Language Model Meta AI), and Tsuzumi may be used.

[0018] The display device 10 can also acquire and display content. The display device 10 may acquire the content from a broadcast station 30, from a distribution device 40, or from a recording medium. The broadcast station 30 may also transmit the content to the display device 10 via a network NW. The broadcast station 30 transmits the content to the display device 10 using, for example, terrestrial digital broadcasting, BS broadcasting, or CS broadcasting. The broadcast station 30 may also transmit the content via the network NW using IP multicast broadcasting.

[0019] In this embodiment, a response sentence is output using an "avatar." However, for example, the response sentence may be displayed on the display screen of the display device 10 using a language model without using an avatar, or the display device 10 may read the response sentence aloud. Furthermore, the display device 10 may be a display system including a device that simply displays a video signal from an external device and a device that outputs the video signal. For example, the display device 10 may include a system including a display and a recording and / or playback device connected via HDMI (registered trademark).

[0020] [1.1.2 Terminology] Hereinafter, terms used in this specification will be used in a way that indicates a meaning that can be understood by a person skilled in the art, for example, as follows.

[0021] An "avatar" is an object that is represented based on a character. It can be selected or created primarily by the user. The avatar may correspond to a specific character, and may represent, for example, a virtual friend, secretary, or advisor, who can be used as a conversation partner. Although it is called an avatar, it does not have to be an alter ego of the user.

[0022] "Conversation" refers to the exchange of sentences, and in this embodiment, particularly refers to the exchange and combination of input sentences and response sentences. Here, an input sentence is a sentence based on the voice uttered by the user. A response sentence is a sentence output or spoken by an avatar. These sentences are also sometimes called messages. An avatar uttering a sentence (response sentence) means, for example, that the control unit displays a sentence (message) or outputs the sentence (message) by voice. In this case, by linking with the display of the avatar, the user can interact with the avatar as if it were actually speaking to them.

[0023] A "prompt" is information for instructing a language model (large-scale language model, small-scale language model) to generate output data including a response sentence.

[0024] "User state" refers to the state of the user, and may refer to, for example, the state of the user when in front of the display device 10, but preferably refers to the state of the user when viewing content. For example, the user state may include the user's facial expression (laughing, surprised, crying, etc.), gaze direction, and face direction that can be recognized from an image. The user state may also include the user's emotion (wow, amazing, I want, etc.) that can be recognized from voice. The user state may also include the user's age, gender, and number of users.

[0025] "Content" includes program content broadcast from broadcast station 30 and distribution content distributed from distribution device 40. Content may also be content recorded on a recording medium (e.g., Blu-ray (registered trademark) or the like). Content includes video including one or more images and audio, and may also include explanatory information, attributes, and subtitle information related to the content. While the content is mainly described using video content as an example, it may also include still images, audio, text, web pages, and the like as needed. For example, video content includes all kinds of images, such as entertainment, education, news, e-commerce (electronic commerce), accommodation information, and travel arrangements.

[0026] [1.2 Hardware Configuration] Next, the hardware configuration of the devices in this embodiment will be described. The display device 10, the server device 20, and the distribution device 40 each include at least a control unit, a storage unit, and a communication unit. Each device also has general functions as needed. Figure 2 describes the hardware configuration of the display device 10.

[0027] The control unit 100 controls the entire display device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the memory unit 110 (e.g., ROM 110A, storage 110C). The control unit 100 may be realized by one or more control devices / arithmetic units (CPUs (Central Processing Units), SoCs (System on a Chip)). The control unit 100 may also be configured by a control circuit.

[0028] The storage unit 110 is one or more storage devices that store necessary data and programs. The storage unit 110 stores data and programs temporarily or non-temporarily. For example, the storage unit 110 includes a ROM 110A, a RAM 110B, and a storage 110C.

[0029] The ROM 110A is a non-volatile memory that can retain programs and data even when the power is turned off.

[0030] The RAM 110B is a main memory that is mainly used by the control unit 100 when executing processes. The RAM 110B is a rewritable memory that temporarily stores programs read from the ROM 110A or the storage 110C, and data including execution results.

[0031] The storage 110C is a non-volatile storage device capable of storing programs and data. For example, the storage 110C may be configured as a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 110C may also be configured as an externally connectable USB memory. The storage 110C may also be, for example, a storage area on the cloud.

[0032] The broadcast control unit 120 receives broadcast waves transmitted by a broadcast station selected by the user, decodes video data from the broadcast waves and outputs the data to the display unit 140, and decodes audio data from the broadcast waves and outputs the data to the audio output unit 160. The broadcast control unit 120 may be configured with, for example, a digital tuner unit (terrestrial / BS / CS, etc.), an OFDM demodulation unit, a DEMUX unit, an MPEG2 decoding unit, etc. Furthermore, the broadcast control unit 120 may have multiple units.

[0033] The operation unit 130 receives operations from the user, issues operation instructions to each functional unit, and notifies the control unit 100 of an operation signal corresponding to the received operation. For example, the operation unit 130 receives an operation signal from a remote control or the like, and operates the display device 10 in accordance with the received operation signal. The operation unit 130 may also perform control based on an operation received using, for example, an operation button provided on the display device 10 or a software key using a touch panel.

[0034] The display unit 140 is a display device capable of displaying images of received programs and various information. The display unit 140 may be, for example, a device capable of displaying images, such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display unit 140 may also be, for example, a projection device such as a projector.

[0035] The audio input unit 150 is an input device that can input sounds from the surrounding area where the display device 10 is installed, such as a microphone. The audio input unit 150 may also be composed of multiple input devices (for example, a microphone array). The audio input unit 150 is primarily used to input the audio of the user watching, but can also input general sounds such as environmental sounds. The display device 10 can recognize the state of the user from the audio input from the audio input unit 150.

[0036] The audio output unit 160 outputs audio included in the content. The audio output unit 160 may be, for example, a device such as a speaker or headphones. The audio output unit 160 may output any sound, and may output general sounds such as music, environmental sounds, etc. A plurality of audio output units 160 may be provided.

[0037] The image capturing unit 170 is an image capturing device that captures images of the surroundings around the display device 10, such as a camera capable of capturing distance images or the three-dimensional shape of the captured object. The image capturing unit 170 may be composed of one or more image capturing devices. The image capturing unit 170 outputs the captured images as image signals. Alternatively, the image capturing unit 170 may output one or more images as a continuous video. The display device 10 can recognize the user's condition from the images captured by the image capturing unit 170. Alternatively, a sensor using infrared rays, millimeter waves, or the like may be provided instead of or in addition to the image capturing unit 170. In this case, the display device 10 can acquire the user's condition, such as body temperature or heart rate, using the infrared rays, millimeter waves, or the like.

[0038] The communication unit 180 is a communication interface for communicating with other devices. For example, the communication unit 180 may be a network interface connectable to a wireless LAN or a network interface connectable to Ethernet (registered trademark) via a wired connection. The communication unit 180 may also be a communication device connectable to a mobile communication network such as LTE / 4G / 5G / 6G.

[0039] The sensor unit 190 is a variety of sensors that acquire information (environmental information) about the user's environment, such as the date and time, the room environment, and weather information.

[0040] 2 may include any necessary components in the embodiment. For example, if the user's state is not acquired from a camera, the image capturing unit 170 may not be included.

[0041] The display device 10 may also be connected to an external device via an interface unit, and the above-described functions may be realized by the external device. For example, the image capturing unit 170 may be realized by a camera device connected via a USB interface.

[0042] [1.3 Software Configuration] [1.3.1 Configuration of Control Unit] Fig. 3 is a diagram illustrating the software configuration, focusing on the configuration of the control unit. For example, the components in Fig. 3 are realized by control unit 100 executing a program stored in storage unit 110. Control unit 100 also functions as a display control unit that displays content, avatars, etc. on display unit 140. The display control unit may include, for example, the functions of content image output unit 1006 and avatar image output unit 1046.

[0043] (Content) The content acquisition unit 1002 acquires content from the broadcast control unit 120 or the communication unit 180. The content acquisition unit 1002 may also acquire stored content from the storage unit 110.

[0044] The content playback unit 1004 plays (displays) the content acquired by the content acquisition unit 1002. The content playback unit 1004, for example, separates multiplexed image data from the content and outputs it to the content image output unit 1006. The content playback unit 1004 also separates multiplexed audio data from the content and outputs it to the content audio output unit 1008. Any known method may be used to separate the image data (image signal) and audio data (audio signal) from the content.

[0045] The content image output unit 1006 outputs image data included in the content as an image (or video as a series of images) to the display unit 140. Here, the display unit 140 displays the image of the content as a first image.

[0046] Furthermore, the content audio output unit 1008 outputs audio data included in the content as audio to the audio output unit 160. Here, the audio output unit 160 outputs the audio of the content as the first audio.

[0047] Furthermore, the content recognition unit 1010 recognizes image data and audio data contained in the content, as well as content information contained in the content (for example, program information, subtitle information, etc. in the case of broadcast waves), and outputs the information to the prompt generation unit 1032. The information related to the content recognized by the content recognition unit 1010 is called content information.

[0048] (User Recognition) The user recognition unit 1022 recognizes a user who is a viewer of the content and recognizes information about the user based on the image captured by the image capturing unit 170. Information about a user is called user information, and the user recognition unit 1022 can recognize the following content as user information.

[0049] (1) Number of users. For example, the user recognition unit 1022 recognizes the number of viewers of the content. The user recognition unit 1022 may also recognize the number of viewers of the content by using the direction of the face and the direction of the line of sight.

[0050] (2) Identifying a user. For example, the user recognition unit 1022 recognizes who is viewing the content. The user recognition unit 1022 can identify a user by acquiring an image of the user's face (face image) and matching it with face images of users registered in advance.

[0051] (3) User's Facial Expression For example, the user recognition unit 1022 can recognize the user's facial expression (facial expression).

[0052] The speech recognition unit 1024 recognizes a conversation (input sentence) uttered by the user based on the speech input by the speech input unit 150. The speech recognition unit 1024 outputs the input sentence to the conversation processing unit 1030.

[0053] Furthermore, the voice recognition unit 1024 may recognize the user's emotions (for example, exclamations, clapping sounds, etc.) as user information.

[0054] That is, the user recognition unit 1022 can recognize user information based on an image, and the voice recognition unit 1024 can recognize user information based on a voice.

[0055] The conversation processing unit 1030 outputs the input sentence, summary information based on the conversation history up to now (hereinafter sometimes simply referred to as summary information), and any other necessary accompanying information to the large-scale language model 22, and obtains output data including a response sentence from the large-scale language model 22. In this way, the conversation processing unit 1030 can realize a conversation using the input sentence entered by the user and the response sentence output by the avatar. Furthermore, because the conversation processing unit 1030 outputs summary information based on the conversation history up to now to the large-scale language model 22, it has the following advantages: (1) it can obtain results that cover the entire past conversation, (2) it can collect information about the speaker's preferences, and (3) it can reflect external information in the conversation.

[0056] More specifically, the conversation processing unit 1030 includes a prompt generating unit 1032 and an output data acquiring unit 1034 .

[0057] The prompt generation unit 1032 generates a prompt to be input into a language model (for example, the large-scale language model 22 or the edge language model 1060) based on the input information. The prompt may be in a natural sentence format or may be in a predetermined format corresponding to the language model 1102 using tags, etc. The prompt generation unit 1032 generates a prompt based on necessary information from the input sentence, summary information, and accompanying information. Note that, hereinafter, information other than that related to the conversation may be referred to as accompanying information.

[0058] Here, the accompanying information other than the input sentence and summary information may include, for example, the following information.

[0059] Content information, which may include, for example, the name of the content, details analyzed from the image of the content, details recognized from the audio of the content, etc.

[0060] User information: For example, in addition to basic information about users (age, sex, number of people, etc.), information about the user's facial expression, user's movements, user's interests, etc. may be included.

[0061] Information about the environment (environment information) For example, information about the user's environment, such as the date and time, the room environment, and weather information, may be included.

[0062] Product information. For example, the information may include one or more pieces of information about products on a specific shopping site. In addition to basic information such as the product name and price, the product information may also include information such as product features, weight, size, and product images that can be obtained from the product sales site.

[0063] The prompt generation unit 1032 may generate a prompt to be input to the large-scale language model 22, for example, by inputting the above-mentioned information once into an edge LLM.

[0064] The conversation processing unit 1030 (prompt generation unit 1032) inputs a prompt including an input sentence as an example of a language model to the large-scale language model 22 of the server device 20. The large-scale language model 22 outputs output data to the conversation processing unit 1030 in accordance with the input prompt.

[0065] The conversation processing unit 1030 (output data acquisition unit 1034) acquires output data from the large-scale language model 22. Here, the output data may include the following information.

[0066] Response sentences responding to input sentences Product recommendation information The conversation processing unit 1030 outputs, for example, the input sentence included in the prompt and the response sentence included in the output data as a conversation to the conversation history storage processing unit 1050. The conversation history storage processing unit 1050 stores the conversation including the input sentence and the response sentence, and information about the conversation, as conversation history information in the conversation history information storage area 1110.

[0067] Furthermore, the conversation processing unit 1030 inputs the conversation history information stored by the conversation history storage processing unit 1050 as spoken text to the edge language model 1060, thereby obtaining summary information based on the conversation history information as output data. The conversation history storage processing unit 1050 stores the summary information obtained as output data.

[0068] Here, the edge language model 1060 is a language model (SLM) with less training data than the large-scale language model 22, which performs preprocessing such as automatically or manually dividing or extracting blocks from utterance text based on input conversation history information, or structuring based on the conversation structure (e.g., topic, argument, reason, etc.), and generates summary information using extraction-type summarization that extracts and summarizes important parts of the conversation, or abstract summarization that succinctly rephrases the conversation (summarizing the main points). Note that the edge language model 1060 may perform morphological analysis, dependency analysis, etc. on the utterance text before dividing or structuring. Furthermore, the edge language model 1060 may generate summary information of the conversation history, using the period from the start of viewing content to the end of the content as a single unit, or may generate summary information of the conversation history in chronological order by date, or may generate summary information for each conversation structure, such as the topic of the conversation.

[0069] Conversation processing unit 1030 outputs the conversation to avatar output control unit 1040. Conversation processing unit 1030 mainly outputs the response sentence included in the output data and information about the avatar included in the output data (e.g., the facial expression of the avatar) to avatar output control unit 1040.

[0070] Avatar output control unit 1040 functions as an avatar generation unit that generates an avatar. Avatar output control unit 1040 controls, for example, the facial expression, movement, and appearance of the avatar. Avatar output control unit 1040 also determines what the avatar will say. Avatar output control unit 1040 then outputs a signal to avatar image generation unit 1042 to generate an avatar image. Here, the avatar image may include not only an image of the avatar but also a message to be spoken by the avatar. The avatar image generated by avatar image generation unit 1042 is then output to display unit 140 via avatar image output unit 1046.

[0071] Furthermore, avatar output control unit 1040 outputs a voice signal for generating a voice to be spoken by the avatar and text data to avatar voice generation unit 1044. Avatar voice generation unit 1044 generates a voice of the avatar in accordance with the response sentence to be spoken by the avatar. Avatar voice output unit 1048 outputs the generated voice of the avatar via voice output unit 160.

[0072] Here, avatar image generation unit 1042 may obtain information about the sound emitted by the avatar from avatar sound generation unit 1044. Also, avatar image generation unit 1042 may obtain information about the sound emitted by the avatar from avatar output control unit 1040.

[0073] Here, the avatar image may change its facial expression or mouth position depending on the response sentence. For example, the response sentence may be received from the avatar image generation unit 1042 or the avatar voice generation unit 1044, and the display of the avatar may be changed depending on the response sentence.

[0074] The avatar image may be displayed superimposed on the content image, or may be displayed separately in a different area.

[0075] 3 are merely examples, and one or more components may be collectively executed by the control unit 100. For example, the user recognition unit 1022, the voice recognition unit 1024, and the content recognition unit 1010 may be combined into one recognition unit.

[0076] Furthermore, content image output unit 1006 and content audio output unit 1008 may be combined into one content output unit. Furthermore, avatar image output unit 1046 and avatar audio output unit 1048 may be combined into one avatar output unit. Furthermore, the content output unit and avatar output unit may be combined into one output unit.

[0077] [1.3.2 Configuration of Storage Unit] The configuration of the storage unit 110 will be described with reference to FIG.

[0078] The language model 1102 is a language model that can be used in the display device 10. The language model 1102 may be, for example, the edge language model 1060 or the large-scale language model 22.

[0079] The user information storage area 1104 stores user information, which is information about users. For example, the user who uses the display device 10 may be stored in advance as user information, or the user may be stored as a user each time the user is recognized. For example, the user information may store the user's name, full name, an icon representing the user, or an image representing the user as information for identifying the user (identification information). The user information may also store the user's age, generation, and gender.

[0080] The content information storage area 1106 stores content information, which is information related to content. The content information may store information about the content currently being played, or may store content information corresponding to content stored in the storage unit 110. The content information may include information such as the content name, the length (time) of the content, and the genre (attributes) of the content. The content information may also include the results of recognizing the image and audio of the content. For example, the content information may include a scene of the content recognized by the content recognition unit 1010 from the image of the content (information recognized from the image, information about objects such as people and objects contained in the image, and location information), words contained in the audio of the content recognized from the audio of the content, information about products introduced in the content, etc. The content information may also be stored in chronological order.

[0081] The avatar information storage area 1108 stores avatar information, which is information about an avatar. The avatar information may include, for example, the gender (male or female), age, face shape, body part arrangement, type of clothing, and type of accessories. The avatar information may also include attributes such as whether the avatar is a human, cat, dog, or anime character. The avatar information may also include information about the avatar's emotions and the sounds the avatar makes.

[0082] The conversation history information storage area 1110 stores conversation history information, which is information related to the history of a conversation. The conversation history information includes input sentences and response sentences as conversations. In addition to the conversation, the conversation history information may store one or more necessary pieces of related information, such as user information, content information, and environmental information. Furthermore, the conversation history information may store a conversation history from the start of viewing content to the end of the content as a single unit, or may store a conversation history in chronological order by date. Note that if a response sentence is not necessary in a conversation, a response sentence may not be stored. Furthermore, the unit or timing for storing the conversation history information may be arbitrarily set by the user.

[0083] The summary information storage area 1112 stores summary information generated by the edge language model 1060 based on the conversation history information.

[0084] [1.4 Overall Description] The overall flow of a conversation between a user and an avatar in this embodiment will now be described.

[0085] [1.4.1 Flow of conversation processing] Fig. 5 is a diagram illustrating the general flow of conversation processing. Note that, although it is preferable that the following processing be appropriately executed by one of the components described in Fig. 3, for the sake of convenience, it will be described as being executed by the control unit 100.

[0086] The control unit 100 receives the content (S10), recognizes the content, and acquires content information (S12).

[0087] When the control unit 100 detects that the user has spoken (S14; Yes), it recognizes the speech and creates an input sentence (S16). The control unit 100 also acquires necessary information from the user information and conversation history information (S18). Here, it is sufficient for the control unit 100 to acquire at least the information necessary for the processing described below.

[0088] The control unit 100 selects one or more necessary pieces of information from the input sentence, content information, user information, summary information, and other associated information, and generates a prompt (S18).The control unit 100 then inputs the generated prompt into the large-scale language model 22 (S20).

[0089] When the control unit 100 acquires output data corresponding to the prompt from the large-scale language model 22 (S22; Yes), it generates a response sentence from the output data (S24).

[0090] The control unit 100 then outputs a response sentence from the avatar (S26). After storing the conversation between the user and the avatar as conversation history information, the control unit 100 inputs the conversation history information into the edge language model 1060 to generate summary information (S28). Since the summary information is generated as the conversation history information is input, the summary information is updated chronologically, for example, by date.

[0091] Then, the control unit 100 stores the updated summary information in the summary information storage area 1112 (S30). The control unit 100 repeats the process until the conversation processing is completed (S32; No → S10).

[0092] 5 is a diagram illustrating an example of a conversation process between a user and an avatar, including an input sentence and a response sentence, and necessary processing is executed as appropriate depending on the process. For example, in a conversation process in which the content is not used, such as when using a shopping site, there is no need to receive the content or acquire the content information. For example, steps S10 and S12 in FIG. 5 do not need to be executed.

[0093] Furthermore, there is no need to receive content or acquire content information when processing conversations on the menu screen, setting screen, etc. In this case, S10 and S12 in FIG. 5 also do not need to be executed.

[0094] 5, the user initiates the speech, but the avatar may initiate the speech. For example, the control unit 100 may execute the process from the process of generating a prompt (S20). In this case, the avatar outputs a response sentence before the user initiates the speech.

[0095] 6A shows an example of a display screen W10. The display screen W10 has a first region R10 for displaying content, a second region R12 for displaying an avatar, and a third region R14 for displaying conversational text. The second region R12 and the third region R14 may be integrated into one region.

[0096] Furthermore, it is preferable that the conversational text be displayed so that the user's speech (input text) and the avatar's speech (response text) can be distinguished from each other. For example, in Fig. 6(a), the user's speech and the avatar's speech are displayed so that they can be distinguished from each other by the direction of the speech bubble. Furthermore, the user's speech and the avatar's speech may be displayed so that they can be distinguished from each other by, for example, the background color, the text color, the font, etc.

[0097] Furthermore, the display screen W10 in FIG. 6A displays a first region R10, a second region R12, and a third region R14, but these display regions can be switched. For example, a user may enable only the second region R12 on the display screen W10 to display only an avatar. In this case, the avatar may be adjusted to a size appropriate for the screen size of the display screen W10. Similarly, a user may enable only the third region R14 on the display screen W10 to display only the dialogue. In this case, the dialogue may be adjusted to a size appropriate for the screen size of the display screen W10, and the font may be enlarged, or more dialogue than usual may be displayed. Furthermore, multiple regions, such as the first region R10 and the third region R14, or the second region R12 and the third region R14, may be selected for display. In the following display screens, the avatar, content, and dialogue may each be selectively displayed.

[0098] For convenience of explanation, the avatars, content, and conversations have been described as switching display areas. However, the control unit 100 may switch the displayed content itself between displayed and hidden while leaving the display area unchanged. The display area also includes a display screen. The control unit 100 may output the content displayed in each display area (display screen) to a different display. For example, the content displayed in the first display area R10 may be displayed on the display unit 140 of the display device 10, and the avatars and conversations displayed in the second display area R12 and the third display area R14 may be output to an external display or another terminal device (e.g., a smartphone) connected via the communication unit 180.

[0099] Fig. 6(b) is a diagram showing an example of a prompt to be input to a language model (e.g., the large-scale language model 22) in natural language, and Fig. 6(c) is a diagram showing an example of output data output from the language model (e.g., the large-scale language model 22) in natural language.

[0100] The prompt preferably includes an input sentence. For example, FIG. 6B includes the input sentence, "Where is Tenri?" The prompt may also include one or more pieces of information, such as a conversation history, summary information, content information, user information, interest information, and product information. The prompt may also include such information that has been optimized by the Edge LLM.

[0101] The output data preferably includes a response sentence. For example, Figure 6(c) includes the response sentence "There are probably a lot of tourists in the northern part of Nara Prefecture." The output data may also include one or more pieces of information such as avatar information (such as the avatar's facial expression), recommendation information, site-related information, and accompanying information.

[0102] 7A is a diagram illustrating an example of a conversation history between a user and an avatar in response to a response sentence (utterance) by the avatar illustrated in FIG. 6C , "I wonder if there are many tourists in the northern part of Nara Prefecture." For example, in response to the avatar's utterance, "I wonder if there are many tourists in the northern part of Nara Prefecture." (Utterance 1), the user utters, "Um, where are the famous tourist spots?" (Utterance 2). In response to Utterance 2, the avatar replies, "The Yamanobe Road is famous. It's the oldest road in Japan, and it's home to numerous cultural assets and historical sites, as well as many ancient tombs." (Utterance 3), and the user utters, "Oh, I didn't know that." (Utterance 4).

[0103] Utterance 5, "The Tenri Factory was established in Tenri City, Nara Prefecture in 1970, the year the Osaka Expo was held...," is an example of audio output of the content.

[0104] Following utterance 5, the user asks, "Is it close to Osaka or Kyoto?" (utterance 6), and in response the avatar utters, "It takes about an hour and a half by train." (utterance 7), to which the user responds, "Hmm, I'd like to go there someday." (utterance 8). Furthermore, the user utters, "What local specialties do they have?" (utterance 9), and in response to utterance 9 the avatar utters, "Strawberries and persimmons, and I'd recommend local sake." (utterance 10).

[0105] In FIG. 7( a), for ease of understanding, the utterances in the conversation history information are displayed divided based on the "topic," such as "utterance 1 to utterance 4," "utterance 5," "utterances 6 to 8," "utterances 9 to 10," etc. However, the conversation history information input to the edge language model 1060 may be utterance text consisting of a series of character strings, and there are no restrictions on the input format.

[0106] Fig. 7(b) is a diagram illustrating an example of summary information generated by the edge language model 1060 based on the conversation history information (utterance text) illustrated in Fig. 7(a). Note that Fig. 7(b) illustrates an example of generating summary information by dividing the utterances in the conversation history information illustrated in Fig. 7(a) based on "topics," which is considered to be blocking. However, when performing blocking, there are no particular limitations on the method used, and blocking may be performed based on, for example, "important parts," "fixed length," or "time series."

[0107] In FIG. 7B, Summary 1 is an example of a summary of "Utterance 1 to Utterance 4." In Summary 1, the utterances related to "Utterance 1 to Utterance 4" are summarized as "Location and Characteristics of Tenri City," and the auxiliary information for Summary 1 is "Northern Nara Prefecture, Yamanobe-no-michi is a famous tourist destination," as indicated by the arrow after the arrow in the figure. The auxiliary information is information that supplements the summary information, and may be content included in the utterance, content learned through prior learning, or information collected from the World Wide Web via the network NW. The edge language model 1060 can output summary information with the auxiliary information added as summary information, or it can output only the summary information or only the auxiliary information.

[0108] Summary 2 is an example of a summary of "Utterance 5." In Summary 2, the utterance related to "Utterance 5" is summarized as "Introduction to the Tenri Factory," and the supplementary information for Summary 2 is "Established in Tenri City, Nara Prefecture in 1970," which is shown after the arrow in the figure.

[0109] Summary 3 is an example of a summary of "Utterance 6 to Utterance 8." In Summary 3, the utterances related to "Utterance 6 to Utterance 8" are summarized as "Access to Tenri City," and supplementary information for Summary 3 is "1 hour and 30 minutes by train from Osaka or Kyoto," which is indicated by the arrow after the figure.

[0110] Summary 4 is an example of a summary of "Utterances 9 to 10." In Summary 4, the utterances related to "Utterances 9 to 10" are summarized as "Tenri City's Specialties," and supplementary information for Summary 4 includes "strawberries, persimmons, and local sake," which are indicated by the arrows in the figure.

[0111] As illustrated in FIG. 7( b), summary information can be generated using either an extraction-type summary or an abstract-type summary. In particular, abstract-type summaries can generate more accurate summary information by utilizing, for example, a neural model such as an Encoder-Decoder model, or a language model such as BERT (Bidirectional Encoder Representations from Transformers) or BART (Bidirectional and Auto-Regressive Transformers), which are obtained by pre-learning conversation history information.

[0112] [1.5 Operation Example] An operation example will be described with reference to FIG.

[0113] FIG. 8 is a diagram showing an example of a display screen W100 that displays an avatar uttering a response sentence generated based on summary information of a conversation history.

[0114] The display screen W100 shows an avatar A100 uttering a response sentence. For example, the control unit 100 generates a prompt including summary information and inputs it into the large-scale language model 22, resulting in output data including a response sentence such as, "Those are the recommended products you mentioned while watching the Tenri video! I recommend the local sake." Here, the output data output from the large-scale language model 22 is output using the summary information exemplified in FIG. 7(b) as input data. The control unit 100 then displays the response sentence as a message M100. The control unit 100 may also output the response sentence as voice.

[0115] In addition, a region R100 displays a plurality of recommended products. This recommended information may be information obtained by inputting summary information into the large-scale language model 22 and acquiring recommended products from the large-scale language model 22.

[0116] In this way, the display device 10 of this embodiment summarizes and stores conversations between the user and the avatar, and reflects the content of the previous conversation with the avatar in the next conversation, allowing the user to obtain appropriate responses. Furthermore, since all conversations can be saved by summarizing, the user's requests can be reflected without omission in the avatar's responses. Furthermore, by processing the generation of summaries using so-called edge LLM, which does not require high-speed processing, the cost of using external resources such as cloud services is not required. Furthermore, as a secondary effect, the user can always obtain appropriate responses from the avatar, thereby improving the pseudo-intimacy between the user and the avatar.

[0117] [2. Second Embodiment] The second embodiment will be described. In the second embodiment, theme words related to a user's preferences along a specific theme are acquired from summary information, and the acquired theme words are used to generate a response sentence for an avatar. The configuration of the second embodiment is almost the same as the hardware configuration and software configuration of the first embodiment. Therefore, the second embodiment will be described mainly focusing on the differences from the first embodiment.

[0118] [2.1 Software Configuration] [2.1.1 Configuration of Control Unit] Fig. 9 is a diagram illustrating the software configuration, focusing on the configuration of the control unit. For example, the control unit 101 executes a program stored in the storage unit 110, thereby realizing each component in Fig. 9.

[0119] Here, the theme word acquisition unit 1070 acquires theme words related to the user's preferences from the summary information stored by the conversation history storage processing unit 1050. The theme words related to preferences are theme words that represent specific things that are the subject of the user's tastes and hobbies, and are also user information that differs for each user (hereinafter, theme words related to preferences may be simply referred to as theme words).

[0120] The theme word acquisition unit 1070 acquires the theme words from the summary information and outputs the acquired theme words to the prompt generation unit 1032. The prompt generation unit 1032 generates prompts to be input into the large-scale language model 22 based on the input information including the theme words.

[0121] [2.2 Processing Flow] The processing flow according to the second embodiment will be described. The processing described in Fig. 10 is preferably executed by the respective components described in Fig. 3, Fig. 9, etc., as appropriate, but for the sake of convenience, it will be described as being executed by the control unit 101.

[0122] The control unit 101 acquires the summary information stored by the conversation history storage processing unit 1050 (S40), and acquires theme words from the acquired summary information (S42).

[0123] Next, the control unit 101 inputs the input sentence, the history information, and a prompt including the theme word into the language model to obtain output data (S44). The output data may directly include the theme word, or may include keywords or the like that indirectly associate with the theme word.

[0124] The control unit 101 generates a response sentence related to the theme word from the output data (S46). In this case, the generated response sentence preferably includes content that prompts the user to speak about the theme word.

[0125] Then, the control unit 101 determines whether or not a user utterance related to the theme word has been acquired in response to the content of the displayed or uttered response sentence (S48).

[0126] If it is determined that the user's utterance has been acquired, the control unit 101 displays or speaks recommendation information related to the theme word, and ends the process (S48; Yes → S50). In this case, the control unit 101 acquires recommendation information related to the theme word from the output data output from the large-scale language model 22 based on the input sentence based on the user's utterance. On the other hand, if it is determined that the user's utterance has not been acquired, the control unit 101 waits until the user's utterance is acquired (S48; No).

[0127] [2.3 Operation Example] An operation example will be described with reference to FIG.

[0128] FIG. 11 is a diagram showing an example of a display screen W110 that displays an avatar that displays or speaks a response sentence related to a theme word acquired from summary information.

[0129] The display screen W110 shows an avatar A110 displaying or speaking a response sentence. The region R110 displays content related to, for example, "New smartphone from X company." For example, the control unit 101 generates a prompt including summary information and a theme word (e.g., smartphone) and inputs the generated prompt into the large-scale language model 22, obtaining output data including content that prompts the user to speak in response to the theme word.

[0130] For example, the control unit 101 displays or speaks a message M112 such as "It has a wide range of functions. Do you have a favorite manufacturer?" to encourage the user to speak about the theme word. For example, when the control unit 101 acquires a user utterance M114 related to the theme word, such as "The one I'm using now is made by △△ company," the control unit 101 displays or speaks recommendation information M116 such as "Then, let's watch a video together about the recently released smartphone made by △△ company."

[0131] In this embodiment, the theme word acquisition unit 1070 has been described as acquiring theme words related to the user's preferences from the summary information stored by the conversation history storage processing unit 1050, but it may also be configured to acquire them from the conversation content before it is summarized.

[0132] In this way, the display device 10 of this embodiment extracts information about the user's preferences based on the summary, and incorporates the extracted information into the conversation or uses it for recommendations. Furthermore, the display device 10 of this embodiment encourages utterances for acquiring information about preferences, i.e., conversations that draw out the user's preferences. This enhances the function of collecting information about the user's preferences compared to the conventional function of simply storing conversation history.

[0133] [3. Third Embodiment] The third embodiment will be described. In addition to the configuration of the first or second embodiment, the third embodiment uses images captured by the image capture unit 170 and environmental information acquired by the sensor unit 190 (hereinafter, sometimes referred to as external information) to generate a response sentence for the avatar. In particular, the third embodiment identifies an object that the user is holding or pointing at based on the external information, and uses the identification result in the avatar's response sentence (utterance). The configuration of the third embodiment is substantially the same as the hardware configuration and software configuration of the first or second embodiment. Therefore, the third embodiment will be described mainly with reference to the differences from the previously described embodiments.

[0134] [3.1 Software Configuration] [3.1.1 Configuration of Control Unit] Fig. 12 is a diagram illustrating the software configuration, focusing on the configuration of the control unit. For example, the control unit 102 executes a program stored in the storage unit 110, thereby realizing each component in Fig. 12.

[0135] Here, the image analysis unit 1080 can be constructed as an object detection model that recognizes objects in an image based on the image (video) captured by the imaging unit 170 and environmental information acquired as needed by the sensor unit 190, and detects the position, type, number, etc. of the recognized objects. Examples of object detection models that can be used include deep learning models such as R-CNN (Regional Convolutional Neural Newworks), YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), DCN (Deformed Convolutional Networks), and DETR (End-to-End Object Detection with Transformers). The image analysis unit 1080 according to the third embodiment identifies an object (object) being held or pointed at by the user. In this case, the image analysis unit 1080 may identify an object located above the image representing the user's hand or on an extension of the fingertips as the object to be identified based on the (coordinate) position of the object detected from the captured image. The image analysis unit 1080 inputs the identification result as analysis information (e.g., an image caption) to the prompt generation unit 1032. Note that the image analysis unit 1080 may be configured such that the control unit 102 executes a program stored in the storage unit 110, as illustrated in the example of FIG. 12 , or the image analysis unit 1080 may be implemented by the photographing unit 170 (e.g., a camera).

[0136] [3.2 Processing Flow] The processing flow according to the third embodiment will be described using Fig. 13. The processing described in Fig. 13 is preferably executed by the respective components described in Fig. 3, Fig. 9, Fig. 12, etc., as appropriate, but for the sake of convenience, it will be described as being executed by the control unit 102.

[0137] The control unit 102 acquires an utterance as an input sentence by the user (S60). In the third embodiment, the description will be given assuming that the input sentence uttered by the user includes, for example, "What is this?", "What is this?", or "What is that thing you are holding in your hand (pointing at)?" as the content of the inquiry to the avatar about the object the user is holding in his / her hand or pointing at.

[0138] The control unit 102 checks whether a camera or the like is connected as the image capturing unit 170 (S62). If it is determined that a camera is connected as the image capturing unit 170, the control unit 102 captures an image including at least the user and the object that the user wishes to identify (S62; Yes → S64). Note that if it is determined that a camera is not connected as the image capturing unit 170, the control unit 102 proceeds to S72 (S62; No → S72).

[0139] After capturing the image, the image analysis unit 1080 analyzes the captured image (S66) and identifies the object the user is holding or pointing at (S68).

[0140] The control unit 102 obtains output data by inputting analysis information based on the object identification results by the image analysis unit 1080, the input sentence (user's speech), historical information, and a prompt including a theme word, if necessary, into the language model (S70).

[0141] The control unit 102 generates a response sentence from the output data, displays or speaks based on the response sentence, and ends the process (S72).

[0142] [3.3 Operation Example] An operation example will be described with reference to FIG.

[0143] FIG. 11 is a diagram showing an example of a display screen W112 that displays an avatar that displays or speaks the result of identifying an object that the user is holding or pointing at.

[0144] 14 shows a state in which, based on the user U1's utterance of "What is this?" to object Ob10, avatar A120 displays or speaks a response sentence such as "It is ____" as message M120. At this time, the control unit 102 identifies the thing (object) the user is holding or pointing at as object ____ based on the image (video) captured by the image capture unit 170 and environmental information acquired as needed by the sensor unit 190. Furthermore, when identifying object ____, if the control unit 102 acquires related information about object Ob10 (e.g., the homepage on which it is posted, links, product information, etc.), it can also display this related information as message M122 (e.g., "The homepage is here.").

[0145] In this way, the display device 10 of this embodiment can apply real-time external information to the input sentence, and therefore can reflect a wide range of information in the response sentence. For example, the display device 10 of this embodiment can reflect in the input sentence an object Ob10 such as an object or animal that is difficult for the user to express in his or her own words by acquiring the object Ob10 using the sensor unit 190, thereby enabling the user to obtain a response sentence that suits his or her needs.

[0146] [4. Fourth Embodiment] The fourth embodiment will be described. As with the third embodiment, the fourth embodiment is an embodiment in which external information, such as an image captured by the image capture unit 170 and environmental information acquired by the sensor unit 190, is used to generate a response sentence for an avatar. The fourth embodiment is an embodiment in which, among other things, the state and attitude of a user are estimated based on external information, and the estimation results are used as analysis information in the response sentence (utterance) of the avatar. The configuration of the fourth embodiment is substantially the same as the hardware configuration and software configuration of the first, second, or third embodiment. Therefore, the third embodiment will be described mainly with respect to the parts that are different from the previously described embodiments.

[0147] [4.1 Software Configuration] [4.1.1 Configuration of Control Unit] The configuration of the control unit according to the fourth embodiment is substantially the same as the configuration of the control unit 102 illustrated in Fig. 12. In addition to (or instead of) the configuration of the image analysis unit 1080 according to the third embodiment, the image analysis unit 1080 according to the fourth embodiment includes an estimation model that estimates the user's state and attitude (degree of positivity and emotional expression) based on the image (video) captured by the image capture unit 170 and environmental information acquired as needed by the sensor unit 190. For example, to detect the user's face as an example of the user's state, an optimal combination of features such as Haar-like features, Joint Haar-like features, and Sparse features that focus on the brightness difference in partial areas of the face can be used as a learning model for face detection. Furthermore, when the captured image is a range image, the image analysis unit 1080 can estimate the user's posture as the user's state by detecting the position coordinates of each part, such as the head, hands, and feet, from the RGB color image using, for example, a convolutional pose machine or open pose, if a CNN or similar is applicable as a posture estimation model. Furthermore, the image analysis unit 1080 can estimate the user's attitude, such as "angry," "enjoyed," or "sad," from the user's facial expression, gaze direction, etc. In this case, the image analysis unit 1080 can estimate the user's attitude by using a trained model that uses the movements and size fluctuations of each part of the face (e.g., eyes, nose, mouth, etc.) and facial muscles, the gaze direction (eye direction), etc. as objective variables.

[0148] [4.2 Processing Flow] The processing flow according to the fourth embodiment will be described using Figure 15. The processing described in Figure 15 is preferably executed by the respective components described in Figures 3, 9, 12, etc., as appropriate, but for the sake of convenience, it will be described as being executed by the control unit 102. Furthermore, processing equivalent to the processing described in Figure 13 will be described using the same step numbers.

[0149] The control unit 102 acquires an utterance by the user as an input sentence (S60).

[0150] The control unit 102 checks whether a camera or the like is connected as the image capturing unit 170 (S62). If it is determined that a camera is connected as the image capturing unit 170, the control unit 102 captures an image including at least the user (S62; Yes → S64). Note that if it is determined that a camera is not connected as the image capturing unit 170, the control unit 102 proceeds to S72 (S62; No → S72).

[0151] After capturing the image, the image analysis unit 1080 analyzes the captured image (S66) and estimates the state and attitude of the user (S74).

[0152] The control unit 102 obtains output data by inputting analysis information based on the estimation results of the user's state and attitude by the image analysis unit 1080, the input sentence (user's speech), historical information, and a prompt including a theme word, if necessary, into the language model (S76).

[0153] The control unit 102 generates a response sentence from the output data, displays or speaks based on the response sentence, and ends the process (S72).

[0154] [4.3 Operation Example] An operation example will be described with reference to FIG.

[0155] FIG. 11 is a diagram showing an example of a display screen W114 that displays an avatar that displays or speaks a response sentence based on the estimation result of the user's state or attitude.

[0156] Figure 16 shows (a) user U2 lying down and browsing his smartphone, uttering the command "Play some music," (b) user U3 appearing angry for some reason, and (c) user U4 enjoying watching the content currently being displayed.

[0157] Then, example responses of avatar A130 to each user (users U2 to U4) are displayed on the display screen W114. For example, (a) in the figure shows a state in which avatar A130 displays or speaks a message M124 (e.g., "Play music from your favorites list") intended to maintain user U2's relaxed state based on user U2's utterance H102, "Play some music," and the inference that user U2 is "lying down." Also, (b) in the figure shows a state in which avatar A130 displays or speaks a message M126 (e.g., "Would you like to play a healing video?") intended to suggest playing content to calm user U3's excited state based on the inference that user U3 is "angry." Furthermore, in (c) of the figure, based on the estimation result that user U4 is "having fun," avatar A130 displays or speaks a message M128 (e.g., "There is a sequel. Would you like to continue playing it?") suggesting playing content that will continue user U4's enjoyable state.

[0158] In this way, the display device 10 of this embodiment applies the user's attitude as real-time external information to the input sentence and reflects it in the response sentence. As described above, the display device 10 of this embodiment reflects the user's attitude in the input sentence, so that the user can obtain a response sentence that is appropriate for their own mental and physical state. This allows the user to obtain a response that is more appropriate to their needs.

[0159] [5. Modifications] The present disclosure is not limited to the above-described embodiments, and various modifications are possible. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of the present disclosure are also included in the technical scope.

[0160] Although the above-mentioned embodiments are described separately for convenience of explanation, they can be combined to the extent possible. Furthermore, the present invention intends to obtain rights to any of the technologies described in the specification through amendments or divisional applications, etc.

[0161] Furthermore, although the above-described embodiment has been described using HDMI as an example of the device connection format, other connection formats may be used. For example, at the time of filing, connection formats that can use EDID indicating display capabilities, such as DVI, VGA, and DisplayPort, may be used. Furthermore, although the above-described embodiment uses EDID as information indicating display capabilities, other information indicating display capabilities may be used. Furthermore, although two HDMI standards, the first standard and the second standard, have been described as examples, three or more standards may be mixed. Even in this case, for example, a device that outputs video can use the highest resolution that the device that displays video can output.

[0162] In addition, the programs that run on each device in each embodiment are programs that control the CPU, etc. (programs that make a computer function) so as to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and then stored in various ROMs and HDDs, from which it is read, modified, and written by the CPU as needed.

[0163] Here, the recording medium for storing the program may be any of semiconductor media (e.g., ROM, non-volatile memory card, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disc), etc.), magnetic recording media (e.g., magnetic tape, flexible disk, etc.), etc.

[0164] Furthermore, when distributing the program on the market, the program can be stored in a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server device is also included in the present disclosure.

[0165] Furthermore, the above-mentioned data may not be stored within the device, but may be stored in an external device and called up as needed. For example, the data may be stored in a network attached storage (NAS) or on the cloud.

[0166] The scope of the present disclosure is not limited to the configurations explicitly described in the specification, but also includes combinations of the technologies disclosed in the specification. The configurations of the present disclosure for which a patent is sought are set forth in the appended claims, but it is not intended to exclude them from the technical scope on the grounds that they are not set forth in the claims.

[0167] Furthermore, in the above-mentioned specification, the statements "in the case of" and "when" are given as examples and are not intended to limit the configuration to the described contents. The disclosure also includes configurations that are not in these cases or when they are obvious to a person skilled in the art, and the patentee intends to obtain the rights to such configurations.

[0168] Furthermore, the processes and data flows described in the specification are not limited to the order in which they are described. For example, the patent also discloses configurations in which some processes are deleted or the order is changed, and the patent holder intends to obtain the rights to such configurations.

[0169] Although the functions described in the embodiments are executed by respective devices, they may be implemented by a single device or may further utilize an external server. For example, a standalone device may be implemented as an electronic device, such as a set-top box. In this case, the electronic device may have at least some of the functions shown in FIGS. 1 to 4 , with the remaining functions provided elsewhere. Alternatively, the electronic device may have all of these functions. In particular, when there are multiple language models, all of the language models may be provided in the electronic device, or at least some of the language models may be provided in the electronic device, with the remaining language models provided elsewhere. Furthermore, the user may be able to select whether or not to display a message. Similarly, the user may be able to select whether or not to display the avatar's speech. Note that the above-mentioned "product information" may also be defined as a product related to the provision of services, such as travel arrangements, and the object of commercial transaction is not limited.

[0170] Furthermore, the display screens described in the above-described embodiments and drawings are merely examples, and are not intended to limit the displayed items, content, or layout. For example, the items displayed on the display screen can be rearranged, or the number of items can be changed. Furthermore, for the sake of convenience, the example display screen displays only a portion of the functions that can actually be realized. Further necessary information and items may be displayed, or conversely, items may be omitted as necessary.

[0171] Furthermore, each functional block or feature of the device used in the above-described embodiments may be implemented or performed by an electrical circuit, for example, an integrated circuit or multiple integrated circuits. The electrical circuit designed to perform the functions described herein may include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof. The general-purpose processor may be a microprocessor, or a conventional processor, controller, microcontroller, or state machine. The electrical circuit may be composed of digital circuits or analog circuits. Furthermore, as advances in semiconductor technology emerge, one or more aspects of the present disclosure may utilize new integrated circuits based on that technology.

[0172] REFERENCE SIGNS LIST 1 System 10 Display device 100 Control unit 110 Memory unit 110A ROM 110B RAM 110C Storage 120 Broadcast control unit 130 Operation unit 140 Display unit 150 Audio input unit 160 Audio output unit 170 Shooting unit 180 Communication unit 190 Sensor unit 20 Server device 22 Large-scale language model 30 Broadcast station 40 Distribution device

Claims

1. A processing device comprising: a display control unit that controls the display of a display screen having a display area capable of displaying an avatar corresponding to a character; a voice input unit that accepts voice input from a user; a recognition unit that recognizes an input sentence from content or the voice input; a prompt generation unit that generates a prompt from the input sentence; an output data acquisition unit that inputs the prompt into a first language model and acquires output data; an output unit that generates a response sentence from the output data and outputs it from the avatar; a memory unit that stores a combination of the input sentence and the response sentence as conversation history information; and a summary information acquisition unit that inputs the conversation history information into a second language model and acquires summary information, wherein the prompt generation unit generates the prompt in response to the input sentence and the summary information.

2. The processing device according to claim 1, wherein the storage unit updates and stores the conversation history information based on the summary information.

3. The processing device according to claim 2, further comprising a theme word acquisition unit that acquires theme words from the summary information, wherein the prompt generation unit generates the prompt in response to the input sentence, the summary information, and the acquired theme words.

4. The processing device according to claim 3, wherein the theme words include theme words related to the user's preferences along a specific theme.

5. The processing device according to claim 4, wherein the first language model outputs, as output data, a response sentence that prompts the user to speak in response to the theme word.

6. A processing device as described in claim 1, comprising: an imaging unit that captures an image including at least the user; and an image analysis unit that analyzes the image captured by the imaging unit; wherein the prompt generation unit generates the prompt according to analysis information based on the analysis results of the captured image, the input sentence, and the summary information.

7. A processing method in a processing device capable of controlling the display of a display screen having a display area capable of displaying an avatar corresponding to a character, comprising: a voice input step of accepting a user's voice input; a recognition step of recognizing an input sentence from content or the voice input; a prompt generation step of generating a prompt from the input sentence; an output data acquisition step of inputting the prompt into a first language model and acquiring output data; an output step of generating a response sentence from the output data and outputting it from the avatar; a storage step of storing a combination of the input sentence and the response sentence as conversation history information; and a summary information acquisition step of inputting the conversation history information into a second language model and acquiring summary information, wherein the prompt generation step includes a step of generating the prompt corresponding to the input sentence and the summary information.

8. A program for a computer capable of controlling the display of a display screen having a display area capable of displaying an avatar corresponding to a character, the program comprising: a voice input function for accepting voice input from a user; a recognition function for recognizing an input sentence from content or the voice input; a prompt generation function for generating a prompt from the input sentence; an output data acquisition function for inputting the prompt into a first language model and acquiring output data; an output function for generating a response sentence from the output data and outputting it from the avatar; a storage function for storing a combination of the input sentence and the response sentence as conversation history information; and a summary information acquisition function for inputting the conversation history information into a second language model and acquiring summary information, the program causing the prompt generation function to realize the function of generating the prompt corresponding to the input sentence and the summary information.

Images