Processing device, voice output method, and program

Abstract

The present invention comprises: a display control unit that controls display of a display screen including a first display region that is capable of displaying acquired content and a second display region that is capable of displaying an avatar which corresponds to a character; a voice input unit that receives voice input from a user; a recognition unit that recognizes an input sentence from the content or the voice input; a processing unit that generates a response sentence corresponding to the input sentence; a voice generation unit that, on the basis of the response sentence, generates a second voice uttered by the avatar; an adjustment unit that adjusts the volume level of the first voice included in the content and the volume level of the second voice; and a voice output unit that outputs at least one of the first voice and the second voice at the volume level adjusted by the adjustment unit. This makes it possible to provide a processing device or the like that is capable of easily manipulating the respective volumes of the voice of the avatar and the voice in the content.

Description

Processing device, audio output method and program

[0001] The present disclosure relates to a processing device, etc. This application is based on Japanese Patent Application No. 2024-152310 filed on September 4, 2024, and the contents of that application are incorporated herein by reference.

[0002] 2. Description of the Related Art Conventionally, there is known a system that displays a plurality of avatar images in a virtual space such as a virtual concert hall, and enables communication via each avatar image.

[0003] Japanese Patent Application Laid-Open No. 2022-065363

[0004] An object of the present disclosure is to provide a processing device or the like that is capable of appropriately manipulating the volume of an avatar's voice and the volume of a content, for example.

[0005] The processing device of the present disclosure includes a display control unit that controls the display of a display screen having a first display area in which acquired content can be displayed and a second display area in which an avatar corresponding to a character can be displayed, a voice input unit that accepts voice input from a user, a recognition unit that recognizes an input sentence from the content or the voice input, a processing unit that generates a response sentence corresponding to the input sentence, a voice generation unit that generates a second voice to be uttered by an avatar based on the response sentence, an adjustment unit that adjusts the volume level of a first voice included in the content and the volume level of the second voice, and a voice output unit that outputs at least one of the first voice and the second voice at the volume level adjusted by the adjustment unit.

[0006] The audio output method in an information processing device disclosed herein is an audio output method in a processing device capable of controlling the display of a display screen having a first display area capable of displaying acquired content and a second display area capable of displaying an avatar corresponding to a character, and includes a voice input step of accepting a user's voice input, a recognition step of recognizing an input sentence from the content or the voice input, a processing step of generating a response sentence corresponding to the input sentence, a voice generation step of generating a second voice to be uttered by an avatar based on the response sentence, an adjustment step of adjusting the volume level of a first voice included in the content and the volume level of the second voice, and an audio output step of outputting at least one of the first voice and the second voice at the adjusted volume level.

[0007] The program of the present disclosure enables a computer capable of controlling the display of a display screen having a first display area capable of displaying acquired content and a second display area capable of displaying an avatar corresponding to a character to implement the following: a voice input function for accepting a user's voice input; a recognition function for recognizing an input sentence from the content or the voice input; a processing function for generating a response sentence corresponding to the input sentence; a voice generation function for generating a second voice to be uttered by an avatar based on the response sentence; an adjustment function for adjusting the volume level of a first voice included in the content and the volume level of the second voice; and a voice output function for outputting at least one of the first voice and the second voice at the adjusted volume level.

[0008] According to the present disclosure, it is possible to provide, for example, a processing device or the like that is capable of appropriately manipulating the volume of the voice of an avatar and the voice of content.

[0009] FIG. 1 is a diagram for explaining an overview of a system in a first embodiment. FIG. 1 is a diagram for explaining the hardware configuration of (a) a display device and (b) a remote control device in the first embodiment. FIG. 2 is a diagram for explaining the software configuration in the first embodiment. FIG. 3 is a diagram for explaining the software configuration in the first embodiment. FIG. 4 is a diagram for explaining the configuration of a storage unit in the first embodiment. FIG. 5 is a diagram for explaining a basic processing flow in the first embodiment. FIG. 6 is a diagram for explaining (a) an example of a display screen, (b) an example of a prompt, and (c) an example of output data in the first embodiment. FIG. 7 is a diagram for explaining the processing flow in the first embodiment. FIG. 8 is a diagram for explaining an example of a display screen as an operation example in the first embodiment. FIG. 9 is a diagram for explaining the software configuration in a second embodiment. FIG. 10 is a diagram for explaining the processing flow in a third embodiment. FIG. 11 is a diagram for explaining the processing flow in the third embodiment. FIG. 12 is a diagram for explaining the processing flow in the third embodiment. FIG. 13 is a diagram for explaining an example of a display screen as an operation example in the third embodiment.

[0010] For example, in a system such as that disclosed in Patent Document 1, an audio signal control device controls the volume of an audio signal so that the volume of the virtual sound source of an avatar image located farther away is reduced relative to the display position of the avatar image of a specific user in a virtual space. In this system, the volume of the virtual sound source of the avatar image changes depending on the display position of the avatar image.

[0011] However, the above system has the following problems: (1) in order to control the volume of the virtual sound source of the avatar image, it is necessary to move the display position of the image by pressing a cursor key; (2) because the sound of the virtual sound source of the avatar image and the sound of the concert are output from the same speaker, it is difficult to distinguish between them, and it is necessary to operate the volume multiple times; and (3) because the output of the audio signal of the virtual sound source of the avatar image and the output of the audio signal of the concert are controlled independently, it is necessary to adjust them individually, which makes the user's operation complicated. As such, previous systems have not been able to appropriately control both the sound of the avatar (hereinafter sometimes referred to as the second sound) and the sound of the video (hereinafter sometimes referred to as the first sound).

[0012] As described above, the processing device and the like of the present disclosure that solve one or more of the above-described problems will be described in the following embodiments with reference to the drawings. Note that the following embodiments describe the invention set forth in the claims as an example, and the technical scope of the present invention is not limited to the description of the following embodiments. Furthermore, in the following embodiments, a case where the processing device of the present disclosure is applied to a display device will be described, but the processing device is not limited to a display device. For example, the processing device may be a standalone set-top box (STB) with a tuner function, or may be applied to a recording device using a hard disk, a display control device that plays back a recording medium, a projector, etc. Furthermore, the processing device may be an information processing device such as a smartphone, tablet, or computer used by a user as a device capable of displaying content, or may be an in-vehicle device such as a car navigation system.

[0013] [1. First Embodiment] [1.1 About the System] [1.1.1 Overview of the System] Fig. 1 is a diagram illustrating an overview of a system 1. The system 1 has a display device 10 and is connected to a network NW. The display device 10 may be capable of recognizing a user U in the vicinity of the device, for example. The display device 10 may have a built-in camera 12 and a microphone 14, and may recognize a user U in front of the display device 10. The display device 10 may be capable of being remotely operated by a user operating a remote control device 300.

[0014] The system 1 may also include a server device 20. The server device 20 is connected to, for example, a network NW. The server device 20 may also store a large-scale language model 22 as a language model.

[0015] The large-scale language model 22 is one type of language model and is generally called an LLM (Large Language Model). The large-scale language model 22 is stored in the server device 20, but may also be stored in the display device 10. The display device 10 may store a language model different from the large-scale language model 22, for example, a small-scale language model (SLM) as an edge LLM. That is, as a language model, a language model (LLM) with a large amount of training data may be stored in the server device 20, and a language model (SLM) with less training data than the large-scale language model 22 may be stored in the display device 10 or another device. In the following specification, a language model stored in the display device 10 may also be referred to as an edge language model.

[0016] The large-scale language model 22 may be prepared by the service provider that builds the system 1, or an external service may be used. For example, large-scale language models such as GPT (Generative Pre-trained Transformers, GPT-3, GPT-4, GPT-4o), PaLM (Pathways Language Model), LLaMA (Large Language Model Meta AI), and Tsuzumi may be used.

[0017] The display device 10 can also acquire and display content. The display device 10 may acquire the content from a broadcast station 30, from a distribution device 40, or from a recording medium. The broadcast station 30 may also transmit the content to the display device 10 via a network NW. The broadcast station 30 transmits the content to the display device 10 using, for example, terrestrial digital broadcasting, BS broadcasting, or CS broadcasting. The broadcast station 30 may also transmit the content via the network NW using IP multicast broadcasting.

[0018] In this embodiment, a response sentence is output using an "avatar." However, for example, the response sentence may be displayed on the display screen of the display device 10 using a language model without using an avatar, or the display device 10 may read the response sentence aloud. Furthermore, the display device 10 may be a display system including a device that simply displays a video signal from an external device and a device that outputs the video signal. For example, the display device 10 may include a system including a display and a recording and / or playback device connected via HDMI (registered trademark).

[0019] [1.1.2 Terminology] Hereinafter, terms used in this specification will be used in a way that indicates a meaning that can be understood by a person skilled in the art, for example, as follows.

[0020] An "avatar" is an object that is represented based on a character. It can be selected or created primarily by the user. The avatar may correspond to a specific character, and may represent, for example, a virtual friend, secretary, or advisor, who can be used as a conversation partner. Although it is called an avatar, it does not have to be an alter ego of the user.

[0021] "Conversation" refers to the exchange of sentences, and in this embodiment, particularly refers to the exchange and combination of input sentences and response sentences. Here, an input sentence is a sentence based on the voice uttered by the user. A response sentence is a sentence output or spoken by an avatar. These sentences are also sometimes called messages. An avatar uttering a sentence (response sentence) means, for example, that the control unit displays a sentence (message) or outputs the sentence (message) by voice. In this case, by linking with the display of the avatar, the user can interact with the avatar as if it were actually speaking to them.

[0022] A "prompt" is information for instructing a language model (large-scale language model, small-scale language model) to generate output data including a response sentence.

[0023] The "user state" refers to the state of the user, and may refer to, for example, the state of the user when in front of the display device 10, but may preferably refer to the state of the user when viewing content. For example, the user state may include the user's facial expression (laughing, surprised, crying, etc.), gaze direction, and face direction that can be recognized from an image. The user state may also include the user's emotion (wow, amazing, I want, etc.) that can be recognized from voice. The user state may also include the user's age, gender, and number of users.

[0024] "Content" includes program content broadcast from broadcast station 30 and distribution content distributed from distribution device 40. Content may also be content recorded on a recording medium (e.g., Blu-ray (registered trademark) or the like). Content includes video including one or more images and audio, and may also include explanatory information, attributes, and subtitle information related to the content. While the content is mainly described using video content as an example, it may also include still images, audio, text, web pages, and the like as needed. For example, video content includes all kinds of images, such as entertainment, education, news, e-commerce (electronic commerce), accommodation information, and travel arrangements.

[0025] [1.2 Hardware Configuration] [1.2.1 Display Device 10] Next, the hardware configuration of the devices in this embodiment will be described using FIG. 2(a). The display device 10, server device 20, and distribution device 40 each include at least a control unit, a storage unit, and a communication unit. Each device also has general functions as needed. Referring to FIG. 2(a), the hardware configuration of the display device 10 will be described.

[0026] The control unit 100 controls the entire display device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the memory unit 110 (e.g., ROM 110A, storage 110C). The control unit 100 may be realized by one or more control devices / arithmetic units (CPUs (Central Processing Units), SoCs (System on a Chip)). The control unit 100 may also be configured by a control circuit.

[0027] The storage unit 110 is one or more storage devices that store necessary data and programs. The storage unit 110 stores data and programs temporarily or non-temporarily. For example, the storage unit 110 includes a ROM 110A, a RAM 110B, and a storage 110C.

[0028] The ROM 110A is a non-volatile memory that can retain programs and data even when the power is turned off.

[0029] The RAM 110B is a main memory that is mainly used by the control unit 100 when executing processes. The RAM 110B is a rewritable memory that temporarily stores programs read from the ROM 110A or the storage 110C, and data including execution results.

[0030] The storage 110C is a non-volatile storage device capable of storing programs and data. For example, the storage 110C may be configured as a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 110C may also be configured as an externally connectable USB memory. The storage 110C may also be, for example, a storage area on the cloud.

[0031] The broadcast control unit 120 receives broadcast waves transmitted by a broadcast station selected by the user, decodes video data from the broadcast waves and outputs the data to the display unit 140, and decodes audio data from the broadcast waves and outputs the data to the audio output unit 160. The broadcast control unit 120 may be configured with, for example, a digital tuner unit (terrestrial / BS / CS, etc.), an OFDM demodulation unit, a DEMUX unit, an MPEG2 decoding unit, etc. Furthermore, the broadcast control unit 120 may have multiple units.

[0032] The operation unit 130 receives operations from the user, issues operation instructions to each functional unit, and notifies the control unit 100 of an operation signal corresponding to the received operation. The operation unit 130 may perform control based on operations received using, for example, operation buttons provided on the display device 10 or software keys using a touch panel.

[0033] The display unit 140 is a display device capable of displaying images of received programs and various information. The display unit 140 may be, for example, a device capable of displaying images, such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display unit 140 may also be, for example, a projection device such as a projector.

[0034] The audio input unit 150 is an input device that can input sounds from the surrounding area where the display device 10 is installed, such as a microphone. The audio input unit 150 may also be composed of multiple input devices (for example, a microphone array). The audio input unit 150 is primarily used to input the audio of the user watching, but can also input general sounds such as environmental sounds. The display device 10 can recognize the state of the user from the audio input from the audio input unit 150.

[0035] The audio output unit 160 outputs audio included in the content. The audio output unit 160 may be, for example, a device such as a speaker or headphones. The audio output unit 160 may output any sound, and may output general sounds such as music, environmental sounds, etc. A plurality of audio output units 160 may be provided.

[0036] The image capturing unit 170 is an image capturing device, such as a camera, that captures images of the surrounding area where the display device 10 is installed. The image capturing unit 170 may be composed of one or more image capturing devices. The image capturing unit 170 outputs the captured image as an image signal. Alternatively, the image capturing unit 170 may output one or more images as a continuous video. The display device 10 can recognize the user's condition from the images captured by the image capturing unit 170. Alternatively, a sensor using infrared rays, millimeter waves, or the like may be provided instead of or together with the image capturing unit 170. In this case, the display device 10 can acquire the user's condition, such as body temperature or heart rate, using the infrared rays, millimeter waves, or the like sensor.

[0037] The communication unit 180 is a communication interface for communicating with other devices. For example, the communication unit 180 may be a network interface connectable to a wireless LAN or a network interface connectable to Ethernet (registered trademark) via a wired connection. The communication unit 180 may also be a communication device connectable to a mobile communication network such as LTE / 4G / 5G / 6G.

[0038] 2A may include any necessary components in the embodiment. For example, if the user's state is not acquired from a camera, the image capturing unit 170 may not be included.

[0039] The display device 10 may also be connected to an external device via an interface unit, and the above-described functions may be realized by the external device. For example, the image capturing unit 170 may be realized by a camera device connected via a USB interface.

[0040] [1.2.2 Remote Control Device 300] Figure 2(b) is a diagram illustrating the hardware configuration of the remote control device 300. The remote control device 300 is a so-called remote controller, and as long as it is located within a distance that allows at least one-way communication (from the remote control device 300 to the display device 10) with the display device 10, it may be a portable type that can be carried around, or a fixed type that is fixed to, for example, a wall surface in a room where the display device 10 is installed. Such a remote control device 300 includes a control unit 301, an operation unit 303, an operation signal switching unit 305, and a communication unit 307. Note that the remote control device 300 is not limited to a conventional remote control, as long as it is a terminal that can communicate with the display device 10.

[0041] The control unit 301 controls the entire remote control device 300. The control unit 301 realizes various functions by reading and executing programs stored in a storage device (not shown). The control unit 301 may be realized by one or more control devices / arithmetic units (CPU, SoC). The control unit 301 may also be configured by a control circuit.

[0042] The operation unit 303 includes various switches such as buttons, dials, knobs, and slide levers provided on the surface of the main body of the remote operation device 300 for adjusting the volume, switching channels, and switching the video input / output destination, and receives operations from the user, issues operation instructions to each functional unit, and notifies the control unit 301 of operation signals corresponding to the received operations. The operation unit 303 is preferably provided with a plurality of operation switches (buttons) that can individually adjust the volume level of the first audio and the volume level of the second audio, but may also be provided with a changeover switch that receives input changeover between an operation instruction for the volume level of the first audio and an operation instruction for the volume level of the second audio, and operates either the volume level of the first audio or the volume level of the second audio based on the changeover switch. In addition, there may be one operation switch (button) for adjusting the volume, and when the operation switch (button) is operated while an avatar is speaking as the second voice, the volume level of the avatar's voice is controlled, and when the operation switch (button) is operated while the voice of the content is being output as the first voice, the volume level of the voice of the content may be controlled.

[0043] Based on an operation signal output from the operation unit 303 (an operation switch related to volume adjustment), the operation signal switching unit 305 switches the operation signal to be output via the communication unit 307. The operation signal switching unit 305 transmits, to the display device 10 via the communication unit 307, an operation signal corresponding to, for example, identification information for the display device 10 to identify the remote operation device 300, switching information of the operation switch related to volume adjustment (information indicating whether adjustment of the volume level of the first audio or adjustment of the volume level of the second audio has been selected), and information related to the operation amount of the operation switch.

[0044] The communication unit 307 is a communication interface that can transmit and receive information to and from the communication unit 180 of the display device 10 .

[0045] The above-described hardware configuration is an example, and at least some of the functions, such as the function of the user's switching operation in the operation signal switching unit 305, may be realized by software. For example, each function may be executed by the user speaking the operation content corresponding to each of the above-described functions into the voice input unit 150. An example of a user's speech is "Turn up (turn down) the volume of the avatar (content)."

[0046] [1.3 Software Configuration] [1.3.1 Configuration of Control Unit] Fig. 3 is a diagram illustrating the software configuration, focusing on the configuration of the control unit. Fig. 4 is a diagram illustrating the configuration of the volume adjustment unit, which is a functional unit of the control unit. For example, the configurations of Figs. 3 and 4 are realized by control unit 100 executing a program stored in storage unit 110. Control unit 100 also functions as a display control unit that displays content, avatars, etc. on display unit 140. The display control unit may include, for example, the functions of content image output unit 1006 and avatar image output unit 1046.

[0047] (Content) The content acquisition unit 1002 acquires content from the broadcast control unit 120 or the communication unit 180. The content acquisition unit 1002 may also acquire stored content from the storage unit 110.

[0048] The content playback unit 1004 plays (displays) the content acquired by the content acquisition unit 1002. The content playback unit 1004, for example, separates multiplexed image data from the content and outputs it to the content image output unit 1006. The content playback unit 1004 also separates multiplexed audio data from the content and outputs it to the content audio output unit 1008. Any known method may be used to separate the image data (image signal) and audio data (audio signal) from the content.

[0049] The content image output unit 1006 outputs image data included in the content as an image (or video as a series of images) to the display unit 140. Here, the display unit 140 displays the image of the content as a first image.

[0050] Furthermore, the content audio output unit 1008 outputs audio data included in the content as audio to the audio output unit 160 via the volume adjustment unit 200. Here, the audio output unit 160 outputs the audio of the content as the first audio.

[0051] Furthermore, the content recognition unit 1010 recognizes image data and audio data contained in the content, as well as content information contained in the content (for example, program information, subtitle information, etc. in the case of broadcast waves), and outputs the information to the prompt generation unit 1032. The information related to the content recognized by the content recognition unit 1010 is called content information.

[0052] (User Recognition) The user recognition unit 1022 recognizes a user who is a viewer of the content and recognizes information about the user based on the image captured by the image capturing unit 170. Information about a user is called user information, and the user recognition unit 1022 can recognize the following content as user information.

[0053] (1) Number of users. For example, the user recognition unit 1022 recognizes the number of viewers of the content. The user recognition unit 1022 may also recognize the number of viewers of the content by using the direction of the face and the direction of the line of sight.

[0054] (2) Identifying a user. For example, the user recognition unit 1022 recognizes who is viewing the content. The user recognition unit 1022 can identify a user by acquiring an image of the user's face (face image) and matching it with face images of users registered in advance.

[0055] (3) User's Facial Expression For example, the user recognition unit 1022 can recognize the user's facial expression (facial expression).

[0056] The speech recognition unit 1024 recognizes a conversation (input sentence) uttered by the user based on the speech input by the speech input unit 150. The speech recognition unit 1024 outputs the input sentence to the conversation processing unit 1030.

[0057] Furthermore, the voice recognition unit 1024 may recognize the user's emotions (for example, exclamations, clapping sounds, etc.) as user information.

[0058] That is, the user recognition unit 1022 can recognize user information based on an image, and the voice recognition unit 1024 can recognize user information based on a voice.

[0059] The conversation processing unit 1030 outputs the input sentence, the conversation history up to now, and any other necessary accompanying information to the large-scale language model 22, and obtains output data including a response sentence from the large-scale language model 22. In this way, the conversation processing unit 1030 can realize a conversation between the input sentence input by the user and the response sentence output by the avatar.

[0060] More specifically, the conversation processing unit 1030 includes a prompt generating unit 1032 and an output data acquiring unit 1034 .

[0061] The prompt generation unit 1032 generates a prompt to be input into a language model (e.g., large-scale language model 22) based on the input information. The prompt may be in a natural sentence format or may be in a predetermined format that is compatible with the language model 1102 and uses tags, etc. The prompt generation unit 1032 generates a prompt based on necessary information from the input sentence, conversation history, and accompanying information. Note that, hereinafter, information other than that related to the conversation may be referred to as accompanying information.

[0062] Here, the accompanying information other than the input sentence and the conversation history may include, for example, the following information.

[0063] Content information, which may include, for example, the name of the content, details analyzed from the image of the content, details recognized from the audio of the content, etc.

[0064] User information: For example, in addition to basic information about users (age, sex, number of people, etc.), information about the user's facial expression, user's movements, user's interests, etc. may be included.

[0065] Information about the environment (environment information) For example, information about the user's environment, such as the date and time, the room environment, and weather information, may be included.

[0066] Product information. For example, the information may include one or more pieces of information about products on a specific shopping site. In addition to basic information such as the product name and price, the product information may also include information such as product features, weight, size, and product images that can be obtained from the product sales site.

[0067] The prompt generation unit 1032 may generate a prompt to be input to the large-scale language model 22, for example, by inputting the above-mentioned information once into an edge LLM.

[0068] The conversation processing unit 1030 (prompt generation unit 1032) inputs a prompt including an input sentence as an example of a language model to the large-scale language model 22 of the server device 20. The large-scale language model 22 outputs output data to the conversation processing unit 1030 in response to the input prompt. Note that the large-scale language model 22 used is the one stored in the server device 20, but when the large-scale language model 22 is stored in the storage unit 110, the control unit 100 can also use the one stored in the storage unit 110. Furthermore, the control unit 100 may select whether to use the large-scale language model 22 or the language model stored in the storage unit 110 depending on the content of the conversation and the type of output data requested.

[0069] The conversation processing unit 1030 (output data acquisition unit 1034) acquires output data from the large-scale language model 22. Here, the output data may include the following information.

[0070] Response sentences responding to input sentences Product recommendation information The conversation processing unit 1030 outputs, for example, the input sentence included in the prompt and the response sentence included in the output data as a conversation to the conversation history storage processing unit 1050. The conversation history storage processing unit 1050 stores the conversation including the input sentence and the response sentence, and information about the conversation, as conversation history information in the conversation history information storage area 1110.

[0071] Furthermore, conversation processing unit 1030 outputs the conversation to avatar output control unit 1040. Conversation processing unit 1030 mainly outputs the response sentence included in the output data and information about the avatar included in the output data (e.g., the facial expression of the avatar) to avatar output control unit 1040.

[0072] Avatar output control unit 1040 functions as an avatar generation unit that generates an avatar. Avatar output control unit 1040 controls, for example, the facial expression, movement, and appearance of the avatar. Avatar output control unit 1040 also determines what the avatar will say. Avatar output control unit 1040 then outputs a signal to avatar image generation unit 1042 to generate an avatar image. Here, the avatar image may include not only an image of the avatar but also a message to be spoken by the avatar. The avatar image generated by avatar image generation unit 1042 is then output to display unit 140 via avatar image output unit 1046.

[0073] Furthermore, avatar output control unit 1040 outputs an audio signal for generating a voice to be spoken by the avatar and text data to avatar voice generation unit 1044. Avatar voice generation unit 1044 generates a voice of the avatar in accordance with the response sentence to be spoken by the avatar. Avatar voice output unit 1048 outputs the generated voice of the avatar to voice output unit 160 via volume adjustment unit 200. Here, the voice of the avatar output by voice output unit 160 is referred to as a second voice.

[0074] Here, avatar image generation unit 1042 may obtain information about the sound emitted by the avatar from avatar sound generation unit 1044. Also, avatar image generation unit 1042 may obtain information about the sound emitted by the avatar from avatar output control unit 1040.

[0075] Here, the avatar image may change its facial expression or mouth position depending on the response sentence. For example, the response sentence may be received from the avatar image generation unit 1042 or the avatar voice generation unit 1044, and the display of the avatar may be changed depending on the response sentence.

[0076] The avatar image may be displayed superimposed on the content image, or may be displayed separately in a different area.

[0077] 3 are merely examples, and one or more components may be collectively executed by the control unit 100. For example, the user recognition unit 1022, the voice recognition unit 1024, and the content recognition unit 1010 may be combined into one recognition unit.

[0078] Furthermore, content image output unit 1006 and content audio output unit 1008 may be combined into one content output unit. Furthermore, avatar image output unit 1046 and avatar audio output unit 1048 may be combined into one avatar output unit. Furthermore, the content output unit and avatar output unit may be combined into one output unit.

[0079] 4, the volume adjustment unit 200 adjusts the volume levels of the sound of the content (first sound) and the sound of the avatar (second sound) based on an operation signal received from a remote control device 300 such as a remote controller. In the first embodiment, a form in which the volume level is adjusted based on an operation signal received from the remote control device 300 will be described. However, the volume adjustment unit 200 may also adjust the volume level by receiving an operation signal output based on an operation received using, for example, an operation button provided on an operation unit 130 provided in the display device 10 or a software key using a touch panel.

[0080] The volume adjustment unit 200 includes a receiving unit 201 , an operation signal determination unit 203 , a voice acquisition unit 205 , a volume level adjustment unit 207 , and a volume level display control unit 209 .

[0081] The receiving unit 201 includes, for example, an antenna (not shown), receives an operation signal transmitted from the remote control device 300, and outputs the operation signal to the operation signal determining unit 203. The receiving unit 201 may also receive an operation signal received using an operation button provided on the operation unit 130 or a software key using a touch panel.

[0082] The operation signal determination unit 203 determines whether the operation signal input from the receiving unit 201 is an operation signal for adjusting the volume level of the first sound or an operation signal for adjusting the volume level of the second sound.

[0083] The audio acquisition unit 205 acquires the first audio (audio of the content) output from the content audio output unit 1008 and the second audio (audio of the avatar) output from the avatar audio output unit 1048. The audio acquisition unit 205 inputs the first audio acquired from the content output unit and the second audio acquired from the avatar output unit to the volume level adjustment unit 207.

[0084] The volume level adjustment unit 207 switches the output between the first audio and the second audio input from the audio acquisition unit 205 and adjusts the volume level of the switched audio based on the determination result by the operation signal determination unit 203.

[0085] The volume level display control unit 209 controls the display of a level gauge that indicates the audio level adjusted by the volume level adjustment unit 207. The volume level display control unit 209 can display the level gauge that indicates the audio level of the first audio and / or the second audio by superimposing it on the content or the like displayed on the display unit 140.

[0086] The volume level adjustment unit 207 and the volume level display control unit 209 may be configured as a single volume level control unit.

[0087] [1.3.2 Configuration of Storage Unit] The configuration of the storage unit 110 will be described with reference to FIG.

[0088] The language model 1102 is a language model that can be used in the display device 10. The language model 1102 may be, for example, an SLM or an LLM. The language model 1102 may function as an edge LLM when the large-scale language model 22 is used. Furthermore, the language model 1102 may be used instead of the large-scale language model 22.

[0089] The user information storage area 1104 stores user information, which is information about users. For example, the user who uses the display device 10 may be stored in advance as user information, or the user may be stored as a user each time the user is recognized. For example, the user information may store the user's name, full name, an icon representing the user, or an image representing the user as information for identifying the user (identification information). The user information may also store the user's age, generation, and gender.

[0090] The content information storage area 1106 stores content information, which is information related to content. The content information may store information about the content currently being played, or may store content information corresponding to content stored in the storage unit 110. The content information may include information such as the content name, the length (time) of the content, and the genre (attributes) of the content. The content information may also include the results of recognizing the image and audio of the content. For example, the content information may include a scene of the content recognized by the content recognition unit 1010 from the image of the content (information recognized from the image, information about objects such as people and objects contained in the image, and location information), words contained in the audio of the content recognized from the audio of the content, information about products introduced in the content, etc. The content information may also be stored in chronological order.

[0091] The avatar information storage area 1108 stores avatar information, which is information about an avatar. The avatar information may include, for example, the gender (male or female), age, face shape, body part arrangement, type of clothing, and type of accessories. The avatar information may also include attributes such as whether the avatar is a human, cat, dog, or anime character. The avatar information may also include information about the avatar's emotions and the sounds the avatar makes.

[0092] The conversation history information storage area 1110 stores conversation history information, which is information related to the history of a conversation. The conversation history information includes input sentences and response sentences as conversations. In addition to the conversation, the conversation history information may store one or more pieces of related information, such as user information, content information, and environmental information, as needed. Furthermore, the conversation history information may store a conversation history from the start of viewing content to the end of the content as a single unit, or may store a conversation history in chronological order by date.

[0093] [1.4 Overall Description] The overall flow of a conversation between a user and an avatar in this embodiment will now be described.

[0094] [1.4.1 Flow of conversation processing] Fig. 6 is a diagram illustrating the general flow of conversation processing. Note that although the following processing is preferably executed by one of the configurations described in Fig. 3 or 4 as appropriate, for the sake of convenience, it will be described as being executed by the control unit 100.

[0095] The control unit 100 receives the content (S10), recognizes the content, and acquires content information (S12).

[0096] When the control unit 100 detects that the user has spoken (S14; Yes), it recognizes the speech and creates an input sentence (S16). The control unit 100 also acquires necessary information from the user information and conversation history information (S18). Here, it is sufficient for the control unit 100 to acquire at least the information necessary for the processing described below.

[0097] The control unit 100 selects one or more necessary pieces of information from the input sentence, content information, user information, conversation history information, and other associated information, and generates a prompt (S18).The control unit 100 then inputs the generated prompt into a language model (e.g., the large-scale language model 22) (S20).

[0098] When the control unit 100 acquires output data corresponding to the prompt from a language model (for example, the large-scale language model 22) (S22; Yes), it generates a response sentence from the output data (S24).

[0099] Then, the control unit 100 outputs a response sentence from the avatar (S26). The control unit 100 repeats the process until the conversation process is completed (S28; No→S10).

[0100] 6, the user initiates the speech, but the avatar may initiate the speech. For example, the control unit 100 may execute the process from the process of generating a prompt (S20). In this case, the avatar will output a response sentence before the user initiates the speech.

[0101] 7A shows an example of a display screen W10. The display screen W10 has a first region R10 for displaying content, a second region R12 for displaying an avatar, and a third region R14 for displaying conversational text. The second region R12 and the third region R14 may be integrated into one region.

[0102] Furthermore, it is preferable that the conversational text be displayed so that the user's speech (input text) and the avatar's speech (response text) can be distinguished from each other. For example, in Fig. 7(a), the user's speech and the avatar's speech are displayed so that they can be distinguished from each other by the direction of the speech bubble. Furthermore, the user's speech and the avatar's speech may be displayed so that they can be distinguished from each other by, for example, the background color, the text color, the font, etc.

[0103] Furthermore, the display screen W10 in FIG. 7A displays a first region R10, a second region R12, and a third region R14, but these display regions can be switched. For example, a user may enable only the second region R12 on the display screen W10 to display only an avatar. In this case, the avatar may be adjusted to a size appropriate for the screen size of the display screen W10. Similarly, a user may enable only the third region R14 on the display screen W10 to display only the dialogue. In this case, the dialogue may be adjusted to a size appropriate for the screen size of the display screen W10, and the font may be enlarged, or more dialogue than usual may be displayed. Furthermore, multiple regions, such as the first region R10 and the third region R14, or the second region R12 and the third region R14, may be selected for display. In the following display screens, the avatar, content, and dialogue may each be selectively displayed.

[0104] For convenience of explanation, the avatars, content, and conversations have been described as switching display areas. However, the control unit 100 may switch the displayed content itself between displayed and hidden while leaving the display area unchanged. The display area also includes a display screen. The control unit 100 may output the content displayed in each display area (display screen) to a different display. For example, the content displayed in the first display area R10 may be displayed on the display unit 140 of the display device 10, and the avatars and conversations displayed in the second display area R12 and the third display area R14 may be output to an external display or another terminal device (e.g., a smartphone) connected via the communication unit 180.

[0105] 7B is a diagram showing an example of a prompt to be input to a language model (e.g., the large-scale language model 22) in natural language, and FIG. 7C is a diagram showing an example of output data output from the language model (e.g., the large-scale language model 22) in natural language.

[0106] The prompt preferably includes an input sentence. For example, FIG. 7B includes the input sentence, "Where is Tenri?" The prompt may also include one or more pieces of information, such as information about conversation history, content information, user information, interest information, and product information. The prompt may also include such information that has been optimized by the Edge LLM.

[0107] The output data preferably includes a response sentence. For example, Figure 7(c) includes the response sentence "There are probably a lot of tourists in the northern part of Nara Prefecture." The output data may also include one or more pieces of information such as avatar information (such as the avatar's facial expression), recommendation information, site-related information, and accompanying information.

[0108] 8 is a diagram illustrating the flow of the volume adjustment process in this embodiment. Note that the following process is executed by either of the configurations described in FIGS. 3 and 4 as appropriate, but will be described as being executed by the control unit 100, as in FIG. 6.

[0109] The control unit 100 plays back the content (S30). The control unit 100 determines whether the user is conversing with the avatar (S32). If it is determined that the user is conversing with the avatar, the control unit 100 determines whether an operation signal has been received from the remote control device 300 (S32; Yes → S34). On the other hand, if it is determined that the user is not conversing with the avatar, the control unit 100 waits until the user starts conversing with the avatar (S32; No).

[0110] When the control unit 100 determines that an operation signal has been received, it determines whether the received operation signal is an operation signal related to the volume level of the sound (first sound) of the content (S34; Yes → S36).On the other hand, when it determines that an operation signal has not been received, the control unit 100 waits until the operation signal is received (S34; No).

[0111] If the control unit 100 determines that the received operation signal is related to adjusting the volume level of the content audio (first audio), it adjusts the volume level of the content audio based on the received operation signal and terminates the process (S36; Yes → S38). On the other hand, if the control unit 100 determines that the received operation signal is not related to adjusting the volume level of the content audio (first audio), it determines that the received operation signal is related to adjusting the volume level of the avatar audio (second audio), adjusts the volume level of the avatar audio based on the received operation signal, and terminates the process (S36; No → S42).

[0112] 9 is a diagram showing an example of a display screen W101 when a volume adjustment operation is performed by the user via the remote control device 300. As shown in FIG.

[0113] The display screen W101 has the same screen configuration as the display screen W10 described in FIG. 7( a) in the first region R10, the second region R12, the third region R14, etc., but differs in that it displays volume level display areas L10 and L12, respectively, that display the volume levels of the content audio (first audio) and the avatar audio (second audio) currently adjusted (set) on the display device 10. The volume level display areas L10 and L12 are displayed in response to an operation signal input from the remote control device 300, and each display area is provided with level gauges L101 and L121 that display the volume levels of the content audio (first audio) and the avatar audio (second audio). By checking the level gauges L101 and L121 displayed in the volume level display areas L10 and L12, respectively, the user can easily grasp the currently set volume levels of the content audio and the avatar audio.

[0114] Note that the display color, display size, and display mode (flashing or lighting) of the volume level display areas L10 and L12 for each sound may be different to indicate whether the sound selected as the target for volume level adjustment is the sound of the content (first sound) or the sound of the avatar (second sound). Figure 9 shows the state in which the sound of the content (first sound) is selected as the target for volume level adjustment (active state). On the other hand, for the sound of the avatar (second sound) that is not selected as the target for volume level adjustment, it is also possible to indicate that the volume level cannot be adjusted (inactive state) by, for example, graying out the volume level display area L12. Note that for sounds not selected as the target for volume level adjustment, the display of the volume level display areas L10 and L12 and the level gauges L101 and L121 may be restricted.

[0115] In this way, the display device 10 of the present embodiment can adjust the volume of the avatar's voice and the content's voice, respectively, and therefore can provide an interface tailored to the user's needs, such as when the user wants to concentrate on watching the content or on a conversation with the avatar. For example, depending on the content scene, the content's voice may temporarily become louder, making the avatar's voice difficult to hear. Even in such a case, the user can easily hear the avatar's voice by simply increasing the volume of the avatar's voice, allowing for a smooth conversation with the avatar.

[0116] [2. Second Embodiment] The second embodiment is an embodiment in which the output destinations of the audio of the content (first audio) and the audio of the avatar (second audio) are respectively assigned to a plurality of output terminals, and the volume levels of the audio of the content (first audio) and the audio of the avatar (second audio) can be adjusted for each assigned output terminal. Note that the configuration of the second embodiment is obtained by replacing Figure 4 of the first embodiment with Figure 10, and the other configurations can be substantially the same as the hardware configuration and software configuration of the first embodiment.

[0117] 10 , the volume level adjustment unit 207 according to the second embodiment can output audio to a plurality of output terminals, such as an audio output unit 1601, an audio output unit 1602, and an external output device 190. In this case, the volume level adjustment unit 207 can, for example, output the audio of the content (first audio) to the audio output unit 1601 (e.g., speaker 1) and output the audio of the avatar (second audio) to the audio output unit 1602 (e.g., speaker 2), and adjust the volume level according to each output terminal.

[0118] The volume level adjustment unit 207 can also adjust the volume level by using at least one of the audio output units 1601 and 1602 as speakers of the display device 10 as an output end for the audio of the content (first audio) and by using an external output device 190 such as Bluetooth (registered trademark) earphones as an output end for the audio of the avatar (second audio). Furthermore, the volume level adjustment unit 207 can also adjust the volume level by using at least one of the audio output units 1601 and 1602 as speakers of the display device 10 as an output end for the audio of the content (first audio) and by using an external output device 190 equipped with a second screen such as a smartphone or tablet as an output end for the audio of the avatar (second audio).

[0119] In addition to the effects of the first embodiment, the present embodiment provides the following advantages. The display device 10 includes multiple output terminals for separately outputting the avatar's voice and the content's voice, thereby providing an interface that allows for smooth conversation with the user. Specifically, the user can listen to the avatar's voice through earphones or listen to the avatar's voice through a second screen at hand, making it easier to hear the avatar's voice and eliminating the need to repeatedly adjust the volume. This allows the user to have a smoother, more realistic conversation with the avatar.

[0120] [3. Third Embodiment] The third embodiment is an embodiment in which the volume levels of the content audio (first audio) and the avatar audio (second audio) are automatically adjusted and linked. Note that the configuration of the third embodiment can be substantially the same as the hardware configuration and software configuration of the first embodiment. Therefore, a description thereof will be omitted here.

[0121] [3.1 Processing Flow] [3.1.1 Automatic Adjustment of Avatar Audio During Content Playback]

[0122] FIG. 11 is a diagram illustrating a process of lowering the volume level of an avatar (second audio) or restricting speech by an avatar when content (first audio) is being played back.

[0123] The following processing is executed by any of the configurations described in FIGS. 3 and 4 as appropriate, but will be described as being executed by the control unit 100, as in FIGS. 6, 8, and the like.

[0124] The control unit 100 plays back the content (S50). Next, the control unit 100 determines whether or not the audio of the content (first audio) is being output (S52). If it is determined that the audio of the content (first audio) is being output, the control unit 100 determines whether or not the avatar has started speaking (S52; Yes→S54).

[0125] If it is determined that the sound of the content (first sound) is not being output, the control unit 100 waits until the sound of the content (first sound) is output (S52; No).

[0126] If it is determined that the avatar has started speaking (second voice), the control unit 100 reduces the volume level of the avatar's voice (second voice) or limits the avatar's speech (S54; Yes → S56).On the other hand, if it is determined that the avatar has not started speaking (second voice), the control unit 100 proceeds to S62 (S54; No → S62).

[0127] Next, the control unit 100 determines whether the output of the sound (first sound) of the content has ended (S58). If it is determined that the output of the sound (first sound) of the content has not ended, the control unit 100 returns the process to S56 and continues to reduce the volume level of the avatar's sound (second sound) or restrict the avatar's speech (S58; No → S56).

[0128] If it is determined that the output of the content's audio (first audio) has ended, the control unit 100 reduces the volume level of the avatar's audio (second audio) or lifts the restriction on the avatar's speaking (S58; Yes → S60).

[0129] The control unit 100 then determines whether or not the playback of the content has finished (S62). If it is determined that the playback of the content has finished, the control unit 100 ends the process (S62; Yes). On the other hand, if it is determined that the playback of the content has not finished, the control unit 100 returns the process to S52 (S62; No → S52).

[0130] 3.1.2 Automatic Adjustment of Avatar Audio in Accordance with Increased Content Audio Volume

[0131] FIG. 12 is a diagram illustrating a process of lowering the volume level of the avatar (second audio) when the volume level of the audio of the content (first audio) is increased.

[0132] The following processing is executed by any of the configurations described with reference to FIGS. 3 and 4 as appropriate, but will be described as being executed by the control unit 100, as in FIG.

[0133] The control unit 100 plays back the content (S70). Next, the control unit 100 determines whether the user is currently conversing with the avatar (S72). If it is determined that the user is currently conversing with the avatar, the control unit 100 determines whether the volume level of the audio (first audio) of the content has been increased by the user (S72; Yes → S74). Note that if it is determined that the user is not currently conversing with the avatar, the control unit 100 waits until the conversation starts (S72; No).

[0134] If it is determined that the user has increased the volume level of the content audio (first audio), the control unit 100 decreases the volume level of the avatar audio (second audio) (S74; Yes → S76). Next, the control unit 100 determines whether the conversation between the user and the avatar has ended (S78). If it is determined that the conversation between the user and the avatar has ended, the control unit 100 determines whether the playback of the content has ended (S78; Yes → S80). If it is determined that the conversation between the user and the avatar has not ended, the control unit 100 returns to S76 (S78; No → S76). Meanwhile, if it is determined that the user has not increased the volume level of the content audio (first audio), the control unit 100 proceeds to step S80 (S74; No → S80).

[0135] The control unit 100 then determines whether or not the playback of the content has finished (S80). If it is determined that the playback of the content has finished, the control unit 100 ends the process (S80; Yes). On the other hand, if it is determined that the playback of the content has not finished, the control unit 100 returns the process to S72 (S80; No → S72).

[0136] Although the present embodiment has been described with reference to a case where the volume level of the content audio is increased, the present invention may also be applied to a case where the volume level of the content audio is decreased as the opposite control. More specifically, if the volume level of the content audio is decreased in step S74, the volume level of the avatar may be increased in step S76. That is, if an operation by a user to adjust the volume level of the content audio is received, the volume level of the avatar audio may be adjusted in accordance with the adjustment of the volume level of the content audio. Furthermore, if an operation to adjust the volume level of the avatar audio is received, the volume level of the avatar audio may be adjusted.

[0137] 3.1.3 Automatic Adjustment of Content Audio Based on Speech Timing to Avatar

[0138] FIG. 13 is a diagram illustrating a process of lowering the volume level of the audio (first audio) of the content at the timing when the user speaks to the avatar.

[0139] The following processing is executed by any of the configurations described in FIGS. 3 and 4 as appropriate, but will be described as being executed by the control unit 100, as in FIGS. 11 and 12, etc.

[0140] The control unit 100 plays back the content (S90). Next, the control unit 100 determines whether the user has spoken to the avatar (S92). If it is determined that the user has spoken to the avatar, the control unit 100 lowers the volume level of the audio (first audio) of the content (S92; Yes → S94). On the other hand, if it is determined that the user has not spoken to the avatar, the control unit 100 proceeds to S100 (S92; No → S100).

[0141] Next, the control unit 100 determines whether the user's speech to the avatar has ended (S96). If it is determined that the user's speech to the avatar has ended, the control unit 100 increases the volume level of the content's audio (first audio) (S96; Yes → S98). If it is determined that the user's speech to the avatar has not ended, the control unit 100 returns to S94 and continues to decrease the volume level of the content's audio (first audio) (S96; No → S94).

[0142] The control unit 100 then determines whether or not the playback of the content has finished (S100). If it is determined that the playback of the content has finished, the control unit 100 ends the process (S100; Yes). On the other hand, if it is determined that the playback of the content has not finished, the control unit 100 returns the process to S92 (S100; No → S92).

[0143] 14 is an example of an operation example illustrating a process of lowering the volume level of the audio (first audio) of the content when the user speaks to the avatar. The operation example illustrated in FIG. 14 corresponds to the process described in FIG. 13.

[0144] Figure 14 shows the state in which a conversation begins on the display screen W101 illustrated in Figure 9, in which the avatar responds to the user's utterance, "Where is Tenri?" with the response, "It's in the northern part of Nara Prefecture, so I guess there are a lot of tourists there."

[0145] At this time, as the user begins speaking to the avatar (start of conversation), the control unit 100 controls the volume level of the content's audio (first audio) by lowering the volume level so that the user can easily hear the avatar's speech.

[0146] Thus, according to the third embodiment, in addition to the effects of the second embodiment, volume adjustment is performed more effectively while viewing content, allowing the user to have a smoother conversation with the avatar.

[0147] [4. Modifications] The present disclosure is not limited to the above-described embodiments, and various modifications are possible. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of the present disclosure are also included in the technical scope.

[0148] Although the above-mentioned embodiments are described separately for convenience of explanation, they can be combined to the extent possible. Furthermore, the present invention intends to obtain rights to any of the technologies described in the specification through amendments or divisional applications, etc.

[0149] Furthermore, although the above-described embodiment has been described using HDMI as an example of the device connection format, other connection formats may be used. For example, at the time of filing, connection formats that can use EDID indicating display capabilities, such as DVI, VGA, and DisplayPort, may be used. Furthermore, although the above-described embodiment uses EDID as information indicating display capabilities, other information indicating display capabilities may be used. Furthermore, although two HDMI standards, the first standard and the second standard, have been described as examples, three or more standards may be mixed. Even in this case, for example, a device that outputs video can use the highest resolution that the device that displays video can output.

[0150] In addition, the programs that run on each device in each embodiment are programs that control the CPU, etc. (programs that make a computer function) so as to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and then stored in various ROMs and HDDs, from which it is read, modified, and written by the CPU as needed.

[0151] Here, the recording medium for storing the program may be any of semiconductor media (e.g., ROM, non-volatile memory card, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disc), etc.), magnetic recording media (e.g., magnetic tape, flexible disk, etc.), etc.

[0152] Furthermore, when distributing the program on the market, the program can be stored in a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server device is also included in the present disclosure.

[0153] Furthermore, the above-mentioned data may not be stored within the device, but may be stored in an external device and called up as needed. For example, the data may be stored in a network attached storage (NAS) or on the cloud.

[0154] The scope of the present disclosure is not limited to the configurations explicitly described in the specification, but also includes combinations of the technologies disclosed in the specification. The configurations of the present disclosure for which a patent is sought are set forth in the appended claims, but it is not intended to exclude them from the technical scope on the grounds that they are not set forth in the claims.

[0155] Furthermore, in the above-mentioned specification, the statements "in the case of" and "when" are given as examples and are not intended to limit the configuration to the described contents. The disclosure also includes configurations that are not in these cases or when they are obvious to a person skilled in the art, and the patentee intends to obtain the rights to such configurations.

[0156] Furthermore, the processes and data flows described in the specification are not limited to the order in which they are described. For example, the patent also discloses configurations in which some processes are deleted or the order is changed, and the patent holder intends to obtain the rights to such configurations.

[0157] Furthermore, although the functions described in the embodiments are described as being executed by each device, they may be realized by a single device or may further utilize an external server. For example, they may be realized as a standalone device, such as an electronic device, including a set-top box as an example. In this case, the electronic device may have at least some of the functions shown in the drawings, with other functions provided in a device other than the electronic device. Alternatively, the electronic device may have all of these functions. In particular, when there are multiple language models, all of the multiple language models may be provided in the electronic device, or at least some of the language models may be provided in the electronic device, with other language models provided in a device other than the electronic device.

[0158] Furthermore, the display screens described in the above-described embodiments and drawings are merely examples, and are not intended to limit the displayed items, content, or layout. For example, the items displayed on the display screen can be rearranged, or the number of items can be changed. Furthermore, for the sake of convenience, the example display screen displays only a portion of the functions that can actually be realized. Further necessary information and items may be displayed, or conversely, items may be omitted as necessary.

[0159] Furthermore, each functional block or feature of the device used in the above-described embodiments may be implemented or performed by an electrical circuit, for example, an integrated circuit or multiple integrated circuits. The electrical circuit designed to perform the functions described herein may include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof. The general-purpose processor may be a microprocessor, or a conventional processor, controller, microcontroller, or state machine. The electrical circuit may be composed of digital circuits or analog circuits. Furthermore, as advances in semiconductor technology emerge, one or more aspects of the present disclosure may utilize new integrated circuits based on that technology.

[0160] 1 System 10 Display device 100 Control unit 110 Memory unit 112 ROM 114 RAM 116 Storage 120 Broadcast control unit 130 Operation unit 140 Display unit 150 Audio input unit 160 Audio output unit 170 Shooting unit 180 Communication unit 200 Volume adjustment unit 201 Receiving unit 203 Operation signal determination unit 205 Audio acquisition unit 207 Volume level adjustment unit 209 Volume level display control unit 20 Server device 22 Large-scale language model 30 Broadcast station 40 Distribution device 300 Remote control device 301 Control unit 303 Operation unit 305 Operation signal switching unit 307 Communication unit

Claims

1. A processing device comprising: a display control unit that controls the display of a display screen having a first display area in which acquired content can be displayed and a second display area in which an avatar corresponding to a character can be displayed; a voice input unit that accepts voice input from a user; a recognition unit that recognizes an input sentence from the content or the voice input; a processing unit that generates a response sentence corresponding to the input sentence; a voice generation unit that generates a second voice to be uttered by an avatar based on the response sentence; an adjustment unit that adjusts the volume level of a first voice included in the content and the volume level of the second voice; and a voice output unit that outputs at least one of the first voice and the second voice at the volume level adjusted by the adjustment unit.

2. The processing device according to claim 1, wherein the processing unit has a prompt generation unit that generates a prompt from the input sentence, and an output data acquisition unit that inputs the prompt into a language model and acquires output data, and generates the response sentence from the output data.

3. A processing device as described in claim 1, further comprising a receiving unit capable of receiving an operation signal for adjusting the volume level of a first sound and an operation signal for adjusting the volume level of a second sound, wherein the adjusting unit adjusts the volume level of at least one of the first sound and the second sound based on the operation signal received via the receiving unit.

4. The processing device according to claim 3, wherein the receiving unit adjusts the volume level of at least one of the first sound and the second sound based on the operation signal received from a remote control device.

5. The processing device according to claim 4, wherein the remote control device includes a switching means for receiving an instruction to switch between an operation signal for adjusting the volume level of the first sound and an operation signal for adjusting the volume level of the second sound, which are output as operation signals.

6. The processing device according to claim 3, wherein the display control unit is capable of displaying information based on the adjusted volume level on the display screen when the volume level is adjusted, and the information based on the volume level is displayed separately for the first audio and the second audio.

7. The processing device according to claim 1, wherein the adjustment unit assigns the output destination of the first audio and the output destination of the second audio to a plurality of output terminals, respectively, and adjusts the volume level of at least one of the first audio and the second audio for each of the assigned output terminals.

8. The processing device according to claim 1, wherein the adjustment unit controls the volume level of the second audio to be lowered when the first audio is being output, thereby restricting acceptance of audio input from the user.

9. The processing device according to claim 1, wherein the adjustment unit, when receiving an operation by the user to adjust the volume level of the first audio, adjusts the volume level of the second audio in accordance with the adjustment of the volume level of the first audio.

10. The processing device according to claim 1, wherein the adjustment unit controls the volume level of the second voice to be lowered when voice input from the user is received.

11. An audio output method in a processing device capable of controlling the display of a display screen having a first display area capable of displaying acquired content and a second display area capable of displaying an avatar corresponding to a character, the audio output method comprising: an audio input step of accepting a user's audio input; a recognition step of recognizing an input sentence from the content or the audio input; a processing step of generating a response sentence corresponding to the input sentence; an audio generation step of generating a second voice to be uttered by an avatar based on the response sentence; an adjustment step of adjusting the volume level of a first voice and the volume level of the second voice included in the content; and an audio output step of outputting at least one of the first voice and the second voice at the adjusted volume level.

12. A program that causes a computer capable of controlling the display of a display screen having a first display area capable of displaying acquired content and a second display area capable of displaying an avatar corresponding to a character to realize the following: a voice input function that accepts user voice input; a recognition function that recognizes an input sentence from the content or the voice input; a processing function that generates a response sentence corresponding to the input sentence; a voice generation function that generates a second voice to be uttered by an avatar based on the response sentence; an adjustment function that adjusts the volume level of a first voice included in the content and the volume level of the second voice; and a voice output function that outputs at least one of the first voice and the second voice at the adjusted volume level.

Images