Information processing apparatus

Abstract

The present technology relates to an information processing device that makes it possible to improve the convenience of a voice AI assistance service used in conjunction with content, comprising a processing circuit configured to control the time of a voice response when using a voice AI assistance service in conjunction with content based on an utterance of a viewer, the processing circuitry is configured to: present a wait state notification that notifies a viewer that the information processing apparatus is in a wait state; and causing the voice response in the waiting state to start when an instruction to start the voice response is received from the viewer; wherein the wait status notification is an indication signal, and wherein the indication signal is turned off when an instruction to start the voice response is received from the viewer.

Description

[0001] This application is a divisional application of the patent application with application number 201980021237.9 and invention name “Information processing device, information processing method, sending device and sending method”, which entered the national stage on September 22, 2020, and has an international application date of March 15, 2019 and international application number PCT / JP2019 / 010762. The entire contents of the application are incorporated herein by reference. Technical Field

[0002] The present technology relates to an information processing device, an information processing method, a sending device, and a sending method. Specifically, the present technology relates to an information processing device, an information processing method, a sending device, and a sending method that can improve the convenience of voice AI assistance services used in conjunction with content. Background Art

[0003] A broadcast application that runs in conjunction with broadcast content has been developed (see, for example, Patent Document 1). For example, use of the broadcast application makes it possible to display information related to the broadcast content.

[0004] Further, a technology for speech recognition that analyzes the content of a user's speech has been developed (for example, see Patent Document 2). For example, this technology applied to a television or a portable terminal device makes it possible to analyze the words uttered by the user and perform processing corresponding to the speech.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open Publication No. 2013-187781

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 2014-153663 Summary of the invention

[0009] Problem that the invention aims to solve

[0010] In recent years, voice AI-assisted services have rapidly become popular. There are cases where voice AI-assisted services are used in conjunction with broadcast content or broadcast applications played back in a receiver such as a television set. In this case, the output of a response voice to the speech of a viewer watching content such as broadcast content can overlap with the voice of the content, which may interfere with the viewer's viewing. As a result, there is a need for a technology that improves the convenience of voice AI-assisted services by eliminating interference with the viewer's viewing when outputting a response voice to the viewer's speech.

[0011] The present technology was made in view of this situation and is committed to improving the convenience of voice AI assisted services used in conjunction with content.

[0012] Solution to the problem

[0013] According to the first aspect of the present technology, an information processing device includes: a control unit, which is configured to control the time of a voice response based on voice response time information when using a voice AI assistance service in conjunction with content, the voice response time information indicating a time suitable for making a voice response to the words of a viewer watching the content.

[0014] The information processing apparatus according to the first aspect of the present technology can be an independent device or can be an internal module constituting one device. In addition, the information processing method according to the first aspect of the present technology is an information processing method corresponding to the above-mentioned information processing apparatus according to the first aspect of the present technology.

[0015] In an information processing method and an information processing device according to the first aspect of the present technology, when a voice AI assistance service is used in conjunction with content, the timing of the voice response is controlled based on voice response time information indicating a time suitable for making a voice response to the speech of a viewer viewing the content.

[0016] According to the second aspect of the present technology, a sending device includes: a generation unit, configured to generate metadata when using a voice AI assistance service in conjunction with content, the metadata including voice response time information indicating a time suitable for making a voice response to the speech of a viewer watching the content; and a sending unit, configured to send the generated metadata.

[0017] The transmission device according to the second aspect of the present technology can be an independent device or can be an internal module constituting one device. In addition, the transmission method according to the second aspect of the present technology is a transmission method corresponding to the above-mentioned transmission device according to the second aspect of the present technology.

[0018] In a sending device and a sending method according to the second aspect of the present technology, when a voice AI assistance service is used in conjunction with content, metadata including voice response time information indicating a time suitable for making a voice response to the speech of a viewer watching the content is generated, and the generated metadata is sent.

[0019] Effects of the Invention

[0020] According to the first and second aspects of the present technology, the convenience of voice AI assistance services used in conjunction with content can be improved.

[0021] It should be noted that the advantageous effects described herein are not restrictive and any advantageous effects described in the present disclosure may be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram showing an example of the configuration of an embodiment of a content-voice AI collaboration system to which the present technology is applied.

[0023] Figure 2 is a diagram illustrating an example of a response time of a voice AI assistance service used in conjunction with content.

[0024] Figure 3 is a diagram showing a first example of the configuration of the first embodiment.

[0025] Figure 4 is a block diagram showing a first example of a detailed configuration of each device according to the first embodiment.

[0026] Figure 5 is a diagram illustrating an example of communication between a voice assistance server and a processing server.

[0027] Figure 6 is a diagram illustrating an example of communication between a voice assistance server and a processing server.

[0028] Figure 7 is a diagram illustrating an example of communication between a voice-user interface device and a voice assistance server.

[0029] Figure 8 : is a flowchart showing the flow of a first example of describing a processing procedure in each device according to the first embodiment.

[0030] Fig. 9 is a diagram showing a second example of the configuration of the first embodiment.

[0031] Fig.10 is a block diagram showing a second example of the detailed configuration of each device according to the first embodiment.

[0032] Fig.11 is a diagram showing an example of metadata exchanged between a broadcast delivery system and a receiving device.

[0033] Fig.12 is a diagram showing an example of description of an MPD.

[0034] Fig.13 : is a flowchart showing the flow of a second example describing the processing procedures in the respective devices according to the first embodiment.

[0035] Fig.14 is a diagram showing an example of the configuration of the second embodiment.

[0036] Fig.15 is a block diagram showing an example of a detailed configuration of each device according to the second embodiment.

[0037] Fig.16 : is a flowchart showing a processing procedure in each device according to the second embodiment.

[0038] Fig.17 is a block diagram showing a first example of detailed configuration of each device according to the third embodiment.

[0039] Fig.18 : is a flowchart showing the flow of a first example describing a processing procedure in each device according to the third embodiment.

[0040] Fig.19 is a block diagram showing a second example of the detailed configuration of each device according to the third embodiment.

[0041] Fig. 20 : is a flowchart showing the flow of a second example describing the processing procedures in the respective devices according to the third embodiment.

[0042] Fig.21 is a block diagram showing an example of another configuration of a receiving device.

[0043] Fig. 22 is a diagram illustrating an example of a configuration of a computer. DETAILED DESCRIPTION

[0044] Embodiments of the present technology will now be described with reference to the accompanying drawings. Note that the description will be made in the following order.

[0045] 1. System Configuration

[0046] 2. Implementation of the present technology

[0047] (1) First embodiment: Configuration for time control of voice response based on voice response time

[0048] (2) Second Embodiment: Configuration for Time Control of Voice Response Based on Instruction Utterances from Viewer

[0049] (3) Third Embodiment: Configuration of performing time control of voice response based on instruction words from a viewer when performing time control of voice response based on voice response time

[0050] 3. Deformation

[0051] 4. Computer configuration

[0052] <1. System Configuration>

[0053] (Configuration of content-voice AI collaboration system)

[0054] Figure 1 is a block diagram showing an example of the configuration of an embodiment of a content-voice AI collaboration system to which the present technology is applied.

[0055] The content-voice AI collaboration system 1 is a system for delivering content and allows the use of voice AI-assisted services in conjunction with the delivered content.

[0056] exist Figure 1 In the embodiment, the content-voice AI collaboration system 1 includes a broadcast transmission system 10, a receiving device 20, a voice-user interface device 30, a voice assistance server 40 and a processing server 50.

[0057] Furthermore, in the content-voice AI collaborative system 1, the receiving device 20 and the voice-user interface device 30 installed in the viewer's house can be connected to various servers (such as a voice assistance server 40 and a processing server 50) installed in a data center, etc. via the Internet 60 for exchanging various data.

[0058] The broadcast delivery system 10 includes one or more broadcast servers, etc. The broadcast delivery system 10 performs necessary processing on the content to be transmitted and transmits a generated broadcast signal (broadcast wave) obtained by the processing from a transmission antenna installed at a transmission station.

[0059] For example, the receiving device 20 is configured as a fixed receiver such as a television and a personal computer, or a mobile receiver such as a smartphone, a cellular phone, and a tablet computer. The receiving device 20 receives and processes a broadcast signal transmitted from the broadcast delivery system 10 and plays back the transmitted content (e.g., such as a broadcast program or AD, etc.).

[0060] For example, the voice-user interface device 30 is a voice processing device (e.g., a speaker) capable of performing wireless communication such as Bluetooth (registered trademark), wireless local area network (LAN), and cellular communication, or wired communication, and is also referred to as a smart speaker or a home agent. For example, in addition to playing music, this type of speaker can be used as a user interface for a voice AI-assisted service, or can perform voice operations on instruments such as lighting devices or air conditioning devices.

[0061] The voice-user interface device 30 can provide voice AI assistance services to end users (viewers of the content) in conjunction with a server such as a voice assistance server 40 on the cloud. In this specification, for example, a voice AI assistance service is a function or service that responds to end users' questions or requests with appropriate replies or operations in conjunction with processing such as voice recognition or natural language analysis.

[0062] The voice assistance server 40 has the function of providing voice AI assistance services, various databases, etc. The voice assistance server 40 performs processing for providing voice AI assistance services in response to a request from the voice-user interface device 30 and replies the processing result (response) to the voice-user interface device 30 via the Internet 60.

[0063] The processing server 50 cooperates with the voice assistance server 40 to perform processing for providing voice AI assistance services, and sends the processing result (response) to the voice assistance server 40 via the Internet 60 or a network 70 (for example, a communication line such as a leased line). Moreover, the processing result obtained by the processing server 50 via the Internet 60 can be sent not only to the voice assistance server 40, but also to the receiving device 20 or the voice-user interface device 30.

[0064] The configuration of the content-voice AI collaboration system 1 is as described above.

[0065] By the way, there is a case where a viewer uses a voice AI assistance service to speak to an application about the content while watching content such as a broadcast program or a TV advertisement (commercial). In this case, the response voice to the viewer's speech overlaps with the sound of the content such as a broadcast program, which causes interference with the viewing / listening of the content.

[0066] For example, Figure 2 A scenario is shown where, in a case where a viewer 2 watches a drama program and wishes to know who an actress who appears in a specific scene is, the viewer uses a voice AI assistance service and utters the words “Who is this actress?”.

[0067] In this case, the voice-user interface device 30 picks up the speech of the viewer 2 through the microphone and sends the voice data of the speech to the voice assistance server 40 via the Internet 60. The voice assistance server 40 performs processing in conjunction with the processing server 50 provided for each broadcasting station to generate voice data for responding to the speech, and sends the voice data to the voice-user interface device 30 via the Internet 60.

[0068] In this description, there is a case where the voice-user interface device 30 receives voice data for response from the voice assistance server 40 and outputs a response voice corresponding to the voice data. In this case, because the output response voice overlaps in time with the words of an actress in the drama program that the viewer 2 watches using the receiving device 20, there is a possibility that the viewer 2 cannot hear the words uttered by the actress.

[0069] exist Figure 2In the example of , the response voice of "The name of this actress is XXXX and her main representative work is..." overlaps with the words "Yes! Let's go eat Indian food today" uttered by the actress. Thus, depending on the timing of the voice response, the voice AI assisted service may interfere with the viewer 2's viewing / listening to the drama program.

[0070] Thus, when using voice AI assisted services in conjunction with content such as broadcast programs, there is a need to improve the convenience of voice AI assisted services by making the voice response to the viewer's words a time required by the viewer.

[0071] Thus, the present technology allows the time of the voice response to be controlled based on information indicating the time suitable for voice response to the viewer's speech (hereinafter referred to as voice response time information). As a result, the convenience of voice AI assistance services used in conjunction with content such as broadcast programs can be improved. The first to third embodiments of the present technology are now described.

[0072] <2. Embodiments of the present technology>

[0073] (1) First Embodiment

[0074] First, refer to Figures 3 to 13 , a configuration for controlling the timing of a voice response based on voice response time information acquired via communication or via broadcasting is described as a first embodiment.

[0075] (First example of configuration)

[0076] Figure 3 is a diagram showing a first example of the configuration of the first embodiment.

[0077] Figure 3 Shown in accordance with the use Figure 3 The time series of parts A to C in the video is processed according to the progress of the broadcast program watched by viewer 2.

[0078] At 10:10:00 (10h:10m:00s), when the viewer 2 utters the words "Who is this actress?" while watching the drama program, the user interface device 30 picks up the viewer 2's words through the microphone and sends the voice data of the words as a question request to the voice assistance server 40 ( Figure 3 Part A).

[0079] The voice assistance server 40 generates voice data of a response to the utterance by performing processing in conjunction with the processing server 50 provided by the broadcasting station that produces the drama program being viewed. Furthermore, in the present description, voice response time information indicating a time suitable for a voice response is generated as voice response time metadata together with the voice data for the response. Then, the voice response time metadata (voice response time information) is sent to the user interface device 30 ( Figure 3 Part B).

[0080] At 10:10:02 (10h:10m:02s), the user interface device 30 receives the voice response time metadata and the voice data for the response from the voice assistance server 40, but the time (time of the day) indicated by the voice response time information is 10:10:10, so it waits until that time (time of the day) ( Figure 3 In other words, for example, in the drama program being watched, a dialogue scenario occurs at about 10:10:02, and therefore, the timing of the voice response is changed to prevent it from overlapping with the actress's words "Yes! Let's go eat Indian food today!"

[0081] Then, when the time reaches 10:10:10, the time (time of the day) being the time indicated by the voice response time information, the user interface device 30 outputs a response voice "The name of this actress is XXXX and the main representative work is ..." based on the response voice data stored in the buffer during the waiting period ( Figure 3 Part C in FIG. 1 ). In this case, the drama program being watched is not a dialogue scene, and therefore, it is impossible for the viewer 2 to interfere with the watching of the drama program.

[0082] Thus, the user interface device 30 can perform a voice response at a time suitable for the voice response, that is, at a time that does not involve interruptions to viewing, such as interruptions in dialogue of the program being viewed, etc. This is performed based on the voice response time metadata (voice response time information) obtained via communication. This enables the viewer 2 to listen to the response voice (the information desired to know) without interrupting viewing of the program.

[0083] Furthermore, for ease of understanding Figure 3 In the description of , a description is given of a case where actual time is used as an example. However, in reality, the time of the voice response can be controlled using the media time of the content played in the receiving device 20.

[0084] (First Example of Detailed Configuration of Each Device)

[0085] Figure 4is a block diagram showing a first example of a detailed configuration of each device according to the first embodiment.

[0086] Figure 4 An example of the configuration of a broadcast delivery system 10 provided on the broadcast station side (transmission side) and a receiving device 20 provided on the local side (reception side) is shown. Figure 4 An example of the configuration of the voice-user interface device 30 provided on the local side and the configuration of the voice assistance server 40 and the processing server 50 provided on the cloud is shown.

[0087] exist Figure 4 In the embodiment, the broadcast delivery system 10 includes a program content processing unit 101, a broadcast-accompanying application generating unit 102, a multiplexer 103, and a transmitting unit 104.

[0088] Program content processing unit 101 performs necessary processing (such as AV encoding, etc.) on content input thereto and supplies the result to multiplexer 103. Also, for example, content is a broadcast program, advertisement, etc., and is acquired from an external server, camera, recording medium, etc.

[0089] The broadcast-accompanying application generating unit 102 generates a broadcast-accompanying application and supplies it to the multiplexer 103. In the present description, the broadcast-accompanying application is an application that runs in conjunction with content such as a broadcast program.

[0090] The multiplexer 103 performs multiplexing on the content supplied from the program content processing unit 101 and the broadcast-accompanying application supplied from the broadcast-accompanying application generating unit 102 and supplies the generated stream to the transmitting unit 104 .

[0091] The transmission unit 104 performs necessary processing (such as error correction coding or modulation processing, for example) on the stream supplied from the multiplexer 103, and transmits the generated broadcast signal (broadcast stream) through a transmission antenna installed at a transmission station.

[0092] exist Figure 4 In the embodiment, the receiving device 20 includes a tuner 201, a demultiplexer 202, an AV decoder 203, an application execution unit 204 accompanying the broadcast, a renderer 205, a display 206, a speaker 207, a media-time management unit 208, and a communication unit 209.

[0093] The tuner 201 receives a broadcast signal (broadcast stream) transmitted from the broadcast delivery system 10 via a receiving antenna installed in a viewer's house or the like, performs necessary processing thereon (such as demodulation processing or error correction decoding, for example), and provides the generated stream to the demultiplexer 202.

[0094] The demultiplexer 202 separates the stream supplied from the tuner 201 into a video stream, an audio stream, and a stream of an application accompanying broadcasting. The demultiplexer 202 supplies the video and audio streams to the AV decoder 203 and supplies the stream of the application accompanying broadcasting to the application execution unit 204 accompanying broadcasting.

[0095] The AV decoder 203 decodes the video stream supplied from the demultiplexer 202 and supplies the result to the renderer 205. Furthermore, the AV decoder 203 also decodes the audio stream supplied from the demultiplexer 202 and supplies the result to the renderer 205.

[0096] The renderer 205 performs rendering on the video data supplied from the AV decoder 203 and causes the generated video to be displayed on a display 206 such as a liquid crystal display (LCD) or an organic light emitting diode (OLED).

[0097] Further, the renderer 205 performs rendering on the voice data supplied from the AV decoder 203 and outputs the generated audio through the speaker 207. In the receiving device 20, this allows displaying a video of content such as a broadcast program on the display 206 and outputting audio synchronized with the video through the speaker 207.

[0098] For example, the broadcasting application execution unit 204 is a browser compatible with HTML5, JavaScript (registered trademark), etc. The broadcasting application execution unit 204 executes the broadcasting application based on the application data supplied from the demultiplexer 202 and supplies video data to the renderer 205 .

[0099] The renderer 205 performs rendering on the video data supplied from the broadcast-accompanying application execution unit 204 and causes the generated video to be displayed on the display 206. In the reception device 20, this allows the video of the broadcast-accompanying application to be displayed on the display 206 in combination with the content such as the broadcast program.

[0100] The media-time management unit 208 manages the media time of the content played in the receiving device 20. The media-time management unit 208 provides the current media time of the content played in response to the query from the application execution unit 204 accompanying the broadcast.

[0101] The communication unit 209 is configured as a communication module compatible with wireless communication such as wireless LAN or cellular communication, or wired communication. The communication unit 209 communicates with the processing server 50 (the communication unit 501 thereof) via the Internet 60 according to a predetermined communication scheme to exchange various data. In the present description, for example, communication is performed using WebSocket or the like, and the communication unit 209 is capable of transmitting the current media time provided from the application execution unit 204 accompanying the broadcast to the processing server 50 (the communication unit 501 thereof).

[0102] exist Figure 4 , the voice-user interface device 30 includes a microphone 301, a speaker 302, a communication unit 303, a voice response output control unit 304, a timer 305, a buffer 306, and a setting management unit 307.

[0103] The microphone 301 picks up the speech of the viewer 2 and supplies voice data of the speech to the communication unit 303 .

[0104] The communication unit 303 is configured as a communication module that is compatible with wireless communication such as wireless LAN and cellular communication, or wired communication.

[0105] The communication unit 303 transmits the voice data of the utterance provided from the microphone 301 to the voice assistance server 40 via the Internet 60. In addition, the communication unit 303 receives the voice data for response and the voice response time metadata (voice response time information) transmitted from the voice assistance server 40 via the Internet 60 and provides them to the voice response output control unit 304. Moreover, although it will be described in detail later, the voice data for response and the voice response time metadata are included and transmitted in the second response message.

[0106] The voice data for response and the voice response time metadata are supplied from the communication unit 303 to the voice response output control unit 304. The voice response output control unit 304 supplies the voice data for response to the speaker 302 at a time corresponding to the time indicated by the voice response time information based on the voice response time metadata acquired via communication.

[0107] In this description, the time indicated by the voice response time information is transmitted from the voice response output control unit 304 to the timer 305 to measure the time, and the buffer 306 temporarily stores the voice data for the response until the time suitable for the voice response is reached. Then, when the time suitable for the voice response is reached, the voice response output control unit 304 reads the voice data for the response temporarily stored in the buffer 306 and provides it to the speaker 302.

[0108] The speaker 302 outputs a response voice corresponding to the voice data for response supplied from the voice response output control unit 304 .

[0109] The setting management unit 307 performs various settings according to the setting input of the viewer 2. In the present description, for example, when outputting a response voice, the setting etc. is constituted by time control of whether to wait until a time suitable for a voice response. Also, in the present description, the setting input is received by the viewer 2 operating input with a button or a touch panel (not shown), or the viewer 2 receives the setting input by voice input of the microphone 301.

[0110] exist Figure 4 , the speech assistance server 40 includes a communication unit 401 , a communication unit 402 , a speech analysis unit 403 , and a message generation unit 404 .

[0111] The communication unit 401 receives voice data of an utterance transmitted from the voice-user interface device 30 via the Internet 60 and supplies it to the voice analysis unit 403 .

[0112] The speech analysis unit 403 analyzes the speech data of the utterance supplied from the communication unit 401 and converts the speech data of the utterance into text data. The speech analysis unit 403 supplies a request message corresponding to the analysis result of the speech data of the utterance to the communication unit 402.

[0113] Communication unit 402 transmits the request message supplied from speech analysis unit 403 to processing server 50 via network 70. Furthermore, communication unit 402 receives the first response message transmitted from processing server 50 via network 70 and supplies the first response message to message generation unit 404.

[0114] The message generation unit 404 converts the first response message provided from the communication unit 402 to generate a second response message for responding to the voice-user interface device 30 , and provides it to the communication unit 401 .

[0115] Moreover, in the present description, for the convenience of description, the response message sent from the processing server 50 and received by the voice assistance server 40 is different from the response message sent from the voice assistance server 40 and received by the voice-user interface device 30. In order to distinguish them, the former is referred to as a first response message, and the latter is referred to as a second response message.

[0116] The message generating unit 404 includes a voice generating unit 404-1 and a metadata merging unit 404-2. In addition, the first response message includes voice response data and voice response time metadata (voice response time information).

[0117] The speech generating unit 404-1 converts the speech response data included in the first response message transmitted from the processing server 50 into speech data for response. The metadata merging unit 404-2 merges the speech response time metadata (speech response time information) included in the first response message transmitted from the processing server 50 with the second response message including the speech data for response.

[0118] Thus, in the message generation unit 404, the voice generation unit 404-1 and the metadata merging unit 404-2 process the first response message (e.g., HTTP response data) including the text-format data (voice response data and voice response time metadata). Thus, a second response message (e.g., HTTP response) including the voice-format data (response voice data) and the text-format data (voice response time metadata) is generated. Figures 5 to 7 Describe the details of the first response message and the second response message.

[0119] The communication unit 410 transmits the second response message provided from the management generation unit 404 to the voice-user interface device 30 via the Internet 60 .

[0120] Furthermore, the communication unit 401 and the communication unit 402 are configured as a communication module, that is, compatible with wireless communication or wired communication. Figure 4 In the present invention, for the convenience of description, the communication unit 401 and the communication unit 402 are described as independent communication modules, but they can be integrated into one communication module.

[0121] exist Figure 4 In the embodiment, the processing server 50 includes a communication unit 501 , a processing unit 502 , and a recording unit 503 .

[0122] The communication unit 501 receives a request message transmitted from the voice assistance server 40 via the network 70 and provides it to the processing unit 502. In addition, the communication unit 501 receives the current media time transmitted from the receiving device 20 by communicating using WebSocket or the like and provides it to the processing unit 502. Moreover, the communication unit 501 is configured as a communication module, which is compatible with wireless communication or wired communication.

[0123] The processing unit 502 extracts the voice response data recorded in the recording unit 503 based on the request message supplied from the communication unit 501 and supplies the result to the communication unit 501. The voice response data is data that voice responds to the utterance of the viewer 2.

[0124] Further, the processing unit 502 extracts the voice response time information indicating the time suitable for the voice response from the voice response time metadata (metadata of the entire program) recorded in the recording unit 503 based on the current media time provided from the communication unit 501. Then, the processing unit 502 provides it to the communication unit 501 as the voice response time metadata.

[0125] The recording unit 503 is a recording medium such as a semiconductor memory, an HDD, or an optical disk, and can record therein various data such as voice response data and voice response time metadata. Moreover, the voice response data and voice response time metadata are not limited to the data recorded in the recording unit 503, and, for example, can be generated by the processing unit 502 based on various types of data such as data acquired from an external server or recorded in the recording unit 503 or data acquired from an external server.

[0126] The communication unit 501 transmits the voice response data and the voice response time metadata provided from the processing unit 502 to the voice assistance server 40 via the network 70 as a first response message.

[0127] Moreover, the content-voice AI collaboration system 1 executes a program called a skill, which makes it possible to use the voice AI assistance service in conjunction with content such as a broadcast program. For example, in the voice assistance server 40, a target skill is specified from among a plurality of skills and the target skill is combined with the processing unit 502 of the processing server 50 through the endpoint URL of the specified target skill. Thus, processing of making a voice response to the viewer's speech is performed. In other words, it can be said that the processing unit 502 constitutes a part of the target skill.

[0128] In this description, skills include information such as what kind of voice is responded to, what function uses what words as parameters, or which processing server (processing program) actually executes the function. In addition, skills refer to a program (computer program) for executing a process corresponding to the voice sent from the voice-user interface device 30 based on the above-mentioned information.

[0129] The configuration of the content-voice AI collaboration system 1 is as described above, and now reference is made to Figures 5 to 7 A description is given in detail of the voice response data and the voice response time metadata exchanged through communication between the voice assistance server 40 and the processing server 50.

[0130] like Figure 5 As shown in , in the processing server 50, the voice response time metadata transmitted via communication is recorded in the recording unit 503. The voice response time metadata is provided as metadata of the entire program of each broadcast program transmitted via broadcasting. Figure 5The example in shows the following four speech-response times.

[0131] First voice response time: 2018-01-29 18:00:05 to 2018-01-29 18:00:55

[0132] Second voice response time: 2018-01-29 18:01:05 to 2018-01-29 18:01:55

[0133] Third voice response time: 2018-01-29 18:02:05 to 2018-01-29 18:02:55

[0134] Fourth voice response time: 2018-01-29 18:03:05 to 2018-01-29 18:03:55 ...

[0136] The processing unit 502 obtains the current media time by using the WebSocket communication between the receiving device 20 and the processing server 50, and thus can extract only one voice response time suitable for voice response from the metadata of the entire program based on the media time. Then, the processing unit 502 sends the extracted one voice response time (voice response time information) as voice response time metadata to the voice assistance server 40 through a message of a predetermined format of an HTTP response, etc.

[0137] like Figure 6 As shown in, in this description, when the third voice response time is extracted from the metadata of the entire program as the time suitable for the voice response, there is a case where an HTTP response is used as the first response message. In this case, text data of the extracted third voice response time is described in the body of the HTTP response. In addition, text data of the voice response data can be described in the body of the HTTP response.

[0138] For example, the text data described above can be described as data in the format of JavaScript (registered trademark) Object Notation (JSON). A JSON format object is expressed by key-value pairs with colons (:), and these pairs are separated by commas (,), zero or more pairs are listed, and the whole is enclosed in curly braces ({}).

[0139] exist Figure 6In the example of , regarding the object of voice response data ("output language"), the "plain text" format is described as the "type" value, and the text data of "XXXX" is described as the "text" value. In addition, regarding the object of voice response time ("responseTiming"), the start time "2018-01-29T18:02:05Z" is described as the "start" value and the end time "2018-01-29T18:02:55Z" is described as the "end" value.

[0140] Further, the voice assistance server 40 receives and processes the HTTP response. For example, in the case of using the HTTP response as the second response message, the following is performed: Figure 7 The processing shown in .

[0141] In other words, the voice assistance server 40 converts the text data of the voice response data described in the body of the HTTP response received from the processing server 50 into voice data for the response, and generates a multi-part HTTP response by merging it with the text data of the voice response time (voice response time information).

[0142] Thus, in the voice assistance server 40, the first response message (HTTP response) received from the processing server 50 is processed, and the text format data (voice response data and voice response time metadata (voice response time information)) is converted into voice format data (response voice data) and text format data (voice response time metadata (voice response time information)). Then, the voice assistance server 40 sends the converted result as a second response message (HTTP response in multi-part format) to the voice-user interface device 30.

[0143] Furthermore, although metadata of the entire program is provided as voice response time metadata transmitted via communication, voice response time can be prepared for all time zones or some time zones of a specific program. In addition, in an embodiment of the present technology, voice response time metadata can represent metadata of the entire program or can represent voice response time information extracted from metadata, and therefore, when a distinction must be made, it is clearly indicated.

[0144] (Procedure of First Example of Processing in Each Device)

[0145] Reference Figure 8 The flowchart in describes the procedure of the first example of the processing in each device of the first embodiment.

[0146] In this description, the process of step S101 is performed as a presetting. In other words, in the voice-user interface device 30, the setting management unit 307 is set to turn on the automatic timing control of the output of the response voice according to the instruction of the viewer 2.

[0147] After the presetting, the broadcast delivery system 10 sends a broadcast stream (S131). In this case, in the broadcast delivery system 10, the application program generation unit 102 accompanying the broadcast generates an application program accompanying the broadcast, and the multiplexer 103 inserts the generated application program accompanying the broadcast into a stream including content such as a broadcast program.

[0148] The receiving device 20 receives the broadcast stream, plays the content such as a broadcast program, and performs the process of step S132. In step S132, the application execution unit 204 accompanying the broadcast executes the application accompanying the broadcast based on the application data obtained from the stream separated by the demultiplexer 202. This allows the receiving device 20 to play back the broadcast program selected by the viewer 2 and execute the application accompanying the broadcast linked to the program.

[0149] Therefore, in the case where the viewer 2 who is viewing the broadcast program utters a question utterance, the processing operations in step S102 and subsequent steps are performed.

[0150] Specifically, in the voice-user interface device 30, the microphone 301 picks up the speech of the viewer 2, and sends the speech data of the speech to the voice assistance server 40. Then, in the voice assistance server 40, the speech data of the speech from the voice-user interface device 30 is analyzed, and a request message corresponding to the analysis result is sent to the processing server 50.

[0151] The processing server 50 receives the request message via the network 70, and performs the processing operations of steps S103 and S104.

[0152] In step S103, the processing unit 502 obtains the current media time based on the request message from the voice assistance server 40. At this stage, for example, the processing server 50 (the communication unit 501 thereof) and the receiving device 20 (the communication unit 209 thereof) communicate with each other according to the communication scheme of WebSocket, etc. Thus, the processing unit 502 can obtain the current media time from the accompanying broadcast application running unit 204 that runs the accompanying broadcast application.

[0153] In step S104, the processing unit 502 extracts voice response time information indicating a time suitable for voice response from metadata of the entire program recorded in the recording unit 503 based on the acquired current media time. Then, the processing unit 502 sends the extracted voice response time information together with the voice response data corresponding to the request message as a first response message to the voice assistance server 40.

[0154] In the present description, for example, in addition to the time that does not interfere with viewing, such as the interruption of the developing dialogue of the program being watched, the time that can be regarded as suitable for voice response also includes the time length of the response voice (response length). In other words, even if it does not interfere with viewing, in the case where the output of the response voice cannot be completed within this time, another longer time can be extracted as the time suitable for voice response. In addition, for example, on the playback timeline of the content being played in the receiving device 20, there is a situation where the latest time zone can be extracted as the time suitable for voice response. Even in this case, when it takes time to output the response voice after the viewer's speech, an earlier time zone can be extracted as the time suitable for voice response.

[0155] In the voice assistance server 40, voice data for response is generated from the voice response data based on the first request message from the processing server 50, and the voice response time information is merged with the voice data for response to generate a second response message. The voice-user interface device 30 receives the second response message via the Internet 60 and performs the processing operations in steps S105 to S108.

[0156] In step S105, the voice response output control unit 304 confirms whether to add voice response time information to the voice data for response based on the second response message from the voice assistance server 40. In the case where it is determined in step S105 that the voice response time information is added ("Yes" in S105), the process proceeds to step S106.

[0157] In step S106, the voice response output control unit 304 confirms (judges) whether the automatic timing control is set to on by making an inquiry to the setting management unit 307. In a case where it is determined in step S106 that the automatic timing control is set to on by, for example, the presetting process ("Yes" in S106), the process proceeds to step S107.

[0158] In step S107, the voice response output control unit 304 waits until a time suitable for voice response is reached based on the added voice response time information. Then, when a time suitable for voice response is reached, the voice response output control unit 304 outputs a response voice by providing voice data for response to the speaker 302 (S108).

[0159] Thus, the voice-user interface device 30 can be made to perform voice response at a time suitable for voice response, that is, at a time that does not interfere with viewing, such as a break in a developing dialogue of a program being viewed, etc. This is performed based on the voice response time metadata (voice response time information) acquired via communication. This makes it possible for the viewer 2 to listen to the response voice (the information he wishes to know) without interfering with the viewer's viewing of the program.

[0160] Furthermore, in the voice-user interface device 3, in the case where it is judged that the voice response time information is not added (in S105, "No"), or in the case where it is judged that the automatic timing control is set to off (in S106, "No"), the process of step S107 is skipped and the process of step S108 is performed. In other words, in this case, the response voice is output immediately without waiting for the voice response time.

[0161] The above description of the processing procedure is given as a procedure of a first example of processing in each device of the first embodiment in the case of controlling the timing of the voice response to the utterance of the viewer 2 based on the voice response time metadata (voice response time information) acquired via communication.

[0162] (Second example of configuration)

[0163] Fig. 9 is a diagram showing a second example of the configuration of the first embodiment.

[0164] and Figure 3 resemblance, Fig. 9 According to the reference Fig. 9 The time series of parts A to C in FIG. 2 shows the processing process according to the progress of the broadcast program being viewed by the viewer 2. On the other hand, Fig. 9 The processing shown in Figure 3 The processing shown in is different in that the voice data and voice response time metadata for the response are sent via broadcast rather than via communication.

[0165] The receiving device 20 acquires and records the voice response time metadata (metadata of the entire program) transmitted from the broadcast delivery system 10 together with the content such as the broadcast program and the application accompanying the broadcast. In response to an inquiry from the user interface device 30, the receiving device 20 is able to extract the voice response time information indicating the time suitable for the voice response from the metadata of the entire recorded program and notify it as the voice response time metadata.

[0166] At 10:10:00 (10h:10m:00s), when the viewer 2 utters the words "Who is this actress?" while watching the drama program, the user interface device 30 picks up the viewer 2's words through the microphone and sends the voice data of the words to the voice assistance server 40 ( Fig. 9 Part A).

[0167] At 10:10:02 (10h:10m:02s), the user interface device 30 receives voice data for response from the voice assistance server 40, but based on the voice response time metadata acquired from the receiving device 20, the time (time of the day) indicated by the voice response time information is 10:10:10, and therefore, it waits until that time (time of the day) ( Fig. 9 In other words, in the drama program being viewed, a dialogue scene occurs at about 10:10:02, and therefore, the time of the voice response is shifted to prevent it from overlapping with the actress' words or the like.

[0168] Then, when the time reaches 10:10:10, which is the time indicated by the voice response time information (time of the day), the user interface device 30 outputs a response voice "The name of this actress is XXXX and the main representative work is ..." based on the voice data for response stored in the buffer during the waiting period ( Fig. 9 Part C).

[0169] Thus, the user interface device 30 can be caused to perform a voice response at a time suitable for the voice response, that is, at a time that does not interfere with viewing, such as a development dialogue interruption of the program being viewed, etc. This is performed based on the voice response time metadata (voice response time information) obtained via the broadcast. This enables the viewer 2 to listen to the response voice (the information desired to know) without interfering with viewing the program.

[0170] Moreover, with Figure 3 Similarly, a description of the case where real time is used is also given as Fig. 9 However, in practice, the timing of the voice response can be controlled using the media timing of the content played in the receiving device 20.

[0171] (Second Example of Detailed Configuration of Each Device)

[0172] Fig.10 is a block diagram showing a second example of the detailed configuration of each device according to the first embodiment.

[0173] and Figure 4 resemblance, Fig.10An example of the configuration of a broadcast delivery system 10 provided on a broadcast station side (transmission side) and a reception device 20 provided on a local side (reception side) is shown. Fig.10 An example of the configuration of the voice-user interface device 30 provided on the local side and the configuration of the voice assistance server 40 and the processing server 50 provided on the cloud is shown.

[0174] exist Fig.10 In the embodiment, the broadcast delivery system 10 includes a program content processing unit 101, a broadcast accompanying application generating unit 102, a multiplexer 103, a transmitting unit 104, and a metadata generating unit 105. In other words, Figure 4 Compared to the configuration shown in Fig.10 In the broadcast delivery system 10 shown, a metadata generating unit 105 is added.

[0175] The metadata generating unit 105 generates voice response time metadata and supplies it to the multiplexer 103. The voice response time metadata is metadata for the entire program of each program transmitted via broadcasting, and includes metadata candidates extracted as voice response time information including a time suitable for a voice response.

[0176] The multiplexer 103 multiplexes the voice response time metadata supplied from the metadata generating unit 105 together with the content and the application accompanying the broadcast, and supplies the generated stream to the transmitting unit 104 .

[0177] exist Fig.10 In the embodiment, the receiving device 20 includes a tuner 201, a demultiplexer 202, an AV decoder 203, an application program execution unit 204 accompanying the broadcast, a renderer 205, a display 206, a speaker 207, a media-time management unit 208, a communication unit 210, a voice response time metadata management unit 211, and a recording unit 212. In other words, Figure 4 Compared to the configuration shown in Fig.10 The receiving device 20 shown in FIG. 2 has a communication unit 210 , a voice response time metadata management unit 211 , and a recording unit 212 as additional components, instead of the communication unit 209 .

[0178] For example, the communication unit 210 is configured as a communication module, which is compatible with wireless communication such as Bluetooth (registered trademark), wireless LAN, and cellular communication, or wired communication. The communication unit 210 communicates with the voice-user interface device 30 (specifically, its communication unit 309) according to a predetermined communication scheme to exchange various data.

[0179] The voice response time metadata management unit 211 acquires the voice response time metadata (metadata of the entire program) obtained from the stream separated by the demultiplexer 202 and records it in the recording unit 212 .

[0180] Further, in the case of receiving a voice response time metadata acquisition request from the voice-user interface device 30 (its voice response output control unit 304), the voice response time metadata management unit 211 inquires the media-time management unit 208 to obtain the current media time. Based on the acquired current media time, the voice response time metadata management unit 211 extracts voice response time information indicating the time suitable for voice response from the voice response time metadata (metadata of the entire program) recorded in the recording unit 212. Then, the voice response time metadata management unit 211 sends the voice response time information as voice response time metadata to the voice-user interface device 30.

[0181] The recording unit 212 is a recording medium such as a semiconductor memory, an HDD, or an optical disk, and can record therein various data including voice response time metadata.

[0182] exist Fig.10 , the voice-user interface device 30 includes a microphone 301, a speaker 302, a communication unit 303, a voice response output control unit 304, a timer 305, a buffer 306, a setting management unit 307, a voice analysis unit 308, and a communication unit 309. In other words, Figure 4 Compared to the configuration shown in Fig.10 In the illustrated voice-user interface device 30 , a voice analysis unit 308 and a communication unit 309 are added.

[0183] The speech analysis unit 308 processes the speech data of the speech supplied from the microphone 301 and supplies it to the communication unit 303. Furthermore, the speech analysis unit 308 analyzes the speech data of the speech and performs timing control target speech setting based on the analysis result of the speech data of the speech in conjunction with the setting management unit 307.

[0184] For example, the communication unit 309 is configured as a communication module that supports wireless communication such as Bluetooth (registered trademark) or wired communication. The communication unit 309 communicates with the receiving device 20 (the communication unit 210 thereof) according to a predetermined communication scheme to exchange various data. Fig.10 In the embodiment, the communication unit 303 and the communication unit 309 are described as independent communication modules, however, the communication unit 303 and the communication unit 309 can be integrated into one communication module.

[0185] The voice response output control unit 304 is supplied with voice data for response from the communication unit 303 and the voice response time metadata is supplied to the voice response output control unit 304 from the communication unit 309. The voice response output control unit 304 supplies the voice data for response to the speaker 302 at a time corresponding to a time indicated by the voice response time information (a time suitable for a voice response) based on the voice response time metadata acquired via the broadcast.

[0186] exist Fig.10 In FIG. 4 , the speech assistance server 40 includes a communication unit 401, a communication unit 402, a speech analysis unit 403, and a speech generation unit 404-1. Fig.10 In the voice assistance server 40 shown, the voice response time metadata is not transmitted via communication and therefore no processing related thereto needs to be performed. Figure 4 The configuration is different in Figure 4 The metadata merging unit 405-2 of the message generating unit 404 shown in FIG.

[0187] Further, in Fig.10 In the example, the processing server 50 includes a communication unit 501, a processing unit 502, and a recording unit 503. In other words, Fig.10 The processing server 50 shown in FIG. Figure 4 A configuration similar to the configuration shown in , but the voice response time metadata is not transmitted via communication, and therefore, in the recording unit 503, the voice response time metadata is not recorded, but only the voice response data is recorded.

[0188] The configuration of the content-voice AI collaboration system 1 is as described above. Fig.11 and Fig.12 A description is given in detail of exchanging voice response time metadata between the broadcast delivery system 10 and the receiving device 20 via broadcasting.

[0189] like Fig.11 As shown in , in the broadcast delivery system 10, the metadata generation unit 105 generates voice response time metadata delivered via broadcasting. For each program delivered via broadcasting (excluding live broadcasting), the voice response time metadata is generated as metadata for the entire program.

[0190] First voice response time: 2018-01-29 18:00:05~2018-01-29 18:00:55

[0191] Second voice response time: 2018-01-29 18:01:05~2018-01-29 18:01:55

[0192] Third voice response time: 2018-01-29 18:02:05~2018-01-29 18:02:55

[0193] Fourth voice response time: 2018-01-29 18:03:05~2018-01-29 18:03:55 ...

[0195] The multiplexer 103 converts the voice response time metadata into a format suitable for the content and inserts it into the stream in which the content and the application accompanying the broadcast are multiplexed.

[0196] In the present description, for example, in the case where content is delivered as a stream compatible with MPEG-Dynamic Adaptive Streaming over HTTP (DASH), voice response time metadata can be sent using a Media Presentation Description (MPD), which is control information for a video or audio file.

[0197] Fig.12 An example in which speech response time metadata is inserted into an MPD is shown. In this example, the MPD includes elements of a period (Period), an adaptation set (AdaptationSet), and a representation (Representation) in a hierarchical structure.

[0198] The element Period is an element that describes the configuration of content such as a broadcast program, etc. Furthermore, the elements AdaptationSet and Representation used for each stream of components such as video, audio, and subtitles constituting the content can describe the attributes of each stream.

[0199] Furthermore, the element EventStream may be described in the element Period. In addition, in the element EventStream, the attributes schemeIdUri and timescale may be described as its attributes.

[0200] In the attribute schemeIdUri, the Uniform Resource Identifier (URI) used to identify the scheme is described. Fig.12 In the MPD description example of , "urn:XXX" is defined as a URI for identifying a scheme for sending voice response time metadata and "urn:XXX" is described as an attribute value of the attribute schemeIdUri. In addition, in the attribute timescale, timescale "1000" is described as its attribute value.

[0201] The element EventStream is an upper element of the element Event. In the element Event, the attribute presentationTime in which the presentation time (start time) is specified and the attribute duration in which the time period from the start time is specified can be described as its attributes.

[0202] exist Fig.12 In the MPD description example, the metadata of the entire program is divided into individual element time periods corresponding thereto and the metadata of the entire program is described as the attribute values ​​of the attributes presentationTime and duration in the element EventStream.

[0203] Specifically, the first voice response time and the second voice response time described in the metadata of the entire program are described in the first Period element (attribute start = 'PT0S'), and the third voice response time and the fourth voice response time are described in the second Period element (attribute start = 'PT2M0S').

[0204] As described above, using the MPD to transmit the voice response time metadata enables the receiving device 20 to obtain the voice response time metadata (metadata of the entire program) from the MPD inserted in the broadcast stream and record it in the recording unit 212. Then, the receiving device 20 can extract the voice response time information indicating the time suitable for the voice response from the voice response time metadata (metadata of the entire program) recorded in the recording unit 212 in response to an inquiry from the voice-user interface device 30 and notify it as the voice response time metadata.

[0205] Furthermore, although metadata of the entire program is provided as the voice response time metadata transmitted via broadcasting, the voice response time can be prepared for all time zones or some time zones of a specific program.

[0206] (Procedure of Second Example of Processing in Each Device)

[0207] Reference Fig.13 The flowchart in describes the procedure of the second example of the processing in each device of the first embodiment.

[0208] In this description, the processing of step S151 is performed as a pre-setting. In other words, in the voice-user interface device 30, the setting management unit 307 is set to turn on the automatic timing control according to the instruction of the viewer 2. In addition, in this description, the call name of the timing control target or the destination for acquiring the voice response time metadata (the destination of the metadata acquisition communication) is set.

[0209] Moreover, the call name is a call name for calling a skill. In other words, the end user (viewer watching the content) needs to say the call name when using the skill. In this description, for example, "TV show" can be set as the call name of the timing control target, and the receiving device 20 can be set as the destination for metadata acquisition communication.

[0210] After the presetting, the processing operations of steps S181 to S182 are performed in the broadcast delivery system 10 .

[0211] In other words, the metadata generation unit 105 generates voice response time metadata (metadata of the entire program), and the multiplexer 103 inserts the generated voice response time metadata into a stream including content such as a broadcast program (S181). In addition, the transmission unit 104 transmits the broadcast stream into which the voice response time metadata is inserted (S182).

[0212] The broadcast stream is received by the receiving device 20, content such as a broadcast program is played, and the process of step S183 is performed. In step S183, the voice response time metadata management unit 211 acquires the voice response time metadata obtained from the stream separated by the demultiplexer 202. The voice response time metadata (metadata of the entire program) is recorded in the recording unit 212.

[0213] Then, in the case where the viewer 2 viewing the broadcast program utters a question utterance, the processing operation of step S152 and subsequent steps are performed.

[0214] Specifically, in the voice-user interface device 30, the microphone 301 picks up the speech of the viewer 2 and sends the speech data of the speech to the voice assistance server 40 (S152). Then, in the voice assistance server 40, the speech data of the speech from the voice-user interface device 30 is analyzed, and a request message corresponding to the analysis result is sent to the processing server 50.

[0215] In this case, in the voice-user interface device 30, the voice analysis unit 308 cooperates with the setting management unit 307 to extract the target call name set by the preset processing from the speech of the viewer 2 based on the analysis result of the voice data of the speech (S151) and sets the speech from which the call name is extracted as the timing control target speech (S153).

[0216] Further, the processing server 50 receives the request message transmitted via the network 70 from the voice assistance server 40, and performs the process of step S154.

[0217] In step S154, the processing unit 502 extracts the voice response data recorded in the recording unit 503 based on the request message from the voice assistance server 40 and sends it to the voice assistance server 40. In the voice assistance server 40, the voice generation unit 404-1 generates voice data for response corresponding to the voice response data and sends it to the voice-user interface device 30.

[0218] The voice-user interface device 30 receives the voice data for response via the Internet 60, and performs the processing operation of step S155 and subsequent steps.

[0219] In step S155, the voice response output control unit 304 confirms (judges) whether the received voice data for response is a response to the timing control target utterance by inquiring the setting management unit 307. In the case where it is judged in step S155 that the received response is a response to the timing control target utterance set in the processing of step S153 ("Yes" in S155), the processing proceeds to step S156.

[0220] In step S156, the voice response output control unit 304 confirms (judges) whether the automatic timing control is set to on by making an inquiry to the setting management unit 307. In a case where it is determined in step S156 that the automatic timing control is set to on by the presetting process (S151) ("Yes" in S156), the process proceeds to step S157.

[0221] In step S157, the voice response output control unit 304 acquires voice response time information by receiving the voice response time metadata from the receiving device 20, which is set as the destination of metadata acquisition communication in the presetting process (S151).

[0222] In other words, in the present description, the voice response output control unit 304 dispatches the voice response time metadata acquisition request to the receiving device 20, and therefore, the receiving device 20 performs the processing operations of steps S158 to S159.

[0223] In step S158 , the voice response time metadata management unit 211 acquires the current media time by inquiring the media-time management unit 208 based on the acquisition request from the voice-user interface device 30 .

[0224] In step S159, the voice response time metadata management unit 211 extracts voice response time information indicating a time suitable for voice response from the voice response time metadata (metadata of the entire program) recorded in the recording unit 212 based on the acquired current media time. Then, the voice response time metadata management unit 211 sends the extracted information as voice response time metadata to the voice-user interface device 30. In the present description, for example, in addition to the time that does not interfere with viewing (such as interruptions in the development of the program being viewed, etc.), the time suitable for voice response can be considered to also include the time length of the responding voice (response length), the time zone on the playback timeline of the content being played, etc.

[0225] The voice-user interface device 30 receives the voice response time metadata (voice response time information), and performs the processing operations of steps S160 to S161.

[0226] In step S160, the voice response output control unit 304 waits until a time suitable for voice response is reached based on the acquired voice response time information. Then, when a time suitable for voice response is reached, the voice response output control unit 304 outputs a response voice by providing voice data for response to the speaker 302 (S161).

[0227] Thus, the voice-user interface device 30 can perform voice response at a time suitable for voice response, that is, at a time that does not interfere with viewing, such as a development dialogue interruption of the program being viewed, etc. This is performed based on voice response time metadata (voice response time information) acquired via the broadcast. This makes it possible for the viewer 2 to listen to the response voice (desired information) without interfering with viewing of the program.

[0228] Furthermore, in the voice-user interface device 30, in the case where it is judged that the received response is not a response to the timing control target utterance (at S155, "No"), or in the case where it is judged that the automatic timing control is set to off (at S156, "No"), the processing of steps S157 to S160 is skipped and the processing of step S161 is performed. In other words, in this case, the response voice is output immediately without waiting for the voice response time.

[0229] The above description of the processing procedure is given as a procedure of a second example of processing in each device of the first embodiment in the case of controlling the timing of the voice response to the utterance of the viewer 2 based on the voice response time metadata (voice response time information) acquired via broadcasting.

[0230] As described above, in the first embodiment, there is a case where the voice response output control unit 304 of the voice-user interface device 30 uses a voice AI assistance service in conjunction with content such as a broadcast program. In this case, the time of the voice response is controlled based on voice response time information indicating a time suitable for a voice response to the utterance of a viewer watching the content. The voice response time information may be information indicating a time suitable for a voice response on a playback timeline of content such as a broadcast program.

[0231] In the present description, in the case where voice response time information (voice response time metadata) is acquired via communication, content such as a broadcast program is played in the receiving device 20 (first device), and the voice response time information is transmitted via communication by the processing server 50 (second device). In the processing server 50 (second device), voice response time information indicating the time suitable for voice response to the content played in the receiving device 20 (first device) is extracted from the voice response time metadata (metadata of the entire program) and transmitted. The voice response time metadata includes voice response time information about all or part of the time on the playback timeline of the content. Therefore, in the voice-user interface device 30, the voice response output control unit 304 controls the time of the voice response based on the voice response time information transmitted via communication.

[0232] Further, in the case where voice response time information (voice response time metadata) is acquired via broadcasting, content such as a broadcast program is played in the receiving device 20 (first device). The voice response time information is transmitted via broadcasting by the broadcast server (second device) of the broadcast transmission system 10. In the broadcast server (second device), voice response time metadata (metadata of the entire program) including voice response time information about all or part of the time on the playback timeline of the content is transmitted. In the receiving device 20 (first device), voice response time information indicating a time suitable for voice response to the content being played is extracted from the voice response time metadata (metadata of the entire program) transmitted via broadcasting. Therefore, in the voice-user interface device 30, the voice response output control unit 304 controls the time of the voice response based on the voice response time information extracted by the receiving device 20 (first device).

[0233] It can be said that the voice-user interface device 30 having this function is an information processing apparatus including the voice response output control unit 304 .

[0234] (2) Second Embodiment

[0235] Incidentally, although the first embodiment described above shows a configuration for controlling the timing of voice response to the viewer's utterance based on the voice response time information on the system side, it is considered that the timing of the voice response differs from the desired timing depending on the viewer 2 .

[0236] Therefore, now refer to Figures 14 to 16 A configuration in which the timing of a voice response is controlled based on an instruction utterance of the viewer 2 is described as a second embodiment.

[0237] (Configuration example)

[0238] Fig.14 is a diagram showing an example of the configuration of the second embodiment.

[0239] Fig.14 Shown in accordance with the use Fig.14 The time series of parts A to D in FIG. 1 is processed according to the progress of the broadcast program watched by viewer 2.

[0240] like Fig.14 As shown in A in FIG. , when the viewer 2 utters the words “Who is this actress?” while watching the drama program, the user interface device 30 picks up the words of the viewer 2 through the microphone and sends the voice data of the words to the voice assistance server 40 .

[0241] Therefore, if Fig.14 As shown in part B in FIG. 1 , the voice assistance server 40 performs processing in conjunction with a processing server 50 provided by a broadcasting station of the drama program being viewed, and thereby generates response voice data to the utterance and transmits it to the user interface device 30 .

[0242] Then, the user interface device 30 outputs the response voice corresponding to the voice data for response from the voice assistance server 40, but the output of the response voice overlaps with the dialogue scene of the drama program being watched. In this case, when the viewer 2 gives a stop instruction (wait instruction) such as "Wait a moment!", the output of the response voice is stopped and an icon indicating that it is in a waiting state (hereinafter referred to as a waiting icon) is presented so that the response is waiting.

[0243] In other words, if Fig.14 As shown in part C in FIG. 1 , the output of the response voice from the user interface device 30 is stopped, and a waiting icon 251 is displayed on the screen of the receiving apparatus 20 to be superimposed on the video of the drama program.

[0244] However, when the drama program being watched continues and, for example, the channel switches to a TV commercial, when the viewer 2 gives a start instruction (wait release instruction) such as "Now!", the output of the response voice that has been stopped is started (restarted) and the wait icon 251 is erased.

[0245] In other words, if Fig.14 As shown in part D in FIG. 1 , the output of the response voice is started (restarted). The user interface device 30 outputs the response voice "The name of this actress is XXXX and the main representative work is ..." based on the voice data for response from the voice assistance server 40.

[0246] In this case, in the receiving device 20, a TV commercial is played, and, for example, a voice response is output at a time that does not interfere with the viewing of the viewer 2 who is watching the drama program. In other words, in this example, the viewer 2 wants to concentrate on the drama program not only in a dialogue scenario but also in a scenario without dialogue, and therefore, the viewer waits to make a response until the TV commercial is played.

[0247] (Example of detailed configuration of each device)

[0248] Fig.15 is a block diagram showing an example of a detailed configuration of each device of the second embodiment.

[0249] and Figure 4 and Fig.10 resemblance, Fig.15 An example of the configuration of the broadcast delivery system 10 and the receiving device 20 is shown. In addition, Fig.15 An example of the configuration of a voice-user interface device 30 provided on the local side and a voice assistance server 40 and a processing server 50 provided on the cloud is shown.

[0250] Moreover, in Fig.15 In the configuration of the broadcast delivery system 10 and Figure 4 The configuration shown in is similar to that shown in , and therefore, its description is omitted here. Fig.15 In the configuration of the voice assistance server 40 and the processing server 50, Fig.10 The configuration shown in is similar to that shown in and thus, its description is omitted here.

[0251] exist Fig.15 In the embodiment, the receiving device 20 includes a tuner 201, a demultiplexer 202, an AV decoder 203, an application program execution unit 204 accompanying the broadcast, a renderer 205, a display 206, a speaker 207, a communication unit 210, and a voice response waiting notification unit 213. In other words, Figure 4 Compared to the configuration shown in Fig.15The configuration of the receiving device 20 shown in is provided with a voice response waiting notification unit 213 and a communication unit 210 instead of the media-time management unit 208 and the communication unit 209.

[0252] In the case of receiving a notification indicating a waiting state from the voice-user interface device 30 (its voice response output control unit 304), the voice response waiting notification unit 213 instructs the renderer 205 to display a waiting icon. In addition, in the case of receiving a notification indicating a released state from the voice-user interface device (its voice response output control unit 304), the voice response waiting notification unit 213 instructs the renderer 205 to delete the waiting icon.

[0253] exist Fig.15 In the embodiment, the voice-user interface device 30 includes a microphone 301, a speaker 302, a communication unit 303, a voice response output control unit 304, a buffer 306, a voice analysis unit 308, and a communication unit 309. In other words, Figure 4 Compared to the configuration shown in Fig.15 In the illustrated voice-user interface device 30, a voice analysis unit 308 and a communication unit 309 are added as a replacement for the excluded timer 305 and the setting management unit 307.

[0254] The speech analysis unit 308 analyzes the speech data of the speech provided from the microphone 301, and based on the result obtained by the analysis, confirms (judges) whether the speech executes a stop instruction (wait instruction) or a start instruction (wait release instruction) for the output of the response speech.

[0255] In the case of judging that the utterance does not execute a stop instruction or a start instruction for the output of the response voice, the voice analysis unit 308 provides the voice data of the utterance to the communication unit 303. This allows the voice data of the utterance to be transmitted to the voice assistance server 40 via the Internet 60. In addition, in the case of judging that the utterance executes a stop instruction or a start instruction for the output of the response voice, the voice analysis unit 308 notifies the voice response output control unit 304 of the judgment result.

[0256] In the case of an instruction to stop outputting the response voice, the voice response output control unit 304 performs control to stop the output of the response voice currently being output based on the notification from the voice analysis unit 308, and in the case of an instruction to start (restart) the output of the response voice, performs control to start (restart) the output of the response voice in the waiting state (stopped). In addition, the voice response output control unit 304 performs control based on the notification from the voice analysis unit 308 so that a notification indicating the waiting state or the released state is provided to the receiving device 20 (the voice response waiting notification unit 213 thereof).

[0257] (Processing in each device)

[0258] Reference Fig.16 The flowcharts in describe the processing procedures in the respective devices of the second embodiment.

[0259] When the viewer 2 who is watching the broadcast program played in the receiving device 20 utters a speech, the microphone 301 picks up the speech (S201) in the voice-user interface device 30. In addition, the speech analysis unit 308 analyzes the speech data of the picked up speech and confirms (judgments) whether to issue a stop instruction (wait instruction) or a start instruction (wait release instruction) regarding the output of the response speech based on the analysis result (S202).

[0260] In this description, one of the following three processing operations is performed according to the judgment processing result in step S202: normal utterance, stop utterance, and start utterance.

[0261] In the first case, in the judgment processing of step S202, when it is judged that normal speech (for example, speech such as "Who is this actress?" etc.) is uttered instead of speech of a stop instruction or a start instruction regarding the output of the response voice, the processing proceeds to step S203 and normal speech processing is performed.

[0262] In this normal utterance process, the voice-user interface device 30 transmits the voice data of the utterance to the voice assistance server 40 (S203A). The voice assistance server 40 analyzes the voice data of the utterance from the voice-user interface device 30 and generates a request message based on the analysis result.

[0263] The processing server 50 receives the request message and performs the process of step S203B. In other words, the processing unit 502 extracts the voice response data based on the request message and sends the extracted data to the voice assistance server 40. In the voice assistance server 40, voice data for response is generated based on the voice response data and sent to the voice-user interface device 30.

[0264] In the voice-user interface device 30, the voice response output control unit 304 causes the speaker 302 to output a response voice corresponding to the voice data for response from the voice assistance server 40. This allows the viewer 2 to confirm the response voice output from the voice-user interface device 30.

[0265] In the second case, in the determination processing of step S202, in the case where it is determined that an instruction to stop the output of the response voice (for example, such as an utterance “Wait a moment!”) is issued, the process proceeds to step S204, and a stop utterance process is performed.

[0266] In this stop utterance processing, the voice-user interface device 30 performs processing operations of steps S204A to S204C. In other words, the voice analysis unit 308 notifies (announces) the voice response output control unit 304 of the stop output of the response voice (S204A).

[0267] This causes the voice response output control unit 304 to stop the current output of the response voice from the speaker 302 (S204B) according to the notification from the voice analysis unit 308. In this case, the voice response output control unit 304 causes the buffer 306 to temporarily store the voice data for the response.

[0268] Further, the voice response output control unit 304 notifies the receiving device 20 that the output of the response voice is in a waiting state (S204C). The receiving device 20 receives the notification indicating the waiting state by using wireless communication such as Bluetooth (registered trademark), and performs the process of step S204D.

[0269] In other words, the voice response waiting notification unit 213 instructs the renderer 205 to display the waiting icon (S204D) based on the notification indicating the waiting state from the voice-user interface device 30 (its voice response output control unit 304). This allows the waiting icon 251 to be displayed on the screen of the display 206 of the receiving device 20, and therefore, the viewer 2 can confirm that the response voice is waiting using the viewer's own words.

[0270] In the third case, in the determination processing of step S202, in the case where it is determined that an instruction to start output of a response voice (for example, such as an utterance “Okay now”) is issued, the process proceeds to step S205 and a start utterance process is executed.

[0271] In this start utterance process, the voice-user interface device 30 performs the process of steps S205A to S205C. In other words, the voice analysis unit 308 causes the voice response output control unit 304 to notify the start of the output of the response voice (S205A).

[0272] This allows the voice response output control unit 304 to cause the speaker 302 to output the response voice corresponding to the voice data for response temporarily stored in the buffer 306 according to the notification from the voice analysis unit 308, and to resume the output of the waiting response voice (S205B).

[0273] Further, the voice response output control unit 304 notifies the receiving device 20 that the output of the waiting response voice is released (S205C). The receiving device 20 receives the notification indicating the released state and performs the process of step S205D.

[0274] In other words, the voice response waiting notification unit 213 instructs the renderer 205 to delete the waiting icon (S205D) based on the notification indicating the release state from the voice-user interface device 30. This allows the waiting icon 251 on the screen of the display 206 of the receiving apparatus 20 to be erased.

[0275] For example, in the case where the output of the response voice is on hold while the viewer 2 is watching a drama program until a TV commercial, the viewer 2 can issue an instruction to start the output of the response voice. Accordingly, the viewer 2 can confirm the response voice "The name of this actress is XXXX and her main representative work is ..." This enables the viewer 2 to listen to the response voice (the information he wants to know) at the time he wants to hear it.

[0276] The above description is given of the processing procedures in the case where the timing of the voice response is controlled based on the instruction utterance of the viewer 2 as the processing procedures in the respective devices of the second embodiment.

[0277] And, the waiting icon is an example of information indicating that the output of the response voice is in a waiting state, and as long as the information that it is in a waiting state can be notified, other types of information can be used. In addition, for example, in the receiving device 20 or the voice-user interface device 30, the flashing light can be used as the waiting information.

[0278] (3) Third Embodiment

[0279] Incidentally, in the first embodiment, the system side controls the timing of the voice response to the viewer's speech based on the voice response time information. Thus, it is assumed that some people want to hear the voice response at a desired time depending on the viewer 2. In other words, similar to the configuration shown in the second embodiment, the configuration shown in the first embodiment can also perform control of the timing of the voice response based on the instruction speech of the viewer 2.

[0280] Therefore, reference Figures 17 to 20 , a configuration in which the timing of a voice response is controlled based on voice response time metadata (voice response time information) acquired via communication or broadcasting and an instruction utterance of the viewer 2 is shown as a third embodiment.

[0281] (First Example of Detailed Configuration of Each Device)

[0282] Fig.17 is a block diagram showing a first example of detailed configuration of each device according to the third embodiment.

[0283] and Figure 4 and Fig.15 resemblance, Fig.17 An example of the configuration of the broadcast delivery system 10 and the receiving device 20 is shown. In addition, Fig.17 An example of the configuration of the voice-user interface device 30 provided on the local side and the configuration of the voice assistance server 40 and the processing server 50 provided on the cloud is shown.

[0284] Moreover, in Fig.17 In the broadcast delivery system 10, the voice assistance server 40, and the processing server 50 are configured similarly to Figure 4 are similar to those shown in , and thus their description is omitted.

[0285] exist Fig.17 In the embodiment, the receiving device 20 includes a tuner 201, a demultiplexer 202, an AV decoder 203, an application program execution unit 204 accompanying the broadcast, a renderer 205, a display 206, a speaker 207, a media-time management unit 208, a communication unit 209, a communication unit 210, and a voice response waiting notification unit 213. In other words, Figure 4 Compared to the configuration shown in Fig.17 The configuration of the receiving device 20 shown in is added with a voice response waiting notification unit 213 and a communication unit 210.

[0286] When receiving a notification indicating a waiting state from the voice-user interface device 30 (its voice response output control unit 304), the voice response waiting notification unit 213 instructs the renderer 205 to display a waiting icon, and when receiving a notification indicating a released state, instructs the renderer 205 to delete the waiting icon.

[0287] exist Fig.17 , the voice-user interface device 30 includes a microphone 301, a speaker 302, a communication unit 303, a voice response output control unit 304, a timer 305, a buffer 306, a setting management unit 307, a voice analysis unit 308, and a communication unit 309. In other words, Figure 4 Compared to the configuration shown in Fig.17 In the configuration of the voice-user interface device 30 shown, a voice analysis unit 308 and a communication unit 309 are added.

[0288] While waiting until the voice response time based on the voice response time metadata (voice response time information) acquired via communication, the voice response output control unit 304 causes the receiving device 20 (its voice response waiting notification unit 213) to issue a notification indicating a waiting state. In addition, when the output of the response voice is instructed to start (restart) based on the notification from the voice analysis unit 308, the voice response output control unit 304 starts (restarts) the output of the response voice in the waiting state (in the stopped state). At this time, the voice response output control unit 304 controls based on the notification from the voice analysis unit 308 so that a notification indicating a released state is provided to the receiving device 20 (its voice response waiting notification unit 213).

[0289] (Procedure of First Example of Processing in Each Device)

[0290] Reference Fig.18 The flowchart in describes the procedure of the first example of the processing in each device of the third embodiment.

[0291] In the case where the viewer 2 viewing the broadcast program broadcasted in the receiving device 20 asks a question, the processing operations of steps S301 to S303 are performed.

[0292] As mentioned above Figure 8 Similar to steps S102 to S107 in , in steps S301 to S303, a request message corresponding to the analysis result of the voice data of the utterance is sent to the processing server 50. In the processing server 50, based on the current media time acquired from the receiving device 20, voice response time information indicating a time suitable for a voice response is extracted from the voice response time metadata (metadata of the entire program), and voice response data corresponding to the request message is sent (S301 and S302).

[0293] Then, the voice data for response and voice response time metadata (voice response time information) are received by the voice-user interface device 30 via the Internet 60. The voice response output control unit 304 waits until the voice response time based on the voice response time metadata (voice response time information) acquired via communication (S303).

[0294] In step S304, the voice response output control unit 304 notifies the receiving device 20 that the output of the response voice is in a waiting state. The receiving device 20 receives the notification indicating that it is in a waiting state and performs the process of step S305.

[0295] In other words, the voice response waiting notification unit 213 instructs the renderer 205 to display the waiting icon (S305) based on the notification indicating the waiting state from the voice-user interface device 30 (its voice response output control unit 304). This allows the waiting icon 251 to be displayed on the screen of the display 206 of the receiving device 20, and therefore, the viewer 2 can confirm that the response voice is waiting.

[0296] In this description, for example, when the viewer 2 confirms the wait icon 251 and the notification voice response is in the wait state, it is deemed necessary to issue an instruction utterance to start output of a response voice (eg, such as “OK now”) (S306 and S307).

[0297] In this case, in the voice-user interface device 30, the voice analysis unit 308 analyzes the voice data of the speech picked up by the microphone 301 and performs the processing operations of steps S308 to S310. In other words, the voice analysis unit 308 notifies the voice response output control unit 304 to start outputting the response voice (S308).

[0298] This allows the voice response output control unit 304 to cause the speaker 302 to output the response voice corresponding to the voice data for response temporarily stored in the buffer 306 according to the notification from the voice analysis unit 308, and to restart the output of the waiting response voice (S309).

[0299] Further, the voice response output control unit 304 notifies the receiving device 20 of releasing the output of the waiting response voice (S310). The receiving device 20 receives the notification indicating the released state and performs the process of step S311.

[0300] In other words, the voice response wait notification unit 213 instructs the renderer 205 to delete the wait icon 251 based on the notification indicating the release state from the voice-user interface device 30 so that the wait state displayed on the screen of the display 206 is deleted (S311).

[0301] Moreover, this example assumes that the viewer 2 noticing the wait icon 251 issues a start instruction. However, in the case where no utterance of the start instruction is issued, for example, when the time indicated by the voice response time information is reached later, a response voice is output from the speaker 302.

[0302] The above description of the processing procedure is given as a procedure of a first example of processing in each device of the third embodiment in the case of controlling the timing of the voice response based on the voice response time metadata (voice response time information) acquired via communication and the instruction utterance of the viewer 2.

[0303] (Second Example of Detailed Configuration of Each Device)

[0304] Fig.19 is a block diagram showing a second example of the detailed configuration of each device according to the third embodiment.

[0305] and Fig.10 and Fig.15 resemblance, Fig.19 An example of the configuration of the broadcast delivery system 10 and the receiving device 20 is shown. In addition, Fig.19 An example of the configuration of the voice-user interface device 30 provided on the local side and the configuration of the voice assistance server 40 and the processing server 50 provided on the cloud is shown.

[0306] Moreover, in Fig.19 In the broadcast delivery system 10, the voice assistance server 40, and the processing server 50 are configured similarly to Fig.10 The configuration shown in is similar, and therefore its description is omitted.

[0307] exist Fig.19 In the embodiment, the receiving device 20 includes a tuner 201, a demultiplexer 202, an AV decoder 203, a broadcast-accompanying application execution unit 204, a renderer 205, a display 206, a speaker 207, a media-time management unit 208, a communication unit 210, a voice response time metadata management unit 211, a recording unit 212, and a voice response waiting notification unit 213. In other words, Fig.10 Compared to the configuration shown in Fig.19 The configuration of the receiving apparatus 20 shown in FIG. 1 adds a voice response waiting notification unit 213 .

[0308] When receiving a notification indicating a waiting state from the voice-user interface device 30 (its voice response output control unit 304), the voice response waiting notification unit 213 instructs the renderer 205 to display a waiting icon, and when receiving a notification indicating a released state, instructs the renderer 205 to delete the waiting icon.

[0309] exist Fig.19 , the voice-user interface device 30 includes a microphone 301, a speaker 302, a communication unit 303, a voice response output control unit 304, a timer 305, a buffer 306, a setting management unit 307, a voice analysis unit 308, and a communication unit 309. In other words, Fig.19 The voice-user interface device 30 shown in FIG. Fig.10 The configuration is similar to the voice-user interface device 30 shown in FIG. 1 , but the processing details of the voice response output control unit 304 and the voice analysis unit 308 are different.

[0310] When waiting until the voice response time based on the voice response time metadata (voice response time information) acquired via broadcasting, the voice response output control unit 304 causes the receiving device 20 (its voice response waiting notification unit 213) to issue a notification indicating a waiting state. In addition, when the output of the response voice is instructed to start (restart) based on the notification from the voice analysis unit 308, the voice response output control unit 304 starts (restarts) the output of the response voice in the waiting state (in the stopped state). At this time, the voice response output control unit 304 controls based on the notification from the voice analysis unit 308 so that a notification indicating a released state is given to the receiving device 20 (its voice response waiting notification unit 213).

[0311] (Procedure of Second Example of Processing in Each Device)

[0312] Reference Fig. 20 The flowchart in describes the procedure of the second example of the processing in each device of the third embodiment.

[0313] In the case where a question is asked by a viewer viewing a broadcast program broadcasted in the receiving device 20, the processing operations of steps S351 to S354 are performed.

[0314] As mentioned above Fig.13 Similar to steps S152 to S160 in , in steps S351 to S354, a request message corresponding to the analysis result of the voice data of the utterance is sent to the processing server 50, and voice response data corresponding to the request message is sent (S351 and S352).

[0315] On the other hand, in the voice-user interface device 30, in the case where the received response voice data is a response to the timing control target utterance, voice response time metadata (voice response time information) is acquired from the receiving device 20 (S353). Accordingly, the voice response output control unit 304 waits until the voice response time based on the voice response time metadata (voice response time information) acquired via the broadcast (S354).

[0316] As mentioned above Fig.18 Similar to steps S304 to S305 in , in steps S355 to S356, the voice response output control unit 304 notifies that the output of the response voice is in a waiting state, and when the receiving device 20 receives the notification, the voice response waiting notification unit 213 instructs to display a waiting icon.

[0317] Furthermore, Fig.18Similarly to steps S306 to S311 in , in steps S357 to S362, in the case where the viewer 2 issues an instruction to start the output of the response voice, the output of the response voice in the waiting state is restarted in the voice-user interface device 30. Further, in this case, the voice response output control unit 304 notifies that the output of the response voice is in the released state, and when the receiving device 20 receives the notification, the voice response waiting notification unit 213 issues an instruction to delete the displayed waiting icon.

[0318] The above description of the processing procedure is given as a procedure of a second example of processing in each device of the third embodiment when the timing of the voice response is controlled based on the voice response time metadata (voice response time information) acquired via broadcasting and the instruction utterance of the viewer 2.

[0319] As described above, in the third embodiment, there is a case where the voice response output control unit 304 of the voice-user interface device 30 uses content such as a broadcast program in combination with a voice AI assistance service. In this case, the time of the voice response is controlled based on the voice response time information indicating the time suitable for making a voice response to the utterance of the viewer who is watching the content and the utterance of the viewer.

[0320] Further, in the voice-user interface device 30, when waiting for the time to reach the voice response, the voice response output control unit 304 can make information indicating that it is in a waiting state appear. Further, in the case where the viewer gives an instruction to start the voice response, the voice response output control unit 304 can make the voice response in the waiting state start outputting.

[0321] Further, in the case of waiting for the time of voice response, the voice response output control unit 304 issues a notification (first message) indicating a waiting state to the receiving device 20 (first device) that plays content such as a broadcast program. In the receiving device 20 (first device), the waiting icon 251 (icon) is displayed based on the notification (first message) indicating the waiting state. In the case where the viewer issues an instruction to start the voice response, a notification (second message) indicating a release state is provided to the receiving device 20 (first device). In the receiving device 20 (first device), the display of the waiting icon 251 (icon) can be deleted based on the notification (second message) indicating the release state.

[0322] It can be said that the voice-user interface device 30 having such a function is an information processing apparatus including the voice response output control unit 304 .

[0323] As mentioned above, in the configuration shown in the first embodiment, the time of voice response is automatically controlled on the system side. Thus, for example, for viewers who wish to hear the response voice earlier, there is the possibility of redundant interference because it overlaps with the dialogue of the main part. Thus, similar to the second embodiment, in the configuration of the third embodiment, even during the period when the voice-user interface device 30 waits for the output of the response voice based on the voice response time information, the receiving device 20 will be made to display the waiting icon. The viewer who wishes to hear the response voice earlier confirms the waiting icon, and when the output of the recognition response voice is in a waiting state, a start instruction such as "now!" etc. of the response voice output is issued, which enables the response voice to be heard immediately. Moreover, unlike the configuration of the second embodiment, if the automatic timing adjustment is suitable for the viewer, there is also the advantage of not needing redundant interaction.

[0324] <3. Transformation>

[0325] (Examples of other configurations)

[0326] The above description is given of the receiving device 20 and the voice-user interface device 30 being different devices. However, for example, Fig.21 As shown in , these devices can be integrated to form one device (information processing apparatus).

[0327] In other words, Fig.21 The receiving device 20 shown in is a device that supports voice AI assistance services, such as a TV or a smartphone. In addition to the tuner 201 to the media-time management unit 208 and the voice response waiting notification unit 213, the receiving device 20 is also provided with a microphone 301 on one side of the voice-user interface device 30, a communication unit 303, a voice response output control unit 304, a timer 305, a buffer 306, a setting management unit 307, and a voice analysis unit 308. However, in Fig.21 In the embodiment, the speaker 207 is also used as the speaker 302 on the side of the voice-user interface device 30. In addition, the communication unit 303 is also used as the communication unit 209 on the side of the receiving device 20.

[0328] exist Fig.21 In the illustrated receiving device 20, the voice response output control unit 304 controls the time of the voice response based on the voice response time information. In addition, the voice response waiting notification unit 213 presents a waiting icon.

[0329] Moreover, reference Fig.21A configuration in which the receiving device 20 is compatible with the voice AI assistance service is described. However, the functions of the voice response waiting notification unit 213 and the like are provided on the side of the voice-user interface device 30, and necessary information can be acquired from the side of the receiving device 20. In addition, although Fig.21 , a configuration corresponding to the third embodiment is shown, however, the receiving device 20 and the voice-user interface apparatus 30 can be integrated, similar to the first and second embodiments.

[0330] Further, as functions for providing voice AI assistance services, there are functions on the local side (for example, functions such as the voice response output control unit 304 or the voice analysis unit 308, etc.) and functions on the cloud (for example, functions such as the voice analysis unit 403 and the voice generation unit 404-1, the metadata merging unit 404-2, etc.). However, all of these functions can be implemented in a device (terminal) on the local side or a device (server) on the cloud, or some of these functions can be implemented in a device (terminal) on the local side or a device (server) on the cloud.

[0331] Furthermore, in the above-mentioned Figure 1 In the content-voice AI collaborative system 1, a situation is shown in which one receiving device 20 (for example, a TV set) and one voice-user interface device 30 (smart speaker) are installed in the viewer's house. However, for example, a receiving device 20 and a voice-user interface device 30 to which the present technology is applied may be provided for each viewer's house. In addition, although it is assumed that the receiving device 20 and the voice-user interface device 30 are installed in the same room of the viewer's house, they can be installed in different rooms. Further, a plurality of voice-user interface devices 30 can be provided for one receiving device 20, or conversely, a plurality of receiving devices 20 can be provided for one voice-user interface device 30.

[0332] Furthermore, in the above-mentioned Figure 1 In the content-voice AI collaboration system 1, a case where one voice assistance server 40 and one processing server 50 are installed is shown, but, for example, a plurality of these servers may be installed for each function or provider (e.g., a broadcaster). On the other hand, all or part of the functions of the voice assistance server 40 and the processing server 50 can be integrated and provided by one or more servers.

[0333] (Example of broadcast system)

[0334] As a broadcast system for sending the broadcast signal (digital broadcast signal) transmitted from the broadcast transmission system 10 to the receiving device 20, for example, the Advanced Television System Committee (ATSC), that is, the system adopted by the United States and the like, can be applied. In addition to this, the Integrated Services Digital Broadcasting (ISDB), that is, the system adopted by Japan and other countries, or the Digital Video Broadcasting (DVB), that is, the system adopted by European countries and the like can be applied. In addition, the transmission path is not limited to terrestrial broadcasting, and for example, it can also be applied to satellite broadcasting using a broadcast satellite (BS), a communication satellite (CS), etc., or cable broadcasting such as cable television (CATV).

[0335] Furthermore, the broadcast delivery system 10 includes one or more broadcast servers and the like. For example, in a general digital broadcast system, the multiplexer 103 (multiplexer server) and the sending unit 104 (sending server) can be installed in different places. More specifically, for example, the multiplexer 103 is installed in a broadcast station, and the sending unit 104 is installed in the sending station. Further, the program content processing unit 101 (program content server), the application generation unit 102 accompanying the broadcast (application server accompanying the broadcast), and the metadata generation unit 105 (metadata server) can be installed in the same place (for example, a place inside the broadcast station) or in a different place (for example, a place outside the broadcast station) as the multiplexer 103 (multiplexer server).

[0336] (Example of content application)

[0337] Moreover, in the above description, programs and broadcasts are exemplified as the content to be transmitted, but the content to which the present technology is applied includes not only moving images or music, but also any type of content application programs such as e-books, games, and advertisements. In addition, in the above, the transmission by the broadcast transmission system 10 via broadcasting is described as the transmission route of the content, but a communication distribution system that provides an over-the-top (OTT) service or the like is capable of transmitting streams via communication. Further, the entire composition (e.g., such as video, audio, or subtitles) including content such as a program can be transmitted via broadcasting or communication, or a part of the composition can be transmitted via broadcasting or communication (the remaining part is transmitted via communication or broadcasting).

[0338] Further, for example, the application accompanying the broadcast may be an application developed in a markup language such as HTML5 or a scripting language such as JavaScript (registered trademark), but is not limited thereto. For example, it may be an application developed in a programming language such as Java (registered trademark). Furthermore, the application accompanying the broadcast is not limited to an application that is run through a browser (the application accompanying the broadcast running unit 204), but can be run as a so-called native application in an operating system (OS) environment or the like.

[0339] Further, the above description is given of a case where an application accompanying a broadcast is transmitted via broadcast by the broadcast delivery system 10, but it is not limited thereto. For example, an application server that generates an application accompanying a broadcast can transmit the application accompanying a broadcast via communication (via the Internet 60). In addition, the above description is given of a case where an application accompanying a broadcast is combined with content transmitted via broadcast, but it can be an application that runs in combination with content transmitted via communication.

[0340] (other)

[0341] Moreover, the terms used here are examples, and in fact, in some cases, other terms are used. However, the difference between these terms is that the form is different, but the essential content of the object is the same. For example, in some cases, the above-mentioned "skill" is referred to as "action", "application", etc. Further, in some cases, the above-mentioned "AI-assisted service" is referred to as "AI-assisted service", etc.

[0342] <4. Computer Configuration>

[0343] The series of processes described above can be executed by hardware or software. In the case where the series of processes are executed by software, a program constituting the software is installed in a computer. Fig. 22 : is a diagram showing an exemplary hardware configuration of a computer that executes the series of processes described above by a program.

[0344] In the computer 1000, a central processing unit (CPU) 1001, a read only memory (ROM) 1002, and a random access memory (RAM) 1003 are connected to each other via a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0345] The input unit 1006 includes a keyboard, a mouse, a microphone, etc. The output unit 1007 includes a display, a speaker, etc. The recording unit 1008 includes a hard disk, a nonvolatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0346] In the computer 1000 of the configuration described above, the CPU 1001 loads the program recorded in the ROM 1002 or the recording unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program. Thereby, the above-described series of processing is executed.

[0347] For example, the program recorded in the removable recording medium 1011 (ie, package medium, etc.) and executed by the computer 1000 (CPU 1001) is provided. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0348] In the computer 1000, the program can be installed on the recording unit 1008 via the input / output interface 1005 by mounting the removable recording medium 1011 on the drive 1010. In addition, the communication unit 1009 can receive the program via a wired or wireless transmission medium and can install the program on the recording unit 1008. In addition, the program can be installed in advance on the ROM 1002 or the recording unit 1008.

[0349] Here, in this specification, the process executed by the computer according to the program does not necessarily have to be executed in the time sequence described in the flowchart. That is, the process executed by the computer according to the program also includes processes executed in parallel or individually (e.g. parallel processes or object processes). In addition, the program may be processed by one computer (processor) or may be distributed and processed by multiple computers.

[0350] Furthermore, the embodiments of the present technology are not limited to the embodiments described above, and various changes and modifications may be made without departing from the scope of the present technology.

[0351] Furthermore, the present technology can also be configured as follows.

[0352] (1) An information processing device comprising:

[0353] A control unit is configured to control the timing of a voice response based on voice response time information indicating a time suitable for a voice response to an utterance of a viewer viewing the content when a voice AI assistance service is used in conjunction with the content.

[0354] (2) The information processing device according to (1),

[0355] The voice response time information is information indicating a time suitable for a voice response on a playback time axis of the content.

[0356] (3) The information processing device according to (1) or (2),

[0357] Wherein, voice response time information is obtained via communication.

[0358] (4) The information processing device according to (3),

[0359] wherein the content is played back through the first device;

[0360] transmitting, by the second device via the communication, the voice response time information;

[0361] The second device extracts voice response time information indicating a time suitable for voice response to the content played in the first device from metadata including voice response time information of an object that is all or part of the time on a playback timeline of the content; and

[0362] The control unit controls the time of the voice response based on the voice response time information transmitted via the communication.

[0363] (5) The information processing device according to (1) or (2),

[0364] The voice response time information is obtained via broadcasting.

[0365] (6) The information processing device according to (5),

[0366] wherein the content is played back through the first device;

[0367] transmitting, by the second device, the voice response time information via broadcast;

[0368] The second device transmits metadata including voice response time information of the object as all or part of the time on the playback timeline of the content;

[0369] The first device extracts voice response time information indicating a time suitable for a voice response to the content being played from metadata transmitted via the broadcast; and

[0370] The control unit controls the time of the voice response based on the voice response time information extracted by the first device.

[0371] (7) The information processing device according to any one of (1) to (6),

[0372] The control unit controls the timing of the voice response based on the viewer's speech.

[0373] (8) The information processing device according to (7),

[0374] When waiting until the time for voice response is reached, the control unit causes information indicating the waiting state to be presented.

[0375] (9) The information processing device according to (7),

[0376] When receiving an instruction to start the voice response from the viewer, the control unit starts the voice response in the waiting state.

[0377] (10) The information processing device according to (9),

[0378] Wherein, while waiting until the time for the voice response is reached, the control unit notifies the first device to play back the content of the first message indicating the waiting state;

[0379] The first device causes an icon indicating a waiting state to be displayed based on the notified first message;

[0380] Upon receiving an instruction to start a voice response from the viewer, the control unit notifies the first device of a second message indicating a waiting state for releasing the voice response; and

[0381] The first device causes the display icon indicating the waiting state to be erased based on the notified second message.

[0382] (11) The information processing device according to (2),

[0383] Among them, the voice response time information includes the time during which the voice of the content being played is not output as the time suitable for the voice response.

[0384] (12) The information processing device according to any one of (1) to (11),

[0385] wherein the content is broadcast content transmitted via broadcast; and

[0386] The voice response is a response to the words of a viewer viewing the broadcast content.

[0387] (13) The information processing device according to (3) or (4),

[0388] Among them, the voice data of the voice response and the voice response time information are transmitted together via the communication using HTTP response.

[0389] (14) The information processing device according to (5) or (6),

[0390] wherein the content is delivered via broadcast as a stream compatible with MPEG-DASH; and

[0391] The voice response time information is transmitted via broadcast using MPD.

[0392] (15) The information processing device according to any one of (1) to (14),

[0393] Among them, the information processing device is configured as a voice processing device, and the voice processing device is configured to be used as a user interface for voice AI assisted services.

[0394] (16) The information processing device according to any one of (1) to (14),

[0395] Among them, the information processing device is configured as a receiving device, and the receiving device is configured to receive and play back content transmitted via broadcasting.

[0396] (17) An information processing method performed by an information processing device, the method comprising:

[0397] When using the voice AI assistance service in conjunction with content, the information processing device controls the timing of the voice response based on voice response time information indicating a time suitable for making a voice response to the utterance of a viewer viewing the content.

[0398] (18) A sending device, comprising:

[0399] a generating unit configured to generate metadata including voice response time information indicating a time suitable for making a voice response to an utterance of a viewer viewing the content when a voice AI assistance service is used in conjunction with the content; and

[0400] The sending unit is configured to send the generated metadata.

[0401] (19) The transmitting device according to (18),

[0402] The generating unit generates an MPD, wherein the MPD is represented by speech response time information of an object that can be identified as all or part of the time on the playback timeline of the content through identification information, and the identification information is used to identify the speech response time information used for the voice AI auxiliary service; and

[0403] The transmission unit transmits the content together with the MPD as a stream compatible with MPEG-DASH via broadcasting.

[0404] (20) A sending method performed by a sending device, the method comprising:

[0405] When a voice AI assistance service is used in conjunction with the content, generating, by the transmitting device, metadata including voice response time information indicating a time suitable for making a voice response to an utterance of a viewer viewing the content; and

[0406] The generated metadata is transmitted by the transmitting device.

[0407] Reference Numbers List

[0408] 1Content-Voice AI Collaboration System

[0409] 10 Broadcast transmission system

[0410] 20 Receiving device

[0411] 30 Voice-User Interface Devices

[0412] 40 Voice Assist Server

[0413] 50 Processing Server

[0414] 60 Internet

[0415] 70 Network

[0416] 101 Program content processing unit

[0417] 102 Application generation unit accompanying broadcast

[0418] 103 Multiplexer

[0419] 104 Sending unit

[0420] 105 Metadata Generation Unit

[0421] 201 Tuner

[0422] 202 Demultiplexer

[0423] 203 AV decoder

[0424] 204 Application program running unit accompanying broadcast

[0425] 205 Renderer

[0426] 206 Display

[0427] 207 Speakers

[0428] 208 Media-Time Management Unit

[0429] 209 Communication Unit

[0430] 210 Communication Unit

[0431] 211 Voice Response Time Metadata Management Unit

[0432] 212 Recording Unit

[0433] 213 Voice response waiting notification unit

[0434] 301 Microphone

[0435] 302 Speakers

[0436] 303 Communication Unit

[0437] 304 Voice response output control unit

[0438] 305 Timer

[0439] 306 Cache

[0440] 307 Setting the Management Unit

[0441] 308 Speech Analysis Unit

[0442] 309 Communication Unit

[0443] 401 Communication Unit

[0444] 402 Communication Unit

[0445] 403 Speech Analysis Unit

[0446] 404 Message Generator

[0447] 404-1 Speech Generation Unit

[0448] 404-2 Metadata Merge Unit

[0449] 501 Communication Unit

[0450] 502 processing units

[0451] 503 Recording Unit

[0452] 1000 Computer

[0453] 1001 CPU

Claims

1. An information processing device, comprising: The processing circuit is configured to control the timing of the voice response when using the voice AI assistance service based on the viewer's words in combination with the content, Wherein, the processing circuit is configured as follows: presenting a waiting state notification notifying the viewer that the information processing apparatus is in a waiting state; and When receiving an instruction to start the voice response from the viewer, starting the voice response in the waiting state; Wherein, the waiting state notification is an indication signal, and When an instruction to start the voice response is received from the viewer, The indication signal is turned off. 2 . The information processing apparatus according to claim 1 , further comprising a user interface, the user interface comprising a setting management unit.

3. The information processing device according to claim 1, wherein: While waiting until the time for the voice response is reached, information indicating the waiting state is generated.

4. The information processing device according to claim 1, wherein: The indication signal is a light.

5. The information processing device according to claim 1, wherein: The indication signal is an icon.

6. The information processing device according to claim 1, in, The processing circuit is further configured to control the timing of the voice response based on voice response timing information indicating a time suitable for the voice response on a playback time axis of the content when there is playback of the content.

7. The information processing device according to claim 6, in, The voice response time information is acquired via communication.

8. The information processing device according to claim 7, wherein The content is played back by the first device, The voice response time information is transmitted by the second device via communication, The second device extracts the voice response time information from metadata, the voice response time information indicating a time suitable for the voice response to the content being played in the first device, the metadata including the voice response time information expected for all or part of the time on the playback timeline of the content, and The processing circuit is configured to control the timing of the voice response based on the voice response timing information transmitted via the communication.

9. The information processing device according to claim 6, wherein: The voice response time information is acquired via broadcasting.

10. The information processing device according to claim 4, wherein: When an instruction to start the voice response is received from the viewer, the light is turned off.

11. The information processing device according to claim 5, wherein: When an instruction to start the voice response is received from the viewer, the icon is turned off.

12. An information processing device, comprising: The processing circuit is configured to control the timing of the voice response when using the voice AI assistance service based on the viewer's words in combination with the content, Wherein, the processing circuit is configured as follows: receiving voice data of the viewer's speech; Transmitting the voice data to a voice AI-assisted server; receiving the voice response to the viewer's utterance from the server; presenting a waiting state notification notifying the viewer that the information processing apparatus is in a waiting state, the waiting state notification being a notification signal; and When an instruction to start the voice response is received from the viewer, the voice response in the waiting state is started.

13. The information processing device according to claim 12, wherein: While waiting until the time of the voice response based on the utterance of the viewer, information indicating the waiting state is generated.

14. The information processing device according to claim 12, wherein: The notification signal is a light.

15. The information processing device according to claim 12, wherein: The notification signal is an icon.

16. The information processing device according to claim 12, in, The processing circuit is further configured to control the timing of the voice response based on voice response timing information indicating a time suitable for the voice response on a playback time axis of the content when there is playback of the content.

17. The information processing device according to claim 16, in, The voice response time information is acquired via communication.

18. The information processing device according to claim 17, wherein: The content is played back by the first device, The voice response time information is transmitted by the second device via communication, The second device extracts the voice response time information from metadata, the voice response time information indicating a time suitable for the voice response to the content being played in the first device, the metadata including the voice response time information expected for all or part of the time on the playback timeline of the content, and The processing circuit is configured to control the timing of the voice response based on the voice response timing information transmitted via the communication.

19. The information processing device according to claim 16, in, The voice response time information is acquired through broadcasting.

20. The information processing device according to claim 12, wherein: When an instruction to start the voice response is received from the viewer, the notification signal is turned off.

Images