Audio output terminal and its control method

Abstract

It appropriately controls the output of audio content to multiple audio output units. [Solution] The voice collaboration system comprises a terminal and a content display device. The terminal comprises a voice acquisition unit that acquires a first voice content and a second voice content, a setting unit that sets information indicating the output destination of the first voice content and the second voice content, a voice output control unit that outputs the first voice content to one of the first and second voice output units and outputs the second voice content to the other based on the information set by the setting unit, and a receiving unit that receives voice data from the content display device as the first voice content. The content display device comprises an acquisition unit that acquires video data and audio data from the received content, a display unit that displays the video data acquired by the acquisition unit, a third audio output unit that outputs the audio data acquired by the acquisition unit, and a transmission unit that transmits the audio data to the terminal.

Description

Technical Field

[0001] The present invention relates to an audio output terminal for outputting audio content and an audio output method.

Background Art

[0002] Conventionally, each of a plurality of sub-receiving terminal devices capable of transmitting and receiving data with a main receiving terminal device is provided with ID information (identification information) unique to each sub-receiving terminal device, and sub-content corresponding to this ID information is transmitted from the main receiving terminal device to the sub-receiving terminal device. There is a technology for this (for example, see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a system including a main receiving terminal device (television receiver) and sub-receiving terminal devices as described in Patent Document 1, each of the main receiving terminal device and the sub-receiving terminal devices has an audio output unit. Also, a single terminal device may have a plurality of audio output units (for example, a speaker and headphones).

[0005] As described above, when having a plurality of audio output units, it is desirable to appropriately output the audio of the content. Therefore, an object is to provide a terminal, an audio cooperation playback system, and a content display device that can appropriately control the output of audio content for a plurality of audio output units.

Means for Solving the Problems

[0006] As means for solving the above problems, the technology described in the claims is used.

[0007] To give one example, the audio output terminal comprises: a wireless communication unit for acquiring audio content; an audio output unit for outputting the audio content; a control unit for controlling the audio output unit based on setting information including first information regarding an output pattern for the audio content, which includes at least a first pattern for outputting the audio content from the audio output unit and a second pattern for not outputting the audio content, and second information regarding the on / off status of the volume control function of the audio output unit; and an audio input unit for acquiring ambient sound of the audio output terminal, wherein the control unit analyzes the ambient sound acquired via the audio input unit to identify the type of ambient sound, and controls the volume of the audio output unit based on the identified type of ambient sound and the setting information. [Effects of the Invention]

[0008] By using the technology of the present invention, it is possible to appropriately control the output of audio content to the audio output unit. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of how the mobile information terminal according to Example 1 can be used. [Figure 2] This is a block diagram of the mobile information terminal according to Example 1. [Figure 3] This diagram illustrates the output pattern from the audio output unit of the portable information terminal according to Example 1. [Figure 4] This is a flowchart showing the processing of a mobile information terminal according to Example 1. [Figure 5A] This is a system configuration diagram (1) of the voice-linked playback system according to Example 2. [Figure 5B] This is a system configuration diagram (2) of the voice-linked playback system according to Example 2. [Figure 5C] This is a system configuration diagram (3) of the voice-linked playback system according to Example 2. [Figure 6] This is an explanatory diagram of the default settings in the audio output section of the portable information terminal of the voice-linked playback system according to Embodiment 2. [Figure 7] It is a block diagram of the content display device according to Example 2. [Figure 8] It is a block diagram of the external transmission device according to Example 2. [Figure 9] It is a flowchart (1) showing the processing of the portable information terminal according to Example 2. [Figure 10] It is a flowchart (2) showing the processing of the portable information terminal according to Example 2. [Figure 11A] It is a diagram (1) showing an application example of the voice-linked playback system according to Example 2. [Figure 11B] It is a diagram (2) showing an application example of the voice-linked playback system according to Example 2. [Figure 12A] It is a diagram (3) showing an application example of the voice-linked playback system according to Example 2. [Figure 12B] It is a diagram (4) showing an application example of the voice-linked playback system according to Example 2. [Figure 13] It is a diagram for explaining the voice control of the voice-linked playback system according to Example 3. [Figure 14] It is a flowchart (1) showing the processing of the portable information terminal according to Example 3. [Figure 15] It is a flowchart (2) showing the processing of the portable information terminal according to Example 3. [Figure 16] It is a flowchart showing the processing of the portable information terminal according to Example 4. [Figure 17] It is a diagram for explaining the synchronization control of the voice-linked playback system according to Example 4. [Figure 18] It is a flowchart showing the processing of the portable information terminal according to Example 5. [Figure 19] It is a flowchart showing the processing of the portable information terminal according to Example 6. [Figure 20] It is a diagram showing a subtitle display example of the voice-linked playback system according to Example 7. [Figure 21] It is a diagram showing various processes required for translation of the voice-linked playback system according to Example 8. [Modes for carrying out the invention]

[0010] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

[0011] (Example 1) First, an example of content playable on the personal digital assistant (PDA) of this embodiment will be described. First, using Figure 1, an example of how the PDA 700 (PDA 700mu) can be used will be explained. The PDA 700 is, for example, a mobile phone, a smartphone, a tablet, etc. The PDA 700 may also be a PDA (Personal Digital Assistant) or a notebook PC. The PDA 700 may also be a digital still camera, a video camera capable of shooting video, a portable game console, etc., or other portable digital device.

[0012] The personal digital assistant (PDA) 700 receives audio content, including audio data, via an external network (e.g., the Internet) and plays the audio content. Here, audio content includes music content, video content, and one-segment broadcast content. The PDA 700 is also capable of receiving one-segment broadcast waves, receiving one-segment broadcast content, and playing the one-segment content.

[0013] The personal digital information terminal 700 has multiple audio output units, and can output audio from these units. Specifically, the personal digital information terminal 700 can output audio from a speaker and from headphones to which it is connected.

[0014] Furthermore, the personal information terminal 700 can make calls via the aforementioned external network or mobile telephone network. In other words, the personal information terminal 700 can make calls using the telephone network. Moreover, the personal information terminal 700 makes calls via the external network or mobile telephone network, receives call audio, and transmits call data to the other party. The personal information terminal 700 can also play pre-stored music content. The above description is not limited to application within Japan only.

[0015] Figure 1 is a schematic diagram illustrating the viewing of two types of content. The mobile information terminal 700 is playing both streaming content and music content. Here, user A 10000mua wants to view streaming content, while user B 10000mub wants to view music content.

[0016] In this case, the headphone audio output unit of the mobile information terminal 700mu plays the audio of the streaming content, and the speaker audio output unit of the mobile information terminal 700mu plays the music content. In this case, user A 10000mua can view the streaming content, and user B 10000mub can view the music content, without interfering with the viewing of the other user's content.

[0017] [Hardware configuration of mobile devices] Next, a specific configuration example in this embodiment will be described. Figure 2 is a block diagram showing an example of the internal configuration of the portable information terminal 700. The portable information terminal 700 comprises a main control unit 701, a system bus 702, a communication processing unit 720 (receiving unit), an operation unit 730, an image processing unit 740 (display control unit), and an audio processing unit 750.

[0018] The main control unit 701 is a microprocessor unit that controls the entire portable information terminal 700. The main control unit 701 acquires multiple audio content, such as broadcast content and video content, via the communication processing unit 720. The main control unit 701 may also store the acquired content. In this way, the main control unit 701 functions as an audio acquisition unit.

[0019] The main control unit 701 has installed applications for playing content (e.g., an audio-linked playback application) and a call application, and executes these applications in response to user requests. The content-playing application plays various audio content, for example, acquired audio content. The audio-linked playback application plays audio data acquired from a television receiver. The operation of this audio-linked application will be described later. The call application controls incoming and outgoing calls.

[0020] When the application that plays the above-mentioned content receives a request to play the content, the main control unit 701 sends the acquired or stored content to the image processing unit 740 or the audio processing unit 750. The main control unit 701 also controls incoming and outgoing calls by telephone, and controls calls using the speaker audio output unit 751 and the audio input unit 754, based on the above-mentioned call application. The system bus 702 is a data communication path for sending and receiving data between the main control unit 701 and each operation block in the portable information terminal 700.

[0021] The communication processing unit 720 includes a LAN communication unit 721, a mobile telephone network communication unit 722, and an NFC communication unit 723. The LAN communication unit 721 transmits and receives data via the Internet. The mobile telephone network communication unit 722 transmits and receives telephone communications (calls) and data via the mobile telephone network.

[0022] The NFC communication unit 723 performs wireless communication when in close proximity to a compatible reader / writer. The LAN communication unit 721, the mobile telephone network communication unit 722, and the NFC communication unit 723 are each equipped with coding circuits, decoding circuits, antennas, etc. The communication processing unit 720 may also be equipped with other communication units, such as a Bluetooth® communication unit or an infrared communication unit. For example, the communication processing unit 720 receives content transmitted from an external device (e.g., a television receiver).

[0023] The operation unit 730 is an instruction input unit that inputs operation instructions to the portable information terminal 700. In this embodiment, it consists of a touch panel and an operation key arranged on top of the display unit 741, with button switches arranged in a row. Either one or the other may be used. The portable information terminal 700 may also be operated using a separate portable terminal device connected via wired or wireless communication. Furthermore, the touch panel function may be provided by the display unit 741.

[0024] The image processing unit 740 consists of a display unit 741, an image signal processing unit 742, a first image input unit 743, and a second image input unit 744. The display unit 741 is, for example, a display device such as a liquid crystal panel, and provides (displays) the image data processed by the image signal processing unit 742 to the user of the portable information terminal 700. The image signal processing unit 742 includes a video RAM (not shown), and the display unit 741 is driven based on the image data input to the video RAM.

[0025] Furthermore, the image signal processing unit 742 shall have functions to perform format conversion, superimposition of menus and other OSD (On Screen Display) signals as needed. The first image input unit 743 and the second image input unit 744 are, for example, camera units. These camera units input image data of the surroundings and objects by converting light input from a lens into electrical signals using electronic devices such as CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensors.

[0026] The audio processing unit 750 consists of a speaker audio output unit 751 (first audio output unit), a headphone audio output unit 752 (second audio output unit), an audio signal processing unit 753 (audio output control unit, analysis unit, calculation unit), and an audio input unit 754 (input unit).

[0027] The speaker audio output unit 751 provides the audio signal processed by the audio signal processing unit 753 to all users within range of being able to hear the output speaker audio. The headphone audio output unit 752 provides the audio signal processed by the audio signal processing unit 753 to the user of the portable information terminal 700. The headphone audio output unit 752 may be equipped with a terminal to which headphones or earphones can be connected. The headphone audio output unit 752 may also wirelessly output the audio signal via Bluetooth® communication (short-range wireless communication) or the like. Thus, the portable information terminal 700 is a terminal equipped with multiple audio output units that output audio content.

[0028] Furthermore, the audio signal processing unit 753 is capable of processing two or more audio signals in order to output different audio signals from the speaker audio output unit 751 and the headphone audio output unit 752. The audio signals output from the speaker audio output unit 751 or the headphone audio output unit 752 are selectively controlled by the main control unit 701.

[0029] The audio signals output by the speaker audio output unit 751 and the headphone audio output unit 752 may be mono or stereo, respectively. The audio input unit 754 is a microphone and converts the user's voice or other audio data into audio data for input.

[0030] The audio signal processing unit 753 sets output pattern information, described later, via the operation unit 730, and stores the set output pattern information. In this way, the audio signal processing unit 753 functions as a setting unit. Furthermore, based on the output pattern information, the audio signal processing unit 753 outputs the first audio content to one of the first and second audio output units, and outputs the second audio content to the other of the first and second audio output units.

[0031] Note that the example configuration of the portable information terminal 700 shown in Figure 2 includes many components that are not essential to this embodiment, but the effectiveness of this embodiment will not be impaired even if these components are not included. Furthermore, additional components not shown, such as a digital broadcasting reception function or an electronic money payment function, may also be included.

[0032] [Audio signal output pattern] Figure 3 shows an example of an output pattern when the first audio signal or the second audio signal is output from the speaker audio output unit 751 and the headphone audio output unit 752 in the portable information terminal 700. Each audio signal has three patterns: "no output", "output from speaker audio output unit 751", and "output from headphone audio output unit 752", and there are 3 x 3 = 9 possible combinations of the two types of audio signals. Both types of audio signals are signals of audio content. The first audio signal is, for example, a signal of video content, and the second audio signal is, for example, a signal of music content.

[0033] Specifically, these are output patterns No. 1 22000 to No. 9 22008 shown in Figure 3. Of these, the four patterns that output both the first and second audio signals are No. 5 output pattern 22004, No. 6 output pattern 22005, No. 8 output pattern 22007, and No. 9 output pattern 22008. In particular, No. 5 output pattern 22004 and No. 9 output pattern 22008 output the first and second audio signals superimposed.

[0034] The user may manually select these output patterns via the operation unit 730, or the system may include a headphone connection detection unit that determines whether headphones are connected, and automatically select the pattern based on the detection result of the headphone connection detection unit. The main control unit 701 implements the headphone connection detection unit by detecting the connection status of the terminals or detecting a headphone connection via short-range wireless communication.

[0035] The settings shown in Figure 3 are merely an example; if the portable information terminal 700 is equipped with other audio output units, the selection of other audio output units is permitted, as is the output of the same audio signal from multiple audio output units.

[0036] Next, using Figure 4, we will explain the procedure for the process by which the portable information terminal 700 switches the output of the audio signal. Figure 4 is a flowchart showing the procedure for the process by which the portable information terminal 700 switches the output of the audio signal. It is assumed that the output pattern has been set in advance.

[0037] First, the main control unit 701 acquires the first audio content (first audio signal) (step S91), and then acquires the second audio content (second audio signal) (step S92). The audio signal processing unit 753 refers to the output pattern (set value) and outputs the first audio content to one of the first and second audio output units, and outputs the second audio content to the other of the first and second audio output units (step S93).

[0038] As described above, according to the portable information terminal 700 described in Embodiment 1, an output pattern is set, and according to this output pattern, a first audio signal is output to one of the first audio output unit and the second audio output unit, and a second audio signal is output to the other of the first audio output unit and the second audio output unit. For example, the portable information terminal 700 can output music content from the speaker audio output unit 751 and video content from the headphone audio output unit 752. In this way, the portable information terminal 700 can appropriately control the output of audio content using multiple audio output units.

[0039] (Example 2) Embodiment 2 of the present invention will now be described. This embodiment is an audio-linked playback system that uses the personal information terminal described in Embodiment 1. Unless otherwise specified, the configuration and effects of the personal information terminal in this embodiment are the same as those in Embodiment 1.

[0040] First, an example of content that can be played back by the audio-linked playback system of this embodiment will be described. In the audio-linked playback system of this embodiment, for example, a content display device, which is a television receiver, can play back BS / terrestrial digital broadcasts by receiving broadcast waves via an antenna.

[0041] Furthermore, if the content display device is capable of connecting to an external network, it can receive and play streaming content via the external network. Conversely, a mobile information terminal can transmit streaming content received via the external network to the content display device and play it.

[0042] Furthermore, if the portable information terminal 700 is capable of receiving one-segment broadcast waves, it can transmit and play one-segment broadcast content on a content display device. The above explanation is not limited to applications within Japan only.

[0043] Next, we will describe a specific example of the configuration of the embodiment in this example. [System Configuration] First, the system configuration of the voice-linked playback system will be explained using Figure 5 (Figures 5A to 5C). Figure 5 is a diagram showing the system configuration of the voice-linked playback system. Figure 5A is a system configuration diagram showing an example of the voice-linked playback system of this embodiment. The voice-linked playback system of this embodiment includes a content display device 100, an antenna 100a, the internet 200 which is a broadband network, a router device 210, a radio tower 300t of a broadcasting station, a base station 600b of a mobile telephone communication network, and a portable information terminal 700.

[0044] The content display device 100 is a television receiver that, in addition to existing digital broadcasting reception functions, also has functions compatible with a broadcast-communication-linked system. The content display device 100 receives broadcast waves transmitted from the radio tower 300t via the antenna 100a. Furthermore, the content display device 100 can connect to the internet 200 via the router device 210, and can transmit and receive data by communicating with various server devices on the internet 200.

[0045] The router device 210 is connected to the internet 200 by wireless or wired communication, and is also connected to the content display device 100 by wireless or wired communication, and to the personal information terminal 700 by wireless communication. This allows each server device on the internet 200, the content display device 100, and the personal information terminal 700 to send and receive data to and from each other via the router device 210. In other words, the content display device 100 and the personal information terminal 700 can communicate with each other. The radio tower 300t transmits broadcast waves, including digital broadcast signals, from the broadcasting equipment of the broadcasting station.

[0046] Figure 5B is a system configuration diagram showing another example of the voice-linked playback system of this embodiment. Unlike the system configuration shown in Figure 5A, it does not include a router device 210. In the system configuration shown in Figure 5B, instead of going through the router device 210, communication between the content display device 100 and the portable information terminal 700 is performed using a method such as Bluetooth®.

[0047] Figure 5C is a system configuration diagram showing another example of the audio linked playback system of this embodiment. Unlike the system configuration shown in Figure 5B, it uses an external transmitter 100b.

[0048] If the content display device 100 cannot connect to the router device 210 or does not have a communication method such as Bluetooth®, it transmits audio signals and the like to the portable information terminal 700 via the external transmission device 100b.

[0049] The three system configurations shown in Figures 5A to 5C differ in the communication method between the content display device 100 and the personal information terminal 700, but the system is not limited to just one communication method; two or more communication methods may be used in combination. As described above, the content display device 100 transmits various information (for example, audio data in digital broadcasting) to the personal information terminal 700 using various communication means (router device 210, external transmission device 100b, etc.). The personal information terminal 700 receives this information accordingly. In addition, the audio data that the content display device 100 transmits to the personal information terminal 700 includes information indicating whether it is a main audio or sub-audio.

[0050] Furthermore, as described above, the main control unit 701 of the personal information terminal 700 stores an application for coordinating with the content display device 100. When the personal information terminal 700 starts this application, it performs authentication processing with the content display device 100. In addition, the personal information terminal 700 stores volume reference information in advance and outputs the received information (audio data) as audio based on this volume reference information.

[0051] [Audio signal output pattern] Incidentally, since the audio linked playback system of this embodiment uses the portable information terminal 700 of Embodiment 1, it is possible to output either a first audio signal or a second audio signal from the speaker audio output unit 751 and the headphone audio output unit 752. For example, it is possible to output the audio from the speaker 174 of the content display device 100 from the speaker audio output unit 751 of the portable information terminal 700, and to output the audio signal of other audio content, such as video content on the network, from the headphone audio output unit 752. The portable information terminal 700 stores in advance output patterns that are defined for each data source to be output.

[0052] Figure 6 shows an example of the default settings for the audio output unit when playing various audio content on the portable information terminal 700. For content 20000 that displays video on the content display device 100, the default setting is to output the main audio, which is listened to by many users, from the speaker audio output unit 751, and the secondary audio from the headphone audio output unit 752.

[0053] On the other hand, in the case of content 20001 that displays video on the portable information terminal 700, since basically only one user, the person who owns the portable information terminal 700, views the content, the default setting is to output audio from the headphone audio output unit 752 regardless of the type of audio content.

[0054] Based on the default settings shown in Figure 6, when headphones are disconnected, the secondary audio of content 20000 and the audio signals of content 20001 cannot be output, so only the main audio of content 20000 is output from the speaker audio output unit 751. When headphones are connected, the audio signal processing unit 753 outputs the main audio signal of content 20000 from the speaker audio output unit 751. The audio signal processing unit 753 also outputs the secondary audio of content 20000 or the audio signal of content 20001 from the headphone audio output unit 752. When the headphones are disconnected, the audio signal processing unit 753 interrupts the output of the audio signal from the headphone audio output unit 752.

[0055] The settings shown in Figure 6 are merely the default settings, and the settings for each audio output in Figure 6 also allow for the selection of the other output unit.

[0056] [Hardware configuration of content display device] Figure 7 is a block diagram showing an example of the internal configuration of the content display device 100. The content display device 100 includes a main control unit 101, a system bus 102, an external communication unit 103, a tuner / demodulation unit 131, a separation unit 132, a video display unit 173, and a speaker 174 (third audio output unit).

[0057] The main control unit 101 is a microprocessor unit that controls the entire content display device 100 according to a predetermined operation program. The system bus 102 is a data communication path for sending and receiving data between the main control unit 101 and each operation block within the content display device 100.

[0058] If the external communication unit 103 includes, for example, a LAN (Local Area Network) communication unit, it can be configured to receive streaming content from the Internet via a router device 210 or to communicate with a mobile information terminal 700, as shown in Figure 5A. Furthermore, if the external communication unit 103 includes, for example, a Bluetooth® communication unit, it can be configured to communicate directly with the mobile information terminal 700, as shown in Figure 5B.

[0059] Furthermore, if the external communication unit 103 includes, for example, an audio output unit, it can transmit audio information to the portable information terminal 700 via the external transmission device 100b, as shown in Figure 5C. The communication method using the LAN communication unit, Bluetooth communication unit, and audio output unit is not limited to just one of these communication methods, and two or more communication methods may be used in combination. In addition, the external communication unit 103 may further include other communication units such as an NFC communication unit or an infrared communication unit.

[0060] When the external communication unit 103 receives audio data from the separation unit 132 (described later), it transmits the audio data to the portable information terminal 700. The external communication unit 103 may also transmit content received by the tuner / demodulation unit 131 to the portable information terminal 700.

[0061] The tuner / demodulator 131 receives broadcast waves from the radio tower 300t via the antenna 100a and tunes (selects) to the channel of the service desired by the user based on the control of the main control unit 101. Furthermore, the tuner / demodulator 131 demodulates the received broadcast signal to obtain a transport stream (TS). In the example shown in Figure 6, a configuration with one tuner / demodulator is illustrated, but the content display device 100 may be configured to have multiple tuners / demodulators for purposes such as simultaneous display of multiple screens or recording of programs while watching another program.

[0062] The separation unit 132 receives the TS output from the tuner / demodulation unit 131 and separates and outputs each data stream, such as a video data stream, an audio data stream, and a subtitle data stream. In this way, the separation unit 132 acquires video data and audio data from the received content (TS). The video display unit 173 is a display device such as an LCD panel and provides the video information and subtitle information separated by the separation unit 132 to the user of the content display device 100. The speaker 174 provides the audio information separated by the separation unit 132 to the user of the content display device 100.

[0063] The content display device 100 may be a television receiver, an optical disc drive recorder such as a DVD (Digital Versatile Disc) recorder, a magnetic disc drive recorder such as an HDD recorder, or an STB (Set Top Box). It may also be a PC (Personal Computer), tablet terminal, or game console equipped with digital broadcasting reception and broadcasting communication linkage functions. If the content display device 100 is a DVD recorder, HDD recorder, STB, etc., the video display unit 173 and speaker 174 do not need to be provided. By connecting an external monitor and external speaker to the external communication unit 103, the same operation as the content display device 100 in this embodiment becomes possible.

[0064] [Hardware configuration of the external transmitter] Figure 8 is a block diagram showing an example of the internal configuration of an external transmitter 100b used when the voice-linked playback system of this embodiment adopts the system configuration shown in Figure 5C, for example. The external transmitter 100b comprises a main control unit 100b1, a system bus 100b2, an audio input unit 100b3, and an external signal transmission unit 100b4.

[0065] The main control unit 100b1 is a microprocessor unit that controls the entire external transmitter 100b according to a predetermined operating program. The system bus 100b2 is a data communication path for sending and receiving data between the main control unit 100b1 and each operating block within the external transmitter 100b.

[0066] The audio input unit 100b3 acquires the audio signal output from the audio output unit of the external communication unit 103 within the content display device 100.

[0067] The external signal transmission unit 100b4 includes, for example, a Bluetooth® communication unit, which converts the audio signal acquired by the audio input unit and transmits it to the outside of the device. The external signal transmission unit 100b4 may further include other communication units such as an NFC communication unit or an infrared communication unit.

[0068] If the external communication unit 103 within the content display device 100 includes, for example, a video output unit, the external transmission device 100b includes a video input unit, enabling it to transmit video signals to the outside of the device, for example, via Bluetooth® communication. Audio signal input and video signal input may be performed using separate terminals, or they may be input simultaneously via, for example, an HDMI terminal. Furthermore, when both audio and video signals are input, the information transmitted by the external signal transmission unit 100b4 may be both audio and video signals, or only audio signals.

[0069] [Operation sequence during voice-linked playback] Figure 9 is an operation sequence diagram showing an example of the operation sequence of the mobile information terminal 700 when operating the voice-linked playback system of this embodiment. The diagram shows a series of events from when the mobile information terminal 700 starts the application for operating the voice-linked playback system of this embodiment, to when it continues to view content and then ends the viewing.

[0070] First, the personal information terminal 700 launches an application to operate the voice-linked playback system of this embodiment (S101) and authenticates the content display device 100 with which it will be linked. If the voice-linked playback system of this embodiment is configured as shown in Figure 5A, the personal information terminal 700 authenticates via the network by entering a password or the like. If the voice-linked playback system of this embodiment is configured as shown in Figure 5B, the content display device 100 is authenticated by a pairing operation or the like.

[0071] If the voice-linked playback system of this embodiment is configured as shown in Figure 5C, the personal information terminal 700 authenticates the external transmission device 100b through a pairing operation or the like. If the content display device 100 or the external transmission device 100b is equipped with an NFC communication unit, the personal information terminal 700 may directly authenticate the content display device 100 or the external transmission device 100b via the NFC communication unit 723.

[0072] Alternatively, the content display device 100 may display a QR code on the video display unit 173 at the user's request, and the image processing unit 740 of the mobile information terminal 700 may process the QR code through image processing to perform authentication. In particular, since the voice-linked playback system of this embodiment is intended for viewing content by multiple users, such as a family, using NFC communication or QR codes is easy to operate, making it possible for anyone from children to the elderly to easily start linking.

[0073] Once the application for operating the voice-linked playback system of this embodiment has been launched and authentication of each device is complete, voice-linked playback is started. The portable information terminal 700 allows setting the playback reference volume via the operation unit 730 to reduce differences in how sound is heard due to individual differences. The voice signal processing unit 753 confirms the setting value set by the operation unit 730, etc. (S103), then confirms the received sound received from the communication processing unit 720 (S105), and outputs the received sound from the speaker sound output unit 751 or the headphone sound output unit 752 (S106).

[0074] Whether or not to continue voice-linked playback is constantly monitored within the processing loop of S103 to S106 (S102). If an application interruption / termination request is input from the operation unit 730, or if the power to the content display device 100 is turned off and communication between the content display device 100 and the mobile information terminal 700 is interrupted, the voice-linked playback system of this embodiment is terminated. Through the above processing, the voice-linked playback system of this embodiment can be operated by the content display device 100 and the mobile information terminal 700.

[0075] Figure 10 is an operation sequence diagram showing a detailed example of the operation sequence of the voice output (S106) in an example of the operation sequence of the portable information terminal 700 shown in Figure 9. First, the voice signal processing unit 753 checks whether the portable information terminal 700 is in call mode before outputting voice (S10601).

[0076] Specifically, the audio signal processing unit 753 queries the main control unit 701 for information on the call status and obtains information from the main control unit 701 indicating whether or not a call is in progress to make a decision. If a call is in progress (S10601: Yes), the audio signal processing unit 753 terminates processing without outputting any audio signals so as not to interfere with listening to the incoming audio.

[0077] On the other hand, if the audio signal processing unit 753 is not in a call state (S10601: No), it makes the audio signal playable and then performs a connection determination by the headphone connection detection unit (S10602). If headphones are connected (S10602: Yes), the audio signal processing unit 753 can play two or more audio signals, so it uses one or both of the speaker audio output unit 751 to the headphone audio output unit 752 to output one or two audio signals (S10603).

[0078] On the other hand, if headphones are not connected, the audio signal processing unit 753 plays a single audio signal from the speaker audio output unit 751 because the only unit capable of outputting audio is the speaker audio output unit 751 (S10604). The selection of the audio signal to be output in processing S10603 to S10604 may be selected in processing S103, or it may be selected based on a pre-set default setting.

[0079] If an incoming call is received on the mobile information terminal 700 while an audio signal is being played back based on the operation sequence shown in Figure 10, the audio signal processing unit 753 changes its decision in the S10601 process and interrupts the output of the audio signal. When the call ends, the audio signal processing unit 753 changes its decision in the S10601 process again and resumes the output of the audio signal.

[0080] Furthermore, if the headphones are disconnected while audio signals are being played from both the speaker audio output unit 751 and the headphone audio output unit 752 with headphones connected, the decision in processing S10602 is changed to interrupt the output of the audio signal from the headphone audio output unit 752. When headphones are connected again, the audio signal processing unit 753 changes its selection in processing S10602 again and resumes the output of the audio signal from the headphone audio output unit 752.

[0081] The operation sequences in Figures 9 and 10 can be partially combined as appropriate, and some operation steps can be rearranged, operated simultaneously, etc., with other operation steps as appropriate. In addition, the operation sequences in Figures 9 and 10 can operate in the background except for user input, and will not interfere with the user's use of other applications.

[0082] [Examples of how to use the audio-linked playback system of this embodiment] Figure 11A is a schematic diagram illustrating an example of the use of the audio-linked playback system of this embodiment, showing two types of content viewing. User A 10000uaa and User B 10000uab are viewing content being played on the video display unit 173 of the content display device 100ua. User B 10000uab is located further away from the content display device 100ua than User A 10000uaa.

[0083] On the other hand, suppose user C 10000uac starts viewing related content on the personal digital assistant 700ua while user B 10000uab is viewing the aforementioned content. Here, the speaker audio output unit 751 of the personal digital assistant 700ua plays the audio played by the speaker 174 of the content display device 100ua. In addition, the headphone audio output unit 752 of the personal digital assistant 700ua plays the audio of the related content. As a result, user A 10000uaa and user B 10000uab can view the content being played on the video display unit 173 of the content display device 100ua, and user C 10000uac can view related content on the personal digital assistant 700ua without interfering with the playback of the aforementioned content.

[0084] Figure 11B is a schematic diagram showing content viewing on a two-screen content display device, which is an example of how the audio-linked playback system of this embodiment can be used. The content display device 100ub is equipped with two tuner / demodulation units 131, and the video display unit 173 is divided into two sections to play the first content and the second content.

[0085] Here, user A 10000uba and user B 10000ubb want to view the first content, while user C 10000ucc wants to view the second content. Therefore, the audio of the first content is played from the speaker 174 of the content display device 100ub and the speaker audio output unit 751 of the personal information terminal 700ub. ​​In addition, the audio of the second content is played from the headphone audio output unit 752 of the personal information terminal 700ub. ​​As a result, user A 10000uba and user B 10000ubb can view the first content, and user C 10000ucc can view the second content, both in terms of video and audio, without interfering with the viewing of the other content.

[0086] Figure 12A is a schematic diagram illustrating a multi-channel audio content viewing scenario, which is an example of how the audio-linked playback system of this embodiment can be used. The content display device 100uc is playing multi-channel audio content, including a first audio track and a second audio track.

[0087] Here, user A 10000uca and user B 10000ucb want to listen to the multi-audio content with the first audio, while user C 10000ucc wants to listen to the multi-audio content with the second audio. Therefore, the first audio is played from the speaker 174 of the content display device 100uc and the speaker audio output unit 751 of the personal information terminal 700uc. The second audio is also played from the headphone audio output unit 752 of the personal information terminal 700uc. As a result, user A 10000uca and user B 10000ucb can listen to the multi-audio content with the first audio, and user C 10000ucc can listen to it with the second audio, without interfering with the listening of the other user.

[0088] Figure 12B is a schematic diagram illustrating an example of how the audio-linked playback system of this embodiment can be used, specifically for viewing surround sound content. The content display device 100ud is playing content that includes surround sound, such as 5.1ch.

[0089] Here, it is assumed that mobile information terminals A 700uda, B 700udb, C 700udc, and D 700udd are arranged around user A 10000uda, user B 10000udb, and user C 10000udc, who are viewing the aforementioned content. The content display device 100ud transmits the surround sound to mobile information terminals A 700uda through D 700udd. Mobile information terminals A 700uda through D 700udd play back each transmitted audio. As a result, user A 10000uda through C 10000udc are able to view the surround sound content in a highly immersive viewing environment.

[0090] Furthermore, as an extension of the application for operating the audio-linked playback system of this embodiment, the number and relative positions of the content display device and multiple mobile information terminals may be registered, and based on the number and relative positions of the mobile information terminals, the application may generate audio for realizing the desired surround sound listening experience and transmit each of the audio to each mobile information terminal.

[0091] As explained above, the audio-linked playback system described in Embodiment 2 transmits audio data from the content display device 100 to the personal information terminal 700. Therefore, the audio can be output not only from the content display device 100 but also from the audio output unit of the personal information terminal 700 (for example, the speaker audio output unit 751). In this way, by using a personal information terminal 700 that can selectively control the output of multiple audio contents to multiple audio output units, it is possible to improve the viewing environment for the first content played on the content display device 100 while simultaneously realizing an audio-linked playback system that allows the user of the personal information terminal 700 to view a second content.

[0092] (Example 3) The following describes Embodiment 3 of the present invention. Unless otherwise specified, the configuration and effects in this embodiment are the same as those in Embodiment 2. Therefore, the following will mainly describe the differences between this embodiment and Embodiment 2, and will omit explanations of common points as much as possible to avoid duplication.

[0093] Although an embodiment of the voice-linked playback system was described in Example 2, a more detailed embodiment will be explained.

[0094] When performing audio-linked playback, it is sometimes desirable to create a listening environment where differences in sound perception due to distance from the content display device are minimized. In this case, it is desirable to control the speaker volume of the mobile information terminal 700 so that it increases as the distance from the content display device to the mobile information terminal 700 increases.

[0095] Therefore, in this embodiment, the portable information terminal 700 receives audio output from the content display device 100 and outputs audio from the speaker audio output unit 751 based on the volume of the input audio.

[0096] Specifically, the voice input unit 754 of the portable information terminal 700 inputs ambient sound (sound from outside the terminal), and the voice signal processing unit 753 acquires the sound. The voice signal processing unit 753 compares the acquired sound with voice data acquired by communication from the content display device 100 to identify the sound output from the content display device 100 from the input sound, and determines the volume of that sound. Based on the identified volume and a set reference volume, the voice signal processing unit 753 determines the volume of the voice data acquired from the content display device 100 and outputs the sound.

[0097] Figure 13 is a schematic diagram showing an example of volume control in the voice-linked playback system of this embodiment. As users of the content played on the content display device 100, there is user A 10000a located close to the content display device 100, and user B 10000b located far from the content display device 100. User A 10000a possesses mobile information terminal A 700a, and user B 10000b possesses mobile information terminal B 700b.

[0098] User A 10000a hears the sound output from the speaker 174 of the content display device 100 at a speaker volume of 100sa at User A's position. On the other hand, User B 10000b hears the sound output from the speaker 174 of the content display device 100 at a speaker volume of 100sb at User B's position. The speaker volume of 100sb at User B's position is lower than the speaker volume of 100sa at User A's position, so User B 10000b will have more difficulty hearing the sound output from speaker 174 compared to User A 10000a.

[0099] Therefore, mobile information terminal A 700a and mobile information terminal B 700b work in conjunction with the content display device 100. Mobile information terminal A 700a plays audio from its speaker audio output unit 751 at a speaker volume of 700sa, and mobile information terminal B 700b plays audio from its speaker audio output unit 751 at a speaker volume of 700sb.

[0100] Compared to the speaker volume of mobile device A (700sa), the speaker volume of mobile device B (700sb) is louder. In this case, the volume heard by user A (800sa) and user B (800sb) will be 800sa and 800sb, respectively. Mobile device A (700a) and mobile device B (700b) control their speaker volumes (700sa and 700sb, respectively) so that the volume heard by user A (800sa) and user B (800sb) are approximately the same.

[0101] Figure 14 is an operation sequence diagram showing an example of an operation sequence that takes into account the volume control of the speaker audio output unit 751, compared to the operation sequence of the portable information terminal 700 shown in Figure 9. While the process in S105 in Figure 9 only involved checking the received audio, the process in S104 in Figure 14 involves checking the input audio from the audio input unit 754 in addition to the received audio.

[0102] After confirming the received audio and the input audio, the audio signal processing unit 753 measures the volume of the input audio input from the audio input unit 754 (S107). As shown in Figure 13, the audio signal processing unit 753 calculates the volume of the audio output from the speaker audio output unit 751 of the portable information terminal 700 so that the sum of the volumes of the output audio output from the speaker 174 of the content display device 100 and the output audio output from the speaker audio output unit 751 of the portable information terminal 700 is always close to the playback reference volume (S108). Then, the portable information terminal 700 outputs the audio at the calculated volume from the speaker audio output unit 751 (S106).

[0103] Figure 15 is an operation sequence diagram showing a detailed example of the operation sequence of the input voice confirmation / received voice confirmation (S104) in an example of the operation sequence of the portable information terminal 700 shown in Figure 14. First, the voice signal processing unit 753 confirms the input voice input from the voice input unit 754 of the portable information terminal 700 (S10401) and stores the input voice data in a buffer that can buffer for a predetermined time (S10402).

[0104] Meanwhile, the communication processing unit 720 of the mobile information terminal 700 checks the received audio (S10403) and stores the received audio data in a buffer capable of buffering for a predetermined time (S10404). Then, it detects the portion between the buffer containing the input audio data and the buffer containing the received audio that is determined to be the same audio data (S10405). By performing this detection, it becomes possible to calculate the magnitude of only the audio signal component of the audio content contained in the input audio data.

[0105] Each operation sequence in Figures 14 and 15 can be partially combined as appropriate, and some operation steps can be reordered, operated simultaneously, etc., with other operation steps as appropriate. In addition, each operation sequence in Figures 14 and 15 can operate in the background except for user input, and will not interfere with the user's use of other applications.

[0106] When the speaker audio output unit 751 of the portable information terminal 700 outputs the same audio signal as the audio output from the speaker 174 of the content display device 100, it is desirable to turn on the volume control by default in order to reduce differences in how the sound is heard due to differences in distance from the content display device 100. On the other hand, when the audio signal is output from the headphone audio output unit 752, the difference in how the sound is heard due to differences in distance from the content display device 100 is very small, so it is desirable to turn it off by default.

[0107] As explained above, the speaker audio output unit 751 receives external audio from the portable information terminal 700, and the audio signal processing unit 753 controls the audio output based on the input audio. Specifically, the audio signal processing unit 753 determines and outputs the volume of the received audio data based on the audio input by the speaker audio output unit 751. This allows audio data to be output from the speaker audio output unit 751 at an appropriate volume even to users who are far away from the content display device 100.

[0108] As described above, the present invention enables an audio-linked playback system that minimizes differences in how sound is heard due to differences in distance from the content display device 100 by controlling the speaker volume of the mobile information terminal 700 to increase as the distance from the content display device 100 increases.

[0109] (Example 4) The following describes Embodiment 4 of the present invention. Unless otherwise specified, the configuration and effects in this embodiment are the same as those in Embodiment 2. Therefore, the following will mainly describe the differences between this embodiment and Embodiment 2, and will omit explanations of common points as much as possible to avoid duplication.

[0110] Although an embodiment of the voice-linked playback system was described in Example 2, a more detailed embodiment will be explained.

[0111] When performing audio-synchronized playback, there may be a timing discrepancy between the playback timing of the audio signal output from the speaker of the content display device 100 and the playback timing of the same audio signal output from the speaker audio output unit 751 of the portable information terminal 700. This is due to a communication delay between the content display device 100 and the portable information terminal 700. In this case, to obtain a suitable viewing environment, it is desirable to perform control to synchronize the playback timing of the same audio signal, as well as the playback timing of the video played back by the content display device.

[0112] The voice-linked playback system of this embodiment adjusts the playback timing described above. Specifically, the voice input unit 754 of the portable information terminal 700 inputs ambient sound, and the voice signal processing unit 753 acquires and stores the sound. The voice signal processing unit 753 also stores the voice data acquired via the communication processing unit 720.

[0113] The audio signal processing unit 753 compares the audio input from the audio input unit 754 with the audio data acquired via the communication processing unit 720 to determine whether there is a match. If there is a match, the audio signal processing unit 753 compares the time of audio input with the time of reception of the received data to calculate the time difference. Based on this time difference, the audio signal processing unit 753 adjusts the audio output timing. Specifically, if the time of audio input is earlier, the audio signal processing unit 753 controls the playback timing of the content display device 100 to be delayed. If the audio data is earlier, it delays the playback timing of the portable information terminal 700.

[0114] Since synchronization control is performed based on the input audio input from the audio input unit 754, the operation sequence of the portable information terminal 700 is as shown in Figure 16. Figure 16 is an example of an operation sequence that takes synchronization control into consideration, in contrast to the detailed operation sequence of input audio confirmation / received audio confirmation (S104) shown in Figure 15. After detecting the identical parts of the input audio data and the received audio data (S10405), the time difference between the audible timing of the identical audio data in the input audio and the playback timing of the identical audio data in the received audio is calculated (S10406).

[0115] Here, if the input audio is faster, the content playback timing in the content display device 100 is delayed (S10408), while if the received audio is faster, the audio playback timing in the speaker audio output unit 751 of the portable information terminal 700 is delayed (S10409). The process in S10408 can only be executed when communication from the portable information terminal 700 to the content display device 100 is possible, so this embodiment of the audio-coordinated playback system is effective when the system configuration shown in Figure 5A or Figure 5B is adopted.

[0116] Figure 17 is a schematic diagram showing an example of synchronization control in the audio-linked playback system of this embodiment. It is assumed that the content is BS / terrestrial digital broadcasting received by the content display device 100. Considering the transmission time between the content display device 100 and the portable information terminal 700, the audio output from the speaker 174 of the content display device 100, i.e., the input audio input from the audio input unit 754 of the portable information terminal 700, is basically faster data in time than the received audio received by the communication processing unit 720 of the portable information terminal 700.

[0117] Therefore, the audio signal processing unit 753 performs the S10408 process to eliminate the time difference between the audible timing 11000a of the input audio 100sp before synchronization and the playback timing 11000b of the received audio 700sp before synchronization. In the S10408 process, first an instruction to delay the content playback timing in the content display device 100 is transmitted to the external communication unit 103 of the content display device 100. Upon receiving the instruction, the content display device 100 delays the playback timing of the speaker 174 and the video display unit 173 with a resolution of approximately 50 msec 100sdt by utilizing lip-sync technology, etc.

[0118] However, if the audible timing of the input audio and the playback timing of the received audio are not synchronized to an even finer order of magnitude, taking into account the propagation delay of sound in the air, the user will perceive an echo. Therefore, the audio signal processing unit 753 delays the playback timing of the speaker 174 based on the resolution 100sdt so that the audible timing of the input audio is later than the playback timing of the received audio, and at the same time delays the playback timing of the speaker audio output unit 751 of the portable information terminal 700 with an even finer resolution of about 20usec, 700sdt.

[0119] The audible timing of the synchronized input audio 100sq, delayed based on a resolution of 100sdt, and the playback timing of the synchronized received audio 700sq, delayed based on a resolution of 700sdt, coincide at the synchronization timing of 11000c, allowing the user to view content in a listening environment with reduced echo.

[0120] In this embodiment of the audio-linked playback system, even when viewing streaming content received by the content display device 100, synchronization control can be performed based on the operation sequence shown in Figure 16. Unlike when viewing BS / terrestrial digital broadcasts, the playback timing of the content display device 100 may be delayed by controlling the program for streaming in the S10408 processing.

[0121] In this embodiment, the audio-linked playback system assumes that the content is streamed content or one-segment broadcast content received by the mobile information terminal 700. In this case, considering the transmission time from the mobile information terminal 700 to the content display device 100, the playback timing of the audio output from the speaker 174 of the content display device 100 will basically be delayed compared to the playback timing of the audio output from the speaker audio output unit 751 of the mobile information terminal 700. Therefore, synchronization control can be performed by processing S10409 alone, which delays the playback timing of the mobile information terminal 700.

[0122] The operation sequence in Figure 16 can be partially combined as needed, and some operation steps can be rearranged, operated simultaneously, etc., with other operation steps as appropriate. In addition, the operation sequence in Figure 16 can operate in the background except for user input, and will not interfere with the user's use of other applications.

[0123] Even when linking two or more portable information terminals 700 to one content display device 100 to construct the audio-linked playback system of this embodiment, it is possible to perform synchronized control of audio, video, etc., in all these devices by performing synchronized control based on the operation sequence shown in Figure 16.

[0124] Synchronization control is desirable to be enabled by default, for the purpose of synchronizing the playback timing with the video played by the content display device 100, regardless of whether the same audio signal output from the speaker 174 of the content display device 100 is output from the speaker audio output unit 751 or headphone audio output unit 752 of the portable information terminal 700.

[0125] As described above, in this embodiment, the audio signal processing unit 753 identifies the timing input by the audio input unit 754 and controls the timing of output to the speaker audio output unit 751 based on that timing. This makes it possible to appropriately adjust the output timing of the audio output from the content display device 100 and the audio output from the portable information terminal 700, and can appropriately adjust even if, for example, there is a discrepancy in the reception timing due to a communication delay.

[0126] (Example 5) The following describes Embodiment 5 of the present invention. Unless otherwise specified, the configuration and effects in this embodiment are the same as those in Embodiments 2 to 4. Therefore, the following will mainly describe the differences between this embodiment and Embodiments 2 to 4, and will omit explanations of common points as much as possible to avoid duplication.

[0127] While embodiments of the voice-linked playback system were described in Examples 2 to 4, more detailed embodiments will be explained below.

[0128] When performing audio-synchronized playback, it may be desirable to analyze ambient sounds in addition to the audio of the content being played and control the volume of the content's audio according to the surrounding environment. For example, if a user of the content is spoken to by someone else, it is desirable to lower the volume of the mobile device's speaker so that the conversation can be heard more clearly. On the other hand, if there is a lot of background noise around the user of the content, it is desirable to raise the volume of the mobile device's speaker so that the audio of the content can be heard more clearly.

[0129] The voice-linked playback system of this embodiment analyzes the type of ambient sound and controls the volume of the output sound based on that type. Specifically, the voice input unit 754 of the portable information terminal 700 inputs ambient sound, and the voice signal processing unit 753 acquires and stores the sound. The voice signal processing unit 753 also stores the voice data acquired via the communication processing unit 720.

[0130] The audio signal processing unit 753 compares the audio input from the audio input unit 754 with the audio data acquired via the communication processing unit 720, extracts the matching portions, and extracts the non-matching portions (ambient sound portions) from the audio input from the audio input unit 754. The audio signal processing unit 753 identifies the type of ambient sound (e.g., conversation, ambient noise), determines the audio output volume of the received audio data according to the identified type of ambient sound, and outputs the audio based on that volume.

[0131] In this embodiment, the procedure for realizing the above detailed embodiment will be described. Figure 18 is an operation sequence diagram showing an example of an operation sequence that takes into account volume control based on ambient sound analysis, compared to the operation sequence of the portable information terminal 700 shown in Figure 9. In the process of S104, when data matching the received audio, i.e., the audio of the content, is detected from the input audio input from the audio input unit 754, the audio of the content and ambient sound are separated in the input audio (S109).

[0132] The audio signal processing unit 753 performs a type analysis based on the separated ambient sound to determine whether the main component of the ambient sound is, for example, conversation or everyday sounds (S110). Meanwhile, the audio signal processing unit 753 measures the volume of the ambient sound (S111), and in the processing of S108, it calculates the volume of the sound to be output from the speaker audio output unit 751 of the portable information terminal 700, taking into account the type analysis results and volume of the ambient sound.

[0133] As an example of a method for controlling the volume, if the type analysis result of the ambient sound is, for example, conversation, the sound output from the speaker sound output unit 751 will be uniformly set to a predetermined low volume, and if the type analysis result of the ambient sound is, for example, everyday noise, the volume of the sound output from the speaker sound output unit 751 will be controlled so that it becomes louder as the volume of the everyday noise increases.

[0134] Here, voice separation (S109) and ambient sound analysis (S110), etc., may be performed inside the mobile information terminal 700 or on a server device on the Internet. If processes S109 and S110, etc. require a long processing time, the maximum processing time may be estimated in advance, and this maximum processing time may be taken into consideration in the synchronization control in the S104 process. Then, in the S10406 ​​process shown in Figure 7B, the time difference with the input voice may be calculated by taking into consideration the maximum processing time, that is, the time from confirming the received voice (S10403) to analyzing the ambient sound and actually outputting (S106).

[0135] Each operation sequence in Figure 18 can be partially combined as appropriate, and some operation steps can be reordered, operated simultaneously, etc., with other operation steps as appropriate. In addition, each operation sequence in Figure 18 can operate in the background except for user input, and will not interfere with the user's use of other applications.

[0136] In this way, the audio signal processing unit 753 extracts ambient sounds, analyzes the type of ambient sound, and controls the output volume based on the analysis results, thereby enabling volume control according to the surrounding conditions.

[0137] As described above, the present invention makes it possible to realize an audio-linked playback system that can analyze ambient sounds and control the volume of the content's audio according to the surrounding environment.

[0138] (Example 6) The following describes Embodiment 6 of the present invention. Unless otherwise specified, the configuration and effects of this embodiment are the same as those of Embodiments 2 to 4. Therefore, the following description will mainly focus on the differences between this embodiment and Embodiments 2 to 4, and common points will be omitted as much as possible to avoid repetition.

[0139] While embodiments of the voice-linked playback system were described in Examples 2 to 4, more detailed embodiments will be explained below.

[0140] There are times when you may want to make a phone call using your mobile device while viewing content via an audio-linked playback system. In this case, to ensure clear audio is delivered to the person you are calling, it is desirable to either increase the amount of call content in the audio input from the microphone, or to reduce the amount of audio content being viewed as much as possible.

[0141] One example of the former method is to apply the audio separation processing (S109) in Example 5 to separate the audio component of the conversation content and the audio component of the content being viewed, and then amplify only the conversation content component. One example of the latter method is to use the received audio information received for output from the speaker audio output unit 751 to cancel out the audio component of the content being viewed, which is included in the audio input from the microphone.

[0142] The voice collaboration system of this embodiment appropriately controls call audio (input audio from a call) when the mobile information terminal 700 is processing a call while outputting audio data received from the content display device 100. Specifically, the voice signal processing unit 753 queries the main control unit 701 for information on the call status and obtains information from the main control unit 701 indicating whether or not a call is in progress to make a determination. The voice signal processing unit 753 inverts the phase of the received audio data and adjusts the amplitude based on the volume of the audio component of the content input to the voice input unit 754, and adds it to the user's outgoing voice.

[0143] In this embodiment, the procedure for realizing the above detailed embodiment will be described. Figure 19 is an operation sequence diagram showing an example of an operation sequence that takes into account the coordinated operation of the application for operating the voice-linked playback system of this embodiment and the call application, in relation to the operation sequence of the mobile information terminal 700 shown in Figure 9. The call application starts a processing loop when an incoming or outgoing call is received (S121). First, it checks the incoming call audio (S123) and outputs the incoming call audio from the earpiece (S124). Meanwhile, it checks the user's outgoing voice from the audio input unit 754 (S125).

[0144] Here, in transmitting the aforementioned voice, the audio of the content output from the speaker audio output unit 751 of the portable information terminal 700, which is operating as a voice-linked playback system, is used. Specifically, in the processing of S106, the audio signal processing unit 753 performs phase inversion of the audio of the content output from the speaker audio output unit 751 and amplitude adjustment based on the volume of the audio component of the content included in the transmitted voice (S120). Then, by adding this to the user's transmitted voice confirmed in the processing of S125 (S126), the audio component of the content included in the transmitted voice is reduced, making it possible to generate a clear transmitted voice. Then, the clear transmitted voice is transmitted (S127).

[0145] Whether or not to continue the call is constantly monitored within the processing loop of S123 to S127 (S122). If a call termination request is input from the operation unit 730 or the other party to the call requests to terminate the call, the call in this embodiment is terminated.

[0146] Each operation sequence in Figure 19 can be partially combined as appropriate, and some operation steps can be reordered, operated simultaneously, etc., with other operation steps as appropriate. In addition, each operation sequence in Figure 19 can operate in the background except for user input, and will not interfere with the user's use of other applications.

[0147] As in the embodiment described above, the audio signal processing unit 753 enhances the call audio contained in the audio input by the audio input unit 754 based on the audio input by the audio input unit 754 and the audio data acquired via the communication processing unit 720. Specifically, as part of this enhancement process, the audio signal processing unit 753 adds the result of inverting the phase of the audio data received by the communication processing unit 720 to the audio input by the audio input unit 754. As a result, the audio signal processing unit 753 weakens the received audio data (audio content) portion, so that clear audio can be used for calls even while viewing content.

[0148] Furthermore, the audio signal processing unit 753 may perform the audio separation processing described in Embodiment 5, as shown in the example above, to separate the call content component from the audio component of the content being viewed, and amplify only the call content component (the call audio portion). In this case, since the audio signal processing unit 753 emphasizes the call content portion, it is possible to make a call with clear audio even while viewing content.

[0149] (Example 7) The following describes Embodiment 7 of the present invention. Unless otherwise specified, the configuration and effects in this embodiment are the same as those in Embodiments 2 to 4. Therefore, the following will mainly describe the differences between this embodiment and Embodiments 2 to 4, and will omit explanations of common points as much as possible to avoid duplication.

[0150] While embodiments of the voice-linked playback system were described in Examples 2 to 4, more detailed embodiments will be explained below.

[0151] When viewing content using an audio-linked playback system, there may be times when users want to display not only audio but also video or subtitles on their mobile devices. In particular, regarding subtitles, when displayed on a content display device used by multiple users, some users may find subtitles superimposed on the video distracting. In such cases, it would be desirable to, for example, not display subtitles on the content display device itself, but only on the mobile devices of users who require them.

[0152] In the voice-linked system of this embodiment, the mobile information terminal 700 receives subtitle data related to the voice data and displays the subtitle data.

[0153] In this embodiment, the procedure for realizing the above-described detailed embodiment will be explained. Figure 20 is an example of the display screen of the display unit 741 of the portable information terminal 700. The display unit 741 has a touch panel function and also serves as the operation unit 730. First, in menu 741b, the user selects whether or not to display video and subtitles. If the user selects to display subtitles, subtitles 741a are displayed.

[0154] The subtitle 741a can be slid up and down using the touch panel function to access subtitle data up to a predetermined time, allowing users to review the missed content if they temporarily miss or mishear something without stopping playback. The displayed subtitle 741a can also be moved by operating the knob 741d on the scroll bar 741c, which indicates the current subtitle position.

[0155] As described above, the present invention makes it possible to realize an audio-linked playback system that can display not only audio but also video or subtitles on a mobile information terminal.

[0156] (Example 8) The following describes Embodiment 8 of the present invention. Unless otherwise specified, the configuration and effects in this embodiment are the same as those in Embodiments 2 to 4. Therefore, the following description will mainly focus on the differences between this embodiment and Embodiments 2 to 4, and common points will be omitted as much as possible to avoid repetition.

[0157] While embodiments of the voice-linked playback system were described in Examples 2 to 4, more detailed embodiments will be explained below.

[0158] When performing audio-synchronized playback, there may be cases where one or more audio tracks or subtitles in the content being played do not include a language that the viewer understands. In such cases, it is desirable to translate using one or more of the aforementioned audio tracks or subtitles and provide them as audio or subtitles in a language that the user understands.

[0159] In this embodiment, the personal information terminal 700 receives subtitle data, requests an external server (translation unit) to translate the received subtitle data, the external server translates the data in response to the translation request, and displays or outputs the translation result. The personal information terminal 700 itself may also be equipped with translation means.

[0160] In this embodiment, the procedure for realizing the above-described detailed embodiments will be explained. Figure 21 is a diagram showing an example of the various processes required for various translation patterns. As for the translation patterns, there are two patterns for the data before translation, "subtitles" and "audio," and there are also two patterns for the data after translation, "subtitles" and "audio," so there are 2 x 2 = 4 possible combinations of translation patterns.

[0161] Specifically, as shown in Figure 21, these are subtitle-to-subtitle translation pattern 23000, subtitle-to-audio translation pattern 23001, audio-to-subtitle translation pattern 23002, and audio-to-audio translation pattern 23003. In addition, since translation is generally done on a text basis, the following processes are required to perform the above four translation patterns: speech-to-text processing 23004 to convert audio to text, text-based translation processing 23005, and text-to-speech processing 23006 to convert text to audio. By applying the above three types of processing as shown in Figure 21, the above four translation patterns become possible.

[0162] The voice-to-subtitle translation pattern 23002 and the voice-to-voice translation pattern 23003 are executed in the operation sequence of the portable information terminal 700 shown in Figure 14, after the received audio is confirmed during processing S104. The subtitle-to-subtitle translation pattern 23000 and the subtitle-to-voice translation pattern 23001 are also performed using subtitles transmitted at the same timing as the audio, and are therefore executed at the same timing as the translation patterns that use audio. Here, the speech-to-text processing 23004 to the text-to-speech processing 23006 may be performed inside the portable information terminal 700 or on a server device on the Internet.

[0163] Since the translated subtitles or audio are data not originally included in the content, the timing of their provision to the user cannot be precisely synchronized as shown in the operation sequence in Figure 16, but it is desirable that they be roughly synchronized with the video content. Therefore, if the speech-to-text processing 23004 to the text-to-speech processing 23006 require a long processing time, the maximum processing time should be estimated in advance. Then, in the synchronization control in processing S104, the maximum processing time may be taken into consideration, and in the processing of S10406 ​​shown in Figure 16, the time difference with the input audio may be calculated by taking into consideration the maximum processing time, that is, the time from confirming the received audio (S10403) to performing the translation and actually outputting it (S106).

[0164] The settings shown in Figure 21 are merely examples; for example, it is also permissible to apply speech-based translation processing, which allows for speech-to-speech translation, instead of text-based translation processing.

[0165] As described above, the present invention makes it possible to realize an audio-linked playback system that performs translation using one or more types of audio or one or more types of subtitles, and provides audio or subtitles in a language that the user can understand.

[0166] The embodiments of the present invention have been described above using Examples 1 to 8. Needless to say, the configurations that realize the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0167] For example, the personal information terminal 700 may use a short-range wireless communication means (e.g., a beacon) to calculate the distance between the personal information terminal 700 and the content display device 100, and control the volume of the received audio data output based on the calculated result. For example, the personal information terminal 700 may store information that defines the volume according to the distance, refer to this information to identify the volume corresponding to the calculated distance, and output the audio data at that volume.

[0168] In this case, if the personal information terminal 700 is configured to increase the volume when the distance is long, it can output audio data at a high volume even when it is far from the content display device 100, thereby improving the listening environment.

[0169] Furthermore, the content display device 100 may, upon receiving notification of a change in the content to be received, transmit the audio data of the changed content to the personal information terminal 700. Alternatively, the content display device 100 may, upon receiving notification of a change in the content to be received, transmit the content before the change to the personal information terminal 700.

[0170] Furthermore, when the content display device 100 receives notification of a change in the content to be received, it may pre-determine whether or not to change the audio data to be transmitted to the mobile information terminal 700 in accordance with the change, or it may inquire with the mobile information terminal 700.

[0171] The functions of the present invention described above may be implemented in hardware, either partially or entirely, by designing them, for example, using an integrated circuit. Alternatively, they may be implemented in software by having a microprocessor unit or the like interpret and execute an operating program that implements each of these functions. Hardware and software may also be used in combination.

[0172] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is reasonable to assume that almost all components are interconnected.

[0173] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0174] 100...Content display device, 100a...Antenna, 100b...External transmission device, 100b1...Main control unit, 100b2...System bus, 100b3...Audio input unit, 100b4...External signal transmission unit, 101...Main control unit, 102...System bus, 103...External communication unit, 131...Tuner / demodulation unit, 132...Separation unit, 173...Video display unit, 174...Speaker, 200...Internet, 210...Router device, 300t...Broadcasting station radio tower, 600b...Mobile telephone communication network Base station, 700…Mobile information terminal, 701…Main control unit, 702…System bus, 720…Communication processing unit, 721…LAN communication unit, 722…Mobile telephone network communication unit, 723…NFC communication unit, 730…Operation unit, 740…Image processing unit, 741…Display unit, 742…Image signal processing unit, 743…First image input unit, 744…Second image input unit, 750…Audio processing unit, 751…Speaker audio output unit, 752…Headphone audio output unit, 753…Audio signal processing unit, 754…Audio input unit.

Claims

[Claim 1] It is an audio output terminal, A wireless communication unit that acquires audio content, The audio output unit outputs the aforementioned audio content, A control unit controls the audio output unit based on setting information including first information regarding an audio content output pattern which includes at least a first pattern which outputs the audio content from the audio output unit and a second pattern which does not output the audio content, and second information regarding the on / off status of the volume control function of the audio output unit. The audio input unit acquires ambient sound from the aforementioned audio output terminal, Equipped with, The control unit is characterized by analyzing ambient sound acquired via the audio input unit to identify the type of ambient sound, and controlling the volume of the audio output unit based on the identified type of ambient sound and the setting information.

Images