Content playback device and content playback method

Abstract

To provide a content playback device that allows users to understand the content even when they look away from the screen, and to view content efficiently. [Solution] The gap detection unit 28 detects gaps in the spoken audio contained in the content. The gaze detection unit 14 detects the gaze of the user looking at the screen of the display unit 21 and determines whether the user's gaze is away from the screen. The audio data generation unit (video analysis / text data generation unit 16 and audio data conversion unit 17) generates explanatory audio data that describes the scenes during the period when the gaze is away from the screen if the state of the gaze being away from the screen continues for a predetermined time or longer. The content playback control unit 20 causes the speaker 22 to output explanatory audio based on the explanatory audio data in the gaps in the content being played.

Description

Technical Field

[0001] The present invention relates to a content playback device and a content playback method.

Background Art

[0002] There is an increasing number of users who watch content while doing other things than just watching the content. Therefore, the user may sometimes look away from the screen of the display unit on which the content is displayed. When the user looks away from the screen, they may not be able to understand the content. In such a case, if the content allows the user to return the playback position, the user may return the playback position and watch the content again.

[0003] Patent Document 1 describes that when the user looks away from the screen, the playback of the content is stopped.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] It is troublesome for the user to return the playback position and watch the content again when they look away from the screen, which causes a time loss. If the playback of the content is stopped every time the user looks away from the screen, the content cannot be watched efficiently. There is a demand for the emergence of a content playback device and a content playback method that can reduce the time loss as much as possible, allow the user to understand the content even if they look away from the screen, and enable the user to watch the content efficiently.

Means for Solving the Problems

[0006] The present invention provides a content playback device comprising: a content playback control unit that displays the video of the content on a display unit and plays the content so as to output the audio of the content from a speaker; a gap detection unit that detects gaps in the spoken audio contained in the content; a gaze detection unit that detects the gaze of a user looking at the screen of the display unit and determines whether or not the user's gaze is away from the screen; and an audio data generation unit that, when the state in which the gaze is away from the screen continues for a predetermined time or longer, generates explanatory audio data that describes the scenes within the period based on the analysis results obtained by analyzing the video of the content during the period in which the gaze is away from the screen; wherein the content playback control unit causes the content playback control unit to output explanatory audio based on the explanatory audio data from the speaker to the gaps detected by the gap detection unit in the content being played.

[0007] The present invention provides a content playback method that displays the video of content on a display unit, plays the content so that the audio of the content is output from a speaker, detects gaps in the spoken audio contained in the content, detects the gaze of the user looking at the screen of the display unit, determines whether the user's gaze is away from the screen, generates explanatory audio data describing the scenes within a predetermined period of time or longer based on the analysis results obtained by analyzing the video of the content during the period in which the gaze is away from the screen, and outputs explanatory audio based on the explanatory audio data from the speaker to fill in the gaps in the content being played. [Effects of the Invention]

[0008] According to the content playback device and content playback method of the present invention, time loss can be reduced as much as possible, the content can be understood even if the user takes their eyes off the screen, and the content can be viewed efficiently. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a block diagram showing a content playback device according to the first embodiment. [Figure 2] Figure 2 shows an example of a scene transition in content. [Figure 3] Figure 3 is a flowchart showing the content playback method according to the first embodiment. [Figure 4] Figure 4 is a timing chart showing an example of the operation of the content playback device according to the first embodiment. [Figure 5] Figure 5 is a block diagram showing an example configuration that is an evolution of the content playback device according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing a content playback device according to the second embodiment. [Figure 7] Figure 7 is a flowchart showing some of the processing in the content playback method according to the second embodiment. [Figure 8] Figure 8 is a flowchart showing the main processes in the content playback method according to the second embodiment. [Figure 9] Figure 9 is a timing chart showing an example of the operation of the content playback device according to the second embodiment. [Modes for carrying out the invention]

[0010] The content playback device and content playback method according to each embodiment will be described below with reference to the attached drawings.

[0011] <First Embodiment> Figure 1 shows a content playback device 100 according to the first embodiment. As an example, the content playback device 100 is a content playback device that receives and plays content provided by the content provision server 50 via the network 30. The content playback device 100 may also be a content playback device that receives and plays broadcast waves from terrestrial digital broadcasting or satellite broadcasting. The content playback device 100 may also be a content playback device that plays packaged media such as DVDs or Blu-ray discs.

[0012] The content playback device 100 includes a communication unit 10, a video buffer 11, an audio buffer 12, a camera 13, a gaze detection unit 14, a video storage unit 15, a video analysis / text data generation unit 16, an audio data conversion unit 17, an audio detection unit 18, a content playback control unit 20, a display unit 21, and a speaker 22. The video buffer 11 and the audio buffer 12 may be composed of a single buffer. The video storage unit 15 is composed of rewritable memory such as RAM. The display unit 21 is a display panel such as a liquid crystal panel or an organic EL panel. Typically, the network 30 is the internet.

[0013] At least a portion of the gaze detection unit 14, the video analysis / text data generation unit 16, the audio data conversion unit 17, the audio detection unit 18, and the content playback control unit 20 may consist of a central processing unit of a computer and a non-temporary storage medium that stores computer programs that cause the central processing unit to perform those processes. At least one of the camera 13, the display unit 21, and the speaker 22 may be an external component of the content playback device 100 and may be externally attached to the content playback device 100.

[0014] The communication unit 10 receives the content provided by the content providing server 50 via the network 30. The video signal constituting the received content is buffered in the video buffer 11 for a predetermined time, and the audio signal constituting the content is buffered in the audio buffer 12 for a predetermined time. The content playback control unit 20 supplies the video signal buffered in the video buffer 11 to the display unit 21 and supplies the audio signal buffered in the audio buffer 12 to the speaker 22. An analog audio signal is supplied to the speaker 22. Thereby, the content playback control unit 20 displays the video of the content on the display unit 21 and outputs the audio of the content from the speaker 22.

[0015] The camera 13 photographs the user who is looking at the screen of the display unit 21. The gaze detection unit 14 detects the user's gaze based on the photographed image of the user supplied from the camera 13 and determines whether the user's gaze is directed at the screen or away from the screen. As a method for detecting the user's gaze by the gaze detection unit 14, a known method may be adopted. As an example, a non-moving part (reference point) and a moving part (moving point) of the eyes are specified, and the gaze can be detected based on the position of the moving point with respect to the reference point. As an example, the reference point can be the inner corner of the eye and the moving point can be the iris.

[0016] When the gaze detection unit 14 determines that the user's gaze has deviated from the screen of the display unit 21, the video storage unit 15 starts storing the video signal supplied from the video buffer 11. The video storage unit 15 is configured to stop storing the video signal if the user's gaze deviates from the screen of the display unit 21 for less than a predetermined time and returns to the screen of the display unit 21 within less than the predetermined time. When the user's gaze deviates from the screen of the display unit 21 again, the video storage unit 15 starts overwriting the video signal. The predetermined time is, for example, 5 seconds. However, the video storage unit 15 may store the video signal for more than 5 seconds.

[0017] The voice detection unit 18 detects whether there is a spoken voice at each timing in the content based on the voice signal buffered in the voice buffer 12, and when there is a spoken voice, detects the timing when the spoken voice ends (the end of the spoken voice). The spoken voice is the lines uttered by an actor if the content is a movie or drama, etc., and the voice uttered by a participant if the content is interview content or news, etc. The end of the spoken voice means the end of one instance of the spoken voice. The voice detection unit 18 may detect the timing when the voice becomes silent as the end of the spoken voice.

[0018] The voice detection unit 18 supplies information indicating whether there is a spoken voice at each timing and information indicating the end of the spoken voice to the video memory unit 15. If information indicating that there is a spoken voice is input when the video memory unit 15 has stored a video signal for 5 seconds, the video memory unit 15 continues to store the video signal. When information indicating the end of the spoken voice is input, the video memory unit 15 stops storing the video signal that exceeds 5 seconds. If information indicating that there is a spoken voice is not input during the period when the video memory unit 15 is storing a video signal for 5 seconds, or if information indicating the end of the spoken voice is input before 5 seconds have elapsed when such information is input, the video memory unit 15 stops storing the video signal within 5 seconds.

[0019] When the video memory unit 15 stops storing the video signal for 5 seconds or more, it notifies the content playback control unit 20 that the storage of the video signal has stopped. Upon receiving the notification from the video memory unit 15, the content playback control unit 20 stops supplying the video signal to the display unit 21 and the audio signal to the speaker 22. As a result, the video of the content displayed on the display unit 21 pauses, and the audio of the content is not output from the speaker 22.

[0020] As described above, if there is a spoken voice when the video memory unit 15 has stored a video signal for 5 seconds, the video is displayed on the display unit 21 and the audio is output from the speaker 22 until the spoken voice ends. Therefore, it is possible to avoid the unpleasant situation where the playback of the content is interrupted in the middle of the spoken voice.

[0021] When the video memory unit 15 stops storing the video signal, the video analysis and text data generation unit 16 analyzes the stored video and generates descriptive text data that explains the scenes within the video memory period. As shown in Figure 2 as an example, suppose that within the video memory period, the scene switches from Sc1, in which a man and a woman are talking in a living room, to Sc2, in which a car is driving on a road. In this case, the video analysis and text data generation unit 16 generates, for example, descriptive text data such as, "The scene has switched from in which a man and a woman are facing each other in a living room to in which a car is driving on a road," or descriptive text data describing the scene after the switch, such as, "A car is driving on a road in a residential area."

[0022] In recent years, artificial intelligence (AI) has developed rapidly, making it easy to analyze video and generate descriptive text data to explain scenes. In particular, by equipping the video analysis and text data generation unit 16 with generation AI, the video analysis and text data generation unit 16 can easily and accurately generate descriptive text data to explain scenes. The video analysis and text data generation unit 16 supplies the descriptive text data to the audio data conversion unit 17.

[0023] The audio data conversion unit 17 converts the input explanatory text data into audio data. The audio data conversion unit 17 can convert the explanatory text data into audio data using a known audio conversion technology called Text to Speech. The audio data conversion unit 17 converts the explanatory text data into audio data and outputs it as explanatory audio data.

[0024] The video analysis / text data generation unit 16 and the audio data conversion unit 17 function as an audio data generation unit that, when the user's gaze is away from the screen for a predetermined period of time or longer, generates explanatory audio data describing the scenes within that period based on the analysis results obtained from analyzing the video content during the period when the user's gaze was away from the screen.

[0025] Instead of the video analysis and text data generation unit 16 analyzing the video using a generation AI, the content playback device 100 may be configured as follows: The content playback device 100 transmits the video signal stored in the video storage unit 15 to a video analysis server that analyzes the video using a generation AI via the network 30. The content playback device 100 receives the descriptive text data describing the scene, which has been generated by the video analysis server after analyzing the video, and supplies it to the audio data conversion unit 17. In this case, the audio data conversion unit 17 functions as an audio data generation unit that generates descriptive audio data.

[0026] The audio data conversion unit 17 supplies explanatory audio data to the content playback control unit 20. The content playback control unit 20 supplies an analog audio signal based on the input explanatory audio data to the speaker 22. As a result, the speaker 22 outputs an explanatory voice message saying, "The scene has switched from a scene of a man and a woman facing each other in a living room to a scene of a car driving on a road."

[0027] The content playback method according to the first embodiment, executed by the content playback device 100, will be explained using the flowchart shown in Figure 3. In Figure 3, the content playback control unit 20 plays the content in step S1. In step S2, the gaze detection unit 14 determines whether the user's gaze is outside the screen of the display unit 21 based on the user's captured image supplied from the camera 13. If the gaze is not outside the screen of the display unit 21 (NO), steps S1 and S2 are repeated.

[0028] If the gaze is away from the screen of the display unit 21 in step S2 (YES), the video storage unit 15 starts storing the video signal in step S3. In step S4, the gaze detection unit 14 determines whether the state of the gaze being away from the screen of the display unit 21 has continued for a predetermined time. If the state of the gaze being away from the screen of the display unit 21 has not continued for a predetermined time (NO), the video storage unit 15 stops storing the video signal in step S5, and the process returns to step S1.

[0029] If, in step S4, the state in which the gaze is away from the screen of the display unit 21 continues for a predetermined time (YES), the video storage unit 15 determines in step S6, based on the information supplied by the sound detection unit 18, whether there is no spoken sound within the predetermined time or whether the spoken sound ended before the predetermined time has elapsed. If there is no spoken sound within the predetermined time or the spoken sound ended before the predetermined time has elapsed (YES), the video storage unit 15 stops storing the video signal after a predetermined time in step S7.

[0030] If step S6 does not indicate that there is no spoken audio within the predetermined time or that the spoken audio has ended before the predetermined time has elapsed (NO), then it means that there is spoken audio within the predetermined time and the spoken audio has not ended when the predetermined time has elapsed. In this case, in step S8, the video storage unit 15 determines whether or not the spoken audio has ended based on the information supplied by the audio detection unit 18. If the spoken audio has not ended (NO), the video storage unit 15 repeats the process in step S8.

[0031] If the spoken audio has finished in step S8 (YES), the video storage unit 15 stops storing the video signal in step S9 at the time the spoken audio has finished. In this case, the video storage unit 15 stores a video signal that exceeds a predetermined time.

[0032] Following step S7 or S8, in step S10, the content playback control unit 20 stops playback of the content in response to the cessation of video signal storage in the video storage unit 15. In step S11, the video analysis and character data generation unit 16 analyzes the video signal stored in the video storage unit 15 and generates explanatory character data. In step S12, the audio data conversion unit 17 converts the explanatory character data into explanatory audio data. In step S13, the content playback control unit 20 plays the explanatory audio.

[0033] In step S14, the content playback device 100 determines whether or not it has been instructed to stop content playback. If it has not been instructed to stop content playback (NO), the content playback device 100 repeats the process in steps S1 to S14. If it has been instructed to stop content playback (YES), the content playback device 100 terminates the process.

[0034] An example of the operation of the content playback device 100 will be further explained using the timing chart shown in Figure 4. In Figure 4, (a) shows the playback of an audio signal by the content playback control unit 20, (b) shows the playback of a video signal by the content playback control unit 20, and (c) shows the gaze detection result by the gaze detection unit 14. In the gaze detection result, "1" indicates that the user's gaze is directed towards the screen of the display unit 21, and "0" indicates that the user's gaze is not directed towards the screen of the display unit 21.

[0035] In Figure 4, (d) shows the state of video signal storage in the video storage unit 15, (e) shows the explanatory text data generated by the video analysis / text data generation unit 16, and (f) shows the explanatory audio data converted by the audio data conversion unit 17.

[0036] As shown in Figure 4(a), the content includes discretely located spoken audio Sa1 to Sa4. As shown in Figure 4(c), assume that at time t1 the user's gaze leaves the screen, and at time t2, less than 5 seconds later the user's gaze returns to the screen. The time indicated by lowercase t is the time in real time. As shown in Figure 4(d), the video storage unit 15 stores the video signal Rv0. The video signal Rv0 is less than 5 seconds long, so the content playback control unit 20 does not stop playback of the content, and the video analysis / character data generation unit 16 does not analyze the video signal Rv0.

[0037] If the user's gaze is diverted from the screen for a short period of time (less than 5 seconds), the video analysis / text data generation unit 16 will not analyze the video signal stored in the video storage unit 15, and the audio data conversion unit 17 will not generate explanatory audio data. In this way, when the user briefly or for a very short time takes their eyes off the screen, content playback will not stop and unnecessary explanatory audio will not be played.

[0038] As shown in Figure 4(c), assume that at time t3 the user's gaze is taken away from the screen, and 5 seconds have passed at time t4. As shown in Figure 4(d), the video memory unit 15 starts storing the video signal Rv1 at time t3 (or immediately thereafter). As shown in Figure 4(a), the spoken audio Sa3 has not finished at time t4. Therefore, the video memory unit 15 continues to store the video signal Rv1 after time t4, and stops storing the video signal Rv1 at time t5 (or immediately thereafter) when the spoken audio Sa3 has finished.

[0039] As shown in Figure 4(c), upon receiving notification that the video signal Rv1 has stopped being stored in the video storage unit 15, the content playback control unit 20 stops playback of the content. As shown in Figure 4(e), the video analysis / character data generation unit 16 analyzes the video signal Rv1 and generates explanatory character data Dc1. As shown in Figure 4(f), the audio data conversion unit 17 converts the explanatory character data Dc1 into explanatory audio data Da1. As shown in Figure 4(a), at time t6 during the period when content playback is stopped, the content playback control unit 20 plays explanatory audio Ea1 based on the explanatory audio data Da1.

[0040] As shown in Figure 4(c), assume the user returns their gaze to the screen at time t7. The content playback control unit 20 resumes content playback at time t7 (or immediately thereafter).

[0041] As described above, with the content playback device 100 and content playback method according to the first embodiment, the content can be understood by listening to the explanatory audio during the period when the user's gaze is diverted, so there is no need to rewind the playback position and watch the content again. Although content playback temporarily stops if the user's gaze is diverted from the screen for a predetermined period of time or longer, time loss can be minimized as much as possible. Furthermore, even if the user looks away from the screen, the content can be understood through the explanatory audio, making it possible to watch the content efficiently.

[0042] Figure 5 shows a content playback device 100', which is an example of a configuration developed from the content playback device 100 according to the first embodiment. In Figure 5, the same reference numerals are used for parts that are the same as in Figure 1, and their descriptions are omitted. The content playback device 100' includes a content attribute acquisition unit 23 and an audio analysis unit 24, which are not included in the content playback device 100.

[0043] The content attribute acquisition unit 23 acquires the attributes of the content received by the communication unit 10. The content attributes indicate the type of content, such as movies, dramas, interview content, or news. The video analysis and text data generation unit 16 may make the explanatory text data more detailed for content where missing the video would likely hinder understanding of the content, such as movies or dramas. The video analysis and text data generation unit 16 may make the explanatory text data more concise for content where missing the video would not significantly hinder understanding of the content, such as interview content or news.

[0044] Thus, the video analysis and text data generation unit 16 may vary the amount of information in the explanatory text data describing the scenes within the video storage period, depending on the type of content. By varying the amount of information in the explanatory text data according to the type of content, the content playback control unit 20 can play back explanatory audio with the optimal amount of information according to the type of content.

[0045] The speech analysis unit 24 analyzes the frequency of the spoken voice. The speech analysis unit 24 may also analyze the frequency of the spoken voice and determine whether the speaker is male or female. The speech analysis unit 24 supplies the frequency analysis result or the determination result of whether the speaker is male or female to the speech data conversion unit 17. The speech data conversion unit 17 may convert the explanatory text data into explanatory voice data with a frequency that is different from the frequency of the spoken voice.

[0046] The voice data conversion unit 17 may convert the explanatory text data into explanatory voice data in a female voice when the utterance is male only, and convert the explanatory text data into explanatory voice data in a male voice when the utterance is female only.

[0047] In the content playback device 100 shown in Figure 1 or the content playback device 100' shown in Figure 5, the gaze detection unit 14 supplies a binary determination result to the video storage unit 15 indicating whether or not the user's gaze is directed towards the screen, as shown in Figure 4(c). The gaze detection unit 14 may also determine the degree to which the user's gaze is away from the screen and supply the determination result to the video analysis / text data generation unit 16. The video analysis / text data generation unit 16 may generate explanatory text data corresponding to the degree to which the gaze is away from the screen.

[0048] For example, the video analysis and text data generation unit 16 may generate explanatory text data that describes the scenes within the video memory period in relatively detail if the degree to which the gaze is away from the screen is large, and generate explanatory text data that describes the scenes relatively concisely if the degree is small. In the latter case, if the scene switches from scene Sc1 to scene Sc2 as shown in Figure 2, it may simply generate concise explanatory text data that says, "The scene has switched."

[0049] In the content playback device 100 or 100', when the content playback control unit 20 plays explanatory audio, it may superimpose additional sounds that enable the user to recognize it as explanatory audio. Alternatively, the content playback control unit 20 may generate additional sounds before playing the explanatory audio so that the user can recognize it as explanatory audio.

[0050] <Second Embodiment> Figure 6 shows a content playback device 200 according to the second embodiment. In Figure 6, the same reference numerals are used for parts that are the same as in Figure 1, and their descriptions may be omitted. The content playback device 200 includes a gap section detection unit 28 instead of an audio detection unit 18. The video buffer 11 and audio buffer 12 of the content playback device 200 buffer the video signal and audio signal for a longer period of time than the video buffer 11 and audio buffer 12 of the content playback device 100 buffer the video signal and audio signal.

[0051] The gap detection unit 28 detects gaps in spoken audio based on the audio signal buffered in the audio buffer 12. The gap detection unit 28 supplies the content playback control unit 20 with the detection results of the gaps in spoken audio, i.e., the end time of each spoken audio and the start time of the next spoken audio that follows each spoken audio. Here, the end time and start time refer to times within the content that are relative to a predetermined timing in the content. The predetermined timing is, for example, the start time of the content.

[0052] In the content playback device 100, the video storage unit 15 notifies the content playback control unit 20 that it has stopped storing the video signal in order to temporarily pause content playback. In the content playback device 200, content playback is not stopped, and even if the video storage unit 15 stops storing the video signal, it does not notify the content playback control unit 20 that it has stopped.

[0053] The content playback method according to the second embodiment, performed by the content playback device 200, will be explained using the flowcharts shown in Figures 7 and 8. In Figures 7 and 8, the same reference numerals are used for steps that are substantially the same as those in Figure 3. Figure 8 shows the main processes in the content playback method according to the second embodiment. Figure 7 shows some of the processes in the content playback method according to the second embodiment, which are auxiliary processes necessary to perform the processes shown in Figure 8. The processes shown in Figure 7 are performed in parallel with the processes shown in Figure 8.

[0054] In Figure 7, the video buffer 11 and audio buffer 12 buffer the content in step S0. The gap detection unit 28 detects gaps in the spoken audio in step S25. In step S26, the gap detection unit 28 supplies information indicating the gaps to the content playback control unit 20.

[0055] In step S14, the content playback device 200 determines whether or not it has been instructed to stop content playback. If it has not been instructed to stop content playback (NO), the content playback device 200 repeats the processes in steps S0, S25, S26, and S14. If it has been instructed to stop content playback (YES), the content playback device 200 terminates the process.

[0056] In Figure 8, the content playback device 200 plays content in step S1 following step S0. In step S2, the gaze detection unit 14 determines whether the user's gaze is away from the screen of the display unit 21 based on the user's captured image supplied by the camera 13. If the gaze is not away from the screen of the display unit 21 (NO), the process in step S2 is repeated.

[0057] If the gaze is away from the screen of the display unit 21 in step S2 (YES), the video storage unit 15 starts storing the video signal in step S3. In step S4, the gaze detection unit 14 determines whether the state of the gaze being away from the screen of the display unit 21 has continued for a predetermined time. In the second embodiment as well, the predetermined time is, for example, 5 seconds. If the state of the gaze being away from the screen of the display unit 21 has not continued for the predetermined time (NO), the video storage unit 15 stops storing the video signal in step S5, and the process returns to step S2.

[0058] If the gaze remains away from the screen of the display unit 21 for a predetermined period of time in step S4 (YES), the video analysis / character data generation unit 16 analyzes the video signal of the stored video for the predetermined period of time stored in the video storage unit 15 in step S21 and generates explanatory character data. The audio data conversion unit 17 converts the explanatory character data into explanatory audio data in step S22. The content playback control unit 20 plays the explanatory audio in the gaps of the spoken audio in step S23, based on the gap detection unit 28 supplying information indicating gaps.

[0059] Furthermore, if the state in step S4 where the user's gaze is away from the screen of the display unit 21 continues for a predetermined time (YES), the gaze detection unit 14, in parallel with steps S21 and onward, determines in step S24 whether or not the user's gaze is still away from the screen. If the state where the user's gaze is away from the screen does not continue (NO), the process from step S2 onward is repeated. If the state where the user's gaze is away from the screen continues (YES), the process from step S4 onward is repeated.

[0060] In step S14, following step S23, the content playback device 200 determines whether or not it has been instructed to stop content playback. If it has not been instructed to stop content playback (NO), the content playback device 200 repeats the process from steps S0 to S14. If it has been instructed to stop content playback (YES), the content playback device 200 terminates the process.

[0061] An example of the operation of the content playback device 200 will be further explained using the timing chart shown in Figure 9. In Figure 9, (a) shows the audio signal buffered in the audio buffer 12, and (b) shows the video signal buffered in the video buffer 11. The gap section detection unit 28 represents the time indicating the timing within the content with a capital T, detects the end time T1 of the spoken audio Sa11 and the start time T2 of the spoken audio Sa12, and detects the time from the end time T1 to the start time T2 as a gap section.

[0062] In Figure 9, (c) shows the playback of an audio signal by the content playback control unit 20, and (d) shows the playback of a video signal by the content playback control unit 20. The content playback control unit 20 plays back the video signal and audio signal that have been buffered for a predetermined time in the video buffer 11 and audio buffer 12, so the video signal and audio signal are played back with a predetermined delay. (e) shows the gaze detection result by the gaze detection unit 14, (f) shows the state of video signal storage in the video storage unit 15, (g) shows the explanatory character data generated by the video analysis / character data generation unit 16, and (h) shows the explanatory audio data converted by the audio data conversion unit 17.

[0063] As shown in Figure 9(e), suppose the user's gaze is away from the screen at time t11, and returns to the screen at time t14, more than 10 seconds later. As shown in Figure 9(f), the video storage unit 15 stores the video signal Rv11 for 5 seconds from time t11 to time t12. Since the state of the user's gaze being away from the screen continues after time t12, the video storage unit 15 stores the video signal Rv12 for 5 seconds from time t12 to time t13.

[0064] Since the user's gaze remains away from the screen even after time t13, the video memory unit 15 continues to store the video signal Rv13. However, at time t14, before 5 seconds have elapsed, the user's gaze returns to the screen, so the video memory unit 15 stops storing the video signal.

[0065] The video storage unit 15 only needs to have the capacity to store multiple groups of video signals, with each 5-second video signal being treated as a single group of video signals. Once the video analysis / character data generation unit 16 analyzes the video of a group of video signals stored in the video storage unit 15 and generates explanatory character data, that group of video signals becomes unnecessary and can be overwritten by a later group of video signals.

[0066] As shown in Figure 9(g), when the video signal Rv11 is stored in the video storage unit 15, the video analysis / character data generation unit 16 analyzes the video signal Rv11 and generates explanatory character data Dc11. When the video signal Rv12 is stored in the video storage unit 15, the video analysis / character data generation unit 16 analyzes the video signal Rv12 and generates explanatory character data Dc12. Since the video signal Rv13 is less than 5 seconds long, the video analysis / character data generation unit 16 does not analyze the video signal Rv13. As shown in Figure 9(h), the audio data conversion unit 17 converts the explanatory character data Dc11 and Dc12 into explanatory audio data Da11 and Da12, respectively.

[0067] As shown in Figure 9(c), the content playback control unit 20 continuously plays explanatory audio Ea11 based on explanatory audio data Da11 and explanatory audio Ea12 based on explanatory audio data Da12 in the gap between the end time T1 of spoken audio Sa11 and the start time T2 of spoken audio Sa12 in the content being played. Continuous playback means that explanatory audio Ea11 is played first, followed by explanatory audio Ea12, and a time interval may be provided between explanatory audio Ea11 and explanatory audio Ea12. The content playback control unit 20 plays explanatory audio Ea11 at time t15 and explanatory audio Ea12 at time t16. As a result, explanatory audio Ea11 and explanatory audio Ea12 are continuously output from the speaker 22 in a single gap.

[0068] The content playback control unit 20 should play each explanatory audio data, such as the explanatory audio data Da11 and Da12, generated by the audio data conversion unit 17, in the nearest gap section that has enough time to play each explanatory audio data after the time each explanatory audio data was generated.

[0069] Suppose the gap between the end time T1 and the start time T2 does not allow for continuous playback of both explanatory audio Ea11 and explanatory audio Ea12. In this case, the content playback control unit 20 should play only explanatory audio Ea11 in the gap between the end time T1 and the start time T2, and play explanatory audio Ea12 in the gap after the end time of spoken audio Sa12. As a result, explanatory audio Ea11 and explanatory audio Ea12 are distributed and output from the speaker 22 to the two gaps. If there are three or more explanatory audios, they may be distributed and output from the speaker 22 to two or more gaps.

[0070] The content playback control unit 20 may choose not to play explanatory audio Ea12 if explanatory audio Ea11 and explanatory audio Ea12 are the same. The video analysis / text data generation unit 16 may achieve non-playback of explanatory audio Ea12 by not supplying explanatory text data Dc12 to the audio data conversion unit 17 when explanatory text data Dc11 and explanatory text data Dc12 are the same data. The audio data conversion unit 17 may achieve non-playback of explanatory audio Ea12 by not supplying explanatory audio data Da12 to the content playback control unit 20 when explanatory audio data Da11 and explanatory audio data Da12 are the same data. The content playback control unit 20 may determine that explanatory audio Ea11 and explanatory audio Ea12 are the same and achieve non-playback of explanatory audio Ea12.

[0071] As described above, with the content playback device 200 and content playback method according to the second embodiment, the content can be understood by listening to the explanatory audio during the period when the user's gaze is diverted, so there is no need to rewind the playback position and watch the content again. Since content playback does not stop even when the user's gaze is diverted from the screen, time loss can be reduced. In addition, even if the user looks away from the screen, the content can be understood through the explanatory audio, allowing for efficient viewing of the content.

[0072] The content playback device 200 may also be equipped with a content attribute acquisition unit 23 or an audio analysis unit 24, similar to the content playback device 100' shown in Figure 5. The effects of the content playback device 200 being equipped with a content attribute acquisition unit 23 are the same as those of the content playback device 100'. The effects of the content playback device 200 being equipped with an audio analysis unit 24 are the same as those of the content playback device 100'.

[0073] In addition, in the content playback device 200, the gaze detection unit 14 may determine the extent to which the user's gaze is away from the screen and supply the determination result to the video analysis / text data generation unit 16. The video analysis / text data generation unit 16 may generate explanatory text data corresponding to the extent to which the user's gaze is away from the screen.

[0074] Furthermore, in the content playback device 200, the content playback control unit 20 may superimpose additional sounds onto the explanatory audio, or it may generate additional sounds before playing the explanatory audio, so that the user can recognize that it is explanatory audio.

[0075] The present invention is not limited to the first or second embodiments described above, and can be modified in various ways without departing from the spirit of the invention. The content playback devices 100, 100', and 200 may be used as in-vehicle devices. [Explanation of Symbols]

[0076] 10 Communications Department 11. Video buffer 12 Audio buffer 13 Cameras 14 Eye-line detection unit 15. Video Memory Unit 16. Video Analysis and Text Data Generation Unit 17. Audio Data Conversion Unit 18. Voice detection unit 20 Content Playback Control Unit 21 Display section 22 speakers 23 Content attribute acquisition unit 24. Voice Analysis Unit 28 Gap detection unit 30 Networks 50 Content Provider Servers 100,100',200 Content Playback Device

Claims

1. A content playback control unit that displays the video of the content on a display unit and plays the content so that the audio of the content is output from a speaker, A gap detection unit for detecting gaps in spoken audio contained in the aforementioned content, A gaze detection unit detects the gaze of a user looking at the screen of the display unit and determines whether or not the user's gaze is directed away from the screen. When the state in which the gaze is away from the screen continues for a predetermined period of time or longer, an audio data generation unit generates explanatory audio data describing the scenes within that period based on the analysis results obtained by analyzing the video of the content during the period in which the gaze is away from the screen, Equipped with, When the content playback control unit determines that the state in which the viewer's gaze is away from the screen continues for a predetermined period of time or longer and the explanatory audio data is generated, the content playback control unit causes the speaker to output explanatory audio based on the explanatory audio data to the gap detected by the gap detection unit in the content being played. Content playback device.

2. The aforementioned audio data generation unit, A video analysis and text data generation unit analyzes the video of the aforementioned content and generates explanatory text data that describes the scenes within the aforementioned period. A voice data conversion unit that converts the aforementioned explanatory text data into the aforementioned explanatory audio data, has The content playback device according to claim 1.

3. The audio data generation unit generates the explanatory audio data for each predetermined time period during which the gaze is away from the screen, whenever the state of the gaze being away from the screen continues for a predetermined time. The content playback control unit causes the speaker to output explanatory audio based on the explanatory audio data at predetermined intervals during the gap section. The content playback device according to claim 1 or 2.

4. When the content playback control unit cannot continuously output the explanatory audio based on the explanatory audio data at predetermined time intervals to one of the gap intervals detected by the gap interval detection unit, it distributes and outputs the explanatory audio based on the explanatory audio data at predetermined time intervals to two or more gap intervals detected by the gap interval detection unit. The content playback device according to claim 3.

5. The content is played back so that the video of the content is displayed on the display unit and the audio of the content is output from the speaker. The gaps between spoken audio segments included in the aforementioned content are detected. The system detects the user's gaze as they view the screen of the display unit and determines whether the user's gaze is directed away from the screen. When the state in which the gaze is away from the screen continues for a predetermined period of time or longer, based on the analysis results obtained by analyzing the video of the content during the period in which the gaze is away from the screen, explanatory audio data describing the scenes during that period is generated. When the state in which the gaze is away from the screen continues for a predetermined period of time or longer, and the explanatory audio data is generated, the speaker is made to output explanatory audio based on the explanatory audio data during the gap in the content being played. How to play content.

Images