Content presentation device, program, and content presentation system

Abstract

To present visual information useful for content understanding of a video content when a viewer under viewing while doing something gazes at a screen.SOLUTION: A content presentation device presents a video content distributed from a content server and comprises: a sensing section for detecting a viewing state of a viewer with respect to the video content; a reproduction control section for switching between a first reproduction mode in which a video is reproduced and a second reproduction mode in which visual information relating to a content of the video is reproduced in accordance with the viewing state of the viewer; an audio / video reproduction section for reproducing an audio stream and a video stream in the first reproduction mode; and a representative image reproduction section for reproducing the visual information while reproducing the audio stream in the second reproduction mode.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a content presentation device, a program, and a content presentation system. [Background technology]

[0002] When a viewer watches video content while simultaneously performing other actions (such as washing the face or washing the dishes, hereafter referred to as "concurrent actions"), this is called "multiple viewing." The following two points can be mentioned as characteristics of multiple viewing: Intermittent viewing: Viewers watch the video for short periods of time while engaged in other activities, getting a rough idea of ​​the content based on audio information alone, without looking at the video, and then watch the video for short periods of time when they have the time to focus their attention on the screen. Switching between concentrated viewing and non-concentrated viewing: When a viewer's interest in a video content exceeds a certain level, he / she may stop other concurrent activities and switch to a state of concentrating on watching the video content (hereafter referred to as concentrated viewing). Also, he / she may switch from concentrated viewing to non-concentrated viewing.

[0003] There is known a content presentation device that presents video content to a viewer who is concurrently performing other activities (see, for example, Non-Patent Document 1). Non-Patent Document 1 discloses a system that supports viewing while concurrently performing other activities. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Nobumu Ogawa, Kinji Matsumura, "The impact of viewing while doing other things on news exposure," Research Report Human-Computer Interaction (HCI), Jun. 2022, vol. 2022-HCI-198, pp.1-8. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has a problem that visual information useful for understanding the video content may not be displayed when the viewer gazes at the screen. In other words, a viewer who is watching something while doing other things gazes at the screen at any time, but the conventional technology does not have a mechanism for presenting appropriate visual information to such a viewer. When the viewer gazes at the screen, the information he or she wanted to see is often replaced by other information, which is an obstacle to the viewer's understanding of the video content.

[0006] In consideration of the above-mentioned problems, the present disclosure provides a technology that presents visual information useful for understanding the content of video content when a viewer who is watching while doing other things looks at the screen. [Means for solving the problem]

[0007] The present disclosure relates to a content presentation device that presents video content distributed from a content server, characterized in having a sensing unit that detects a viewer's viewing status of the video content, a playback control unit that switches between a first playback mode in which a video is played and a second playback mode in which visual information related to the content of the video is played depending on the viewer's viewing status, an audio / video playback unit that plays an audio stream and a video stream in the first playback mode, and a representative image playback unit that plays the visual information while playing the audio stream in the second playback mode. Effect of the Invention

[0008] The present disclosure can present visual information useful for understanding the content of video content when a viewer is watching something else and focuses their attention on the screen. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of concurrent behavior in multitasking. [Diagram 2] FIG. 13 is a diagram showing an example of timing at which a viewer watches video content while engaging in a parallel action. [Diagram 3]FIG. 1 is a diagram illustrating an outline of a content presentation device. [Figure 4] FIG. 1 is a diagram illustrating a system configuration of an example of a content presentation system. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a content server and a content presentation device. [Figure 6] FIG. 2 shows an example of a functional block diagram illustrating functions of a content server, a content presentation device, and a sensor in blocks. [Figure 7] FIG. 2 is a diagram illustrating a unit section and a representative image. [Figure 8] FIG. 13 is a diagram showing an example of a metatable for displaying a representative image. [Figure 9] 11A and 11B are diagrams illustrating a method for detecting a gaze direction. [Figure 10] 11A and 11B are diagrams illustrating a normal playback mode, a representative image playback mode, and a method of switching between the two modes. [Figure 11] 11 is a flowchart illustrating an example of a process in which the content presentation device switches between a normal playback mode and a representative image playback mode. [Figure 12] 11 is a flowchart illustrating an example of a process in which the content presentation device switches between a normal playback mode and a representative image playback mode. [Figure 13] 11 is a flowchart illustrating a process performed by a representative image playback unit in a representative image playback mode. FIG. [Figure 14] 11A and 11B are diagrams illustrating switching from a normal playback mode to a representative image playback mode. [Figure 15] 11 is a flowchart illustrating an example of a process in which the content presentation device switches between a normal playback mode and a representative image playback mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A content presentation device and an information presentation method performed by the content presentation device will be described below with reference to the drawings as an example of an embodiment of the present invention.

[0011] <An example of parallel actions> FIG. 1 shows an example of concurrent behavior in multitasking. In FIG. 1(a), a content presentation device displays video content on a display 101 (smart table) in a table. A viewer 9 watches the video content displayed on the display 101 while reading a book. In FIG. 1(b), a content presentation device displays video content on a display 102 (smart mirror) embedded in a mirror. A viewer 9 watches the video content displayed on the display 102 while adjusting his / her clothes and hairstyle. In FIG. 1(c), a content presentation device controls a projector to project an image 103 on a wall. The projector is a smart projector that is placed on a ceiling or the like and projects an image on the wall where the viewer is located. A viewer 9 watches the image 103 projected by the projector while choosing clothes.

[0012] <An example of when viewers might look at the screen while engaged in parallel activities> FIG. 2 shows an example of the timing at which a viewer gazes at video content during concurrent activities. In FIG. 2(a), a viewer gazes at the video content after noticing a voice 105 saying, "This is nationwide pollen information." The viewer wanted to check the pollen dispersion situation in Tokyo on a pollen forecast chart 106. However, as shown in FIG. 2(b), when the viewer gazed at the video content, the scene of the video content switched to a studio scene 107, and the viewer 9 missed the pollen forecast chart in the weather forecast (visual information useful for understanding the content). Thus, conventionally, there were many cases where visual information useful for understanding the content of the video content was not displayed when the viewer 9 gazed at the video content during concurrent activities.

[0013] <Outline of the operation of the content presentation device> Therefore, the content presentation device of the present embodiment presents appropriate visual information (helpful in understanding the details of the video content) at the timing when the viewer gazes at the video content during a parallel action.

[0014] Fig. 3 is a diagram for explaining an outline of the content presentation device of this embodiment. In Fig. 3, a display 506 and a sensor 20 are installed in a kitchen. Video content distributed from a content server described later is displayed on the display 506. The sensor 20 is, for example, one or more of a camera, a speaker, and a touch panel, and is used by the content presentation device to detect the viewing situation of the viewer 9. The viewing situation is detected by the face image, voice, finger operation, etc. of the viewer 9.

[0015] The content presentation device of this embodiment plays back video content in two playback modes according to the viewing state of the viewer 9. Normal playback mode 111 (an example of the first playback mode) The normal playback mode is a playback mode in which the video content transmitted by the content server is played back as is. Representative image playback mode 112 (an example of the second playback mode) The representative image playback mode is a playback mode in which the audio stream of the video content is played back, and the representative image 113 (still image) is switched and displayed for each unit section. The representative image 113 may be played back by pausing the video content, or by playing a representative image prepared in advance.

[0016] According to the content presentation device of the present embodiment, during intensive viewing, the video content is played back in normal playback mode, and during multitasking viewing, the audio stream and the representative image are played back in representative image playback mode. Therefore, when the viewer 9 gazes at the video content while concurrently acting, the representative image is displayed, so that visual information useful for understanding the content of the video content can be presented at the timing when the viewer gazes at the video content while concurrently acting. Also, when the viewer 9 views the representative image for a certain period of time, it is determined that the viewer is viewing attentively, and the content presentation device switches to normal playback mode. In this way, the viewer can understand the content even while concurrently watching, and the video content can be played back by automatically switching to normal playback mode. By appropriately switching the playback mode at the timing when the viewer 9 gazes at the video content while concurrently acting, visual information useful for understanding the content of the video content can be presented.

[0017] <Terminology> Video content refers to information conveyed by moving image data and audio data. Moving image data is sometimes called a video stream, and audio data is sometimes called an audio stream.

[0018] The viewing status is a status related to whether or not the viewer is directing their gaze to the video content, and if so, how long they have been directing their gaze, or if not, how long they have not been directing their gaze. In this embodiment, the viewing status is determined by the gaze, and also by the viewer's operation to determine whether or not the viewer is directing their gaze to the video content.

[0019] Metadata is data for explaining the main data. In this embodiment, metadata is used to explain the video content. In this embodiment, the metadata is explained by the term representative image display meta table.

[0020] The visual information is information that can be visually grasped regarding the contents of the video and is provided for each unit section. The visual information may be any information that is useful for understanding the contents of the video content.

[0021] Multitasking refers to a situation where a viewer watches video content while doing something else.

[0022] <System configuration example> A system configuration of the content presentation system 100 according to the present embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram showing a system configuration of an example of the content presentation system 100. The content presentation system 100 includes a content server 10, a content presentation device 30, and a sensor 20. The content server 10 and the content presentation device 30 are communicatively connected via a wide-area network N1 such as the Internet. Furthermore, the content presentation device 30 is connected to a network N2. The network N2 is a LAN in a home or facility where the content presentation device 30 exists, or a communication network of a mobile phone network such as 3G, 4G, 5G, or 6G. The content presentation device 30 is communicatively connected to the content server 10 via the network N2 and the network N1.

[0023] The content server 10 is realized by one or more information processing devices. The content server 10 provides a video distribution service that distributes content to the content presentation device 30 via the network N1. The video distribution service is a service that mainly uses the Internet to provide movies, TV programs (animations, dramas), self-produced videos, anime, sports, etc. at the time the viewer 9 wants to watch them. The video distribution service may be a service that distributes video content live, or may be a VOD (Video On Demand) service that distributes pre-stored video content in response to a request from the viewer 9.

[0024] The content server 10 may be realized by cloud computing or by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The content server 10 may exist on the Internet or on-premise. An example of an on-premise location is when the content server 10 is a home server or a television recording device.

[0025] The content presentation device 30 is, for example, an information processing device used for viewing content by the viewer 9. The content presentation device 30 may be located indoors and connected to Wi-Fi (registered trademark) or a wired LAN, or may be located outdoors.

[0026] The content presentation device 30 may be a smart table, a smart mirror, a projector, etc., as shown in Fig. 1, or may be, for example, a tablet terminal, a display-integrated PC, a notebook PC, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), etc. The content presentation device 30 may be anything that can receive and play back content, and may run a web browser or a native application for playback.

[0027] Furthermore, the content presentation device 30 may be a device in which the main body and the display are separated, such as a desktop PC or a set-top box, etc. Furthermore, the content presentation device 30 may be a projector that does not have a display.

[0028] The content presentation device 30 may be automatically connected to the content server 10 upon startup, or the viewer may connect the content presentation device 30 to the content server 10 through manual operation or voice operation.

[0029] A sensor 20 is connected to the content presentation device 30 via a network N2 or a dedicated line such as a USB cable. The sensor 20 is, for example, one or more of a Web camera 22, a microphone 21, and a touch panel. The Web camera 22 repeatedly captures an image of the viewer 9 and transmits image data showing the viewer 9 to the content presentation device 30. The content presentation device 30 analyzes the image data to determine the viewing situation. Therefore, the content presentation device 30 can automatically switch the playback mode without any operation by the viewer.

[0030] The microphone 21 acquires the audio data and transmits the audio data to the content presentation device 30. The content presentation device 30 recognizes the audio data and determines the viewing situation. The touch panel is integrated with the display 506 of the content presentation device 30 and receives operations from the viewer 9. A keyboard may be substituted for the touch panel. The display 506 displays various menu buttons. The touch panel transmits the touch coordinates touched by the viewer 9 to the content presentation device 30. The content presentation device 30 detects operation information (switching of playback mode) based on the touch coordinates and determines the viewing situation based on this. Therefore, the viewer can explicitly operate to switch the playback mode. For example, if the viewer 9 wishes to save on packet communication volume, the viewer 9 can set the representative image playback mode.

[0031] <Hardware configuration example> The hardware configuration of the content server 10 and the content presentation device 30 according to this embodiment will be described with reference to FIG.

[0032] <<Content Server 10 and Content Presentation Device 30>> Fig. 5 is a diagram showing an example of a hardware configuration of the content server 10 and the content presentation device 30 according to the present embodiment. As shown in Fig. 5, the content server 10 and the content presentation device 30 are constructed by a computer 500, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0033] Among these, the CPU 501 controls the operation of the entire computer 500. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 5.

[0034] Moreover, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be any of a CD, a DVD, and Blu-Ray (registered trademark). The media I / F 516 controls reading and writing (storing) of data from a storage medium 515 such as a flash memory.

[0035] <About the function> Next, a functional configuration of the content presentation system 100 will be described with reference to Fig. 6. Fig. 6 shows a functional block diagram illustrating the functions of the content server 10, the content presentation device 30, and the sensor 20 in blocks.

[0036] <<Sensor>> The sensor 20 has a sensing unit 25. The sensing unit 25 senses the viewing state of the viewer 9. Specifically, the sensing unit 25 generates RGB (color) image data captured by a Web camera and transmits the image data to the content presentation device 30. The sensing unit 25 also transmits audio data acquired by a microphone to the content presentation device 30. The sensing unit 25 also transmits touch coordinates on the display acquired by a touch panel to the content presentation device 30.

[0037] <<Content presentation device>> The content presentation device 30 has a Web application 32 and a native application 33 that run on a Web browser 31 .

[0038] First, the native application 33 has an image processing unit 34, a voice recognition unit 35, an operation detection unit 36, and a viewing status determination unit 37. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 5 executing instructions included in one or more programs installed in the content presentation device 30.

[0039] The image processing unit 34 processes the image data transmitted from the sensing unit 25. The image processing unit 34 detects the line of sight of the viewer, and judges whether or not the line of sight is directed toward the content presentation device 30. Details will be described later.

[0040] The voice recognition unit 35 performs voice recognition processing on the voice data sent from the sensing unit 25 and converts it into text data. Voice recognition may be performed by a voice recognition server on the network. The operation detection unit 36 ​​detects operation information based on the voice-recognized text data. For example, if the text data is a "still image," the operation information detects an operation to switch to representative image playback mode. If the text data is a "video," the operation information detects an operation to switch to normal playback mode.

[0041] Furthermore, the operation detection unit 36 ​​converts the touch coordinates transmitted from the sensing unit 25 into operation information. That is, since menu buttons are displayed on the playback screen of the video content, the operation detection unit 36 ​​determines which menu button was pressed based on the touch coordinates. The operation information is an operation to switch to the representative image playback mode or an operation to switch to the normal playback mode.

[0042] The viewing status determination unit 37 determines the viewing status of the viewer 9 based on the information on the line of sight direction determined by the image processing unit 34. The viewing status is whether the viewer is directing their gaze at the video content, and if so, how long they have been directing their gaze at the video content, or if not, how long they have not been directing their gaze at the video content. The viewing status determination unit 37 also determines the viewing status of the viewer 9 based on operation information. In the case of operation information for switching to the representative image playback mode, the viewing status is deemed to be that the viewer is not directing their gaze at the video content, and in the case of operation information for switching to the normal playback mode, the viewing status is deemed to be that the viewer is directing their gaze at the video content.

[0043] The Web application 32 has a representative image playback unit 41, an audio / video playback unit 43, a playback control unit 42, and a content acquisition unit 44. Each of these functional units is a function or means that is transmitted from the content server 10 as the Web application 32 and is realized by the CPU 501 shown in FIG. 5 executing an instruction included in the Web application.

[0044] The content acquisition unit 44 acquires video content from the content server 10 according to a video content communication protocol (mpeg-dash, HLS (HTTP Live Streaming), HDS (HTTP Dynamic Streaming), etc.). The video content to be acquired may be specified by the viewer 9 or may be automatically determined. The video content may be live-streamed or may be pre-recorded.

[0045] The audio / video playback unit 43 plays back the video content as a video under the control of the playback control unit 42. The audio / video playback unit 43 decodes the video content according to the codec of the video content (MPEG-4, H.264, H.265, AV1, etc.) and displays it on the display 506.

[0046] The representative image playback unit 41 plays back the representative image of the video content as a still image under the control of the playback control unit 42. The representative image is played back, for example, by pausing the video content. There is no need to store the representative image separately. Details of the representative image will be described later. However, the representative image may be stored separately. The representative image does not necessarily have to be the same as the video content, and a representative image with higher resolution than the video content may be prepared.

[0047] The playback control unit 42 controls whether the audio / video playback unit 43 performs playback in normal playback mode or in representative image playback mode, depending on the viewing situation determined by the viewing situation determination unit 37 .

[0048] <<Content Server>> The content server 10 is, for example, a Web server. In response to an HTTP request sent from the content presentation device 30, the content server 10 embeds video content in an HTTP response and delivers it. The content server 10 has an audio stream 11, a video stream 12, and a metatable 13 for displaying a representative image. The audio stream 11 and the video stream 12 are included in the video content. The audio stream 11 and the video stream 12 do not have to be separated from each other, and may be prepared as one video content. The metatable 13 for displaying a representative image will be described with reference to FIG. 8.

[0049] <Metatable for displaying representative images> First, a unit section and a representative image will be described in detail with reference to Fig. 7. A "representative image" is a still image included in a video stream of a certain section of video content (hereinafter referred to as a unit section) that is useful for understanding the contents of the unit section. For example, a "weather chart" or a "pollen dispersion forecast chart" in a weather forecast is a representative image.

[0050] The representative image and unit section are determined by the video editor according to the following rules. One representative image is always associated with one unit interval. A single video content can have any number of unit segments. A unit interval may overlap with other unit intervals.

[0051] 7 shows the section start points, section end points, and representative images of multiple unit sections 1 to 3. The representative image of unit section 1 is displayed from the section start point to the section end point of unit section 1. The representative image of unit section 2 is displayed from the section start point to the section end point of unit section 2. The representative image of unit section 3 is displayed from the section start point to the section end point of unit section 3. In other words, the representative image is the representative image of the section to which the playback time of the audio stream belongs.

[0052] Furthermore, there is a portion of overlap between unit interval 1 and unit interval 2. An overlapping section may be set when, for example, there is a still image in the latter half of unit interval 1 that is useful for understanding the content, and there is a still image in the first half of unit interval 2 that is useful for understanding the content, and the relationship between unit interval 1 and unit interval 2 is close. If overlapping sections are not allowed when there is a time period in which representative images are densely packed, one unit interval may become extremely short. In this case, the presentation time for the representative images will also be short.

[0053] The representative images of the overlapping sections are displayed with priority in the case where the section start point is earlier. The representative image 1 of the unit section 1 is displayed until the end point of the unit section 1, and when the end point of the unit section 1 arrives, the representative image 2 of the unit section 2 with the later section start point is displayed. However, the representative image that has not been viewed is prioritized over the representative image that has not been viewed. If the representative image 1 of the unit section 1 has been viewed and the representative image 2 of the unit section 2 has not been viewed, the representative image 1 of the unit section 1 is displayed until the start point of the unit section 2, and then the representative image 2 of the unit section 2 is displayed. This ensures the presentation time of the representative images even if the representative images are densely packed. If both the representative images 1 and 2 have been viewed, the one with the later section start point is displayed with priority. This is because if the one with the earlier section start point is displayed with priority, the viewer 9 will view the images in the order of image 1 → image 2, and then return to image 1 again, which is difficult to understand. Note that two representative images may be displayed simultaneously in the overlapping section.

[0054] On the other hand, there is a blank space between unit interval 2 and unit interval 3, and they are not continuous. Such blank spaces can occur when there is no suitable representative image. Since there is no representative image between unit interval 2 and unit interval 3, a blank message (for example, a time period in which no representative image is set) or the like may be displayed. Alternatively, the representative image of the immediately preceding unit interval (unit interval 2 in the case of FIG. 7) or the immediately succeeding unit interval (unit interval 3 in the case of FIG. 7) may continue to be played. It is preferable for the viewer 9 to be able to set whether a message or a representative image of the immediately preceding or immediately succeeding unit interval is to be presented during the blank time period.

[0055] In Fig. 7, news is given as an example of video content, but the video content may be any content other than news. For example, if the video content is a music program, the representative image may be a scene in which an artist or song is introduced. If the video content is sports, the representative image may be the scores of both teams, a scoring scene, spectators in the stadium, etc. If the video content is a drama, the representative image may be a scene in which many cast members meet, a close-up of the cast, a scene with a lot of action, etc.

[0056] 8 shows an example of the representative image display metatable 13. The representative image display metatable 13 is a table that defines the representative image and the timing for switching the representative image to be played back. The representative image playback metatable is acquired by the content acquisition unit 44 together with the video content from the content server 10. The playback control unit 42 uses the representative image playback metatable in the playback process in the representative image playback mode.

[0057] The meanings of the items in the representative image display meta table 13 are as follows: The "Index" field contains identification information (index) of the records that make up the representative image display meta table 13. The index increases as the interval start points increase, and may be, for example, a consecutive number. The start point of the unit interval that the representative image represents is set in the section start point item. The section end point item is set to the end point of the unit section that the representative image represents. The playback position item sets the position on the video stream where the representative image is played (playback time from the start of the video stream). The viewed item is set with a true / false value (TRUE: viewed, FALSE: not viewed) indicating whether the representative image has been viewed by the viewer. When the content acquisition unit 44 acquires the representative image playback meta table from the content server 10, the values ​​of all records are FALSE. The viewing judgment time item sets the time for the playback control unit 42 to judge whether or not each unit section has been viewed. The viewing judgment time corresponds to Nc (an example of a first threshold value) described later. By making the viewing judgment time different for each unit section, it is possible to set an appropriate viewing judgment time depending on the complexity, difficulty, etc. of the representative image.

[0058] The representative image display metatable 13 may be prepared separately according to the attributes (gender, age, place of residence, occupation, family structure, etc.) of the viewer 9. For example, a viewer 9 who represents the attributes may create the representative image display metatable 13. Therefore, different unit intervals and representative images are set in the representative image display metatable 13 according to the attributes of the viewer 9.

[0059] The content server 10 can also create the meta table 13 for displaying the representative image. The content server 10 is assumed to be capable of acquiring attributes of the viewer 9. The content server 10 acquires the gaze direction of each viewer 9 during distribution of video content from the content presentation device 30, and identifies a still image (time) on which the gaze stays for a certain period of time or more. The content server 10 creates a histogram with the playback time on the horizontal axis and the number of times the gaze stayed for a certain period of time or more on the height. If the attribute is gender, histograms are created for males and females. The content server 10 regards a peak above a certain height as a still image of the representative image. The unit section corresponding to this representative image is determined, for example, by dividing the unit section at the midpoint between the playback position of the representative image of the previous unit section and the playback position of the representative image of the next unit section.

[0060] By preparing the representative image display meta table 13 for each attribute, the unit section and the representative image are selected as follows, for example. For example, when the video content is a morning news program, if the viewer 9 is a woman, a scene of today's horoscope is presented as the representative image. When the viewer 9 is a man, a scene of market information is presented as the representative image. For example, when the video content is baseball, if the viewer 9 is a woman, a scene of a close-up of a player popular with women is presented as the representative image. When the viewer 9 is a man, a scene of a score is presented as the representative image. For example, when the video content is a drama, if the viewer 9 is a woman, a scene of a close-up of a male actor is presented as the representative image. When the viewer 9 is a man, a scene of a close-up of an actress is presented as the representative image. In this way, by preparing the representative image display meta table 13 for each attribute, a representative image that the viewer is interested in is presented, and a representative image that the viewer is not interested in is less likely to be presented. This is particularly effective for video content in which the representative image is expected to differ depending on the viewer's attributes.

[0061] <Detection of gaze direction> Next, FIG. 9 is a diagram for explaining a method for detecting the gaze direction. FIG. 9(a) shows a detection result of facial feature points 121. The facial feature points 121 are eyebrows, pupils, nose bridge, nose tip, nostrils, mouth corners, contours, and the like. The image processing unit 34 detects the face by performing pattern matching using HAAR-LIKE features and the like, searches for other feature points from features such as pupils, and detects the facial feature points 121 by further searching for other feature points. Alternatively, the image processing unit 34 may detect the facial feature points 121 using a facial feature detection model generated using learning data for machine learning in which the coordinates of the feature points 121 are annotated.

[0062] FIG. 9(b) shows an enlarged view of the detected pupils 122, 123. The upper row shows an RGB image, and the lower row shows a binary image. It is considered that the position of the Purkinje images 124, 125 relative to the pupils 122, 123 represents the gaze direction. The image processing unit 34 detects the gaze direction using a gaze direction detection model generated using learning data annotated with the gaze direction for the image in FIG. 9(b). The gaze direction may be either the coordinates of the viewpoint on the display 506 (2D gaze) or the gaze direction in three-dimensional space (3D gaze). Which gaze direction is obtained depends on the annotation.

[0063] Many methods have been proposed for detecting the gaze direction, and the method in Fig. 9 is merely one example. For example, the image processing unit 34 may detect the facial direction of the viewer 9 and use the facial direction as the gaze direction. Alternatively, the viewer 9 may wear glasses for detecting the gaze direction, and the glasses may transmit the gaze direction to the content presentation device 30.

[0064] <Normal playback mode and representative image playback mode> Next, the normal playback mode and the representative image playback mode will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining the normal playback mode and the representative image playback mode, and a method for switching between the two. In the normal playback mode, the audio / video playback unit 43 plays back the audio stream and the video stream 131 as a moving image. In the normal playback mode, the representative image playback unit 41 does not play back the representative image 132. In the representative image playback mode, the audio / video playback unit 43 plays back only the audio stream 133, and does not play back the video stream. In the representative image playback mode, the representative image playback unit 41 plays back the representative image 134.

[0065] In the normal playback mode, if the viewer 9 is not looking at the screen for Nn seconds, the playback control unit 42 determines that the viewer 9 is watching while doing something else, and switches to the representative image playback mode. In the representative image playback mode, if the viewer 9 is looking at the screen for Ns seconds, the playback control unit 42 determines that the viewer 9 is watching intently, and switches to the normal playback mode. Ns seconds is a sufficient time for the viewer 9 to understand the representative image.

[0066] That is, the normal playback mode and the representative image playback mode are switched as follows. Representative image playback mode when watching while doing other things -Normal playback mode when watching with full concentration Fig. 11 is a flowchart for explaining the process in which the content presentation device 30 switches between the normal playback mode and the representative image playback mode. The process in Fig. 11 is repeatedly executed, for example, during playback of video content. Note that the flowchart in Fig. 11 is a control example in which the playback mode is automatically switched without the viewer 9 operating by voice or finger.

[0067] The viewing status determination unit 37 periodically (for example, every second) notifies the playback control unit 42 of the viewer's current viewing status (TRUE / FALSE) (S11). TRUE indicates that the line of sight is directed toward the video content, and FALSE indicates that the line of sight is not directed toward the video content.

[0068] Each time the playback control unit 42 receives the above notification, it executes the following process: First, the playback control unit 42 determines whether or not the current playback mode is the representative image playback mode (S12).

[0069] If the determination in step S12 is Yes (in the case of the representative image playback mode), the playback control unit 42 determines whether the viewing situation value is TRUE and has continued for Ns seconds or more (S13). Ns is an arbitrary positive value, for example, about 10 seconds. Ns is an example of a third threshold value. If the determination in step S13 is No, the process ends.

[0070] If the determination in step S13 is Yes, the playback control unit 42 determines that the viewer has switched to intensive viewing, and switches the playback mode to normal playback mode (S14). The audio / video playback unit 43 displays the video stream, and the representative image playback unit 41 hides the representative image. The audio stream continues to be played. In this way, if the viewer is intensively viewing, the playback mode can be automatically switched to normal playback mode.

[0071] If the determination in step S12 is No (in the case of normal playback mode), the playback control unit 42 determines whether the viewing situation value is FALSE and this has continued for Nn seconds or more (S15). Nn is an arbitrary positive value, for example, about 10 seconds. Nn is an example of a second threshold value. If the determination in step S15 is No, the process ends.

[0072] If the determination in step S15 is Yes, the playback control unit 42 determines that the viewer has switched to multitasking viewing, and switches the playback mode to representative image playback mode (S16). The audio / video playback unit 43 first hides the video stream. The audio stream continues to be played. The representative image playback unit 41 determines a unit section that corresponds to the current playback time of the audio stream based on the representative image display metatable 13. The representative image playback unit 41 acquires the playback position in this unit section from the representative image display metatable 13, and plays the representative image by pausing at the playback position. If the unit sections overlap, the representative image with the earlier start point of the unit section is given priority. Therefore, when the viewer returns to multitasking viewing, the mode can be automatically switched to representative image playback mode, which can reduce power consumption and communication data volume.

[0073] By repeating the above process, the normal playback mode and the representative image playback mode are switched depending on the viewing situation of the viewer 9.

[0074] <<Switching playback modes through explicit viewer operations>> Fig. 12 is a flowchart for explaining the process of switching between the normal playback mode and the representative image playback mode by the content presentation device 30. The flowchart in Fig. 12 is a control example in which the viewer 9 explicitly switches the playback mode by voice or touch panel operation. The processes in steps S11 and S12 may be the same as those in Fig. 11.

[0075] If the determination in step S12 is Yes (representative image playback mode), the playback control unit 42 determines whether or not an operation to switch to normal playback mode has been detected (S13-2). The operation to switch to normal playback mode is a voice operation of "video" or pressing of the menu button by the viewer 9.

[0076] If the determination in step S13-2 is Yes, the playback control unit 42 determines that the viewer has shifted to intensive viewing, and switches the playback mode to the normal playback mode (S14). That is, the playback control unit 42 switches to the normal playback mode without waiting Ns seconds because of the operation by the viewer 9.

[0077] If the determination in step S12 is No (in the case of the normal playback mode), the playback control unit 42 determines whether or not an operation to switch to the representative image playback mode has been detected (S15-2).

[0078] If the determination in step S15-2 is Yes, the playback control unit 42 determines that the viewer has switched to multitasking viewing, and switches the playback mode to the representative image playback mode (S16). That is, the playback control unit 42 switches to the representative image playback mode without waiting for Nn seconds because the viewer 9 operated it.

[0079] <Processing during playback in representative image playback mode> Next, the playback process in the representative image playback mode will be described with reference to Fig. 13. Fig. 13 is a flow chart for explaining the process performed by the representative image playback unit 41 in the representative image playback mode. The process in Fig. 13 may be common regardless of whether the playback mode is switched by the line of sight or by operation.

[0080] The viewing status determination unit 37 periodically (for example, every second) notifies the playback control unit 42 of the current screen gaze status (TRUE / FALSE) (S21). Every time the playback control unit 42 receives the notification, it performs the following process.

[0081] The reproduction control unit 42 determines whether or not the state in which the value of the screen viewing status is TRUE has continued for Nc seconds (Nc is an arbitrary positive number and Nc < Ns.) or more (S22). If the determination in step S22 is No, the process of FIG. 13 ends.

[0082] If the determination in step S22 is Yes, the reproduction control unit 42 determines that the representative image currently being reproduced has been viewed by the viewer 9, and sets TRUE in the viewed item of the record (S23).

[0083] The audio-visual reproduction unit 43 notifies the reproduction control unit 42 of the reproduction position of the current audio stream at regular intervals (for example, every 1 second) (S24). Each time the reproduction control unit 42 receives the above notification, it performs the following process.

[0084] The reproduction control unit 42 checks each record in the representative image display meta-table 13, interval start point < current reproduction position < interval end point and extracts a list A of records that satisfy (S25). The reason why multiple records can be extracted in step S25 is that there are overlapping unit intervals. If list A is empty, the reproduction control unit 42 skips the following process (S26).

[0085] The reproduction control unit 42 extracts a list B of records in list A whose "viewed" value is FALSE (S27). If the list B of records with FALSE is empty, since the viewer 9 has viewed the representative image and the same representative image is to be displayed again, the reproduction control unit 42 extracts a list B of records whose "viewed" value is TRUE (S27).

[0086] The reproduction control unit 42 extracts a record C with the minimum index value from list B (S28).

[0087] The reproduction control unit 42 compares record C with the representative image currently being reproduced by the representative image reproduction unit 41 (S29). The process of step S29 is necessary for FIG. 13 to be repeatedly executed.

[0088] If record C and the representative image currently being played back by the representative image playback unit 41 are the same (Yes in S29), the playback control unit 42 skips the process of step S30.

[0089] If record C and the representative image currently being played back by the representative image playback unit 41 are different (No in S29), the playback control unit 42 requests the representative image playback unit 41 to change the representative image. The representative image playback unit 41 changes the playback position to the playback position of record C and pauses the playback state (S30). By pausing the playback state, the representative image is displayed as a still image.

[0090] In addition, for representative images of unit sections that overlap in part, the overlapping unit sections may be combined to equalize the playback time of each representative image. For example, in FIG. 7, unit sections 1 and 2 overlap, so the section from the section start point of unit section 1 to the section end point of unit section 2 is considered as one unit section, and the representative image playback unit 41 switches from the representative image of unit section 1 to the representative image of unit section 2 at the midpoint from the section start point of unit section 1 to the section end point of unit section 2. In this way, the presentation time of the representative image of each overlapping unit section can be equalized regardless of the playback position at which the representative image is displayed in the actual video content. The same control can be performed when three or more unit sections overlap in part. In this case, the unit section in which the representative image has been viewed will not be displayed until the other overlapping representative image is viewed.

[0091] <Major Effects> According to the content presentation device 30 of the present embodiment, during intensive viewing, the video content is played back in normal playback mode, and during multitasking viewing, the audio stream and the representative image are played back in representative image playback mode. Therefore, when the viewer 9 gazes at the video content while concurrently acting, the representative image is displayed, so that visual information useful for understanding the content of the video content can be presented at the timing when the viewer gazes at the video content while concurrently acting. Furthermore, when the viewer 9 views the representative image for a certain period of time, it is determined that the viewer is viewing attentively, and the content presentation device switches to normal playback mode. In this way, the viewer can understand the content even while concurrently watching, and the video content can be played back by automatically switching to normal playback mode. By appropriately switching the playback mode at the timing when the viewer 9 gazes at the video content while concurrently acting, visual information useful for understanding the content of the video content can be presented.

[0092] <When playing in normal playback mode while watching something else> As a modified example, switching between the normal playback mode and the representative image playback mode may be performed as follows, inversely to the flow chart of FIG. Normal playback mode for multitasking and immersive viewing Representative image playback mode for a certain period of time FIG. 14 is a diagram for explaining switching from the normal playback mode to the representative image playback mode in this modified example. (1) First, during multitasking viewing, the mode is normal playback mode, and as soon as the viewer 9 moves his / her line of sight to the video content, the mode is switched to representative image playback mode. (2) The viewer 9 views the representative images for a certain period of time and understands the content of the video. (3) After a certain period of time has elapsed since the representative image playback mode was switched to, it is determined that the viewer 9 wishes to concentrate on viewing, and the mode is switched to normal playback mode.

[0093] Fig. 15 is a flowchart for explaining the process in which the content presentation device 30 switches between the normal playback mode and the representative image playback mode. The process in Fig. 15 is repeatedly executed, for example, during playback of video content. Note that the flowchart in Fig. 15 can be applied to both the case in which the playback mode is automatically switched depending on the line of sight and the case in which the viewer 9 operates the mode by voice or finger.

[0094] The viewing status determination unit 37 periodically (for example, every second) notifies the playback control unit 42 of the current viewing status (TRUE / FALSE) of the viewer (S31).

[0095] Each time the playback control unit 42 receives the above notification, it executes the following process: First, the playback control unit 42 determines whether or not the current playback mode is the normal playback mode (S32).

[0096] If the determination in step S32 is Yes (in the case of normal playback mode), the playback control unit 42 determines whether the value of the viewing situation has changed from FALSE to TRUE (S33). If the determination in step S33 is No, the process ends.

[0097] If the determination in step S33 is Yes, the playback control unit 42 determines that the viewer wants to view the representative image, and switches the playback mode to representative image playback mode (S34). The audio / video playback unit 43 first hides the video stream. The audio stream continues to be played. The representative image playback unit 41 determines a unit section that corresponds to the current playback time of the audio stream based on the metatable 13 for displaying representative images. The representative image playback unit 41 obtains the playback position in this unit section from the metatable 13 for displaying representative images, and plays the representative image by pausing at the playback position. If unit sections overlap, the representative image that starts at the earlier unit section is given priority.

[0098] Next, the playback control unit 42 starts a timer for switching to the normal playback mode (S35).

[0099] If the determination in step S32 is No (if the representative image playback mode is selected), the playback control unit 42 determines whether or not the timer has timed out (S36).

[0100] If the time is up (Yes in S36), the playback control unit 42 determines that the viewer 9 is fully engaged in viewing, and determines to switch to normal playback mode (S37).

[0101] If the time has not expired (No in S36), the playback control unit 42 judges whether the value of the viewing situation has changed from TRUE to FALSE (S38). That is, it is judged whether the viewing state has returned to multitasking viewing in the representative image playback mode. If the judgment in step S38 is No, the viewer 9 is still viewing the representative image, so the timer continues to count.

[0102] If the determination in step S38 is Yes, since the viewer 9 is no longer viewing the video content, the playback control unit 42 switches to normal playback mode (S39).

[0103] 15, when the viewer 9 moves his / her gaze to the video content, the representative image is displayed, so that the viewer 9 can understand the details of the video content. Also, if the viewer 9 continues to watch the video content, the mode switches to normal playback mode, so that the viewer 9 can watch the video content as a video.

[0104] Therefore, during multitasking, the video content is played in normal playback mode, and during concentrated viewing, the representative image is played for a certain period of time and then the mode returns to normal playback mode. Therefore, when the viewer 9 gazes at the video content while doing multiple actions, the representative image is displayed, so that visual information useful for understanding the content of the video content can be presented at the timing when the viewer gazes at the video content while doing multiple actions.

[0105] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0106] For example, in the normal playback mode and the representative image playback mode, the representative image playback unit may display thumbnails of the representative images on the time axis. For example, as shown in Fig. 7, thumbnails of the representative images of the respective unit sections are displayed on the time axis.

[0107] Furthermore, when the unit interval is changed, the representative image of the previous unit interval may be gradually reduced in size and the representative image of the next unit interval may be gradually increased in size over a certain period of time, instead of being changed instantly from the representative image of the previous unit interval to the representative image of the next unit interval. This allows the viewer 9 to know that the representative image will soon be changed.

[0108] Further, the content server 10 may present the video content on a display connected to the content presentation device 30 locally, rather than presenting the video content to the content presentation device 30 via a network. The video content may be held by the content server 10.

[0109] Furthermore, the representative image does not have to be a still image. The representative image may be a video in which a short time range of a unit section (for example, less than 10 seconds) is repeatedly played. Such a representative image can present visual information that is difficult to convey with a still image. Furthermore, the representative image may not be part of the video content, but may be a still image or a short video that summarizes the video content by a human or machine.

[0110] Moreover, the configuration example in Fig. 6 and the like is divided according to main functions in order to facilitate understanding of the processing by the content presentation device 30. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing of the content presentation device 30 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0111] Each function of the present disclosure described above can be realized not only by software processing by executing a program, but also by one or more processing circuits. Here, the "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a device such as a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0112] 10 Content Server 20 Sensors 30 Content presentation device 100 Content Presentation System

Claims

1. A content presentation device that presents video content distributed from a content server, a sensing unit for detecting a viewing state of the video content by a viewer; a playback control unit that switches between a first playback mode in which a video is played and a second playback mode in which visual information related to the content of the video is played according to a viewing situation of the viewer; an audio / video playback unit that plays an audio stream and a video stream in the first playback mode; a representative image reproduction unit that reproduces the visual information while reproducing an audio stream in the second reproduction mode; A content presentation device comprising:

2. a content acquisition unit that acquires, together with the video content, metadata in which a plurality of unit sections set in the video content and playback positions of the visual information in the unit sections are registered; The content presentation device of claim 1, characterized in that, in the second playback mode, the audio-video playback unit identifies the unit section that includes the playback time of the audio stream, obtains from the metadata the playback position of the visual information corresponding to the unit section, and pauses the video stream at the obtained playback position.

3. When two consecutive unit intervals partially overlap each other, The content presentation device according to claim 2, characterized in that in the second playback mode, the representative image playback unit preferentially plays back the visual information of the unit interval whose start point is earlier, and when the end point of the unit interval whose start point is earlier arrives, plays back the visual information of the unit interval whose start point is later.

4. When two consecutive unit intervals partially overlap each other, The content presentation device described in claim 2, characterized in that in the second playback mode, the representative image playback unit treats two consecutive unit intervals as one unit interval, plays the visual information of the unit interval whose start point is earlier up to the midpoint of the unit interval, and plays the visual information of the unit interval whose start point is later after the midpoint.

5. the sensing unit is a camera that captures an image of the viewer, an image processing unit that detects a line of sight direction from image data captured by the sensing unit; While the representative image playback unit is playing back the visual information in the second playback mode, When the time during which the line of sight is directed toward the visual information continues to be equal to or longer than a first threshold, the playback control unit records the visual information as having been viewed, 4. The content presentation device according to claim 3, wherein the representative image playback unit plays back the visual information that has not been viewed with priority when two successive unit sections partially overlap each other.

6. The metadata is prepared according to attributes of the viewer, 3. The content presentation device according to claim 2, wherein the representative image playback unit plays back different pieces of visual information for each of the unit sections that differ according to an attribute of the viewer.

7. the sensing unit is a camera that captures an image of the viewer, an image processing unit that detects a line of sight direction from image data captured by the sensing unit; the playback control unit switches to the second playback mode when, in the first playback mode, the line of sight direction is not directed toward the video for a period of time equal to or longer than a second threshold; A content presentation device according to any one of claims 1 to 3, which switches to the first playback mode when, in the second playback mode, the gaze direction is directed toward the visual information for a period of time equal to or longer than a third threshold value.

8. the sensing unit is a microphone that detects audio data of the viewer or a touch panel that detects touch coordinates, an operation detection unit that detects operation information from the voice data or touch coordinates detected by the sensing unit; the reproduction control unit, when in the first reproduction mode, the operation information indicates an instruction to switch to the second reproduction mode, switches to the second reproduction mode; 7. The content presentation device according to claim 1, wherein, in the second playback mode, when the operation information indicates that the first playback mode should be switched, the content presentation device switches to the first playback mode.

9. A content presentation device that presents video content distributed from a content server, a playback control unit that switches between a first playback mode in which a video is played and a second playback mode in which visual information related to the content of the video is played, depending on a viewing state of the video content by a viewer detected by a sensing unit; an audio / video playback unit that plays an audio stream and a video stream in the first playback mode; a representative image reproduction unit that reproduces the visual information while reproducing an audio stream in the second reproduction mode; A program to function as a

10. A content presentation system including a content server and a content presentation device that presents video content distributed from the content server, The content server includes: Delivering the video content to the content presentation device; The content presentation device includes: a sensing unit for detecting a viewing state of the video content by a viewer; a playback control unit that switches between a first playback mode in which a video is played and a second playback mode in which visual information related to the content of the video is played according to a viewing situation of the viewer; an audio / video playback unit that plays an audio stream and a video stream in the first playback mode; a representative image reproduction unit that reproduces the visual information while reproducing an audio stream in the second reproduction mode; A content presentation system comprising:

Images