AI information processing device and AI information processing method

Abstract

The present invention provides an artificial intelligence information processing device that generates information related to scenes by artificial intelligence.　The artificial intelligence information processing device comprises a gaze degree estimation unit for estimating the gaze degree of a user viewing content by artificial intelligence on the basis of sensor information, an acquisition unit for acquiring the video of a scene in the content which the user is gazing at and information relating to the content on the basis of the estimation result of the gaze degree estimation unit, and a scene information estimation unit for estimating information relating to the scene which the user is gazing at by artificial intelligence on the basis of the video of the scene which the user is gazing at and information relating to the content.

Description

[Technical field]

[0001] The technology disclosed in this specification relates to an artificial intelligence information processing device and an artificial intelligence information processing method that process information related to content using artificial intelligence. [Background technology]

[0002] Television broadcasting services have been widely available for a long time. Currently, television receivers are widely available, with one or more installed in every household. Recently, broadcast-type video distribution services that use networks, such as IPTV (Internet Protocol TV) and OTT (Over-The-Top), are also becoming more widespread.

[0003] Recently, research and development has also been conducted on technology that combines television receivers with sensing technology to measure "viewing quality," which indicates the degree to which viewers pay attention to video content (see, for example, Patent Document 1). Viewing quality can be used in a variety of ways. For example, based on the results of viewing quality measurements, it is possible to evaluate the effectiveness of video content or advertisements, or recommend other content or products to viewers. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2017 / 120469 [Patent Document 2] Patent No. 4840393 [Patent Document 3] JP 2007-143010 A [Patent Document 4] JP 2008-236779 A Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the technology disclosed in this specification is to provide an artificial intelligence information processing device and an artificial intelligence information processing method that process information accompanying content using artificial intelligence. [Means for solving the problem]

[0006] A first aspect of the technology disclosed in this specification is an attention level estimation unit that estimates an attention level of a user viewing content by using artificial intelligence based on sensor information; an acquisition unit that acquires a video of a scene in the content that the user is gazing at and information about the content based on an estimation result of the gaze degree estimation unit; A scene information estimation unit that estimates information about the scene that the user is gazing at using artificial intelligence based on a video of the scene that the user is gazing at and information about the content; The present invention is an artificial intelligence information processing device equipped with the above.

[0007] The scene information estimation unit estimates information correlated with the scene the user is gazing at by using an artificial intelligence-based neural network that has learned the correlation between the scene image, information about the content, and information about the scene.

[0008] In addition, the attention level estimation unit estimates the degree of attention that is correlated with the sensor information regarding the user watching the content, using a neural network that has learned the correlation between sensor information and the user's attention level as an estimation based on artificial intelligence.

[0009] A second aspect of the technology disclosed in this specification is: an input unit for inputting sensor information relating to a user viewing the content; a scene information estimation unit that estimates information correlated with a scene that a user is gazing at by utilizing a neural network that has learned correlations between sensor information, content, and content information, and information about a scene that a user is gazing at; The present invention is an artificial intelligence information processing device equipped with the above.

[0010] A third aspect of the technology disclosed in this specification is: an attention level estimating step of estimating an attention level of a user viewing the content by artificial intelligence based on sensor information; an acquisition step of acquiring a video of a scene in the content that the user is gazing at and information about the content based on a result of the estimation step of the gaze degree estimation step; a scene information estimation step of estimating information about the scene that the user is gazing at by artificial intelligence based on a video of the scene that the user is gazing at and information about the content; The present invention relates to an artificial intelligence information processing method. Effect of the Invention

[0011] According to the technology disclosed in this specification, it is possible to provide an artificial intelligence information processing device and an artificial intelligence information processing method that estimate metadata related to some scenes of content using artificial intelligence.

[0012] Note that the effects described in this specification are merely examples, and the effects brought about by the technology disclosed in this specification are not limited to these. Furthermore, the technology disclosed in this specification may have additional effects in addition to the effects described above.

[0013] Further objects, features and advantages of the technology disclosed in this specification will become apparent from the following detailed description based on the embodiments and the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a system for viewing video content. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of the television receiving device 100. As shown in FIG. [Diagram 3]FIG. 3 is a diagram showing an example of the configuration of a sensor group 300 equipped in the television receiver 100. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of a scene acquisition and scene information estimation system 400. As shown in FIG. [Diagram 5] FIG. 5 is a diagram showing an example of the configuration of a neural network 500 used in the gaze degree estimation unit 406. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of the configuration of a neural network 600 used in the scene information estimation unit 408. As shown in FIG. [Figure 7] FIG. 7 is a flowchart showing the processing steps performed by the scene acquisition and scene information estimation system 400. As shown in FIG. [Figure 8] FIG. 8 illustrates a variation of a scene capture and scene information estimation system 800. In FIG. [Figure 9] FIG. 9 is a diagram showing an example of the configuration of a neural network 900 used in the scene information estimation unit 806. As shown in FIG. [Figure 10] FIG. 10 is a diagram showing how a user watching television is photographed by a camera. [Figure 11] FIG. 11 is a diagram showing how a face is recognized from an image captured by a camera. [Figure 12] FIG. 12 is a diagram showing how a gaze scene is captured according to the estimation result of the user's gaze degree. [Figure 13] FIG. 13 is a diagram showing an example of the configuration of a list screen of scenes gazed upon by the user. [Figure 14] FIG. 14 is a diagram showing an example of the configuration of a screen to which a transition occurs after a scene is selected. [Figure 15] FIG. 15 is a diagram showing an example of the configuration of a screen that presents related content. [Figure 16] FIG. 16 is a diagram showing how a scene that a user is gazing at is fed back on the screen of a smartphone. [Figure 17] FIG. 17 is a diagram showing an example of the arrangement of a plurality of scenes. [Figure 18]FIG. 18 is a diagram showing an example of the arrangement of a plurality of scenes. [Figure 19] FIG. 19 shows an example of application of the panel speaker technology. [Figure 20] FIG. 20 is a diagram showing an example of the configuration of an artificial intelligence system 2000 that uses a cloud. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the technology disclosed in this specification will be described in detail with reference to the drawings.

[0016] A. System Configuration FIG. 1 shows a schematic configuration example of a system for viewing video content.

[0017] The television receiver 100 is equipped with a large screen for displaying video content and a speaker for outputting audio. The television receiver 100 has, for example, a built-in tuner for selecting and receiving broadcast signals, or is connected to a set-top box, and can use broadcast services provided by television stations. The broadcast signals may be either terrestrial or satellite waves.

[0018] The television receiving device 100 can also use broadcast-type video distribution services that use a network, such as IPTV and OTT. For this reason, the television receiving device 100 is equipped with a network interface card and is interconnected to an external network, such as the Internet, via a router or an access point, using communication based on existing communication standards, such as Ethernet (registered trademark) and Wi-Fi (registered trademark).

[0019] A stream distribution server that distributes video streams is installed on the Internet, and provides a broadcast-type video distribution service to the television receiving device 100.

[0020] In addition, a large number of servers that provide various services are installed on the Internet. One example of the server is a stream distribution server. The television receiving device 100 can use a Web service by starting a browser function and issuing, for example, an HTTP (Hyper Text Transfer Protocol) request to the stream distribution server.

[0021] In addition, in this embodiment, it is assumed that there is an artificial intelligence server (not shown) that provides clients with artificial intelligence functions over the Internet (or on the cloud). Here, the artificial intelligence functions refer to functions that are generally performed by the human brain, such as learning, inference, data creation, and planning, which are artificially realized by software or hardware. In addition, the artificial intelligence server is equipped with a neural network that performs deep learning (DL) using a model that imitates the neural circuits of the human brain. The neural network has a mechanism in which artificial neurons (nodes) that form a network by synaptic connections acquire problem-solving capabilities while changing the connection strength of the synapses through learning. The neural network can automatically infer problem-solving rules by repeated learning. Note that the "artificial intelligence server" referred to in this specification is not limited to a single server device, and may be in the form of a cloud that provides cloud computing services, for example.

[0022] 2 shows an example of the configuration of the television receiving device 100. The television receiving device 100 includes a main control unit 201, a bus 202, a storage unit 203, a communication interface (IF) unit 204, an expansion interface (IF) unit 205, a tuner / demodulation unit 206, a demultiplexer (DEMUX) 207, a video decoder 208, an audio decoder 209, a superimposed text decoder 210, a subtitle decoder 211, a subtitle synthesis unit 212, a data decoder 213, a cache unit 214, an application (AP) control unit 215, a browser unit 216, a sound source unit 217, a video synthesis unit 218, a display unit 219, an audio synthesis unit 220, an audio output unit 221, and an operation input unit 222.

[0023] The main control unit 201 is composed of, for example, a controller, a ROM (Read Only Memory) (however, this includes a rewritable ROM such as an EEPROM (Electrically Erasable Programmable ROM)), and a RAM (Random Access Memory), and controls the overall operation of the television receiving device 100 according to a predetermined operation program. The controller is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General Purpose Graphic Processing Unit), etc., and the ROM is a non-volatile memory in which basic operation programs such as an operating system (OS) and other operation programs are stored. Operation setting values ​​required for the operation of the television receiving device 100 may be stored in the ROM. The RAM serves as a work area when the OS and other operation programs are executed. The bus 202 is a data communication path for transmitting and receiving data between the main control unit 201 and each unit in the television receiving device 100.

[0024] The storage unit 203 is composed of a non-volatile storage device such as a flash ROM, a solid state drive (SSD), or a hard disc drive (HDD). The storage unit 203 stores the operation program and operation setting values ​​of the television receiving device 100, personal information of the user who uses the television receiving device 100, and the like. It also stores operation programs downloaded via the Internet and various data created by the operation programs. The storage unit 203 can also store content such as videos, still images, and audio acquired via broadcast waves or the Internet.

[0025] The communication interface unit 204 is connected to the Internet via a router (described above) or the like, and transmits and receives data to and from each server device on the Internet and other communication devices. It is also assumed that it acquires a data stream of a program transmitted via a communication line. The router may be either a wired connection such as Ethernet (registered trademark) or a wireless connection such as Wi-Fi (registered trademark).

[0026] The tuner / demodulator 206 receives broadcast waves such as terrestrial broadcast or satellite broadcast via an antenna (not shown), and tunes (selects) a channel of a service (such as a broadcast station) desired by the user under the control of the main controller 201. The tuner / demodulator 206 also demodulates the received broadcast signal to obtain a broadcast data stream. Note that the television receiver 100 may be configured to include multiple tuners / demodulators (i.e., a multiple tuner) for the purpose of simultaneous display of multiple screens, recording of a program on another channel, etc.

[0027] The demultiplexer 207 distributes the video data stream, audio data stream, superimposed text data stream, and subtitle data stream, which are real-time presentation elements, to the video decoder 208, audio decoder 209, superimposed text decoder 210, and subtitle decoder 211, respectively, based on the control signal in the input data stream. The data streams input to the demultiplexer 207 include broadcast data streams from broadcast services and distribution data streams from distribution services such as IPTV and OTT. The former are input to the demultiplexer 207 after being selected and received and demodulated by the tuner / demodulator 206, and the latter are input to the demultiplexer 207 after being received by the communication interface 204. The demultiplexer 207 also reproduces multimedia applications and file-based data, which are components thereof, and outputs them to the application control unit 215 or temporarily stores them in the cache unit 214.

[0028] The video decoder 208 decodes the video stream input from the demultiplexer 207 and outputs video information. The audio decoder 209 decodes the audio stream input from the demultiplexer 207 and outputs audio information. In digital broadcasting, video streams and audio streams each coded according to, for example, the MPEG2 System standard are multiplexed and transmitted or distributed. The video decoder 208 and audio decoder 209 decode the coded video stream and coded video stream demultiplexed by the demultiplexer 207 according to their respective standardized decoding methods. Note that the television receiver 100 may include multiple video decoders 208 and audio decoders 143 in order to simultaneously decode multiple types of video data streams and audio data streams.

[0029] The subtitle decoder 210 decodes the subtitle data stream input from the demultiplexer 207 and outputs subtitle information. The subtitle decoder 211 decodes the subtitle data stream input from the demultiplexer 207 and outputs subtitle information. The subtitle synthesis unit 212 synthesizes the subtitle information output from the subtitle decoder 210 and the subtitle information output from the subtitle decoder 211.

[0030] The data decoder 213 decodes a data stream that is multiplexed together with video and audio into an MPEG-2 TS stream. For example, the data decoder 213 notifies the main control unit 201 of the result of decoding a general-purpose event message stored in a descriptor area of ​​a PMT (Program Map Table), which is one of the PSI (Program Specific Information) tables.

[0031] The application control unit 215 inputs control information contained in the broadcast data stream from the demultiplexer 207, or obtains the control information from a server device on the Internet via the communication interface unit 204, and interprets the control information.

[0032] The browser unit 216 presents the multimedia application file and its component file data acquired from a server device on the Internet via the cache unit 214 or the communication interface unit 204, in accordance with an instruction from the application control unit 215. The multimedia application file referred to here is, for example, an HTML (Hyper Text Markup Language) document or a BML (Broadcast Markup Language) document. The browser unit 216 also plays back the audio information of the application by influencing the sound source unit 217.

[0033] The video synthesis unit 218 inputs the video information output from the video decoder 208, the subtitle information output from the subtitle synthesis unit 212, and the application information output from the browser unit 216, and performs processing for appropriately selecting or superimposing them. The video synthesis unit 218 includes a video RAM (not shown), and drives the display unit 219 based on the video information input to this video RAM. The video synthesis unit 218 also performs superimposing processing of screen information such as an EPG (Electronic Program Guide) screen and graphics generated by an application executed by the main control unit 201, as necessary, based on the control of the main control unit 201.

[0034] The display unit 219 is a display device consisting of, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and presents to the user the image information that has been subjected to the selection or superimposition process by the image synthesis unit 218. Alternatively, as the display unit 219, a liquid crystal display device of a type in which a transmissive liquid crystal panel is divided into a plurality of display areas (blocks) and light is individually incident using a backlight for each display area may be used (for example, see Patent Document 2). This type of display device has the advantage that the amount of incident light for each display area can be controlled to expand the dynamic range of the luminance of the displayed image.

[0035] The voice synthesis unit 220 receives the voice information output from the voice decoder 209 and the voice information of the application reproduced by the sound source unit 217, and performs processing such as appropriate selection or synthesis.

[0036] The audio output unit 221 is used to output the audio of program content or data broadcasting content selected and received by the tuner / demodulator unit 206, and to output audio information (including the synthesized voice of the voice guidance or voice agent) processed by the audio synthesis unit 220. The audio output unit 221 is configured with a sound generating element such as a speaker. For example, the audio output unit 221 may be a speaker array (a multi-channel speaker or an ultra-multi-channel speaker) combining a plurality of speakers, and some or all of the speakers may be externally connected to the television receiving device 100.

[0037] In addition to the cone-type speaker, a flat panel speaker (see, for example, Patent Document 3) can be used for the audio output unit 221. Of course, a speaker array in which different types of speakers are combined can also be used as the audio output unit 221. The speaker array may also include one that outputs audio by vibrating the display unit 219 with one or more vibrators (actuators) that generate vibrations. The vibrators (actuators) may be in a form that is retrofitted to the display unit 219. FIG. 19 shows an example of application of panel speaker technology to a display. The display 1900 is supported by a stand 1902 on the back. A speaker unit 1901 is attached to the back surface of the display 1900. A vibrator 1901-1 is disposed at the left end of the speaker unit 1901, and a vibrator 1901-2 is disposed at the right end, forming a speaker array. Each of the vibrators 1901-1 and 1901-2 can vibrate the display 1901 based on the left and right audio signals, respectively, to output sound. A subwoofer that outputs low-frequency sound may be built into the stand 1902. The display 1900 corresponds to the display unit 219 that uses an organic EL element.

[0038] Returning to Fig. 2 again, the configuration of the television receiving device 100 will be described. The operation input unit 222 is an instruction input unit through which a user inputs operation instructions to the television receiving device 100. The operation input unit 222 is composed of, for example, a remote control receiving unit that receives commands transmitted from a remote control (not shown) and operation keys in the form of an array of button switches. The operation input unit 222 may also include a touch panel superimposed on the screen of the display unit 219. The operation input unit 222 may also include an external input device such as a keyboard connected to the expansion interface unit 205.

[0039] The expansion interface unit 205 is a group of interfaces for expanding the functions of the television receiving device 100, and is configured, for example, by an analog video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The expansion interface unit 205 may include a digital interface configured by a DVI terminal, an HDMI (registered trademark) terminal, a Display Port (registered trademark) terminal, etc.

[0040] In this embodiment, the expansion interface 205 is also used as an interface for importing sensor signals from various sensors included in a sensor group (described later and see FIG. 3). The sensors include both sensors equipped inside the main body of the television receiving device 100 and sensors externally connected to the television receiving device 100. The externally connected sensors also include sensors built into other CE (Consumer Electronics) devices and IoT (Internet of Things) devices that exist in the same space as the television receiving device 100. The expansion interface 205 may import the sensor signals after performing signal processing such as noise removal and further digital conversion, or may import the sensor signals as unprocessed RAW data (analog waveform signals).

[0041] B. Sensing function One of the purposes of equipping the television receiving device 100 with various sensors is to measure or estimate the degree of attention (viewing quality) when a user watches video content displayed on the display unit 219. Generally, if the satisfaction level with video content is high, the degree of attention to a particular scene tends to be high. Therefore, the degree of attention can be rephrased as "satisfaction level" with respect to the video content. In other words, the "degree of attention" that appears in this specification is synonymous with the expression "satisfaction level." In this specification, when the term "user" is used simply, it refers to a viewer who watches (or plans to watch) the video content displayed on the display unit 219, unless otherwise specified.

[0042] 3 shows an example of the configuration of a sensor group 300 equipped in the television receiving device 100. The sensor group 300 is made up of a camera unit 310, a status sensor unit 320, an environment sensor unit 330, a device status sensor unit 340, and a user profile sensor unit 350.

[0043] The camera unit 310 includes a camera 311 that captures an image of a user watching the video content displayed on the display unit 219, a camera 312 that captures the video content displayed on the display unit 219, and a camera 313 that captures the interior (or installation environment) in which the television receiving device 100 is installed.

[0044] The camera 311 is installed, for example, near the center of the upper edge of the screen of the display unit 219, and preferably captures an image of a user viewing video content. In this embodiment, the camera 311 is essential.

[0045] The camera 312 is installed, for example, facing the screen of the display unit 219, and captures the video content being viewed by the user. Alternatively, the user may wear goggles equipped with the camera 312. The camera 312 also has a function of recording (audio) the audio of the video content. However, if the television receiving device 100 has a buffer (described later) for temporarily storing the video stream and audio stream to be output, the camera 312 is not essential.

[0046] The camera 313 is composed of, for example, a panoramic camera or a wide-angle camera, and captures the room (or the installation environment) in which the television receiving device 100 is installed. Alternatively, the camera 313 may be a camera mounted on a camera table (platform head) that can be rotated around the roll, pitch, and yaw axes. However, when sufficient environmental data can be acquired by the environmental sensor 330 or when the environmental data itself is not necessary, the camera 310 is not necessary.

[0047] The state sensor unit 320 is composed of one or more sensors that acquire state information related to the state of the user. The state sensor unit 320 is intended to acquire, as state information, for example, the user's working state (whether or not the user is watching video content), the user's behavior state (movement state such as stationary, walking, running, etc., eyelid opening / closing state, gaze direction, pupil size), mental state (emotion level, excitement level, awakening level, emotion, feeling, etc., such as whether the user is immersed in or concentrated on the video content), and further physiological state. The state sensor unit 320 may include various sensors such as a sweat sensor, an electromyogram sensor, an electrooculography sensor, an electroencephalogram sensor, a breath sensor, a gas sensor, an ion concentration sensor, an IMU (Inertial Measurement Unit) that measures the user's behavior, and an audio sensor (such as a microphone) that picks up the user's speech.

[0048] The environment sensor unit 330 is made up of various sensors that measure information about the environment, such as the room in which the television receiving device 100 is installed. For example, the environment sensor unit 330 includes a temperature sensor, a humidity sensor, a light sensor, an illuminance sensor, an airflow sensor, an odor sensor, an electromagnetic wave sensor, a geomagnetic sensor, a GPS (Global Positioning System) sensor, and an audio sensor (such as a microphone) that picks up ambient sound.

[0049] The device state sensor unit 340 is composed of one or more sensors that acquire the internal state of the television receiving device 100. Alternatively, circuit components such as the video decoder 208 and the audio decoder 209 may have a function of outputting the state of an input signal, the processing status of the input signal, and the like, to the outside, and may function as a sensor that detects the internal state of the device. Furthermore, the device state sensor unit 340 may detect operations performed by the user on the television receiving device 100 or other devices, and may store the user's past operation history.

[0050] The user profile sensor unit 350 detects profile information related to a user who watches video content on the television receiving device 100. The user profile sensor unit 350 does not necessarily have to be composed of a sensor element. For example, a user profile such as the age and sex of a user may be detected based on a face image of the user captured by the camera 311 or a speech of the user captured by a sound sensor. In addition, a user profile acquired on a multifunctional information terminal carried by a user such as a smartphone may be acquired by cooperation between the television receiving device 100 and the smartphone. However, the user profile sensor unit 350 does not need to detect sensitive information related to the privacy or confidentiality of the user. In addition, it is not necessary to detect the profile of the same user every time the video content is watched, and the user profile information acquired once may be stored, for example, in an EEPROM (described above) in the main control unit 201.

[0051] Furthermore, a multi-function information terminal carried by a user, such as a smartphone, may be used as the status sensor unit 320, the environment sensor unit 330, or the user profile sensor unit 350 through cooperation between the television receiving device 100 and the smartphone. For example, sensor information acquired by a sensor built into the smartphone, and data managed by applications such as a health care function (such as a pedometer), a calendar or a schedule book / memorandum, email, and a social network service (SNS) may be added to the user's status data and environment data.

[0052] C. Scene information estimation based on gaze The television receiving device 100 according to the present embodiment can measure or estimate the degree of attention of a user to video content by combining with a sensing function as shown in Fig. 3. There are various ways to use the degree of attention. For example, an information providing device has been proposed that derives the user's preferences and interests based on the results of tallying up attribute information of the content the user has viewed, and provides the user with viewing support information and value-added information (see Patent Document 4).

[0053] The content attribute information referred to here is, for example, the genre of the content, the performers, keywords, etc., and the attribute information can be extracted from so-called metadata, which is information accompanying video content. There are various routes for obtaining metadata, such as when the metadata is superimposed on the video content, when it is distributed as broadcast data accompanying the main broadcast content, and when it is obtained via a route different from the broadcast content (for example, when the metadata of the broadcast content is obtained via the Internet). In any case, the content attribute information is often given to the entire content by the content producer or distributor.

[0054] Attribute information about content that a user finds interesting can be used for a variety of purposes, including providing viewing support to users (e.g., automatic recording scheduling and recommendations for other content), providing feedback to content creators and distributors, as well as rating content, and promoting related products.

[0055] However, the attribute information on the entire content is generally information that characterizes the entire content (attribute information included in the metadata of the content), and does not necessarily indicate attribute information that characterizes each scene in the content. For a user, information that characterizes each scene (attribute information or metadata) may be of higher interest to the user who is interested in that scene than information that characterizes the entire content. For example, if a content creator, distributor, advertisement distributor, etc. performs viewing support, content evaluation, product recommendation, etc. based on the measurement result of the user's attention degree for each scene in the content and the correlation that characterizes the entire content, it may not be possible to accurately reflect the user's interest. For this reason, it is expected that content or products will be recommended according to attribute information in which the user is not directly interested. Therefore, in order to realize more accurate content recommendation, etc., it is necessary to obtain information that characterizes each scene in which the user is interested. However, it is difficult for a content creator to attach different attribute information to all scenes when creating the content.

[0056] Therefore, in this specification, we propose a technology that extracts specific scenes from video content that a user is focusing on, and uses artificial intelligence to infer and automatically output metadata (also called labels when a neural network is used in artificial intelligence), which is information about the specific scenes.

[0057] Fig. 4 shows a configuration example of a scene acquisition and scene information estimation system 400. The illustrated system 400 is configured using components in the television receiving device 100 shown in Fig. 2 and an external device (such as a server device on the cloud) of the television receiving device 100 as necessary.

[0058] The receiving unit 401 receives video content and metadata associated with the video content. The video content includes broadcast content sent from a broadcast station (such as a radio tower or a broadcast satellite) and streaming content distributed from a stream distribution server such as an OTT service. The metadata received by the receiving unit 401 is metadata that is assigned to the entire content by the content creator or distributor. The receiving unit 401 then separates (demultiplexes) the received signal into a video stream, an audio stream, and metadata, and outputs them to the signal processing unit 402 and the buffer unit 403 in the subsequent stages.

[0059] The receiving unit 401 is configured by, for example, the tuner / demodulator 206, the communication interface unit 204, and the demultiplexer 207 in the television receiving device 100.

[0060] The signal processing unit 402 is composed of, for example, the video decoder 2080 and the audio decoder 209 in the television receiving device 100, and decodes the video data stream and the audio data stream input from the receiving unit 401, and outputs the video information and the audio information to the output unit 404. In addition, the signal processing unit 402 may output the decoded video data stream and the audio data stream to the buffer unit 403.

[0061] The output unit 404 is composed of, for example, the display unit 219 and the audio output unit 221 in the television receiver 100, and displays and outputs video information on a screen, and outputs audio information from a speaker or the like.

[0062] The buffer unit 403 has a video buffer and an audio buffer, and temporarily holds the video information and audio information decoded by the signal processing unit 402 for a certain period of time. The certain period of time corresponds to, for example, the processing time required to acquire a scene that the user is gazing at from the video content. The buffer unit 403 may be, for example, a RAM in the main control unit 201, or another buffer memory (not shown).

[0063] The sensor unit 405 is basically composed of the sensor group 300 shown in Fig. 3. However, only the camera 311 that captures an image of a user viewing video content is essential, and other cameras, a state sensor 320, an environment sensor 330, and the like are optional. For example, the scene acquisition and scene information estimation system 400 shown in Fig. 4 includes a buffer unit 403, and therefore can identify and record the scene of the video content being viewed by the user. Therefore, the camera 312 for recording the display unit 219 of the television receiver 100 from outside the television receiver 100 is not necessary to identify the scene of the video content.

[0064] The sensor unit 405 outputs a face image of the user captured by the camera 311 to the gaze degree estimation unit 406 while the user is viewing video content output from the output unit 404. The sensor unit 405 may also output to the gaze degree estimation unit 406 an image captured by the camera 313, state information of the user sensed by the state sensor unit 320, and indoor environmental information sensed by the environmental sensor unit 330.

[0065] The gaze degree estimation unit 406 estimates the degree of attention of the user to the video content being viewed by the user using artificial intelligence based on the sensor signal output from the sensor unit 405. In this embodiment, the gaze degree estimation unit 406 basically estimates the degree of attention of the user using artificial intelligence based on the recognition result of the user's face image captured by the camera 311. For example, the gaze degree estimation unit 406 estimates and outputs the degree of attention of the user based on the image recognition result of the user's facial expression such as dilated pupils or wide open mouth. Of course, the gaze degree estimation unit 406 may also input a sensor signal other than the image captured by the camera 311 to estimate the degree of attention of the user using artificial intelligence.

[0066] The gaze estimation unit 406 may use a trained neural network to provide an artificial intelligence inference function. FIG. 5 shows an example of the configuration of a gaze estimation neural network 500 used in the gaze estimation unit 406. The gaze estimation neural network 500 includes an input layer 510 that inputs an image signal captured by the camera 311 and other sensor signals, an intermediate layer 520, and an output layer 530 that outputs the gaze of the user. In the illustrated example, the intermediate layer 520 includes a plurality of intermediate layers 521, 522, ..., and the neural network 500 can perform deep learning. In consideration of processing time-series information such as video images and audio, the intermediate layer 520 may have a recurrent neural network (RNN) structure including recurrent connections.

[0067] The input layer 510 includes, as elements of an input vector, a video image stream (or still images) captured by the camera 311. Basically, the image signal captured by the camera 311 is input to the input layer 510 in the form of RAW data.

[0068] When sensor signals from sensors other than the image captured by the camera 311 are also used to measure the gaze level, input nodes corresponding to the respective sensor signals are additionally arranged in the input layer 510. In addition, a Convolutional Neural Network (CNN) may be used to condense feature points for input of image signals and audio signals.

[0069] The output layer is made up of output nodes 530 that output the results of estimating the degree of attention from the image signal and the sensor signal as an attention level expressed as a continuous value (or discrete value) from 0 to 100, for example. When the output node 530 outputs the attention level consisting of a continuous value, it may be determined that the user has "gazed" or "not gazed" at the scene depending on whether the output attention level exceeds a predetermined value. Alternatively, the output node 530 may be configured to output a discrete value such as "gazed" or "not gazed".

[0070] In the process of learning the gaze estimation neural network 500, a huge number of combinations of face images, other sensor signals, and the user's gaze are input to the gaze estimation neural network 500, and the weighting coefficient (inference coefficient) of each node in the intermediate layer 520 is updated so that the strength of connection with the output node of the likely gaze for the face image and other sensor signals is increased, thereby learning the correlation between the user's face image (which may include other sensor signals) and the user's gaze. In the process of using the gaze estimation neural network 500 (estimating the gaze), when the face image captured by the camera 311 and other sensor information are input to the trained gaze estimation neural network 500, the user's gaze is output with high accuracy.

[0071] 5 is realized, for example, in the main control unit 201. For this reason, a processor dedicated to the neural network may be included in the main control unit 201. The attention degree estimation neural network 500 may be provided in a cloud on the Internet, but in order to estimate the attention degree for video content in real time, it is preferable that the attention degree estimation neural network 500 is disposed in the television receiving device 100.

[0072] For example, the television receiver 100 is shipped incorporating the gaze estimation neural network 500 that has completed learning using an expert teaching database. The gaze estimation neural network 500 may continue learning using an algorithm such as backpropagation. Alternatively, the result of learning performed based on data collected from a huge number of users on the cloud side on the Internet may be updated to the gaze estimation neural network 500 in the television receiver 100 installed in each home, but this point will be described later.

[0073] Referring again to FIG. 4, the description of the scene capture and scene information inference system 400 continues.

[0074] The scene acquisition unit 407 acquires from the buffer unit 403 the video stream and audio stream of the section where the attention estimation unit 406 determines that the user is gazing (or the section where the attention level exceeds a predetermined value), as well as metadata regarding the entire content, and outputs them to the scene information estimation unit 408.

[0075] The video stream and audio stream acquired by the scene acquisition unit 407 from the buffer unit 403 based on the user's degree of attention can be said to be the scene the user was gazing at. On the other hand, metadata about the entire content does not necessarily apply to some scenes in the content. This is because the user may be very interested in a feature unique to a particular scene (for example, an object that appears only in a particular scene) that is not very meaningful for the entire content.

[0076] Therefore, in the scene acquisition and scene information estimation system 400 of this embodiment, the scene information estimation unit 408 is configured to input the video stream and audio stream of the scene on which the user is gazing and metadata related to the entire content from the scene acquisition unit 407, use artificial intelligence to estimate what (likely) metadata (also called a label) characterizes the scene on which the user is gazing, and output the metadata as information appropriate to a specific scene rather than the entire content.

[0077] The scene information estimation unit 408 may use a trained neural network to provide an artificial intelligence inference function. FIG. 6 shows an example of the configuration of a scene information estimation neural network 600 used in the scene information estimation unit 408. The scene information estimation neural network 600 includes an input layer 610 that inputs a video stream and an audio stream of a scene gazed upon by a user and metadata of the entire content, an intermediate layer 620, and an output layer 630 that outputs metadata that is information characterizing the scene gazed upon by the user. The intermediate layer 620 includes a plurality of intermediate layers 621, 622, . . . , and it is preferable that the neural network 600 is capable of performing deep learning. In consideration of time-series information such as a video stream and an audio stream, the intermediate layer 620 may have an RNN structure including a recurrent connection.

[0078] The input layer 610 includes the video stream and audio stream stored in the buffer unit 403 after decoding as elements of the input vector. However, if the buffer unit 403 holds the video stream and audio stream in the form of RAW data, the RAW data is input to the input layer 610 as is. For the audio stream, the input waveform signal for each window that is continuous in the time axis direction is included as an element of the input vector. There may be an overlapping section between previous and next windows. Also, a frequency signal obtained by FFT (Fast Fourier Transform) processing of the waveform signal for each window may be used as an element of the input vector.

[0079] The input layer 610 receives metadata, which is information that characterizes the entire content. If the content is a broadcast program, the metadata consists of text data such as the program name, the names of the performers, a summary of the program content, and keywords. The input layer 610 has input nodes corresponding to each piece of text data. CNN may be used to input image and audio signals, and a feature point condensation process may be performed.

[0080] The output layer 630 generates an output that can infer (plausible) metadata (also called a label), which is information that characterizes the scene that the user is gazing at. The metadata for each scene is assumed to include metadata of the entire original content, such as the program name, the names of the performers, a summary of the program contents, and keywords, as well as things that are not included in the metadata of the original content, such as objects that appear in the scene (for example, the brand name of the clothes and accessories worn by the performers, the name of the store of the paper cup of coffee that the performers are holding, and the location), the name of the BGM, and the lines of the performers. In the output layer 630, output nodes corresponding to each text data of these metadata are arranged. Then, the output node corresponding to the plausible metadata for the video stream and audio stream of the scene input to the input layer 610 is fired.

[0081] In the learning process of the scene information estimation neural network 600, the correlation between the scene and metadata is learned by updating the weight coefficient (inference coefficient) of each node of the intermediate layer 620 consisting of multiple layers so that the connection strength with the output node of plausible metadata for the video stream and audio stream of the scene gazed upon by the user is increased. Then, in the use of the scene information estimation neural network 600, that is, in the process of scene information estimation, plausible metadata is output with high accuracy for the video stream and audio stream of the scene gazed upon by the user.

[0082] The scene information estimation neural network 600 as shown in FIG. 6 is realized, for example, in the main control unit 201. For this reason, the main control unit 201 may include a processor dedicated to the neural network. For example, the television receiving device 100 is shipped with the scene information estimation neural network 600 that has completed learning using an expert teaching database incorporated therein. The scene information estimation neural network 600 may learn using an algorithm such as backpropagation. Also, the scene information estimation neural network 600 in the television receiving device 100 installed in each home may be updated with a learning result based on data collected from a huge number of users on the cloud side on the Internet, but this point will be described later.

[0083] If real-time performance is not required, the scene information estimation neural network 600 may be provided via a cloud on the Internet.

[0084] The final output of the scene acquisition and scene information estimation system 400 is the user's gaze scene acquired by the scene acquisition unit 407 and metadata (hereinafter also referred to as "scene metadata") as information characterizing the scene estimated for each scene output by the scene information estimation unit 408.

[0085] The scene gazed at by the user and the metadata of the scene by the scene acquisition and scene information estimation system 400 may be output to various destinations. For example, the scene may be stored in the television receiving device 100 where the user viewed the content, or may be uploaded to a server on the Internet. The upload destination server may be an artificial intelligence server in which the scene information estimation neural network 600 is constructed, or may be another server that collects metadata. The scene gazed at by the user and the metadata of the scene may be output to an information terminal such as a smartphone owned by the user. For example, an application (temporarily called a "companion application") linked to the scene acquisition and scene information estimation system 400 may be started on the smartphone, so that the scene gazed at by the user and the metadata of the scene can be viewed.

[0086] There are various ways to use the metadata of the scene that the user gazed upon. For example, it can be used to evaluate the video content that the user has viewed, or to recommend other video content. It can also be used for marketing, such as recommending products related to the scene to the user.

[0087] FIG. 7 shows, in the form of a flowchart, a processing procedure executed by the scene acquisition and scene information estimation system 400 for acquiring a scene from video content and outputting metadata, which is information characterizing the scene.

[0088] First, the receiving unit 401 inputs video content and metadata (or attribute data included in the metadata), which is information about the entire content (step S701). The video content includes broadcast content sent from a broadcast station (such as a radio tower or a broadcast satellite) and streaming content distributed from a stream distribution server such as an OTT service. The metadata received by the receiving unit 401 is metadata that is assigned to the entire content by the content creator or distributor.

[0089] Next, the video stream and audio stream of the content inputted by the receiving unit 401 are processed by the signal processing unit 402 and outputted by the output unit 404, and the content and metadata are buffered (step S702).

[0090] Then, the attention level of the user while viewing the content presented by the output unit 404 is estimated by the attention level estimation unit 406 using artificial intelligence (step S703).

[0091] The gaze degree estimation unit 406 basically measures the gaze degree of the user or estimates it by artificial intelligence based on the recognition result of the face image of the user captured by the camera 311. The gaze degree estimation unit 406 can also use sensor signals of sensors other than the captured image of the camera 311 to measure the gaze degree. In addition, the gaze degree estimation unit 406 uses an gaze degree estimation neural network 500 (see FIG. 5) that has learned the correlation between the face image and other sensor information and the gaze degree of the user in order to perform estimation by artificial intelligence.

[0092] Next, when the gaze degree estimation unit 406 determines that the user has gazed (Yes in step S70), the scene acquisition unit 407 acquires the video stream and audio stream of the section where the user gazed from the buffer unit 403 as the scene where the user gazed, and also acquires metadata of the entire content from the buffer unit 403 and outputs it to the scene information estimation unit 408 (step S705).

[0093] Next, the scene information estimation unit 408 inputs the video stream and audio stream of the scene gazed upon by the user and metadata related to the entire content from the scene acquisition unit 407, and estimates what (likely) metadata characterizing the scene gazed upon by the user is by artificial intelligence (step S706). In order to perform estimation by artificial intelligence, the scene information estimation unit 408 uses a scene information estimation neural network 600 (see FIG. 6) that has learned the correlation between the scene gazed upon by the user and the likely metadata for the scene.

[0094] Then, the scene gazed upon by the user and the scene metadata characterizing (presumably) that scene are output to a predetermined output destination (step S707), and this process ends.

[0095] Fig. 8 shows a modified example of a scene acquisition and scene information estimation system 800. The illustrated system 800 is configured using components within the television receiving device 100 shown in Fig. 2 and an external device (such as a server device on the cloud) of the television receiving device 100 as necessary.

[0096] The configurations and operations of the receiving unit 801, signal processing unit 802, buffer unit 803, output unit 804, and sensor unit 805 are similar to those of the scene information estimation system 400 shown in FIG. 4, so detailed description thereof will be omitted here.

[0097] The scene information estimation unit 806 collectively realizes the processes performed by the gaze degree estimation unit 406, the scene acquisition unit 407, and the scene information estimation unit 408 in the scene information estimation system 400 shown in FIG. 4 using a single neural network.

[0098] 9 shows an example of the configuration of a scene information estimation neural network 900 used in the scene information estimation unit 806. The scene information estimation neural network 900 includes an input layer 910, an intermediate layer 920, and an output layer 930. In the example shown in the figure, the intermediate layer 920 includes a plurality of intermediate layers 921, 922, ..., and the scene information estimation neural network 900 can perform deep learning (DL). Note that, taking into account time-series information such as video images and audio, the intermediate layer 920 may have a recurrent neural network (RNN) structure including recurrent connections.

[0099] The input layer 910 includes nodes for inputting image signals captured by the camera 311 and other sensor information, video streams and audio streams temporarily stored in the buffer unit 803, and metadata of content.

[0100] The input layer 910 includes elements of an input vector, such as a video image stream (or still images) captured by the camera 311, and a video stream and audio stream of content. Basically, image signals and audio signals are input to the input layer 910 in the form of raw data. For audio streams, input waveform signals for each window that are continuous in the time axis direction are used as input vectors to be input to each node of the input layer 910. There may be overlapping sections between previous and next windows. Also, a frequency signal obtained by FFT processing the waveform signal for each window may be used as the input vector (same as above). Also, a convolutional neural network CNN may be used to input image signals and audio signals, and a feature point condensation process may be performed.

[0101] In addition, when metadata consisting of text data such as the program name, performer names, program summary, and keywords of the broadcast content is input to the input layer 910, input nodes corresponding to each piece of text data are placed in the input layer 910.

[0102] On the other hand, the output layer 930 generates an output that can estimate the scene that the user has focused on and (likely) metadata (also called a label) that characterizes the scene. The metadata for each scene is assumed to include not only the metadata of the entire original content, such as the program name, the names of the performers, a summary of the program contents, and keywords, but also things that are not included in the metadata of the original content, such as objects that appear in the scene (for example, the brand name of the clothes and accessories worn by the performers, the name of the store of the paper cup of coffee that the performers are holding, and the location), the name of the BGM, and the lines of the performers. Therefore, the output layer 930 is provided with output nodes corresponding to each text data of these metadata. Then, for the video stream and audio stream input to the input layer 910, the output nodes corresponding to the video stream and audio stream of the scene that the user has focused on and the likely metadata for that scene are fired.

[0103] In the learning process of the scene information estimation neural network 900, the correlation between scenes and metadata is learned by updating the weight coefficients (inference coefficients) of each node of the intermediate layer 920 consisting of multiple layers so that the connection strength between the output node of the plausible gaze scene and the plausible metadata for that scene is increased for the face image and other sensor signals, and the video stream and audio stream inputted from moment to moment. Then, in the use of the scene information estimation neural network 900, that is, in the process of scene information estimation, the plausible gaze scene and the plausible metadata for that scene are outputted with high accuracy for the face image and other sensor signals, and the video stream and audio stream inputted from moment to moment.

[0104] The scene information estimation neural network 900 as shown in FIG. 9 is realized, for example, in the main control unit 201. For this reason, a processor dedicated to the neural network may be included in the main control unit 201 or in a processing circuit separate from the main control unit 201. For example, a television receiver 100 incorporating the scene information estimation neural network 900 that has completed learning using an expert teaching database is shipped. The scene information estimation neural network 900 may learn using an algorithm such as backpropagation. Also, the scene information estimation neural network 900 in the television receiver 100 installed in each home may be updated with a learning result based on data collected from a huge number of users on the cloud side on the Internet, but this point will be described later.

[0105] If real-time performance is not required, the scene information estimation neural network 900 may be provided via a cloud on the Internet.

[0106] The plausible user gaze scene and plausible metadata for that scene identified by the scene information estimation unit 806 are the final output of the scene capture and scene information estimation system 800.

[0107] D. Feedback of scene information estimation results based on gaze level Here, a method of feeding back to the user the scene acquisition and scene information estimation results by the scene information estimation system 400 will be described together with the operation of the scene information estimation system 400.

[0108] As shown in FIG. 10, while video (such as broadcast content or streaming video of an OTT service) is displayed on the screen of the television receiving device 100, a user is continuously photographed by a camera 311 installed, for example, near the center of the upper edge of the screen.

[0109] The gaze degree estimation unit 406 measures the gaze degree of the user by performing face recognition from the image captured by the camera 311. FIG. 11 shows a state where face images 1101 and 1102 of two users are recognized from the image captured by the camera 311. The gaze degree estimation unit 406 performs gaze degree estimation for at least one of the face images 1101 and 1102. The gaze degree estimation neural network 500 shown in FIG. 5 may be used for estimating the gaze degree of the user to the content by artificial intelligence. The gaze degree estimation neural network 500 may estimate the gaze degree of the user based on a sensor signal of another sensor in addition to the image captured by the camera 311.

[0110] When the gaze degree estimation unit 406 estimates that the face image 1101 or 1102 recognized from the image captured by the camera 311 indicates gazing, the video stream and audio stream stored in the buffer unit 403 during the section estimated to be gazing are captured as the user's gaze scene. Fig. 12 illustrates an example of capturing a gaze scene in conjunction with measurement of the user's gaze degree.

[0111] The scene of the user's gaze and metadata characterizing the scene output by the scene acquisition and scene information estimation system 400 are uploaded to a server on the Internet. For example, the content recommendation server searches for similar or related content to be recommended to the user based on the scene of the user's gaze and the metadata of the scene, and provides the user with information on the recommended content. This type of content recommendation server may use an algorithm such as CF (Collaborative Filtering) or CBF (Content Based Filtering) to search for similar or related content from the scene of the user's gaze. Alternatively, the content recommendation server may be the artificial intelligence server described above, and extract similar or related content using a neural network that has learned the correlation between metadata and content.

[0112] Regardless of which algorithm the content recommendation server uses to search for similar or related content, it searches for similar or related content based on the metadata of a specific scene in the content that the user focuses on, rather than on the entire content. This makes it possible to recommend to the user content that is less related to metadata related to the entire content, such as the program name or cast members, but is related to small phenomena that appear only in the scene that the user focuses on (for example, the brand of clothing worn by the cast members or accessories such as sunglasses and watches).

[0113] FIG. 13 shows an example of a screen configuration that feeds back scenes that a user has previously gazed upon. On the screen shown in the figure, a list of representative images of scenes that a user who has watched video content on the television receiving device 100 has gazed upon in the past is displayed in a matrix form. The "representative image" referred to here may be, for example, the first image of a scene cut out from the content, or an image captured from a scene according to a predetermined algorithm or randomly. The display format on the screen is not limited to a matrix form, and may be a horizontal or vertical line that virtually forms an infinitely long ring. Also, many scenes arranged at arbitrary positions in a three-dimensional space may be displayed as a two-dimensional plane that is visually displayed when the three-dimensional space is viewed from a predetermined viewpoint, and the viewpoint may be changed.

[0114] The user can select a representative image of a scene of interest from the list by using, for example, a cross key on a remote control. Then, when one of the representative images is selected, for example, by pressing a decision button on the remote control, the selection of that scene is confirmed. The above remote control operation for selecting a specific scene from the list of representative images can also be performed by voice instruction from a microphone through a voice agent function such as an AI (Artificial Intelligence) speaker with an artificial intelligence function. Alternatively, the remote control operation can be performed by gestures. The gestures are not limited to hands, but can be performed using any part of the body including the head. In addition, when smart glasses or the like are worn, the line of sight can be recognized and a selection confirmation instruction can be given by a gesture using the eyes, such as a wink. Furthermore, wireless headphones equipped with a motion sensor that can detect head movement by an artificial intelligence function can be used. In wireless devices such as wireless headphones, a selection confirmation instruction can be given using a touch sensor.

[0115] Fig. 14 shows an example of the screen configuration to which the transition occurs after a certain representative image is selected. In the example shown, one representative image selected by the user from a list of representative images (see Fig. 13) is enlarged and displayed, along with displaying part or all of the scene metadata (MD#1, MD#2, ...), which is information estimated for that scene by the scene information estimation system 400. When the representative image is displayed, the video stream and audio stream of the section gazed upon by the user may be reproduced and output, instead of a still image.

[0116] By viewing a representative image or video of a gazed scene, the user can recall memories of scenes gazed upon in the past. In addition, by viewing the representative image of a scene gazed upon together with the metadata, the user can confirm the reason or basis for his / her gaze upon that scene. If the user wishes to change the metadata that characterizes a scene and that has been automatically assigned to the scene by the scene information estimation unit 408 using artificial intelligence, the user can edit (change, add, delete, etc.) the metadata through a remote control operation or a voice agent function, and customize the metadata associated with the scene for personal use.

[0117] Furthermore, the user can instruct the presentation of similar or related content recommended based on the selected gaze scene through a remote control operation or a voice agent. FIG. 15 shows an example of the configuration of a screen that presents similar or related content recommended based on the metadata of a selected scene. The left half of the illustrated screen displays the user's gaze scene together with the metadata of the scene. In addition, a list of similar or related content is displayed on the right half of the screen. In the list of similar or related content, a representative image and metadata of each similar or related content are displayed. The user can determine whether or not he or she wants to view each similar or related content based on the representative image and metadata. Then, when the user finds similar or related content that he or she wants to view, he or she can instruct the playback of the content through a remote control operation or a voice agent function, and start viewing.

[0118] In addition, the user can view feedback of the gazed-at scene not only on the large screen of the television receiver 100 but also on the small screen of an information terminal such as a smartphone. For example, a companion application that works in conjunction with the scene acquisition and scene information estimation system 400 may be started on the smartphone to enable the user to view the scene that the user gazed at and the metadata of that scene.

[0119] FIG. 16 illustrates an example of how a scene that a user is gazing at is fed back on the screen of a smartphone. The screen of a smartphone is small, and it is not possible to display a list of representative images of multiple scenes like the television screen shown in FIG. 13. For this reason, only the representative image of one scene is displayed on one screen. However, the representative images of multiple scenes may be virtually arranged in a carousel shape as shown in FIG. 17, or virtually arranged in a matrix shape as shown in FIG. 18, and when the user flicks a representative image displayed on the screen of the smartphone in the horizontal or vertical direction, the display may transition to an adjacent representative image in the direction of the flick on the carousel or matrix.

[0120] Returning to Fig. 16 again, the configuration of the feedback screen on the smartphone will be described. When the representative image is displayed on the screen, the video stream and audio stream of the section the user is gazing at may be reproduced and output instead of a still image. In addition, a part or all of the metadata estimated by the scene information estimation system 400 for that scene is displayed below the representative image.

[0121] According to the scene information estimation system 400 disclosed in this specification, the user can recall the memory of the scene that he / she gazed at in the past by viewing the representative image or video of the gazed scene. According to the scene information estimation system 400 disclosed in this specification, the user can confirm the reason or basis for his / her gaze at the scene by viewing the representative image of the scene that he / she gazed at together with the metadata. Furthermore, according to the scene information estimation system 400 disclosed in this specification, when the user wants to change the metadata assigned to the scene, he / she can edit (change, add, delete, etc.) the metadata by using the editing function of the smartphone using the touch panel. Furthermore, according to the scene information estimation system 400 disclosed in this specification, the scene with high gaze of the user is accumulated and learned, so that the tendency of the video content with high gaze of the user, i.e., high "satisfaction" can be learned. Therefore, the artificial intelligence has the effect of increasing the possibility of providing the user with video content with high "satisfaction" through the learning function.

[0122] E. Neural network updates So far, we have explained the gaze estimation neural network 500, the scene information estimation neural network 600, and the scene information estimation neural network 900, which are used in the process of estimating metadata related to a scene in video content that a user gazes at using artificial intelligence.

[0123] These neural networks are used as functions of artificial intelligence, and operate in an operating environment (hereinafter also referred to as a "local environment") such as a home where a user can directly operate a television receiving device 100 installed in each home. One of the advantages of operating a neural network in a local environment as a function of artificial intelligence is that, for example, learning can be easily realized in real time using an algorithm such as backpropagation for these neural networks, and using feedback from users as teacher data. The feedback from the user is, for example, the user's gaze degree estimated by the gaze degree estimation neural network 500, or the user's evaluation of metadata, which is information characterizing a scene estimated by the scene information estimation neural network 600 or 900. The user feedback may be simple, such as OK (good) or NG (bad). The user feedback is input to the television receiving device 100 via, for example, the operation input unit 222, a remote control, a voice agent which is a form of artificial intelligence, or a linked smartphone. Thus, another aspect of the power of having neural networks running in local environments as these artificial intelligence functions is that the neural networks can be customized or personalized to specific users by learning using user feedback.

[0124] On the other hand, a method is also conceivable in which data is collected from a huge number of users in one or more server devices (hereinafter simply referred to as "cloud") operating on a cloud, which is a collection of server devices on the Internet, and neural network learning is accumulated as an artificial intelligence function, and the learning results are used to update the neural network in the television receiving device 100 in each home. One of the effects of updating the neural network that functions as an artificial intelligence in the cloud is that a more accurate neural network can be constructed by learning from a large amount of data.

[0125] 20 is a schematic diagram showing an example of the configuration of a cloud-based artificial intelligence system 2000. The illustrated cloud-based artificial intelligence system 2000 is made up of a local environment 2010 and a cloud 2020.

[0126] The local environment 2010 corresponds to an operating environment (home) in which the television receiving device 100 is installed, or the television receiving device 100 installed in the home. For simplicity, only one local environment 2010 is illustrated in FIG. 20, but in reality, it is assumed that a huge number of local environments are connected to one cloud 2020. In addition, in this embodiment, the television receiving device 100 or an operating environment such as a home in which the television receiving device 100 operates is mainly exemplified as the local environment 2010, but the local environment 2010 may be any device that can be directly operated by a user, such as a smartphone or a wearable device, or an environment in which the device operates (including public facilities such as stations, bus stops, airports, and shopping centers, and work facilities such as factories and workplaces).

[0127] As described above, the television receiving device 100 includes, as artificial intelligence, the gaze estimation neural network 500 and the scene information estimation neural network 600, or the scene information estimation neural network 900. These neural networks that are installed in the television receiving device 100 and are actually used are collectively referred to as the operational neural network 2011. It is assumed that the operational neural network 2011 has been previously trained using an expert teaching database consisting of a huge amount of sample data.

[0128] Meanwhile, the cloud 2020 is equipped with an artificial intelligence server (described above) (consisting of one or more server devices) that provides an artificial intelligence function. The artificial intelligence server is provided with an operational neural network 2021 and an evaluation neural network 2022 that evaluates the operational neural network 2022. The operational neural network 2021 has the same configuration as the operational neural network 2011 arranged in the local environment 2010, and is assumed to have been trained in advance using an expert teaching database consisting of a huge amount of sample data. The evaluation neural network 2022 is a neural network used to evaluate the learning status of the operational neural network 2021.

[0129] On the local environment 2010 side, the operational neural network 2011 receives sensor information such as images captured by the camera 311 and a user profile, and outputs metadata for each scene and a gaze level that matches the user profile. However, when the operational neural network 2011 is the scene information estimation neural network 600, the input is a video stream of a scene that the user gazes at and metadata of the original content. For simplicity, the input to the operational neural network 2011 will be simply called an "input value," and the output from the operational neural network 2012 will be simply called an "output value."

[0130] A user of the local environment 2010 (e.g., a viewer of the television receiving device 100) evaluates the output value of the operational neural network 2011 and feeds back the evaluation result to the television receiving device 100 via, for example, the operation input unit 222, a remote control, a voice agent, a linked smartphone, etc. Here, for simplicity of explanation, it is assumed that the user feedback is either OK (0) or NG (1).

[0131] Feedback data consisting of a combination of input values ​​and output values ​​of the operational neural network 2011 and user feedback is transmitted from the local environment 2010 to the cloud 2020. In the cloud 2020, the feedback data transmitted from a huge number of local environments is accumulated in a feedback database 2023. A huge amount of feedback data describing the correspondence between the input values ​​and output values ​​of the operational neural network 2011 and users is accumulated in the feedback database 2023.

[0132] The cloud 2020 also owns or has access to an expert teaching database 2024 consisting of a huge amount of sample data used in pre-learning of the operational neural network 2011. Each piece of sample data is teaching data describing the correspondence between sensor information and user profiles and output values ​​of the operational neural network 2011 (or 2021).

[0133] When feedback data is retrieved from the feedback database 2023, an input value included in the feedback data (e.g., a combination of sensor information and a user profile) is input to the operational neural network 2021. In addition, an output value of the operational neural network 2021 and an input value included in the corresponding feedback data (e.g., a combination of sensor information and a user profile) are input to the evaluation neural network 2022, which outputs user feedback.

[0134] Within the cloud 2020, training of an evaluation neural network 2022 as a first step and training of an operational neural network 2021 as a second step are alternately performed.

[0135] The evaluation neural network 2022 is a network that learns the correspondence between the input value to the operational neural network 2021 and the user feedback for the output of the operational neural network 2021. Therefore, in the first step, the evaluation neural network 2022 inputs the output value of the operational neural network 2021 and the user feedback included in the corresponding feedback data, and learns so that the user feedback that it outputs for the output value of the operational neural network 2021 matches the real user feedback for the output value of the operational neural network 2021. As a result, the evaluation neural network 2022 is learned so as to output user feedback (OK or NG) similar to that of a real user for the output of the operational neural network 2021.

[0136] In the subsequent second step, the evaluation neural network 2022 is fixed, and learning is then performed on the operational neural network 2021. As described above, when feedback data is retrieved from the feedback database 2023, the input value contained in the feedback data is input to the operational neural network 2021, and the output value of the operational neural network 2021 and the user feedback data contained in the corresponding feedback data are input to the evaluation neural network 2022, which outputs user feedback equivalent to that of the actual user.

[0137] At this time, the operational neural network 2021 applies an evaluation function (e.g., a loss function) to the output from the output layer of the neural network, and performs learning using backpropagation so that the value is minimized. For example, when user feedback is used as training data, the operational neural network 2021 learns so that the output of the evaluation neural network 2022 becomes OK (0) for all input values. By performing such learning, the operational neural network 2021 becomes able to output an output value (attention level, metadata for a scene, etc.) that the user feeds back as OK for any input value (sensor information, user profile, etc.).

[0138] Moreover, when training the operational neural network 2021, the expert teaching database 2024 may be used as training data. Training may be performed using two or more pieces of training data, such as user feedback and the expert teaching database 2024. In this case, the operational neural network 2021 may be trained to minimize the weighted sum of loss functions calculated for each training data.

[0139] As described above, the learning of evaluation neural network 2022 as a first step and the learning of operational neural network 2021 as a second step are alternately performed, thereby improving the accuracy of operational neural network 2021. Then, by providing the inference coefficients in operational neural network 2021, the accuracy of which has been improved by learning, to operational neural network 2011 in local environment 2010, the user can also enjoy operational neural network 2011 with further advanced learning.

[0140] For example, the bit stream of the inference coefficients of the operational neural network 2011 may be compressed and downloaded to the local environment from the cloud 2020. When the size of the bit stream is large even after compression, the inference coefficients may be divided for each layer or region, and the compressed bit stream may be downloaded in multiple installments. [Industrial Applicability]

[0141] The technology disclosed in this specification has been described in detail above with reference to specific embodiments. However, it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the technology disclosed in this specification.

[0142] In this specification, the embodiment in which the technology disclosed in this specification is applied to a television receiver has been mainly described, but the gist of the technology disclosed in this specification is not limited to this. The technology disclosed in this specification can also be applied to various types of content playback devices that present various playback contents such as video and audio to users.

[0143] In short, the technology disclosed in this specification has been described in the form of examples, and the contents of the specification should not be interpreted in a limiting manner. In order to judge the gist of the technology disclosed in this specification, the claims should be taken into consideration.

[0144] The technology disclosed in this specification can also be configured as follows.

[0145] (1) an attention level estimation unit that estimates an attention level of a user viewing content by using artificial intelligence based on sensor information; an acquisition unit that acquires a video of a scene in the content that the user is gazing at and information about the content based on an estimation result of the gaze degree estimation unit; A scene information estimation unit that estimates information about the scene that the user is gazing at using artificial intelligence based on a video of the scene that the user is gazing at and information about the content; An artificial intelligence information processing device comprising:

[0146] (2) The scene information estimation unit estimates information correlated with the scene that the user is gazing at by using a neural network that has learned the correlation between the scene image, information about the content, and information about the scene, as an estimation by artificial intelligence. The artificial intelligence information processing device according to (1) above.

[0147] (3) The gaze degree estimation unit estimates a gaze degree that is correlated with the sensor information regarding the user viewing the content by using a neural network that has learned a correlation between the sensor information and the gaze degree of the user as an estimation based on artificial intelligence. An artificial intelligence information processing device according to any one of (1) or (2) above.

[0148] (4) the sensor information includes at least a captured image of the user viewing the content, The gaze degree estimation unit estimates a gaze degree that is correlated with the face recognition result in the captured image by using a neural network that has learned a correlation between the face recognition result and the gaze degree of the user as an estimation by artificial intelligence. The artificial intelligence information processing device according to (3) above.

[0149] (5) The content includes at least one of broadcast content and stream content. An artificial intelligence information processing device according to any one of (1) to (4) above.

[0150] (6) outputting information about the scene the user is gazing at to an external device; An artificial intelligence information processing device according to any one of (1) to (5) above.

[0151] (7) Presenting to the user a scene that the user is gazing at and information about the scene; An artificial intelligence information processing device according to any one of (1) to (6) above.

[0152] (8) an input unit for inputting sensor information regarding a user viewing the content; a scene information estimation unit that estimates information correlated with a scene that a user is gazing at by utilizing a neural network that has learned correlations between sensor information, content, and content information, and information about a scene that a user is gazing at; An artificial intelligence information processing device comprising:

[0153] (9) The sensor information includes at least a captured image of the user viewing the content, The scene information estimation unit estimates information correlated with the scene the user is gazing at by utilizing a neural network that has learned correlations between face images, content, and content information, and information about the scene the user is gazing at. The artificial intelligence information processing device according to (8) above.

[0154] (10) an attention level estimation step of estimating an attention level of a user viewing the content by using artificial intelligence based on sensor information; an acquisition step of acquiring a video of a scene in the content that the user is gazing at and information about the content based on a result of the estimation step of the gaze degree estimation step; a scene information estimation step of estimating information about the scene that the user is gazing at by artificial intelligence based on a video of the scene that the user is gazing at and information about the content; An artificial intelligence information processing method comprising the steps of: [Explanation of symbols]

[0155] 100...television receiving device, 201...control unit, 202...bus 203: storage unit, 204: communication interface (IF) unit 205...Expansion interface (IF) section 206 ... tuner / demodulation unit, 207 ... demultiplexer 208...video decoder, 209...audio decoder 210... Superimposed text decoder, 211... Subtitle decoder 212: subtitle synthesis unit, 213: data decoder, 214: cache unit 215... application (AP) control unit, 216... browser unit 217: sound source unit, 218: video synthesis unit, 219: display unit 220: voice synthesis unit, 221: voice output unit, 222: operation input unit 300: sensor group, 310: camera unit, 311 to 313: cameras 320: condition sensor unit, 330: environment sensor unit 340... device state sensor unit, 350... user profile sensor unit 400: Scene acquisition and scene information estimation system, 401: Receiving unit 402: signal processing section, 403: buffer section, 404: output section 405: sensor unit, 406: gaze degree estimation unit, 407: scene acquisition unit 408…Scene information estimation unit 500…Gaze estimation neural network 510...input layer, 520...intermediate layer, 530...output layer 600…Scene information estimation neural network 610...input layer, 620...intermediate layer, 630...output layer 800: Scene acquisition and scene information estimation system, 801: Receiving unit 802: signal processing section, 803: buffer section, 804: output section 805: sensor unit; 806: scene information estimation unit 900…Scene information estimation neural network 910...input layer, 920...intermediate layer, 930...output layer 1900…Display, 1901…Speaker unit 1902: Stand, 1901-1, 1902-2: Shaker 2000…Cloud-based artificial intelligence system 2010…Local environment, 2011…Operational neural network 2020: Cloud; 2021: Operational Neural Networks 2022…Evaluation Neural Networks 2023…Feedback Database 2024…Expert teaching database

Claims

1. an attention level estimation unit that estimates an attention level of a user viewing content by using artificial intelligence based on sensor information; an acquisition unit that acquires a video of a scene in the content that the user is gazing at and information about the content based on an estimation result of the gaze degree estimation unit; a scene information estimation unit that estimates information about the scene that the user is gazing at using artificial intelligence based on a video of the scene that the user is gazing at and information about the content; An artificial intelligence information processing device comprising:

2. The scene information estimation unit estimates information correlated with the scene that the user is gazing at by using a neural network that has learned the correlation between the image of the scene, information about the content, and information about the scene, as an estimation by artificial intelligence. The artificial intelligence information processing device according to claim 1.

3. The gaze degree estimation unit estimates a gaze degree that is correlated with the sensor information related to the user viewing the content by using a neural network that has learned a correlation between the sensor information and the gaze degree of the user as an estimation by artificial intelligence. The artificial intelligence information processing device according to claim 1.

4. The sensor information includes at least a captured image of the user viewing the content, the captured image being captured by a camera, The gaze degree estimation unit estimates a gaze degree that is correlated with the face recognition result in the captured image by using a neural network that has learned a correlation between the face recognition result and the gaze degree of the user as an estimation by artificial intelligence. The artificial intelligence information processing device according to claim 3.

5. The content includes at least one of broadcast content and streamed content. The artificial intelligence information processing device according to claim 1.

6. outputting information about the scene that the user is gazing at to an external device; The artificial intelligence information processing device according to claim 1.

7. presenting to the user a scene that the user is gazing at and information about the scene; The artificial intelligence information processing device according to claim 1.

8. An attention level estimation step of estimating an attention level of a user viewing content by artificial intelligence based on sensor information; an acquisition step of acquiring a video of a scene in the content that the user is gazing at and information about the content based on a result of the estimation step of the gaze degree estimation step; a scene information estimation step of estimating information about the scene that the user is gazing at by artificial intelligence based on a video of the scene that the user is gazing at and information about the content; An artificial intelligence information processing method comprising the steps of:

Images