Control program, control device, and control method, as well as audio-visual quality output program, audio-visual quality output device, and audio-visual quality output method

Abstract

To provide a control program, a control device and a control method, and a view quality output program, a view quality output device and a view quality output method which image a viewing area of a television and perform various processes.SOLUTION: According to one aspect of the present invention, a control device comprises: a viewer detection unit which detects a state of a viewer within a viewing area of a television on the basis of an image obtained from a camera arranged to image the viewing area; and an equipment control unit which controls equipment to be controlled on the basis of the state of the viewer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control program, a control device and a control method, as well as a viewing quality output program, a viewing quality output device and a viewing quality output method. [Background technology]

[0002] Patent document 1 discloses a technology in which a camera is positioned to capture an image of a television viewing area and the number of people present in the viewing area and the number of people engaged with the video in the viewing area are quantified from the obtained image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-507533 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a control program, a control device and a control method for photographing a television viewing area and performing various processes, as well as a viewing quality output program, a viewing quality output device and a viewing quality output method. [Means for solving the problem]

[0005] According to one aspect of the present invention, a control device is provided which includes a viewer detection unit that detects the status of a viewer within a viewing area of ​​a display based on an image from a camera positioned to capture the viewing area, and an equipment control unit that controls a controlled device based on the status of the viewer.

[0006] The display device may further include a display state detection unit that detects a state of the display based on audio information from the display, and the device control unit may control the controlled device based on a state of the viewer and a state of the display.

[0007] The display state detection unit may detect content being played on the display as the state of the display, and the device control unit may control the control target device based on the content being played on the display.

[0008] The device to be controlled may be a mobile terminal of the viewer, and the device control unit may perform control so that information related to the content being reproduced on the display is displayed on the mobile terminal of the viewer.

[0009] The device to be controlled may be an interactive speaker, and the device control unit may perform control such that information relating to content being reproduced on the display is output from the interactive speaker.

[0010] The viewer detection unit may detect, as the state of the viewer, whether or not the viewer is gazing at the display, and the device control unit may control the controlled device based on whether or not the viewer is gazing at the display.

[0011] The device to be controlled may be a mobile terminal of the viewer, the viewer detection unit may detect, as the state of the viewer, whether or not the viewer is looking at the mobile terminal, and the device control unit may control the mobile terminal if the viewer is looking at the mobile terminal.

[0012] According to one aspect of the present invention, a control program is provided that causes a computer to function as a viewer detection unit that detects the status of a viewer within a viewing area of ​​a display based on images from a camera positioned to capture the viewing area, and an equipment control unit that controls equipment to be controlled based on the status of the viewer.

[0013] According to one aspect of the present invention, a control method is provided in which a status of a viewer within a viewing area of ​​a display is detected based on an image from a camera positioned to capture the viewing area, and a controlled device is controlled based on the status of the viewer.

[0014] According to one aspect of the present invention, there is provided a viewing quality output program that causes a computer to function as: setting receiving means for receiving at least one setting of a broadcast station, a period, content, and viewer attributes; acquisition means for acquiring a viewing quality value corresponding to the accepted setting from storage means that stores viewing quality values ​​in association with viewer attributes for each broadcast station and date / time, or for each content; and output control means for outputting information relating to the viewing quality value according to the accepted setting.

[0015] The output control means may cause information relating to the audience quality value to be displayed on a display.

[0016] The output control means may cause the display to display viewing quality information in which days of the week are set in a predetermined direction, a time axis is set in a direction perpendicular to the predetermined direction, and the display mode of positions corresponding to each day of the week and each time period corresponds to information regarding viewing quality values.

[0017] The setting receiving means may receive a setting of a collection time period, and the unit of the time axis may correspond to the collection time period.

[0018] When the received settings include a selection of broadcast stations, the output control means may cause a display to display the audience quality information for each selected broadcast station.

[0019] According to one aspect of the present invention, there is provided a viewing quality output program that causes a computer to function as: acquisition means for acquiring viewing quality values ​​from storage means that stores viewer viewing quality values ​​for each broadcast station and date and time, or for each piece of content; and output control means for outputting viewing quality information in which days of the week are set in a predetermined direction and a time axis is set in a direction perpendicular to the predetermined direction, and the display mode of positions corresponding to each day of the week and each time period corresponds to information related to the viewing quality value.

[0020] The output control means may cause the audience quality information to be displayed on a display.

[0021] The positions corresponding to each day of the week and each time slot may be colored darker the higher the audience quality value, or may be colored in accordance with information relating to the audience quality value.

[0022] The output control means may cause information relating to audience quality values ​​to be displayed at positions corresponding to each day of the week and each time slot.

[0023] The output control means may cause a display to display the audience quality information for each of all broadcast stations.

[0024] According to one aspect of the present invention, there is provided a viewing quality output program that causes a computer to function as: an acquisition means for acquiring a viewing quality value from a storage means that stores a viewer's viewing quality value for each piece of content that is played a number of times; and an output control means for outputting, in a graph, changes in information relating to the viewing quality value for each piece of content that is played a number of times.

[0025] A first axis of the graph may indicate time, and a second axis of the graph may indicate a value related to the audience quality value of the content reproduced at the time indicated by the first axis.

[0026] A first axis of the graph may represent an integrated value of the viewing rate when a particular piece of content is displayed, and a second axis of the graph may represent a value related to the viewing quality value relative to the value indicated by the first axis.

[0027] According to one aspect of the present invention, there is provided a viewing quality output device comprising: a setting receiving means for receiving at least one setting of a broadcast station, a period, content, and viewer attributes; an acquisition means for acquiring a viewing quality value corresponding to the accepted setting from a storage means that stores viewing quality values ​​in association with viewer attributes for each broadcast station and date / time, or for each content; and an output control means for outputting information relating to the viewing quality value according to the accepted setting.

[0028] According to one aspect of the present invention, there is provided a viewing quality output method which accepts at least one setting of a broadcast station, a period, content, and viewer attributes, obtains a viewing quality value corresponding to the accepted setting from a storage means which stores viewing quality values ​​in association with viewer attributes for each broadcast station and date / time, or for each content, and outputs information relating to the viewing quality value according to the accepted setting.

[0029] According to one aspect of the present invention, there is provided a viewing quality output device comprising: acquisition means for acquiring viewing quality values ​​from storage means which stores viewer viewing quality values ​​for each broadcast station and date and time, or for each content; and output control means for outputting viewing quality information in which days of the week are set in a predetermined direction and a time axis is set in a direction perpendicular to the predetermined direction, and in which the display mode at positions corresponding to each day of the week and each time period corresponds to information regarding the viewing quality value.

[0030] According to one aspect of the present invention, there is provided a viewing quality output method which obtains viewing quality values ​​from a storage means which stores viewer viewing quality values ​​for each broadcast station and date and time, or for each piece of content, sets days of the week in a predetermined direction, sets a time axis in a direction perpendicular to the predetermined direction, and outputs viewing quality information in a display mode at a position corresponding to each day of the week and each time slot which corresponds to information relating to the viewing quality value.

[0031] According to one aspect of the present invention, there is provided a viewing quality output device comprising: an acquisition means for acquiring a viewing quality value from a storage means storing a viewer's viewing quality value for each piece of content that is played a number of times; and an output control means for outputting, in a graph, changes in information relating to the viewing quality value for each piece of content that is played a number of times.

[0032] According to one aspect of the present invention, there is provided a viewing quality output method comprising the steps of: acquiring a viewing quality value for each piece of content that is played multiple times from a storage means that stores a viewer's viewing quality value; and outputting, in a graph, changes in information relating to the viewing quality value for each piece of content that is played multiple times. Effect of the Invention

[0033] The television viewing area can be photographed and various processing can be performed. [Brief description of the drawings]

[0034] [Figure 1] FIG. 1 is a diagram illustrating a control system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a schematic configuration of a control system according to a first embodiment. [Figure 3A] 1 is a block diagram showing a schematic configuration of an audience quality acquisition system. [Figure 3B] FIG. 2 is a diagram showing visitor information. [Figure 3C] FIG. 4 is a diagram showing a schematic diagram of viewer information. [Figure 3D] FIG. 2 is a diagram showing television status information. [Figure 4A] FIG. 2 is a diagram showing an example of the structure of a channel and broadcast time database 2. [Figure 4B] FIG. 2 is a diagram showing an example of the structure of a program database 36. [Diagram 5] FIG. 2 shows an example of the structure of a view quality database 37. [Figure 6] FIG. 11 is a block diagram showing a schematic configuration of an audience quality display system according to a second embodiment. [Figure 7A]5 is an example of a screen that is displayed on the display 52 to accept station settings. [Figure 7B] 13 is an example of a screen displayed on the display 52 for accepting a period setting. [Figure 7C] 13 is an example of a screen displayed on the display 52 for accepting attribute settings. [Figure 7D] 13 is an example of a screen displayed on the display 52 for accepting a counting time setting. [Figure 8A] 13 is an example of a screen displaying information indicating a viewing quality value. [Figure 8B] 13 is another example of a screen displaying information indicating a viewing quality value. [Figure 9] 13 is another example of a screen displaying information indicating a viewing quality value. [Figure 10A] An example of a screen displaying viewing quality values ​​as a line graph. [Figure 10B] An example of a screen displaying viewing quality values ​​as a line graph. [Figure 10C] An example of a screen displaying viewing quality values ​​as a line graph. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Hereinafter, an embodiment of the present invention will be specifically described with reference to the drawings.

[0036] (First embodiment) In the first embodiment, a television viewing area is photographed by a camera, and an arbitrary device is controlled based on the obtained image. Fig. 1 is a diagram showing a schematic diagram of a control system according to a first embodiment. The control system includes a camera 21, a microphone 22, and a control device 23. Fig. 1 also shows a television 100 installed in a living room and a viewing area in front of it.

[0037] Camera 21 is positioned so as to capture the viewing area of ​​television 100, such as the frame of television 100 or the ceiling of the living room. Microphone 22 is positioned so as to capture audio from television 100, such as near the speaker of television 100. Control device 23 controls any control target device 24 based on the image captured by camera 21 and / or the audio captured by microphone 22. Note that if control device 23 can directly capture audio information from television 100 (for example, if an earphone jack or audio output terminal of television 100 can be connected to control device 23), microphone 22 does not need to be provided.

[0038] 2 is a block diagram showing a schematic configuration of a control system according to the first embodiment. The control device 23 in this control system includes a communication unit 231, a viewer detection unit 232, a television state detection unit 233, and a device control unit 234. The control device 23 may be a computer having a processor and a memory, and some or all of the functions of these units may be realized by the processor executing a predetermined program stored in the memory.

[0039] The communication unit 231 communicates with the camera 21, the microphone 22, and the control target device 24. There is no particular limit to the type of communication, and it may be wired communication or wireless communication such as short-distance wireless communication or wireless LAN. Specifically, the communication unit 231 receives information from the camera 21 and the microphone 22. In addition, the communication unit 231 transmits information for controlling the camera 21, the microphone 22, and the control target device 24.

[0040] The viewer detection unit 232 detects the state of the viewer in the viewing area based on the image from the camera 21. As a specific processing example, the viewer detection unit 232 identifies the viewer in the viewing area (or that there is no viewer in the viewing area) by performing face recognition from the image from the camera 21. The viewer detection unit 232 may perform human body recognition, which is a part of the object recognition function, extract a face included in the detected human body, and compare it with a face photo registered in advance. The viewer detection unit 232 then detects whether each viewer is gazing at the television 100 or whether each viewer is looking at their own mobile terminal (such as a smartphone or tablet). In this specification, a person in the viewing area of ​​the television (regardless of whether they are actually watching) may be called a "viewer".

[0041] Furthermore, the viewer detection unit 232 may detect the attributes (gender, age, etc.) and facial expressions such as joy, anger, sadness, and happiness of each identified viewer. In addition, the viewer detection unit 232 may detect the number of viewers in the viewing area, the time each viewer is gazing at the television 100, the distance between the television 100 and the viewer, etc. as the state of the viewer. The viewer detection unit 232 may perform the detection process using artificial intelligence, or may perform the detection process using a known method.

[0042] The television state detection unit 233 detects the state of the television 100 based on the audio information from the television 100. The state of the television 100 includes whether the television 100 is on or off, which content is being played, and the like. The content may be a program, a commercial, or a specific scene (such as a certain incident or accident), and the unit of the content is arbitrary. The content here may be something currently being broadcast (real time) or something that has been recorded. As the content, the content may be specified, or the genre of the content (sports, movies, news, etc.) may be detected. The content is not limited to that provided by broadcast waves, and includes content provided via the Internet (in this case, it may be a simple display, not the television 100). An example of a method for specifying the content being played on the television 100 will be described in the second embodiment.

[0043] The device control unit 234 controls the control target device 24 based on the detection results by the viewer detection unit 232 and / or the television state detection unit 233. More specifically, the device control unit 234 generates control information for controlling the control target device 24 and controls the control target device 24 by transmitting the control information via the communication unit 231. The control target device 24 may be, for example, the television 100 itself, a speaker other than the television, a light (in the living room where the television is installed or in another room), a mobile terminal of a viewer, an interactive speaker (a so-called "smart speaker" having an AI assistant function), or the like.

[0044] Here, the device control unit 234 may directly control the control target device 24 (such as a television, speaker, or light installed in the home) that is not network-connected to an external device by transmitting a control signal via short-range wireless communication or a home LAN. Also, for the control target device 24 (such as a mobile terminal or an interactive speaker) that is network-connected to an external device such as a server, the device control unit 234 may indirectly control the control target device 24 by transmitting a control signal to the external device such as the server via an Internet line (i.e., the external device controls the control target device 24 in response to a control signal from the device control unit 234).

[0045] In addition, in order to control the mobile terminal of an identified viewer, it is desirable for the device control unit 234 to previously grasp the correspondence between a viewer who can be identified by the viewer detection unit 232 (e.g., a father or a mother in a household where a television is installed) and the mobile terminal used by that viewer.

[0046] As described above, the camera 21 can be used to detect viewers in the viewing area, and the microphone 22 can be used to detect the state of the television 100. Therefore, the device control unit 234 may use both of them to control the control-target device 24. Alternatively, the device control unit 234 may use only one of them to control the control-target device 24. Specific examples of control by the device control unit 234 will be given below.

[0047] For example, if the television 100 is on even though there is no viewer in the viewing area, the device control unit 234 may turn off the television 100 or turn off the lights in the room in which the television 100 is installed. Alternatively, if a state in which there is no viewer in the viewing area changes to a state in which there is a viewer in the viewing area, the device control unit 234 may turn on the television 100. Also, if a state in which there is no viewer in the viewing area changes to a state in which there is a viewer in the viewing area and the viewer is gazing at the television 100, the device control unit 234 may turn on the television 100.

[0048] Furthermore, the device control unit 234 may control the content of the television 100. For example, when the viewer is not watching the television 100, an image for attracting the viewer's attention may be displayed on at least a part of the display of the television 100, or sound may be output from the speaker of the television 100. Such images and sounds may be predetermined or may be tailored to the attributes and preferences of the viewer.

[0049] Furthermore, the device control unit 234 may control the control target device 24 based on the content being played on the television 100. As a specific example, the device control unit 234 may cause the interactive speaker to output information about the content, or cause the content to be displayed on the viewer's mobile terminal. More specifically, the device control unit 234 transmits information indicating the content being played to an external server connected to the mobile terminal or the interactive speaker. As a result, under control of the external server, information about the content is recommended from the interactive speaker, or an advertisement related to the content is displayed on the viewer's mobile terminal.

[0050] Furthermore, the device control unit 234 may control a mobile terminal when the viewer is watching the mobile terminal. Alternatively, when the viewer is not watching the television, the device control unit 234 may control the mobile terminal, assuming that the viewer is likely watching the mobile terminal. As a specific example, the device control unit 234 may display information about a commercial being played on the television in real time on the mobile terminal. This allows the viewer to see an advertisement with the same content as or related to the commercial being played on the television, even if the viewer is not watching the commercial.

[0051] As described above, in the first embodiment, the viewing area of ​​the television 100 is photographed by the camera 21, and the control target device 24 is controlled based on the obtained image. Therefore, the control target device 24 can be appropriately controlled according to various conditions of the viewing area, such as whether or not there are viewers in the viewing area, and if there are viewers, what kind of viewers there are, and whether or not they are watching television. Furthermore, in the first embodiment, by controlling the control target device 24 in consideration of the state of the television 100 acquired from the sound of the television 100, more diverse control is possible.

[0052] Second embodiment Traditionally, the "household viewership rating" has been widely used as a measure of the percentage of households that watch a particular television program. However, the household viewership rating only corresponds to the percentage of households that have the television program tuned in, and does not necessarily correlate with the percentage of households that watch the television program.

[0053] In recent years, viewer ratings are also obtained on an individual basis, rather than on a household ratio basis. For individual viewer ratings, a specific device is provided with buttons corresponding to each member of a family, and when a person watching television selects the button corresponding to himself or herself, it is recorded as having watched a program. However, even with regard to individual viewer ratings, it is not necessarily the case that the person is actually watching television, and there is a possibility that they just leave the television on for some reason.

[0054] In contrast, the above-mentioned Patent Document 1 focuses on the "quality" of viewing and discloses a technique for quantifying the number of people present in the viewing area in front of the display and the number of people engaged with the video in the viewing area. However, Patent Document 1 merely discloses quantifying these numbers, and does not give any consideration to how to present them to the user.

[0055] The second embodiment described next is intended to make it possible to present audience quality in a more easily understandable manner.

[0056] In the second embodiment, viewing quality is quantified as a viewing quality value. As examples of viewing quality, an index of how many viewers were able to watch television (Viewability Index, hereinafter referred to as the "VI value") and an index of how many viewers were paying attention to television (Attention Index, hereinafter referred to as the "AI value") are used. The VI value indicates the degree to which viewers stayed, and the higher the VI value, the more viewers were in front of a television set. On the other hand, the AI ​​value indicates the degree to which viewers paid attention, and the higher the AI ​​value, the more viewers paid attention to the television content at that audience rating.

[0057] First, an example of a method for obtaining VI and AI values ​​will be described. 3A is a block diagram showing a schematic configuration of a viewing quality acquisition system. The viewing quality acquisition system comprises a viewing information acquisition device 1, a channel and broadcast time database 2, and a viewing quality management device 3. It is assumed that a viewing information acquisition device 1 is installed in each household, and information from the viewing information acquisition device 1 in each household is aggregated by the viewing quality management device 3.

[0058] The viewing information acquisition device 1 is installed in each household. The people (household members) who belong to each household are determined in advance. In the following description, it is assumed that the household in which the viewing information acquisition device 1 is installed includes three household members A, B, and C.

[0059] The viewing information acquisition device 1 has a camera 11, an IR sensor 12, a viewer detection unit 13, a microphone 14, a television state detection unit 15, and a transmission unit 16.

[0060] The camera 11 captures an image of the television viewing area (in front of the television) and supplies the camera image to a viewer detection unit 13.

[0061] The IR sensor 12 has a light-emitting unit that irradiates the viewing area with infrared rays and a light-receiving unit that detects infrared rays reflected from an object (particularly, a viewer). The IR sensor 12 generates a depth image indicating the distance between the light-emitting unit and the object based on the infrared rays detected by the light-receiving unit, and supplies the depth image to the viewer detection unit 13.

[0062] The viewer detection unit 13 includes a visitor detection unit 131 and a gazing person detection unit 132 .

[0063] The visitor detection unit 131 detects a visitor in the viewing area by performing face recognition from the camera image and the depth image, or based on the walking style, body type, etc. More specifically, the visitor detection unit 131 identifies which of the household members A to C the visitor is. Then, the visitor detection unit 131 generates visitor information indicating which viewer was visiting at each time.

[0064] FIG. 3B shows a schematic diagram of visitor information. As shown in the figure, the visitor information indicates whether or not each of household members A to C was in the viewing area at each time. A circle in the figure indicates that they were present, and a cross indicates that they were not present. For example, household member A was in the viewing area at 19:00:00 on January 3, 2020, 19:00:05 on the same day, and 19:00:10 on the same day.

[0065] The visitor information can be said to indicate the number of visitors at each time. Note that, although visitors are detected every 5 seconds in Figure 3B, there are no particular limitations on the unit.

[0066] The gazing person detection unit 132 detects people gazing at the television by analyzing the gestures, movements, face direction, and gaze movements of the viewers from the camera image and the depth image. More specifically, the gazing person detection unit 132 identifies which of the household members A to C the gazing person is. Then, the gazing person detection unit 132 generates gazing person information indicating which viewer was gazing at the television at each time.

[0067] FIG. 3C shows a schematic diagram of the gazer information. As shown in the figure, the gazer information indicates whether or not each of household members A to C was gazing at the television at each time. A circle in the figure indicates that they were gazing, and a cross indicates that they were not gazing. For example, household member A was gazing at the television at 19:00:00 and 19:00:10 on January 3, 2020, but was not gazing at the television at 19:00:05 on the same day.

[0068] The gazer information can be said to indicate the number of gazers at each time. Note that, although gazers are detected every 5 seconds in Fig. 3B, there is no particular limit to the unit.

[0069] Returning to FIG. 3A, the microphone 14 collects the sound reproduced from the television and supplies it to the television state detection unit 15.

[0070] The television status detection unit 15 acquires the status of the television based on audio information from the television. Specifically, the television status detection unit 15 has a fingerprint generation unit 151 and a channel / broadcast time information acquisition unit 152.

[0071] Fingerprint generation unit 151 processes the audio from microphone 14 and generates an audio fingerprint (hereinafter simply referred to as a “fingerprint”) F.

[0072] Channel and broadcast time information acquisition unit 152 acquires the state of the television at each time based on channel and broadcast time database 2 that stores the correspondence between fingerprint F and channels (broadcast stations) and broadcast times, and generates television state information. The television state information indicates the broadcast station and broadcast date and time of the content that was displayed on the television at each time.

[0073] As a specific processing example, channel and broadcast time information acquisition unit 152 transmits the generated fingerprint F to a service having channel and broadcast time database 2. Then, channel and broadcast time information acquisition unit 152 receives information indicating the channel and broadcast time identified by the service based on fingerprint F.

[0074] Fig. 4A is a diagram showing an example of the structure of channel / broadcast time database 2. As shown in the figure, fingerprints are stored in association with broadcast stations, broadcast dates and times, and viewing dates and times. For example, if fingerprint F2 was generated by fingerprint generation unit 151 in Fig. 3A at a specific time, it can be seen that the program being played on the television was a program broadcast by broadcast station Q from 19:00:00 to 19:00:05 on January 3, 2020.

[0075] Television status information is shown diagrammatically in Figure 3D. For example, at 19:00:00 on January 3, 2020 and 19:00:05 on the same day, the television is displaying the content that was broadcast by broadcasting station X at 19:00:00 on January 3, 2020 and 19:00:05 on the same day, respectively. Note that, although Figure 3D detects the television status in 5-second intervals, there are no particular limitations on the unit.

[0076] Returning to Fig. 3A, transmission unit 16 of viewing information acquisition device 1 transmits visitor information (Fig. 3B), gazer information (Fig. 3C), and television state information (Fig. 3D) to viewing quality management device 3. Similar information is transmitted to viewing quality management device 3 from each viewing information acquisition device 1.

[0077] 3A does not necessarily require all of the components shown in the figure. For example, the IR sensor 12 may be omitted, and the gazing person detection unit 132 may detect a gazing person only from the camera image.

[0078] Next, we will describe audience quality management device 3. Viewing quality management device 3 has an attribute detection section 31, a program information acquisition section 32, a VI value calculation section 33, an AI value calculation section 34, a database writing section 35, a program database 36, and an audience quality database 37.

[0079] The attribute detection unit 31 detects the attributes of the viewers. Specifically, the attribute detection unit 31 sets the attributes (gender, age, annual income, etc.) of the household members of the household in which the viewing information acquisition device 1 is installed in association with each viewing information acquisition device 1. For example, the attributes of the above-mentioned three household members A to C are set. Then, the attribute detection unit 31 specifies the attributes of the viewers who stayed in the viewing area at each time and the viewers who watched the television at each time, based on the visitor information and the gazing person information from the viewing information acquisition device 1. As for the age and gender, it may be specified as being one of the categories (C, T, M1 to M3, F1 to F3) used in marketing, for example.

[0080] The program information acquisition unit 32 refers to a program database 36 that stores the correspondence between channels (broadcast stations), broadcast times, and programs, and identifies the programs that were broadcast on the channel (broadcast station) and at the broadcast time indicated by the television status information.

[0081] 4B is a diagram showing an example of the structure of the program database 36. As shown in the figure, broadcasting stations and broadcasting times are stored in association with programs broadcast by each broadcasting station at each broadcasting time. For example, the program broadcast by broadcasting station Q from 7:00 p.m. to 9:00 p.m. on January 3, 2020 is professional baseball game B.

[0082] In this way, television state detection unit 15 and program information acquisition unit 32 work together to acquire the content being played on the television. Depending on the structure of the database, the program may be directly identified from fingerprint F without identifying the channel and broadcast time. Alternatively, only the channel and broadcast time may be identified.

[0083] According to the technique of identifying a program using the fingerprint F, not only programs that are being broadcast in real time but also programs that have been recorded and are being viewed using time-shifted viewing can be identified.

[0084] Note that, while FIG. 4B shows an example in which content is stored in program database 36 on a content-by-content basis, content may also be stored in program database 36 on a finer scale than content (so-called program, content in the narrow sense) (for example, a commercial or a specific scene in the content (content in the broad sense)).

[0085] Returning to Fig. 3A, VI value calculation unit 33 counts the number of visitors based on visitor information from each viewing information acquisition device 1, and calculates a VI value for each broadcast station and date and time (or for each content, the same applies below) in association with viewer attributes. AI value calculation unit 34 counts the number of viewers based on viewer information from each viewing information acquisition device 1, and calculates an AI value in association with viewer attributes for each broadcast station and date and time. Database writing unit 35 writes the VI value and AI value into viewing quality database 37 in association with viewer attributes for each broadcast station and date and time.

[0086] 5 is a diagram showing an example of the structure of audience quality database 37. In the example shown, for each program (broadcast date and time and broadcast station), the VI value and AI value are stored in association with the viewer's attributes (here, gender, age, and income). For example, for "News A" broadcast on broadcast station P from 7:00 to 7:30 p.m. on January 3, 2020, the AI ​​value for viewers in the M1 demographic with an income of 3 million yen or less is 0.2, and the VI value is 0.15.

[0087] In this manner, audience quality value information is stored in audience quality database 37. Note that the above is merely an example, and any appropriate modification is possible as long as audience quality values ​​can be stored for each broadcast station and date and time. For example, one or two of gender, age, and annual income may be used as attributes, other elements may be included, or attributes may be omitted. Indices other than VI and AI values ​​may also be applied as audience quality values.

[0088] Next, the display of audiovisual quality will be described. 6 is a block diagram showing a schematic configuration of an audience quality display system according to the second embodiment. The audience quality display system comprises an audience quality display device 5 and the audience quality database 37 of FIG.

[0089] Audience quality display device 5 may be any computer, such as a personal computer, a smartphone, or a tablet, and has input interface 51, display 52, setting acceptance section 53, audience quality acquisition section 54, and output control section 55. At least some of the functions of setting acceptance section 53, audience quality acquisition section 54, and output control section 55 may be realized by a processor of audience quality display device 5 executing a predetermined program.

[0090] Setting reception unit 53 receives a setting for what audience quality to display (i.e., the target of the audience quality to be displayed) from the user via input interface 51. This setting may include at least one of the broadcast station, period, program, viewer attributes, and aggregation time, for example. Viewing quality acquisition unit 54 accesses audience quality database 37 and acquires an audience quality value according to the setting received by setting reception unit 53. Output control unit 55 outputs a screen for receiving the above settings and information indicating the audience quality value acquired by audience quality acquisition unit 54.

[0091] Output control section 55 may display information indicating the audience quality value on display 52, may print it out on a printer (not shown), may generate and output an electronic file such as CSV, or may output it externally as electronic data using an API. The following mainly describes an example in which output control section 55 displays information indicating the audience quality value on display 52, but similar printing may be performed, or an electronic file or electronic data for displaying a similar screen may be output.

[0092] 7A is an example of a screen displayed on display 52 for accepting broadcast station settings. As shown in the figure, a check box is provided for each broadcast station, and one or more check boxes can be checked. Via input interface 51, the user selects a broadcast station by checking the check box of the broadcast station for which the user wants to display the audience quality value. Audience quality acquisition unit 54 acquires the audience quality value for the programs of the selected broadcast station.

[0093] Incidentally, the setting of a broadcast station is arbitrary, and audience quality display device 5 may display audience quality values ​​of all broadcast stations available for broadcasting in a specific area without setting a broadcast station.

[0094] Fig. 7B is an example of a screen displayed on display 52 to accept a period setting. Fig. 7B is an example of specifying a period in one-week units, and provides fields for inputting the year, month, and week for which viewing quality values ​​are to be displayed. The user specifies the period by inputting the period for which viewing quality is to be displayed via input interface 51. Viewing quality acquisition section 54 acquires viewing quality values ​​for the specified period.

[0095] There is no limit to the method of setting the period, and the start and end periods may be specified by date or time, and the unit of the period is not limited to one week, but may be a specific day or month. A calendar or clock may be displayed to allow the start and end periods of the period to be selected. Furthermore, the period may be set arbitrarily, and audience quality display device 5 may display audience quality values ​​for a predetermined period, such as the most recent week, without setting the period.

[0096] The setting reception unit 53 may receive a selection on a program-by-program basis instead of specifying a broadcast station and a period. Also, the setting reception unit 53 may receive a selection not only on an individual program basis but also on a genre basis, such as news or sports.

[0097] FIG. 7C is an example of a screen displayed on the display 52 for accepting attribute settings.

[0098] As shown in Fig. 7C(a), radio buttons are provided for each of "Male," "Female," and "Both" in relation to the setting of the viewer's gender, and only one radio button can be selected exclusively. Via input interface 51, the user selects the gender for which he or she wishes to display the audience quality value. Viewing quality acquisition unit 54 acquires the audience quality value for the selected gender (or for any gender if "Both" is selected).

[0099] As shown in FIG. 7C(b), for selection of the viewer's age, a check box is provided for each age group, and one or more check boxes can be checked. Via input interface 51, the user specifies the age group by checking the check box for the age group for which the viewer wants to display the viewer quality value. Viewer quality acquisition unit 54 acquires the viewer quality value for the specified age group. Note that the user may be able to select the age arbitrarily in increments of one year.

[0100] It should be noted that the gender and age group may be set simultaneously in categories such as C, T, M1 to M3, and F1 to F3.

[0101] As shown in Fig. 7C(c), for setting the income of a viewer, a check box is provided for each income tier, and one or more check boxes can be checked. Via input interface 51, the user specifies the income tier by checking the check box for the income tier for which the viewer wishes to display the viewer quality value. Viewer quality acquisition unit 54 acquires the viewer quality value for the specified income tier.

[0102] Incidentally, setting of attributes is optional, and audience quality display device 5 may display audience quality values ​​of all viewers without setting attributes. Also, one or two of gender, age, and income may be able to be set, and other attributes may be able to be set depending on the configuration of audience quality database 37.

[0103] FIG. 7D is an example of a screen displayed on display 52 for accepting the setting of the tally time. As shown in the figure, a field for inputting the tally time is provided. Via input interface 51, the user inputs the time that is the unit for tallying audience quality values. Viewer quality acquisition unit 54 acquires audience quality values ​​and tallys up the audience quality values ​​in the set tally time unit. For example, if the tally time is one hour, audience quality acquisition unit 54 tallys up audience quality values ​​in one-hour units, and if the tally time is one day, audience quality acquisition unit 54 tallys up audience quality values ​​in one-day units. The unit of tally is arbitrary, and may be in one-minute units, ten-minute units, one-hour units, or one-day units.

[0104] It should be noted that the tallying time can be set arbitrarily, and audience quality display device 5 may tally audience quality values ​​over a predetermined tallying time, such as two hours, without setting a tallying time.

[0105] 8A is an example of a screen displaying information indicating audience quality values. In the figure, it is assumed that the setting reception unit 53 has set broadcast station P, the first week of March 2020, and one hour as the aggregation time. In addition, it is assumed that M2 (male, aged 35 to 49) is specified as the attribute, and for convenience, the audience quality values ​​(here, VI values) are shown for only some dates and times.

[0106] As audience quality information 6, days of the week are set horizontally, and a time axis is set vertically. The time axis is in units corresponding to the counting time (in this example, the counting time is one hour, so in units of one hour). The display mode of the position corresponding to each day of the week and each time slot corresponds to the audience quality value for that day of the week and time slot.

[0107] As a display format, the audience quality value itself may be displayed at a position corresponding to each day of the week and each time slot. For example, in Fig. 8A, the audience quality value for Monday at 5:00 is 0.3, and the audience quality value for Tuesday at 5:00 is 0.35 (other days of the week and other time slots are not shown).

[0108] Furthermore, as a display mode, the color or color intensity of the position (cell) corresponding to each day of the week and each time slot may correspond to the audience quality value. As a specific example, the higher the audience quality value, the darker the color. There may be a one-to-one correspondence between the audience quality value and the display mode, or a specific display mode may be used when the audience quality value is within a certain range (for example, the color becomes one level darker for every 0.1 increase in the audience quality value).

[0109] In Fig. 8A, for the sake of convenience, the higher the audience quality value, the denser the spots are. This type of display allows the viewer to intuitively grasp the overview of the audience quality value for each day of the week and each time slot. Fig. 8A shows that the audience quality is generally high after 7pm when people return home from work, followed by 7-8am on weekends and before work.

[0110] Fig. 8B is another example of a screen displaying information indicating audience quality values. The difference from Fig. 8A is that F2 (female, age 35-49) is specified as the attribute. Unlike Fig. 8A where M2 is specified as the attribute, it can be seen that the audience quality is high during the daytime.

[0111] In this way, by setting attributes, the user can grasp the trends in audience quality according to the attributes. For example, when considering placing a TV ad for a product targeted at middle-aged men, it is recommended to display the audience quality information 6 in Fig. 8A. On the other hand, when considering placing a TV ad for a product targeted at middle-aged women, it is recommended to display the audience quality information 6 in Fig. 8B.

[0112] When multiple broadcast stations are selected, output control unit 55 displays audience quality information 6 for each of them, as shown in Fig. 8. When there are not many broadcast stations, output control unit 55 may display audience quality information 6 for all broadcast stations (that can broadcast in a specific area).

[0113] Fig. 9 is another example of a screen displaying information indicating audience quality values. The difference with audience quality information 6 in Fig. 8 is that the days of the week set in the horizontal direction are divided into two sections: weekdays and Saturday and Sunday. That is, a value obtained by summarizing the audience quality values ​​from Monday to Friday (for example, an average value) is set as the weekday audience quality value, and a value obtained by summarizing the audience quality values ​​from Saturday and Sunday (for example, an average value) is set as the Saturday and Sunday audience quality value. Also, the time axis set in the vertical direction does not have to be at equal intervals.

[0114] As explained above, the user sets the target for which the audience quality value is to be displayed, and output control unit 55 displays audience quality information 6 in accordance with that setting. This makes it possible for the user to display the audience quality value that he or she desires. Also, in this embodiment, output control unit 55 displays the audience quality values ​​in a matrix for each day of the week and time slot, making it easy for the user to grasp the audience quality value for each day of the week and each time slot.

[0115] It should be noted that audience quality values ​​for a plurality of broadcasting stations or periods may be collected and displayed. For example, audience quality values ​​collected for a specific morning information program over a specified period may be displayed. Or, audience quality values ​​collected across broadcasting stations for programs of a particular genre may be displayed. Settings for the type of collection to be performed can be accepted from the user.

[0116] Next, a description will be given of a mode in which the transition of the audience quality of a content that is played multiple times is displayed in a line graph. Here, the content that is played multiple times can be, for example, a repeatedly broadcast commercial, a serial drama, a rebroadcast program, etc., but in the following, it is assumed to be a commercial.

[0117] Fig. 10A is an example of a screen displaying audience quality as a line graph. The solid line indicates commercials for drink A, the dashed line indicates commercials for drink B, and the dotted line indicates the average of these. The horizontal axis of the line graph indicates time (in Fig. 10A, in units of one month), and one month is the unit of time in Fig. 10A. The vertical axis of the line graph indicates values ​​related to the audience quality value (in Fig. 10A, the product of the VI value and the AI ​​value) of the content played at the time indicated by the corresponding horizontal axis.

[0118] For example, arrow F in Fig. 10A is the product of the VI and AI values ​​of a commercial for drink A played in June. If a commercial for drink A was played multiple times in the same month, the average value of each time may be used, or the cumulative value may be used.

[0119] Such a line graph is displayed by output control section 55 by storing the audience quality values ​​of television viewers for each piece of content that is played multiple times in audience quality database 37 (FIG. 6) and acquiring the necessary audience quality information by audience quality acquisition section 54. Note that "for each piece of content" here may refer to each playback, or may refer to a total over a specific period (e.g., one day, one week, one month, etc.).

[0120] The time unit on the horizontal axis is not limited to one month, but may be one week or one year. Furthermore, the setting reception section 53 may receive a time unit setting from the user via the input interface 51, and the output control section 55 may set the horizontal axis according to the received time unit. Furthermore, the audience quality value on the vertical axis may be a total value obtained by collectively collecting data for a plurality of broadcasting stations or periods.

[0121] Furthermore, the setting reception unit 53 may receive a designation of any part in the line graph from the user via the input interface 51, and the output control unit 55 may display detailed information corresponding to the designated part.

[0122] For example, as shown in Fig. 10B, when the user designates April (for example, when the user moves the cursor to an area on display 52 corresponding to April), output control unit 55 may display numerically the audience quality values ​​of beverages A and B. Also, as shown in Fig. 10C, when the user designates beverage A (for example, when the user moves the cursor to an area on display 52 corresponding to beverage A), output control unit 55 may display numerically the audience quality values ​​of beverage A for each month.

[0123] Additionally, the output control unit 55 may display the commercial, the program in which the commercial was played, and the time at which the commercial was played. For example, if the user specifies September for beverage A, the program and time at which the commercial for beverage A was played may be displayed. This type of display allows the user to infer the reason why the audience quality value was high or low.

[0124] In this way, by displaying the audience quality values ​​as a line graph, it becomes possible to grasp the transition of the audience quality values.

[0125] In addition, when the viewing quality is stored in association with an attribute as shown in Fig. 5, it is also possible to track the viewing trends of viewers with specific attributes. For example, by tracking the content with a high viewing quality value for the M1 demographic, it is possible to understand the content that is recommended to the M1 demographic.

[0126] In addition, as attributes, the hobbies and preferences of each viewer can be obtained by conducting a questionnaire survey of each viewer in advance. In this case, it is also possible to track the viewing trends of viewers with specific hobbies and preferences. For example, by tracking commercials with high viewing quality values ​​for smokers, it is possible to understand what brands smokers are paying attention to.

[0127] In the above-mentioned second embodiment, the VI value and the AI ​​value are mainly exemplified as the audience quality, but other indices may be used. For example, at least one of the indices regarding the number of viewers (unique reach number), the attention rate (unique reach rate), the cumulative number of attentions (Gross Attention), the cumulative attention rate, the attention content rate, and the number of attentions may be used as the audience quality. Also, as in the first embodiment, instead of a television, a display provided via the Internet may be used.

[0128] The number of viewers is the total number of people who have gazed at a particular content. For example, the gaze rate is the ratio of the number of viewers to a reference value (for example, the total number of panels or the number of people with a particular attribute). Here, although people who have gazed at a particular content for even a very short time may be included in the number of viewers, it is preferable to include only people who have gazed at the particular content for a predetermined time or more in the number of viewers. For example, the predetermined time may be set to 3 seconds when the content is a CM of about 15 seconds. This is because, according to the findings of the inventors of the present application, recognition of the CM increases when the gaze time is 3 seconds or more. This predetermined time may be a fixed value, a value according to the length of the content, or may be arbitrarily set by the user. Furthermore, a person may be included in the number of viewers when he or she has gazed at a particular content a predetermined number of times or more, and this predetermined number of times may be arbitrarily set by the user.

[0129] For example, if the above-mentioned predetermined time is 3 seconds, and the reference value is 2,000 people, and 100 people gaze at a particular piece of content for 3 seconds or more, the number of people gazing is 100, and the gaze rate is 5% (100 / 2,0000).

[0130] The cumulative attention rate is a value obtained by accumulating the attention rate of each content in a content that is played multiple times. The attention content rate is a cumulative attention rate for a value obtained by accumulating the viewer ratings of each content in a content that is played multiple times. Content that is played multiple times includes, for example, repeatedly broadcasted commercials, serial dramas, rebroadcast programs, etc.

[0131] For example, suppose a certain commercial is broadcast three times, with viewer ratings of 10%, 15%, and 20%, respectively, and attention rates of 5%, 6%, and 7%, respectively. In this case, the cumulative attention rate is 18 (= 5 + 6 + 7)%, and the attention content rate is 60 (= (5 + 6 + 7) / (10 + 15 + 20))%.

[0132] The index for the number of fixations is also known as "frequency." Frequency refers to the number or percentage of people who have fixed a certain number of times (e.g., three times) or more, with one fixation being defined as watching a specific piece of content (typically a commercial) for a certain period of time (e.g., three seconds). There is a view that commercials do not lead to recognition unless they are viewed a certain number of times, but frequency can be an index to measure the extent to which a commercial is recognized.

[0133] To obtain the gaze rate, cumulative gaze rate, gaze content rate, and frequency, for example, the gazer counting unit 132 in the viewer detection unit 13 in Fig. 3A measures the time each viewer spends gazing at the television and transmits the gaze time of each viewer to the viewing information management device 3. Then, a gaze information calculation unit is provided in the viewing information management device 3 in Fig. 3A, and the gaze rate, cumulative gaze rate, and gaze content rate can be calculated based on the gaze time of each viewer.

[0134] The gaze rate, cumulative gaze rate, gaze content rate, and frequency can be said to be the audience quality related to the number of viewers who gazed at the content for a predetermined time or more. Such audience quality can also be presented to the user through an interface similar to that of Figures 7A to 7D, as in Figures 8A, 8B, and 9.

[0135] Other examples of the audience quality value include the product of the AI ​​value and the VI value, the ratio of the AI ​​value or the VI value to a reference value, and the sum of household audience ratings.

[0136] Furthermore, although the horizontal axis in FIG. 10A and FIG. 10B is time, other parameters may be used as the horizontal axis.

[0137] For example, output control unit 55 may display a line graph with GRP (a value obtained by accumulating the viewing ratings when specific content, etc. is displayed (cumulative viewing rating)) on the horizontal axis and viewing quality value (e.g. AI value) on the vertical axis. Output control unit 55 may also display a line graph with the number of encounters (the number of times a specific piece of content was encountered) on the horizontal axis and the viewing quality value on the vertical axis. Alternatively, output control unit 55 may display a line graph with elapsed time (the amount of time elapsed since the start of specific content) on the horizontal axis and the viewing quality value on the vertical axis. In this way, the horizontal axis can be set arbitrarily and changes in information relating to viewing quality value can be displayed in a graph.

[0138] Note that a part or all of the first embodiment described above may be incorporated into the second embodiment, and a part or all of the second embodiment may be incorporated into the first embodiment. Also, in the above-mentioned embodiments, processing for each program has been mainly described, but processing may be performed in any unit as content, such as for each commercial or for each specific scene.

[0139] The above-described embodiments have been described for the purpose of enabling a person having ordinary skill in the art to practice the present invention. Various modifications of the above-described embodiments are naturally possible for a person skilled in the art, and the technical idea of ​​the present invention can be applied to other embodiments. Therefore, the present invention is not limited to the described embodiments, but should be accorded the broadest scope in accordance with the technical idea defined by the claims. [Explanation of symbols]

[0140] 21 Camera 22. Mike 23 Control device 231 Communications Department 232 Viewer Detection Unit 233 TV Status Detection Unit 234 Equipment Control Unit 24 Controlled Equipment 1. Viewing information acquisition device 11 Camera 12 IR Sensor 13 Viewer detection unit 131 Visitor detection unit 132 Gazing Person Detection Unit 14. Mike 15 TV status detection unit 151 Fingerprint Generation Unit 152 Channel and broadcast time information acquisition unit 16 Transmitter 2 Channel and broadcast time database 3. Viewing quality management device 31 Attribute detection section 32 Program Information Acquisition Unit 33 VI value calculation section 34 AI value calculation section 35 Database writing section 36 Program Database 37 Viewing Quality Database 5. Audio quality display device 51 Input Interface 52 Display 53 Settings reception section 54 Viewing Quality Acquisition Department 55 Output control section 6. Viewing quality information

Claims

1. a viewer detection unit that detects the state of a viewer in a viewing area of ​​a display based on an image from a camera that is arranged to capture the viewing area; a display state detection unit that detects a state of the display based on audio information from the display; a device control unit that controls a control target device based on a state of the viewer and a state of the display.

2. the display state detection unit detects content being played on the display as a state of the display; The control device according to claim 1 , wherein the device control unit controls the control target device based on content being reproduced on the display.

3. the control target device is a mobile terminal of the viewer, The control device according to claim 2 , wherein the device control unit performs control so that information related to the content being reproduced on the display is displayed on a mobile terminal of the viewer.

4. the control target device is an interactive speaker, The control device according to claim 2 , wherein the device control unit performs control so that information relating to the content being reproduced on the display is output from the interactive speaker.

5. the viewer detection unit detects, as the state of the viewer, whether the viewer is gazing at the display; The control device according to claim 1 , wherein the device control unit controls the control target device based on whether the viewer is gazing at the display.

6. A control device for controlling a mobile terminal of a viewer, a viewer detection unit that detects whether or not the viewer in the viewing area is looking at a mobile terminal based on an image from a camera arranged to capture the viewing area of ​​a display; A control device comprising: a device control unit that controls the mobile terminal when the viewer is watching the mobile terminal.

7. Computer, a viewer detection unit that detects the state of a viewer in a viewing area of ​​a display based on an image from a camera that is arranged to capture the viewing area; a display state detection unit that detects a state of the display based on audio information from the display; a control program that causes the device to function as a device control unit that controls a control target device based on the state of the viewer and the state of the display.

8. A computer that controls a viewer's mobile terminal, a viewer detection unit that detects whether or not the viewer in the viewing area is looking at a mobile terminal based on an image from a camera arranged to capture the viewing area of ​​a display; a control program that causes the portable terminal to function as a device control unit that controls the portable terminal when the viewer is watching the portable terminal;

9. Detecting the state of viewers in a viewing area of ​​a display based on an image from a camera arranged to capture the viewing area; Detecting a state of the display based on audio information from the display; A control method for controlling a control target device based on a state of the viewer and a state of the display.

10. A control method for controlling a viewer's mobile terminal, comprising: Detecting whether the viewer in the viewing area is looking at the mobile terminal based on an image from a camera arranged to capture the viewing area of ​​the display; A control method for controlling a mobile terminal when the viewer is watching the mobile terminal.

Images