Monitoring device and monitoring method

Abstract

The present invention is provided with: an acquisition unit (102B) that acquires visual recognition data that can grasp positions on a display screen of a display device, said positions being visually recognized by a plurality of viewers; and a determination unit (104B) that obtains a determination result as to whether or not there is a failure in the display device by inputting the visual recognition data acquired by the acquisition unit into the learning completion model. The learning completion model uses visual recognition data which can grasp positions in a display screen of the display device visually recognized by a plurality of viewers, and data indicating whether or not there is a defect in the display device. And a model obtained by learning a condition indicating the relationship between the visual recognition data and whether or not there is a defect in the display device.

Description

Technical Field

[0001] The present invention relates to a monitoring device and a monitoring method. Background Art

[0002] In public facilities and stores, there are various solutions for displaying advertisements and other notifications using image display devices such as monitors and projectors (hereinafter referred to as displays). In this case, if a malfunction occurs in the display device or a malfunction occurs in a supply device that supplies input signals to the display device, the desired image may not be displayed correctly.

[0003] If the image cannot be displayed correctly, it is disadvantageous to the display user (hereinafter referred to as user) such as the advertiser. Therefore, it is necessary to detect whether the display content of the display has a problem. If a problem occurs, it is necessary to immediately urge the replacement and equipment confirmation. However, it is costly to go to the site where the display is installed to confirm the display status. Therefore, there is a demand for users and managers to use remote applications to understand whether there is an abnormality in the display content.

[0004] Here, there is a technology for detecting the line of sight of a viewer viewing a display (see, for example, Patent Document 1).

[0005] [Prior art literature]

[0006] [Patent Document]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2009-244949 Summary of the invention

[0008] [Problems to be Solved by the Invention]

[0009] However, in the case of monitoring the display state of a display, there is a method of using a sensor built into the display to monitor the internal state of the display, thereby detecting whether a fault has occurred only on the display side. However, in this method, there are types of faults that cannot be detected depending on the content of the fault, such as "faults on the imaging device side" and "abnormalities of the display device such as the final panel that cannot be detected by the sensor alone".

[0010] The problem to be solved in the present disclosure is that, when using a sensor inside a detection device, the number of defects of an object that can be detected is insufficient.

[0011] [Methods for solving the problem]

[0012] One embodiment of the present invention is a monitoring device comprising: an acquisition unit that acquires shooting data from a camera that shoots a location where a display device can be visually identified; and a determination unit that determines whether the display device has a malfunction based on the viewing behavior of multiple viewers determined based on the shooting data.

[0013] One embodiment of the present invention is a monitoring device comprising: an acquisition unit, which acquires visual recognition data of a position located in a display screen of a display device that can be visually recognized by multiple viewers; and a determination unit, which obtains a determination result of whether the display device is in a malfunction by inputting the visual recognition data acquired by the acquisition unit into a learned model, wherein the learned model is a model obtained by using the visual recognition data and data indicating whether the display device is in a malfunction to learn conditions indicating the relationship between the visual recognition data and whether the display device is in a malfunction.

[0014] One embodiment of the present invention is a learning device, which uses visual recognition data of a position in a display screen of a display device that can be visually recognized by multiple viewers and data indicating whether the display device has a malfunction to learn conditions indicating the relationship between the visual recognition data and whether the display device has a malfunction.

[0015] One embodiment of the present invention is a monitoring method, wherein shooting data is obtained from a camera that shoots a location where a display device can be visually recognized, and whether the display device has a malfunction is determined based on the viewing behavior of multiple viewers determined based on the shooting data.

[0016] One embodiment of the present invention is a monitoring method, wherein visual recognition data of a position located in a display screen of a display device that can be visually recognized by multiple viewers is obtained, and a judgment result of whether the display device is in a malfunction is obtained by inputting the visual recognition data into a learned model. The learned model is a model obtained by using the visual recognition data and data indicating whether the display device is in a malfunction to learn conditions indicating the relationship between the visual recognition data and whether the display device is in a malfunction, and an alarm signal is output based on the judgment result.

[0017] [Effects of the invention]

[0018] According to the present invention, it is possible to use shooting data obtained from a camera that shoots a location where a display device can be visually identified, and determine whether the display device has a malfunction based on the viewing behavior of viewers based on the shooting data. Therefore, it is possible to determine whether a malfunction exists by a method different from the internal sensor of the detection device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a system configuration diagram showing the configuration of the remote control system S.

[0020] Figure 2 It is a schematic functional block diagram showing the structure of the multi-screen display 30.

[0021] Figure 3 It is a schematic functional block diagram showing the configuration of the monitoring device 10 .

[0022] Figure 4 It is a flowchart explaining the operation of the monitoring device 10 .

[0023] Figure 5 It is a schematic functional block diagram showing the configuration of a monitoring device 10A in another embodiment.

[0024] Figure 6 : is a schematic functional block diagram showing the functions of the learning device 50 .

[0025] Figure 7 This is a flowchart for explaining the operation of the monitoring device 10A.

[0026] Figure 8 This is a schematic functional block diagram showing the configuration of a monitoring device 10B in another embodiment.

[0027] Fig. 9 It is a schematic functional block diagram showing the configuration of a monitoring device 10C in another embodiment. DETAILED DESCRIPTION

[0028] Hereinafter, a remote monitoring system S according to an embodiment of the present invention will be described with reference to the drawings.

[0029] Figure 1 1 is a system configuration diagram showing the configuration of the remote control system S.

[0030] In the remote monitoring system S, a monitoring device 10 , a content providing device 20 , and a multi-screen display 30 are connected via a network N so as to be communicable.

[0031] The monitoring device 10 controls the multi-screen display 30 via the network N, and acquires information related to the multi-screen display 30 via the network N.

[0032] The user who uses the monitoring device 10 can remotely monitor and control the multi-screen display 30 by utilizing the monitoring device 10 .

[0033] The content providing device 20 stores content and provides the content to the multi-screen display. The content may also be advertisements, notifications, instructions, etc.

[0034] The multi-screen display 30 includes a plurality of displays disposed adjacent to each other, and displays a video signal corresponding to the content supplied from the content providing device 20 .

[0035] The multi-screen display 30 is provided with a camera 31. The camera may be built in the multi-screen display 30 or may be provided outside the multi-screen display 30. Here, the multi-screen display 30 is provided with a total of 9 displays (display 30a, display 30b, display 30c, display 30d, display 30e, display 30f, display 30g, display 30h, display 30i) arranged adjacently in a manner of 3 displays arranged vertically and 3 displays arranged horizontally.

[0036] The multi-screen display 30 can display content as one large display screen including the display screens of each of such multiple displays.

[0037] The camera 31 captures a range in which the multi-screen display 30 can be visually recognized. More specifically, the camera 31 captures a viewer who is located near the display screen of the multi-screen display 30.

[0038] Here, in the case where a camera is provided in a display, it is rare that the camera that captures the display screen of the display is provided in a location different from the display. On the other hand, there are increasing cases where a camera is provided in the display and a camera that captures a range in which the display screen of the display can be visually recognized from the position of the display, that is, a camera that can capture a user who visually recognizes the display screen of the display is provided. For example, the following scheme is provided: the action of a person (viewer) standing in front of the display is captured by a camera, the action of the captured person is analyzed, and the object included in the displayed content is changed according to the action. In addition, the following scheme is also provided: the viewer is captured by a camera provided in the display, the age group is estimated based on the facial image of the person (viewer) obtained from the captured image, and an advertisement corresponding to the estimated age group is displayed on the display.

[0039] In the present embodiment, such a generally used camera is used, and based on the captured image obtained from the camera, it is determined whether a malfunction has occurred in any of the displays included in the multi-screen display based on the viewing behavior of viewing the multi-screen display. Therefore, in the case where the display is provided with a camera, by using this camera, there is no need to install a new camera. In addition, there is no need to construct a system that monitors the display state of the display by installing a camera for capturing the display screen of the display at a location different from the display and monitoring the captured image obtained from the camera, so there is no need to study which location of the location different from the display to install the camera, and there is no need to incur installation costs for installing the camera at a location different from the display.

[0040] The network N can be a LAN (local area network) or other communication networks.

[0041] Figure 2 It is a schematic functional block diagram showing the structure of the multi-screen display 30.

[0042] The multi-screen display 30 includes a communication unit 301 , a display unit 302 , a display control unit 303 , a camera 304 , a line of sight detection unit 305 , a storage unit 306 , and a control unit 307 .

[0043] The communication unit 301 communicates with the monitoring device 10 and the content providing apparatus 20 via the network N.

[0044] The display unit 302 displays content based on the video signal. The display unit 302 is, for example, a liquid crystal display panel. Here, the multi-screen display 30 includes nine displays, but for simplicity of description, the nine displays are collectively described as one display unit 302.

[0045] The display control unit 303 controls a drive circuit that drives a liquid crystal display panel as the display unit 302 , thereby reading out the content stored in the storage unit 306 and displaying it on the display unit 302 .

[0046] The camera 304 captures a range in which the display screen of the multi-screen display 30 can be visually recognized.

[0047] The sight line detection unit 305 extracts a person (viewer) included in the captured image based on the captured image captured by the camera 304, and detects the person's sight line. In addition, the sight line detection unit 305 detects the movement of the person's sight line or the position at which the person is looking in the multi-screen display 30 based on the detected sight line. In addition, the sight line detection unit 305 generates sight line data based on the result of detecting the sight line, and sends it to the monitoring device 10 via the communication unit 301. The sight line data represents the viewing behavior of the viewer who views the content displayed on the display device by visually recognizing the display screen of the display device. Specifically, the sight line data is data based on the result of detecting the sight line of the person who visually recognizes the display screen, and is data representing the visually recognized position in the display screen of the multi-screen display from the start time of detecting the sight line to the end time of the measurement.

[0048] The storage unit 306 stores the content provided by the content providing device 20 .

[0049] The control unit 307 controls each unit of the multi-display 30 .

[0050] Figure 3 It is a schematic functional block diagram showing the configuration of the monitoring device 10 .

[0051] The monitoring device 10 includes a communication unit 101 , an acquisition unit 102 , a storage unit 103 , a determination unit 104 , an output unit 105 , and a control unit 106 .

[0052] The communication unit 101 communicates with the multi-screen display 30 via the network N.

[0053] The acquisition unit 102 acquires the line of sight data from the multi-screen display 30 .

[0054] The storage unit 103 stores the line-of-sight data acquired by the acquisition unit 102 .

[0055] The determination unit 104 determines whether a malfunction occurs in the display device (multi-screen display 30) based on the viewing behavior of the plurality of viewers determined based on the shooting data. For example, the determination unit 104 determines whether a malfunction occurs in the multi-screen display 30 based on the viewing behavior indicated by the line of sight data received from the multi-screen display 30.

[0056] Here, let the display screen of one of the nine displays in the multi-screen display 30 (for example, the display 30f) be A. When a malfunction occurs in the display screen A and the content becomes non-displayed (for example, the screen becomes black without signal (black screen)), one content can continue to be displayed other than the display screen A, so only the display screen A is black, and thus only the display screen A looks abnormal to the viewer. Therefore, while a large image is displayed as one content on the screens of the eight displays, only the display screen A becomes black due to the malfunction, so the display screen A alone displays a unique image different from the others, and stands out compared to the other eight display screens.

[0057] If a viewer who is viewing the multi-screen display 30 while moving across the front of the display system visually recognizes the multi-screen display 30 in such a display state, the display screen A is conspicuous, so it is expected that the viewer will pay special attention to the display screen A. This can also be said to correspond to the "similar element" of the Bragnans law.

[0058] Based on the viewing behavior of such viewers, the determination unit 104 can distinguish between a situation in which the viewer views content through a multi-screen display 30 in normal operation and a situation in which the viewer views content through a multi-screen display 30 in which an abnormal situation occurs on the display screen A, by utilizing the characteristics of their respective line of sight movements.

[0059] The determination unit 104 performs the following two main determinations.

[0060] (1) Determination of the viewing characteristics of viewers when using the display in normal operation

[0061] (2) Determination of the viewer's sight characteristics when the display is in an abnormal state

[0062] These two determinations are described below.

[0063] (1) Determination of the viewing characteristics of viewers when using a display in normal operation

[0064] When the display is operating normally (when no malfunction occurs), the movement of the viewer's line of sight has the following characteristics, for example, a1 and a2.

[0065] (a1) Not fixating on one point. The viewer's line of sight moves slightly in accordance with the subtitles and images displayed as content. For example, the viewer's line of sight moves when reading the subtitle character string. In addition, the point of fixation in the image displayed as content changes according to interest.

[0066] (a2) When different viewers watch content, the initial gaze position (the initial gaze position) and the gaze position after a certain period of time are different for each viewer and therefore have no similarity. In particular, when the content is a video, the initial gaze position of the displayed content is different, and the subsequent gaze movement is different depending on the initial gaze start position.

[0067] (2) Determination of the viewer's sight characteristics when an abnormal state occurs during use of the display

[0068] The movement of the viewer's line of sight when a malfunction occurs in at least one of the displays constituting the multi-screen display has the following characteristics, for example, b1 and b2.

[0069] (b1) Focus on a point.

[0070] (b2) When different viewers watch content, there are similarities between the first gaze position (the initial gaze position) and the gaze position after a certain period of time. For example, the initial gaze position varies depending on the viewer, but since the viewer notices a display with an abnormal display mode, the gaze will be directed toward the abnormal display. Therefore, the gaze position after a certain period of time is similar between different viewers.

[0071] (b3) In b1 above, a high percentage of the displays being viewed are displays that have a malfunction.

[0072] Regarding the above-mentioned condition a1, for example, in the case of content such as displaying text information such as the release date only on a specific display in a multi-screen display and displaying a large character on other displays, if there are multiple viewers who want to confirm the release date, it is possible that multiple viewers who look at the display showing the release date will have different characteristics of the change in the position of attention, because there are viewers who look at the character and then move their eyes to the display showing the release date, and viewers whose eyes change with respect to the character according to the movement of the character. Therefore, based on condition a2, it is possible to distinguish whether the display is in normal operation or a malfunction has occurred.

[0073] Considering such conditions, the determination unit 104 obtains the sight line data from the sight line detection unit 305 that detects the sight line data including the gaze part (the gaze position) and the gaze time (the time of continuous gaze at the same position), and when the number of viewers for which the sight line data is obtained reaches a certain number, the determination process is performed using the above conditions and the reference value determined for the conditions. By performing this determination process, it is possible to estimate whether a malfunction has occurred in the display. When it is determined that a malfunction has occurred, an alarm message is output from the output unit 105.

[0074] In addition, the determination unit 104 may also determine that a defect exists in a specific area when the positions in the display screen of the display device visually recognized by multiple viewers are concentrated in a specific area in the display screen regardless of the change of the image displayed on the display device. The so-called change of the image is, for example, based on whether the image of the content displayed on the display screen changes during the reproduction time from the reproduction of the content to the end. For example, in the case where the content is a static image, the image does not change during the display of the static image. In addition, in the case where the content is a video, it can be said that there is no change in the image when there is no scene switching in the reproduction of the content, the people, scenery, goods, character strings, etc. displayed as content on the display screen do not move, there is no change in color, there is almost no movement, etc.

[0075] In addition, the determination unit 104 may determine whether the data is concentrated in a specific area based on whether there is similarity. For example, if there is similarity, the data is concentrated in a specific area, and if there is no similarity, the data is not concentrated in a specific area.

[0076] The output unit 105 outputs the determination result. For example, when the output unit 105 determines that a malfunction has occurred in the display in the determination result, it outputs an alarm. When the output unit 105 outputs an alarm, it displays a screen indicating the alarm on a display device built into the monitoring device 10A or a display device provided outside the monitoring device 10. In addition, the output unit 105 sends an alarm to a terminal device (for example, a smartphone, etc.) carried by an administrator of the display or a user of the display, displays an alarm screen, or plays an alarm sound.

[0077] The control unit 106 controls each unit of the monitoring device 10 .

[0078] Figure 4 It is a flowchart explaining the operation of the monitoring device 10 .

[0079] The multi-screen display 30 is photographed by the camera 31. When a person is extracted from the photographed image, the sight line detection unit 305 detects the sight line of the person, generates sight line data indicating the position of the sight line during the measurement target time (here, for example, 15 seconds) from the time when the sight line is detected, and transmits the sight line data to the monitoring device 10. The sight line detection unit 305 performs sight line detection processing on each person extracted from the photographed image, and transmits the sight line data to the monitoring device 10 each time the sight line data is generated.

[0080] If the sight line data is sent from the multi-screen display 30, the acquisition unit 102 of the monitoring device 10 receives (acquires) the sight line data (step S101). The storage unit 103 stores the acquired sight line data. The determination unit 104 determines whether the number of received sight line data reaches the number of people to be determined (step S102). If it does not reach the number of people to be determined (step S102-No), the process moves to step S101. Here, the number of people to be determined can be any number as long as it is 2 or more, but here, it is preferably a number of people that can grasp the degree of tendency of the viewing behavior of multiple viewers, for example, 10 people.

[0081] In step S102, when the number of received sight line data reaches the number of people to be determined (step S102-Yes), the determination unit 104 determines whether there is sight line data in which the time of looking at one display is longer than the reference time (step S103). The reference time is a time shorter than the measurement target time, preferably a time that can grasp the extent to which the sight line is continuously directed toward a certain position of the display, for example, 1 second.

[0082] When there is no sight line data indicating that the sight lines are directed to the same position for more than 1 second among the sight line data of the 10 persons (step S103-No), the determination unit 104 determines that the display is in a normal operation state (step S104). In this case, it is estimated that there is no person who continuously stares at a specific position of the display for more than 1 second among the 10 viewers. In addition, in this case, it is estimated that the above-mentioned condition a1 is satisfied. At this time, the acquisition unit 102 performs the acquisition process of the sight line data (step S101).

[0083] In step S103, if there is sight line data indicating that the sight lines are directed to the same position for more than 1 second in the sight line data of the 10 persons (step S103-Yes), the determination unit 104 moves the processing to step S105. Then, the determination unit 104 determines whether there is sight line data with similarity for the sight line action based on the sight line data (step S105). For example, the determination unit 104 compares the sight line data of the 10 persons respectively for the movement from the initial gaze position to the sight line position after a certain period of time, and determines that the sight line data is similar when the sight line position after at least a certain period of time is at the same position or within a certain range, and determines that there is no similarity when the sight line position after a certain period of time is not at the same position.

[0084] Regarding the gaze action based on the gaze data, if there is no similar gaze data (step S105-No), the determination unit 104 determines that the display is in a normal operation state (step S104). In this case, only a specific viewer is looking at a specific position of the display, and it can be inferred that the gaze is not due to a malfunction or the like.

[0085] On the other hand, regarding the gaze action based on the gaze data, if there is similar gaze data (step S105-Yes), the determination unit 104 determines whether the similar gaze data exceeds the number of reference persons (step S106). The reference number of persons is a number less than the predetermined number of persons in step S102, and any number of persons who can be estimated to have a common viewing tendency, for example, 6 persons.

[0086] In step S106, if the number of similar sight line data does not exceed the reference number of people (step S106-No), the determination unit 104 determines that the display is in a normal operation state (step S104). In this case, it can be inferred that although a plurality of viewers are looking at the same position, there are a plurality of viewers looking at the portion where the interesting content is displayed.

[0087] In step S106 , when the number of similar sight line data exceeds the reference number of people (step S106 —Yes), the determination unit 104 determines that a malfunction has occurred in the display (step S107 ).

[0088] When it is determined that a malfunction has occurred in the display, the output unit 105 outputs an alarm (step S108 ).

[0089] In this way, it is possible to capture the viewer from the camera installed in the multi-screen display 30, and determine whether a malfunction has occurred in the display based on the viewer's viewing behavior on the multi-screen display 30.

[0090] Next, a monitoring device 10A according to another embodiment will be described.

[0091] Figure 5 1 is a schematic functional block diagram showing the structure of a monitoring device 10A in another embodiment. Figure 3 The same functions as those of the monitoring device 10 are marked Figure 3 The monitoring device 10A can replace Figure 1 The monitoring device 10 is set in the remote control system S.

[0092] The acquisition unit 102A acquires the imaged data from the camera 304 that captures an image of a place where the multi-screen display 30 can be visually recognized.

[0093] The storage unit 103A stores the learned model.

[0094] The learned model is a model generated by performing supervised learning on the learning model. The learned model is a model obtained by learning conditions indicating the relationship between visual recognition data and whether the display is malfunctioning, using visual recognition data of positions on the display screen of the display device that can be visually recognized by multiple viewers and label data indicating whether the display is malfunctioning.

[0095] The visual recognition data may be, for example, photographing data or stay data indicating a visual recognition position and duration.

[0096] The photographed data is obtained by photographing a place where the display can be visually recognized, and for example, photographed data obtained from the camera 31 is used. The photographed data includes the viewer who visually recognizes the display, and it is possible to understand which position on the display screen of the display the viewer visually recognizes.

[0097] The captured data may be captured data obtained from a camera installed in a multi-screen display of the same model as the multi-screen display to be monitored, or a multi-screen display having the same number of displays arranged in the vertical and horizontal directions.

[0098] The stay data is data including a visual position and a duration. The visual position is a position on the display screen of the display device that is visually recognized by the viewer, and the duration is the time that the line of sight continues to be directed toward the visual position.

[0099] As a method of obtaining the stay data, for example, there is a method of obtaining the stay data by measuring the duration of the visually recognized position visually recognized by the viewer for each visually recognized position based on the shooting data. In addition, as another method of obtaining the stay data, there is a method of obtaining the visually recognized position and duration by inputting the shooting data obtained from the camera 31 into a learning model, and the learning model can obtain the visually recognized position and duration by inputting the shooting data.

[0100] The learned model is learned so as to be able to predict whether a malfunction has occurred in the multi-screen display to be monitored based on the input visual recognition data.

[0101] The learned model includes a first learned model, which is a model obtained by using shooting data obtained by shooting a place where the display can be visually identified and data indicating whether the display device has a malfunction, to learn conditions indicating the relationship between the shooting data and whether the display device has a malfunction.

[0102] In addition, the learning completion model has a second learning completion model, which is a model obtained by learning the conditions representing the relationship between the stay data and whether there is a bad condition in the display device. The stay data is a combination of a visually recognizable position of a position located on the display screen of the display device visually recognized by the viewer and the time, that is, the duration, when the line of sight continues to be directed toward the visually recognizable position.

[0103] Furthermore, the learned model can predict whether a malfunction has occurred or not, and can also predict the extent to which a malfunction has occurred based on probability or the like.

[0104] The model (learning model) that causes the learned model to learn may be a model to which any machine learning method is applied. For example, the learning model may be a model based on deep learning (deep learning) such as DNN (Deep Neural Network) and CNN (Convolutional Neural Network), which is known as an image classification model for recognizing and classifying images.

[0105] The determination unit 104A obtains a determination result of whether the multi-screen display 30 has a malfunction by inputting the captured data into the learned model stored in the storage unit 103A. That is, the determination unit 104A obtains a determination result of whether the display device has a malfunction by inputting the visual recognition data into the learned model, which is a model obtained by learning the conditions indicating the relationship between the visual recognition data and whether the display device has a malfunction using the visual recognition data of the positions in the display screen of the display device that can be visually recognized by multiple viewers and the data indicating whether the display device has a malfunction.

[0106] The determination unit 104A may determine whether or not there is a malfunction in the display device using either the first learned model or the second learned model.

[0107] When using the first learned model, the determination unit 104A inputs the externally acquired imaging data to the first learned model, obtains a result of whether the display device has a malfunction, and thereby determines whether the display device has a malfunction.

[0108] When using the second learned model, the determination unit 104A inputs the stay data obtained based on the imaging data into the second learned model to obtain a result of whether the display device has a malfunction, thereby determining whether the display device has a malfunction.

[0109] When the second learned model is used, the determination unit 104A may obtain the stay data from a measuring device provided outside the monitoring device 10A and obtaining the stay data from the captured data. Alternatively, the monitoring device 10A may be provided with a measuring function of such a measuring device and the stay data may be obtained from the measuring function.

[0110] In addition, when the second learned model is used, the determination unit 104A can also obtain the stay data by inputting it into the third learned model, and the third learned model can obtain the visual recognition position and duration by inputting the shooting data. The third learned model is a learned model obtained by learning the relationship between the shooting data and the visual recognition position and duration. The function of obtaining the stay data using the third learned model can be set in an external device, and the determination unit 104A obtains the stay data, or it can be set inside the monitoring device 10A, and the determination unit 104A obtains the stay data from the third learned model. In this case, in the monitoring device 10A, the stay data is obtained from the shooting data using the third learned model, and the stay data is input into the second learned model, thereby determining whether the display device has a malfunction. Therefore, in the monitoring device 10A, the learned model is used in two stages to determine whether the display device has a malfunction.

[0111] The output unit 105A outputs an alarm signal based on the determination result of the determination unit 104A.

[0112] Figure 6 : is a schematic functional block diagram showing the functions of the learning device 50 .

[0113] The learning device 50 includes an input unit 501 , a learning unit 502 , and an output unit 503 .

[0114] The input unit 501 obtains training data obtained by establishing a correspondence between shooting data obtained by shooting a place where the display device can be visually identified and label data indicating whether the display device has a malfunction. The training data includes, for example: first training data obtained by establishing a correspondence between the viewing behavior (movement of sight, etc.) of the viewer when a malfunction occurs in the multi-screen display and the label data indicating that the malfunction has occurred; and second training data obtained by establishing a correspondence between the viewing behavior of the viewer when no malfunction occurs in the multi-screen display and the label data indicating that no malfunction has occurred. Preferably, the first training data and the second training data are a large amount of data obtained in different scenes and at different times.

[0115] For example, the input unit 501 of the learning device 50 collects captured data of viewers' viewing behavior when no malfunction occurs by using a camera to capture a location where the display device can be visually recognized when no malfunction occurs in the display device. Then, the learning unit 502 assigns label data indicating that no malfunction occurs to the captured data. Here, a plurality of captured data are collected by acquiring different captured data for each viewer, and label data is assigned to each of the captured data to generate training data.

[0116] Then, when a problem occurs, the input unit 501 extracts the captured data of the viewing behavior of the viewers captured by the camera during the period when the problem occurs. The learning unit 502 adds label data indicating that a problem occurs to the captured data to generate training data.

[0117] Then, the learning unit 502 performs learning using the generated training data.

[0118] The learning unit 502 uses visual recognition data of a position on a display screen of a display device that can be visually recognized by a plurality of viewers and data indicating whether the display is in a malfunctioning state to learn a condition indicating a relationship between the visual recognition data and whether the display is in a malfunctioning state, thereby generating a learned model. The learned model generated by the learning unit 502 may be any one or more of the first learned model, the second learned model, and the third learned model.

[0119] The output unit 503 outputs the learned model generated by the learning unit 502 to an external device. For example, the output unit 503 outputs the learned model to the monitoring device 10A. In this case, the output unit 503 of the learning device 50 and the monitoring device 10A are connected in a communicative manner via a communication cable or a communication network, and the data as the learned model is sent from the output unit 503 of the learning device 50 to the monitoring device 10A for output.

[0120] Figure 7 This is a flowchart for explaining the operation of the monitoring device 10A.

[0121] The acquisition unit 102A of the monitoring device 10A acquires the imaged data from the camera 31 of the multi-screen display 30 (step S201 ).

[0122] The determination unit 104A inputs the acquired imaging data into the learned model (step S202 ), and acquires the determination result from the learned model (step S203 ). Then, the determination unit 104A determines whether the determination result indicates that a malfunction has occurred in the display (step S204 ).

[0123] When the judgment result obtained from the learned model does not indicate that a malfunction has occurred in the display (step S204-No), the judgment unit 104A causes the processing to enter step S201. When the judgment result obtained from the learned model indicates that a malfunction has occurred in the display (step S204-Yes), an alarm is output through the output unit 105A (step S205).

[0124] In the above-described embodiment, the monitoring object is described as a case where the multi-screen display 30 is used as the monitoring object, but the monitoring object is not limited to the multi-screen display. As long as there is a camera, it can also be a display group in which multiple display devices are arranged in adjacent or close positions, or a digital signage. In the display group in this case, each display can display one content. In addition, it can also be applied to a multi-screen display system in which images projected by multiple projectors are adjacent to each other and these multiple projected images are used to display one content.

[0125] Figure 8 It is a schematic functional block diagram showing the configuration of a monitoring device 10B in another embodiment.

[0126] The monitoring device 10B includes an acquisition unit 102B and a determination unit 104B.

[0127] The acquisition unit 102B acquires visual recognition data of a position in the display screen of the display device that can be visually recognized by multiple viewers. As a camera in this case, a camera installed in the display device can be used. The determination unit 104B obtains a determination result of whether the display device has a malfunction by inputting the visual recognition data acquired by the acquisition unit into a learned model. The learned model is a model obtained by learning the conditions indicating the relationship between the visual recognition data and whether the display device has a malfunction using the visual recognition data of the position in the display screen of the display device that can be visually recognized by multiple viewers and data indicating whether the display device has a malfunction. The visual recognition data includes, for example, data that can grasp which position in the display screen of the display device the viewer is directing his / her sight toward, or which position he / she is looking at. For example, in the case where there is a tendency that multiple viewers are looking at the same position of the display screen, it can be determined that a malfunction has occurred at that position in the display screen.

[0128] Fig. 9 It is a schematic functional block diagram showing the configuration of a monitoring device 10C in another embodiment.

[0129] The monitoring device 10C includes an acquisition unit 102C, a determination unit 104C, and an output unit 105C.

[0130] The acquisition unit 102C acquires visual recognition data that can grasp the position on the display screen of the display device that is visually recognized by a plurality of viewers.

[0131] The determination unit 104C obtains a determination result of whether the display device has a malfunction by inputting visual recognition data into the learned model, wherein the learned model is a model obtained by learning conditions indicating the relationship between visual recognition data and whether the display device has a malfunction using visual recognition data of a position in the display screen of the display device that can be visually recognized by multiple viewers and data indicating whether the display device has a malfunction.

[0132] The output unit 105C outputs an alarm signal according to the determination result.

[0133] In the embodiments described above, the storage unit 103, the storage unit 103A, and the storage unit 306 are composed of storage media such as HDD (Hard Disk Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read / write Memory), ROM (Read Only Memory), or any combination of these storage media.

[0134] For example, nonvolatile memories can be used as these storage units.

[0135] In addition, in the embodiments described above, the acquisition unit 102, the acquisition unit 102A, the determination unit 104, the determination unit 104A, the control unit 106, the display control unit 303, the line of sight detection unit 305, the control unit 307, the input unit 501, and the learning unit 502 can be composed of, for example, a processing device such as a CPU (central processing unit) or a dedicated electronic circuit.

[0136] In addition, it can also be used to implement Figure 1 The program of the function of the processing unit in the system is recorded in a computer-readable recording medium, and the computer system reads and executes the program recorded in the recording medium, thereby performing construction management. In addition, the "computer system" mentioned here includes hardware such as OS and peripheral devices.

[0137] In addition, if the WWW system is used, the "computer system" also includes the homepage providing environment (or display environment).

[0138] In addition, "computer-readable recording medium" refers to removable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" include media that retain programs for a certain period of time, such as volatile memories inside computer systems that become servers and clients. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, or a program that can realize the above-mentioned functions by combining with programs already recorded in the computer system. In addition, the above-mentioned program may be stored in a specified server, and the program may be distributed (downloaded, etc.) via a communication line in response to requests from other devices.

[0139] As mentioned above, although the embodiment of the present invention is described in detail with reference to the drawings, the specific structure is not limited to this embodiment, and also includes the design etc. within the scope which does not deviate from the summary of the present invention.

[0140] [Explanation of Reference Numerals]

[0141] 10, 10A, 10B, 10C…monitoring device, 20…content providing device, 30…multi-screen display, 30a, 30b, 30c, 30d, 30e, 30f, 30g, 30h, 30i…display, 31…camera, 50…learning device, 101…communication unit, 102, 102A, 102B, 102C…acquisition unit, 103, 103A, 306…storage unit, 104, 104A, 104B, 104C…determination unit, 105, 105A, 105C, 503…output unit, 106, 307…control unit, 301…communication unit, 302…display unit, 303…display control unit, 304…camera, 305…sight line detection unit, 501…input unit, 502…learning unit

Claims

1. A monitoring device comprising: an acquisition unit that acquires image data from a camera that captures an image of a location where the display device can be visually recognized; and The determination unit determines whether the display device has a malfunction based on the viewing behaviors of a plurality of viewers determined based on the captured data.

2. The monitoring device according to claim 1, wherein: The determination unit determines whether the display device has a malfunction based on positions on the display screen of the display device that are visually recognized by the plurality of viewers and are determined to be viewing behaviors of the plurality of viewers.

3. The monitoring device according to claim 2, wherein: The determination unit determines that a malfunction exists in a specific area on the display screen when positions on the display screen of the display device visually recognized by the plurality of viewers are concentrated in a specific area on the display screen regardless of changes in an image displayed on the display device.

4. The monitoring device according to claim 2, wherein: The determination unit determines whether there is similarity based on a history of positions of the display screen visually recognized by the viewer, and determines whether there is a malfunction in the display device based on a result of the determination.

5. The monitoring device according to claim 4, wherein: The determination unit determines whether a second position visually recognized by the viewer after the first position is visually recognized has similarity.

6. The monitoring device according to claim 5, wherein: The determination unit determines whether or not the display device has a malfunction based on whether or not the number of viewers considered to have the similarity reaches a reference number of viewers.

7. The monitoring device according to any one of claims 1 to 6, wherein: The acquisition unit acquires an image of a viewer visually recognizing content displayed on a display screen of the display device from a camera mounted on the display device.

8. A monitoring device comprising: an acquisition unit that acquires visual recognition data capable of grasping a position on a display screen of a display device that is visually recognized by a plurality of viewers; and The determination unit obtains a determination result of whether the display device is in a malfunctioning condition by inputting the visual recognition data obtained by the acquisition unit into the learned model. The learned model is a model obtained by using the visual recognition data and the data indicating whether the display device is in a malfunctioning condition to learn the conditions indicating the relationship between the visual recognition data and whether the display device is in a malfunctioning condition.

9. A learning device, wherein: Using visual recognition data that can grasp the position on the display screen of the display device that can be visually recognized by multiple viewers and data indicating whether the display device has a malfunction, a condition indicating the relationship between the visual recognition data and whether the display device has a malfunction is learned.

10. A monitoring method, wherein: Acquire image data from a camera that captures an image of a location where the display device can be visually recognized, Based on the viewing behaviors of a plurality of viewers determined based on the photographic data, it is determined whether the display device has a malfunction.

11. A monitoring method, wherein: Acquiring visual recognition data capable of grasping positions on a display screen of a display device that are visually recognized by a plurality of viewers, The determination result of whether the display device is in a malfunction is obtained by inputting the visual recognition data into a learned model, wherein the learned model is a model obtained by learning a condition indicating a relationship between the visual recognition data and the data indicating whether the display device is in a malfunction, using the visual recognition data and the data indicating whether the display device is in a malfunction. An alarm signal is outputted according to the determination result.

Images