Video viewing analysis system, method for analyzing viewing analysis, and video viewing analysis program

Abstract

To provide a detailed insight on video from the attitude of a viewer when the viewer is watching the video based on the biological information of the viewer.SOLUTION: A video viewing analysis system for analyzing the insight of a viewer on video acquires a plurality of pieces of noted point data corresponding to a plurality of viewers, generates a noted point visualization map which visualizes a noted point when a viewer who corresponds to the noted point data watched the video on the basis of at least some of the plurality of pieces of noted point data, and displays and superposes, the noted point visualization map on the video.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program. [Background technology]

[0002] Due to the impact of the COVID-19 pandemic, most aspects of social life are now moving online (in other words, digital transformation (DX)) at an accelerating pace. In particular, in the field of marketing, digital marketing has become more mainstream than traditional marketing using paper media, etc. In the field of digital marketing, advertising videos on video platforms such as YouTube (registered trademark) are currently becoming mainstream.

[0003] Patent Document 1 discloses an advertisement evaluation system for evaluating the effectiveness of an advertisement video. The advertisement evaluation system includes a user terminal, an e-mail server, a video distribution server, and a management server. These elements are communicatively connected via a communication network. The advertisement evaluation system determines whether or not a user has purchased a product introduced in an advertisement video based on the user's transaction data, and evaluates the effectiveness of the advertisement video according to the determination result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7057606 Summary of the Invention [Problem to be solved by the invention]

[0005] In the advertisement evaluation system disclosed in Patent Document 1, the effectiveness of an advertisement video is evaluated according to the sales record of whether or not a user actually purchased a product introduced in the advertisement video. For this reason, the advertisement evaluation system cannot evaluate an advertisement video at a stage where there is no actual sales record of the advertised product. On the other hand, there is a need for companies that perform digital marketing for products to grasp detailed insights about the video, such as an advertisement video, before the launch of the video. Here, "insight" means a deep understanding of a customer's potential needs, emotions, motivations, values, behavioral patterns and tendencies, which is presented in a refined form by analyzing customer data and feedback. In particular, there is a need for companies to grasp insights about the video based on the biometric information (e.g., gaze, facial expression, etc.) of the viewer who watches the video before the launch of the advertisement video. Thus, there is room for consideration of an advertisement evaluation system that can provide the above needs.

[0006] The present disclosure aims to provide a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program that are capable of providing detailed insights into videos from viewers' behavior while watching videos based on their biometric information. [Means for solving the problem]

[0007] A video viewing analysis system according to one embodiment of the present disclosure is a video viewing analysis system that analyzes viewer insights into a video, obtains a plurality of gaze point data corresponding to each of a plurality of viewers, generates a gaze point visualization map based on at least some of the plurality of gaze point data, which visualizes the gaze point when the viewer corresponding to the gaze point data watched the video, and displays the gaze point visualization map superimposed on the video.

[0008] According to the above configuration, since the gaze point visualization map is displayed superimposed on the video, the company side can grasp the viewers' viewing behavior for the video in detail by checking the gaze point visualization map. For example, the company side can grasp the areas of attention of the viewers in the video for each playback time in detail. In this way, it is possible to provide detailed insights about the video to the company side that desires video analysis.

[0009] A video viewing analysis system according to one embodiment of the present disclosure is a video viewing analysis system that analyzes viewer insights into a video, obtains multiple gaze point data corresponding to each of multiple viewers, and based on at least some of the multiple gaze point data, identifies an object that was the focus of the viewer corresponding to the gaze point data when they watched the video, and displays information about the focus object.

[0010] According to the above configuration, the company can grasp information about at least one attention object in the video, and thus it is possible to provide detailed insights about the video to the company that desires to analyze the video.

[0011] A video viewing analysis system according to one embodiment of the present disclosure is a video viewing analysis system that analyzes viewer insights into a video, obtains a plurality of facial expression data corresponding to each of a plurality of viewers, generates comprehensive facial expression information indicating the overall facial expression of the viewer corresponding to the facial expression data when watching the video based on at least some of the plurality of facial expression data, and displays the comprehensive facial expression information.

[0012] According to the above configuration, since the overall facial expression information is displayed, the company side can grasp the viewers' viewing behavior for the video in detail by checking the overall facial expression information. For example, the company side can grasp the viewers' emotions (e.g., happy, sad, etc.) for the video for each playback time in detail. In this way, it is possible to provide detailed insights about the video to the company side that desires video analysis.

[0013] A video viewing analysis system according to one embodiment of the present disclosure is a video viewing analysis system that analyzes viewer insights into a video, obtains a plurality of facial expression data corresponding to each of a plurality of viewers, generates overall interest level information indicating the overall interest level of the viewer corresponding to the facial expression data when watching the video based on at least some of the plurality of facial expression data, and displays the overall interest level information.

[0014] According to the above configuration, since the overall interest level information is displayed, the company can grasp the viewers' viewing behavior for the video in detail by checking the overall interest level information. For example, the company can grasp the viewers' interest in the video for each playback time in detail. In this way, it is possible to provide detailed insights about the video to the company that desires video analysis.

[0015] The video viewing analysis system may also input extraction conditions for the plurality of viewers, and extract at least some of the plurality of gaze point data based on the extraction conditions.

[0016] According to the above configuration, a gaze point visualization map is generated based on the extracted gaze point data. In this way, it is possible to grasp the viewing behavior of viewers who meet the extraction conditions in detail, and therefore it is possible to provide the company with more detailed insights about the video.

[0017] In addition, the extraction conditions may be associated with at least one of age information of the multiple viewers, gender information of the multiple viewers, address information of the multiple viewers, occupation information of the multiple viewers, number of people in each household of the multiple viewers, marital information of the multiple viewers, and educational background information of the multiple viewers.

[0018] According to the above configuration, it is possible to obtain a detailed understanding of the viewing behavior of viewers who meet extraction conditions associated with at least one of the following information: age information, gender information, address information, occupation information, number of people in household, marital information, and educational background information of multiple viewers, thereby making it possible to provide companies with more detailed insights regarding videos.

[0019] The video may also include advertising content.

[0020] Based on the above, it is possible to provide companies with detailed insights into digital marketing.

[0021] The gaze point visualization map may be a gaze point heat map.

[0022] According to the above, the company can grasp the viewers' viewing behavior for the video in detail by checking the gaze point heat map. In particular, by checking the intensity distribution of the gaze point heat map, it becomes possible to more clearly grasp the viewer's attention area on the video.

[0023] The gaze point data may indicate a temporal change in a gaze point corresponding to the line of sight of the viewer on a display area of ​​the video.

[0024] The video viewing analysis system may display the attention objects in a ranking format.

[0025] The video viewing analysis system may display at least one of the attention level, appearance time, and average attention level of the object of attention in association with the object of attention.

[0026] The comprehensive facial expression information may indicate a probability of each of a plurality of facial expression elements throughout the entire video.

[0027] Based on the above, the company can check the probability of each of a plurality of facial expression elements throughout the entire video, thereby gaining a detailed understanding of the viewers' viewing behavior with respect to the video.

[0028] The comprehensive facial expression information may indicate, for each of a plurality of facial expression elements, a change in probability corresponding to a time series of the video.

[0029] Based on the above, the company can grasp in detail the viewing behavior of the viewer with respect to the video by checking the transition of the probability for each of a plurality of facial expression elements corresponding to the time series of the video.

[0030] The overall interest level information may indicate a change in the interest level corresponding to a time series of the video.

[0031] According to the above, the company can grasp the viewers' viewing behavior for the video in detail by checking the change in interest level corresponding to the wording of the video.

[0032] The video viewing analysis system may also calculate the viewer's positivity or negativity toward the video based on the probability of each of a plurality of facial expression elements, and display the positivity or negativity.

[0033] Based on the above, the company can grasp the viewers' viewing behavior toward the video in detail by checking the viewers' positive or negative ratings.

[0034] A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insights into a video, and includes the steps of acquiring a plurality of gaze point data corresponding to each of a plurality of viewers, generating a gaze point visualization map based on at least some of the plurality of gaze point data, which visualizes the gaze points when the viewer corresponding to the gaze point data watched the video, and displaying the gaze point visualization map superimposed on the video, and is executed by a computer.

[0035] A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insights into a video, and includes the steps of acquiring a plurality of gaze point data corresponding to each of a plurality of viewers, identifying an object of attention when the viewer corresponding to the gaze point data watched the video based on at least some of the plurality of gaze point data, and displaying information related to the object of attention, and is executed by a computer.

[0036] A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insights into a video, and includes the steps of acquiring a plurality of facial expression data corresponding to each of a plurality of viewers, generating comprehensive facial expression information indicating the comprehensive expression of the viewer corresponding to the facial expression data when watching the video based on at least some of the plurality of facial expression data, and displaying the comprehensive facial expression information, and is executed by a computer.

[0037] A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insights into a video, and includes the steps of acquiring a plurality of facial expression data corresponding to each of a plurality of viewers, generating overall interest level information indicating the overall interest level of the viewer corresponding to the facial expression data when watching the video based on at least some of the plurality of facial expression data, and displaying the overall interest level information, and is executed by a computer.

[0038] Also, a video viewing analysis program for causing a computer to execute the video viewing analysis method may be provided. Effect of the Invention

[0039] According to the present disclosure, it is possible to provide a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program that are capable of providing detailed insights into videos from the viewer's behavior while watching a video based on their biometric information. [Brief description of the drawings]

[0040] [Figure 1] 1 is a diagram illustrating an example of a configuration of a video viewing analysis system according to an embodiment of the present disclosure (hereinafter, the present embodiment). [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a viewer terminal. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a company terminal. [Diagram 5] FIG. 2 is a sequence diagram for explaining a series of processes executed between a viewer terminal and a server in the video viewing analysis system according to the present embodiment. [Figure 6] FIG. 2 is a diagram for explaining a viewer's gaze point on a video display area. [Figure 7] FIG. 2 is a diagram for explaining facial feature points of a viewer. [Figure 8] FIG. 2 is a sequence diagram for explaining a series of processes executed between a server and a company terminal in the video viewing analysis system according to the embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a video viewing analysis screen. [Figure 10] 1A is a diagram showing an example of a moving image, FIG. 1B is a diagram showing an example of a gaze point heat map, and FIG. 1C is a diagram showing an example of a composite moving image in which the gaze point heat map is superimposed on the moving image. [Figure 11] These figures show a schematic diagram of each frame constituting a video and each frame constituting a gaze point heat map, where (a) shows a state in which the frame rate of the video and the frame rate of the gaze point heat map match, and (b) shows a state in which the frame rate of the video and the frame rate of the gaze point heat map do not match. [Figure 12] FIG. 13 is a diagram showing an example of an extraction condition display screen. [Figure 13] FIG. 13 is a diagram showing an example of a video viewing analysis screen after extraction conditions have been specified. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Hereinafter, the video viewing analysis system 1 according to the present embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of the video viewing analysis system 1 according to the present embodiment. As shown in FIG. 1, the video viewing analysis system 1 includes viewer terminals 2a and 2b, a server 3, and a company terminal 4. These are connected to a communication network 8. Each of the viewer terminals 2a and 2b is communicatively connected to the server 3 via the communication network 8. The company terminal 4 is communicatively connected to the server 3 via the communication network 8. The communication network 8 is composed of at least one of a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a wireless core network.

[0042] The viewer terminal 2a is a terminal associated with the viewer Va and is operated by the viewer Va. The viewer terminal 2b is a terminal associated with the viewer Vb and is operated by the viewer Vb. In this embodiment, for convenience of explanation, the viewer terminals 2a and 2b may be collectively referred to as the viewer terminal 2. Similarly, the viewers Va and Vb may be collectively referred to as the viewer V. In this embodiment, a large number of viewer terminals 2 associated with a large number of viewers are provided in the video viewing analysis system 1, but for convenience of explanation, only two of the large number of viewer terminals, the viewer terminals 2a and 2b, are illustrated in FIG. 1. The type of the viewer terminal 2 is not particularly limited, and the viewer terminal 2 may be, for example, a smartphone, a personal computer, a tablet, or a wearable device (for example, a head-mounted display or an AR display), etc.

[0043] Next, the hardware configuration of the viewer terminal 2 will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of the hardware configuration of the viewer terminal 2. As shown in Fig. 2, the viewer terminal 2 includes a control unit 20, a storage device 21, an imaging unit 22, a communication unit 23, an input operation unit 24, a display unit 25, a speaker 26, and an RTC (Real Time Clock) 28. These elements constituting the viewer terminal 2 are connected to a communication bus 29.

[0044] The control unit 20 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is configured from a ROM (Read Only Memory) in which various programs and the like are stored, and a RAM (Random Access Memory) having multiple work areas in which various programs and the like executed by the processor are stored. The processor is configured from at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The CPU may be configured from multiple CPU cores. The GPU may be configured from multiple GPU cores. The processor may be configured to load a program designated from various programs incorporated in the storage device 21 or the ROM onto the RAM and execute various processes in cooperation with the RAM.

[0045] The storage device 21 is, for example, a storage device (storage) such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and is configured to store programs and various data. The imaging unit 22 is configured to acquire video data showing the surrounding environment of the viewer terminal 2 through shooting. In particular, the imaging unit 22 is a camera configured to generate image data or video data showing the surrounding environment of the viewer terminal 2 through shooting, and includes an image sensor (for example, a CCD sensor or a CMOS sensor) and an image sensor drive processing circuit. In this embodiment, the control unit 20 functions as a gaze tracking unit that detects a change in the gaze of the viewer V based on the video data acquired by the imaging unit 22. Furthermore, the control unit 20 functions as an expression tracking unit that detects a change in the expression of the viewer V based on the video data.

[0046] The communication unit 23 includes a wireless communication module and / or a wired communication module for communicating with an external device connected to the communication network 8. The wireless communication module is configured to wirelessly communicate with an external device such as a base station or a wireless LAN router, and includes a transmitting / receiving antenna and a wireless transmitting / receiving circuit. The wireless communication module may be a wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a wireless communication module compatible with an X-generation mobile communication system (for example, a fourth-generation mobile communication system such as LTE) using a SIM (Subscriber Identity Module).

[0047] The input operation unit 24 is, for example, a touch panel, a mouse, and / or a keyboard arranged over the video display of the display unit 25, and is configured to receive an input operation of the viewer V and generate an operation signal corresponding to the input operation. The display unit 25 is, for example, configured of a video display and a video display circuit that drives and controls the video display. The display unit 25 has a display screen 27 on which moving images are displayed (see FIG. 6).

[0048] The speaker 26 is configured to output the audio of the video to the outside based on the audio data included in the video. The RTC 28 is configured to obtain information indicating the current time.

[0049] Returning to FIG. 1, the server 3 is communicatively connected to the viewer terminal 2 and the company terminal 4 via the communication network 8. The server 3 transmits video data to each of the multiple viewer terminals 2 via the communication network 8, and transmits a composite video (described later) in which a gaze point heat map is superimposed on the video to the company terminal 4. The server 3 may be composed of multiple servers. The server 3 functions as a WEB server configured to provide a video viewing analysis application as a WEB application. In this regard, the server 3 is configured to transmit data (e.g., HTML files, CSS files, image and video files, program files, etc.) for displaying a video viewing analysis screen on the WEB browser of the company terminal 4. In this way, the server 3 functions as a server for providing SaaS (System as a Service). The server 3 may be built on-premise or may be a cloud server. The server 3 also functions as a data management server that manages multiple video data, gaze point data of each viewer V, and facial expression data of each viewer V.

[0050] The hardware configuration of the server 3 will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the server 3. As shown in Fig. 3, the server 3 includes a control unit 30, a storage device 31, an input / output interface 32, a communication unit 33, an input operation unit 34, and a display unit 35. These elements constituting the server 3 are connected to a communication bus 36.

[0051] The control unit 30 includes a memory and a processor. The memory is configured to store computer-readable instructions. In particular, the memory may store a video viewing analysis program for causing the processor to execute a series of processes (video viewing analysis method) executed by the server 3 in Figs. 5 and 8. The memory is configured with a ROM and a RAM. The processor is configured with at least one of a CPU, an MPU, and a GPU.

[0052] The storage device 31 is, for example, a storage device (storage) such as an HDD, SSD, or flash memory, and is configured to store programs and various data. The storage device 31 stores a plurality of video data, gaze point data of each viewer V, and facial expression data of each viewer V. The storage device 31 also stores a viewer information table related to information of each viewer V and a user information table related to each user U who uses a video viewing analysis application. The viewer information table includes attribute information of each viewer V. For example, the viewer information table may include at least one of identification information, gender information, age information, household number information, address information, and occupation information of each viewer V. The user information table may include identification information, attribute information, login information, and the like of each user U.

[0053] The input / output interface 32 is an interface that enables a connection between an external device and the server 3, and includes an interface conforming to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 33 may include various wired communication modules for communicating with an external terminal on the communication network 8. The input operation unit 34 is, for example, a touch panel, a mouse, and / or a keyboard, and is configured to receive an input operation by an operator and generate an operation signal according to the input operation by the operator. The display unit 35 is, for example, configured by an image display and an image display circuit.

[0054] The company terminal 4 is a terminal operated by a user U who uses a video viewing analysis application provided by the server 3. In this embodiment, a plurality of company terminals 4 are provided in the video viewing analysis system 1 (in other words, in this embodiment, a plurality of users U use the video viewing analysis application), but for convenience of explanation, only one company terminal 4 is illustrated in FIG.

[0055] The hardware configuration of the company terminal 4 will be described below with reference to FIG. 4. FIG. 4 is a diagram showing an example of the hardware configuration of the company terminal 4. As shown in FIG. 4, the company terminal 4 may be, for example, a personal computer, a smartphone, a tablet, or a wearable device attached to a user U. The company terminal 4 has a web browser. The video viewing analysis application runs on the web browser of the company terminal 4. The company terminal 4 includes a control unit 40, a storage device 41, an input / output interface 42, a communication unit 43, an input operation unit 44, and a display unit 45. These elements are connected to a communication bus 46.

[0056] The control unit 40 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is configured with a ROM and a RAM. For example, the processor is configured with at least one of a CPU, an MPU, and a GPU.

[0057] The storage device 41 is, for example, a storage device such as an HDD, SSD, or flash memory, and is configured to store programs and various data. The input / output interface 42 is an interface (for example, USB, HDMI, etc.) that enables a connection between an external device and the company terminal 4. The communication unit 43 is configured to connect the company terminal 4 to the communication network 8, and includes a wireless communication module and / or a wired communication module. The input operation unit 44 is, for example, a touch panel, a mouse, and / or a keyboard, and is configured to accept an input operation of the user U and generate an operation signal according to the input operation of the user U. The display unit 45 is, for example, configured by a video display and a video display circuit. In this embodiment, a video viewing analysis screen 70 (see FIG. 9) is displayed on the display unit 45.

[0058] Next, a series of processes executed between each of the multiple viewer terminals 2 and the server 3 in the video viewing analysis system 1 according to this embodiment will be described below with reference to FIG. 5. FIG. 5 is a sequence diagram for explaining a series of processes executed between the viewer terminal 2 and the server 3 in the video viewing analysis system 1. For convenience of explanation, a series of processes executed between a single viewer terminal 2 and the server 3 is illustrated in FIG. 5, but in this embodiment, each of the multiple viewer terminals 2 executes the series of processes shown in FIG. 5. In addition, in the series of processes shown in FIG. 5, the viewer V provides the server 3 with gaze point data and facial expression data of the viewer V for a predetermined video through viewing the predetermined video provided by the server 3. In this regard, the operating company that operates the server 3 and provides the video viewing analysis application may provide the viewer V with an incentive for viewing the video. For example, the company may grant the viewer V a predetermined number of points or electronic money for each video viewing.

[0059] As shown in FIG. 5, in step S1, the viewer terminal 2 transmits a video viewing request to the server 3 to view a video in response to an input operation on the input operation unit 24 (e.g., a touch panel) by the viewer V. Next, the server 3 receives the video viewing request from the viewer terminal 2 and transmits the video stored in the storage device 31 to the viewer terminal 2 (step S2). The video may be transmitted in streaming format or in download format to the viewer terminal 2. The video transmitted to the viewer terminal 2 (i.e., the video subject to viewing analysis) may include advertising content. For example, the video may be an advertising video played on a video platform such as YouTube (registered trademark).

[0060] Next, the viewer terminal 2 starts playing the video (step S3). As shown in FIG. 6, the viewer terminal 2 displays the video on the display screen 125 of the display unit 25. The viewer terminal 2 also starts acquiring gaze point data and facial expression data of the viewer V at the same time as starting the playback of the video (step S4). As shown in step S5, the viewer terminal 2 continues to acquire the gaze point data and facial expression data of the viewer V until the playback of the video ends. On the other hand, the viewer terminal 2 stops acquiring the gaze point data and facial expression data of the viewer V when the playback of the video ends, and then transmits the gaze point data and facial expression data to the server 3 (step S6).

[0061] <Method of acquiring gaze point data> The viewer terminal 2 acquires video data showing the viewer V through shooting using the imaging unit 22, and then acquires gaze point data of the viewer V based on the video data. In this regard, the control unit 20 of the viewer terminal 2 functions as an eye tracking unit that detects changes in the gaze of the viewer V based on the video data acquired by the imaging unit 22. As an example of a method of acquiring gaze point data, the control unit 20 identifies the viewer V and the viewer V's eyes shown in a predetermined frame (still image) of the video data, and then acquires position information indicating the position of the viewer V's eyes relative to the imaging unit 22. Next, the control unit 20 identifies the rotation angle of the pupil of the viewer V's eyeball based on the predetermined frame, and then acquires gaze information indicating the direction of the viewer V's gaze L (see FIG. 6). Furthermore, the control unit 20 acquires face direction information indicating the direction of the viewer V's face based on the predetermined frame. The control unit 20 may acquire gaze information based on a three-dimensional eyeball model (passive method) or may acquire gaze information based on a corneal reflex method (active method). When the line of sight information is acquired based on the corneal reflex method, the viewer terminal 2 may further include an infrared light emitting element (for example, an infrared LED) configured to emit infrared light, and an infrared camera (infrared image sensor).

[0062] Next, the control unit 20 acquires the position (X, Y) (XY coordinates) of the gaze point P, which is the point where the gaze L of the viewer V intersects with the display screen 125 of the viewer terminal 2, based on the position information, the gaze information, and the face direction information. Here, the position of the gaze point P where the gaze L intersects with the display screen 125 may be acquired as a position on the XY coordinate space set on the display screen 125. In this way, the control unit 20 can acquire the position of the gaze point P where the gaze L intersects with the display screen 125 from a predetermined frame of the video data. The control unit 20 may acquire the position of the gaze point P at an acquisition rate corresponding to the frame rate of the video data, or may acquire the position of the gaze point P at a rate different from the frame rate. For example, when the frame rate of the video data is 30 fps, the acquisition rate of the gaze point P may be 30 Hz, or may be an acquisition rate lower than 30 Hz (for example, 15 Hz).

[0063] Next, the control unit 20 converts the position of the gaze point P of the viewer V from a position in the XY coordinate space set on the display screen 125 to a position in the XY coordinate space set in the display area S of the video. In this regard, when the video is not displayed on the display screen 125 in full screen mode, the XY coordinate space set in the display area S of the video does not match the XY coordinate space set on the display screen 125. For this reason, the position of the gaze point P is converted from a position in the XY coordinate space set on the display screen 125 to a position in the XY coordinate space set in the display area S of the video. Note that when the video is displayed on the display screen 125 in full screen mode, the XY coordinate space set in the display area S of the video matches the XY coordinate space set on the display screen 125, so this coordinate conversion process does not need to be performed.

[0064] Furthermore, the control unit 20 acquires the position of the gaze point P on the display area S and the playback time of the video when the position of the gaze point P is acquired. For example, if the playback time of the video when the position (X1, Y1) of the gaze point P1 is acquired is t1, the position (X1, Y1) of the gaze point P1 on the display area S and the playback time t1 are stored in a state in which the position (X1, Y1) of the gaze point P1 and the playback time t1 are associated with each other. In this way, the control unit 20 can acquire gaze point data indicating a temporal change in the position of the gaze point P on the display area S. For example, if the acquisition rate of the gaze point is 30 Hz and the playback time of the video is 30 seconds, the number of gaze points P included in the gaze point data is 900.

[0065] <Method of acquiring facial expression data> The viewer terminal 2 acquires video data showing the viewer V through shooting using the imaging unit 22, and then acquires facial expression data of the viewer V based on the video data. In this regard, the control unit 20 of the viewer terminal 2 functions as an expression tracking unit that detects changes in the expression of the viewer V based on the video data acquired by the imaging unit 22. Here, the facial expression data is data indicating temporal changes in the facial expression information of the viewer V. More specifically, the facial expression data is data indicating temporal changes in probability information of the facial expression elements of the viewer V (in this example, five types of facial expression elements: happy, sad, neutral (no expression), anger, and surprise).

[0066] As an example of a method for acquiring facial expression data, the control unit 20 identifies the face of the viewer V shown in a predetermined frame (still image) of the video data M, and then acquires a plurality of facial feature points K included in the face of the viewer V (see FIG. 7). As shown in FIG. 7, in this example, 24 facial feature points K related to the eyebrows, eyes, nose, and mouth are acquired. Next, the control unit 20 calculates a plurality of facial feature amounts based on the acquired plurality of facial feature points K, and acquires probability information of each facial expression element of the viewer V based on the calculated plurality of facial feature amounts and a facial expression recognition model (learning model). In the facial expression recognition model, the value of each unit of the input layer indicates a corresponding one of the plurality of facial feature amounts, and the value of each unit of the output layer indicates probability information of a corresponding one of the plurality of facial expression elements. The probability information of each facial expression element may be expressed as a percentage %. In particular, the probability of the facial expression element of "happy", the probability of the facial expression element of "sad", the probability of the facial expression element of "neutral", the probability of the facial expression element of "anger", and the probability of the facial expression element of "surprise" are each expressed as a percentage %. In this way, the control unit 20 can obtain probability information of each facial expression element of the viewer V from a predetermined frame.

[0067] The control unit 20 may acquire facial expression information of the viewer V at an acquisition rate corresponding to the frame rate of the video data, or may acquire facial expression information of the viewer V at a rate different from the frame rate. For example, when the frame rate of the video data is 30 fps, the acquisition rate of the facial expression information may be 30 Hz, or may be a rate lower than 30 Hz (e.g., 15 Hz).

[0068] In addition, the control unit 20 acquires the playback time of the video when the facial expression information is acquired together with the facial expression information. For example, if the playback time of the video when the probability information of each facial expression element is acquired is t1, the probability information of each facial expression element and the playback time t1 are stored in a state in which the probability information and the playback time t1 are associated with each other. In this way, the control unit 20 can acquire facial expression data indicating the temporal change in the facial expression information of the viewer V.

[0069] Returning to FIG. 5, in step S7, the server 3 receives the gaze point data and facial expression data of the viewer V related to the video from each viewer terminal 2, and stores the gaze point data and facial expression data of each viewer V in the storage device 31 in a state in which the gaze point data and facial expression data of each viewer V are associated with the video to be viewed. In the example shown in FIG. 5, for convenience of explanation, one type of video is transmitted to each viewer terminal 2, but multiple types of videos may be transmitted to each viewer terminal 2. In this case, the gaze point data and facial expression data may be associated with both the identification information of the viewer and the identification information of the video. For example, the gaze point data and facial expression data of viewer Va transmitted from viewer terminal 2a may be associated with the identification information of viewer Va and the identification information of the video to be viewed. Similarly, the gaze point data and facial expression data of viewer Vb transmitted from viewer terminal 2b may be associated with the identification information of viewer Vb and the identification information of the video to be viewed. In this way, the gaze point data and facial expression data of each viewer V can be stored in the server 3 through the series of processes shown in FIG. 5.

[0070] Next, referring to FIG. 8, a series of processes executed between the server 3 and the company terminal 4 in the video viewing analysis system 1 according to this embodiment will be described below. FIG. 8 is a sequence diagram for explaining a series of processes executed between the server 3 and the company terminal 4 in the video viewing analysis system 1. In this example, a user U who operates the company terminal 4 is engaged in digital marketing work related to videos, and wishes to grasp detailed insights of the videos based on biometric information (gazing points and facial expressions) of multiple viewers V who have viewed the videos. The videos viewed by each viewer V in the series of processes shown in FIG. 5 are videos related to the user U's digital marketing work (for example, advertising videos displayed on a video platform, etc.).

[0071] As shown in FIG. 8, in step S10, the company terminal 4 transmits the login information of the user U (e.g., a login ID and a login password) to the server 3 through an input operation by the user U. In step S11, the server 3 authenticates the user U based on the login information transmitted from the company terminal 4. Note that the authentication method for the user U is not limited to login authentication. After authenticating the user U, the server 3 identifies the video associated with the user U by referring to the user information table stored in the storage device 31.

[0072] In step S12, the server 3 transmits data for displaying the video viewing analysis screen 70 shown in Fig. 9 (hereinafter, video viewing analysis screen data) to the company terminal 4 via the communication network 8. The video viewing analysis screen data may be a file (e.g., an HTML file, a CSS file, a program file, etc.) for displaying the video viewing analysis screen 70 on the WEB browser of the company terminal 4. Here, the program file may be a program file executable on the WEB browser such as JavaScript.

[0073] As shown in FIG. 9, the video viewing analysis screen 70 has a video display area 71, an object of interest display area 76, a facial expression information display area 72, and a facial expression / interest level information display area 73. Furthermore, the video viewing analysis screen 70 has a viewer selection button 74 and a video selection button 75. The video display area 71 displays a composite video 63, which will be described later. The object of interest display area 76 displays information about an object of interest in the video. The facial expression information display area 72 displays comprehensive facial expression information in a radar chart format. The facial expression / interest level information display area 73 displays comprehensive facial expression information in a time series graph and comprehensive interest level information in a time series graph.

[0074] In step S13, the company terminal 4 selects the selected video to be displayed on the video viewing analysis screen 70 and the extraction conditions for the viewer V in response to the input operation of the user U on the video viewing analysis screen 70. Specifically, the company terminal 4 selects the selected video to be displayed in the video display area 71 in response to the input operation of the user U on the video selection button 75. The selected video that can be selected through the video selection button 75 is the video associated with the user U. The extraction conditions for the viewer V are selected in response to the user's input operation on the viewer selection button 74. More specifically, as shown in FIG. 12, the company terminal 4 displays an extraction condition display screen 80 for specifying the extraction conditions for the viewer V on the video viewing analysis screen 70 in response to the input operation of the user U on the viewer selection button 74.

[0075] In the extraction condition display screen 80 shown in FIG. 12, the extraction condition of the viewer V is associated with the gender information of the viewer V, the age information of the viewer V, and the number of people in the household of the viewer V. For example, when the viewer V in his / her 40s is selected as the extraction condition, in step S15, the server 3 acquires the gaze point data of the viewer V in his / her 40s. In addition, the user U can specify an extraction condition associated with at least one of the gender information, age information, and household information through the extraction condition display screen 80. For example, as shown in FIG. 12, when the user U specifies an extraction condition (item name: target) related to a woman in her 20s living in a single-person household, in step S15, the server 3 acquires the gaze point data of the viewer V that matches the woman in her 20s living in a single-person household.

[0076] 12, three items, namely, gender information, age information, and household number information, are exemplified as the extraction conditions for viewer V, but this embodiment should not be limited to these. In this respect, address information, occupation information, marital information, and educational background information of viewer V may be further associated with the extraction conditions for viewer V. In this manner, the extraction conditions for viewer V may be associated with at least one of age information, gender information, address information, occupation information, household number information, marital information, and educational background information of viewer V.

[0077] Furthermore, the extraction conditions for the viewer V may be further associated with at least one of the viewer V's questionnaire information, survey information, and past behavior history information (eg, frequency of going to movie theaters, etc.).

[0078] Next, in step S14, the company terminal 4 requests the server 3 to transmit the composite video in response to an input operation by the user U. At this point, in response to an input operation by the user U on the composite video play button provided on the video viewing analysis screen 70, the company terminal 4 transmits to the server 3 a request to transmit the composite video associated with the selected video selected via the video selection button 75. Furthermore, the company terminal 4 transmits to the server 3 information regarding the extraction conditions of the viewer V together with the request to transmit the composite video.

[0079] In step S15, the server 3 receives a request to send a composite video and information on the extraction conditions of the viewer V from the company terminal 4, and then extracts the gaze point data of the viewer V that matches the extraction conditions from the storage device 31. In this regard, the server 3 extracts the identification information of the viewer V that matches the extraction conditions by referring to a viewer information table related to the information of each viewer V. The server 3 then extracts the gaze point data associated with both the identification information of the selected video and the identification information of the extracted viewer V. For example, as shown in FIG. 13, when the extraction conditions of the viewer V are age: 20s, gender: female, and number of people in the household: one-person household, the server 3 identifies the identification information of the viewer V that meets all the conditions of age: 20s, gender: female, and number of people in the household: one-person household, and then extracts the gaze point data of the viewer V that is associated with both the identification information of the identified viewer V and the identification information of the selected video. On the other hand, as shown in FIG. 9, when the extraction conditions of the viewer V are all viewers, the gaze point data of the viewer V that is associated with the identification information of the selected video is extracted.

[0080] In step S16, the server 3 generates a gaze point heat map (one example of a gaze point visualization map) that visualizes the temporal change of the position of the gaze points P of the multiple viewers V that meet the extraction condition based on the extracted gaze point data. In the gaze point heat map, the temporal change of the aggregate of the gaze points P of the multiple viewers V is visualized. The frame rate of the gaze point heat map may be the same as the acquisition rate of the gaze point data. For example, when the gaze points of the viewers V1 to Vn (n is an integer of 2 or more) at the playback time t1 of the video are P1 to Pn, the gaze points P1 to Pn are visualized in the gaze point heat map at the playback time t1. In the gaze point heat map, when the number of gaze points overlapping a predetermined area is large (i.e., when the predetermined area is an area gazed at by multiple viewers V), the predetermined area may be colored in a warm color such as red. On the other hand, when the number of gaze points overlapping a predetermined area is small (i.e., when the predetermined area is an area not gazed at much by multiple viewers V), the predetermined area may be colored in a cool color such as blue. In this way, the color of the predetermined region may gradually change from a cool color to a warm color as the number of gaze points overlapping the predetermined region increases. Furthermore, if there are no gaze points included in the predetermined region (i.e., if the predetermined region is an area that is not gazed upon by any of the viewers V), the predetermined region may not be colored. In this regard, a region that includes only gaze points that are treated as noise (outliers) may not be colored at all.

[0081] In this embodiment, a gaze point heat map is described as an example of a gaze point visualization map that visualizes the temporal changes in the gaze points of multiple viewers V, but the gaze point visualization map is not limited to the gaze point heat map.

[0082] Furthermore, in this embodiment, when the extraction conditions for the viewer V are all viewers, a gaze point heat map is generated that visualizes the temporal change in the positions of the gaze points P of all viewers V. In this way, the gaze point heat map generated changes according to the extraction conditions for the viewer V (in other words, the attributes of the viewer V).

[0083] In step S17, the server 3 generates a composite video 63 in which the gaze point heat map 62 is superimposed on the video 60 (see FIG. 10). In particular, the server 3 generates a composite video 63 in which the gaze point heat map 62 is superimposed on the video 60 in a state in which the gaze point heat map 62 and the video 60 are synchronized in time. In this regard, when the frame rate of the video 60 and the frame rate of the gaze point heat map 62 are the same, as shown in FIG. 11(a), each of the frames Fb1 to Fb6 constituting the gaze point heat map 62 is superimposed on a corresponding one of the multiple frames Fa1 to Fa6 constituting the video 60. For example, the frame Fb3 of the gaze point heat map 62 corresponding to the playback time t3 of the video 60 is superimposed on the frame Fa3 of the video 60 corresponding to the playback time t3. On the other hand, when the frame rate of the gaze point heat map 62 is half the frame rate of the video 60, as shown in FIG. 11(b), each of the multiple frames Fb1 to Fb3 constituting the gaze point heat map 62 is superimposed on at least one corresponding one (two frames in this example) of the multiple frames Fa1 to Fa6 constituting the video 60. In the example shown in FIG. 11(b), the frame Fb1 of the gaze point heat map 62 is superimposed on the frames Fa1 and Fa2 of the video 60. Thus, in this embodiment, the "state in which the gaze point heat map 62 and the video 60 are synchronized in time" includes not only the state shown in FIG. 11(a) but also the state shown in FIG. 11(b).

[0084] In step S18, the server 3 transmits the composite video 63 to the company terminal 4. The composite video 63 may be transmitted in a streaming format or in a download format. After receiving the composite video 63 from the server 3, the company terminal 4 displays the composite video 63 in a video display area 71 of a video viewing analysis screen 70. In this manner, the composite video 63 is played back on the company terminal 4 (step S19).

[0085] In step S20, the server 3 acquires information about an object of attention (area of ​​attention) in the video 60, and then transmits the information about the object of attention to the company terminal 4. In this regard, the server 3 identifies an object of attention (area of ​​attention) in the video 60 that overlaps with the gaze points of multiple viewers V that match the extraction conditions based on the gaze point data of the viewer V extracted in step S15, and acquires information about the object of attention. The information may be transmitted after the composite video 63 is played. As shown in FIG. 9, the information about the object of attention in the video 60 is displayed in the object of attention display area 76 of the video viewing analysis screen 70 (step S21).

[0086] The information on the attention object may include information indicating the specified attention object (attention area), attention level information on the attention level of the attention object, appearance time information on the appearance time of the attention object, and average attention level information on the average attention level of the attention object. The information indicating the specified attention object may be displayed as a trimmed image 176 indicating a partial area of ​​the video 60, as shown in FIG. 9. For example, the server 3 may specify the top three attention objects in the attention level ranking as the attention objects in the video 60 that overlap with the gaze points of multiple viewers V frequently, and then display the trimmed images 176 of the three attention objects in the attention object display area 76. In this case, the attention object that overlaps with the gaze points of the viewers V most frequently becomes the attention object with the highest attention level ranking.

[0087] The server 3 may also determine the attention level of each attention object based on the number of gaze points of the viewer V that overlap the attention object (in this example, the top three attention objects in the attention level ranking) in each frame of the video 60. The attention level of the attention object may be displayed as a percentage % or as a grade (A, B, C, etc.).

[0088] The server 3 may determine whether or not an attention object (in this example, the top three attention objects in the attention ranking) is present in each frame of the video 60, and then determine the appearance time of the attention object. For example, if the frame rate is 60 fps and the playback time is 30 seconds, the total number of frames in the video is 1800. Here, if the attention object O1 with the highest attention ranking is present within 600 frames, the appearance time of the attention object O1 is 30 seconds x (600 frames / 1800 frames) = 10 seconds.

[0089] The server 3 identifies the number of gaze points of the viewer V overlapping the attention object (in this example, the top three attention objects in the attention ranking) in each frame of the video 60, and then identifies the average number of gaze points of the viewer V overlapping the attention object. The server 3 may then determine the average attention degree of the attention object based on the average number of gaze points of the viewer V overlapping the attention object. The average attention degree of the attention object may be displayed as a percentage % or as a grade (A, B, C, etc.).

[0090] Next, in step S22, the server 3 extracts facial expression data of the viewer V that matches the extraction condition from the storage device 31. In this regard, the server 3 extracts the identification information of the viewer V that matches the extraction condition by referring to a viewer information table related to the information of each viewer V. Then, the server 3 extracts facial expression data associated with both the identification information of the selected video and the identification information of the extracted viewer V. For example, as shown in FIG. 13, when the extraction conditions of the viewer V are age: 20s, gender: female, and number of people in the household: one-person household, the server 3 identifies the identification information of the viewer V that satisfies all the conditions of age: 20s, gender: female, and number of people in the household: one-person household, and then extracts the facial expression data of the viewer V that is associated with both the identification information of the identified viewer V and the identification information of the selected video. On the other hand, as shown in FIG. 9, when the extraction conditions of the viewer V are all viewers, the facial expression data of the viewer V that is associated with the identification information of the selected video is extracted.

[0091] In step S23, the server 3 generates overall facial expression information and overall interest level information based on the facial expression data of the viewer V extracted in step S22. After that, the server 3 transmits the generated overall facial expression information and overall interest level information to the company terminal 4. As for the timing of transmitting these pieces of information, these pieces of information may be transmitted after the composite video 63 is played back. As shown in FIG. 9, the company terminal 4 displays the overall facial expression information displayed in a radar chart format in the facial expression information display area 72 of the video viewing analysis screen 70, and also displays the overall facial expression information displayed in a time series graph format in the facial expression and interest level information display area 73 of the video viewing analysis screen 70 (step S24). Furthermore, the company terminal 4 displays the overall interest level information displayed in a time series graph in the facial expression and interest level information display area 73.

[0092] The comprehensive facial expression information indicates comprehensive facial expression information of the viewer V that matches the extraction condition, and includes comprehensive facial expression information in a radar chart format and comprehensive facial expression information in a time series graph format. The comprehensive facial expression information in the radar chart format indicates average probability information (percentage %) of each comprehensive facial expression element (happy, sad, neutral, anger, surprise) of the viewer V that matches the extraction condition. For example, the comprehensive facial expression information in the radar chart format may include the average probability of the facial expression element "happy" of multiple viewers V that match the extraction condition, the average probability of the facial expression element "sad" of the multiple viewers V, the average probability of the facial expression element "neutral" of the multiple viewers V, the average probability of the facial expression element "anger" of the multiple viewers V, and the average probability of the facial expression element "surprise" of the multiple viewers V.

[0093] For example, if the extracted viewer V includes viewers Va, Vb, and Vc, the server 3 determines the average probability of each facial expression element of the viewer Va based on the facial expression data showing the temporal change of each facial expression element of the viewer Va. Similarly, the server 3 determines the average probability of each facial expression element of the viewer Vb based on the facial expression data showing the temporal change of each facial expression element of the viewer Vb, and determines the average probability of each facial expression element of the viewer Vc based on the facial expression data showing the temporal change of each facial expression element of the viewer Vc. After that, the server 3 determines the overall average probability of each facial expression element of the viewers Va to Vc based on the average probability of each facial expression element of the viewers Va to Vc. At this point, if the average probabilities of the facial expression element "happy" of the viewers Va, Vb, and Vc are Ra1, Rb1, and Rc1, the average probability Rave1 of the overall facial expression element "happy" of the viewers Va to Vc is Rave1=(Ra1+Rb1+Rc1) / 3. Similarly, if the average probabilities of the facial expression element “sad” for viewers Va, Vb, and Vc are Ra2, Rb2, and Rc2, the average probability Rave2 of the overall facial expression element “sad” for viewers Va to Vc is Rave2 = (Ra2 + Rb2 + Rc2) / 3.

[0094] The server 3 may also calculate the positivity and negativity of the extracted viewer V for the video 60 as a percentage based on the average probability of each overall facial expression element of the extracted viewer V. Furthermore, the server 3 may transmit information indicating the positivity and negativity to the company terminal 4. As shown in FIG. 9, the information indicating the positivity and negativity may be displayed in the facial expression information display area 72. For example, when the average probability of the facial expression element "happy" is high, the value of the positivity may be large. On the other hand, when the average probability of the facial expression element "anger" is high, the value of the negativity may be large. The server 3 may calculate the positivity and negativity from the average probability of each overall facial expression element by using a relational expression or a learning model in which the average probability of each overall facial expression element is set as an input value and the values ​​of the positivity and negativity are set as output values.

[0095] The comprehensive facial expression information in the form of a time series graph indicates the time change of each comprehensive facial expression element (happy, sad, neutral, angry, surprised) of the viewer V who matches the extraction condition. For example, the comprehensive facial expression information in the form of a time series graph may include information indicating the time change of the average probability of the facial expression element "happy" of multiple viewers V who match the extraction condition, information indicating the time change of the average probability of the facial expression element "sad" of the multiple viewers V, information indicating the time change of the average probability of the facial expression element "neutral" of the multiple viewers V, information indicating the time change of the average probability of the facial expression element "anger" of the multiple viewers V, and information indicating the time change of the average probability of the facial expression element "surprise" of the multiple viewers V.

[0096] For example, if the extracted viewer V includes viewers Va, Vb, and Vc, the server 3 calculates information indicating the change over time of the average probability of the facial expression element "happy" for viewers Va to Vc, information indicating the change over time of the average probability of the facial expression element "sad" for viewers Va to Vc, information indicating the change over time of the average probability of the facial expression element "neutral" for viewers Va to Vc, information indicating the change over time of the average probability of the facial expression element "anger" for viewers Va to Vc, and information indicating the change over time of the average probability of the facial expression element "surprise" for viewers Va to Vc, and transmits this information to the company terminal 4 as comprehensive facial expression information in the form of a time series graph.

[0097] The server 3 may also calculate overall interest level information in the form of a time series graph based on the overall facial expression information in the form of a time series graph. The overall interest level information indicates a time change in the overall interest level (percentage %) of the viewer V who meets the extraction condition in the video 60. For example, if the average probability of each overall facial expression element at the playback time t1 is R1 to R5, the overall interest level I at the playback time t1 may be calculated based on the average probability R1 to R5. In this regard, the server 3 may calculate the overall interest level I from the average probability R1 to R5 of each overall facial expression element by using a relational expression or a learning model in which the average probability R1 to R5 of each overall facial expression element is set as an input value and the overall interest level I is set as an output value.

[0098] Furthermore, in this embodiment, when the extraction conditions for the viewer V are all the viewers V, the overall facial expression information indicates the overall facial expression information of all the viewers V, and the overall interest level information indicates the change over time in the overall interest level of all the viewers V in the video 60. In this way, the overall facial expression information and the overall interest level information change according to the extraction conditions for the viewer V (in other words, the attributes of the viewer V to be extracted).

[0099] In addition, in this embodiment, the overall interest level information of the multiple viewers V is calculated based on the overall facial expression information (particularly, the change over time in the average probability of each facial expression element of the viewer V), but the overall interest level information may be calculated based on the gaze point data of each viewer V. Specifically, the overall interest level information is calculated based on the interest level data of each viewer V, and the interest level data of the viewer V may be calculated based on the gaze point data of the viewer V, information on the blinking of the viewer V, and drowsiness information of the viewer V. The information on the blinking of the viewer V may be information indicating the number of times the viewer V blinks or information indicating whether or not the viewer V blinks. The information on the blinking of the viewer V may be specified based on each frame of the video data of the viewer V. The drowsiness information of the viewer V may be information indicating whether or not the viewer V is drowsy. The drowsiness information of the viewer V may be specified based on each frame of the video data of the viewer V. For example, the interest level data of the viewer V may be calculated based on information on whether or not the position of the gaze point P of the viewer V is within the display area S of the video, information on whether or not the viewer V is drowsy, and information on whether or not the viewer V blinks. Based on the interest level data of each viewer V, it is possible to calculate overall interest level information of a plurality of viewers V.

[0100] In addition, in this embodiment, information indicating the positivity and negativity of multiple viewers V is calculated based on facial expression data of each viewer V, but this information may also be calculated based on gaze point data of each viewer V.

[0101] In this embodiment, the order of the processes shown in Fig. 8 is not particularly limited. In the examples shown in Figs. 9 and 13, a composite video 63 in which a gaze point heat map 62 is superimposed on a video 60 is displayed in the video display area 71, but the video 60 may be displayed in the video display area 71 instead of the composite video 63. Even in this case, information on an object of interest in the video 60 may be displayed in the object of interest display area 76. Furthermore, comprehensive facial expression information in a radar chart format may be displayed in the facial expression information display area 72, and comprehensive facial expression information and comprehensive interest level information in a time series graph format may be displayed in the facial expression and interest level information display area 73.

[0102] In this example, two pieces of information, the overall facial expression information and the overall interest level information, are displayed on the video viewing analysis screen 70, but only one of the two pieces of information, the overall facial expression information and the overall interest level information, may be displayed on the video viewing analysis screen 70. In this example, five types of facial expression elements, happy, sad, neutral, angry, and surprised, are used as the facial expression information, but the types of facial expression elements are not limited to these. For example, fear, happiness, enjoyment, disgust, etc. may be further used as other facial expression elements.

[0103] According to this embodiment, the company terminal 4 is presented with a composite video 63 in which the gaze point heat map 62 is superimposed on the video 60 in a state in which the gaze point heat map 62 and the video 60 are synchronized in time, so that the user U can grasp in detail the viewing behavior of the multiple viewers V for the video 60 by checking the composite video 63 displayed on the video viewing analysis screen 70. In particular, the user U can grasp in detail the attention objects (attention areas) in the video 60 by the multiple viewers V for each playback time of the video 60. In this way, it is possible to provide detailed insights on the video 60 (more specifically, detailed insights on the digital marketing of video advertisements) to the company that desires video analysis.

[0104] Furthermore, since information regarding the object of interest in the video 60 is displayed on the video viewing analysis screen 70, the user U can obtain a more detailed insight regarding the object of interest present in the video 60 by checking information regarding the object of interest (particularly, a cropped image of the object of interest, attention level information or average attention level information of the object of interest, the appearance time of the object of interest, etc.).

[0105] In particular, in this embodiment, a gaze point heat map is generated that visualizes the temporal change in the position of the gaze point of the viewer V who satisfies the attribute conditions based on gaze point data of the viewer V that matches the extraction conditions associated with the age information, gender information, household number information, etc. of the multiple viewers V. In this way, it is possible to provide the company with more detailed insights about the video 60 according to the viewer attributes.

[0106] Furthermore, according to this embodiment, the overall facial expression information and the overall interest level information in the form of a radar chart and a time series graph are presented on the company terminal 4, so that the user U can check this information displayed on the video viewing analysis screen 70 to grasp in detail the viewing behavior of multiple viewers V for the video 60. In particular, the user U can grasp in detail the facial expressions (happy, sad, neutral, angry, surprised, etc.) and interest levels of the viewers V for the video 60 for each playback time. In this way, it is possible to provide detailed insights about videos (more specifically, detailed insights about the digital marketing of video advertisements) to companies that wish to analyze videos.

[0107] In particular, in this embodiment, based on facial expression data of viewers V that match extraction conditions associated with age information, gender information, household size information, and the like of multiple viewers V, comprehensive facial expression information is generated that indicates probability information of each comprehensive facial expression element of the viewers V that satisfy these attribute conditions, and comprehensive interest level information is generated that indicates the comprehensive interest level of the viewers V that match the extraction conditions in the video 60. In this way, through this information, the viewing behavior of the viewers V that match the extraction conditions can be grasped in detail, making it possible to provide the company with more detailed insights into the video 60.

[0108] Although the embodiment of the present invention has been described above, the technical scope of the present invention should not be interpreted as being limited by the description of the embodiment. The embodiment is an example, and it is understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0109] 1: video viewing analysis system, 2, 2a, 2b: viewer terminal, 3: server, 4: company terminal, 8: communication network, 20: control unit, 21: storage device, 22: imaging unit, 23: communication unit, 24: input operation unit, 25: display unit, 26: speaker, RTC: 28, 30: control unit, 31: storage device, 32: input / output interface, 33: communication unit, 34: input operation unit, 35: display unit, 40: control unit, 41: storage device, 42: input / output interface, 43: communication unit, 44: input Operation unit, 45: display unit, 60: video, 62: gaze point heat map, 63: composite video, 70: video viewing analysis screen, 71: video display area, 72: facial expression information display area, 73: interest level information display area, 74: viewer selection button, 75: video selection button, 76: attention object display area, 80: extraction condition display screen, 125: display screen, 176: trimmed image, K: facial feature points, L: gaze, M: video data, P: gaze point, S: display area, U: user, V, Va, Vb: viewer

Claims

1. A video viewing analysis system that analyzes viewer insights into a video, Acquire a plurality of gaze point data corresponding to each of a plurality of viewers, generating a gaze point visualization map based on at least some of the plurality of gaze point data, the gaze point visualization map visualizing a gaze point when a viewer corresponding to the gaze point data viewed the video; The gaze point visualization map is displayed superimposed on the video. Video viewing analysis system.

2. A video viewing analysis system that analyzes viewer insights into a video, Acquire a plurality of gaze point data corresponding to each of a plurality of viewers, Identifying an object of attention of a viewer who viewed the video corresponding to the gaze point data based on at least some of the plurality of gaze point data; displaying information about the object of interest; Video viewing analysis system.

3. A video viewing analysis system that analyzes viewer insights into a video, Acquire a plurality of pieces of facial expression data corresponding to each of a plurality of viewers, generating comprehensive facial expression information indicating a comprehensive facial expression of a viewer corresponding to the facial expression data when viewing the video, based on at least some of the plurality of facial expression data; Displaying the comprehensive facial expression information. Video viewing analysis system.

4. A video viewing analysis system that analyzes viewer insights into a video, Acquire a plurality of pieces of facial expression data corresponding to each of a plurality of viewers, generating overall interest level information indicating an overall interest level of a viewer corresponding to the facial expression data when viewing the video, based on at least some of the plurality of facial expression data; Displaying the overall interest level information. Video viewing analysis system.

5. Inputting extraction conditions for the plurality of viewers; extracting at least some of the plurality of gaze point data based on the extraction condition; The video viewing analysis system according to claim 1 or 2.

6. Inputting extraction conditions for the plurality of viewers; extracting at least some of the plurality of facial expression data based on the extraction conditions; The video viewing analysis system according to claim 3 or 4.

7. The extraction conditions are: Age information of the plurality of viewers; Gender information of the plurality of viewers; address information of the plurality of viewers; Occupational information of the plurality of viewers; Information on the number of viewers in a household; Marriage information of the plurality of viewers; Educational background information of the plurality of viewers; is associated with at least one of The video viewing analysis system according to claim 5 .

8. The extraction conditions are: Age information of the plurality of viewers; Gender information of the plurality of viewers; address information of the plurality of viewers; Occupational information of the plurality of viewers; Information on the number of viewers in a household; Marriage information of the plurality of viewers; Educational background information of the plurality of viewers; is associated with at least one of The video viewing analysis system according to claim 6.

9. The video includes advertising content. The video viewing analysis system according to any one of claims 1 to 4.

10. The gaze point visualization map is a heat map. The video viewing analysis system according to claim 1 .

11. The gaze point visualization map and the video are displayed in a time-synchronized state. The video viewing analysis system according to claim 1 .

12. The gaze point data indicates a temporal change in a gaze point corresponding to a line of sight of the viewer on a display area of ​​the moving image. The video viewing analysis system according to claim 1 or 2.

13. Displaying the attention objects in a ranking format. The video viewing analysis system according to claim 2 .

14. displaying at least one of the attention level, appearance time, and average attention level of the attention object in association with the attention object; The video viewing analysis system according to claim 2 .

15. The comprehensive facial expression information indicates a probability of each of a plurality of facial expression elements throughout the entire video. The video viewing analysis system according to claim 3 .

16. The comprehensive facial expression information indicates a change in probability of each of a plurality of facial expression elements corresponding to a time series of the video. The video viewing analysis system according to claim 3 .

17. the overall interest level information indicates a transition of the interest level corresponding to a time series of the video; The video viewing analysis system according to claim 4.

18. Calculating a positive or negative degree of the viewer with respect to the video based on the probability of each of a plurality of facial expression elements; Displaying the positive or negative degree; The video viewing analysis system according to claim 3 .

19. A video viewing analysis method for analyzing viewer insights regarding a video, comprising: acquiring a plurality of gaze point data corresponding to a plurality of viewers; generating a gaze point visualization map based on at least some of the plurality of gaze point data, the gaze point visualization map visualizing a gaze point when a viewer corresponding to the gaze point data viewed the video; A step of displaying the gaze point visualization map by superimposing it on the video; A video viewing analysis method executed by a computer, comprising:

20. A video viewing analysis method for analyzing viewer insights regarding a video, comprising: acquiring a plurality of gaze point data corresponding to a plurality of viewers; Identifying an object of attention of a viewer who viewed the video corresponding to at least some of the plurality of gaze point data based on the plurality of gaze point data; displaying information about the object of interest; A video viewing analysis method executed by a computer, comprising:

21. A video viewing analysis method for analyzing viewer insights regarding a video, comprising: acquiring a plurality of pieces of facial expression data corresponding to a plurality of viewers; generating comprehensive facial expression information indicating a comprehensive facial expression of a viewer corresponding to the facial expression data when viewing the video, based on at least some of the plurality of facial expression data; displaying the integrated facial expression information; A video viewing analysis method executed by a computer, comprising:

22. A video viewing analysis method for analyzing viewer insights regarding a video, comprising: acquiring a plurality of pieces of facial expression data corresponding to a plurality of viewers; generating overall interest level information indicating an overall interest level of a viewer corresponding to the facial expression data when viewing the video, based on at least some of the plurality of facial expression data; displaying the overall interest level information; A video viewing analysis method executed by a computer, comprising:

23. A video viewing analysis program for causing a computer to execute the video viewing analysis method according to any one of claims 19 to 22.

Images