Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program for outputting information regarding a line-of-sight of a subject with respect to a place that the subject looked at first, a place that the subject looked at in the middle, or a place that the subject looked at last.SOLUTION: In an information processing system in which a camera, a model generating apparatus 110, and a line-of-sight estimating apparatus 140 are communicatively connected via a network, the model generating apparatus 110 inputs image data and time information into a learned model. The learned model learns line-of-sight data with respect to the image data as output data by using the image data and the time information as input data. The line-of-sight estimating apparatus 140 acquires the line-of-sight data relating to the input image data at the input time information from the learned model, and outputs information relating to the line-of-sight with respect to the image data based on the acquired line-of-sight data.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique aimed at estimating the gaze direction of a subject with high accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7310931 Summary of the Invention [Problem to be solved by the invention]

[0004] Depending on the information contained in the target image, there may be cases where the user wishes to know the location the subject first looked at, the location the subject looked at in the middle of looking at the image, or the location the subject last looked at. [Means for solving the problem]

[0005] According to one aspect of the present invention, an information processing system is provided. The information processing system acquires image data. The image data and time information are input to a trained model. The trained model is a trained model trained using the image data and time information as input data and gaze data with respect to the image data as output data. The gaze data is actual gaze data of a person who viewed the image data. Gaze data related to the input image data at the input time information is acquired from the trained model. Information regarding the gaze with respect to the image data is output based on the acquired gaze data. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the model generating device. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the gaze estimation device. [Figure 4] FIG. 4 is a sequence diagram showing an example of information processing related to generation of a trained model in an information processing system. [Diagram 5] FIG. 5 is a diagram illustrating an example of a learning dataset. [Figure 6] FIG. 6 is a sequence diagram showing an example of information processing regarding gaze estimation in the information processing system. [Figure 7] FIG. 7 is a diagram showing an example of a target image acquired by the control unit. [Figure 8] FIG. 8 is a diagram showing an example of a superimposed image output to an output device. [Figure 9] FIG. 9 is a diagram showing an example of changes in heat map data output when the time information input to the trained model is changed. [Figure 10] FIG. 10 is a diagram showing an example of a learning dataset in the first modification. [Figure 11] FIG. 11 is a diagram showing an example of a superimposed image. [Figure 12] FIG. 12 is a diagram showing an example of a superimposed image. [Figure 13] FIG. 13 is a diagram illustrating an example of a system configuration of an information processing system according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of the gaze estimation device according to the second embodiment. [Figure 15] FIG. 15 is a flowchart illustrating an example of information processing in the gaze estimation device of the second embodiment. [Figure 16] FIG. 16 is a diagram showing an example of a target image divided into N parts. [Figure 17] FIG. 17 is a diagram showing an example of a superimposed image. [Figure 18] FIG. 18 is a diagram showing an example of a screen for changing the division line based on a user operation. [Figure 19] FIG. 19 is a diagram showing an example of dividing a target image by automatically changing the position of a dividing line. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0008] <Embodiment 1> 1. System configuration diagram Fig. 1 is a diagram showing an example of a system configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes, as a system configuration, a model generating device 110, a camera 120, and a gaze estimating device 140. The model generating device 110 and the gaze estimating device 140 are communicatively connected via a network 150. The model generating device 110 and the camera 120 are also communicatively connected via a network or the like.

[0009] A subject 130 visually views a learning target image (hereinafter, simply referred to as a learning target image). Although only one subject is shown in Fig. 1 for the sake of simplicity, there are a plurality of subjects with different attributes.

[0010] The camera 120 captures an image of the subject 130 looking at a learning target image. The camera 120 transmits the captured data to the model generating device 110. Although only one camera is shown in FIG. 1 for the sake of simplicity, the subject 130 may be captured by multiple cameras. The model generating device 110 extracts gaze data of the subject based on the received captured data, and executes a process of generating a trained model.

[0011] The gaze estimation device 140 acquires a trained model from the model generation device 110, acquires heat map data of the gaze for the image data using the trained model, and outputs the acquired heat map data.

[0012] Here, the information processing system described in the claims may be composed of multiple devices or may be composed of one device. When the information processing system described in the claims is composed of one device, an example of the device is the gaze estimation device 140. When the information processing system described in the claims is composed of multiple devices, an example of the multiple devices is the model generation device 110 and the gaze estimation device 140.

[0013] In the specification, the model generating device 110 and the gaze estimating device 140 are described as separate entities, but they may be integrated into one device. Also, in the specification, the model generating device 110 and the camera 120 are described as separate entities, but they may be integrated into one device.

[0014] 2. Hardware Configuration (1) Hardware Configuration of Model Generator 110 FIG. 2 is a diagram illustrating an example of a hardware configuration of the model generating device 110. As shown in FIG. As shown in FIG. 2, the model generating device 110 includes, as its hardware configuration, a control unit 210, a storage unit 220, an input unit I / F 230, an output unit I / F 240, and a communication unit 250.

[0015] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the model generating device 110 as a whole.

[0016] The storage unit 220 is any one of a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), a solid state drive (SSD), etc., or an arbitrary combination of these, and stores a program, data used when the control unit 210 executes a process based on the program, etc. The storage unit 220 is an example of a storage medium. In the specification, the data used when the control unit 210 executes a process based on the program is described as being stored in the storage unit 220, but the data may be stored in a storage unit of another device that can communicate with the model generation device 110. In other words, the data may be stored in a storage unit of any device as long as the control unit 210 can refer to the data. The control unit 210 executes a process based on the program stored in the storage unit 220, thereby realizing the function of the model generation device 110 and the process of the model generation device 110 in the sequence diagram shown in FIG. 4 described later.

[0017] The input unit I / F 230 is an interface (I / F) that connects the model generation device 110 to an input device. The input device is, for example, a keyboard and / or a mouse, and is a device that inputs information to the model generation device 110 in response to an operation by an operator.

[0018] The output unit I / F 240 is an interface (I / F) that connects the model generating device 110 and an output device. The output device is, for example, a display, and is a device that outputs information.

[0019] The communication unit 250 connects the model generation device 110 to the network 150 and is responsible for communication with other devices.

[0020] (2) Hardware configuration of the gaze estimation device 140 FIG. 3 is a diagram showing an example of a hardware configuration of the gaze estimation device 140. As shown in FIG. As shown in FIG. 3, gaze estimation device 140 includes, as its hardware configuration, a control unit 310, a storage unit 320, an input unit I / F 330, an output unit I / F 340, and a communication unit 350.

[0021] The control unit 310 is a CPU or the like, and controls the entire gaze estimation device 140.

[0022] The storage unit 320 is any one of HDD, ROM, RAM, SSD, etc., or any combination thereof, and stores a program, data used when the control unit 310 executes a process based on the program, etc. The storage unit 320 is an example of a storage medium. In the specification, the data used when the control unit 310 executes a process based on the program is described as being stored in the storage unit 320, but the data may be stored in a storage unit of another device that can communicate with the gaze estimation device 140. That is, the data may be stored in a storage unit of any device as long as the control unit 310 can refer to the data. The control unit 310 executes a process based on the program stored in the storage unit 320, thereby realizing the function of the gaze estimation device 140 and the process of the gaze estimation device 140 in the sequence diagram shown in FIG. 6 described later.

[0023] The input unit I / F 330 is an interface (I / F) that connects the gaze estimation device 140 and an input device. The input device is, for example, a keyboard and / or a mouse, and is a device that inputs information to the gaze estimation device 140 in response to an operation by an operator.

[0024] The output unit I / F 340 is an interface (I / F) that connects the gaze estimation device 140 and an output device. The output device is, for example, a display, and is a device that outputs information.

[0025] The communication unit 350 connects the gaze estimation device 140 to the network 150 and controls communication with other devices.

[0026] 3. Information Processing The information processing of the first embodiment will be described below.

[0027] (1) Generating a trained model FIG. 4 is a sequence diagram showing an example of information processing related to generation of a trained model in the information processing system 1000. In sequence SQ401, the camera 120 captures an image of the subject 130 looking at the learning target image. The camera 120 captures an image of the subject 130 looking at the learning target image for a predetermined period of time (for example, one minute). In sequence SQ402, the camera 120 transmits the captured image data to the model generating device 110. The model generating device 110 receives the captured image data transmitted from the camera 120.

[0028] Upon receiving the photographed data transmitted from the camera 120, in sequence SQ403, the control unit 210 of the model generating device 110 stores the photographed data in the storage unit 220 or the like. In sequence SQ404, the control unit 210 extracts data on the line of sight of the subject 130 looking at the learning target image (gaze data) from the photographic data stored in the storage unit 220 or the like. The gaze data is gaze data on where and for how long the subject looked at the actual learning target image when looking at the learning target image (image data). The gaze data is data that holds time information. The control unit 210 generates a learning data set 540 by associating a learning target image 510, time information 520 related to the measurement of the gaze data, and gaze data 530 measured at a time indicated by the time information for the learning target image. In one measurement, the learning data set 540 is generated with time information 502 of 0 to 10 seconds, 10 to 20 seconds, 20 to 30 seconds, 30 to 40 seconds, 40 to 50 seconds, and 50 to 60 seconds. As another example, the control unit 210 may generate a learning data set using the time information itself (for example, 10 seconds, 20 seconds, 30 seconds, etc.). As yet another example, the control unit 210 may generate a learning dataset using time information itself input via a screen displayed on an output device, etc. Fig. 5 is a diagram showing an example of a learning dataset 540. A learning target image is an image that includes at least two or more pieces of information among text information, photo information, illustration information, graph information, and table information. In sequence SQ405, the control unit 210 stores the generated learning data set 540 in the storage unit 220 or the like.

[0029] There are a plurality of learning target images. There are also a plurality of subjects who view the learning target images. The processes from sequence SQ401 to sequence SQ405 are repeatedly executed the number of times corresponding to the number of learning target images and subjects. By repeatedly executing the processes from sequence SQ401 to sequence SQ405, a plurality of learning data sets 540 are generated.

[0030] When a predetermined number of learning data sets 540 are generated, in sequence SQ406, the control unit 210 executes machine learning of the estimation model. Executing machine learning of the estimation model means training the estimation model so that, for the learning data set 540, a learning target image and time information are input, gaze data indicating where on the learning target image the user looked at at the time indicated by the time information is output, thereby generating a learned model. The learned model is a learned model that has been trained using image data and time information as input data, and gaze data on the image data as output data.

[0031] (2) Gaze estimation FIG. 6 is a sequence diagram showing an example of information processing related to gaze estimation in the information processing system 1000. Upon receiving a request from the gaze estimation device 140, in sequence SQ601, the control unit 210 transmits the trained model to the gaze estimation device 140. Upon receiving the trained model transmitted from the model generation device 110, in sequence SQ602, the control unit 310 stores the trained model in the memory unit 320.

[0032] In sequence SQ603, control unit 310 acquires a target image for estimating the line of sight. Control unit 310 may acquire a target image stored in storage unit 320 in response to an operation by an operator via an input device or the like, or may acquire a target image to be processed based on a file name or the like of the target image stored in storage unit 320. The target image (image data) is an image that includes at least two or more pieces of information among character information, photo information, illustration information, graph information, and table information. FIG. 7 is a diagram showing an example of a target image 700 acquired by control unit 310. In target image 700, as an example, character information 710, photo information 720, and illustration information 730 are included in the image. The process of sequence SQ603 is an example of a process of acquiring image data.

[0033] In sequence SQ604, control unit 310 acquires time information. Control unit 310 may acquire time information input via a screen or the like in response to an operation by an operator via an input device or the like, or may acquire time information set in association with the target image acquired in sequence SQ603 from storage unit 320 or the like.

[0034] In sequence SQ605, the control unit 310 inputs the target image acquired in sequence SQ603 and the time information acquired in sequence SQ604 into the learned model, and acquires gaze data related to the input image data at the input time information from the learned model.

[0035] In sequence SQ606, the control unit 310 outputs heat map data of the gaze for the image data based on the acquired gaze data. More specifically, the control unit 310 generates heat map data based on the acquired gaze data, and outputs the generated heat map data to the output device by superimposing it on the target image acquired in sequence SQ603. The heat map data is a graph that represents the certainty of the gaze for the image data by color shading. The heat map data is an example of information about the gaze. The information about the gaze is not limited to heat map data, and may be, for example, time series data of the gaze. The time series data of the gaze may include a number (e.g., 1, 2, 3, etc.) indicating the gaze transition so that the order of the gaze transition can be known, or may include information indicating the actual movement of the gaze transition, such as from where to where the gaze changed, like a footprint. Furthermore, the control unit 310 may, for example, acquire gaze data for each specified time information, compare the gaze data for each time information, and determine that gaze data that matches more than a first threshold has a high similarity, and gaze data that does not match more than a second threshold has a low similarity. Then, the control unit 310 may superimpose gaze data with a high similarity (a location that attracts gazes at all times) on the target image in response to a request, or superimpose gaze data with a low similarity (a location that does not attract gazes much at different times) on the target image.

[0036] Fig. 8 is a diagram showing an example of a superimposed image 800 output to an output device. In Fig. 8, data 810 indicates heat map data. In the example of Fig. 8, it is shown that the text information in the lower right is viewed the most.

[0037] FIG. 9 is a diagram showing an example of changes in heat map data output when the time information input to the trained model is changed. The heat map data superimposed on the superimposed image 900 is heat map data when the time information input to the trained model is 0 to 10 seconds. The heat map data superimposed on the superimposed image 910 is heat map data when the time information input to the trained model is 10 to 20 seconds. The heat map data superimposed on the superimposed image 920 is heat map data when the time information input to the trained model is 20 to 30 seconds. As shown in FIG. 9, the location of the target image being viewed varies depending on the time.

[0038] According to the process of the first embodiment, information on the line of sight of the target image at various times can be obtained. For example, it is possible to know the place where the subject looked first, the place where the subject looked in the middle, or the place where the subject looked last. That is, information on the line of sight at different times (for example, information on the field of view) can be obtained.

[0039] (Variation 1) A first modification of the first embodiment will be described. The control unit 210 of the first modification generates a learning dataset 540 including attribute information 1010 of the subject. The attribute information of the subject is information indicating the attributes of the subject. The attributes of the subject include age (teens, twenties, thirties, forties, fifties, sixties, seventies or more), gender, and personality traits. The personality traits are information indicating the characteristics of the subject's personality, such as extroversion, neuroticism, openness, diligence, and cooperation. For example, the control unit 210 displays a Web screen on the terminal device for conducting a questionnaire on personality traits in response to a request from the terminal device of the subject, and identifies the personality traits of the subject based on the results of the questionnaire via the Web screen. As another example, a questionnaire or the like containing a questionnaire is distributed to the subject, and the results of the questionnaire written on the collected questionnaire are input by a person in charge or the like via an input device or the like. The control unit 210 identifies the personality traits of the subject based on the results of the questionnaire input via the input device. FIG. 10 is a diagram showing an example of the learning dataset 540 in the first modification. The learning dataset 540 of the first modification includes attribute information 1010 of the subjects. An example of the attribute information 1010 is "50s," "male," "considerateness," and the like.

[0040] The control unit 210 trains an estimation model for the learning dataset 540 of variant example 1 so that, in response to input of a learning target image, time information, and attribute information, it outputs gaze data indicating where on the learning target image a subject with attributes indicated by the attribute information was looking at at the time indicated by the time information, and generates a learned model.

[0041] The control unit 310 of the first modification acquires attribute information to be input to the trained model. The control unit 310 may acquire attribute information input via a screen or the like in response to an operation by an operator via an input device or the like, or may acquire attribute information set in association with the target image acquired in sequence SQ603 from the storage unit 320 or the like.

[0042] The control unit 310 inputs the acquired target image, time information, and attribute information into the trained model, and acquires gaze data related to the input image data in the input time information and attribute information from the trained model.

[0043] Then, the control unit 310 generates a heat map of the gaze for the image data based on the acquired gaze data, generates an image in which the generated heat map data is superimposed on the target image, and outputs the image to an output device.

[0044] Fig. 11 is a diagram showing an example of a superimposed image 1100. In the superimposed image 1100, heat map data of time information shown from 0 to 10 seconds of a woman in her 30s who has an extroverted personality trait is superimposed on the target image. Fig. 12 is a diagram showing an example of a superimposed image 1200. In the superimposed image 1200, heat map data of time information shown from 0 to 10 seconds of a man in his 30s who has an extroverted personality trait is superimposed on the target image.

[0045] Another example of the gaze-related information in the first modification is information on the similarity of gazes in different attributes. For example, the control unit 310 acquires gaze data for each specified attribute, compares the gaze data for each attribute, and determines that gaze data that matches more than a first threshold has a high degree of similarity, and gaze data that does not match more than a second threshold has a low degree of similarity. Then, the control unit 310 may superimpose gaze data with a high degree of similarity (a portion that attracts the gazes of all attributes) on the target image in response to a request, or superimpose gaze data with a low degree of similarity (a portion that does not attract much gaze if the attributes are different) on the target image.

[0046] According to the process of the first modification, it is possible to obtain information related to the line of sight according to the attribute of the target image. That is, it is possible to obtain information related to the line of sight (for example, information on the field of view) with different attributes.

[0047] (Variation 2) A second modification of the first embodiment will be described. The measured gaze data of the subject includes gaze parameters, which include information on pupil diameter, saccades, blinks, and fixations. For example, the control unit 210 of the model generating device 110 of the second modified example generates a trained model by using gaze data in which the pupil diameter is equal to or larger than a threshold among the measured gaze data. This is because the pupil becomes larger when viewing an object that the user is interested in or has a liking for.

[0048] Also, for example, the control unit 210 generates a trained model by using gaze data in which saccades (or saccades) are below a threshold among the measured gaze data. This is because it is said that visual information is almost completely blocked during the saccade period due to processing in the visual cortex of the brain. Also, for example, the control unit 210 generates a trained model by using gaze data in which the number of blinks is equal to or less than a threshold among the measured gaze data. This is because the number of blinks decreases when the user is concentrating compared to when the data is not collected.

[0049] Furthermore, for example, the control unit 210 generates a trained model using data in which the gaze retention time is equal to or longer than a threshold among the measured gaze data. This is because it can be determined that visual information processing is being performed when the gaze remains in a certain area for a certain period of time.

[0050] According to the second variant, a trained model can be generated using gaze data indicating interest or concentration. Note that any combination of the gaze parameters described in Modification 2 may be used. For example, a trained model may be generated using data in which the pupil diameter is equal to or greater than a first threshold and the gaze retention time is equal to or greater than a second threshold.

[0051] (Other variations) For example, some or all of the processing of the model generator 110 may be performed by the camera 120 . The model generation device 110 may also provide the use of the generated trained model in the form of a Web service. For example, the model generation device 110 accepts input data to be input to the trained model from a terminal device or the like. The model generation device 110 then inputs the accepted input data to the trained model, and transmits output data output from the trained model to the requesting terminal device or the like. Even with such a configuration, the above-mentioned effects can be achieved.

[0052] <Embodiment 2> 1. System configuration diagram Hereinafter, the second embodiment will be described. Note that the second embodiment will mainly be described with respect to the differences from the first embodiment. The second embodiment is an embodiment in which further functions and the like are added to the first embodiment, and the configuration and processing and the like that are not described below are the same as those of the first embodiment. As described later, in the second embodiment, an example is shown in which the gaze of a target image is estimated while taking into consideration the actual gaze of a user captured by a web camera, and information related to the gaze is output. In particular, an example is shown in which the gaze of a target image is estimated using photographed data captured by a camera of a normal PC (Personal Computer) or the like, without using an infrared sensor or the like used in eye tracking technology.

[0053] 13 is a diagram showing an example of a system configuration of an information processing system 1000 according to the second embodiment. The second embodiment differs from the first embodiment in that a user 160 is present and views a target image for which gaze estimation is to be performed. For the sake of simplicity, the second embodiment will be described assuming that the target image is displayed on an output device connected to the output unit I / F 340 of the gaze estimation device 140 according to the second embodiment.

[0054] 2. Hardware Configuration 14 is a diagram showing an example of a hardware configuration of the gaze estimation device 140 of the embodiment 2. Compared to the hardware configuration of the gaze estimation device of the embodiment 1, the hardware configuration of the gaze estimation device 140 of the embodiment 2 additionally includes an imaging unit 360.

[0055] The photographing unit 360 is a camera without an infrared sensor, more specifically, a Web camera. The photographing unit 360 may be built into the gaze estimation device 140, or may be externally attached via a USB (Universal Serial Bus) connection or the like. The photographing unit 360 photographs the user 160 as a subject. More specifically, the photographing unit 360 photographs the face, particularly the eyes, of the user 160 as a subject. In the case where the photographed data photographed by the photographing unit 360 does not include the subject's eyes, the control unit 310 of the second embodiment may output a message to a predetermined screen or the like to adjust the inclination of the photographing unit 360 so that the eyes are included. Taking a PC or a notebook PC as a more specific example of the gaze estimation device 140, the image capturing unit 360 is a Web camera that is built into the PC or notebook PC in advance. Also, the image capturing unit 360 can be said to be a camera having a degree of accuracy (resolution) sufficient to distinguish area division of the target image.

[0056] 3. Information Processing The information processing of the second embodiment will be described below. 15 is a flowchart showing an example of information processing in the gaze estimation device 140 of embodiment 2. For example, when a predetermined operation is received via the screen, the control unit 310 starts the process of step S1510. In step S1510, the control unit 310 determines whether or not the image capturing data has been received from the image capturing unit 360. The image capturing data is image capturing data of a user viewing a target image displayed on an output device connected to the output unit I / F 340. As described above, the image capturing data includes the user's eyes as a subject. If the control unit 310 determines that the image capturing data has been received from the image capturing unit 360, the control unit 310 advances the process to step S1520. If the control unit 310 determines that the image capturing data has not been received from the image capturing unit 360, the control unit 310 repeats the process of step S1510.

[0057] In step S1520, control unit 310 acquires the user's line of sight data relating to the target image, based on the photographing data and position information indicating the display position on the output device of the target image being output to the output device. More specifically, control unit 310 divides the target image into N (N is a natural number equal to or greater than 2) areas (area division). FIG. 16 is a diagram showing an example of a target image divided into N areas. In the example of FIG. 16, control unit 310 shows an example in which the target image is divided into 20 areas. Each of the divided areas is called a segment. The target image is divided in order to estimate which segment the user's line of sight is on.

[0058] The control unit 310 inputs the divided target image and the user's photographed data into the trained model. The trained model is trained using the image divided into segments and the photographed data of the user looking at the image as input data, and gaze data showing which segment of the image the user's gaze is in as a probability distribution as output data. An example of gaze data shown as a probability distribution is heat map data. The control unit 310 acquires the gaze data output from the trained model. As another example, the control unit 310 may perform image processing on the captured data to determine eye movement contained in the captured data, and obtain gaze data that indicates, as a probability distribution, which segment of the image the user's gaze is on from the eye movement. That is, control unit 310 divides the image data into N pieces (N is a natural number of 2 or more), and acquires line-of-sight data for each region of the image data divided into N pieces based on the captured data. The process of step S1520 is an example of a process for acquiring gaze data of the user regarding the target image based on the captured data.

[0059] In step S1530, control unit 310 outputs heat map data of the gaze for the image data, based on the gaze data acquired in sequence SQ605 in the first embodiment and the gaze data acquired in step S1520. The process of step S1530 is an example of a process of outputting information about the gaze of the user with respect to image data, based on gaze data acquired from a trained model and gaze data of the user acquired based on the captured data.

[0060] 17 is a diagram showing an example of a superimposed image 1700. In the superimposed image 1700, heat map data 1710 based on the gaze data acquired in sequence SQ605 of the first embodiment and heat map data 1720 based on the gaze data acquired in step S1520 are superimposed on a target image. That is, the control unit 310 superimposes information regarding the gaze at the image data based on gaze data acquired from the trained model shown in embodiment 1 and information regarding the gaze at the image data based on the user's gaze data acquired based on the shooting data shown in embodiment 2 onto the target image and outputs the same. The control unit 310 may, for example, in response to a predetermined switching operation via the screen, superimpose only the heat map data 1710 on the target image, superimpose only the heat map data 1720 on the target image, or superimpose both the heat map data 1710 and the heat map data 1720 on the target image.

[0061] (Variation 1) A first modification of the second embodiment will be described. The control unit 310 of the first modification displays, on an output device connected to the line of sight estimation device 140 via the output unit I / F 340, a screen configured to allow a user to operate a dividing line (segment boundary line) of a target image divided into N pieces. Fig. 18 is a diagram showing an example of a screen 1800 for changing a dividing line based on a user operation. The dividing line dividing the target image on the screen 1800 is configured to be changeable in response to an operation of a movement instruction object 1830. The movement instruction object 1820 is an example of a GUI (Graphical User Interface) component for moving the dividing line. For example, the user moves the movement instruction object 1830 onto the dividing line to which the user wants to move (dividing line 1810 in the example of FIG. 18), and moves the dividing line to the left or right. Since dividing line 1810 is a line that divides the target image vertically, it can be moved to the left and sideways. In the case of dividing line 1820, which is a line that divides the target image horizontally, the user moves dividing line 1820 by moving it up or down via the movement instruction object 1830. These processes are an example of a process for dividing image data into N pieces based on a setting operation via a predetermined screen.

[0062] According to the first modification, the target image can be divided in response to a user's operation. As a result, for example, it is possible to prevent the target image from being divided on text information or photo information that is likely to attract the user's attention.

[0063] (Variation 2) A second modification of the second embodiment will be described. The control unit 310 of the second modification performs image analysis on the target image and identifies the positions of character information, photo information, illustration information, graph information, table information, and the like included in the target image. The control unit 310 may then divide the target image into N pieces based on this position information. That is, the control unit 310 may divide the target image into N pieces so that the character information, photo information, illustration information, graph information, table information, and the like included in the target image are not divided. Fig. 19 is a diagram showing an example in which the position of the dividing line is automatically changed to divide the target image. These processes are examples of processes for dividing image data into N pieces based on character information, photo information, or illustration information contained in the image data. Moreover, these processes can be said to be an example of a process for dividing image data into N pieces so as not to divide an area that includes character information, photographic information, or illustration information contained in the image data.

[0064] According to the second modification, the target image can be automatically divided so as to prevent the target image from being divided on text information, photo information, or the like that is likely to attract the user's attention.

[0065] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. For example, the modifications of the above-described embodiments may be implemented in any combination. [Explanation of symbols]

[0066] 110: Model generation device 120: Camera 130: Subject 140: Gaze estimation device 150: Network 210: Control section 220: Storage section 230: Input I / F 240: Output I / F 250: Communications Department 310: Control section 320: Storage section 330: Input I / F 340: Output I / F 350: Communications Department 502: Time information 510: Learning target image 520: Time information 530: Gaze data 540: Training dataset 700: Target image 710: Text information 720: Photo information 730: Illustration information 800: Overlay image 810: Data 900: Overlay image 910: Overlay image 920: Overlay image 1000: Information processing systems 1010: Attribute information 1100: Overlay image 1200: Overlay image

Claims

1. An information processing system, Acquire image data; The image data and time information are input into a trained model; The trained model is a trained model trained using image data and time information as input data and gaze data with respect to the image data as output data, The gaze data is actual gaze data of a person viewing the image data, Obtaining gaze data relating to the input image data at the input time information from the trained model; outputting information regarding a line of sight with respect to the image data based on the acquired line of sight data; Information processing system.

2. 2. The information processing system according to claim 1, The information about the line of sight is heat map data of the line of sight. Information processing system.

3. 2. The information processing system according to claim 1, The image data, time information, and attribute information are input into a trained model; The trained model is a trained model trained using image data, time information, and attribute information of a person who viewed the image data as input data, and gaze data on the image data as output data, The gaze data is actual gaze data of a person viewing the image data, Obtaining gaze data related to the input image data in the input time information and attribute information from the trained model; outputting information regarding a line of sight with respect to the image data based on the acquired line of sight data; Information processing system.

4. 4. The information processing system according to claim 3, The information about the line of sight is heat map data of the line of sight. Information processing system.

5. 4. The information processing system according to claim 3, The information about the line of sight is information about the similarity of the line of sight in different attributes. Information processing system.

6. 4. The information processing system according to claim 3, The attribute information includes the subject's age, sex, or personality traits. Information processing system.

7. 2. The information processing system according to claim 1, the gaze data includes gaze parameters; The gaze parameters include information regarding pupil diameter, saccades, blinks, or gaze, The trained model is a trained model trained using gaze data selected based on a threshold value for the gaze parameter. Information processing system.

8. 2. The information processing system according to claim 1, The image data includes at least two of text information, photo information, illustration information, graph information, and table information. Information processing system.

9. 2. The information processing system according to claim 1, acquiring gaze data of a user regarding the image data based on photographed data of the user viewing the image data; The photographing data is photographed by a camera, The photographed data includes the user's eyes as a subject, Outputting information regarding the gaze of the user with respect to the image data based on the gaze data acquired from the trained model and the gaze data of the user acquired based on the photographed data. Information processing system.

10. 10. The information processing system according to claim 9, Information regarding the gaze of the image data based on the gaze data acquired from the trained model and information regarding the gaze of the image data based on the gaze data of the user acquired based on the shooting data are superimposed and output. Information processing system.

11. 11. The information processing system according to claim 10, Divide the image data into N pieces (N is a natural number equal to or greater than 2); acquiring line-of-sight data for each of the N divided regions of the image data based on the photographing data; outputting information regarding a line of sight with respect to the image data based on the line of sight data; Information processing system.

12. 12. The information processing system according to claim 11, Dividing the image data into N pieces based on a setting operation via a predetermined screen; Information processing system.

13. 12. The information processing system according to claim 11, Dividing the image data into N pieces based on character information, photo information, illustration information, graph information, or table information contained in the image data; Information processing system.

14. 14. The information processing system according to claim 13, Dividing the image data into N pieces so as not to divide an area including character information, photographic information, illustration information, graph information, or table information included in the image data. Information processing system.

15. An information processing method executed by an information processing system, comprising: Acquire image data; The image data and time information are input into a trained model; The trained model is a trained model trained using image data and time information as input data and gaze data with respect to the image data as output data, The gaze data is actual gaze data of a person viewing the image data, Obtaining gaze data relating to the input image data at the input time information from the trained model; outputting information regarding a line of sight with respect to the image data based on the acquired line of sight data; Information processing methods.

16. A program, Computer, A program for causing the information processing system according to any one of claims 1 to 14 to function as such.

Images