Model generation method and inference program

Abstract

A model generation device according to one aspect of the present disclosure acquires eyeball-related data measured from a subject who is viewing content, uses the acquired eyeball-related data to perform machine learning of an inference model, and outputs the results of the machine learning. The machine learning includes training the inference model to acquire, from the eyeball-related data, the ability to infer a semantic representation in an information space corresponding to content included in the viewed content. Thus, the present disclosure provides a technique for easily inferring, at low cost, content perceived by an individual while viewing content.

Description

Model generation method and inference program

[0001] The present invention relates to a model generation method and an inference program.

[0002] How to quantify the effects of diverse content on diverse individuals and how to optimize content for diverse individuals are important issues in a wide range of fields, including education, entertainment, advertising, and policy.

[0003] With the rapid development of artificial intelligence technology, including large-scale language models, techniques are being developed to quantitatively represent the content contained in content expressed in various modalities as fixed-length vectors in any latent space, such as image feature space, semantic space, concept space, etc. For example, Non-Patent Document 1 proposes an architecture for calculating continuous vector representations of words from a large-scale dataset.

[0004] Furthermore, with the recent advances in quantitative modeling techniques for brain activity, techniques for decoding the content of perceptual experiences as vectors in a latent space are also being realized. For example, Patent Literature 1 proposes a method for measuring brain activity and estimating perceived semantic content by analyzing the measured brain activity.

[0005] Japanese Patent Application Laid-Open No. 2016-195716

[0006] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, “Efficient estimation of word representations in vector space”, [online], [Retrieved January 29, 2024] Internet <URL: https: / / arxiv.org / abs / 1301.3781>

[0007] In the information analysis method based on brain activity proposed in Patent Document 1 and the like, a large-scale measuring device such as fMRI (functional magnetic resonance imaging) is used to measure brain activity. This large-scale measuring device is expensive and highly restrictive. Therefore, there is a problem in that it is difficult to infer the content perceived by an individual in a simple and low-cost manner.

[0008] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide a technology for easily inferring what is perceived by an individual while viewing content at low cost.

[0009] In order to solve the above-mentioned problems, the present invention employs the following configurations, which can be combined as appropriate.

[0010] A model generation method according to one aspect of the present invention is an information processing method in which a computer executes the steps of acquiring eye-related data measured from a subject viewing content, performing machine learning on an inference model using the acquired eye-related data, and outputting a result of the machine learning, wherein the machine learning includes training the inference model to acquire the ability to infer, from the eye-related data, a semantic representation in an information space corresponding to content included in the content.

[0011] The present inventors have found through experimental examples described below that it is possible to infer from eye activity what an individual perceives while viewing content (i.e., eye-related data can be used as a signal source to replace brain activity data). Based on this finding, the present configuration generates a trained inference model (trained model) that has acquired the ability to infer what an individual perceives from content from eye-related data. Compared to brain activity, eye activity can be measured easily and at low cost. Therefore, with the present configuration, it is possible to use the generated trained model to infer what an individual perceives while viewing content easily and at low cost.

[0012] In the model generation method according to the above aspect, the computer may further execute a step of acquiring behavioral data indicating the behavior of the subject while viewing the content. The acquired behavioral data may be further used in machine learning of the inference model. In the machine learning, the inference model may be trained to acquire the ability to infer the semantic representation from the eye-related data and the behavioral data.

[0013] For example, behaviors while viewing content, such as searching for words that appear in the content or cheering for a player that appears in the content, may be triggered as a result of perceiving the content. In other words, because behaviors while viewing content are correlated with perceptual results, behavioral data of individuals while viewing content can provide clues for inferring perceived content. This configuration allows for the generation of a trained model that further accepts input of behavioral data. The generated trained model is expected to improve the accuracy of inferring what individuals perceive while viewing content by using the behavioral data together with eye-related data as explanatory variables.

[0014] In the model generation method according to the above aspect, the acquired eye-related data may be composed of gaze data, blink data, pupil diameter data, or a combination thereof. Gaze, blink data, and pupil diameter accurately represent eye activity. Therefore, according to this configuration, by using at least one of the gaze data, blink data, and pupil diameter data as eye-related data, it is possible to generate a trained model that has acquired the ability to appropriately infer perceptual content from eye activity.

[0015] Furthermore, aspects of the present invention may not be limited to the model generation stage. The present invention may also be directed to an inference stage that uses a trained inference model generated by the model generation method. For example, one aspect of the present invention may be an inference program that uses a trained inference model generated by the model generation method.

[0016] According to one aspect of the present invention, there is provided an inference program for causing a computer to execute the steps of: acquiring eye-related data measured from a subject viewing target content; inferring a semantic representation in an information space corresponding to content included in the target content from the acquired eye-related data using a trained inference model; and outputting a result of inferring the semantic representation. With this configuration, it is possible to infer the content perceived by an individual while viewing content simply and at low cost.

[0017] The inference program according to the above aspect may further cause the computer to execute a step of acquiring behavioral data indicating the behavior of the subject while viewing the target content. In the inference step, the computer may infer the semantic representation from the acquired eye-related data and behavioral data. This configuration is expected to improve the accuracy of inferring content perceived by an individual (subject).

[0018] Note that the present invention does not need to be limited to the above-described model generation method. As another aspect of the model generation method according to the above-described aspects, one aspect of the present invention may be an information processing device (model generation device) that realizes all or part of the above-described configurations, a program, or a storage medium readable by a machine such as a computer on which such a program is stored. A storage medium readable by a machine such as a computer is a medium that stores information such as a program through electrical, magnetic, optical, mechanical, or chemical action. Furthermore, the present invention does not need to be limited to the above-described inference program. As another aspect of the inference program according to the above-described aspects, one aspect of the present invention may be an information processing device (inference device) that realizes all or part of the above-described configurations, an information processing method (inference method), or a storage medium readable by a machine such as a computer on which a program is stored.

[0019] For example, a model generation device according to an aspect of the present invention may include a control unit. The control unit may be configured to execute the steps of acquiring eye-related data measured from a subject viewing content, performing machine learning on an inference model using the acquired eye-related data, and outputting a result of the machine learning. The machine learning may include training the inference model to acquire the ability to infer, from the eye-related data, a semantic representation in an information space corresponding to content included in the content.

[0020] For example, an inference device according to an aspect of the present invention may include a control unit configured to execute the steps of acquiring eye-related data measured from a subject viewing target content, inferring a semantic representation in an information space corresponding to content included in the target content from the acquired eye-related data using a trained inference model, and outputting a result of inferring the semantic representation.

[0021] For example, an inference method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of acquiring eye-related data measured from a subject viewing target content, inferring a semantic representation in an information space corresponding to the content contained in the target content from the acquired eye-related data using a trained inference model, and outputting the result of inferring the semantic representation.

[0022] According to the present invention, it is possible to provide a technology for easily and at low cost inferring what is perceived by an individual while viewing content.

[0023] FIG. 1 schematically illustrates an example of a scenario in which the present invention is applied. FIG. 2 schematically illustrates an example of the hardware configuration of a model generation device. FIG. 3 schematically illustrates an example of the hardware configuration of an inference device. FIG. 4 schematically illustrates an example of the software configuration of a model generation device. FIG. 5 schematically illustrates an example of the software configuration of an inference device. FIG. 6 is a flowchart illustrating an example of the processing procedure of the model generation device. FIG. 7 is a flowchart illustrating an example of the processing procedure of the inference device. FIG. 8 illustrates the relationship between principal component vectors and decoding models obtained in an experimental example. FIG. 9 illustrates the results of calculating the average inference accuracy of the inference model in an experimental example for each dimension of the principal component vector. FIG. 10 illustrates the results of expressing perceptual content inferred from pupil diameter using t-SNE. FIG. 11 illustrates the results of calculating the change in inference accuracy between individuals of the encoding model in an experimental example.

[0024] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0025] §1 Application Example Fig. 1 schematically shows an example of a scenario in which the present invention is applied. The system according to this embodiment includes a model generation device 1 and an inference device 2. The model generation device 1 is one or more computers configured to generate a trained inference model 5 by controlling the implementation of machine learning. The inference device 2 is one or more computers configured to infer the perceptual content of a subject TU using the trained inference model 5.

[0026] The model generation device 1 according to this embodiment acquires training eye-related data 31 measured from a subject LU viewing training content LC. The training content LC may also be referred to as training content. The model generation device 1 performs machine learning on an inference model 5 using the acquired eye-related data 31. The machine learning includes training the inference model 5 to acquire the ability to infer, from the eye-related data 31, a semantic representation 39 in an information space corresponding to the content included in the content LC. The semantic representation in the information space corresponds to the perceived content of the content. Therefore, this machine learning can generate a trained inference model 5 that has acquired the ability to infer the perceived content from the eye-related data measured while the content is being viewed. The model generation device 1 outputs the results of the machine learning. The generated trained inference model 5 may be provided to the inference device 2 in any manner and at any timing.

[0027] Meanwhile, the inference device 2 according to this embodiment acquires target eye-related data 41 measured from a target person TU viewing target content TC. Using a trained inference model 5, the inference device 2 infers a semantic representation 49 in information space corresponding to the content included in the target content TC from the acquired eye-related data 41. This semantic representation 49 corresponds to the content perceived by the target person TU with respect to the target content TC. The inference device 2 outputs the result of inferring the semantic representation 49.

[0028] Experimental examples described below have revealed that it is possible to infer from eye activity what is perceived by an individual while viewing content. Based on this finding, the model generation device 1 according to this embodiment generates a trained inference model 5 that has acquired the ability to infer from eye-related data what is perceived from content. By using this trained inference model 5, it is possible to infer simply and at low cost what is perceived by an individual while viewing content. The inference device 2 according to this embodiment can infer simply and at low cost what is perceived by a subject TU while viewing target content TC.

[0029] (Content) In one example, the content (content LC, target content TC) may be any information that can be viewed by a person. Viewing may mean at least one of watching and listening. As long as the content can be viewed, the data format of the content is not particularly limited and may be selected appropriately depending on the embodiment. For example, the content (content LC, target content TC) may be composed of text data, image data (video, still image), sound data, other numerical data, or a combination of these. As a typical example, the content may be video content that includes audio (e.g., video content for education / medical support, etc.). Note that the training content LC and the target content TC may be the same or different.

[0030] (Eye-related Data) In one example, the eye-related data (eye-related data 31, eye-related data 41) may be configured to indicate eye activity of a person (subject LU, target TU) while the person is viewing content. The activity may include a state. The eye activity may be expressed, for example, by gaze, blinking, pupil diameter, etc. As long as the eye-related data is configured in this manner, the configuration of the eye-related data is not particularly limited and may be selected appropriately depending on the embodiment.

[0031] In one example, the eyeball-related data (eyeball-related data 31, eyeball-related data 41) may be composed of gaze data, blink data, pupil diameter data, or a combination thereof. Gaze, blink data, and pupil diameter accurately represent eye activity. Therefore, according to one example of this embodiment, the model generation device 1 can be expected to generate a trained inference model 5 that has acquired the ability to properly infer perceptual content from eye activity. Furthermore, by using this trained inference model 5 in the inference device 2, proper inference of the subject TU's perceptual content in relation to the target content TC can be expected.

[0032] The eye-related data may be generated by measuring the activity of a person's eyeballs using any sensor. In a training data collection situation, a sensor SA may be used to appropriately collect training samples of the eyeball-related data 31. In an inference situation, a sensor SC may be used to acquire samples of the eyeball-related data 41. As long as the sensor can measure the activity of the eyeballs, the type of the sensor (sensor SA, sensor SC) is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the sensor (sensor SA, sensor SC) may include an imaging device such as an infrared camera. The sensor (sensor SA, sensor SC) may further include a device that assists in measurement, such as an infrared projector.

[0033] The data format of the eyeball-related data is not particularly limited and may be appropriately selected depending on the type of sensor and the embodiment. Furthermore, at least a portion of the eyeball-related data may be composed of sensing data obtained by a sensor, or may be composed of analysis data (analysis results) obtained by analyzing the sensing data. The analysis may include, for example, feature extraction, calculation of feature amounts, etc. The feature extraction may include, for example, identification of gaze direction, measurement of blinks, measurement of pupil diameter, etc. The feature amounts may include, for example, statistics such as maximum value, minimum value, median, mean value, variance, standard deviation, and n-th percentile value.

[0034] The model generation device 1 may acquire data directly from the sensor SA, or may acquire the data indirectly via another computer. When acquiring data directly from the sensor SA, the sensor SA may be connected to the model generation device 1. When acquiring data indirectly via another computer, the model generation device 1 may acquire the target data from the other computer via a network, a storage medium, etc. The model generation device 1 may acquire data measured in the past by the sensor SA, or may acquire data measured in real time by the sensor SA.

[0035] Similarly, the inference device 2 may acquire data directly from the sensor SC, or may acquire it indirectly via another computer. When acquiring data directly from the sensor SC, the sensor SC may be connected to the inference device 2. When acquiring data indirectly via another computer, the inference device 2 may acquire the target data from the other computer via a network, a storage medium, etc. The inference device 2 may acquire data that was measured in the past by the sensor SC, or may acquire data that is being measured in real time by the sensor SC.

[0036] (Other Data) In one example, data other than the eye-related data may be further used to infer a semantic expression. Figure 1 illustrates a situation in which, as an example of the other data, behavioral data indicating behavior while viewing content is further used to infer a semantic expression.

[0037] 1 , the model generation device 1 may be further configured to acquire behavioral data 33 indicating the behavior of the subject LU while viewing the content LC. The acquired behavioral data 33 may be further used in machine learning of the inference model 5. Accordingly, in the machine learning, the inference model 5 may be trained to acquire the ability to infer a semantic representation 39 from the eye-related data 31 and the behavioral data 33.

[0038] As described above, behaviors while viewing content, such as searching for words that appear in the content or cheering for a player that appears in the content, may be triggered as a result of perceiving the content. In other words, because behaviors while viewing content are correlated with perceptual results, behavioral data of individuals while viewing content can provide clues for inferring perceived content. Therefore, according to an example of this embodiment, by generating a trained inference model 5 that further accepts input of behavioral data, it is expected that the accuracy of inferring semantic expressions in the generated trained inference model 5 can be improved.

[0039] In one example, the inference device 2 may further be configured to acquire behavioral data 43 indicating the behavior of the subject TU while viewing the target content TC. In the inferring step, the inference device 2 may infer a semantic representation 49 from the acquired eye-related data 41 and behavioral data 43. According to one example of the present embodiment, by further taking the behavioral data 43 into consideration when inferring the semantic representation 49, improvement in the accuracy of the inference can be expected.

[0040] In one example, the behavioral data (behavioral data 33, behavioral data 43) may be configured to include information about behavior that may be related to the perception of content. As long as the behavioral data is configured in this manner, the configuration of the behavioral data is not particularly limited and may be determined appropriately depending on the embodiment. The behavioral data may be generated by measuring human behavior with any sensor. In a training data collection situation, the sensor SB may be used to appropriately collect training samples of the behavioral data 33. In an inference situation, the sensor SD may be used to acquire samples of the behavioral data 43.

[0041] As long as the sensors (SB, SD) can measure human behavior, the type of the sensors (SB, SD) is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the behavior of the subject observed while viewing content may include, for example, terminal operation (key operation, mouse operation, etc.), physical movement (gestures, etc.), vocalization (including speech), vital signs (body temperature, heart rate, pulse, blood pressure, electrocardiogram, electromyogram, sweating, activity level, etc.). Accordingly, the sensors (SB, SD) may include a computer, an imaging device, an acceleration sensor, an angular acceleration sensor, a gyro sensor, a motion capture, a microphone, a vital sensor, etc. The number of modalities in the behavioral data may be selected appropriately depending on the embodiment.

[0042] The information on terminal operation may be acquired as appropriate from any computer (user terminal). When collecting training data, the model generation device 1 is used as a user terminal and the behavioral data 33 is configured to include information on terminal operation, the model generation device 1 may be used as a computer (sensor SB) for acquiring the information on terminal operation. Similarly, the inference device 2 may be used as a computer (sensor SD) for acquiring the information on terminal operation. Furthermore, the vital sensor may include, for example, a thermometer, a heart rate monitor, a pulse rate monitor, a blood pressure monitor, an electrocardiograph, an electromyograph, a skin electrodermal response monitor, an activity monitor, etc. The vital sensor may be configured as a wearable terminal such as a smart watch.

[0043] The data format of the behavioral data is not particularly limited and may be appropriately selected depending on the type of sensor and the embodiment. Furthermore, at least a portion of the behavioral data may be composed of sensing data obtained by a sensor, or may be composed of analysis data (analysis results) obtained by analyzing the sensing data. As with the eye-related data, the analysis may include, for example, feature extraction, feature amount calculation, etc. Feature extraction may include, for example, identifying operation content, gesture estimation, vital sign estimation, etc.

[0044] The model generation device 1 may acquire data directly from the sensor SB, or may acquire the data indirectly via another computer. When acquiring data directly from the sensor SB, the sensor SB may be connected to the model generation device 1. When acquiring data via another computer, the model generation device 1 may acquire the target data from the other computer via a network, a storage medium, or the like. The model generation device 1 may acquire data measured in the past by the sensor SB, or may acquire data measured in real time by the sensor SB. As described above, the model generation device 1 may also function as at least a part of the sensor SB.

[0045] Similarly, the inference device 2 may acquire data directly from the sensor SD, or may acquire it indirectly via another computer. When acquiring data directly from the sensor SD, the sensor SD may be connected to the inference device 2. When acquiring data via another computer, the inference device 2 may acquire the target data from the other computer via a network, a storage medium, etc. The inference device 2 may acquire data measured in the past by the sensor SD, or may acquire data measured in real time by the sensor SD. The inference device 2 may also function as at least a part of the sensor SD.

[0046] The sensors (sensors SA and SC) that measure eye-related data and the sensors (sensors SB and SD) that measure behavioral data may be different or may at least partially overlap. When the sensors that measure eye-related data and the sensors that measure behavioral data at least partially overlap, the eye-related data may include at least a portion of the behavioral data. For example, when eye activity and human behavior are measured using an imaging device, the image data obtained by the imaging device may at least partially serve as both the eye-related data and the behavioral data.

[0047] Note that the manner in which semantic expressions are inferred is not limited to this example. In another example, data other than behavioral data may be used in addition to or instead of behavioral data to infer semantic expressions. In another example, semantic expressions may be inferred using only eye-related data.

[0048] Inference: In one example, inferring may include identifying and / or regressing. Also, content up to a certain time may have correlation with content from a certain time to a future time. Therefore, inferring may include predicting the future.

[0049] (Information Space / Semantic Representation) In one example, the information space may be configured to represent the content included in the content as a vector. A vector is composed of one or more numerical values. The data format of the vector may be selected arbitrarily. In one example, the vector may simply be numerical data. The content represented in the information space may correspond to the meaning of, for example, a word, a context, or the like.

[0050] A phrase may be, for example, a word, and may correspond to a meaning at a point in time. A phrase may be directly included in the content, or may be indirectly derived from the content. A phrase indirectly derived from the content may include, for example, a phrase associated with the content, a phrase derived by interpreting the content, etc. A phrase may be represented by a vector of a point in information space.

[0051] A context may correspond to a sequence of meanings at two or more points in time, such as a sentence, a video scene, etc. Similar to a phrase, a context may be directly included in the content or indirectly derived from the content. A context may be represented by one or more vectors of points in an information space. In one example, the information space may be divided into clusters. Each cluster may be configured to represent a context. Each point in each cluster may be configured to represent a phrase. This allows semantic representations to be differentiated by context.

[0052] In one example, the semantic representation (semantic representation 39, semantic representation 49) may be a vector in an information space and indicate the content included in the content. The relationship between the semantic representation and the content of the content may be determined appropriately depending on the embodiment. In one example, the semantic representation may directly indicate the content of the content. For example, a numerical representation (such as a binary representation) of a phrase such as a word may be used as the semantic representation. In this case, in machine learning scenarios, the correct answer to the semantic representation 39 may be obtained directly from the content LC. In another example, the semantic representation may indirectly indicate the content of the content. For example, the semantic representation may be obtained by using any projection method that converts the content into a vector representation, such as the method proposed in Non-Patent Document 1 above. In this case, the information space may also be referred to as a latent space. In machine learning scenarios, the correct answer to the semantic representation 39 may be obtained by projecting the content LC using a predetermined projection method.

[0053] The projection method is not particularly limited and may be selected appropriately depending on the embodiment. For example, a trained machine learning model may be used to project the content onto a semantic representation. The machine learning model may have any configuration, such as a Transformer or a Vision Transformer. A known method may be used as the projection method. In one example, the content may be converted into a vector representation by a single projection. In another example, the content may be converted into a vector representation by multiple projections, such as by performing an operation (image2txt) to convert an image into text and then an operation (word2vec) to convert words (text) into vectors in a distributed manner.

[0054] In an inference scenario, the semantic representation 49 corresponds to what the subject TU perceives with respect to the target content TC. As described above, the perceived content may include, for example, content perceived directly from the content, content indirectly derived from the content, etc. The content indirectly derived from the content may include, for example, content associated with the content, or content derived by interpreting the content. The perceived content may also include content predicted to appear at a future time based on the content up to the current time. In other words, perceiving may include predicting.

[0055] (Inference Model) The inference model 5 is configured to infer a semantic expression from a given input (e.g., eye-related data). As long as such inference processing can be performed, the configuration of the inference model 5 is not particularly limited and may be determined appropriately depending on the embodiment. The inference model 5 may be configured using any machine learning model.

[0056] The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference (inference of semantic expression in this embodiment). The machine learning model may be configured, for example, by a neural network, a regression model, a decision tree model, a support vector machine, or other functional formula (calculation model). The machine learning method may be selected appropriately depending on the machine learning model to be adopted.

[0057] In one example, the inference model 5 may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, and the connection relationships between the nodes in each layer. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a long short-term memory (LSTM). The neural network may include any type of model such as a diffusion model, a transformer model, or a generative model. The neural network may include a model capable of in-context learning, such as a large language model (LLM) or a large vision-language model (LVLM). In one example, by including a self-attention mechanism and an autoregressive model, the neural network can acquire the ability to perform in-context learning. The weights of the connections between the nodes included in the neural network and the thresholds of the nodes are examples of calculation parameters. The data format of the input and output of the inference model 5 is not particularly limited and may be selected appropriately depending on the embodiment.

[0058] Machine learning (i.e., training the inference model 5) involves adjusting (optimizing) the values ​​of computational parameters using training samples. Typically, the model generation device 1 may perform supervised learning as a machine learning process using multiple training datasets, each of which is composed of a combination of input samples (training samples) and output samples (teacher signals, labels). The input samples are samples of input data such as the eye-related data 31 and behavioral data 33. The output samples are correct semantic representations 39 corresponding to the input samples. As described above, the correct semantic representations 39 may be obtained directly from the input samples, by transforming the input samples using a predetermined projection method, or by other methods. In supervised learning, the values ​​of computational parameters of the machine learning model may be adjusted so that the output obtained from the machine learning model when an input sample is provided matches the corresponding output sample. However, the method of generating a trained model is not limited to this example and may be changed as appropriate depending on the embodiment. The training dataset is not limited to the above example and may be selected as appropriate depending on the embodiment. For example, when acquiring in-context learning ability, data other than the above may be used as the training dataset. Furthermore, the learning method does not have to be limited to supervised learning, and other methods such as unsupervised learning (including self-supervised learning) and reinforcement learning may also be used.

[0059] Furthermore, the input / output format of the inference model 5 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the input data may be provided to the inference model 5 as is, or may be provided after being preprocessed. In another example, the output of the inference model 5 may be configured to directly or indirectly represent a semantic representation (semantic representation 39, semantic representation 49). When the output of the inference model 5 is configured to indirectly represent a semantic representation, the semantic representation may be obtained by performing any information processing (such as interpretation processing) on ​​the output of the inference model 5.

[0060] Note that the machine learning processing does not necessarily have to be executed within the model generation device 1. The model generation device 1 performing machine learning of the inference model 5 may include executing the machine learning processing within the model generation device 1, and giving instructions to a computer other than the model generation device 1 to cause the other computer to execute the machine learning processing. In the latter case, for example, the model generation device 1 may be appropriately connected to the other computer, and may cause the other computer to execute the machine learning processing while performing data communication.

[0061] Furthermore, experimental examples described below suggest that, although there are individual differences, semantic representations can be inferred even using trained models generated from eye-related data of others. Therefore, the subject LU in the training stage and the subject TU in the inference stage do not necessarily need to be the same person. The subject LU and the subject TU may be the same person, or they may be different people. That is, in the inference process of inferring the semantic representation 49 for the subject TU, the inference device 2 may use a trained inference model 5 generated from data obtained from the subject TU (e.g., eye-related data 31), or may use a trained inference model 5 generated from data obtained from someone other than the subject TU. Furthermore, the trained inference model 5 may be generated from data obtained from multiple subjects LU (e.g., eye-related data 31), or may be generated from data obtained from a single subject LU. In one example, the subjects LU and the subjects TU may be classified into groups, and the trained inference model 5 may be shared within the same group.

[0062] (System Configuration) In one example, as shown in Figure 1, the model generation device 1 and the inference device 2 may be connected to each other via a network. The type of network may be selected as appropriate from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between the model generation device 1 and the inference device 2 is not limited to this example and may be selected as appropriate depending on the embodiment. In another example, data may be exchanged using a storage medium.

[0063] 1, the model generation device 1 and the inference device 2 are separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. In another example, the model generation device 1 and the inference device 2 may be configured as a single computer. In yet another example, at least one of the model generation device 1 and the inference device 2 may be configured as multiple computers.

[0064] §2 Configuration Example [Hardware Configuration] (Model Generation Device) Fig. 2 shows a schematic diagram of an example of the hardware configuration of the model generation device 1 according to this embodiment. In this example of the present embodiment, the model generation device 1 is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0065] The control unit 11 includes a hardware processor such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0066] The storage unit 12 may be configured with, for example, a hard disk drive, a solid state drive, a semiconductor memory, etc. The storage unit 12 (and RAM, ROM) is an example of a memory resource. In this embodiment, the storage unit 12 stores various information such as a model generation program 81, eye-related data 31, behavioral data 33, and learning result data 50.

[0067] The model generation program 81 is a program for causing the model generation device 1 to execute information processing (see FIG. 6 , described below) related to machine learning of the inference model 5. The model generation program 81 includes a series of instructions for the information processing. The learning result data 50 is configured to indicate information related to the generated trained inference model 5. In this embodiment, the learning result data 50 may be generated as a result of executing the model generation program 81. Note that the configuration of the learning result data 50 is not particularly limited as long as it can hold information for executing the calculation processing of the trained inference model 5, and may be determined appropriately depending on the embodiment. In one example, the learning result data 50 may be configured to include information indicating the values ​​of calculation parameters adjusted by machine learning. In some cases, the learning result data 50 may be configured to include information indicating the configuration of the inference model 5 (e.g., the structure of a neural network, etc.).

[0068] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, or the like. The type and number of external interfaces 13 may be determined appropriately depending on the embodiment. If the external interface 13 includes a communication port, the model generation device 1 may perform data communication with another computer (e.g., the inference device 2, etc.) via a network. The communication standard of the communication port may be selected arbitrarily. Furthermore, in this embodiment, the model generation device 1 may be connected to at least one of the sensor SA and the sensor SB via the external interface 13.

[0069] The input device 14 is a device for inputting, for example, a mouse, a keyboard, an operator, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. A user can operate the model generation device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrated into one device, for example, a touch panel display, etc.

[0070] The drive 16 is a device for reading various information such as programs stored in a storage medium 91. At least one of the model generation program 81, the eyeball-related data 31, the behavioral data 33, and the learning result data 50 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (such as stored programs) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The model generation device 1 may acquire at least one of the model generation program 81, the eyeball-related data 31, the behavioral data 33, and the learning result data 50 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface 13. Furthermore, the storage medium 91 may include a memory resource provided in another computer, such as a network attached storage (NAS). When the storage medium 91 is a memory resource of another computer, the drive 16 may be omitted.

[0071] Note that, with regard to the specific hardware configuration of the model generation device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. If the behavioral data 33 is not used, the behavioral data 33 may be omitted from the storage unit 12. The model generation device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be identical. Furthermore, the model generation device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a terminal device, or the like.

[0072] 3 is a schematic diagram showing an example of the hardware configuration of the inference device 2 according to this embodiment. In this example of this embodiment, the inference device 2 is a computer to which a control unit 21, a storage unit 22, an external interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0073] The control unit 21 to the drive 26 and the storage medium 92 of the inference device 2 may be configured similarly to the control unit 11 to the drive 16 and the storage medium 91 of the model generation device 1. The control unit 21 (CPU) is an example of a processor resource of the inference device 2, and the storage unit 22 (and RAM, ROM) is an example of a memory resource of the inference device 2. In this embodiment, the storage unit 22 stores various information such as an inference program 82 and learning result data 50.

[0074] The inference program 82 is a program for causing the inference device 2 to execute information processing (see FIG. 7 described below) related to the inference of the content perceived from the content. The inference program 82 includes a series of instructions for the information processing. The learning result data 50 may be managed separately from the inference program 82, or may be incorporated into the inference program 82. At least one of the inference program 82 and the learning result data 50 may be stored in a storage medium 92 instead of or together with the storage unit 22. The inference device 2 may acquire at least one of the inference program 82 and the learning result data 50 from the storage medium 92.

[0075] If the external interface 23 includes a communication port, the inference device 2 may perform data communication with another computer (e.g., the model generation device 1, etc.) via a network. The inference device 2 may also be connected to at least one of the sensor SC and the sensor SD via the external interface 23. An operator can operate the inference device 2 by using the input device 24 and the output device 25.

[0076] Note that, with regard to the specific hardware configuration of the inference device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the external interface 23, input device 24, output device 25, and drive 26 may be omitted. The inference device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The inference device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0077] 4 schematically shows an example of the software configuration of the model generation device 1 according to this embodiment. The control unit 11 of the model generation device 1 loads a model generation program 81 stored in the storage unit 12 into RAM, and executes instructions included in the model generation program 81 using the CPU. As a result, the model generation device 1 operates as a computer including an acquisition unit 111, a training unit 112, and an output processing unit 113 as software modules.

[0078] The acquisition unit 111 is configured to acquire eye-related data 31 measured from a subject LU who is viewing content LC. The training unit 112 is configured to perform machine learning of the inference model 5 using the acquired eye-related data 31. The machine learning includes training the inference model 5 to acquire the ability to infer, from the eye-related data 31, a semantic representation 39 in an information space corresponding to the content included in the content LC. The output processing unit 113 is configured to output the results of the machine learning.

[0079] In one example, the acquisition unit 111 may be configured to further acquire behavioral data 33 indicating the behavior of the subject LU while viewing the content LC. In response to this, the training unit 112 may be configured to perform machine learning of the inference model 5 using the acquired eye-related data 31 and behavioral data 33. The machine learning may include training the inference model 5 to acquire the ability to infer a semantic representation 39 in an information space corresponding to content included in the content LC from the eye-related data 31 and the behavioral data 33. In one example, the acquisition unit 111 may be configured to acquire eye-related data 31 composed of gaze data, blink data, pupil diameter data, or a combination thereof.

[0080] 5 shows a schematic diagram of an example of the software configuration of the inference device 2 according to this embodiment. The control unit 21 of the inference device 2 loads an inference program 82 stored in the storage unit 22 into RAM, and executes instructions included in the inference program 82 using the CPU. As a result, the inference device 2 operates as a computer that includes an acquisition unit 211, an inference unit 212, and an output processing unit 213 as software modules.

[0081] The acquisition unit 211 is configured to acquire eyeball-related data 41 measured from a subject TU viewing target content TC. The inference unit 212 has a trained inference model 5 by holding learning result data 50. The inference unit 212 is configured to infer a semantic representation 49 in an information space corresponding to the content included in the target content TC from the acquired eyeball-related data 41, using the trained inference model 5. The output processing unit 213 is configured to output a result of inferring the semantic representation 49.

[0082] In one example, the acquisition unit 211 may be configured to further acquire behavioral data 43 indicating the behavior of the subject TU while viewing the target content TC. In response, the inference unit 212 may be configured to infer the semantic representation 49 from the acquired eye-related data 41 and behavioral data 43 using a trained inference model 5. In one example, the acquisition unit 211 may be configured to acquire eye-related data 41 composed of gaze data, blink data, pupil diameter data, or a combination thereof.

[0083] (Other) In this embodiment, an example is described in which each software module of the model generating device 1 and the inference device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the model generating device 1 and the inference device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0084] §3 Operational Example [Model Generation Device] Figure 6 is a flowchart showing an example of the processing procedure of the model generation device 1 according to this embodiment. The following processing procedure is an example of a model generation method (information processing method) executed by a computer. However, the processing procedure of the model generation device 1 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0085] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires the training eye-related data 31 measured from the subject LU who is viewing the content LC.

[0086] In one example, the control unit 11 may further acquire behavioral data 33 indicating the behavior of the subject LU while viewing the content LC. The sensor SA may be used to acquire the eyeball-related data 31, and the sensor SB may be used to acquire the behavioral data 33. At least a portion of the eyeball-related data 31 and the behavioral data 33 may be generated by the model generation device 1 or by another computer. The control unit 11 may acquire at least a portion of the eyeball-related data 31 and the behavioral data 33 via a network, an external storage device, a storage medium 91, or the like. The order in which the eyeball-related data 31 and the behavioral data 33 are acquired is not particularly limited and may be appropriately selected depending on the embodiment. The eyeball-related data 31 and the behavioral data 33 may be acquired at least partially in parallel. In another example, the acquired eyeball-related data 31 may be composed of gaze data, blink data, pupil diameter data, or a combination thereof. After acquiring the eyeball-related data 31, the control unit 11 proceeds to the next step S102.

[0087] (Step S102) In step S102, the control unit 11 operates as the training unit 112 and performs machine learning of the inference model 5 using the acquired eyeball-related data 31. In the machine learning, the control unit 11 trains the inference model 5 so that it acquires the ability to infer, from the eyeball-related data 31, a semantic representation 39 in the information space corresponding to the content included in the content LC. This makes it possible to generate a trained inference model 5 that has acquired the ability to infer a semantic representation from the eyeball-related data within the category of the training sample.

[0088] In one example, when behavioral data 33 has been acquired, the control unit 11 may further use the acquired behavioral data 33 in machine learning of the inference model 5. That is, in the machine learning process, the control unit 11 may train the inference model 5 so that when the eye-related data 31 and behavioral data 33 are given, the inference model 5 outputs an inference result that matches the correct answer of the corresponding semantic representation 39. This makes it possible to generate a trained inference model 5 that has acquired the ability to infer semantic representations from the eye-related data and behavioral data.

[0089] In one example, training the inference model 5 may be optimizing the values ​​of the calculation parameters of the inference model 5 according to the training samples. The machine learning method may be determined appropriately depending on the embodiment, such as the type and structure of the machine learning model used in the inference model 5. Any method, such as backpropagation or solving an optimization problem, may be adopted as a method for adjusting the calculation parameters. In another example, the control unit 11 may execute the machine learning calculation process within the model generation device 1, or may instruct another computer to execute the machine learning calculation process. When the machine learning (training the inference model 5) is completed, the control unit 11 proceeds to the next step S103.

[0090] (Step S103) In step S103, the control unit 11 operates as the output processing unit 113 and outputs the results of the machine learning.

[0091] The output destination and the content of the output information may be selected appropriately depending on the embodiment. In one example, the control unit 11 may generate learning result data 50 indicating the results of machine learning as an output process and store the generated learning result data 50 in a predetermined storage area. The predetermined storage area may be, for example, RAM within the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, or the like. The external storage device may be, for example, a data server such as a NAS. The external storage device may be, for example, an external storage device. When machine learning of the inference model 5 is executed on another computer, the learning result data 50 may be generated on the other computer. In another example, the control unit 11 may output information (e.g., loss, etc.) obtained during the machine learning calculation process as an output process. The output destination may be, for example, RAM within the control unit 11, the storage unit 12, the output device 15, an external computer, an external storage device, a storage medium, or a combination thereof.

[0092] When the output of the machine learning results is completed, the control unit 11 ends the processing procedure of the model generation device 1 according to this operation example.

[0093] The generated learning result data 50 may be provided to the inference device 2 at any timing and by any method. For example, the control unit 11 may transmit the learning result data 50 to the inference device 2 as part of the processing of step S103 or separately from the processing of step S103. The inference device 2 may acquire the learning result data 50 (trained inference model 5) by receiving it. For example, the inference device 2 may acquire the learning result data 50 by accessing the model generation device 1 or a data server via a network. For example, the inference device 2 may acquire the learning result data 50 via a storage medium 92. For example, the learning result data 50 may be pre-installed in the inference device 2.

[0094] Furthermore, the control unit 11 may periodically or irregularly repeatedly execute the series of processes from step S101 to step S103 to update or generate new learning result data 50. During this repetition, at least a portion of the training samples used for machine learning may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 11 may update the learning result data 50 held by the inference device 2 by providing the updated or newly generated learning result data 50 to the inference device 2 in any manner.

[0095] [Inference Device] Figure 7 is a flowchart showing an example of the processing procedure of the inference device 2 according to this embodiment. The following processing procedure is an example of an inference method (information processing method) executed by a computer. However, the processing procedure of the inference device 2 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0096] (Step S201) In step S201, the control unit 21 operates as the acquisition unit 211 and acquires the eyeball-related data 41 measured from the subject TU who is viewing the target content TC.

[0097] In one example, the control unit 21 may further acquire behavioral data 43 indicating the behavior of the subject TU while viewing the target content TC. The sensor SC may be used to acquire the eyeball-related data 41, and the sensor SD may be used to acquire the behavioral data 43. At least a portion of the eyeball-related data 41 and the behavioral data 43 may be generated by the inference device 2 or by another computer. The control unit 21 may acquire at least a portion of the eyeball-related data 41 and the behavioral data 43 via a network, an external storage device, a storage medium 92, or the like. The order in which the eyeball-related data 41 and the behavioral data 43 are acquired is not particularly limited and may be selected appropriately depending on the embodiment. The eyeball-related data 41 and the behavioral data 43 may be acquired at least partially in parallel. In another example, the acquired eyeball-related data 41 may be composed of gaze data, blink data, pupil diameter data, or a combination thereof. After acquiring the eyeball-related data 41, the control unit 21 proceeds to the next step S202.

[0098] (Step S202) In step S202, the control unit 21 operates as an inference unit 212 and uses a trained inference model 5 to infer a semantic representation 49 in the information space corresponding to the content contained in the target content TC from the acquired eyeball-related data 41.

[0099] In one example, when behavioral data 43 is acquired, the control unit 21 may use a trained inference model 5 to infer a semantic expression 49 from the acquired eye-related data 41 and behavioral data 43.

[0100] The computational processing of the trained inference model 5 may be performed as appropriate depending on the embodiment, such as the type and configuration of the inference model 5. For example, if the inference model 5 is configured as a neural network, the control unit 21 may input the eyeball-related data 41 (and the behavioral data 43) to the trained inference model 5 and perform forward computational processing of the trained inference model 5. This allows the control unit 21 to obtain an output corresponding to the result of inferring the semantic representation 49 from the trained inference model 5. When the inference of the semantic representation 49 is completed, the control unit 21 proceeds to the next step S203.

[0101] (Step S203) In step S203, the control unit 21 operates as the output processing unit 213 and outputs the result of inferring the semantic expression 49.

[0102] The content of the information to be output as the inference result is not particularly limited as long as it is related to the inference result of the semantic representation 49, and may be determined appropriately depending on the embodiment. In one example, the control unit 21 may output the inference result of the semantic representation 49 as is. Outputting the inference result of the semantic representation 49 as is may include outputting a vector obtained as the semantic representation 49 by inference, and outputting content (phrases, context, etc.) corresponding to the obtained vector. If the semantic representation is obtained by transforming the content using a predetermined projection method, the control unit 21 can obtain the content corresponding to the vector by performing an inverse transformation of the predetermined projection method on the inference result of the semantic representation 49.

[0103] In another example, the control unit 21 may perform any information processing (e.g., a determination process, an analysis process, etc.) on the obtained inference result, and then the control unit 21 may output the result of the information processing as information related to the inference result.

[0104] For example, the inferred semantic representation 49 corresponds to the content perceived by the subject TU. Therefore, the control unit 21 may determine the degree of match between the inferred semantic representation 49 (the inference result of the perceived content) and the content actually included in the target content TC. The control unit 21 may evaluate whether the subject TU is properly perceiving the target content TC based on the determined degree of match. Proper perception may include, for example, concentrating on watching the content or accurately understanding the content of the content. If the determined degree of match is low, the control unit 21 may evaluate that the subject TU is not properly perceiving the target content TC (e.g., not concentrating on watching the target content TC or not understanding the content of the target content TC). On the other hand, if the determined degree of match is high, the control unit 21 may evaluate that the subject TU is properly perceiving the target content TC (e.g., concentrating on watching the target content TC or understanding the content of the target content TC). Whether the degree of match is high or low may be determined, for example, by comparing it with a threshold value. The control unit 21 may output the result of this evaluation as an output process of the inference result. As an example of an application scenario, this output format may be adopted in an educational setting to evaluate whether a student is concentrating on watching educational content, whether the student has understood the educational content, etc.

[0105] Furthermore, for example, the control unit 21 may categorize the subject TU according to the inference result of the obtained semantic representation 49. Categorization involves determining the group to which the subject TU belongs. The control unit 21 may output the categorization result (i.e., the result of determining the group to which the subject TU belongs) as an inference result output process. Groups are presumed to correspond to perceptual tendencies. In other words, belonging to the same group is presumed to indicate similar perceptual tendencies. Therefore, as an example of an application scenario, this output format may be adopted in medical settings to infer the severity of a neurological disorder that affects content perception. The categorization result may correspond to the severity of the neurological disorder. This enables medical assistance (support). As another example of an application scenario, it is possible to estimate content that is likely to be noticed in the target content TC from the semantic representation 49 perceived by each group. Therefore, the control unit 21 may output the inferred semantic representation 49 for each group. The output information on the inference results may be used as content analysis material in production situations such as optimizing the target content TC or creating new content. Note that the output format of this analysis material may be adopted regardless of categorization. That is, the control unit 21 may output the inference results of the semantic representation 49 without performing categorization. The output inference results may be used as the above-mentioned analysis material.

[0106] Furthermore, for example, a computational model that projects from the information space of the semantic representation to another space (such as another latent space) may be further prepared. The computational model may be, for example, appropriately configured using a trained machine learning model or the like. The control unit 21 may convert the inference result of the semantic representation 49 into another representation by projecting the inference result of the semantic representation 49 into another space using the computational model. The control unit 21 may output the obtained another representation as an output process of the inference result. The another representation may be selected arbitrarily. The another representation may be, for example, an emotional representation, a future semantic representation (future prediction), etc. When an emotional representation is used as the another representation, the control unit 21 can obtain an inference result of the emotions of the subject TU when viewing the target content TC via the inference result of the semantic representation 49. When a future prediction is used as the another representation, the control unit 21 can obtain a future prediction result of the content perceived by the subject TU regarding the target content TC via the inference result of the semantic representation 49. When inferring the semantic representation 49 includes future prediction, the another representation may be a semantic representation further in the future than the semantic representation 49. In one example, a plurality of computational models may be prepared, each of which projects onto a plurality of different spaces, by machine learning the projection relationships between the information space of the semantic representation and each of the plurality of different spaces. By selectively using the plurality of computational models, the control unit 21 can obtain an expression in any other space starting from the information space in which direct projection from the eyeball-related data has been learned.

[0107] The output destination is not particularly limited and may be selected appropriately depending on the embodiment. The output destination may be, for example, the RAM in the control unit 21, the storage unit 22, the output device 25, an external computer, an external storage device, a storage medium, or a combination thereof.

[0108] When the output of the inference result is completed, the control unit 21 ends the processing procedure of the inference device 2 according to this operation example. In one example, the control unit 21 may execute a series of processes from step S201 to step S203 in real time. That is, the control unit 21 may execute a process of inferring a semantic expression 49 from the eyeball-related data 41 (and behavioral data 43) obtained in real time in step S202 and a process of outputting the inference result in step S203. In another example, the control unit 21 may infer a semantic expression 49 retroactively from the eyeball-related data 41 (and behavioral data 43) measured in the past. That is, the control unit 21 may acquire the eyeball-related data 41 (and behavioral data 43) measured in the past in step S201, and execute the processes of step S202 and step S203 on the acquired eyeball-related data 41 (and behavioral data 43).

[0109] [Features] In this embodiment, the model generation device 1 generates a trained inference model 5 that has acquired the ability to infer the content perceived from eyeball-related data through the processing of step S102. By using this trained inference model 5, it is possible to infer the content perceived by an individual while viewing content in a simple and low-cost manner. The inference device 2 can infer the content perceived by the subject TU while viewing the target content TC in a simple and low-cost manner through the processing of steps S201 and S202 described above.

[0110] §4 Modifications Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in every respect. It goes without saying that various improvements or modifications can be made without departing from the scope of the present disclosure. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0111] For example, in the above embodiment, the inference device 2 holds the trained inference model 5 and executes the process of inferring the semantic representation 49. However, the entity executing the inference process does not have to be limited to the inference device 2. The trained inference model 5 (learning result data 50) may be held in another computer, and the inference device 2 may request the other computer to execute the inference process. In this case, the inference device 2 does not need to hold the trained inference model 5, and may obtain the results of inferring the semantic representation 49 from the other computer.

[0112] §5 Experimental Example The following experiment was conducted to verify whether it is possible to infer the semantic expression of the content contained in the content from the eye activity while viewing the content. However, the present invention is not limited to the following experimental example.

[0113] (Preparation) First, 10-minute videos (7 videos) were prepared. General video works were used for the videos. The prepared videos were viewed by subjects, and pupil diameter (Y) was measured while the subjects were watching the videos to obtain pupil diameter data. The pupil diameter data consisted of one-dimensional numerical data for each unit of time. In addition, a person other than the subjects (annotator) was asked to annotate each scene in the prepared videos. Using a computational model (word2vec), the annotations were converted into a 1000-dimensional vector (X ｗ２ｖ ) was converted into a vector of the pupil diameter (Y) and the semantic expression (X). A known model was used as the calculation model. This resulted in the correct value of the semantic expression in the information space (1000-dimensional numerical data for each unit time). ｗ２ｖ ) Each of the seven videos was divided into training samples and evaluation samples. Specifically, the evaluation experiment was repeated seven times, using samples obtained from six of the seven videos as training samples and a sample obtained from one video as an evaluation sample, and the evaluation results were obtained by averaging the seven evaluations. 58 subjects participated in this experiment.

[0114] Next, in order to take into account the delay in pupil response, four time points were set to be used to infer the pupil diameter at the target time. The target time was set to 0 seconds, and the four time points were -2 seconds, +2 seconds, +6 seconds, and +10 seconds. Then, by performing regression analysis using the training samples, each parameter (W ｗ２ｖ The regression model used was the following equation 1:

[0115] a is a constant (noise term). The values ​​of each parameter of 4000 dimensions obtained by regression analysis are averaged over time, and each parameter of 1000 dimensions (W ｗ２ｖ＿ａｖｒｇ ) values ​​were obtained.

[0116] Next, the obtained 1000-dimensional parameters (W ｗ２ｖ＿ａｖｒｇ ) values ​​and evaluated the contribution of each parameter to the inference of pupil diameter, generating a principal component vector (W). The dimensions of the principal component vector (W) were set to 2, 10, 12, 14, 16, 18, and 20. The principal component vector (W) for each dimension was used to generate a semantic representation vector (X ｗ２ｖ ) into the principal component space, a compressed vector (Xe ｗ２ｖ ) training samples were obtained.

[0117] Next, to take time delay into account, 21 time points were set to be used to infer the vector of the semantic expression of the target time. The target time was set to 0 seconds, and the 21 time points were taken in 2-second increments from 20 seconds before to 20 seconds after. Then, by performing regression analysis using the training samples, a vector (Xe ｗ２ｖ The values ​​of each parameter (D) of the regression model (decoding model) that infers the above-mentioned hypothesis were specified for each dimension of the principal component vector.

[0118] FIG. 8 shows the relationship between the principal component vectors and the decoding model obtained in the experiment. As shown in FIG. 8, the transposed matrix (W ｔ ) to obtain a compressed vector (Xeｗ２ｖ ) is projected (inversely transformed) to the original 1000-dimensional semantic representation vector (Xd ｗ２ｖ ) can be obtained. Therefore, the decryption model and the transpose matrix of the principal component vectors (W ｔ ) to convert the pupil diameter (Y) into a semantic representation vector (X ｗ２ｖ An inference model for inferring the above was constructed for each dimension of the principal component vector.

[0119] (Verification) Next, using the evaluation sample, the accuracy of the obtained inference model was evaluated for each dimension of the principal component vector and for each subject. Specifically, using the above-mentioned decoding model among the obtained inference models, the compressed semantic representation vector (Xe ｗ２ｖ ) is calculated from the pupil diameter (Y), and the vector (Xe ｗ２ｖ ) is the original vector (X ｗ２ｖ ) is a compressed semantic representation vector (Xe ｗ２ｖ ) and calculated the inference accuracy (correlation coefficient) of the inference model according to whether or not the principal component vectors matched the corresponding values. In this accuracy evaluation, each of the 58 subjects was designated as a first subject, and the remaining 57 subjects were designated as second subjects. The principal component vectors obtained from the 57 second subjects were averaged to obtain the principal component vectors to be applied to the first subjects. A decoding model was generated from the training samples obtained from the first subjects, and the generated decoding model was applied to the evaluation samples. The above evaluation was then repeated for each dimension of the principal component vector for the 58 subjects, and the obtained accuracies were averaged to calculate the average inference accuracy for each dimension of the principal component vector (Figure 9). In addition, using the obtained inference model, a semantic expression vector (Xd ｗ２ｖ ) was calculated. Then, using t-SNE (t-distributed Stochastic Neighbor Embedding), the calculated vector (Xd ｗ２ｖ ) in space, the vector (Xd ｗ２ｖ ) is expressed in space (Figure 10).

[0120] Furthermore, to verify individual differences, the coding model (parameters (W ｗ２ｖ ) was used on a second subject, separate from the first subject, to evaluate whether the second subject's pupil diameter at a target time could be accurately inferred from the semantic representation vector. Each of the 30 subjects was designated as a first subject, and the remaining 29 subjects were designated as second subjects. The second subject's pupil diameter was then inferred from the encoding model of the first subject, and accuracy was evaluated based on the inference results (29 times). For each of the 30 subjects, the difference between the accuracy of the first subject's own pupil diameter inferred using the encoding model of the first subject and the average accuracy of the 29 times was calculated (30 times) to calculate the variation in inference accuracy between individuals (Figure 11). Note that a commercially available PC was used for all of the above calculations.

[0121] (Results) Figure 9 shows the results of calculating the average inference accuracy of the inference model for each dimension of the principal component vector in the experimental example. Figure 10 shows the results of expressing the perceptual content inferred from pupil diameter using t-SNE. Figure 11 shows the results of calculating the change in inference accuracy between individuals for the coding model in the experimental example.

[0122] As shown in Figure 9, for example, when the number of dimensions of the principal component vector was three, the average accuracy (correlation coefficient) of inferring a 1,000-dimensional semantic expression from one-dimensional pupil diameter was approximately 0.18. These results indicate that the resulting inference model can achieve statistically significantly higher inference power (correlation coefficient of approximately 0.18) than a model without inference power. Furthermore, as shown in Figure 10, the content content could be restored from one-dimensional pupil diameter. Pupil diameter is an example of eye activity, and other eye activities such as gaze and blinking are also related to pupil diameter. Therefore, these results suggest that the content perceived by an individual while viewing content can be inferred from eye activity.

[0123] In addition, as shown in Figure 11, the inference accuracy of the coding model deteriorated for some subjects but not for the remaining subjects. These results suggest that, although there are individual differences, eye activity is similar at least to some extent, and that a trained inference model generated from other people's eye-related data can infer semantic representations. Furthermore, it is possible to categorize the inference results of semantic representations and analyze the tendencies of subjects by group. In other words, it is possible to group the inference results of semantic representations and apply them to the above-mentioned diagnosis, content optimization, and other applications.

[0124] 1...model generation device, 11...control unit, 12...storage unit, 13...external interface, 14...input device, 15...output device, 16...drive, 81...model generation program, 91...storage medium, 111...acquisition unit, 112...training unit, 113...output processing unit, LU...subject, LC...content, SA / SB...sensor, 31...eye-related data, 33...behavioral data, 39...semantic representation, 2...inference device, 21...control unit, 22...storage unit, 23...external interface, 24...input device, 25...output device, 26...drive, 82...inference program, 92...storage medium, 211...acquisition unit, 212...inference unit, 213...output processing unit, TU...subject, TC...target content, SC / SD...sensor, 41...eye-related data, 43...behavioral data, 49...semantic representation, 5...inference model

Claims

1. A model generation method, in which a computer executes the steps of: acquiring eye-related data measured from a subject viewing content; performing machine learning on an inference model using the acquired eye-related data; and outputting the results of the machine learning, wherein the machine learning includes training the inference model to acquire the ability to infer, from the eye-related data, semantic representations in an information space corresponding to content included in the content.

2. The model generation method of claim 1, wherein the computer further performs a step of acquiring behavioral data indicating the behavior of the subject while viewing the content, and the acquired behavioral data is further used in machine learning of the inference model, and in the machine learning, the inference model is trained to acquire the ability to infer the semantic representation from the eye-related data and the behavioral data.

3. The model generation method according to claim 1, wherein the acquired eye-related data is composed of gaze data, blink data, pupil diameter data, or a combination thereof.

4. An inference program for causing a computer to execute the steps of: acquiring eye-related data measured from a subject viewing target content; inferring a semantic expression in information space corresponding to the content contained in the target content from the acquired eye-related data using a trained inference model; and outputting the result of inferring the semantic expression.

5. The inference program of claim 4, further comprising the step of causing the computer to acquire behavioral data indicating the behavior of the subject while viewing the target content, and in the inference step, causing the computer to infer the semantic expression from the acquired eye-related data and behavioral data.

Images