Learning device, estimation device, learning method, estimation method, and program
The learning device predicts gaze areas outside visible image frames using trained models, addressing the limitation of existing techniques by accurately anticipating human attention in real-world environments.
Patent Information
- Application Number
- JP2021083943
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-05-18
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2041-05-18
AI Technical Summary
Existing image and video saliency estimation techniques are limited to predicting gaze areas that are visible in the image, failing to account for attention to invisible regions such as areas behind obstacles, which is crucial for AI systems operating in real-world environments.
A learning device and method that utilizes training data pairs of image data with correct gaze area data for invisible regions, updating a mathematical model to predict gaze areas outside the visible image frame using convolutional neural networks, long short-term memory networks, and reinforcement learning to enhance prediction accuracy.
Enables the prediction of gaze regions in spaces not visible in images or videos, enhancing the capability of AI systems to anticipate human attention in real-world scenarios, improving safety and functionality in applications like autonomous vehicles and robotics.
Smart Images

Figure 0007578955000007 
Figure 0007578955000008 
Figure 0007578955000009
Abstract
Description
[Technical field]
[0001] The present invention relates to a learning device, an inference device, a learning method, an inference method, and a program. [Background technology]
[0002] Technologies have been developed to predict the Focus of Attention (FoA), which is the area within a space such as a scene that humans are likely to focus their attention on. A representative example is a technology called saliency estimation, which detects the area of attention within an image or video by analyzing the image or video. This technology has been researched for many years in the technical fields of robot vision and computer vision, and has been used in a wide range of fields such as object recognition, image or video editing, image coding, image quality evaluation, autonomous driving, and online marketing.
[0003] AI (artificial intelligence) technology is being implemented in society, and one of the basic requirements for AI technology is the ability to perceive or recognize the real world in the same way as humans, and technology to predict gaze areas is one of the core technologies. In particular, for AI robots and AI agents that operate in the real world in the same way as humans, it is expected that they will perceive the outside world using images or videos from a first-person perspective taken by an attached camera as input. Therefore, there are high expectations for technology to predict gaze areas that target images or videos from a first-person perspective. In the following, for simplicity of explanation, the technology to predict gaze areas will be explained using an image as an example, but since a video is a collection of multiple images, the following explanation also applies to video instead of an image.
[0004] With the recent development of deep learning, great progress has been made in saliency estimation based on first-person perspective images, and excellent prediction performance has been achieved. For example, [1] proposed a model based on 3D convolutional networks that can estimate the gaze area in first-person perspective images with high accuracy in autonomous driving scenarios.
[0005] In addition, Non-Patent Document 2 focuses on the difference in gaze areas depending on age and proposes a technology that uses image conversion technology to convert the results of estimation of the gaze area by adults into the results of estimation of the gaze area by elderly people. This technology also targets first-person perspective images from autonomous driving scenarios and pedestrian viewpoints, enabling highly accurate saliency prediction. [Prior art documents] [Non-patent literature]
[0006] [Non-Patent Document 1] Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara, “Predicting the drivers focus of attention: the dr(eye)ve project”, IEEE transactions on pattern analysis and machine intelligence, 2018. [Non-Patent Document 2] Onkar Krishna, Go Irie, Takahito Kawanishi, Kunio Kashino, and Kiyoharu Aizawa, “Translating Adult's Focus of Attention to Elderly's”, In Proceedings of International Conference on Pattern Recognition, 2020. Summary of the Invention [Problem to be solved by the invention]
[0007] However, these technologies are only designed to estimate the gaze area in an image, and are only technologies to estimate the gaze area in an image. Humans who are active in the real world do not necessarily pay attention to observable areas in the scene. Humans sometimes predict or judge areas that should be paid attention to, even if they are invisible, such as behind them or behind a wall, and take preparatory actions or take actions to avoid danger.
[0008] Therefore, AI systems that work alongside humans in the real world are expected to have similar functions to humans, specifically, the ability to predict areas of gaze in areas not captured in images.
[0009] However, existing image saliency estimation techniques are all realized as techniques for estimating salient areas only in areas that appear in an image, and are not techniques for predicting gaze areas in areas that do not appear in an image. As mentioned above, the same is true for video.
[0010] In view of the above circumstances, an object of the present invention is to provide a technique for predicting a gaze area in an area in space that is not captured in an image or video. [Means for solving the problem]
[0011] One aspect of the present invention is a learning device that includes a control unit that updates a mathematical model indicating the relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data that is a pair of data consisting of image data and correct gaze area data that is data indicating a gaze area in a space not shown in the image represented by the image data, and the control unit uses the mathematical model to estimate a gaze area based on image data included in the training data, and updates the mathematical model so that the estimated result approaches the gaze area indicated by the correct gaze area data included in the training data.
[0012] One aspect of the present invention is an estimation device that includes a control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space that is not shown in an image represented by the image data, based on training data that is a pair of data consisting of image data and correct gaze area data that is data indicating a gaze area in a space that is not shown in the image represented by the image data, and the control unit uses the mathematical model to estimate a gaze area based on image data included in the training data, and updates the mathematical model so that the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data, and a control unit that estimates a gaze area in a space that is not shown in an image represented by the image data based on input image data using an updated mathematical model, which is the mathematical model updated until a predetermined termination condition is satisfied, by a learning device that updates the mathematical model so that the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data.
[0013] One aspect of the present invention is a learning method comprising a control step of updating a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data which is a pair of data consisting of image data and correct gaze area data which is data indicating a gaze area in a space not shown in the image represented by the image data, in which in the control step, a gaze area is estimated based on image data contained in the training data using the mathematical model, and the mathematical model is updated so that the estimated result approaches the gaze area indicated by the correct gaze area data included in the training data.
[0014] One aspect of the present invention is an estimation method comprising: a control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data which is a pair of data consisting of image data and correct gaze area data which is data indicating a gaze area in a space not shown in the image represented by the image data; the control unit uses the mathematical model to estimate a gaze area based on image data included in the training data, and updates the mathematical model so that the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data; and a control step of estimating a gaze area in a space not shown in an image represented by the image data, based on input image data, using the updated mathematical model, which is the mathematical model updated until a predetermined termination condition is satisfied, by a learning device that updates the mathematical model so that the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data.
[0015] One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device. Effect of the Invention
[0016] The present invention makes it possible to predict a gaze area in a region in space that is not captured in an image or video. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a prediction system 100 according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram for explaining a first example of a prediction result expression format in the embodiment. [Diagram 3] FIG. 11 is an explanatory diagram for explaining a second example of a prediction result expression format in the embodiment. [Figure 4] FIG. 11 is an explanatory diagram for explaining a third example of a prediction result expression format in the embodiment. [Diagram 5] FIG. 11 is an explanatory diagram for explaining a fourth example of a prediction result expression format in the embodiment. [Figure 6] FIG. 2 is a diagram showing an example of the configuration of a predictor 10 according to the embodiment. [Figure 7]4 is a flowchart showing an example of the flow of processing executed by a predictor 10 in the embodiment. [Figure 8] 11 is a flowchart showing an example of the flow of a reinforcement prediction process in the embodiment. [Figure 9] 11 is a flowchart showing an example of the flow of a reward acquisition process in an embodiment. [Figure 10] FIG. 2 is a diagram showing an example of the hardware configuration of a learning device 1 in an embodiment. [Figure 11] FIG. 2 is a diagram showing an example of the functional configuration of a control unit 11 in the embodiment. [Figure 12] FIG. 2 is a diagram showing an example of a hardware configuration of a prediction device 2 in the embodiment. [Figure 13] FIG. 2 is a diagram showing an example of the functional configuration of a control unit 21 in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] (Embodiment) FIG. 1 is a diagram showing an example of the configuration of a prediction system 100 according to an embodiment. The prediction system 100 includes a learning device 1 and a prediction device 2. Hereinafter, for ease of explanation, the prediction system 100 will be described taking as an example a case where the data input to the learning device 1 and the prediction device 2 is image data. However, since a video is a time series of images, the data input to the learning device 1 and the prediction device 2 may be video data, which is video data, instead of image data. Hereinafter, for ease of explanation, the prediction system 100 will be described taking as an example a case where image data is input to the learning device 1 and the prediction device 2, but video data may be input to the learning device 1 and the prediction device 2 instead of image data.
[0019] The learning device 1 accepts input of image data. Based on the input image data, the learning device 1 updates a prediction model by a machine learning method. The prediction model is a mathematical model that indicates the relationship between the input image data and a focus of attention (FoA) in a space that is not shown in an image represented by the input image data. In other words, the focus of attention in a space outside the space represented by the image of the image data. The focus of attention is an area within a space (hereinafter referred to as an "intra-space area") that humans are likely to direct their eyes to.
[0020] The prediction device 2 accepts input of image data. The prediction device 2 predicts a gaze area based on the input image data, using the trained prediction model acquired by the learning device 1. The prediction device 2 includes a predictor control unit 211. The predictor control unit 211 will be described in detail later, but the predictor control unit 211 controls the operation of a circuit that expresses the trained prediction model.
[0021] Note that "trained" means that learning has been performed until a predetermined end condition (hereinafter referred to as "learning end condition") is satisfied. Therefore, a trained mathematical model is a mathematical model at the time when the learning end condition is satisfied. The learning end condition may be, for example, a condition that learning has been performed a predetermined number of times, or the learning end condition may be, for example, a condition that the change in the learning model due to learning is smaller than a predetermined change.
[0022] A mathematical model is a set of one or more processes whose execution conditions and order are predetermined. For example, a process included in a mathematical model is a process of inputting a value to a predetermined function to obtain the value of the function.
[0023] Note that performing learning means updating a mathematical model. Updating a mathematical model means that the values of the parameters of the circuit representing the mathematical model are updated. At least some of the processes included in the mathematical model are represented by, for example, a neural network. Note that a neural network is an example of a circuit such as an electronic circuit, an electrical circuit, an optical circuit, or an integrated circuit that represents at least some of the processes of a mathematical model. That the mathematical model is updated by learning means that the values of the parameters of the circuit representing the mathematical model are updated. When a part of the circuit representing the mathematical model is a neural network, the parameters of the neural network are suitably adjusted based on a predefined quantity. The predefined quantity is, for example, the value of a predefined objective function (i.e., loss).
[0024] <Example of the form of expression of the prediction result> A form of expression (hereinafter referred to as "prediction result expression form") for expressing the result of prediction by a learned prediction model or a prediction model will be described.
[0025] FIG. 2 is an explanatory diagram for explaining a first example of the prediction result expression form in the embodiment. FIG. 2 is an example of a prediction result expression form that represents the fixation area of the prediction result as the position of a point that is easy to attract attention (hereinafter referred to as "fixation point"). FIG. 2 shows an image with a height of 2H + 1 pixels and a width of 2W + 1 pixels. In FIG. 2, for example, if the pixel at the center of the image is expressed as (0, 0), the position (i.e., coordinates) of any pixel in the image is expressed as (x, y) using the horizontal position x and the vertical position y.
[0026] The prediction result expression form of FIG. 2 can represent the fixation point by coordinates even when the fixation point exists outside the space shown in the image of FIG. 2. That is, even when the x-coordinate of the fixation point is outside the range of -W < x < W and the y-coordinate is outside the range of -H < y < H, the prediction result expression form of FIG. 2 can represent the fixation point by coordinates. The x-axis and the y-axis are predefined coordinate axes that are perpendicular to each other.
[0027] Figure 2 shows an example of a fixation point, which is located at a position (2W, 0) that is 2W pixels away from the center to the right. Since it is 2W away from the center to the right, the point located at position (2W, 0) is outside the range of the image.
[0028] Fig. 3 is an explanatory diagram for explaining a second example of a prediction result representation format in an embodiment. Fig. 3 shows an example of a prediction result representation format that represents the position of the gaze point as a result of prediction using a distance r from the center and an angle θ (i.e., polar coordinates). Note that in the prediction result representation format of Fig. 3, the distance r does not necessarily need to be used to represent the position of the gaze point. In the prediction result representation format of Fig. 3, the position of the gaze point may be represented only by the angle θ (i.e., the gaze direction).
[0029] FIG. 4 is an explanatory diagram for explaining a third example of a prediction result representation format in an embodiment. The example in FIG. 4 is an example of a prediction result representation format in which the position of a point is discretized and expressed using the result of discretization. The discretization method is, for example, as shown in FIG. 4, a method of dividing the inner and outer surfaces of an image into 24 regions in units of W×H. Hereinafter, each region after division is referred to as a discretized region. Since the inner and outer surfaces of the image are divided into 24 regions, the size of each discretized region is W×H. Each discretized region is assigned one of the identifiers of the division number from 1 to the division number, and any point is represented by the value of the identifier of each discretized region to which it belongs.
[0030] For example, point A in FIG. 4 belongs to a discretization region with an identifier of 5, and is therefore output as "5". In a discretization region to which no identifier is assigned, the point is represented, for example, by the identifier of the closest discretization region among the discretization regions to which identifiers are assigned. In the prediction result representation format, when the discretization region to which point A belongs is a discretization region to which no identifier is assigned, for example, information "no corresponding region" may be used to represent that the discretization region to which point A belongs is a discretization region to which no identifier is assigned. In addition, the size of each discretization region does not necessarily have to be W×H. In addition, the number of discretization regions does not necessarily have to be 24. The size of each discretization region and the number of discretization regions may be appropriately determined in advance by a user according to the scene to which the prediction system 100 is applied.
[0031] 5 is an explanatory diagram illustrating a fourth example of a prediction result representation format in the embodiment. FIG. 5 is an example of an output that represents a gaze area as a distribution. FIG. 5 shows the mean (W, H), variance σ 2 1 shows an example of an output in which the gaze area is expressed as an isotropic two-dimensional normal distribution of . The variance does not necessarily have to be isotropic. The variance may be different in the x-axis direction and in the y-axis direction. In addition, the distribution does not necessarily have to be a normal distribution. The distribution may be any probability distribution as long as it can indicate the position and range of the gaze area. The distribution may be, for example, a probability distribution.
[0032] The prediction result expression format is not limited to the examples of Figures 2 to 5, and may be any expression format that can express the trained prediction model or the results of prediction using a prediction model and can express a spatial region in at least an outer space of the space shown in the image.
[0033] Returning to the explanation of FIG. 1, the learning device 1 may update the prediction model by any method as long as the method is capable of learning the prediction model. The prediction model may be expressed by using a neural network, for example. Hereinafter, the circuit that expresses the prediction model is referred to as a predictor 10.
[0034] The learning device 1 updates the prediction model by updating the parameters of the predictor 10. The parameters of the predictor 10 at the time when the learning end condition is satisfied are transmitted to the prediction device 2. The prediction device 2 includes a circuit similar to the predictor 10, and operates the circuit using the parameters acquired from the learning device 1. As a result, the prediction device 2 executes the trained prediction model acquired by the learning device 1. Therefore, the circuit similar to the predictor 10 included in the prediction device 2 is a circuit to be controlled by the predictor control unit 211. Hereinafter, for ease of explanation, a circuit included in the prediction device 2 and similar to the predictor 10 will also be referred to as the predictor 10.
[0035] The method by which the learning device 1 acquires a trained prediction model will be described in detail later. To explain the method by which the learning device 1 acquires a trained prediction model, the predictor 10 will first be described.
[0036] <Explanation of predictor 10> The predictor 10 is a circuit that expresses a prediction model, and may be any circuit that can output the result of prediction by the prediction model (i.e., a gaze area in a space not shown in an image represented by image data) in a predetermined prediction result representation format. In other words, the predictor 10 may be any circuit that can predict a gaze area in a space not shown in an image represented by image data based on image data, and output the predicted result in a predetermined prediction result representation format. The predictor 10 is, for example, a circuit configured with a convolutional neural network (CNN), a long short-term memory (LSTM) network, and two fully connected layers (FC).
[0037] When the predictor 10 is a circuit composed of a CNN, an LSTM, and two fully connected layers, the network structure of the CNN constituting the predictor 10 may be any network structure as long as the predictor 10 can express a prediction model and can output a result in a predetermined prediction result expression format. The CNN constituting the predictor 10 may be, for example, ResNet described in Reference 1 below. ResNet is suitable for the prediction system 100 because it can directly input and acquire image data.
[0038] Reference 1: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[0039] For ease of explanation, the prediction system 100 will be described below taking as an example a case in which the predictor 10 is a circuit configured with a CNN, an LSTM, and two fully connected layers.
[0040] Fig. 6 is a diagram showing an example of the configuration of a predictor 10 in an embodiment. The predictor 10 shown in Fig. 6 includes a CNN 101, an LSTM 102, a prediction FC 103, and a stop determination FC 104. Image data is input to the CNN 101. The CNN 101 is a convolutional neural network (CNN), and acquires feature amounts (hereinafter referred to as "image features") of the input image data based on the input image data.
[0041] The LSTM 102 is a long short-term memory network, and acquires region features based on the image features and auxiliary information acquired by the CNN 101. The region features are outputs of the LSTM 102, and are input to the prediction FC 103 and the stop determination FC 104 located downstream of the LSTM 102. Therefore, the region features are a type of intermediate output. The region features are information including at least a part of the information indicated by the image features and the auxiliary information, and at least a part of the information indicated by the history of the results of prediction of the past gaze region. The type or amount of information included in the region information is updated by learning.
[0042] The auxiliary information is information input to LSTM102. The auxiliary information indicates a value according to the timing of input to LSTM102. The value indicated by the auxiliary information is a value that follows a predetermined distribution having a non-zero variance. The auxiliary information indicates, for example, a vector determined depending on the number of times input to LSTM102. The auxiliary information may indicate, for example, a region feature obtained before the input at the previous timing. The auxiliary information may indicate, for example, a statistical quantity of the distribution of a plurality of region features obtained before the input at the previous timing. The auxiliary information may indicate, for example, a vector (hereinafter referred to as an "auxiliary vector") having the same dimensionality d as the image feature, and whose element values are defined by the following formulas (1) and (2).
[0043]
number
[0044]
number
[0045] a(i, k) represents the value of the k-th element of the auxiliary vector at the i-th input. The i-th input refers to the i-th timing when auxiliary information is input to the LSTM 102. c is a constant. The value of the constant c is, for example, 10,000.
[0046] For simplicity of explanation, the prediction system 100 will be explained below taking as an example a case where the auxiliary information indicates an auxiliary vector.
[0047] <Effects of supplementary information> The effect of the auxiliary information will be described. To explain the effect of the auxiliary information, the LSTM (i.e., the LSTM 102) included in the predictor 10 will be described first. The LSTM 102 receives image features that are the output of the CNN 101 as input, and outputs region features. The number of attention regions in one input image is not necessarily one. Also, the number can change depending on the input image.
[0048] The predictor 10 outputs one or more attention regions by inputting image features to the LSTM 102 multiple times. The LSTM is a type of recursive neural network, and is a neural network that has internal state variables and determines the output based on both the input and the state variables. The state variables of the LSTM are updated each time an input is received. Therefore, even if the same image features are input to the LSTM 102, the output may change with each input. Therefore, the predictor 10 using the LSTM can predict attention regions that are not necessarily the same with each input, even if image features obtained from the same image data are input multiple times.
[0049] When auxiliary information is used, even if the input image features are the same as the image features at the time of past learning, information different from the information previously input to the predictor 10 is input to the predictor 10. As a result, even if the image features input to the predictor 10 are the same as the image features at the time of past learning, the predictor 10 outputs a result different from the past result more frequently. In this way, the auxiliary information has the effect of increasing the frequency with which the prediction results are not the same.
[0050] In this way, the predictor 10 is a circuit that predicts a gaze area in a space not shown in an image represented by the image data, for example, based on image data and auxiliary information, and outputs the predicted result in a predetermined prediction result representation format. Hereinafter, the process of predicting a gaze area in a space not shown in an image represented by the image data, based at least on image data, is referred to as a prediction process.
[0051] The area features are input to the prediction FC 103 and the stop determination FC 104. The prediction FC 103 outputs the gaze area based on the input area features. The output format of the prediction FC 103 (i.e., the prediction result expression format) is a predetermined format. The prediction result expression format is, for example, a format predetermined by a user according to the area features input to the prediction FC 103.
[0052] Prediction FC103 outputs two-dimensional coordinate values (x, y) when the gaze area is expressed as a coordinate value of the gaze point as in the example shown in FIG. 2. Prediction FC103 outputs the probability that the gaze area exists in each discretized area when the position of the point is discretized as in the example shown in FIG. 4. Prediction FC103 outputs the parameters of the distribution when the position and range of the gaze area are expressed using a distribution as in the example shown in FIG. 5. The parameters of the distribution output by Prediction FC103 are, for example, the average x and y values and the variance σ in the example shown in FIG. 5. 2 and the value of
[0053] The stop determination FC104 determines whether or not to stop the prediction process based on the region feature. More specifically, the stop determination FC104 outputs stop determination information t, which is a binary value indicating whether or not to stop the prediction process based on the region feature. The condition for the stop determination FC104 to stop the prediction process is obtained by learning. Obtaining by learning means, for example, obtaining the condition by updating the condition so that the loss function is minimized.
[0054] When the predictor 10 is configured to output one or more attention regions by inputting image features to the LSTM multiple times, one or more attention regions are output. However, as described above, it is not self-evident how many outputs should be obtained. Therefore, the stop determination FC 104 makes a determination according to the situation based on the area features. Specifically, "according to the situation" means based on information generated during the prediction process.
[0055] The number of times the output is obtained may be, for example, a predetermined number (hereinafter referred to as the "maximum predicted number of times") T. In such a case, the stop determination FC104 determines to stop the prediction process when the number of times the output is obtained reaches the maximum predicted number of times T, and determines not to stop the prediction process when the number of times the output is obtained is less than the maximum predicted number of times T.
[0056] The appropriate number of predictions (i.e., the number of gaze regions) varies for each image data input to the predictor 10. Therefore, the determination of whether or not to stop the prediction process by the stop determination FC104 is preferably based on the above-mentioned region features rather than a determination using a predetermined maximum number of predictions T.
[0057] 7 is a flowchart showing an example of the flow of processing executed by the predictor 10 in the embodiment. Image data is input to the CNN 101 (step S101). The image data is, for example, image data of an image captured by a camera. The image data may be, for example, image data of a frame extracted from a video, or may be, for example, image data of a frame of a video captured by an on-board camera of an automobile.
[0058] Next, the predictor control unit 211 sets the number of times i that the prediction process has started to 1 (step S102). Setting the number of times i to 1 means initializing the auxiliary information. Next, based on the image data of the image input to the CNN 101, the image feature f is obtained (step S103). Next, based on the image feature f and the auxiliary information ai, the LSTM 102 obtains the region feature (step S104). Next, based on the region feature, the prediction FC 103 predicts the fixation region (step S105). Next, the prediction FC 103 outputs a prediction result Oi indicating the result of the prediction (step S106).
[0059] Next, the stop determination FC 104 determines whether to stop the prediction process based on the region feature (step S107). Next, the stop determination FC 104 outputs information indicating the result of the determination (that is, the stop determination information ti) (step S108). Next, the predictor control unit 211 determines whether the prediction end condition is satisfied based on the stop determination information (step S109). The prediction end condition is a condition regarding the end of the prediction process and is at least a condition based on the stop determination information. The prediction end condition is, for example, a condition that either one of the condition that the stop determination information ti indicates stop and the condition that the number of times the prediction process has started is greater than or equal to the maximum number of predictions T (that is, i < T) is satisfied.
[0060] If the prediction end condition is satisfied, the process ends. On the other hand, if the prediction end condition is not satisfied, the predictor control unit 211 updates the auxiliary information ai according to a predetermined update rule (step S110). After step S110, the process returns to the process of step S104.
[0061] Note that step S106 may be executed at any timing as long as it is executed after step S105 and before step S109. Note that step S108 may be executed at any timing as long as it is executed after step S107 and before step S109. Note that the process of step S105 may be executed after the process of step S107 as long as it is executed after step S104 and before step S109.
[0062] The maximum prediction count T is any positive number, for example, T = 5. The maximum prediction count may be any positive number, but is preferably a value larger than the maximum value of the number of correct fixation regions indicated by the teacher data used when the learning device 1 learns the prediction model.
[0063] In this way, the predictor 10 executes image feature extraction processing, prediction processing, and stop determination processing. The image feature extraction processing is processing for acquiring image features based on image data. In the example of FIG. 7, this is the processing of step 103. In the example of FIG. 7, the prediction processing is processing indicated by a series of steps S104, S105, and S106. The stop determination processing is processing for determining whether or not to stop the prediction processing based at least on image data. In the example of FIG. 7, the stop determination processing is processing indicated by a series of steps S104, S107, and S108.
[0064] By such processing, the predictor 10 can obtain a maximum of T prediction results through a maximum of T prediction processes. Hereinafter, the number of prediction results by the predictor 10 will be represented as N.
[0065] <Method in which the learning device 1 obtains a trained predictive model> The learning device 1 uses training data to learn a prediction model. The training data is a pair of image data and data showing a set of correct gaze areas (hereinafter referred to as "correct gaze area data"). The correct gaze areas are gaze areas in space that are not captured in the image shown by the corresponding image data. Therefore, the set of correct gaze areas is a set of gaze areas included in the corresponding image.
[0066] More specifically, the training data D is data that includes image data as learning data (i.e., data on the explanatory variable side) and correct gaze area data as teacher data (i.e., data on the objective variable side). The image represented by the image data of the learning data may be an image that includes only one gaze area, or an image that includes multiple gaze areas. Hereinafter, the image data of the learning data will be referred to as image data I, and the set of correct gaze areas will be referred to as set {Sj} (j=1, , M). Therefore, the training data is set D={(I, {Sj})}. Therefore, hereinafter, the training data will be referred to as training data D.
[0067] The method of learning the prediction model may be any method using training data, for example, a reinforcement learning method. An example of a process of learning the prediction model by reinforcement learning (hereinafter referred to as "prediction model reinforcement learning process") will be described below.
[0068] The prediction model reinforcement learning process includes a reinforcement prediction process, a reward acquisition process, a reinforcement prediction learning process, and a stop determination learning process. Before describing each process, an overview of reinforcement learning will be provided.
[0069] Reinforcement learning is a learning method for learning optimal behavior decision-making policies for agents that decide their actions in a certain situation. Reinforcement learning is a learning method for learning agents by prescribing rewards according to whether the final situation brought about as a result of a series of agent actions is desirable or not.
[0070] In the prediction model reinforcement learning process, a predictor 10 is used as an agent in reinforcement learning. In the prediction model reinforcement learning process, a prediction of a gaze area and a decision to stop the prediction process are used as actions in reinforcement learning. In the prediction model reinforcement learning process, image data I, the correctness or incorrectness of past prediction results, and the correctness or incorrectness of the decision to stop are used as situations in reinforcement learning.
[0071] As described above, reinforcement learning is a search-type learning by trial and error. More specifically, in reinforcement learning, after obtaining a result using a mathematical model to be updated, a reward is calculated based on the obtained result, and the mathematical model to be updated is updated based on the reward. Therefore, in reinforcement learning, first, a process of obtaining a result using a mathematical model to be updated is performed. In the prediction model reinforcement learning process, the process of obtaining a result using a mathematical model to be updated is the reinforcement prediction process.
[0072] Therefore, the enhanced prediction process is a process in which the predictor 10 is used to execute the prediction process and the stop determination process.
[0073] FIG. 8 is a flowchart showing an example of the flow of the reinforcement prediction process in the embodiment. The learning control unit 111, which will be described later, acquires training data D (step S201). Next, the learning control unit 111 sets the number of times i at which the execution of the reinforcement prediction process has been started to 1 (step S202). Setting the number of times i to 1 means initializing auxiliary information. Next, the CNN 101 acquires an image feature f based on the image data of the input image (step S203). Next, the LSTM 102 acquires a region feature based on the image feature f and the auxiliary information ai (step S204). Next, the prediction FC 103 predicts the gaze region based on the region feature (step S205). Next, the prediction FC 103 outputs a prediction result Oi indicating the result of the prediction. The output prediction result Oi is recorded by the learning control unit 111 in a predetermined storage device such as the storage unit 13, which will be described later (step S206).
[0074] Next, a stop determination FC104 determines whether to stop the prediction process based on the region features (step S207). Next, the stop determination FC104 outputs information indicating the result of the determination (i.e., stop determination information ti). The output stop determination information ti is recorded in a predetermined storage device by the learning control unit 111 (step S208). Next, the learning control unit 111 determines whether the learning-time prediction end condition is satisfied based on the stop determination information (step S209). The reinforcement prediction end condition is a condition regarding the end of the reinforcement prediction process and is at least a condition based on the stop determination information. The reinforcement prediction end condition is a condition different from the prediction end condition in that the process to be determined whether to stop is the reinforcement prediction process instead of the prediction process. The reinforcement prediction end condition is, for example, a condition that either one of the condition that the stop determination information ti indicates stop and the condition that the number of times the execution of the reinforcement prediction process has started is equal to or greater than the maximum prediction number T (i.e., i < T) is satisfied.
[0075] When the reinforcement prediction end condition is satisfied, the process ends. On the other hand, when the reinforcement prediction end condition is not satisfied, the learning control unit 111 updates the auxiliary information ai according to a predetermined update rule (step S210). After step S210, the process returns to the process of step S204.
[0076] Note that step S206 may be executed at any timing after the execution of step S205 and before the execution of step S209. Note that step S208 may be executed at any timing after the execution of step S207 and before the execution of step S209. Note that the process of step S205 may be executed after the process of step S207 as long as it is after the execution of step S204 and before the execution of step S209.
[0077] The process of FIG. 8 is executed for all the prepared training data D.
[0078] The reward acquisition process includes a true / false judgment process. The true / false judgment process is a process for judging whether or not the set of prediction results {Oi} obtained by the reinforced prediction process correctly predicts the set of normal gaze areas {Sj}. In the reward acquisition process, after the true / false judgment process is executed, a reward is acquired based on the result of the true / false judgment process. The reward is acquired, for example, by calculation.
[0079] FIG. 9 is a flowchart showing an example of the flow of the reward acquisition process in the embodiment. The learning control unit 111 initializes the number of successful predictions Q (step S301). As a result of the initialization, 0 is substituted for the number of successful predictions Q. Next, the learning control unit 111 selects one prediction result Oi for which the success or failure of the prediction has not yet been determined from the set of prediction results {Oi} obtained by the reinforcement prediction process (step S302). Next, the learning control unit 111 determines the success or failure of the prediction of the prediction result Oi selected in step S302 (step S303). Specifically, determining the success or failure of the prediction means determining whether or not at least one of the correct gaze areas {Sj} has been predicted. If at least one of the correct gaze areas {Sj} has been predicted, the prediction is successful, and if none of the correct gaze areas {Sj} has been predicted, the prediction is not successful (i.e., it is not successful).
[0080] The method of judging whether the prediction is successful or not depends on the representation format of the prediction result. As an example, an example of the method of judging whether the prediction is successful or not will be described for the case where the gaze area is given by the gaze point as shown in FIG. 2 or FIG. 3. In this case, both Oi and {Sj} represent points. Therefore, the success or failure of the prediction is judged by determining S* as the closest one to Oi among {Sj}, and judging whether the prediction is successful or not if the distance between S* and Oi is equal to or less than a certain value, and judging whether the prediction is unsuccessful if it is greater than the certain value.
[0081] As another example of the method of determining whether prediction is successful or not, an example of the method of determining whether prediction is successful or not will be described for the case where the region is discretized as shown in Fig. 4. In this case, the success or failure of the prediction is determined by a method in which the prediction is determined to be successful if Oi and at least one S* of {Sj} indicate the same region, and the prediction is determined to be unsuccessful if they do not. In addition, in the case where the region is expressed by a distribution as shown in Fig. 5, the success or failure of the prediction is determined by a method in which the prediction is determined to be successful if the overlap between the region covered by the distribution of Oi and the region covered by at least one S* of {Sj} is equal to or greater than a certain amount, and the prediction is determined to be unsuccessful if it is less than a certain amount.
[0082] After step S303, the learning control unit 111 updates the number of successful predictions Q and removes the correct gaze area predicted by the prediction result Oi from the set of correct gaze areas {Sj} (step S304). Specifically, the update of the number of successful predictions Q is a process of increasing the value of the number of successful predictions by 1.
[0083] Next, the learning control unit 111 judges whether or not a reward acquisition condition is satisfied (step S305). The reward acquisition condition is a condition that at least one of the following conditions is satisfied: the success or failure of prediction has been judged for all prediction results Oi obtained by the reinforcement prediction process; and the set of normal gaze regions {Sj} is an empty set. The process of judging the success or failure of prediction is specifically the process of step S303.
[0084] If the reward acquisition condition is not satisfied (step S305: NO), the process returns to step S302. On the other hand, if the reward acquisition condition is satisfied (step S305: YES), the learning control unit 111 acquires a predefined reward value (step S306).
[0085] <About rewards> The reward is, for example, the amount R defined by the following formula (3): pred It is.
[0086]
number
[0087] The more appropriate the reward, the higher the probability of obtaining a desirable prediction model. In addition, it is preferable to perform learning that can predict as many gaze areas as possible among multiple correct gaze areas with as few predictions as possible as accurately. Therefore, by giving a larger reward when as many gaze areas as possible can be predicted as accurately as possible with as few predictions as possible, the probability of performing desirable learning is increased.
[0088] The value of the left side of the above formula (3) is between 0 and 1. The value of the left side of the above formula (3) is the maximum value 1.0 when the number of predictions N is the same as the number of correct gaze areas M and all predictions are successful, that is, when M = N = Q. Therefore, the reward R defined in formula (3) is pred satisfies the property that a high reward is given when as many correct gaze areas as possible are predicted as accurately as possible with as few predictions as possible. Therefore, the reward R defined in Equation (3) is pred is suitable for training predictive models.
[0089] The reinforcement prediction learning process will be described. The reinforcement prediction learning process updates the prediction model based on the reward obtained in the reward acquisition process. For example, when the predictor 10 includes a CNN 101, an LSTM 102, a prediction FC 103, and a stop determination FC 104, the reinforcement prediction learning process updates the NN 101, the LSTM 102, and the prediction FC 103 based on the reward. Hereinafter, for ease of explanation, the deep neural network combining the NN 101, the LSTM 102, and the prediction FC 103 is represented as φ.
[0090] The reinforcement prediction learning process is a process of updating the network φ based on a reward. The process of updating the network φ based on a reward in the reinforcement prediction learning process may be any method based on a reward. For example, the update method may be a policy gradient method. When the update method is the policy gradient method, the weight Wφ of the network φ is updated based on the following formulas (4) and (5).
[0091]
number
[0092]
number
[0093] α is an arbitrary positive real value that represents the learning rate. For example, α is 0.01.
[0094] In this way, in the prediction model reinforcement learning process, a process of estimating a gaze area based on image data included in the training data (i.e., reinforced prediction process) is executed using a prediction model. In the prediction model reinforcement learning process, the prediction model is updated by executing the reward acquisition process and the reinforced prediction learning process so that the estimation result obtained by the reinforced prediction process approaches the gaze area indicated by the correct gaze area data included in the training data.
[0095] Here, when reinforcement learning is used to train the network φ, a prediction model with higher estimation accuracy can be generated than when supervised learning is used. The reason why reinforcement learning has higher estimation accuracy will be explained.
[0096] One of the reasons is that it is not obvious which correct gaze area among the multiple correct gaze areas {Sj} should be compared with the i-th prediction result Oi by the predictor 10. One of the reasons is that the prediction by the predictor 10 is repeatedly performed until a prediction end condition is satisfied, so that it is necessary to comprehensively judge the results of multiple predictions to proceed with learning. For example, even if the first prediction result O1 correctly indicates the first one S1 of the correct gaze areas, if the second and subsequent prediction results O2, ..., ON indicate the exact same correct gaze area S1, it is difficult to say that the second and subsequent prediction results are meaningful prediction results.
[0097] In this way, the quality of a prediction model cannot be judged based on each prediction result, but must be judged based on multiple prediction results (i.e., a comprehensive judgment). Furthermore, it is desirable for a prediction model to be able to predict the correct gaze area accurately. One reason for this is that it is desirable for not only the gaze area prediction result {Oi} but also the stop judgment information {ti} to be highly accurate.
[0098] The stop judgment learning process will be described. The stop judgment learning process is a process for updating the contents of the stop judgment process. The stop judgment learning process updates, for example, the CNN 101, the LSTM 102, and the stop judgment FC 104. It is preferable that the stop judgment process is a process for determining the timing when all correct gaze regions have been predicted as a stop. The timing when all correct gaze regions have been predicted is, for example, the timing when {Sj} becomes an empty set in step S305.
[0099] Unlike the prediction process, the stop judgment process has clear teaching data for the output. Therefore, even if the stop judgment process is updated using a supervised machine learning method, the process generated has the same or higher accuracy as when it is updated using reinforcement learning.
[0100] The stop judgment learning process updates the CNN 101, LSTM 102, and stop judgment FC 104 provided in the predictor 10. For ease of explanation, the deep neural network combining the NN 101, LSTM 102, and stop judgment FC 104 will be represented as ψ below. The learning method performed by the stop judgment learning process may be any method as long as it is a supervised learning method. The stop judgment learning process updates the contents of the stop judgment process by, for example, a gradient method. As an example, the stop judgment learning process will be described in the case where the learning method performed by the stop judgment learning process is a gradient method.
[0101] For ease of explanation, the stop judgment information whose contents are correct (hereinafter referred to as "correct stop judgment information") is represented as ui, which is the correct stop judgment information of identifier i. For example, the correct stop judgment information ui of identifier i indicates 1 when {Sj} becomes empty, and indicates 0 otherwise. The stop judgment learning process updates the weight Wψ of ψ so that the loss function of the following equation (6) becomes smaller.
[0102]
number
[0103] The loss function in equation (6) is a loss function called Binary Cross Entropy. The loss function in equation (6) takes a small value when the stop determination information ti matches the correct stop determination information ui. Therefore, by updating Wψ so that the value (i.e., the loss) of the loss function in equation (6) becomes small, ψ that can output ti close to ui can be obtained. The update method is, for example, a gradient method.
[0104] The learning device 1 performs learning of a prediction model by repeatedly executing each of the subroutines of a reinforcement prediction process, a reward acquisition process, a reinforcement prediction learning process, and a stop judgment learning process in sequence.
[0105] 10 is a diagram showing an example of a hardware configuration of a learning device 1 in an embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. The learning device 1 functions as a device including the control unit 11, an input / output interface 12, and a storage unit 13 by executing the program.
[0106] More specifically, in the learning device 1, the processor 91 reads a program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the learning device 1 functions as a device including the control unit 11, the input / output interface 12, and the storage unit 13.
[0107] The control unit 11 controls the operation of various functional units of the learning device 1. The input / output interface 12 includes an interface for connecting the learning device 1 to an external device. The input / output interface 12 communicates with the external device via wired or wireless communication. The external device is, for example, the prediction device 2. The input / output interface 12 also includes input devices such as a mouse, keyboard, and touch panel. The input / output interface 12 may also include an interface for connecting these input devices to the learning device 1.
[0108] The input / output interface 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The input / output interface 12 may include an interface that connects these display devices to the learning device 1. For example, training data is input to the input / output interface 12.
[0109] The storage unit 13 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information resulting from the processing executed by the control unit 11, for example. The storage unit 13 stores, for example, parameter values of the predictor 10. The storage unit 13 stores, for example, auxiliary information in advance. The storage unit 13 may store, for example, the maximum number of predictions T in advance.
[0110] 11 is a diagram illustrating an example of a functional configuration of the control unit 11 in the embodiment. The control unit 11 includes a learning control unit 111 and a predictor 10. The learning control unit 111 controls the operation of the predictor 10 and executes learning of the predictor 10. The learning control unit 111 executes, for example, a prediction model reinforcement learning process.
[0111] 12 is a diagram showing an example of a hardware configuration of a prediction device 2 in an embodiment. The prediction device 2 includes a control unit 21 including a processor 93 such as a CPU (Central Processing Unit) and a memory 94 connected by a bus, and executes a program. The prediction device 2 functions as a device including the control unit 21, an input / output interface 22, and a storage unit 23 by executing the program.
[0112] More specifically, in the prediction device 2, the processor 93 reads a program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, whereby the prediction device 2 functions as a device including the control unit 21, the input / output interface 22, and the storage unit 23.
[0113] The control unit 21 controls the operation of various functional units included in the prediction device 2. The input / output interface 22 includes an interface for connecting the prediction device 2 to an external device. The input / output interface 22 communicates with the external device via wired or wireless communication. The external device is, for example, the learning device 1. The input / output interface 22 also includes input devices such as a mouse, keyboard, and touch panel. The input / output interface 22 may include an interface for connecting these input devices to the prediction device 2.
[0114] The input / output interface 22 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The input / output interface 22 may include an interface that connects these display devices to the prediction device 2. To the input / output interface 22, for example, image data of an estimation target is input.
[0115] The storage unit 23 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the prediction device 2. The storage unit 23 stores various information resulting from the processing executed by the control unit 21, for example. The storage unit 23 stores, for example, a learned prediction model. The storage unit 23 stores, for example, auxiliary information in advance. The storage unit 23 may store, for example, the maximum number of predictions T in advance.
[0116] 13 is a diagram illustrating an example of a functional configuration of the control unit 21 in the embodiment. The control unit 21 includes a predictor control unit 211 and a predictor 10. The predictor control unit 211 controls the operation of the predictor 10 included in the prediction device 2. In other words, the predictor control unit 211 causes the predictor 10 to execute a learned prediction model.
[0117] <Application Examples> An application example of the prediction system 100 will be described. The prediction device 2 is installed in a vehicle, for example. In this case, the prediction device 2 predicts an area to be watched outside the visible area based on an image acquired by a camera mounted on the vehicle. After the prediction, the prediction device 2 indicates the area to be watched or paid attention to, for example, via a visual or audio warning.
[0118] The prediction device 2 may be mounted on, for example, a robot. In this case, the prediction device 2 predicts an area to be watched outside the visible area from an image captured by a camera mounted on the robot. After the prediction, the prediction device 2 controls the behavior of the robot by transmitting the prediction result to a control device that controls the behavior of the robot based on the prediction result.
[0119] The prediction system 100 configured in this way includes a learning device 1 that obtains a mathematical model that indicates the relationship between image data and a gaze area in a space that is not shown in the image represented by the image data. Therefore, the prediction system 100 can predict a gaze area in a space that is not shown in the image. As described above, the same applies to video data instead of image data.
[0120] The prediction system 100 also includes a prediction device 2 that predicts a gaze area using the mathematical model obtained by the learning device 1. Therefore, the prediction system 100 can predict a gaze area in an area in space that is not captured in an image. As described above, the same applies to video data instead of image data.
[0121] (Modification) Each of the prediction system 100, the learning device 1, and the prediction device 2 may be implemented using a plurality of information processing devices communicatively connected via a network. All or part of the functions of the prediction system 100, the learning device 1, and the prediction device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded in a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system. The program may be transmitted via an electric communication line. The learned prediction model is an example of an updated mathematical model. The prediction device 2 is an example of an estimation device. Prediction is an example of estimation.
[0122] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]
[0123] 100... prediction system, 1... learning device, 2... prediction device, 10... predictor, 11... control unit, 12... input / output interface, 13... memory unit, 111... learning control unit, 21... control unit, 22... input / output interface, 23... memory unit, 211... predictor control unit, 91... processor, 92... memory, 93... processor, 94... memory
Claims
1. A control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data that is a pair of data of image data and correct gaze area data that is data indicating a gaze area in a space not shown in the image represented by the image data; Equipped with The mathematical model estimates a gaze area in a space not shown in the image based on information having a value corresponding to the image data and a timing at which the image data is input, The control unit estimates a gaze area based on image data included in training data using the mathematical model, and updates the mathematical model so that a result of the estimation approaches the gaze area indicated by correct gaze area data included in the training data. Learning device.
2. a control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data which is a pair of data of image data and correct gaze area data which is data indicating a gaze area in a space not shown in an image represented by the image data, the mathematical model estimates a gaze area in a space not shown in the image, based on information having a value according to the image data and a timing of input, the control unit uses the mathematical model to estimate a gaze area based on image data included in training data, and updates the mathematical model so that the result of the estimation approaches the gaze area indicated by the correct gaze area data included in the training data, a learning device includes a control unit that estimates a gaze area in a space not shown in an image represented by the image data, based on input image data, using an updated mathematical model which is the mathematical model updated until a predetermined termination condition is satisfied; An estimation device comprising:
3. A control step of updating a mathematical model showing a relationship between the image data and a gaze area in a space not shown in the image represented by the image data, based on training data which is a pair of data of the image data and correct gaze area data which is data showing a gaze area in a space not shown in the image represented by the image data; having The mathematical model estimates a gaze area in a space not shown in the image based on the image data and information having a value according to the input timing, In the control step, a gaze area is estimated based on image data included in training data using the mathematical model, and the mathematical model is updated so that the estimation result approaches the gaze area indicated by correct gaze area data included in the training data. How to learn.
4. a control unit that updates a mathematical model indicating a relationship between the image data and the gaze area in the space not shown in the image represented by the image data, based on training data which is a pair of data of image data and correct gaze area data which is data indicating a gaze area in the space not shown in the image represented by the image data, the mathematical model estimates the gaze area in the space not shown in the image, based on information having a value according to the image data and a timing of input, the control unit uses the mathematical model to estimate the gaze area based on the image data included in the training data, and updates the mathematical model so that the result of the estimation approaches the gaze area indicated by the correct gaze area data included in the training data, a control step of estimating the gaze area in the space not shown in the image represented by the image data, based on the input image data, using the updated mathematical model, which is the mathematical model updated until a predetermined termination condition is satisfied, by a learning device; The estimation method has the following structure:
5. A control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data that is a pair of data of image data and correct gaze area data that is data indicating a gaze area in a space not shown in the image represented by the image data; Equipped with The control unit estimates a gaze area based on image data included in training data using the mathematical model, and updates the mathematical model so that a result of the estimation approaches the gaze area indicated by correct gaze area data included in the training data. The mathematical model is represented using a long short-term memory network, The mathematical model estimates the gaze area based on auxiliary information, which is information input to the long short-term memory network, and whose indicated value is a value according to the timing of input to the long short-term memory network, and the value is a value following a predetermined distribution having a variance other than 0. Learning device.
6. The control unit updates the mathematical model by a reinforcement learning method. The learning device according to claim 5 .
7. a control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data which is a pair of data of image data and correct gaze area data which is data indicating a gaze area in a space not shown in an image represented by the image data, the control unit uses the mathematical model to estimate a gaze area based on image data included in the training data, and updates the mathematical model so that a result of the estimation approaches the gaze area indicated by the correct gaze area data included in the training data, the mathematical model being expressed using a long short-term memory network, the mathematical model estimating the gaze area based on auxiliary information which is information input to the long short-term memory network, a value indicated by the mathematical model being a value according to a timing at which the value is input to the long short-term memory network, and the value is a value following a predetermined distribution having a variance other than 0, the control unit estimating a gaze area in a space not shown in an image represented by the image data, based on the input image data, using an updated mathematical model which is the mathematical model updated by a learning device until a predetermined termination condition is satisfied; An estimation device comprising:
8. A control step of updating a mathematical model showing a relationship between the image data and a gaze area in a space not shown in the image represented by the image data, based on training data which is a pair of data of the image data and correct gaze area data which is data showing a gaze area in a space not shown in the image represented by the image data; having In the control step, a gaze area is estimated based on image data included in training data using the mathematical model, and the mathematical model is updated so that the estimation result approaches the gaze area indicated by correct gaze area data included in the training data. The mathematical model is expressed using a long short-term memory network, and the mathematical model estimates the gaze area based on auxiliary information, which is information input to the long short-term memory network, and whose indicated value is a value according to a timing at which the value is input to the long short-term memory network, and the value is a value according to a predetermined distribution having a variance other than 0. How to learn.
9. a control unit that updates a mathematical model indicating a relationship between image data and a gaze area in a space not shown in an image represented by the image data, based on training data that is a pair of data of image data and correct gaze area data that is data indicating a gaze area in a space not shown in an image represented by the image data, wherein the control unit uses the mathematical model to estimate a gaze area based on image data included in the training data, and updates the mathematical model so that a result of the estimation approaches the gaze area indicated by the correct gaze area data included in the training data, and the mathematical model is expressed using a long short-term memory network, and the mathematical model estimates the gaze area based on auxiliary information that is information input to the long short-term memory network, a value indicated by the mathematical model is a value according to a timing at which the mathematical model is input to the long short-term memory network, and the value is a value according to a predetermined distribution having a variance other than 0; The estimation method has the following structure:
10. A program for causing a computer to function as either one of the learning device according to claim 1, 5 or 6 and the estimation device according to claim 2 or 7.
Citation Information
Patent Citations
Position estimation device, position learning device and program
JP2021064025A
