Learning device, estimation device, learning method, estimation method, and program

Abstract

To accurately estimate a target position specified by referring to a reference position.SOLUTION: A learning device for performing machine learning comprises: an acquisition unit for acquiring teacher data including input data and correct answer data, wherein the input data includes an input image containing a reference object and an input text that relatively specifies a target position by referring to the reference object; a generation unit foe generating output data for identifying the target position by inputting the input data into a model; and an update unit for updating parameters of the model to reduce the loss obtained by inputting the output data and the correct answer data into a loss function. The model includes: a first sub-model that generates a plurality of feature amounts representing the reference object and having mutually different resolutions based on the input image and the input text, and a second sub-model that generates output data based on the plurality of feature amounts and the input text. Each of the plurality of feature amounts is input to the second sub-model.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program. [Background technology]

[0002] Various technologies for controlling vehicle driving using models generated by machine learning have been proposed. Patent Document 1 describes training a neural network using sensor data acquired by a vehicle. A technology has also been proposed for estimating a position in an image specified by language using a multimodal model that takes images and language as input. Proposed multimodal models include FIBER (Fusion-In-the-Backbone-based transformER) (Non-Patent Document 1), CLIP (Contrastive Language-Image Pre-training) (Non-Patent Document 2), and PWAN (Pixel-Word Attention Module) (Non-Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-513866 [Non-patent literature]

[0004] [Non-Patent Document 1] Zi-Yi Dou, et al., “Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone”, [online], November 18, 2020, arXiv, [Retrieved March 16, 2020], Internet<URL:https: / / arxiv.org / pdf / 2206.07643.pdf> [Non-patent document 2] Alec Radford, et al., “Learning Transferable Visual Models From Natural Language Supervision”, [online], February 26, 2021, arXiv, [searched March 16, 2021], Internet<URL:https: / / arxiv.org / pdf / 2103.00020.pdf> [Non-patent document 3] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation Zhao Yang, et al., “LAVT: Language-Aware Vision Transformer for Referring Image Segmentation”, [online], April 5, 2022, arXiv, [Retrieved March 16, 2024], Internet<URL:https: / / arxiv.org / pdf / 2112.02244.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] A target position in an input image can be specified by referring to a reference object included in the input image. The reference object can have various sizes in the input image. One aspect of the present invention aims to accurately estimate the target position specified by referring to the reference object. [Means for solving the problem]

[0006] According to some embodiments, there is provided a learning device for performing machine learning, comprising: an acquisition means for acquiring teacher data including input data and ground truth data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object; a generation means for generating output data for identifying the target position by inputting the input data into a model; and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the output data and the ground truth data into a loss function, wherein the model includes: a first sub-model that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions from each other; and a second sub-model that generates the output data based on the plurality of feature quantities and the input text, wherein each of the plurality of feature quantities is input to the second sub-model. [Effects of the Invention]

[0007] According to some embodiments, the target location specified can be refined with reference to a reference location. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a computer according to some embodiments. [Figure 2] FIG. 10 is a schematic diagram illustrating an example of input data according to some embodiments. [Figure 3] FIG. 10 is a schematic diagram illustrating an example of correct answer data according to some embodiments. [Figure 4] FIG. 10 is a schematic diagram illustrating an example of the configuration of a model according to some embodiments. [Figure 5] FIG. 4 is a schematic diagram illustrating an example of the configuration of a feature extraction unit according to some embodiments. [Figure 6] FIG. 4 is a schematic diagram illustrating an example of the configuration of a target position estimation unit according to some embodiments. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of a loss function according to some embodiments. [Figure 8] FIG. 1 is a flow diagram illustrating an example of a training method according to some embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating an example estimation method according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] With reference to FIG. 1 , an example hardware configuration of a computer 100 according to some embodiments will be described. As will be described in detail below, the computer 100 is used to train a model by machine learning. Therefore, the computer 100 may be referred to as a learning device. The computer 100 may be, for example, a server computer or a personal computer (e.g., a desktop or laptop computer). The computer 100 may also be a computer resource located in a cloud environment.

[0011] The computer 100 may include the hardware devices shown in Fig. 1. The processor 101 controls the overall operation of the computer 100. The processor 101 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or a combination of these. The processor 101 may be a single processor or a collection of multiple processors connected to each other so that they can communicate with each other.

[0012] The memory 102 stores programs and data used in the processing of the computer 100. The memory 102 may be configured, for example, by a combination of random access memory (RAM) and read-only memory (ROM).

[0013] The input device 103 is a device for obtaining instructions from a user of the computer 100. The input device 103 may be configured, for example, by a combination of one or more of a keyboard, buttons, a touchpad, and a microphone. The display device 104 is a device for visually presenting information to a user of the computer 100. The display device 104 may be, for example, a dot-matrix display such as a liquid crystal display. The computer 100 may have a device (e.g., a touch screen) in which the input device 103 and the display device 104 are integrated. The input device 103 and the display device 104 may be external to the computer. In this case, the computer 100 may have an interface for communicating with the external input device 103 and display device 104.

[0014] The communication device 105 is a device for communicating with devices external to the computer 100. When the computer 100 performs wired communication, the communication device 105 may be a network interface card (NIC) having a connector for connecting a cable. When the computer 100 performs wireless communication, the communication device 105 may be a wireless communication module including an antenna and a baseband processing circuit.

[0015] The secondary storage device 106 is a device for non-volatilely storing programs and data used in the processing of the computer 100. The secondary storage device 106 is configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0016] The computer 100 may be capable of communicating with an external database 110. The database 110 may store training data 111 used for machine learning by the computer 100. The computer 100 may acquire the training data 111 from the database 110. Alternatively or in addition, the training data 111 may be stored in the secondary storage device 106 of the computer 100. In machine learning, a plurality of different training data 111 is used. Two sets of training data 111 are different in that the input data 112 included in these sets of training data 111 are different (for example, at least one of the input text 201 and the input image 202 described below is different). A portion of the training data 111 may be used as verification data and test data.

[0017] The training data 111 includes input data 112 and supervised data 113. The input data 112 may be data input to a model (e.g., model 400 in FIG. 4) to train the model. The supervised data 113 may be data to be output by the model.

[0018] An example of the input data 112 will be described with reference to Fig. 2. The input data 112 may include a pair of an input image 202 including a reference object 203 and input text 201 that relatively specifies a target position by referencing the reference object 203. The input text 201 may represent an instruction for the operation of the vehicle 210 by an occupant of the vehicle 210.

[0019] The input image 202 may be any image including an object. The input image 202 may be an image captured by a camera 211 of the vehicle 210. For example, the input image 202 may be an image captured by a camera 211 attached to the vehicle so as to capture an image in front of the vehicle 210. Alternatively, the input image 202 may be an image captured by a camera attached to the vehicle so as to capture an image in another direction (e.g., rearward) of the vehicle 210. The camera 211 of the vehicle 210 may be a camera 211 attached to the vehicle 210 or a camera brought into the vehicle (e.g., a smartphone of a vehicle occupant). The input image 202 may be an image not related to the vehicle.

[0020] The reference object 203 may be any object included in the input image 202. In the example of Fig. 2, a vehicle is used as the reference object 203. Alternatively, the reference object 203 may be a traffic participant other than a vehicle, a road sign, a traffic light, a guardrail, an intersection, a crosswalk, a building, a signboard, etc.

[0021] The input text 201 may be expressed in a natural language, for example, "Park in front of the black car on the right." In this example, "the black car on the right" in the input text 201 specifies a reference object 203, and "in front of" in the input text 201 specifies a target position relative to the reference object 203. Instead of being expressed in a natural language, the input text 201 may be expressed in other formats. For example, the input text 201 may be selected from a plurality of candidates for combinations of a reference object and a positional relationship that are set in advance.

[0022] An example of the correct answer data 113 will be described with reference to Fig. 3. The correct answer data 113 may be data for representing the correct position of the target position specified by the input text 201. The correct answer data 113 may be set manually for the input data 112 or may be set by a computer. In the following description, the correct position of the target position will be referred to as the correct target position.

[0023] The correct target position may be represented as a point 301 in the input image 202. Alternatively, the correct target position may be represented as an area centered on the point 301. The point 301 may be represented by each coordinate value in a two-dimensional coordinate system set in the input image 202 (hereinafter simply referred to as the "coordinate system of the input image 202"). The correct data 113 may include the coordinate value of the point 301 as the correct target position.

[0024] The correct target position may be specified by the correct position of the reference object 203 and a vector extending from the reference object 203 to the target position. In this case, the correct data 113 may include the correct position of the reference object 203 and a vector extending from the reference object 203 to the target position. The correct position of the reference object 203 is referred to as the correct reference position. The correct reference position may be specified by a region 302. The region 302 may be a rectangle having an outer edge circumscribing the reference object 203. The region 302 may be represented by a center, a width, and a height. The center of the region 302 may be represented by each coordinate value in the coordinate system of the input image 202. Alternatively, the region 302 may be represented by the coordinate values ​​of the upper left corner and the coordinate values ​​of the lower right corner. The region 302 may be a shape other than a rectangle, for example, a circle. The shape of the region 302 may vary depending on the shape of the reference object 203. The vector extending from the reference object 203 to the target position may be a vector extending from the center of the region 302 to the point 301. This vector may be represented by each coordinate value in the coordinate system of the input image 202.

[0025] Referring to FIG. 4, a model 400 learned by machine learning by the computer 100 will be described. The model 400 generates output data for identifying a target location specified by the input text 201 based on the input data 112. In the following description, the target location identified by the output data of the model 400 will be referred to as an estimated target location. The output data may include coordinate values ​​of the estimated target location. Alternatively, the output data may include coordinate values ​​of the position of the reference object 203 and a vector extending from the reference object 203 to the target location. The model 400 may have any structure that affects the output data of the model by processing both the input text 201 and the input image 202 with the model's parameters. The model 400 in FIG. 4 is an example of such a model. The model's parameters include at least one of weights and biases.

[0026] In Figure 4, the input text 201 is txt and the input image 202 is represented as x img and the output data of model 400 is expressed as y T and the correct data 113 is expressed as g T It is expressed as x txt is text data such as "Park in front of the black car on the right." img is the image data. img may be color image data or monochrome image data. img may be expressed as a three-dimensional array of data of H (height) × W (width) × C (channel). T g may be the coordinate value of the estimated target position, and may be expressed as a two-dimensional vector, for example. T may be the coordinate values ​​of the correct target position, and may be expressed as a two-dimensional vector, for example.

[0027] The output data (y T ) is input to a loss function 405 during training of the model 400. The loss function 405 includes the correct answer data 113 (g T) is also input. The loss function 405 outputs a loss based on the error between the output data and the correct answer data 113.

[0028] The model 400 may include a feature extraction unit 401, a text extraction unit 402, a reference object encoding unit 403, and a target position estimation unit 404. The feature extraction unit 401, the text extraction unit 402, the reference object encoding unit 403, and the target position estimation unit 404 may each be a model that can be machine-learned separately. The feature extraction unit 401, the text extraction unit 402, the reference object encoding unit 403, and the target position estimation unit 404 may each be called a sub-model. Each sub-model may be pre-trained independently before training the model 400. In training the model 400, parameters of each pre-trained model may be updated or maintained.

[0029] The feature extraction unit 401 extracts x txt and x img Based on V and y L and z loc V is a plurality of feature quantities V1 to V2 each representing a reference object 203 included in the input image 202. K (K is an integer equal to or greater than 2, e.g., K=5). K have different resolutions. For example, V i (1≦i≦K) is H i (Height) x Width i (Width) x C i The size of the three-dimensional array data is V i For example, 0.5 x H j =H j+1 , 0.5×W j =W j+1 , 0.5×C j =C j+1 (In both cases, 1≦j≦K−1) may be satisfied.

[0030] y L represents the position of the reference object 203 included in the input image 202. For example, y Lmay be expressed as a four-dimensional vector (for example, the coordinate values ​​of the center of the region representing the position of the reference object 203, and the height and width of the region).

[0031] z loc represents the position of the reference object 203 included in the input image 202. For example, z loc is the y L The reliability of the estimation of and the aspect ratio of the region representing the position of the reference object 203 are expressed as y L It may be expressed as a four-dimensional vector added to

[0032] The feature extraction unit 401 may be configured with any multimodal model having a hierarchical structure and receiving an image and a language as input. The output of any layer of the feature extraction unit 401 may be any feature included in V (V i ) is output from the feature extraction unit 401. The feature extraction unit 401 may be trained in advance so as to output the position of the reference object 203 included in the supervised answer data 113. An example of a specific configuration of the feature extraction unit 401 will be described later.

[0033] The text extraction unit 402 txt Based on x L and x T Generate x L is a text representing the reference object 203 included in the input image 202. That is, the text extraction unit 402 extracts x txt Extract the text representing the reference object 203 contained in the input image 202 from x L is x txt It can also be a partial text of x. txt is "Park in front of the black car on the right", then x L could also be "the black car on the right". L is x txt It may be text other than the partial text of

[0034] x T is a text that represents a target position relative to the reference object 203 included in the input image 202. That is, the text extraction unit 402 extracts x txt, extract text representing the target position relative to the reference object 203 contained in the input image 202. T is x txt It can also be a partial text of x. txt is "Park in front of the black car on the right", then x T could also be "in front of the black car on the right." T is x txt It may be text other than the partial text of

[0035] The text extraction unit 402 may be configured with any language model. For example, the text extraction unit 402 may be a large-scale language model such as GPT4. The text extraction unit 402 inputs a prompt such as "Please extract information representing an object contained in the text 'Park in front of the black car on the right'" into the language model, and L You may also obtain x T The same is true for .

[0036] The reference object coding unit 403 img and x L and, y L and based on z L Generate z L is a feature representing the reference object 203 included in the input image 202. For example, z L may be represented by a 1024-dimensional vector. The reference object encoding unit 403 may be trained in advance so that an output obtained by inputting the feature output from the reference object encoding unit 403 to the output layer represents the type and position of the reference object 203 included in the supervised answer data 113.

[0037] The reference object coding unit 403 includes a pre-processing unit and a multi-modal unit. img Of which y L The partial image is a portion of the input image 202 that includes the reference object 203. Then, the multimodal unit extracts the partial image extracted by the preprocessing unit and the region indicated by x L and based on z LThe multimodal unit may be configured with CLIP (Non-Patent Document 2), for example. As described above, the multimodal unit generates a partial image in which the reference object 203 is extracted from the input image 202 and the input text (x txt ) from which information representing the reference object 203 is extracted (x L ) improves the estimation accuracy of the multimodal part.

[0038] In the model 400 of Fig. 4, the reference object coding unit 403 may not include a pre-processing unit. In this case, the multi-modal unit may include img and x L and based on z L In this case, the feature extraction unit 401 may generate y L Instead of the example of FIG. 4, the reference object encoding unit 403 may generate x L instead of x txt In this case, the text extractor 402 may use x L does not need to be generated.

[0039] The target position estimation unit 404 calculates V and z loc and x T and z L Based on and, y T As mentioned above, x T is the input text 201(x txt ) is based on the T is x txt The target position estimation unit 404 generates V1 to V K An example of a specific configuration of the feature extraction unit 401 will be described later.

[0040] An example configuration of the feature extraction unit 401 will be described with reference to FIG. 5. The feature extraction unit 401 may have a configuration similar to that of FIBER, but may differ in that it further outputs V. The feature extraction unit 401 includes an image input layer 510, a text input layer 520, an image coding layer 530, a text coding layer 540, and an output layer 550. The image input layer 510 converts the input image 202 into a format that is input to the image coding layer 530. For example, the image input layer 510 converts the input image 202 into a plurality of vectors. For example, the image input layer 510 may divide the input image 202 into a plurality of patch images and rearrange the pixel values ​​of each patch image into a one-dimensional vector.

[0041] The image coding layer 530 encodes the input image 202 (specifically, the input image 202 expressed as a plurality of vectors) input from the image input layer 510. The specific configuration of the image coding layer 530 will be described later. The output layer 550 encodes z loc As will be described later, the image coding layer 530 outputs a matrix in which multiple row vectors are combined. The output layer 550 multiplies this output matrix by a weight matrix from the right to generate z loc Furthermore, the output layer 550 may calculate z loc Some components of y L Output as

[0042] A specific configuration of the image coding layer 530 will be described. The image coding layer 530 may include one or more independent coding layers 560 (two in the example of FIG. 5) and one or more collaborative coding layers 570 (two in the example of FIG. 5). When the image coding layer 530 includes multiple independent coding layers 560, these may be connected in series. When the image coding layer 530 includes multiple collaborative coding layers 570, these may be connected in series. One or more independent coding layers 560 may be arranged together in the first half of the image coding layer 530, and one or more collaborative coding layers 570 may be arranged together in the second half of the image coding layer 530. Alternatively, the independent coding layers 560 and the collaborative coding layers 570 may be arranged interleaved.

[0043] The independent coding layer 560 included in the image coding layer 530 encodes each of the multiple vectors input from the previous layer in the image coding layer 530, without using the feature determined by the text coding layer 540 as input. The independent coding layer 560 may include a self-attention layer 561 and a fully connected layer 562.

[0044] The multiple vectors input to the independent coding layer 560 are converted into multiple different vectors by the self-attention layer 561. The multiple vectors output from the self-attention layer 561 are converted into multiple different vectors by the fully connected layer 562. The multiple vectors output from the fully connected layer 562 are output from the independent coding layer 560. Each of the multiple output vectors of the self-attention layer 561 represents the relationship of each input vector to other input vectors in the multiple input vectors of the self-attention layer 561.

[0045] The fully connected layer 562 outputs multiple different vectors by combining all of the multiple input vectors. For example, the fully connected layer 562 multiplies the matrix Y output from the self-attention layer 561 by a weight matrix from the right and adds a bias vector to each row of the resulting matrix. The weight matrix and bias vector are parameters determined by machine learning. The fully connected layer 562 then outputs a matrix obtained by applying an activation function to each element of the matrix calculated in this way. The weight matrix of the fully connected layer 562 has a size such that the matrix output from the fully connected layer 562 (i.e., the matrix output from the independent coding layer 560) is the same size as the input matrix of the next independent coding layer 560.

[0046] The joint coding layer 570 included in the image coding layer 530 uses the features determined by the text coding layer 540 as additional inputs to encode each of the multiple vectors input from the previous layer in the image coding layer 530. The joint coding layer 570 may further include a cross-attention layer 571 in addition to the self-attention layer 561 and the fully connected layer 562 described above.

[0047] The multiple vectors input to the joint coding layer 570 are converted into multiple different vectors by the self-attention layer 561. Some of the features determined by the self-attention layer 561 are input to the cross-attention layer 571. Some of the features determined by the joint coding layer 570 (specifically, the self-attention layer 561) included in the text coding layer 540 are also input to the cross-attention layer 571. The cross-attention layer 571 generates and outputs multiple vectors based on these inputs.

[0048] The multiple vectors output from the self-attention layer 561 and the multiple vectors output from the cross-attention layer 571 are added together and input to the fully connected layer 562. The fully connected layer 562 converts the input multiple vectors into multiple different vectors. The multiple vectors output from the fully connected layer 562 are output from the joint coding layer 570.

[0049] Each of the multiple output vectors of the cross-attention layer 571 represents the relationship of multiple output vectors from the self-attention layer 561 included in the image coding layer 530 with respect to each of the multiple output vectors from the self-attention layer 561 included in the text coding layer 540.

[0050] The output of one or more independent coding layers 560 and one or more collaborative coding layers 570 included in the image coding layer 530 is V i 5, the output from the most upstream collaborative coding layer 570 (i.e., closest to the image input layer 510) is output as V1, and the output from the second most upstream collaborative coding layer 570 is output as V2. V1 and V2 have different resolutions. For example, V1 and V2 have different data sizes. In the example of FIG. 5, V is composed of two features, but the number of features included in V is not limited to this.

[0051] 6, a configuration example of the target position estimation unit 404 will be described. The target position estimation unit 404 may include an encoding unit 601, a conversion unit 602, an integration unit 603, and an output unit 604. Each of these components may include parameters determined by machine learning.

[0052] The encoding unit 601 T Based on L T Specifically, the encoding unit 601 generates x T By encoding L T Generate L T is the text data (x T ) is a feature that represents L T may be represented by two-dimensional array data of D (dimension of feature) × T (maximum token length). The encoding unit 601 may be configured with any language model, for example, BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa (Robustly Optimized BERT Pretraining Approach).

[0053] The conversion unit 602 converts V and L T F is generated based on the above. F is a set of intermediate features F1 to V, each of which represents a target position. K (K is an integer equal to or greater than 2, for example, K=5). The conversion unit 602 converts L T Using V i F i Convert to F i is V i In this case, the plurality of intermediate features F1 to F2 may have the same resolution (for example, data size) as the plurality of intermediate features F1 to F2. K have different resolutions. For example, F i (1≦i≦K) is H i (Height) x Width i (Width) x C i The conversion unit 602 converts V i independently (i.e., Vj (without using j≠i)F i may be converted to

[0054] The conversion unit 602 may be configured by, for example, PWAM (Non-Patent Document 3). For example, the conversion unit 602 calculates V using the following equation: i F i may be converted to

[0055] Q iq = flatten(w iq (V i )), L ik = w ik (L T ), L iv = w iv (L T ), G′ i = softmax(C i -1 / 2 t Q iq L ik ) t L iv , G i = w iw (unflatten( t G′ i )), V im = w im (V i ), F i = w io (V im *G i ).

[0056] Here, "w iq "," "w ik "," "w iv " and "w iw " represents a 1x1 convolution operation. im " and "w io " each represents a projection function, which may be an operation input to an activation function (e.g., ReLU) after a 1x1 convolution operation. t " represents the transposition operation. "Ci " represents the number of channels. "flatten()" represents a function that flattens a multidimensional array into a one-dimensional array. "unflatten()" represents the inverse function of flatten(). "softmax()" represents the softmax function. "*" represents element-wise multiplication. In the above formulas, variables with "i" in the index are used in F i In the above equation, the conversion unit 602 may include different parameters for V i at the pixel level T Combine with.

[0057] The integration unit 603 integrates each element of F, that is, F1 to F K By integrating z T For example, the integration unit 603 may perform generalized average pooling on each element of F and then concatenate them. For example, z T is (ΣC i It is represented by a (i=1,…,K)-dimensional vector.

[0058] The output unit 604 is T and z loc and z L Based on and, y T The output unit 604 may be configured with, for example, a multi-layer perceptron. For example, T and z loc and z L and may be input to the multi-layer perceptron as separate channels.

[0059] According to the above-described model 400, feature quantities V1 to V2 having multiple resolutions representing the reference object 203 are K Therefore, even when the target position is specified by referring to the reference object 203, which may have various sizes, the target position can be estimated with high accuracy.

[0060] In the above model 400, the target position estimation unit 404 calculates z loc It does not have to be based on z loc instead of y LIn this case, the feature extraction unit 401 may be based on z loc In the above model 400, the target position estimation unit 404 does not need to generate z L In this case, the reference object coding unit 403 may be omitted. In the above model 400, the target position estimation unit 404 is based on x T instead of x txt In this case, the text extractor 402 may be based on x T does not need to be generated.

[0061] Next, the loss function 405 used in some embodiments will be described with reference to Fig. 7. As described in Fig. 3, the supervised data 113 may include a point 301 representing a supervised target position in the input image 202 and a region 302 representing a supervised reference position of the reference object 203 in the input image 202. Furthermore, the supervised data 113 may include a penalty region 705 in the input image 202. The penalty region 705 may be a set of positions that should not be estimated as target positions.

[0062] In FIG. 7, the target position identified by the output data of model 400, i.e., the estimated target position, is represented by point 701. The center of region 302 is represented by point 702. Point 702 is based on the correct reference position of reference object 203. The vector extending from point 702 to point 301 is represented as correct vector 703. Correct vector 703 is a vector whose starting point is a point based on the correct reference position of reference object 203 and whose ending point is the correct target position. The vector extending from point 702 to point 701 is represented as estimated vector 704. Estimated vector 704 is a vector whose starting point is a point based on the correct reference position of reference object 203 and whose ending point is the estimated target position.

[0063] The loss function 405 may calculate the loss L using the following formula:

[0064] L = λ d L d +λ a L a +λ n L n+λ p L p .

[0065] Here, "λ d ", "λ a ", "λ n " and "λ p " are positive constants and may be determined as hyperparameters. d "," "L a "," "L n " and "L p " are losses determined based on the output data from the model 400 and the correct answer data 113. Each loss will be explained in detail below.

[0066] L d is the loss based on the distance between the correct target position (point 301) and the estimated target position (point 701). For example, L d may be the distance between point 301 and point 701, or may be the result of applying some function to this distance. The distance between point 301 and point 701 may be the L2 norm, the smooth L1 norm, or some other distance.

[0067] L a is a loss based on the angle between the correct vector 703 and the estimated vector 704. For example, L a may be the angle between the correct vector 703 and the estimated vector 704, or may be the result of applying some function to this angle. The angle between the correct vector 703 and the estimated vector 704 may be calculated by applying the inverse cosine function to the dot product of the unit vectors of each vector.

[0068] L n is a loss based on the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704. For example, L n may be the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704, or may be the result of applying some function to this difference. The magnitude of each vector may be, for example, the L2 norm.

[0069] L p is the loss based on whether the estimated target position is included in the penalty region 705. When the estimated target position is included in the penalty region 705, L p is the L when the estimated target position is not included in the penalty region 705 p The penalty area 705 may include the reference object 203. For example, the area 302 representing the correct position of the reference object 203 may be included as part of the penalty area 705. p may be calculated.

[0070] As described above, the loss function 405 is a loss (L d ), but also a loss (L a and L n ) is also included. This allows the model 400 to be trained so that the target position can be estimated with high accuracy. Specifically, the loss (L a and L n ) is based on the difference between the correct vector 703 and the estimated vector 704. As described above, the difference between the correct vector 703 and the estimated vector 704 may include the angle between the correct vector 703 and the estimated vector 704, or may include the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704, or may include both. In addition, the loss function 405 is a loss (L p This makes it possible to prevent an inappropriate position from being estimated as the target position.

[0071] In the loss function 405, the loss L is expressed as "L d ","L a ","L n " and "L p " is given as the sum of four terms of constant multiples of "L d ","L a ","L n " and "L p " may include only some terms. For example, L=λa L a +λ n L n may be.

[0072] In the loss function 405 described above, the point 702 based on the correct position of the reference object 203 is used as the starting point of the correct vector 703 and the estimated vector 704. Alternatively, a point independent of the correct position of the reference object 203, for example, the center of the input image 202, may be used as the starting point of the correct vector 703 and the estimated vector 704.

[0073] An example of a learning method for learning the model 400 will be described with reference to Fig. 8. Each step of the method in Fig. 8 may be processed by, for example, the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method in Fig. 8 may be executed by a dedicated circuit such as an application specific integrated circuit (ASIC). At the start of Fig. 8, the parameters of the model 400 may be values ​​set randomly or may be values ​​determined by pre-learning.

[0074] In S801, the computer 100 acquires one piece of training data 111. The training data 111 may be read out from the database 110 at this point, or may be stored in advance in the secondary storage device 106. Instead of using each piece of training data 111 one by one, multiple pieces of training data 111 may be used together as a batch.

[0075] In S802, the computer 100 generates output data by inputting the input data 112 included in the training data 111 acquired in S801 into the model 400. As described above, the output data is data for identifying a target position (e.g., point 301).

[0076] In S803, the computer 100 updates the parameters of the model 400 so as to reduce the loss obtained by inputting the output data generated in S802 and the supervised data 113 included in the training data 111 acquired in S801 into the loss function 405. The parameter update may be performed using an existing method such as Adam. For example, the loss function 405 receives the estimated target position included in the output data and the supervised target position included in the supervised data 113.

[0077] In S804, the computer 100 determines whether a condition for terminating the iteration of parameter updates (hereinafter, the termination condition) is satisfied. If it is determined that the termination condition is satisfied ("YES" in S804), the computer terminates the process. Otherwise ("NO" in S804), the computer transitions the process to S801. The termination condition may be that the parameters have been updated a predetermined number of times (i.e., S804 has been executed). After executing the process of FIG. 8, the computer 100 may store the trained model 400 in the secondary storage device 106 for future processing or may transmit it to another device (e.g., the database 110).

[0078] Next, with reference to FIG. 9 , an example of an estimation method for estimating a target position using the model 400 will be described. The estimation method of FIG. 9 may be executed by, for example, the computer 100. Therefore, the computer 100 may be referred to as an estimation device. The computer 100 that executes the estimation method of FIG. 9 may be different from the computer 100 that executes the learning method of FIG. 8. Each step of the method of FIG. 9 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method of FIG. 9 may be executed by a dedicated circuit such as an ASIC. It is assumed that a trained model 400 is available to the computer 100 at the start of FIG. 9 . For example, the trained model 400 may be stored in the secondary storage device 106 of the computer 100.

[0079] In S901, the computer 100 acquires input data to be input to the model 400. The input data may include a pair of an input image 202 including a reference object 203 and input text 201 that relatively specifies a target position by referring to the reference object 203. A vehicle according to some embodiments acquires voice input from an occupant through a microphone and converts the voice input into the input text 201. In response to acquiring the voice input, the vehicle acquires the input image 202 by capturing a scene in front of the vehicle.

[0080] In S902, the computer 100 generates output data by inputting the input image acquired in S901 into the model 400. As described above, the output data of the model 400 includes information for identifying the target position. The computer 100 identifies the target position using the output data of the model 400.

[0081] In S903, the computer 100 performs processing using the target position identified in S902. For example, the vehicle executes processing specified by voice input with respect to the target position. For example, if an instruction to "park in front of the black car on the right" is input by voice, the vehicle identifies the position in front of the black car on the right as the target position and controls the vehicle's driving so as to stop at this target position.

[0082] In the above description of FIG. 9 , the model 400 is used by a vehicle. Alternatively, the model 400 may be used by another device. For example, the model 400 can be used for operating a robot in addition to driving a vehicle. In this case, an image captured by a camera provided on a robot instead of a vehicle may be input to the model 400, and the target position included in the user's instruction may be identified. For example, when an instruction such as "Place the cup in front of the black box on the right" is input by voice, the robot identifies the position in front of the black box as the target position, just as in the case of a vehicle. Thereafter, the robot controls the robot's actuator to place the object (cup) it is holding at the target position.

[0083] <Summary of the embodiment> [Item 1] A learning device (100) that performs machine learning, An acquisition means for acquiring teacher data (111) including input data (112) and correct answer data (113), the input data including an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object; a generating means for generating output data for identifying the target position by inputting the input data into a model (400); an update means for updating the parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function (405) is reduced; The model is a first sub-model (401) that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model (404) that generates the output data based on the plurality of features and the input text; a learning device in which each of the plurality of feature amounts is input to the second sub-model. According to this item, a model can be generated that accurately estimates a target position specified by referring to a reference object. [Item 2] the model further includes a third sub-model (402) for extracting from the input text text representing the target position relative to the reference object; 2. The learning device according to claim 1, wherein the second sub-model generates the output data based on each of the plurality of feature amounts and the text extracted by the third sub-model. According to this item, a model can be generated that estimates a target position specified with more accuracy by referring to a reference object. [Item 3] The model is a third sub-model (402) for extracting text representing the reference object from the input text; a fourth sub-model (403) that generates features representing the reference object based on the input image and the text extracted by the third sub-model; 3. The learning device according to item 1 or 2, wherein the second sub-model generates the output data further based on the feature generated by the fourth sub-model. According to this item, a model can be generated that estimates a target position specified with more accuracy by referring to a reference object. [Item 4] the first sub-model further generates data representing a position (302) of the reference object based on the input image and the input text; 4. The learning device according to claim 3, wherein the fourth sub-model generates the feature representing the reference object further based on the data generated by the first sub-model. According to this item, a model can be generated that estimates a target position specified with more accuracy by referring to a reference object. [Item 5] the first sub-model further generates data representing a position (302) of the reference object based on the input image and the input text; 5. The learning device according to any one of items 1 to 4, wherein the second sub-model generates the output data further based on the data generated by the first sub-model. According to this item, a model can be generated that estimates a target position specified with more accuracy by referring to a reference object. [Item 6] The model is a third sub-model (402) for extracting text representing the reference object from the input text; a fourth sub-model (403) that generates features representing the reference object based on the input image and the text extracted by the third sub-model; the first sub-model further generates data representing a position (302) of the reference object based on the input image and the input text; The second sub-model is generating a plurality of intermediate features by transforming each of the plurality of features based on the input text; 6. The learning device according to any one of items 1 to 5, wherein the output data is generated based on each of the plurality of intermediate features, the data generated by the first sub-model, and the features generated by the fourth sub-model. According to this item, a model can be generated that estimates a target position specified with more accuracy by referring to a reference object. [Item 7] 7. The learning device according to any one of items 1 to 6, wherein the input image includes an image captured by a camera (211) of a vehicle (210). According to this item, a model can be generated that accurately estimates a target position suitable for vehicle control. [Item 8] 8. The learning device according to any one of items 1 to 7, wherein the input text is expressed in a natural language. According to this item, a model can be generated that accurately estimates a target position specified in natural language. [Item 9] A program for causing a computer to function as each means of the learning device described in any one of items 1 to 8. According to this item, the above-mentioned effects can be obtained in the form of a program. [Item 10] An estimation device (100) for estimating a target position, an acquisition means for acquiring input data (112), the input data including an input image (202) including a reference object (203) and input text (201) specifying a target position relatively with reference to the reference object; generating means for generating output data for identifying the target position by inputting the input data into a model (400); The model is a first sub-model (401) that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model (404) that generates the output data based on the plurality of features and the input text; an estimation device, wherein each of the plurality of feature amounts is input to the second sub-model. According to this item, the target position specified by referring to the reference object can be estimated with high accuracy. [Item 11] A program for causing a computer to function as each of the learning and estimation means described in item 10. According to this item, the above-mentioned effects can be obtained in the form of a program. [Item 12] A learning method for machine learning, comprising: Acquiring (S801) training data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object; generating output data for identifying the target location by inputting the input data into a model (400) (S802); updating (S803) the parameters of the model so that a loss obtained by inputting the output data and the ground truth data into a loss function (405) is reduced; The model is a first sub-model (401) that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model (404) that generates the output data based on the plurality of features and the input text; a learning method in which each of the plurality of feature amounts is input to the second sub-model. According to this item, a model can be generated that accurately estimates a target position specified by referring to a reference object. [Item 13] 1. A method for estimating a target position, comprising: acquiring (S901) input data (112), the input data including an input image (202) including a reference object (203) and input text (201) specifying a target position relative to the reference object; and generating (S902) output data for identifying the target location by inputting the input data into a model (400); The model is a first sub-model (401) that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model (404) that generates the output data based on the plurality of features and the input text; an estimation method in which each of the plurality of feature amounts is input to the second sub-model. According to this item, the target position specified by referring to the reference object can be estimated with high accuracy.

[0084] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0085] 100 Computer, 201 Input text, 202 Input image, 400 Model

Claims

1. A learning device that performs machine learning, an acquisition means for acquiring teacher data including input data and correct answer data, the input data including an input image including a reference object and input text that relatively specifies a target position by referring to the reference object; a generating means for generating output data for identifying the target position by inputting the input data into a model; an update means for updating parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function is reduced; The model is a first sub-model that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model that generates the output data based on the plurality of features and the input text; a learning device in which each of the plurality of feature amounts is input to the second sub-model.

2. the model further includes a third sub-model for extracting from the input text text representing the target position relative to the reference object; The learning device according to claim 1 , wherein the second sub-model generates the output data based on each of the plurality of feature amounts and the text extracted by the third sub-model.

3. The model is a third sub-model for extracting text representing the reference object from the input text; a fourth sub-model that generates features representing the reference object based on the input image and the text extracted by the third sub-model; The learning device according to claim 1 , wherein the second sub-model generates the output data further based on the feature quantity generated by the fourth sub-model.

4. the first sub-model further generates data representing a position of the reference object based on the input image and the input text; The learning device according to claim 3 , wherein the fourth sub-model generates the feature quantity representing the reference object further based on the data generated by the first sub-model.

5. the first sub-model further generates data representing a position of the reference object based on the input image and the input text; The learning device according to claim 1 , wherein the second sub-model generates the output data further based on the data generated by the first sub-model.

6. The model is a third sub-model for extracting text representing the reference object from the input text; a fourth sub-model that generates features representing the reference object based on the input image and the text extracted by the third sub-model; the first sub-model further generates data representing a position of the reference object based on the input image and the input text; The second sub-model is generating a plurality of intermediate features by transforming each of the plurality of features based on the input text; 2. The learning device according to claim 1, wherein the output data is generated based on each of the plurality of intermediate features, the data generated by the first sub-model, and the features generated by the fourth sub-model.

7. The learning device according to claim 1 , wherein the input image includes an image captured by a camera of a vehicle.

8. The learning device according to claim 1 , wherein the input text is expressed in a natural language.

9. A program for causing a computer to function as each of the means of the learning device according to any one of claims 1 to 8.

10. An estimation device for estimating a target position, an acquisition means for acquiring input data, the input data including an input image including a reference object and input text specifying a target position relatively with reference to the reference object; generating means for generating output data for identifying the target position by inputting the input data into a model; The model is a first sub-model that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model that generates the output data based on the plurality of features and the input text; the second sub-model is input with each of the plurality of feature quantities.

11. A program for causing a computer to function as each means of the estimation device according to claim 10.

12. A learning method for machine learning, comprising: acquiring training data including input data and correct answer data, the input data including an input image including a reference object and input text that relatively specifies a target position by referring to the reference object; inputting the input data into a model to generate output data for identifying the target location; updating parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function is reduced; The model is a first sub-model that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model that generates the output data based on the plurality of features and the input text; a learning method in which each of the plurality of feature amounts is input to the second sub-model.

13. 1. A method for estimating a target position, comprising: acquiring input data, the input data including an input image including a reference object and input text specifying a target position relatively with reference to the reference object; and generating output data for identifying the target location by inputting the input data into a model; The model is a first sub-model that generates a plurality of feature quantities representing the reference object based on the input image and the input text, the plurality of feature quantities having different resolutions; a second sub-model that generates the output data based on the plurality of features and the input text; an estimation method in which each of the plurality of feature quantities is input to the second sub-model.

Images