Training device, training method, and program

Abstract

This training device that carries out machine learning comprises: an acquisition unit that acquires training data including input data and ground truth data, the input data including an input image containing a reference object and input text specifying a target position relative to a reference object, the ground truth data including a ground truth target position; an estimation unit that determines an estimated target position by inputting the input data into a model; and an update unit that updates parameters of the model so as to reduce a loss obtained as a result of inputting the estimated target position and the ground truth target position into a loss function. The loss function is based on the angle between an estimated vector extending from a starting point in the input image to the estimated target position and a ground truth vector extending from the starting point to the ground truth target position.

Description

Learning device, learning method, and program

[0001] The present invention relates to a learning device, a learning method, and a program.

[0002] Various technologies for controlling vehicle driving using models generated by machine learning have been proposed. Patent Literature 1 describes training a neural network using sensor data acquired by a vehicle. A technology has also been proposed for estimating a position in an image indicated by language using a multimodal model that takes images and language as input. Examples of multimodal models proposed include FIBER (Fusion-In-the-Backbone-based transformER) (Non-Patent Literature 1), CLIP (Contrastive Language-Image Pre-training) (Non-Patent Literature 2), and PWAN (Pixel-Word Attention Module) (Non-Patent Literature 3).

[0003] Special Publication No. 2022-513866

[0004] Zi-Yi Dou, et al., “Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone”, [online], November 18, 2020, arXiv, [Retrieved March 16, 2020], Internet <URL: https: / / arxiv.org / pdf / 2206.07643.pdf> Alec Radford, et al., “Learning Transferable Visual Models From Natural Language Supervision”, [online], February 26, 2021, arXiv, [Retrieved March 16, 2020], Internet <URL: https: / / arxiv.org / pdf / 2103.00020.pdf> LAVT: Language-Aware Vision Transformer for Referring Image Segmentation Zhao Yang, et al., “LAVT: Language-Aware Vision Transformer for Referring Image Segmentation”, [online], April 5, 2020, arXiv, [searched on March 16, 2020], Internet <URL: https: / / arxiv.org / pdf / 2112.02244.pdf>

[0005] A target position in an input image can be specified by referring to a reference object included in the input image. The target position can be located at various relative positions with respect to the reference object. One aspect of the present invention aims to provide a model capable of accurately estimating a target position specified by referring to the reference object.

[0006] According to some embodiments, there is provided a learning device that performs machine learning, comprising: an acquisition means for acquiring teacher data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object, and the correct answer data includes a correct target position; an estimation means for determining an estimated target position by inputting the input data into a model; and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the estimated target position and the correct answer target position into a loss function, wherein the loss function is based on the angle between an estimated vector extending from a starting point in the input image to the estimated target position and a correct answer vector extending from the starting point to the correct target position.

[0007] According to some embodiments, a model is provided that can accurately estimate a target position that is specified with reference to a reference position. Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar components are designated by the same reference numerals.

[0008] The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention. A block diagram illustrating an example hardware configuration of a computer according to some embodiments. A schematic diagram illustrating example input data according to some embodiments. A schematic diagram illustrating example supervised data according to some embodiments. A schematic diagram illustrating an example configuration of a model according to some embodiments. A schematic diagram illustrating an example configuration of a feature extraction unit according to some embodiments. A schematic diagram illustrating an example configuration of a target position estimation unit according to some embodiments. A schematic diagram illustrating an example loss function according to some embodiments. A flow diagram illustrating an example training method according to some embodiments. A flow diagram illustrating an example estimation method according to some embodiments.

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] Referring to FIG. 1 , an example hardware configuration of a computer 100 according to some embodiments will be described. As described in detail below, the computer 100 is used to train a model through machine learning. Therefore, the computer 100 may be referred to as a learning device. The computer 100 may be, for example, a server computer or a personal computer (e.g., a desktop or laptop computer). The computer 100 may also be a computer resource located in a cloud environment.

[0011] The computer 100 may include the hardware devices shown in Fig. 1. The processor 101 controls the overall operation of the computer 100. The processor 101 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or a combination of these. The processor 101 may be a single processor or a collection of multiple processors connected to each other so that they can communicate with each other.

[0012] The memory 102 stores programs and data used in the processing of the computer 100. The memory 102 may be configured, for example, by a combination of random access memory (RAM) and read-only memory (ROM).

[0013] The input device 103 is a device for obtaining instructions from a user of the computer 100. The input device 103 may be configured, for example, by a combination of one or more of a keyboard, buttons, a touchpad, and a microphone. The display device 104 is a device for visually presenting information to a user of the computer 100. The display device 104 may be, for example, a dot-matrix display such as a liquid crystal display. The computer 100 may have a device (e.g., a touch screen) in which the input device 103 and the display device 104 are integrated. The input device 103 and the display device 104 may be external to the computer. In this case, the computer 100 may have an interface for communicating with the external input device 103 and display device 104.

[0014] The communication device 105 is a device for communicating with devices external to the computer 100. When the computer 100 performs wired communication, the communication device 105 may be a network interface card (NIC) having a connector for connecting a cable. When the computer 100 performs wireless communication, the communication device 105 may be a wireless communication module including an antenna and a baseband processing circuit.

[0015] The secondary storage device 106 is a device for non-volatilely storing programs and data used in the processing of the computer 100. The secondary storage device 106 is configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0016] The computer 100 may be capable of communicating with an external database 110. The database 110 may store teacher data 111 used for machine learning by the computer 100. The computer 100 may acquire the teacher data 111 from the database 110. Alternatively or in addition, the teacher data 111 may be stored in the secondary storage device 106 of the computer 100. In machine learning, multiple different teacher data 111 are used. Two pieces of teacher data 111 are different in that the input data 112 included in these pieces of teacher data 111 are different (for example, at least one of an input text 201 and an input image 202, which will be described later, is different). A portion of the teacher data 111 may be used as verification data and test data.

[0017] The training data 111 includes input data 112 and correct answer data 113. The input data 112 may be data input to a model (e.g., model 400 in FIG. 4) to train the model. The correct answer data 113 may be data to be output by the model.

[0018] An example of the input data 112 will be described with reference to Figure 2. The input data 112 may include a pair of an input image 202 including a reference object 203 and input text 201 that relatively specifies a target position by referencing the reference object 203. The input text 201 may represent an instruction for operation of the vehicle 210 by an occupant of the vehicle 210.

[0019] The input image 202 may be any image including an object. The input image 202 may be an image captured by a camera 211 of the vehicle 210. For example, the input image 202 may be an image captured by a camera 211 attached to the vehicle so as to capture an image in front of the vehicle 210. Alternatively, the input image 202 may be an image captured by a camera attached to the vehicle so as to capture an image in another direction (e.g., rearward) of the vehicle 210. The camera 211 of the vehicle 210 may be a camera 211 attached to the vehicle 210 or a camera brought into the vehicle (e.g., a smartphone of a vehicle occupant). The input image 202 may be an image not related to the vehicle.

[0020] The reference object 203 may be any object included in the input image 202. In the example of Fig. 2, a vehicle is used as the reference object 203. Alternatively, the reference object 203 may be a traffic participant other than a vehicle, a road sign, a traffic light, a guardrail, an intersection, a crosswalk, a building, a signboard, etc.

[0021] The input text 201 may be expressed in natural language, for example, "Park in front of the black car on the right." In this example, "the black car on the right" in the input text 201 specifies a reference object 203, and "in front of" in the input text 201 specifies a target position relative to the reference object 203. Instead of being expressed in natural language, the input text 201 may be expressed in other formats. For example, the input text 201 may be selected from a plurality of candidates for combinations of a reference object and a positional relationship that are set in advance.

[0022] An example of the correct answer data 113 will be described with reference to Fig. 3. The correct answer data 113 may be data for representing the correct position of the target position specified by the input text 201. The correct answer data 113 may be set manually for the input data 112 or may be set by a computer. In the following description, the correct position of the target position will be referred to as the correct target position.

[0023] The correct target position may be represented as a point 301 in the input image 202. Alternatively, the correct target position may be represented as an area centered on the point 301. The point 301 may be represented by each coordinate value in a two-dimensional coordinate system set in the input image 202 (hereinafter simply referred to as the "coordinate system of the input image 202"). The correct data 113 may include the coordinate value of the point 301 as the correct target position.

[0024] The correct target position may be specified by the correct position of the reference object 203 and a vector extending from the reference object 203 to the target position. In this case, the correct data 113 may include the correct position of the reference object 203 and a vector extending from the reference object 203 to the target position. The correct position of the reference object 203 is referred to as the correct reference position. The correct reference position may be specified by a region 302. The region 302 may be a rectangle having an outer edge circumscribing the reference object 203. The region 302 may be represented by a center, a width, and a height. The center of the region 302 may be represented by each coordinate value in the coordinate system of the input image 202. Alternatively, the region 302 may be represented by the coordinate values ​​of the upper left corner and the coordinate values ​​of the lower right corner. The region 302 may be a shape other than a rectangle, for example, a circle. The shape of the region 302 may vary depending on the shape of the reference object 203. The vector extending from the reference object 203 to the target position may be a vector extending from the center of the region 302 to the point 301. This vector may be represented by each coordinate value in the coordinate system of the input image 202.

[0025] Referring to FIG. 4 , a model 400 machine-learned by the computer 100 will be described. The model 400 generates output data for identifying a target position specified by the input text 201 based on the input data 112. In the following description, the target position identified by the output data of the model 400 will be referred to as an estimated target position. The output data may include coordinate values ​​of the estimated target position. Alternatively, the output data may include coordinate values ​​of the position of the reference object 203 and a vector extending from the reference object 203 to the target position. The model 400 may have any structure that affects the output data of the model by processing both the input text 201 and the input image 202 using model parameters. The model 400 in FIG. 4 is an example of such a model. The model parameters include at least one of weights and biases.

[0026] In FIG. 4, the input text 201 is txt and the input image 202 is expressed as x img and the output data of the model 400 is expressed as y T and the correct answer data 113 is expressed as g T It is expressed as x txt is text data such as "Park in front of the black car on the right." img is the image data. img may be color image data or monochrome image data. img may be expressed as three-dimensional array data of H (height) × W (width) × C (channel). T may be the coordinate values ​​of the estimated target position, and may be expressed as a two-dimensional vector, for example. T may be the coordinate values ​​of the correct target position, and may be expressed as a two-dimensional vector, for example.

[0027] The output data (y T ) is input to a loss function 405 during training of the model 400. The loss function 405 includes the correct answer data 113 (g T ) is also input. The loss function 405 outputs a loss based on the error between the output data and the correct answer data 113.

[0028] The model 400 may include a feature extraction unit 401, a text extraction unit 402, a reference object encoding unit 403, and a target position estimation unit 404. The feature extraction unit 401, the text extraction unit 402, the reference object encoding unit 403, and the target position estimation unit 404 may each be a model that can be machine-learned separately. The feature extraction unit 401, the text extraction unit 402, the reference object encoding unit 403, and the target position estimation unit 404 may each be called a sub-model. Each sub-model may be pre-trained independently before training the model 400. In training the model 400, parameters of each pre-trained model may be updated or maintained.

[0029] The feature extraction unit 401 txt and x img Based on this, V and y L And z loc V is a plurality of feature quantities V to V each representing a reference object 203 included in the input image 202. K (K is an integer equal to or greater than 2. For example, K=5). K have different resolutions. For example, V i (1≦i≦K) is H i (Height) x W i (Width) x C i The size of the three-dimensional array data may be expressed as V i For example, 0.5×H j = H j+1 , 0.5×W j =W j+1 , 0.5×C j =C j+1 (In both cases, 1≦j≦K−1) may be satisfied.

[0030] y L represents the position of the reference object 203 included in the input image 202. For example, y L may be expressed as a four-dimensional vector (for example, the coordinate values ​​of the center of the region representing the position of the reference object 203, and the height and width of the region).

[0031] z locrepresents the position of the reference object 203 included in the input image 202. For example, z loc is the y L and the aspect ratio of the region representing the position of the reference object 203 are expressed as y L may be expressed as a four-dimensional vector added to

[0032] The feature extraction unit 401 may be configured with any multimodal model having a hierarchical structure and receiving an image and a language as input. The output of any layer of the feature extraction unit 401 may be any feature included in V (V i ) is output from the feature extraction unit 401. The feature extraction unit 401 may be trained in advance so as to output the position of the reference object 203 included in the supervised answer data 113. An example of a specific configuration of the feature extraction unit 401 will be described later.

[0033] The text extraction unit 402 txt Based on x L and x T Generate x L is a text representing the reference object 203 included in the input image 202. That is, the text extraction unit 402 extracts x txt , the text representing the reference object 203 contained in the input image 202 is extracted from x. L is x txt For example, x txt is "Park in front of the black car on the right", then x L could also be "the black car on the right". L is x txt It may be text other than the partial text of

[0034] x T is a text that represents a target position relative to the reference object 203 included in the input image 202. That is, the text extraction unit 402 extracts x txt , extract text representing the target position relative to the reference object 203 contained in the input image 202. T is x txt For example, x txt is "Park in front of the black car on the right", then xT could also be "in front of the black car on the right". T is x txt It may be text other than the partial text of

[0035] The text extraction unit 402 may be configured with any language model. For example, the text extraction unit 402 may be a large-scale language model such as GPT4. The text extraction unit 402 inputs a prompt such as "Please extract information representing an object contained in the text 'Park in front of the black car on the right'" into the language model, and L You may also obtain x T The same is true for .

[0036] The reference object coding unit 403 img and x L And, y L Based on this, z L Generate z L is a feature representing the reference object 203 included in the input image 202. For example, z L may be represented by a 1024-dimensional vector. The reference object encoding unit 403 may be trained in advance so that an output obtained by inputting the feature output from the reference object encoding unit 403 to the output layer represents the type and position of the reference object 203 included in the supervised answer data 113.

[0037] The reference object coding unit 403 includes a pre-processing unit and a multi-modal unit. img Of which y L The partial image is a portion of the input image 202 that includes the reference object 203. Then, the multimodal unit extracts the partial image extracted by the preprocessing unit and the region indicated by x L Based on this, z L The multimodal unit may be configured with CLIP (Non-Patent Document 2), for example. As described above, the partial image in which the reference object 203 is extracted from the input image 202 and the input text (x txt ) from which information representing the reference object 203 is extracted. L ) improves the estimation accuracy of the multimodal part.

[0038] In the model 400 of FIG. 4, the reference object coding unit 403 may not include a pre-processing unit. In this case, the multi-modal unit may img and x L Based on this, z L In this case, the feature extraction unit 401 may generate y L Instead of the example of FIG. 4, the reference object encoding unit 403 may generate x L instead of x txt In this case, the text extractor 402 may use x L does not need to be generated.

[0039] The target position estimation unit 404 calculates V and z loc and x T And z L Based on this, y T As mentioned above, x T is the input text 201 (x txt ) is based on the T is x txt The target position estimation unit 404 generates the target position V based on V K An example of a specific configuration of the feature extraction unit 401 will be described later.

[0040] An example configuration of the feature extraction unit 401 will be described with reference to FIG. 5 . The feature extraction unit 401 may have a configuration similar to that of FIBER, but may differ in that it further outputs V. The feature extraction unit 401 includes an image input layer 510, a text input layer 520, an image coding layer 530, a text coding layer 540, and an output layer 550. The image input layer 510 converts the input image 202 into a format that is input to the image coding layer 530. For example, the image input layer 510 may convert the input image 202 into a plurality of vectors. For example, the image input layer 510 may divide the input image 202 into a plurality of patch images and rearrange the pixel values ​​of each patch image into a one-dimensional vector.

[0041] The image coding layer 530 encodes the input image 202 (specifically, the input image 202 expressed as a plurality of vectors) input from the image input layer 510. The specific configuration of the image coding layer 530 will be described later. The output layer 550 encodes z loc As will be described later, the image coding layer 530 outputs a matrix in which multiple row vectors are combined. The output layer 550 multiplies this output matrix by a weight matrix from the right to generate z loc Furthermore, the output layer 550 may calculate z loc Some components of y L Output as

[0042] A specific configuration of the image coding layer 530 will be described. The image coding layer 530 may include one or more independent coding layers 560 (two in the example of FIG. 5 ) and one or more collaborative coding layers 570 (two in the example of FIG. 5 ). When the image coding layer 530 includes multiple independent coding layers 560, these may be connected in series. When the image coding layer 530 includes multiple collaborative coding layers 570, these may be connected in series. One or more independent coding layers 560 may be arranged together in the first half of the image coding layer 530, and one or more collaborative coding layers 570 may be arranged together in the second half of the image coding layer 530. Alternatively, the independent coding layers 560 and the collaborative coding layers 570 may be arranged interleaved.

[0043] The independent coding layer 560 included in the image coding layer 530 encodes each of the multiple vectors input from the previous layer in the image coding layer 530, without using the feature determined by the text coding layer 540 as input. The independent coding layer 560 may include a self-attention layer 561 and a fully connected layer 562.

[0044] The multiple vectors input to the independent coding layer 560 are converted into multiple different vectors by the self-attention layer 561. The multiple vectors output from the self-attention layer 561 are converted into multiple different vectors by the fully connected layer 562. The multiple vectors output from the fully connected layer 562 are output from the independent coding layer 560. Each of the multiple output vectors from the self-attention layer 561 represents the relationship of each input vector to other input vectors in the multiple input vectors of the self-attention layer 561.

[0045] The fully connected layer 562 outputs multiple different vectors by combining all of the multiple input vectors. For example, the fully connected layer 562 multiplies the matrix Y output from the self-attention layer 561 by a weight matrix from the right and adds a bias vector to each row of the resulting matrix. The weight matrix and bias vector are parameters determined by machine learning. The fully connected layer 562 then outputs a matrix obtained by applying an activation function to each element of the matrix calculated in this way. The weight matrix of the fully connected layer 562 has a size such that the matrix output from the fully connected layer 562 (i.e., the matrix output from the independent coding layer 560) is the same size as the input matrix of the next independent coding layer 560.

[0046] The joint coding layer 570 included in the image coding layer 530 uses the feature values ​​determined by the text coding layer 540 as additional inputs to encode each of the multiple vectors input from the previous layer in the image coding layer 530. In addition to the self-attention layer 561 and the fully connected layer 562 described above, the joint coding layer 570 may further include a cross-attention layer 571.

[0047] The multiple vectors input to the joint coding layer 570 are converted into multiple different vectors by the self-attention layer 561. Some of the features determined by the self-attention layer 561 are input to the cross-attention layer 571. Some of the features determined by the joint coding layer 570 (specifically, the self-attention layer 561) included in the text coding layer 540 are also input to the cross-attention layer 571. The cross-attention layer 571 generates and outputs multiple vectors based on these inputs.

[0048] The multiple vectors output from the self-attention layer 561 and the multiple vectors output from the cross-attention layer 571 are added together and input to the fully connected layer 562. The fully connected layer 562 converts the multiple input vectors into multiple different vectors. The multiple vectors output from the fully connected layer 562 are output from the joint coding layer 570.

[0049] Each of the multiple output vectors of the cross-attention layer 571 represents the relationship of multiple output vectors from the self-attention layer 561 included in the image coding layer 530 with respect to each of the multiple output vectors from the self-attention layer 561 included in the text coding layer 540.

[0050] The output of one or more independent coding layers 560 and one or more joint coding layers 570 included in the image coding layer 530 is V i 5, the output from the most upstream joint coding layer 570 (i.e., closest to the image input layer 510) is output as V1, and the output from the second most upstream joint coding layer 570 is output as V2. V1 and V2 have different resolutions. For example, V1 and V2 have different data sizes. In the example of FIG. 5, V is composed of two features, but the number of features included in V is not limited to this.

[0051] 6, a configuration example of the target position estimation unit 404 will be described. The target position estimation unit 404 may include an encoding unit 601, a conversion unit 602, an integration unit 603, and an output unit 604. Each of these components may include parameters determined by machine learning.

[0052] The encoding unit 601 T Based on L T Specifically, the encoding unit 601 generates x T By encoding L T Generate L T is the text data (x T ) is a feature value representing Tmay be represented by two-dimensional array data of D (dimension of feature) × T (maximum token length). The encoding unit 601 may be configured with any language model, such as BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa (Robustly Optimized BERT Pretraining Approach).

[0053] The conversion unit 602 converts V and L T F is generated based on a plurality of intermediate features F1 to V K (K is an integer equal to or greater than 2, for example, K=5). The conversion unit 602 converts L T Using V i F i Convert to F i is V i In this case, the plurality of intermediate features F1 to F2 may have the same resolution (for example, data size) as the plurality of intermediate features F1 to F2. K have different resolutions. For example, F i (1≦i≦K) is H i (Height) x W i (Width) x C i The conversion unit 602 converts V i independently (i.e., V j (without using j≠i) F i may be converted to

[0054] The conversion unit 602 may be configured by, for example, PWAM (Non-Patent Document 3). For example, the conversion unit 602 calculates V using the following equation: i F i may be converted to

[0055] Q iq = flatten(w iq (V i )), L ik = w ik (L T ), L iv = w iv (LT ), G′ i = softmax(C i -1 / 2 t Q iq L ik ) t L iv , G i = w iw (unflatten( t G′ i )), V im = w im (V i ), F i = w io (V im *G i ).

[0056] Here, "w iq "," "w ik "," "w iv " and "w iw " respectively represent 1x1 convolution operations. im " and "w io " each represents a projection function, which may be an operation input to an activation function (e.g., ReLU) after a 1x1 convolution operation. t " represents the transposition operation. "C i " represents the number of channels. "flatten()" represents a function that flattens a multidimensional array into a one-dimensional array. "unflatten()" represents the inverse function of flatten(). "softmax()" represents the softmax function. "*" represents element-wise multiplication. In the above formulas, variables with "i" in the index are used in F i In the above equation, the conversion unit 602 may include different parameters for V i At the pixel level, T Combine with.

[0057] The integration unit 603 integrates each element of F, that is, F1 to F K By integrating z T For example, the integration unit 603 may perform generalized average pooling on each element of F and then concatenate them. For example, z T is (ΣC i(i=1, . . . , K)-dimensional vector.

[0058] The output unit 604 T And z loc And z L Based on this, y T The output unit 604 may be configured with, for example, a multi-layer perceptron. T And z loc And z L and may be input to the multi-layer perceptron as separate channels.

[0059] According to the above-described model 400, the feature quantities V to V having multiple resolutions representing the reference object 203 are K Therefore, even when the target position is specified by referring to the reference object 203, which may have various sizes, the target position can be estimated with high accuracy.

[0060] In the above model 400, the target position estimation unit 404 calculates z loc It does not have to be based on z loc instead of y L In this case, the feature extraction unit 401 may be based on z loc In the above model 400, the target position estimation unit 404 does not need to generate z L In this case, the reference object coding unit 403 may be omitted. In the above model 400, the target position estimation unit 404 may be based on x T instead of x txt In this case, the text extractor 402 may use x T does not need to be generated.

[0061] Next, a loss function 405 used in some embodiments will be described with reference to Fig. 7. As described in Fig. 3, the supervised data 113 may include a point 301 representing a supervised target position in the input image 202 and a region 302 representing a supervised reference position of the reference object 203 in the input image 202. Furthermore, the supervised data 113 may include a penalty region 705 in the input image 202. The penalty region 705 may be a set of positions that should not be estimated as target positions.

[0062] 7, the target position identified by the output data of the model 400, i.e., the estimated target position, is represented by point 701. The center of the region 302 is represented by point 702. Point 702 is based on the correct reference position of the reference object 203. A vector extending from point 702 to point 301 is represented as a correct vector 703. The correct vector 703 is a vector whose starting point is a point based on the correct reference position of the reference object 203 and whose ending point is the correct target position. The vector extending from point 702 to point 701 is represented as an estimated vector 704. The estimated vector 704 is a vector whose starting point is a point based on the correct reference position of the reference object 203 and whose ending point is the estimated target position.

[0063] The loss function 405 may calculate the loss L using the following formula:

[0064] L = λ d L d +λ a L a +λ n L n +λ p L p .

[0065] Here, "λ d ", "λ a ", "λ n " and "λ p " are positive constants and may be determined as hyperparameters. d "," "L a "," "L n " and "L p " are losses determined based on the output data from the model 400 and the correct answer data 113. Each loss will be described in detail below.

[0066] L d is a loss based on the distance between the correct target position (point 301) and the estimated target position (point 701). For example, L d may be the distance between point 301 and point 701, or may be the result of applying some function to this distance. The distance between point 301 and point 701 may be the L norm, the smooth L norm, or some other distance.

[0067] L a is a loss based on the angle between the correct vector 703 and the estimated vector 704. For example, L a may be the angle between the correct vector 703 and the estimated vector 704, or may be the result of applying some function to this angle. The angle between the correct vector 703 and the estimated vector 704 may be calculated by applying the inverse cosine function to the dot product of the unit vectors of each vector.

[0068] L n is a loss based on the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704. For example, L n may be the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704, or may be the result of applying some function to this difference. The magnitude of each vector may be, for example, the L norm.

[0069] L p is the loss based on whether the estimated target position is included in the penalty region 705. When the estimated target position is included in the penalty region 705, L p is the L when the estimated target position is not included in the penalty region 705 p The penalty area 705 may include the reference object 203. For example, the area 302 representing the correct position of the reference object 203 may be included as part of the penalty area 705. p may be calculated.

[0070] As described above, the loss function 405 is a loss (L d ), but also a loss (L a and L n ) is also included. This allows the model 400 to be trained so that the target position can be estimated with high accuracy. Specifically, the loss (L a and L n) is based on the difference between the correct vector 703 and the estimated vector 704. As described above, the difference between the correct vector 703 and the estimated vector 704 may include the angle between the correct vector 703 and the estimated vector 704, or may include the difference between the magnitude of the correct vector 703 and the magnitude of the estimated vector 704, or may include both. In addition, the loss function 405 is a loss (L p This makes it possible to prevent an inappropriate position from being estimated as the target position.

[0071] In the loss function 405, the loss L is expressed as "L d "," "L a "," "L n " and "L p " Alternatively, the loss L can be expressed as "L d "," "L a "," "L n " and "L p For example, L=λ a L a +λ n L n may be.

[0072] In the loss function 405 described above, the point 702 based on the correct position of the reference object 203 is used as the starting point of the correct vector 703 and the estimated vector 704. Alternatively, a point independent of the correct position of the reference object 203, for example, the center of the input image 202, may be used as the starting point of the correct vector 703 and the estimated vector 704.

[0073] An example of a learning method for learning the model 400 will be described with reference to Fig. 8. Each step of the method in Fig. 8 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method in Fig. 8 may be executed by a dedicated circuit such as an application specific integrated circuit (ASIC). At the start of Fig. 8, the parameters of the model 400 may be set to random values ​​or may be values ​​determined by pre-learning.

[0074] In S801, the computer 100 acquires one piece of teacher data 111. The teacher data 111 may be read out from the database 110 at this point, or may be stored in advance in the secondary storage device 106. Instead of using each piece of teacher data 111 one by one, multiple pieces of teacher data 111 may be used together as a batch.

[0075] In S802, the computer 100 generates output data by inputting the input data 112 included in the teacher data 111 acquired in S801 into the model 400. As described above, the output data is data for identifying a target position (e.g., point 301).

[0076] In S803, the computer 100 updates the parameters of the model 400 so as to reduce the loss obtained by inputting the output data generated in S802 and the supervised data 113 included in the training data 111 acquired in S801 into the loss function 405. The parameter update may be performed using an existing method such as Adam. For example, the loss function 405 is input with the estimated target position included in the output data and the supervised target position included in the supervised data 113.

[0077] In S804, the computer 100 determines whether a condition for terminating the repeated parameter update (hereinafter, the termination condition) is satisfied. If the termination condition is satisfied (YES in S804), the computer terminates the process. Otherwise (NO in S804), the computer transitions the process to S801. The termination condition may be that the parameters have been updated a predetermined number of times (i.e., S804 has been executed). After executing the process of FIG. 8, the computer 100 may store the trained model 400 in the secondary storage device 106 for future processing or may transmit the trained model 400 to another device (e.g., the database 110).

[0078] Next, with reference to FIG. 9 , an example of an estimation method for estimating a target position using the model 400 will be described. The estimation method of FIG. 9 may be executed by, for example, the computer 100. Therefore, the computer 100 may be referred to as an estimation device. The computer 100 that executes the estimation method of FIG. 9 may be different from the computer 100 that executes the learning method of FIG. 8. Each step of the method of FIG. 9 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method of FIG. 9 may be executed by a dedicated circuit such as an ASIC. It is assumed that a trained model 400 is available to the computer 100 at the start of FIG. 9 . For example, the trained model 400 may be stored in the secondary storage device 106 of the computer 100.

[0079] In S901, the computer 100 acquires input data to be input to the model 400. The input data may include a pair of an input image 202 including a reference object 203 and input text 201 that relatively specifies a target position by referring to the reference object 203. A vehicle according to some embodiments acquires voice input from an occupant through a microphone and converts the voice input into the input text 201. In response to acquiring the voice input, the vehicle acquires the input image 202 by capturing a scene in front of the vehicle.

[0080] In S902, the computer 100 generates output data by inputting the input image acquired in S901 into the model 400. As described above, the output data of the model 400 includes information for identifying the target position. The computer 100 identifies the target position using the output data of the model 400.

[0081] In S903, the computer 100 performs processing using the target position identified in S902. For example, the vehicle executes processing specified by voice input with respect to the target position. For example, if an instruction to "park in front of the black car on the right" is input by voice, the vehicle identifies the position in front of the black car on the right as the target position and controls the vehicle's driving so as to stop at this target position.

[0082] In the above description of FIG. 9 , the model 400 is used by a vehicle. Alternatively, the model 400 may be used by another device. For example, the model 400 can be used for operating a robot in addition to driving a vehicle. In this case, an image captured by a camera provided on a robot instead of a vehicle may be input to the model 400 to identify the target position included in the user's instruction. For example, when an instruction such as "Place the cup in front of the black box on the right" is input by voice, the robot identifies the position in front of the black box as the target position, just as in the case of a vehicle. The robot then controls its actuators to place the object (cup) it is holding at the target position.

[0083] Summary of Embodiments [Item 1] A learning device (100) that performs machine learning, comprising: an acquisition means for acquiring teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object, and the correct answer data includes a correct target position (301); an estimation means for determining an estimated target position (701) by inputting the input data to a model (400); and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the estimated target position and the correct answer target position into a loss function (405), wherein the loss function is based on the difference between an estimated vector (704) extending from a starting point (702) in the input image to the estimated target position, and a correct answer vector (703) extending from the starting point to the correct answer target position. According to this item, it is possible to generate a model that accurately estimates a target position specified by referring to a reference object. [Item 2] The learning device according to item 1, wherein the correct answer data further includes a correct position (302) of the reference object, and the starting point is based on the correct position of the reference object. According to this item, it is possible to generate a model that more accurately estimates a target position specified by referring to a reference object. [Item 3] The learning device according to item 1 or 2, wherein the correct answer data further includes a penalty region (705) included in the input image, and the loss function makes the loss larger when the estimated target position is included in the penalty region than the loss when the estimated target position is not included in the penalty region. According to this item, it is possible to generate a model that more accurately estimates a target position specified by referring to a reference object. [Item 4] The learning device according to item 3, wherein the penalty region includes the reference object. According to this item, it is possible to generate a model that more accurately estimates a target position specified by referring to a reference object. [Item 5] The learning device according to any one of items 1 to 4, wherein the difference between the estimated vector and the correct vector includes an angle between the estimated vector and the correct vector.According to this item, it is possible to generate a model that estimates with greater accuracy a target position specified with reference to a reference object. [Item 6] The learning device according to any one of items 1 to 5, wherein the difference between the estimated vector and the correct vector includes the difference between the magnitude of the estimated vector and the magnitude of the correct vector. According to this item, it is possible to generate a model that estimates with greater accuracy a target position specified with reference to a reference object. [Item 7] The learning device according to any one of items 1 to 6, further based on the distance between the estimated target position and the correct target position. According to this item, it is possible to generate a model that estimates with greater accuracy a target position specified with reference to a reference object. [Item 8] The learning device according to any one of items 1 to 7, wherein the loss function includes a term obtained by multiplying the angle between the estimated vector and the correct vector by a constant. According to this item, it is possible to generate a model that estimates with greater accuracy a target position specified with reference to a reference object. [Item 9] A learning device (100) that performs machine learning, comprising: an acquisition means for acquiring teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object, and the correct answer data includes a correct target position (301) and a penalty region (705) included in the input image; an estimation means for determining an estimated target position (701) by inputting the input data into a model (400); and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the estimated target position and the correct answer target position into a loss function (405), wherein the penalty region includes the reference object, and the loss function makes the loss greater when the estimated target position is included in the penalty region than when the estimated target position is not included in the penalty region. According to this item, it is possible to generate a model that accurately estimates a target position specified by referring to a reference object. [Item 10] The learning device according to any one of items 1 to 9, wherein the input image includes an image captured by a camera (211) of a vehicle (210).According to this item, it is possible to generate a model that accurately estimates a target position suitable for vehicle control. [Item 11] The learning device according to any one of items 1 to 10, wherein the input text is expressed in natural language. According to this item, it is possible to generate a model that accurately estimates a target position specified in natural language. [Item 12] A program for causing a computer to function as each means of the learning device according to any one of items 1 to 11. According to this item, the above-mentioned effects can be obtained in the form of a program. [Item 13] A learning method for performing machine learning, comprising: acquiring (S801) teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object, and the correct answer data includes a correct target position (301); determining (S802) an estimated target position (701) by inputting the input data into a model (400); and updating (S803) parameters of the model so that a loss obtained by inputting the estimated target position and the correct answer target position into a loss function (405) is reduced, wherein the loss function is based on a difference between an estimated vector (704) extending from a starting point (702) in the input image to the estimated target position, and a correct answer vector (703) extending from the starting point to the correct answer target position. According to this item, a model can be generated that accurately estimates a target position specified by referring to a reference object.[Item 14] A learning method for performing machine learning, comprising: acquiring (S801) teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position by referring to the reference object, and the correct answer data includes a correct target position (301) and a penalty region (705) included in the input image; determining (S803) an estimated target position (701) by inputting the input data into a model (400); and updating (S803) parameters of the model so that a loss obtained by inputting the estimated target position and the correct answer target position into a loss function (405) is reduced, wherein the penalty region includes the reference object, and the loss function makes a loss when the estimated target position is included in the penalty region greater than a loss when the estimated target position is not included in the penalty region. According to this item, a model can be generated that accurately estimates a target position specified by referring to a reference object.

[0084] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0085] 100 Computer, 201 Input text, 202 Input image, 400 Model

Claims

1. A learning device that performs machine learning, comprising: an acquisition means for acquiring teacher data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object, and the correct answer data includes a correct target position; an estimation means for determining an estimated target position by inputting the input data into a model; and an update means for updating parameters of the model so that a loss obtained by inputting the estimated target position and the correct answer target position into a loss function is reduced, wherein the loss function is based on the difference between an estimated vector extending from a starting point in the input image to the estimated target position and a correct answer vector extending from the starting point to the correct answer target position.

2. The learning device according to claim 1, wherein the correct answer data further includes a correct answer position of the reference object, and the starting point is based on the correct answer position of the reference object.

3. A learning device as described in claim 1 or 2, wherein the correct answer data further includes a penalty area included in the input image, and the loss function makes the loss when the estimated target position is included in the penalty area greater than the loss when the estimated target position is not included in the penalty area.

4. The learning device according to claim 3, wherein the penalty region includes the reference object.

5. A learning device according to any one of claims 1 to 4, wherein the difference between the estimated vector and the correct vector includes an angle between the estimated vector and the correct vector.

6. A learning device according to any one of claims 1 to 5, wherein the difference between the estimated vector and the correct vector includes a difference between the magnitude of the estimated vector and the magnitude of the correct vector.

7. The learning device according to claim 1, further based on a distance between the estimated target position and the ground truth target position.

8. The learning device according to any one of claims 1 to 7, wherein the loss function includes a term obtained by multiplying the angle between the estimated vector and the correct vector by a constant.

9. A learning device for performing machine learning, comprising: an acquisition means for acquiring teacher data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referring to the reference object, and the correct answer data includes a correct answer target position and a penalty area included in the input image; an estimation means for determining an estimated target position by inputting the input data into a model; and an update means for updating parameters of the model so that a loss obtained by inputting the estimated target position and the correct answer target position into a loss function is reduced, wherein the penalty area includes the reference object, and the loss function makes the loss when the estimated target position is included in the penalty area greater than the loss when the estimated target position is not included in the penalty area.

10. A learning device according to any one of claims 1 to 9, wherein the input images include images taken by a vehicle camera.

11. A learning device according to any one of claims 1 to 10, wherein the input text is expressed in a natural language.

12. A program for causing a computer to function as each means of the learning device according to any one of claims 1 to 11.

13. A learning method for performing machine learning, comprising: acquiring training data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object, and the correct answer data includes a correct target position; determining an estimated target position by inputting the input data into a model; and updating parameters of the model so that a loss obtained by inputting the estimated target position and the correct answer target position into a loss function is reduced, wherein the loss function is based on the difference between an estimated vector extending from a starting point in the input image to the estimated target position and a correct answer vector extending from the starting point to the correct target position.

14. A learning method for performing machine learning, comprising: acquiring training data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object, and the correct answer data includes a correct target position and a penalty area included in the input image; determining an estimated target position by inputting the input data into a model; and updating parameters of the model so that a loss obtained by inputting the estimated target position and the correct target position into a loss function is reduced, wherein the penalty area includes the reference object, and the loss function makes the loss when the estimated target position is included in the penalty area greater than the loss when the estimated target position is not included in the penalty area.

Images