Learning device, learning method, and program

Abstract

This learning device, which performs machine learning, includes: an acquiring unit that acquires teacher data including input data and correct answer data, the input data including an input image including a reference object and input text that relatively specifies a target position by referring to the reference object; a generating unit that inputs the input data into a model to generate output data for specifying the target position, a reference position that is the position of the reference object, and a positional relationship of the target position with respect to the reference position; and an updating unit that updates a parameter of the model such that a loss obtained by inputting the output data and the correct answer data into a loss function decreases. The loss function is based on at least two of: a first error between the target position specified by the output data and the target position specified by the correct answer data; a second error between the reference position specified by the output data and the reference position specified by the correct answer data; and a third error between the positional relation specified by the output data and the positional relation specified by the correct answer data.

Description

Learning device, learning method, and program

[0001] The present invention relates to a learning device, a learning method, and a program.

[0002] Various technologies have been proposed for controlling vehicle driving using models generated by machine learning. Patent Literature 1 describes training a neural network using sensor data acquired by a vehicle. Non-Patent Literature 1 describes a model that uses images and language as inputs.

[0003] Special Publication No. 2022-513866

[0004] Zi-Yi Dou, et al., “Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone”, [online], November 18, 2020, arXiv, [searched on May 24, 2020], Internet <URL: https: / / arxiv.org / pdf / 2206.07643.pdf>

[0005] It may be useful to train a model for inferring the target location using an image and text specifying the target location by referencing an object contained in the image as input data. One aspect of the present invention aims to generate a model capable of accurately inferring the target location.

[0006] According to some embodiments, there is provided a learning device that performs machine learning, comprising: an acquisition means for acquiring teacher data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referencing the reference object; a generation means for generating output data by inputting the input data into a model to identify the target position, a reference position that is the position of the reference object, and a positional relationship of the target position with respect to the reference position; and an update means for updating parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function is reduced, wherein the loss function is based on at least two of: a first error between the target position identified by the output data and the target position identified by the correct answer data; a second error between the reference position identified by the output data and the reference position identified by the correct answer data; and a third error between the positional relationship identified by the output data and the positional relationship identified by the correct answer data.

[0007] Some embodiments allow for the generation of a model that allows for accurate estimation of target positions.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. A block diagram illustrating an example hardware configuration of a computer according to some embodiments. A schematic diagram illustrating example input data according to some embodiments. A schematic diagram illustrating example ground truth data according to some embodiments. A schematic diagram illustrating an example configuration of a model according to some embodiments. A schematic diagram illustrating an example configuration of a self-attention layer according to some embodiments. A schematic diagram illustrating an example configuration of a cross-attention layer according to some embodiments. A flow diagram illustrating an example learning method according to some embodiments.

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0011] Referring to FIG. 1 , an example hardware configuration of a computer 100 according to some embodiments will be described. As described in detail below, the computer 100 is used to train a model through machine learning. Therefore, the computer 100 may be referred to as a learning device. The computer 100 may be, for example, a server computer or a personal computer (e.g., a desktop or laptop computer). The computer 100 may also be a computer resource located in a cloud environment.

[0012] The computer 100 may include the hardware devices shown in Fig. 1. The processor 101 controls the overall operation of the computer 100. The processor 101 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or a combination of these. The processor 101 may be a single processor or a collection of multiple processors connected to each other so that they can communicate with each other.

[0013] The memory 102 stores programs and data used in the processing of the computer 100. The memory 102 may be configured, for example, by a combination of random access memory (RAM) and read-only memory (ROM).

[0014] The input device 103 is a device for obtaining instructions from a user of the computer 100. The input device 103 may be configured, for example, by a combination of one or more of a keyboard, buttons, a touchpad, and a microphone. The display device 104 is a device for visually presenting information to a user of the computer 100. The display device 104 may be, for example, a dot-matrix display such as a liquid crystal display. The computer 100 may have a device (e.g., a touch screen) in which the input device 103 and the display device 104 are integrated. The input device 103 and the display device 104 may be external to the computer. In this case, the computer 100 may have an interface for communicating with the external input device 103 and display device 104.

[0015] The communication device 105 is a device for communicating with devices external to the computer 100. When the computer 100 performs wired communication, the communication device 105 may be a network interface card (NIC) having a connector for connecting a cable. When the computer 100 performs wireless communication, the communication device 105 may be a wireless communication module including an antenna and a baseband processing circuit.

[0016] The secondary storage device 106 is a device for non-volatilely storing programs and data used in the processing of the computer 100. The secondary storage device 106 is configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0017] The computer 100 may be capable of communicating with an external database 110. The database 110 may store teacher data 111 used for machine learning by the computer 100. The computer 100 may acquire the teacher data 111 from the database 110. Alternatively or in addition, the teacher data 111 may be stored in the secondary storage device 106 of the computer 100. In machine learning, multiple different teacher data 111 are used. Two pieces of teacher data 111 are different in that the input data 112 included in these pieces of teacher data 111 are different (for example, at least one of an input text 201 and an input image 202, which will be described later, is different). A portion of the teacher data 111 may be used as verification data and test data.

[0018] The training data 111 includes input data 112 and correct answer data 113. The input data 112 may be data input to a model (e.g., model 400 in FIG. 4) to train the model. The correct answer data 113 may be data to be output by the model.

[0019] An example of the input data 112 will be described with reference to Fig. 2. The input data 112 may include a pair of an input image 202 including a reference object 203 and input text 201 that relatively specifies a target position by referencing the reference object 203.

[0020] The input image 202 may be any image that includes an object. The input image 202 may be an image captured by a camera 211 of the vehicle 210. For example, the input image 202 may be an image captured by a camera 211 attached to the vehicle so as to capture an image in front of the vehicle 210. Alternatively, the input image 202 may be an image captured by a camera attached to the vehicle so as to capture an image in another direction (e.g., rearward) of the vehicle 210. The camera 211 of the vehicle 210 may be a camera 211 attached to the vehicle 210 or a camera brought into the vehicle (e.g., a smartphone of a vehicle occupant). The input image 202 may be an image that is not related to a vehicle.

[0021] The reference object 203 may be any object included in the input image 202. In the example of Fig. 2, a vehicle is used as the reference object 203. Alternatively, the reference object 203 may be a traffic participant other than a vehicle, a road sign, a traffic light, a guardrail, an intersection, a pedestrian crossing, etc.

[0022] The input text 201 may be expressed in natural language, for example, "Park in front of the black car on the right." In this example, "the black car on the right" in the input text 201 specifies a reference object 203, and "in front of" in the input text 201 specifies a target position relative to the reference object 203. Instead of being expressed in natural language, the input text 201 may be expressed in other formats. For example, the input text 201 may be selected from a plurality of candidates for combinations of a reference object and a positional relationship that are set in advance.

[0023] An example of the supervised answer data 113 will be described with reference to FIG. 3 . The supervised answer data 113 may be data for specifying a target position specified by the input text 201, a reference position which is the position of the reference object 203, and a positional relationship of the target position with respect to the reference object 203. The supervised answer data 113 may be set manually for the input data 112 or may be set by a computer. In the example of FIG. 3 , the supervised answer data 113 includes a point 301 in the input image 202 and an area 302 in the input image 202. The target position specified by the input text 201 is specified by the point 301. For example, the point 301 may be the target position. Alternatively, an area centered on the point 301 may be the target position. The point 301 may be represented by directional components in a two-dimensional coordinate system set in the input image 202 (hereinafter simply referred to as the "coordinate system of the input image 202").

[0024] The reference position, which is the position of the reference object 203, is specified by the region 302. For example, the region 302 may be the reference position. The region 302 may be a rectangle having an outer edge that circumscribes the reference object 203. The region 302 may be represented by a center, a width, and a height. The center of the region 302 may be represented by components in each direction in the coordinate system of the input image 202. Alternatively, the region 302 may be represented by the coordinates of the upper left corner and the coordinates of the lower right corner. The region 302 may be a shape other than a rectangle, and may be, for example, a circle. The shape of the region 302 may differ depending on the shape of the reference object 203.

[0025] The positional relationship of the target position with respect to the reference object 203 is specified by a point 301 and an area 302. For example, the positional relationship may be represented by a two-dimensional vector extending from the center of the area 302 to the point 301. This vector may be represented by components in each direction in the coordinate system of the input image 202.

[0026] In the above example, the correct answer data 113 includes a point 301 and an area 302. Alternatively, the correct answer data 113 may include an area 302 and a two-dimensional vector. In this case, the target position specified by the input text 201 may be specified by a point that is shifted from the center of the area 302 by the amount of the two-dimensional vector. Alternatively, the correct answer data 113 may include a point 301 and two two-dimensional vectors. The upper left corner and the upper right corner of the area 302 may be specified by points that are shifted from the point 301 by the amount of each of the two two-dimensional vectors.

[0027] In the above example, the supervised answer data 113 explicitly represents two of the three pieces of information: the target position specified by the input text 201, the reference position which is the position of the reference object 203, and the positional relationship of the target position with respect to the reference object 203, and the remaining piece of information is determined from the other two pieces of information. Alternatively, the supervised answer data 113 may explicitly represent these three pieces of information.

[0028] Referring to Fig. 4, a model 400 that is machine-learned by the computer 100 will be described. Based on the input data 112, the model 400 generates output data for identifying a target position specified by the input text 201, a reference position that is the position of the reference object 203, and the positional relationship of the target position with respect to the reference object 203. The model 400 may have any structure that affects the output data of the model by processing both the input text 201 and the input image 202 included in the input data 112 using the parameters of the model. The model 400 in Fig. 4 is an example of such a model.

[0029] The model 400 includes an image input layer 410, a text input layer 420, an image coding layer 430, a text coding layer 440, and an output layer 450. The image input layer 410 converts the input image 202 into a format that can be input to the image coding layer 430. For example, the image input layer 410 converts the input image 202 into a plurality of vectors. For example, the image input layer 410 may divide the input image 202 into a plurality of patch images and rearrange the pixel values ​​of each patch image into a one-dimensional vector.

[0030] The image encoding layer 430 encodes the input image 202 (specifically, the input image 202 expressed as multiple vectors) input from the image input layer 410. The specific configuration of the image encoding layer 430 will be described later. The output layer 450 generates output data for identifying a target position specified by the input text 201, a reference position that is the position of the reference object 203, and the positional relationship of the target position with respect to the reference object 203, based on the data encoded by the image encoding layer 430. The output data of the model 400 may have the same configuration as the above-described correct answer data 113. For example, the output data of the model 400 may be a point 301 and a region 302. As will be described later, the image encoding layer 430 outputs a matrix in which multiple row vectors are combined. The output layer 450 may calculate a one-dimensional column vector by multiplying this output matrix by a weighting matrix from the right. This weighting matrix is ​​one of the parameters determined by machine learning. The multiple components of the calculated column vector are the coordinate values ​​of the point 301 and the values ​​for specifying the area 302 .

[0031] A specific configuration of the image coding layer 430 will be described. The image coding layer 430 may include one or more independent coding layers 460 (two in the example of FIG. 4 ) and one or more collaborative coding layers 470 (two in the example of FIG. 4 ). When the image coding layer 430 includes multiple independent coding layers 460, these may be connected in series. When the image coding layer 430 includes multiple collaborative coding layers 470, these may be connected in series. One or more independent coding layers 460 may be arranged together in the first half of the image coding layer 430, and one or more collaborative coding layers 470 may be arranged together in the second half of the image coding layer 430. Alternatively, the independent coding layers 460 and the collaborative coding layers 470 may be arranged interleaved.

[0032] The independent coding layer 460 included in the image coding layer 430 encodes each of the multiple vectors input from the previous layer in the image coding layer 430, without using the feature determined by the text coding layer 440 as input. The independent coding layer 460 may include a self-attention layer 461 and a fully connected layer 462.

[0033] The multiple vectors input to the independent coding layer 460 are converted into multiple different vectors by the self-attention layer 461. The multiple vectors output from the self-attention layer 461 are converted into multiple different vectors by the fully connected layer 462. The multiple vectors output from the fully connected layer 462 are output from the independent coding layer 460.

[0034] Each of the multiple output vectors of the self-attention layer 461 represents the relationship of each input vector to other input vectors in the multiple input vectors of the self-attention layer 461. A specific configuration of the self-attention layer 461 will be described with reference to Figure 5. The self-attention layer 461 combines multiple input row vectors into one two-dimensional input matrix X. The self-attention layer 461 applies a weight matrix W to the input matrix X. Ｑ , weight matrix W Ｋ , weight matrix W Ｖ The query Q, key K, and value V are calculated by multiplying them from the right. Ｑ , weight matrix WＫ , and the weight matrix W Ｖ is a parameter determined by machine learning.

[0035] The self-attention layer 461 includes a score calculation unit 501. The score calculation unit 501 calculates a score S based on a query Q and a key K. Specifically, the score calculation unit 501 calculates an intermediate matrix by multiplying the query Q by the transposed matrix of the key K from the right and dividing each component by a predetermined value (e.g., the square root of the number of columns of the key K). Then, the score calculation unit 501 calculates a score S by applying a Softmax function to each row of the intermediate matrix. Then, the self-attention layer 461 calculates a matrix Y by multiplying the score S by a value V from the right. The self-attention layer 461 outputs the matrix Y calculated in this manner. The multiple rows of the matrix Y correspond to the multiple row vectors output from the self-attention layer 461.

[0036] The fully connected layer 462 outputs multiple different vectors by combining all of the multiple input vectors. For example, the fully connected layer 462 multiplies the matrix Y output from the self-attention layer 461 by a weight matrix from the right and adds a bias vector to each row of the resulting matrix. The weight matrix and bias vector are parameters determined by machine learning. The fully connected layer 462 then outputs a matrix obtained by applying an activation function to each element of the matrix calculated in this way. The weight matrix of the fully connected layer 462 has a size such that the matrix output from the fully connected layer 462 (i.e., the matrix output from the independent coding layer 460) is the same size as the input matrix of the next independent coding layer 460.

[0037] The joint coding layer 470 included in the image coding layer 430 uses the feature determined by the text coding layer 440 as an additional input to encode each of the multiple vectors input from the previous layer in the image coding layer 430. In addition to the self-attention layer 461 and the fully connected layer 462 described above, the joint coding layer 470 may further include a cross-attention layer 471.

[0038] The multiple vectors input to the joint coding layer 470 are converted into multiple different vectors by the self-attention layer 461. Some of the features determined by the self-attention layer 461 are input to the cross-attention layer 471. Some of the features determined by the joint coding layer 470 (specifically, the self-attention layer 461) included in the text coding layer 440 are also input to the cross-attention layer 471. The cross-attention layer 471 generates and outputs multiple vectors based on these inputs.

[0039] The multiple vectors output from the self-attention layer 461 and the multiple vectors output from the cross-attention layer 471 are added together and input to the fully connected layer 462. The fully connected layer 462 converts the multiple input vectors into multiple different vectors. The multiple vectors output from the fully connected layer 462 are output from the joint coding layer 470.

[0040] Each of the multiple output vectors of the cross-attention layer 471 represents the relationship of multiple output vectors from the self-attention layer 461 included in the image coding layer 430 with respect to each of the multiple output vectors from the self-attention layer 461 included in the text coding layer 440.

[0041] A specific configuration of the cross-attention layer 471 will be described with reference to Fig. 6. The cross-attention layer 471 receives a query Q from the self-attention layer 461a included in the image coding layer 430, and a key K and a value V from the self-attention layer 461b included in the text coding layer 440. The query Q is part of the features determined by the self-attention layer 461a. The key K and the value V are part of the features determined by the self-attention layer 461b.

[0042] The score calculation unit 501 calculates the score S based on the query Q and the key K in the same manner as described above. Then, the cross-attention layer 471 calculates the matrix Z by multiplying the score S by the value V from the right. The cross-attention layer 471 outputs the matrix Z calculated in this manner. The multiple rows of the matrix Z correspond to the multiple row vectors output from the cross-attention layer 471.

[0043] The text input layer 420 converts the input text 201 into a format that can be input to the text coding layer 440. For example, the text input layer 420 segments the input text 201 into multiple words and converts each word into a vector. An existing technique, such as word2vec, may be used to vectorize the words. The text coding layer 440 encodes the input text 201 (specifically, the input text 201 expressed as multiple vectors) input from the text input layer 420. The text coding layer 440 may have the same layer structure as the image coding layer 430. Alternatively, the text coding layer 440 may have a layer structure different from that of the image coding layer 430.

[0044] For example, the text coding layer 440 may include one or more independent coding layers 460 (two in the example of FIG. 4 ) and one or more joint coding layers 470 (two in the example of FIG. 4 ), similar to the image coding layer 430. The independent coding layers 460 included in the text coding layer 440 encode each of the multiple vectors input from the previous layer in the text coding layer 440 without using the feature values ​​determined by the image coding layer 430 as input. The joint coding layer 470 included in the text coding layer 440 encodes each of the multiple vectors input from the previous layer in the text coding layer 440, using the feature values ​​determined by the image coding layer 430 as additional input.

[0045] The output data output from the model 400 is input to a loss function 480 during training of the model 400. The supervised data 113 corresponding to the input data 112 is also input to the loss function 480. The loss function 480 outputs a loss based on the error between the output data and the supervised data 113.

[0046] An example of a learning method for learning the model 400 will be described with reference to Fig. 7. Each step of the method in Fig. 7 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method in Fig. 7 may be executed by a dedicated circuit such as an application specific integrated circuit (ASIC). At the start of Fig. 7, the parameters of the model 400 may be set to random values.

[0047] In S701, the computer 100 acquires one piece of teacher data 111. The teacher data 111 may be read out from the database 110 at this point, or may be stored in advance in the secondary storage device 106. Instead of using each piece of teacher data 111 one by one, multiple pieces of teacher data 111 may be used together as a batch.

[0048] In S702, the computer 100 generates output data by inputting the input data 112 included in the teacher data 111 acquired in S701 into the model 400. As described above, the output data is data for specifying a target position (e.g., point 301), a reference position (e.g., area 302) which is the position of the reference object 203, and the positional relationship of the target position with respect to the reference position.

[0049] In S703, the computer 100 updates the parameters of the model 400 so as to reduce the loss obtained by inputting the output data generated in S702 and the supervised answer data 113 included in the training data 111 acquired in S701 into the loss function 480. The parameter updating may be performed using an existing method such as Adam.

[0050] The loss function 480 used in some embodiments will be described in detail. The loss function 480 is based on at least two of: (1) the error between the target position identified by the output data of the model 400 and the target position identified by the supervised data 113 (hereinafter referred to as the target error); (2) the error between the reference position identified by the output data of the model 400 and the reference position identified by the supervised data 113 (hereinafter referred to as the reference error); and (3) the error between the positional relationship identified by the output data of the model 400 and the positional relationship identified by the supervised data 113 (hereinafter referred to as the relationship error). For example, the loss function 480 may be based on the target error and the reference error, or may be based on the target error and the relationship error, or may be based on the reference error and the relationship error. Furthermore, the loss function 480 may be based on all of the target error, the reference error, and the relationship error. In this way, by basing the loss function 480 on at least two of the target error, the reference error, and the relationship error, it is possible to generate a model 400 that can accurately estimate the target position.

[0051] The loss function 480 used in some other embodiments will be described in detail. The loss function 480 is based on (3) the error (the above-mentioned relationship error) between the positional relationship identified by the output data of the model 400 and the positional relationship identified by the ground truth data 113. The relationship error is an error in the positional relationship of the target position with respect to the reference position, and is therefore based on both the target position and the reference position. Therefore, even if the loss function 480 is based only on the relationship error, it is possible to generate a model 400 that can accurately infer the target position.

[0052] When the target position is represented by a point 301 as described above, the target error may be the difference between each coordinate value of the point 301 specified by the output data of the model 400 and each coordinate value of the point 301 specified by the supervised data 113. When the reference position is represented by an area 302 as described above, the reference error may be the difference between each coordinate value of the center, width, and height of the area 302 specified by the output data of the model 400 and each coordinate value of the center, width, and height of the area 302 specified by the supervised data 113. When the positional relationship between the target position and the reference position is represented by a two-dimensional vector as described above, the relationship error may be the difference between each component of the two-dimensional vector specified by the output data of the model 400 and each component of the two-dimensional vector specified by the supervised data 113.

[0053] The loss calculated by the loss function 480 may be a sum of at least two of a loss based on a target error, a loss based on a reference error, and a loss based on a relation error. The sum may be a weighted sum, and the coefficients of the weighted sum may be determined as hyperparameters.

[0054] The loss based on the target error may be, for example, an L1 loss of the target error. The loss based on the reference error may be, for example, a sum of an L1 loss of the reference error and a GIoU loss of the reference error. This sum may be a weighted sum, and the coefficient of the weighted sum may be determined as a hyperparameter. The loss based on the relation error may be, for example, an L1 loss of the relation error. The above-mentioned methods for determining the losses based on each error are examples, and losses may be determined by other methods.

[0055] In S704, the computer 100 determines whether a condition for terminating the parameter update iterations (hereinafter, the termination condition) is satisfied. If the termination condition is satisfied (YES in S704), the computer terminates the process. Otherwise (NO in S704), the computer transitions the process to S701. The termination condition may be that the parameters have been updated a predetermined number of times (i.e., S704 has been executed). After executing the process of FIG. 7, the computer 100 may store the trained model 400 in the secondary storage device 106 for future processing or may transmit the trained model 400 to another device (e.g., the database 110).

[0056] The parameters of the model 400 at the start of learning may be set randomly. Alternatively, the parameters of the model 400 at the start of learning may be parameters of the image encoding layer 430 determined by another machine learning process that uses the input image 202 as input data and the position of the reference object 203 as ground truth data. In other words, the parameters of the model 400 may be determined by fine tuning that uses parameters determined by another machine learning process.

[0057] Next, a method for using the model 400 will be described. A vehicle according to some embodiments acquires voice input from a passenger through a microphone and converts the voice input into text. In response to acquiring the voice input, the vehicle acquires images by capturing a scene in front of the vehicle. The vehicle generates output data by inputting the acquired text and image into the model 400, and identifies a target position using the output data. The vehicle then executes the processing specified by the voice input with respect to the target position. For example, when an instruction such as "park in front of the black car on the right" is given by voice input, the vehicle identifies the position in front of the black car on the right as the target position and controls the vehicle's driving so as to stop at this target position. In this way, by using the model 400, it is possible to simultaneously predict a target position and a reference object specified in the input text.

[0058] Summary of Embodiments [Item 1] A learning device (100) that performs machine learning, comprising: an acquisition means for acquiring teacher data (111) including input data (112) and supervised answer data (113), the input data including an input image (202) including a reference object (203), and input text (201) that relatively specifies a target position (301) by referencing the reference object; a generation means for inputting the input data into a model (400) to generate output data for specifying the target position, a reference position (302) that is the position of the reference object, and a positional relationship of the target position with respect to the reference position; and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the output data and the supervised answer data into a loss function (480), the loss function comprising: a first error between the target position specified by the output data and the target position specified by the supervised answer data; A learning device based on at least two of: a second error between the reference position identified by the output data and the reference position identified by the supervised data; and a third error between the positional relationship identified by the output data and the positional relationship identified by the supervised data. According to this item, a model capable of accurately inferring a target position can be generated. [Item 2] The learning device according to item 1, wherein the input image includes an image captured by a camera (211) of a vehicle (210). According to this item, a model capable of accurately inferring a target position used for vehicle control can be generated. [Item 3] The learning device according to item 1 or 2, wherein the input text is expressed in natural language. According to this item, a model capable of accurately inferring a target position specified in natural language can be generated. [Item 4] The learning device according to any one of items 1 to 3, wherein the loss function is based on at least the first error. According to this item, because it is based on the first error, a model capable of even more accurately inferring a target position can be generated. [Item 5] The learning device according to any one of items 1 to 4, wherein the loss function is based on all of the first error, the second error, and the third error.According to this item, a model capable of inferring a target position with greater accuracy can be generated. [Item 6] The learning device according to any one of Items 1 to 5, wherein the model includes: an image coding layer (430) for coding the input image; and a text coding layer (440) for coding the input text, wherein a portion of the feature quantities determined by the text coding layer is input to the image coding layer; and a portion of the feature quantities determined by the image coding layer is input to the text coding layer. According to this item, both the input image and the input text can be reflected in the output data. [Item 7] The learning device according to Item 6, wherein the image coding layer and the text coding layer have the same layer structure. According to this item, model implementation is easy. [Item 8] The learning device according to Item 6 or 7, wherein the machine learning of the model is a first machine learning, and the parameters of the model at the start of learning are parameters of the image coding layer determined by a second machine learning that uses the input image as input data and the position of the reference object as correct answer data. According to this item, the learning time of the model can be shortened. [Item 9] A learning device (100) that performs machine learning, comprising: an acquisition means for acquiring teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position (301) by referencing the reference object; a generation means for inputting the input data into a model (400) to generate output data for identifying the target position, a reference position (302) that is the position of the reference object, and a positional relationship of the target position with respect to the reference position; and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the output data and the correct answer data into a loss function (480), wherein the loss function is based on an error between the positional relationship identified by the output data and the positional relationship identified by the correct answer data. According to this item, a model capable of accurately inferring a target position can be generated.[Item 10] A program for causing a computer to function as each means of the learning device described in any one of items 1 to 9. According to this item, the above items can be provided in the form of a program. [Item 11] A learning method for performing machine learning, comprising: an acquisition step of acquiring teacher data (111) including input data (112) and correct answer data (113), wherein the input data includes an input image (202) including a reference object (203) and input text (201) that relatively specifies a target position (301) by referencing the reference object; a generation step of inputting the input data into a model (400) to generate output data for specifying the target position, a reference position (302) that is the position of the reference object, and a positional relationship of the target position with respect to the reference position; and an update step of updating parameters of the model so as to reduce a loss obtained by inputting the output data and the correct answer data into a loss function (480), wherein the loss function is determined based on: a first error between the target position specified by the output data and the target position specified by the correct answer data; a second error between the reference position specified by the output data and the reference position specified by the correct answer data; and a third error between the positional relationship specified by the output data and the positional relationship specified by the ground truth data, or the loss function is based on the third error. According to this item, it is possible to generate a model that can accurately infer a target position.

[0059] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0060] This application claims priority to U.S. Provisional Patent Application No. 18 / 213,980, filed June 26, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A learning device that performs machine learning, Acquisition means for acquiring training data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referring to the reference object, A generation means that generates output data for specifying the target position, the reference position which is the position of the reference object, and the positional relationship of the target position with respect to the reference position by inputting the aforementioned input data into the model, The model includes an update means for updating the model parameters such that the loss obtained by inputting the output data and the ground truth data into the loss function decreases. The aforementioned loss function is, The first error between the target position identified by the output data and the target position identified by the correct data, A second error between the reference position identified by the output data and the reference position identified by the correct data, A third error between the positional relationship identified by the output data and the positional relationship identified by the correct data, A learning device based on at least two of the following.

2. The learning device according to claim 1, wherein the input image includes an image captured by a vehicle camera.

3. The learning device according to claim 1, wherein the input text is expressed in natural language.

4. The learning device according to claim 1, wherein the loss function is based on at least the first error.

5. The learning device according to claim 1, wherein the loss function is based on all of the first error, the second error, and the third error.

6. The aforementioned model, An image encoding layer for encoding the aforementioned input image, A text encoding layer for encoding the input text, A portion of the features determined by the text encoding layer is input to the image encoding layer. The learning device according to claim 1, wherein a portion of the features determined by the image encoding layer is input to the text encoding layer.

7. The learning device according to claim 6, wherein the image coding layer and the text coding layer have the same layer structure.

8. The machine learning described above for the aforementioned model is a first machine learning method, The learning device according to claim 6, wherein the parameters of the model at the start of learning are the parameters of the image coding layer determined by a second machine learning method that uses the input image as input data and the position of the reference object as ground truth data.

9. A learning device that performs machine learning, Acquisition means for acquiring training data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referring to the reference object, A generation means that generates output data for specifying the target position, the reference position which is the position of the reference object, and the positional relationship of the target position with respect to the reference position by inputting the aforementioned input data into the model, The model includes an update means for updating the model parameters such that the loss obtained by inputting the output data and the ground truth data into the loss function decreases. A learning device in which the loss function is based on the error between the positional relationship identified by the output data and the positional relationship identified by the ground truth data.

10. A program for causing a computer to function as one of the means of a learning device described in any one of claims 1 to 9.

11. A learning method for performing machine learning, An acquisition step for acquiring training data including input data and correct answer data, wherein the input data includes an input image including a reference object and input text that relatively specifies a target position by referring to the reference object. A generation step of inputting the aforementioned input data into a model to generate output data for specifying the target position, the reference position which is the position of the reference object, and the positional relationship of the target position with respect to the reference position, The system includes an update step of updating the model parameters so that the loss obtained by inputting the output data and the ground truth data into the loss function decreases, The aforementioned loss function is, The first error between the target position identified by the output data and the target position identified by the correct data, A second error between the reference position identified by the output data and the reference position identified by the correct data, A third error between the positional relationship identified by the output data and the positional relationship identified by the correct data, Based on at least two of the following, The loss function is a learning method based on the third error.