Learning device, learning method, and program

Abstract

A learning device that performs machine learning includes: an acquisition unit that acquires training data including input data and correct answer data, the input data including an input image including a reference object and input text that relatively specifies a target position with reference to the reference object, and the correct answer data including a correct answer target position; an estimation unit that determines an estimated target position by inputting the input data to a model; and an update unit that updates a parameter of the model so as to reduce a loss obtained by inputting the estimated target position and the correct answer target position to a loss function. The loss function is based on an angle constituted by an estimated vector extending from a start point in the input image to the estimated target position and a correct answer vector extending from the start point to the correct answer target position.

Description

Technical Field

[0001] This invention relates to learning devices, learning methods, and programs. Background Technology

[0002] Various techniques for vehicle driving control using models generated by machine learning have been proposed. Patent Document 1 describes a case of learning a neural network using sensor data acquired from the vehicle. Additionally, techniques for estimating the location within an image indicated by language input using a multimodal model that takes both image and language input have been proposed. As multimodal models, FIBER (Fusion-In-the-Backbone-based transformER) (non-patent document 1), CLIP (Contrastive Language-Image Pre-training) (non-patent document 2), and PWAN (Pixel-Word Attention Module) (non-patent document 3) have been proposed.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Publication No. 2022-513866.

[0004] Non-patent literature Non-Patent Document 1: Zi-Yi Dou, et al., “Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone”, [online], November 18, 2005 (Japan), arXiv, [accessed March 16, 2006 (Japan)], Internet <URL: https: / / arxiv.org / pdf / 2206.07643.pdf> Non-Patent Document 2: Alec Radford, et al., “Learning Transferable Visual Models From Natural Language Supervision”, [online], February 26, 2005 (Japan), arXiv, [accessed March 16, 2006 (Japan)], Internet <URL: https: / / arxiv.org / pdf / 2103.00020.pdf> Non-Patent Document 3: LAVT: Language-Aware Vision Transformer for Referring Image Segmentation Zhao Yang, et al., “LAVT: Language-Aware Vision Transformer for Referring Image Segmentation”, [online], April 5, 2005 (Japan), arXiv, [accessed March 16, 2006 (Japan)], Internet <URL: https: / / arxiv.org / pdf / 2112.02244.pdf>. Summary of the Invention

[0005] The problem that the invention aims to solve It is possible to specify a target location in an input image by referring to a reference object contained within the input image. The target location can be situated in a variety of relative positions with respect to the reference object. One aspect of the object of the present invention is to provide a model capable of estimating a target location specified by referring to a reference object with high accuracy.

[0006] means for solving problems According to one embodiment, a learning apparatus is provided for performing machine learning, wherein the learning apparatus comprises: an acquisition unit that acquires training data including input data and forward retrieval data, wherein the input data includes an input image containing a reference object and input text that specifies a target position relative to the reference object, and the forward retrieval data includes a forward retrieval target position; an estimation unit that determines an estimated target position by inputting the input data into a model; and an update unit that updates the parameters of the model to reduce the loss obtained by inputting the estimated target position and the forward retrieval target position into a loss function, wherein the loss function is based on an angle formed by an estimation vector extending from a starting point in the input image to the estimated target position and a forward retrieval vector extending from the starting point to the forward retrieval target position.

[0007] Invention Effects According to one implementation method, a model is provided that can accurately estimate the target position specified by a reference position.

[0008] Other features and advantages of the invention will become apparent from the following description with reference to the accompanying drawings. Furthermore, in the drawings, the same reference numerals are used to denote the same or identical structures. Attached Figure Description

[0009] The accompanying drawings are included in and form a part of the specification, illustrating embodiments of the invention and serving, together with the description, to explain the principles of the invention.

[0010] Figure 1 This is a block diagram illustrating an example of the hardware configuration of a computer involved in one of the implementation methods.

[0011] Figure 2 This is a schematic diagram illustrating an example of input data involved in one of the implementation methods.

[0012] Figure 3 This is a schematic diagram illustrating an example of correct solution data involved in a portion of the implementation methods.

[0013] Figure 4 This is a schematic diagram illustrating an example of the configuration of a model involved in one of the implementation methods.

[0014] Figure 5 This is a schematic diagram illustrating an example of the configuration of the feature extraction unit involved in a partial implementation.

[0015] Figure 6 This is a schematic diagram illustrating an example of the configuration of the target location estimation unit involved in a certain implementation method.

[0016] Figure 7 This is a schematic diagram illustrating an example of the loss function involved in one of the implementation methods.

[0017] Figure 8 This is a flowchart illustrating an example of a learning method involved in one of the implementation methods.

[0018] Figure 9 This is a flowchart illustrating an example of the presumed method involved in a portion of the implementation methods. Detailed Implementation

[0019] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the invention to which the technical solution pertains. Additionally, not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Furthermore, identical or identical components are labeled with the same reference numerals, and repeated descriptions are omitted.

[0020] Reference Figure 1 The hardware configuration example of the computer 100 involved in a portion of the implementation will be described below. As detailed in the following description, the computer 100 is used to learn a model through machine learning. Therefore, the computer 100 can be referred to as a learning device. The computer 100 can be, for example, a server computer, a personal computer (e.g., a desktop or laptop computer). The computer 100 can also be a computer resource configured in a cloud environment.

[0021] Computer 100 can have Figure 1 The hardware device shown. Processor 101 controls the overall operation of computer 100. Processor 101 may be composed of, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a combination of a CPU and a GPU. Processor 101 may be a single processor or a collection of multiple processors connected together and capable of communicating with each other.

[0022] The memory 102 stores programs and data used for processing by the computer 100. The memory 102 may also be composed of, for example, a combination of random access memory (RAM) and read-only memory (ROM).

[0023] Input device 103 is a device for obtaining instructions from the user of computer 100. Input device 103 may be, for example, a combination of one or more of a keyboard, buttons, touchpad, and microphone. Display device 104 is a device for visually prompting information to the user of computer 100. Display device 104 may be, for example, a dot-matrix display such as an LCD. Computer 100 may have a device that integrates input device 103 and display device 104 (e.g., a touchscreen). Input device 103 and display device 104 may be located externally to the computer. In this case, computer 100 may have an interface for communicating with external input device 103 and display device 104.

[0024] The communication device 105 is a device for communicating with external devices of the computer 100. When the computer 100 performs wired communication, the communication device 105 may be a network interface card (NIC) with a connector for connecting a cable. When the computer 100 performs wireless communication, the communication device 105 may be a wireless communication module including an antenna and baseband processing circuitry.

[0025] Secondary storage device 106 is a device used for non-volatile storage of programs and data used for processing by computer 100. Secondary storage device 106 may be, for example, a hard disk drive (HDD) or a solid disk drive (SSD).

[0026] Computer 100 may be able to communicate with an external database 110. Database 110 may store training data 111 for machine learning based on computer 100. Computer 100 may retrieve training data 111 from database 110. Alternatively, or on this basis, training data 111 may also be stored in secondary storage device 106 of computer 100. Multiple different training data sets 111 may be used in machine learning. Two training data sets 111 may be different if the input data 112 contained in these training data sets 111 are different (e.g., at least one of input text 201 and input image 202 described later is different). A portion of the training data 111 may be used as validation data and test data.

[0027] Training data 111 includes input data 112 and correct answer data 113. Input data 112 can be used to learn the model (e.g., ...). Figure 4 The model 400) is the data input to the model. The correct solution data 113 can be the data that the model should output.

[0028] Reference Figure 2 An example of input data 112 will be described. Input data 112 may include a pair of input image 202 containing reference object 203 and input text 201 that specifies the target position relative to reference object 203. Input text 201 may represent an instruction for the action of vehicle 210 issued by the occupant of vehicle 210.

[0029] The input image 202 can be any image including an object. The input image 202 can also be an image captured by a camera 211 of the vehicle 210. For example, the input image 202 can be an image captured by a camera 211 mounted on the vehicle 210 in a manner that captures images from the front of the vehicle 210. Alternatively, the input image 202 can also be an image captured by a camera mounted on the vehicle 210 in a manner that captures images from other directions (e.g., the rear). The camera 211 of the vehicle 210 can be a camera mounted on the vehicle 210 or a camera brought into the vehicle (e.g., a smartphone of a vehicle occupant). The input image 202 can also be an image unrelated to the vehicle.

[0030] The reference object 203 can be any object contained in the input image 202. Figure 2 In the example, the vehicle is used as reference object 203. Alternatively, reference object 203 can also be other traffic participants besides vehicles, road signs, traffic lights, guardrails, intersections, pedestrian crossings, buildings, signs, etc.

[0031] The input text 201 can be expressed in natural language, for example, as "stop in front of the black car on the right". In this example, "black car on the right" in the input text 201 specifies the reference object 203, and "in front" in the input text 201 specifies the target position relative to the reference object 203. The input text 201 can also be expressed in other forms instead of in natural language. For example, the input text 201 can be selected from multiple candidates of predefined combinations of reference objects and positional relationships.

[0032] Reference Figure 3 An example of correct answer data 113 will be explained below. Correct answer data 113 can be data used to represent the correct answer position of the target location specified by input text 201. Correct answer data 113 can be set manually or by a computer relative to input data 112. In the following description, the case of the correct answer position of the target location will be referred to as the correct answer target position.

[0033] The target location for the forward solving can be represented as point 301 within the input image 202. Alternatively, the target location can be represented as a region centered at point 301. Point 301 can be represented by various coordinate values ​​in a two-dimensional coordinate system defined in the input image 202 (hereinafter simply referred to as "the coordinate system of the input image 202"). The forward solving data 113, as the target location for the forward solving, can include the coordinate values ​​of point 301.

[0034] The target position can be determined by the target position of the reference object 203 and the vector extending from the reference object 203 to the target position. In this case, the target data 113 may include the target position of the reference object 203 and the vector extending from the reference object 203 to the target position. The target position of the reference object 203 is represented as the target reference position. The target reference position can be determined by region 302. Region 302 can be a rectangle with an outer edge circumscribed to the reference object 203. Region 302 can be represented by its center, width, and height. The center of region 302 can be represented by various coordinate values ​​in the coordinate system of the input image 202. Alternatively, region 302 can be represented by the coordinate values ​​of its upper left corner and lower right corner. Region 302 can also be a shape other than a rectangle, such as a circle. The shape of region 302 can vary depending on the shape of the reference object 203. The vector extending from the reference object 203 to the target position can be a vector from the center of region 302 toward point 301. This vector can be represented by the coordinate values ​​in the coordinate system of the input image 202.

[0035] Reference Figure 4The following describes a machine learning model 400 generated by computer 100. Model 400 generates output data based on input data 112 to determine a target location specified by input text 201. In the following description, the target location determined by the output data of model 400 is referred to as the estimated target location. The output data may include the coordinate values ​​of the estimated target location. Alternatively, the output data may include the coordinate values ​​of the position of reference object 203 and a vector extending from reference object 203 to the target location. Model 400 may have any structure where both input text 201 and input image 202 are processed by the model's parameters, thereby affecting the model's output data. Figure 4 Model 400 is an example of such a model. The parameters of the model include at least one of the weights and biases.

[0036] exist Figure 4 In the input text, 201 is represented as x. txt The input image 202 is represented as x img The output data of model 400 is represented as y T The correct answer, 113, is represented as g. T x txt It's text data like "stop in front of the black car on the right". img It is image data. img It can be color image data or black and white image data. img It can be represented by three-dimensional arrangement data of H (height) × W (width) × C (channels). T It can be the coordinates of the estimated target location, for example, it can be represented by a two-dimensional vector. g T It can be the coordinates of the target location, for example, it can be represented by a two-dimensional vector.

[0037] Output data from model 400 (y T The input data is fed into the loss function 405 during the learning process of model 400. The positive solution data 113 (g) corresponding to the input data 112 is also input into the loss function 405. T The loss function 405 outputs the loss based on the error between the output data and the correct solution data 113.

[0038] Model 400 may include a feature extraction unit 401, a text extraction unit 402, a reference object encoding unit 403, and a target position estimation unit 404. Each of these units can be an independent machine learning model. Each of these units can be referred to as a sub-model. Each sub-model can be pre-learned individually before learning model 400. During the learning process of model 400, the parameters of the pre-learned models can be updated or maintained.

[0039] Feature extraction unit 401 based on x txt With x img And generate V, y L z loc V represents multiple feature quantities V1~V3 of the reference object 203 contained in the input image 202. K A set of multiple features V1~V (where K is an integer greater than 2, e.g., K = 5). K They have different resolutions. For example, V i (1≤i≤K) can be derived from H i (Height) × W i (Width) × C i (Channel) three-dimensional arrangement data representation. The size of the three-dimensional arrangement data can be according to each V i However, they are different. For example, it can satisfy 0.5×H. j =H j+1 0.5×W j =W j+1 0.5×C j =C j+1 (All are 1≤j≤K-1).

[0040] y L This indicates the position of the reference object 203 contained within the input image 202. For example, y L It can be represented by a four-dimensional vector (e.g., the coordinates of the center of the region representing the position of the reference object 203, the height of the region, and the width of the region).

[0041] z loc This indicates the position of the reference object 203 contained within the input image 202. For example, z loc The y-based feature extraction unit 401 can be used to extract the y-based feature extraction unit 401. L The reliability of the estimate, and the aspect ratio of the area representing the position of the reference object 203, are added to y. L The subsequent four-dimensional vector representation.

[0042] The feature extraction unit 401 can be constructed from any multimodal model that takes images and language as input and has a hierarchical structure. The output of any layer of the feature extraction unit 401 is used as any feature quantity (V) contained in V. i The feature extraction unit 401 outputs the position of the reference object 203 contained in the correct solution data 113. An example of the specific configuration of the feature extraction unit 401 will be described later.

[0043] Text extraction unit 402 based on x txt And generate x L With x T x L This refers to the text representing the reference object 203 contained in the input image 202. That is, the text extraction unit 402 extracts text from x... txt Extract the text representing the reference object 203 contained in the input image 202. L It can be x txt Part of the text. For example, in x txt In the case of "stopping in front of the black car on the right", x L It could be "the black car on the right". L It can also be x txt Text other than the text in the part.

[0044] x T This refers to text that represents the relative target position with respect to the reference object 203 contained in the input image 202. That is, the text extraction unit 402 extracts text from x... txt Extract text representing the relative target position with respect to the reference object 203 contained in the input image 202. T It can be x txt Part of the text. For example, in x txt In the case of "stopping in front of the black car on the right", x T It could be "before the black car on the right". x T It can also be x txt Text other than the text in the part.

[0045] The text extraction unit 402 can be constructed from any language model. For example, the text extraction unit 402 can be a large-scale language model like GPT4. The text extraction unit 402 can input a prompt such as "Please extract information representing the objects contained in the text 'In front of the black car parked on the right'" into the language model and obtain x. L Regarding x T It's the same.

[0046] Reference object coding unit 403 based on x img x Ly L And generate z L z L These are the feature quantities that represent the reference object 203 contained in the input image 202. For example, z L It can be represented by a 1024-dimensional vector. The reference object encoding unit 403 can be pre-learned so that the output obtained by inputting the feature quantities output from the reference object encoding unit 403 to the output layer represents the type and position of the reference object 203 contained in the forward data 113.

[0047] The reference object encoding unit 403 includes a processing unit and a multimodal unit. The preprocessing unit will process x img The y in L The represented region is extracted as a partial image. This partial image is the portion of the input image 202 that includes the reference object 203. Then, the multimodal unit, based on the partial image extracted by the preprocessing unit and x... L And generate z L The multimodal unit can be constructed, for example, by CLIP (Non-Patent Document 2). As described above, a partial image of the reference object 203 extracted from the input image 202 and the image from the input text (x...) are used. txt The extracted text (x) representing the information of the reference object 203. L This improves the estimation accuracy based on the multimodal part.

[0048] exist Figure 4 In model 400, the reference object encoding unit 403 may also omit the preprocessing unit. In this case, the multimodal unit can be based on x img With x L And generate z L In this case, the feature extraction unit 401 may also not generate y. L .replace Figure 4 For example, the reference object coding unit 403 can use x txt to replace x L In this case, the text extraction unit 402 may also not generate x. L .

[0049] Target location estimation unit 404 is based on V, z loc x T z L And generate y T As mentioned above, x T Based on input text 201(x) txt Therefore, y T Generated based on xtxt. Inputs V1~V1 to the target position estimation unit 404 respectively. K An example of the specific structure of the feature extraction unit 401 will be described later.

[0050] Reference Figure 5 Here is an example of the configuration of the feature extraction unit 401. The feature extraction unit 401 has the same configuration as FIBER, but differs in its further output V. The feature extraction unit 401 includes an image input layer 510, a text input layer 520, an image encoding layer 530, a text encoding layer 540, and an output layer 550. The image input layer 510 converts the input image 202 into a form input to the image encoding layer 530. For example, the image input layer 510 converts the input image 202 into multiple vectors. For example, the image input layer 510 can segment the input image 202 into multiple patch images and sort the pixel values ​​of each patch image into a one-dimensional vector.

[0051] Image coding layer 530 encodes the input image 202 (specifically, represented as an input image 202 consisting of multiple vectors) input from image input layer 510. The specific configuration of image coding layer 530 will be described later. Output layer 550 generates z based on the data encoded by image coding layer 530. loc As described later, the image coding layer 530 outputs a matrix containing multiple row vectors. The output layer 550 can calculate z by multiplying the weight matrix from the right by this output matrix. loc Furthermore, the output layer 550 will z loc A portion of the components as y L And the output.

[0052] The specific structure of image coding layer 530 is described below. Image coding layer 530 may include more than one independent coding layer 560 (in... Figure 5 In the example, there are two), and more than one cooperative coding layer 570 (in Figure 5 (In the example, there are two). When the image coding layer 530 includes multiple independent coding layers 560, they can be connected in series. When the image coding layer 530 includes multiple cooperative coding layers 570, they can be connected in series. More than one independent coding layer 560 can be centrally configured in the first half of the image coding layer 530, and more than one cooperative coding layer 570 can be centrally configured in the second half of the image coding layer 530. Alternatively, independent coding layers 560 and cooperative coding layers 570 can also be configured in a mixed manner.

[0053] The independent coding layer 560 contained in the image coding layer 530 does not use the feature quantities determined by the text coding layer 540 as input, but instead encodes multiple vectors from the layer inputs of the preceding part of the image coding layer 530. The independent coding layer 560 may include a self-attention layer 561 and a fully combined layer 562.

[0054] Multiple vectors input to the independent coding layer 560 are transformed into different multiple vectors by the self-attention layer 561. Multiple vectors output from the self-attention layer 561 are transformed into different multiple vectors by the fully combined layer 562. Multiple vectors output from the fully combined layer 562 are output from the independent coding layer 560. The multiple output vectors of the self-attention layer 561 respectively represent the relationships between other input vectors related to each of the multiple input vectors of the self-attention layer 561.

[0055] The fully combined layer 562 combines all the input vectors, thereby outputting multiple distinct vectors. For example, the fully combined layer 562 multiplies the weight matrix from the right by the matrix Y output from the self-attention layer 561, and adds a bias vector to each row of the resulting matrix. The weight matrix and the bias vector are parameters determined by machine learning. The fully combined layer 562 then outputs a matrix obtained by applying an activation function to each element of the thus computed matrix. The weight matrix of the fully combined layer 562 has the following dimensions: the matrix output from the fully combined layer 562 (i.e., the matrix output from the independent coding layer 560) has the same size as the input matrix of the next independent coding layer 560.

[0056] The cooperative coding layer 570 included in the image coding layer 530 uses the feature quantities determined by the text coding layer 540 as additional inputs to encode multiple vectors from the layer inputs of the preceding layer in the image coding layer 530. In addition to the self-attention layer 561 and the fully combined layer 562 mentioned above, the cooperative coding layer 570 may also include a cross-attention layer 571.

[0057] Multiple vectors input to the cooperative coding layer 570 are transformed into different vectors by the self-attention layer 561. A portion of the feature values ​​determined by the self-attention layer 561 is input to the cross-attention layer 571. A portion of the feature values ​​determined by the cooperative coding layer 570 (specifically, the self-attention layer 561) contained within the text coding layer 540 is also input to the cross-attention layer 571. Based on these inputs, the cross-attention layer 571 generates and outputs multiple vectors.

[0058] Multiple vectors output from self-attention layer 561 are added to multiple vectors output from cross-attention layer 571 and then input to full-association layer 562. Full-association layer 562 transforms the input multiple vectors into different multiple vectors. The multiple vectors output from full-association layer 562 are output from cooperative coding layer 570.

[0059] The multiple output vectors of the cross-attention layer 571 respectively represent the relational relationship between the multiple output vectors of the self-attention layer 561 contained in the image coding layer 530 and each of the multiple output vectors of the self-attention layer 561 contained in the text coding layer 540.

[0060] The output of any one of the independent coding layers 560 and the cooperative coding layer 570 contained in the image coding layer 530 is used as V. i The output is from the feature extraction unit 401. Figure 5 In this example, the output from the upstream (i.e., closest to the image input layer 510) cooperative coding layer 570 is output as V1, and the output from the second upstream cooperative coding layer 570 is output as V2. V1 and V2 have different resolutions. For example, V1 and V2 have different data sizes. Figure 5 In the example, V consists of two features, but the number of features contained in V is not limited to this.

[0061] Reference Figure 6 Here is an example of the configuration of the target position estimation unit 404. The target position estimation unit 404 may include an encoding unit 601, a conversion unit 602, an integration unit 603, and an output unit 604. These components may each include parameters determined by machine learning.

[0062] Encoding Unit 601 is based on x T And generate L T Specifically, the encoding unit 601, through x T L is generated by encoding. T L T It represents text data (x) T The characteristic quantity of L. T It can be represented by a two-dimensional permutation of data, D (dimensionality of features) × T (maximum token length). The encoding unit 601 can be composed of any language model, such as BERT (Bidirectional Encoder Representations from Transformers), or RoBERTa (Robustly Optimized BERT Pretraining Approach).

[0063] The converter 602 is based on V and L T F is generated. F consists of multiple intermediate feature quantities F1~V representing the target location, respectively. K The set of (K is an integer greater than 2. For example, K = 5). The transformation unit 602 uses L for each i (1 ≤ i ≤ K). T V i Convert to F i F i It can have the same characteristics as V. iSame resolution (e.g., data size). In this case, multiple intermediate feature quantities F1~F K They have different resolutions. For example, F i (1≤i≤K) by H i (Height) × Width × C i The three-dimensional arrangement data representation of (channels). The conversion unit 602 can convert V for each i (1≤i≤K). i Individually (i.e., without using V) j (j≠i) is converted to F i .

[0064] The conversion unit 602 may be constructed, for example, by PWAM (Non-Patent Document 3). For example, the conversion unit 602 uses the following formula to convert V... i Convert to F i .

[0065] Q iq =flatten(w iq (V) i )), L ik =w ik (L) T ), L iv =w iv (L) T ), G′ i =softmax(C i -1 / 2t Q iq L ik ) t L iv , G i =w iw (unflatten( t G′ i )), V im =w im (V) i ), F i =w io (V) im G i ). Here, "w" iq “w” ik “w” iv "and "w iw "represents a 1×1 convolution operation." im "and "w io"These represent projection functions, or operations that are input to an activation function (e.g., ReLU) after a 1×1 convolution operation." t " indicates the transpose operation. "C i " " indicates the number of channels. "flatten()" represents the function that flattens a multidimensional arrangement into a one-dimensional arrangement. "unflatten()" represents the inverse function of flatten(). "softmax()" represents the soft maximum function. "*" indicates element-wise multiplication. In the above formulas, variables containing "i" in the index can include those multiplied by each F. i Different parameters. In the above formula, the conversion unit 602 converts V at the pixel level. i Combined with L T .

[0066] Integration Department 603 integrates the various elements of F, namely F1~F K And generate z T For example, integration unit 603 can join the elements of F after applying a general average pooling. For example, z T By (ΣC i A vector representation of (i = 1, ..., K) dimensions.

[0067] Output section 604 based on z T z loc z L And generate y T The output unit 604 can, for example, be constructed from a multilayer perceptron. For example, z T z loc z L It can be used as an independent channel to input to a multilayer sensor.

[0068] Based on the aforementioned model 400, and using feature quantities V1~V1~V2 with multiple resolutions representing the reference object 203... K Therefore, even when the target position is specified with reference to a reference object 203 that may have various sizes, the target position can be estimated with high accuracy.

[0069] In the aforementioned model 400, the target position estimation unit 404 may not be based on z. loc It can also be based on y L To replace z loc In this case, the feature extraction unit 401 may not generate z. loc In the aforementioned model 400, the target position estimation unit 404 may not be based on z. L In this case, the reference object encoding unit 403 can be omitted. In the model 400 described above, the target position estimation unit 404 can be based on x txtto replace x T In this case, the text extraction unit 402 may not generate x. T .

[0070] Next, refer to Figure 7 The loss function 405 used in one part of the implementation will be explained. Like in... Figure 3 As explained, the forward solution data 113 may include a point 301 representing the forward solution target position in the input image 202, and a region 302 representing the forward solution reference position of the reference object 203 in the input image 202. Further, the forward solution data 113 may include a penalty region 705 in the input image 202. The penalty region 705 may be a set of locations that should not be presumed as the target position.

[0071] exist Figure 7 In this diagram, point 701 represents the estimated target position determined by the output data of model 400. Point 702 represents the center of region 302. Point 702 is based on the forward reference position of reference object 203. The vector extending from point 702 to point 301 is represented as the forward vector 703. The forward vector 703 is a vector that starts at the point based on the forward reference position of reference object 203 and ends at the forward target position. The vector extending from point 702 to point 701 is represented as the estimated vector 704. The estimated vector 704 is a vector that starts at the point based on the forward reference position of reference object 203 and ends at the estimated target position.

[0072] The loss function 405 can be used to calculate the loss L using the following formula.

[0073] L=λ d L d +λ a L a +λ n L n +λ p L p . Here, “λ d “λ” a “λ” n "and "λ p "These are positive constants, which can be determined as hyperparameters." L d “L” a “L” n "and "L p "These are the losses defined based on the output data from model 400 and the forward solution data 113, respectively. The following is a detailed explanation of each loss."

[0074] L dIt is the loss based on the distance between the correct target location (point 301) and the estimated target location (point 701). For example, L d It can be the distance between point 301 and point 701, or the result of applying any function to that distance. The distance between point 301 and point 701 can be the L2 norm, the smoothed L1 norm, or any other distance.

[0075] L a This is the loss based on the angle between the positive solution vector 703 and the inferred vector 704. For example, L a It can be the angle between the forward vector 703 and the inferred vector 704, or it can be the result of applying any function to that angle. The angle between the forward vector 703 and the inferred vector 704 can be calculated by applying the inverse cosine function to the inner product of the unit vectors of each vector.

[0076] L n It is the loss based on the difference between the magnitude of the positive solution vector 703 and the magnitude of the inferred vector 704. For example, L n It could be the difference between the magnitude of the positive solution vector 703 and the magnitude of the inferred vector 704, or it could be the result of applying any function to that difference. The magnitudes of the vectors could be, for example, the L2 norm.

[0077] L p The loss is based on whether the presumed target location is included in penalty region 705. L is calculated when the presumed target location is included in penalty region 705. p It can be greater than L when the presumed target location is not included in the penalty region 705. p The penalty region 705 may include the reference object 203. For example, the region 302 representing the correct position of the reference object 203 can also be used as part of the penalty region 705 to calculate L. p .

[0078] As described above, loss function 405 includes not only the loss (L) that explicitly compares the correct target position with the estimated target position. d ), and also includes a loss (L) that explicitly represents the relational relationship with the positive solution position of the reference object 203. a and L n Therefore, model 400 can be learned in a way that allows for high-precision estimation of the target position. Specifically, the loss (L) explicitly represents the relational relationship with the positive solution position of the reference object 203. a and L nThe loss function 405 is based on the difference between the positive solution vector 703 and the estimated vector 704. As described above, the difference between the positive solution vector 703 and the estimated vector 704 can include the angle between them, the difference between the magnitudes of the positive solution vector 703 and the estimated vector 704, or both. Additionally, the loss function 405 includes a loss based on the penalty region 705 (L...). p Therefore, it is possible to prevent unsuitable locations from being presumed as target locations.

[0079] In loss function 405, the loss L is assigned as "L". d “L” a “L” n "and "L p The sum of four terms that are constant multiples of "L". Alternatively, the loss L can also include only "L". d “L” a “L” n "and "L p A term that is part of a phrase. For example, it could be L = λ. a L a +λ n L n .

[0080] In the loss function 405 described above, point 702, based on the positive solution position of reference object 203, is used as the starting point for both the positive solution vector 703 and the inference vector 704. Alternatively, a point independent of the positive solution position of reference object 203, such as the center of input image 202, can also be used as the starting point for both the positive solution vector 703 and the inference vector 704.

[0081] Reference Figure 8 An example of a learning method used to learn Model 400 will be given. Figure 8 The various steps of the method can be processed, for example, by executing a program read into memory 102 using the processor 101 of computer 100. Alternatively, Figure 8 Some or all of the steps of the method can be performed by special-purpose circuits such as application-specific integrated circuits (ASICs). Figure 8 At the start time, the parameters of model 400 can be randomly set or determined by pre-learning.

[0082] In S801, computer 100 acquires training data 111. Training data 111 can be read from database 110 at that point in time, or it can be pre-stored in secondary storage device 106. Training data 111 can be used in batches instead of one at a time.

[0083] In S802, computer 100 inputs the input data 112 contained in the training data 111 acquired in S801 into model 400, thereby generating output data. As described above, the output data is data used to determine the target location (e.g., point 301).

[0084] In S803, computer 100 updates the parameters of model 400 to reduce the loss obtained by inputting the output data generated in S802 and the forward solution data 113 contained in the training data 111 acquired in S801 into the loss function 405. The parameter updates can also use existing methods like Adam. For example, inputting the estimated target location contained in the output data and the forward solution target location contained in the forward solution data 113 into the loss function 405.

[0085] In S804, computer 100 determines whether the condition for ending the repetition of parameter updates (hereinafter, the termination condition) is met. If the termination condition is met ("Yes" in S804), the computer ends the process; if the termination condition is not met ("No" in S804), the computer transfers the process to S801. The termination condition can be the case where the parameters are updated only a predetermined number of times (i.e., the case where S804 is executed). Figure 8 After the processing is completed, the computer 100 can store the learned model 400 in the secondary storage device 106 for future processing, or send the learned model 400 to other devices (e.g., database 110).

[0086] Next, refer to Figure 9 An example of an estimation method for estimating the target location using Model 400 is given. Figure 9 The estimation method can be executed by computer 100, for example. Therefore, computer 100 can be called an estimation device. Figure 9 The computer 100 of the presumption method can be used with the execution Figure 8 The learning methods of computer 100 are different. For example, the processor 10 of computer 100 can execute programs read into memory 102 to process... Figure 9 The various steps of the method. Alternatively, Figure 9 Some or all of the steps of the method can be executed by dedicated circuitry such as an ASIC. Figure 9 At the start time, the computer 100 can utilize the learned model 400. For example, the learned model 400 can be stored in the secondary storage device 106 of the computer 100.

[0087] In S901, computer 100 acquires input data to model 400. The input data may include a pair of input image 202 containing reference object 203 and input text 201 that relatively specifies a target location by referring to reference object 203. In one embodiment, the vehicle acquires audio input from occupants via a microphone and converts it into input text 201. Based on the acquired audio input, the vehicle photographs the scenery in front of it, thereby acquiring input image 202.

[0088] In S902, computer 100 generates output data by inputting the input image acquired in S901 to model 400. As described above, the output data of model 400 includes information for determining the target location. Computer 100 uses the output data of model 400 to determine the target location.

[0089] In S903, the computer 100 performs processing using the target position determined in S902. For example, the vehicle executes processing specified by voice input for the target position. For example, if an instruction to "stop in front of the black car on the right" is given via voice input, the vehicle's movement is controlled in such a way that the position in front of the black car on the right is determined as the target position and the vehicle stops at that target position.

[0090] In the above Figure 9 In the description, model 400 uses a vehicle. Alternatively, model 400 can be used with other devices. For example, in addition to driving a vehicle, model 400 can be used for operating a robot. In this case, instead of a vehicle, images captured by a camera on the robot can be input to model 400, and the target position contained in the user's instruction can be determined. For example, when an instruction such as "Place the cup in front of the black box on the right" is given via voice input, the robot determines the position in front of the black box as the target position, just as it would when using a vehicle. Then, the robot controls its actuators to place the object it is holding (the cup) in the target position.

[0091] <Summary of Implementation Methods> [Project 1] A learning device (100) that performs machine learning, The learning device (100) includes: The acquisition unit acquires training data (111) including input data (112) and forward data (113), wherein the input data includes an input image (202) containing a reference object (203) and input text (201) that specifies the target position relative to the reference object, and the forward data includes the forward target position (301). The estimation unit determines the estimated target location (701) by inputting the input data into the model (400); and An update unit updates the parameters of the model to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function (405). The loss function is based on the difference between the inferred vector (704) extending from the starting point (702) in the input image to the inferred target position and the positive vector (703) extending from the starting point to the positive target position.

[0092] According to this project, it is possible to generate a model that can accurately estimate the target position by referring to a reference object.

[0093] [Project 2] According to the learning device described in Project 1 The positive solution data also includes the positive solution position (302) of the reference object. The starting point is based on the correct solution position of the reference object.

[0094] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0095] [Project 3] The learning device according to Project 1 or 2 The positive solution data also includes the penalty region (705) contained in the input image. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

[0096] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0097] [Project 4] The learning device according to Project 3 The penalty area includes the reference object.

[0098] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0099] [Project 5] The learning device according to any one of items 1 to 4 The difference between the estimated vector and the positive solution vector includes the angle formed by the estimated vector and the positive solution vector.

[0100] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0101] [Project 6] The learning device according to any one of items 1 to 5 The difference between the estimated vector and the positive solution vector includes the difference between the magnitude of the estimated vector and the magnitude of the positive solution vector.

[0102] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0103] [Project 7] The learning device according to any one of items 1 to 6, The learning device is also based on the distance between the estimated target position and the correct solution target position.

[0104] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0105] [Project 8] The learning device according to any one of items 1 to 7 The loss function includes a term formed by multiplying the angle formed by the estimated vector and the positive solution vector by a constant multiple.

[0106] According to this project, it is possible to generate models that can further estimate the target position by referring to a reference object with higher accuracy.

[0107] [Project 9] A learning device (100) that performs machine learning, The learning device (100) includes: The acquisition unit acquires training data (111) including input data (112) and positive solution data (113), wherein the input data includes an input image (202) containing a reference object (203) and input text (201) that specifies the target position relative to the reference object, and the positive solution data includes the positive solution target position (301) and the penalty region (705) contained in the input image. The estimation unit determines the estimated target location (701) by inputting the input data into the model (400); and An update unit updates the parameters of the model to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function (405). The penalty area includes the reference object. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

[0108] According to this project, it is possible to generate a model that can accurately estimate the target position by referring to a reference object.

[0109] [Project 10] The learning device according to any one of items 1 to 9 The input images include images captured by the camera (211) of the vehicle (210).

[0110] According to this project, it is possible to generate a high-precision model that can estimate the target position suitable for vehicle control.

[0111] [Project 11] The learning apparatus according to any one of items 1 to 10, wherein the input text is expressed in natural language.

[0112] According to this project, it is possible to generate models that can accurately estimate the location of a target specified by natural language.

[0113] [Project 12] A program for enabling a computer to function as a unit of any of the learning devices described in any of items 1 to 11.

[0114] According to this project, the above-mentioned effects are achieved through a procedural approach.

[0115] [Project 13] A learning method that performs machine learning. The learning method has the following steps: Acquire training data (111) (S801) including input data (112) and forward data (113), wherein the input data includes an input image (202) containing a reference object (203) and input text (201) that specifies the target position relative to the reference object, and the forward data includes the forward target position (301). The estimated target location (701) is determined by inputting the input data into the model (400) (S802); and The parameters of the model are updated to reduce the loss obtained by inputting the estimated target position and the correct target position into the loss function (405) (S803). The loss function is based on the difference between the inferred vector (704) extending from the starting point (702) in the input image to the inferred target position and the positive vector (703) extending from the starting point to the positive target position.

[0116] According to this project, it is possible to generate a model that can accurately estimate the target position by referring to a reference object.

[0117] [Project 14] A learning method that performs machine learning. The learning method has the following steps: Training data (111) (S801) is obtained, including input data (112) and positive solution data (113), wherein the input data includes an input image (202) containing a reference object (203) and input text (201) that specifies the target position relative to the reference object, and the positive solution data includes the positive solution target position (301) and the penalty region (705) contained in the input image. The estimated target location (701) is determined by inputting the input data into the model (400) (S803). The parameters of the model are updated to reduce the loss obtained by inputting the estimated target position and the correct target position into the loss function (405) (S803). The penalty area includes the reference object. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

[0118] According to this project, it is possible to generate a model that can accurately estimate the target position by referring to a reference object.

[0119] Explanation of reference numerals in the attached figures 100: Computer; 201: Input text; 202: Input image; 400: Model.

Claims

1. A learning device that performs machine learning, wherein, The learning device includes: The acquisition unit acquires training data including input data and forward solution data, wherein the input data includes an input image containing a reference object and input text that specifies the target position relative to the reference object, and the forward solution data includes the forward solution target position; The estimation unit determines the estimated target location by inputting the input data into the model; as well as An update unit updates the parameters of the model to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function. The loss function is based on the difference between the inferred vector extending from the starting point in the input image to the inferred target position and the positive vector extending from the starting point to the positive target position.

2. The learning device according to claim 1, wherein, The positive solution data also includes the positive solution position of the reference object. The starting point is based on the correct solution position of the reference object.

3. The learning device according to claim 1 or 2, wherein, The positive solution data also includes the penalty region contained in the input image. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

4. The learning device according to claim 3, wherein, The penalty area includes the reference object.

5. The learning device according to any one of claims 1 to 4, wherein, The difference between the estimated vector and the positive solution vector includes the angle formed by the estimated vector and the positive solution vector.

6. The learning device according to any one of claims 1 to 5, wherein, The difference between the estimated vector and the positive solution vector includes the difference between the magnitude of the estimated vector and the magnitude of the positive solution vector.

7. The learning device according to any one of claims 1 to 6, wherein, It is also based on the distance between the estimated target position and the correct target position.

8. The learning device according to any one of claims 1 to 7, wherein, The loss function includes a term formed by multiplying the angle formed by the estimated vector and the positive solution vector by a constant multiple.

9. A learning device that performs machine learning, wherein, The learning device includes: The acquisition unit acquires training data including input data and positive solution data, wherein the input data includes an input image containing a reference object and input text that specifies the target position relative to the reference object, and the positive solution data includes the target position of the positive solution and the penalty region contained in the input image; The estimation unit determines the estimated target location by inputting the input data into the model; as well as An update unit updates the parameters of the model to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function. The penalty area includes the reference object. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

10. The learning device according to any one of claims 1 to 9, wherein, The input images include images captured by the vehicle's camera.

11. The learning device according to any one of claims 1 to 10, wherein, The input text is expressed in natural language.

12. A program, wherein, The program is a program for enabling the computer to function as a unit of the learning device according to any one of claims 1 to 11.

13. A learning method that performs machine learning, wherein, The learning method comprises the following steps: Acquire training data including input data and forward solution data, wherein the input data includes an input image containing a reference object and input text that specifies the target position relative to the reference object, and the forward solution data includes the forward solution target position; The estimated target location is determined by inputting the input data into the model; The parameters of the model are updated to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function; as well as The loss function is based on the difference between the inferred vector extending from the starting point in the input image to the inferred target position and the positive vector extending from the starting point to the positive target position.

14. A learning method that performs machine learning, wherein, The learning method comprises the following steps: Acquire training data including input data and positive solution data, wherein the input data includes an input image containing a reference object and input text that specifies the target position relative to the reference object, and the positive solution data includes the target position and the penalty region contained in the input image; The estimated target location is determined by inputting the input data into the model; as well as The parameters of the model are updated to reduce the loss obtained by inputting the estimated target position and the positive solution target position into the loss function. The penalty area includes the reference object. The loss function makes the loss greater when the estimated target location is included in the penalty region than when the estimated target location is not included in the penalty region.

Images