Learning device, estimate device, learning method, estimate method and program

Abstract

To accurately estimate parallax between two images.SOLUTION: A learning device which performs machine learning comprises: an acquisition unit which acquires teacher data including input data and correct answer data, and in which the input data includes a first image and a second image; a generation unit which generates output data which shows parallax between the first image and the second image, by inputting the input data to a model; and an update unit which updates a parameter of the model so that loss obtained by inputting the output data and the correct answer data to a loss function is reduced. The model includes: a feature amount generation unit which generates a first feature amount on the basis of the first image, and generates a second feature amount on the basis of the second image; and a map generation unit which generates a parallax map between the first image and the second image, on the basis of the first feature amount and the second feature amount. The map generation unit includes a cross attention layer which receives input based on the first feature amount, and the input based on the second feature amount. The parallax map is based on the output from the cross attention layer.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program. [Background technology]

[0002] In order to estimate the distance to an object, the disparity between two images of the object captured from two different positions is estimated. Patent Documents 1 and 2 describe methods for estimating the disparity between two images by machine learning. Non-Patent Document 1 describes a model called HITNet (Hierarchical Iterative Tile Refinement Network) that generates a disparity map of two images and then fine-tunes the disparity map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-526818 [Patent Document 2] Special Publication No. 2021-519983 [Non-patent literature]

[0004] [Non-Patent Document 1] Vladimir Tankovich, “HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching”, [online], January 19, 2020, arXiv, [searched on March 16, 2020], Internet<URL:https: / / arxiv.org / pdf / 2007.12140.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] The accuracy of estimating disparity between two images has been improved by utilizing machine learning. However, there is still room for improvement in the accuracy of disparity estimation. One aspect of the present invention aims to provide a technique for accurately estimating disparity between two images. [Means for solving the problem]

[0006] According to some embodiments, there is provided a learning device for performing machine learning, comprising: an acquisition means for acquiring teacher data including input data and ground truth data, the input data including a first image and a second image; a generation means for generating output data representing a disparity between the first image and the second image by inputting the input data into a model; and an update means for updating parameters of the model so as to reduce a loss obtained by inputting the output data and the ground truth data into a loss function, wherein the model comprises: a feature generation unit that generates first features based on the first image and second features based on the second image; and a map generation unit that generates a disparity map between the first image and the second image based on the first features and the second features, wherein the map generation unit includes a cross-attention layer that receives input based on the first features and input based on the second features, and the disparity map is based on an output from the cross-attention layer. [Effects of the Invention]

[0007] According to some embodiments, the disparity between two images can be estimated with high accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a computer according to some embodiments. [Figure 2] FIG. 10 is a schematic diagram illustrating an example of input data according to some embodiments. [Figure 3] FIG. 10 is a schematic diagram illustrating an example of the configuration of a model according to some embodiments. [Figure 4] FIG. 4 is a schematic diagram illustrating an example configuration of a feature generator according to some embodiments. [Figure 5] FIG. 1 is a schematic diagram illustrating an example configuration of a self-attention layer according to some embodiments. [Figure 6] FIG. 10 is a schematic diagram illustrating an example configuration of a cross-attention layer according to some embodiments. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of the configuration of a model according to some embodiments. [Figure 8] FIG. 1 is a flow diagram illustrating an example of a training method according to some embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating an example estimation method according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] With reference to FIG. 1 , an example hardware configuration of a computer 100 according to some embodiments will be described. As will be described in detail below, the computer 100 is used to train a model by machine learning. Therefore, the computer 100 may be referred to as a learning device. The computer 100 may be, for example, a server computer or a personal computer (e.g., a desktop or laptop computer). The computer 100 may also be a computer resource located in a cloud environment.

[0011] The computer 100 may include the hardware devices shown in Fig. 1. The processor 101 controls the overall operation of the computer 100. The processor 101 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or a combination of these. The processor 101 may be a single processor or a collection of multiple processors connected to each other so that they can communicate with each other.

[0012] The memory 102 stores programs and data used in the processing of the computer 100. The memory 102 may be configured, for example, by a combination of random access memory (RAM) and read-only memory (ROM).

[0013] The input device 103 is a device for obtaining instructions from a user of the computer 100. The input device 103 may be configured, for example, by a combination of one or more of a keyboard, buttons, a touchpad, and a microphone. The display device 104 is a device for visually presenting information to a user of the computer 100. The display device 104 may be, for example, a dot-matrix display such as a liquid crystal display. The computer 100 may have a device (e.g., a touch screen) in which the input device 103 and the display device 104 are integrated. The input device 103 and the display device 104 may be external to the computer. In this case, the computer 100 may have an interface for communicating with the external input device 103 and display device 104.

[0014] The communication device 105 is a device for communicating with devices external to the computer 100. When the computer 100 performs wired communication, the communication device 105 may be a network interface card (NIC) having a connector for connecting a cable. When the computer 100 performs wireless communication, the communication device 105 may be a wireless communication module including an antenna and a baseband processing circuit.

[0015] The secondary storage device 106 is a device for non-volatilely storing programs and data used in the processing of the computer 100. The secondary storage device 106 is configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0016] The computer 100 may be capable of communicating with an external database 110. The database 110 may store training data 111 used for machine learning by the computer 100. The computer 100 may acquire the training data 111 from the database 110. Alternatively or in addition, the training data 111 may be stored in the secondary storage device 106 of the computer 100. In machine learning, multiple different training data 111 are used. Two training data 111 being different may mean that the input data 112 included in these training data 111 are different. A portion of the training data 111 may be used as verification data and test data.

[0017] The training data 111 includes input data 112 and supervised data 113. The input data 112 may be data input to a model (e.g., model 300 in FIG. 3) to train the model. The supervised data 113 may be data to be output by the model.

[0018] An example of the input data 112 will be described with reference to FIG. 2. The input data 112 may include a pair of two images. Hereinafter, a pair of two images will be referred to as an image pair. The image pair included in the input data 112 may be two images captured by a stereo camera 211. For example, the stereo camera 211 may include a right camera 211R and a left camera 211L that are spatially spaced apart. The image pair included in the input data 112 may be a right image 201R captured by the right camera 211R and a left image 201L captured by the left camera 211L. Typically, the right image 201R and the left image 201L have the same resolution. The right image 201R and the left image 201L may be color images or monochrome images.

[0019] The stereo camera 211 may be attached to the vehicle 210. The vehicle 210 may be a vehicle that can accommodate an occupant or a micromobility. Alternatively, the stereo camera 211 may be attached to a moving body other than the vehicle 210. For example, the stereo camera 211 may be attached to a robot that transports luggage or leads people. For example, the stereo camera 211 may be attached to the vehicle 210 so as to capture an image of the area ahead of the vehicle 210. Alternatively, the image pair included in the input data 112 may be images captured by a camera brought into the vehicle 210 (for example, a smartphone of a vehicle occupant). The image pair included in the input data 112 may be images unrelated to the vehicle. Furthermore, the image pair included in the input data 112 may be two images of the same subject captured by a single camera at different times.

[0020] The supervised data 113 may include a disparity map of the image pair included in the input data 112. The disparity map may be an image that represents the disparity at each pixel between the right image 201R and the left image 201L. The disparity map may be generated based on one of the right image 201R and the left image 201L. In the following description, a case where the disparity map is represented based on the right image 201R will be described. Alternatively, the disparity map may be represented based on the left image 201L.

[0021] The pixel value of a specific pixel in the disparity map represents the distance between a pixel in the right image 201R at the same position as the specific pixel and a pixel in the left image 201L that represents the same object as the pixel in the right image 201R. The disparity map may have the same resolution (i.e., the same number of pixels) as the right image 201R. In this case, one pixel in the disparity map corresponds to one pixel in the right image 201R. The disparity of one pixel in the right image 201R is represented by the pixel value of the corresponding pixel in the disparity map. Alternatively, the disparity map may have a lower resolution (i.e., fewer pixels) than the right image 201R. In this case, one pixel in the disparity map corresponds to multiple pixels in the right image 201R. The disparity of each of the multiple pixels in the right image 201R is represented by the pixel value of the corresponding pixel in the disparity map.

[0022] Referring to FIG. 3, a model 300 that is machine-learned by the computer 100 will be described. The model 300 generates output data based on input data 112. As described above, the input data 112 may include a right image 201R and a left image 201L. The output data may include a disparity map. The disparity map represents the disparity between the right image 201R and the left image 201L. The output data output from the model 300 is input to a loss function 310 during training of the model 300. The correct answer data 113 corresponding to the input data 112 is also input to the loss function 310. The output data of the model 300 may have the same data structure as the correct answer data 113. The loss function 310 outputs a loss based on the error between the output data and the correct answer data 113.

[0023] The model 300 includes two feature generators 301R and 301L and a map generator 302. The model 300 may include other components. The feature generator 301R generates a feature representing the right image 201R based on the right image 201R. In the following description, the feature representing the right image 201R is referred to as a right feature y R The right image 201R may be expressed as a three-dimensional array of (height) x (width) x (number of channels). The right feature quantity y R may be expressed as a three-dimensional array of (height) x (width) x (number of channels). R The resolution of the right image 201R may be the same as or smaller than the resolution of the right image 201R.

[0024] The feature generating unit 301L generates a feature representing the left image 201L based on the left image 201L. In the following description, the feature representing the left image 201L is referred to as a left feature y L The data structure of the left image 201L may be the same as the data structure of the right image 201R. L The data structure of the right feature y R The data structure may be the same as that of

[0025] The map generation unit 302 generates the right feature value y R and left feature y L A disparity map z between the right image 201R and the left image 201L is generated based on the above. The disparity map z may be represented as a two-dimensional array of (height) x (width), for example. The resolution of the disparity map z may be the same as or smaller than the resolution of the right image 201R.

[0026] Next, with reference to FIG. 4, an example configuration of the feature generation unit 301R will be described. The feature generation unit 301L may have the same configuration as the feature generation unit 301R. The feature generation unit 301R includes an image input layer 410, multiple encoder layers 420, and multiple decoder layers 430. The feature generation unit 301R may also include other layers. In the example of FIG. 4, the feature generation unit 301R includes two consecutive encoder layers 420. Alternatively, the feature generation unit 301R may include other numbers of encoder layers 420, for example, only one encoder layer 420. In the example of FIG. 4, the feature generation unit 301R includes two consecutive decoder layers 430. Alternatively, the feature generation unit 301R may include other numbers of decoder layers 430, for example, only one decoder layer 430. 4, an image input layer 410 is followed by a plurality of encoder layers 420 connected in series, which are then followed by a plurality of decoder layers 430. Alternatively, the plurality of encoder layers 420 and the plurality of decoder layers 430 may be interleaved.

[0027] The image input layer 410 converts the right image 201R into a format to be input to the encoder layer 420. The image input layer 410 may have a configuration similar to that of the input layer of a ViT (Vision Transformer). For example, the image input layer 410 converts the right image 201R into multiple vectors. For example, the image input layer 410 may divide the right image 201R into multiple patch images and rearrange the pixel values ​​of each patch image into a one-dimensional vector. Furthermore, the image input layer 410 may embed the positions of the patch images into the one-dimensional vector, similar to the input layer of a ViT. The image input layer 410 outputs multiple one-dimensional vectors representing the right image 201R. The image input layer 410 may further output class tokens of the same size as the patch images.

[0028] The encoder layer 420 encodes each of the multiple vectors input from the previous layer. This extracts features from the data input to the encoder layer 420. The encoder layer 420 may generate output data with a lower resolution than the input data. In this case, the resolution of the data decreases as it passes through one encoder layer 420.

[0029] The encoder layer 420 may have a configuration similar to that of the encoder block of ViT. For example, the encoder layer 420 may include a self-attention layer 421 and a fully connected layer 422. A plurality of vectors input to the encoder layer 420 are converted into a plurality of different vectors by the self-attention layer 421. A plurality of vectors output from the self-attention layer 421 are converted into a plurality of different vectors by the fully connected layer 422. A plurality of vectors output from the fully connected layer 422 are output from the encoder layer 420. The input to each layer (e.g., the self-attention layer 421) of the feature generator 301R is based on the right image 201R. The output from the feature generator 301R (i.e., the right feature y R ) is based on the output of each layer (e.g., self-attention layer 421) of feature generator 301R.

[0030] The encoder layer 420 may include a path 423 that bypasses the self-attention layer 421. In this case, the input to the self-attention layer 421 is added to the output to the self-attention layer 421. Alternatively, the encoder layer 420 may not include the path 423. The encoder layer 420 may include a path 424 that bypasses the fully connected layer 422. In this case, the input to the fully connected layer 422 is added to the output to the fully connected layer 422. Alternatively, the encoder layer 420 may not include the path 424. The encoder layer 420 may further include a normalization layer before the self-attention layer 421. The encoder layer 420 may further include a normalization layer before the fully connected layer 422.

[0031] The decoder layer 430 encodes each of the multiple vectors input from the previous layer. This allows features to be extracted from the data input to the decoder layer 430. The decoder layer 430 may generate output data with a higher resolution than the input data. In this case, the resolution of the data increases as the data passes through one decoder layer 430.

[0032] The decoder layer 430 may have a configuration similar to that of the decoder block of HITNet. For example, the decoder layer 430 may include a convolutional layer 431. A plurality of vectors input to the decoder layer 430 are converted into a plurality of different vectors by the convolutional layer 431. In FIG. 4, the decoder layer 430 includes one convolutional layer 431. Alternatively, the decoder layer 430 may include a plurality of convolutional layers with different parameters (e.g., filter size and stride).

[0033] An example of the configuration of the self-attention layer 421 will be described with reference to FIG. 5. Each of the multiple output vectors of the self-attention layer 421 represents the relationship of each input vector with respect to other input vectors in the multiple input vectors of the self-attention layer 421. The self-attention layer 421 combines multiple input row vectors into one two-dimensional input matrix X. The self-attention layer 421 applies a weight matrix W to the input matrix X. Q , weight matrix W K , weight matrix W V The query Q, key K, and value V are calculated by multiplying them from the right. Q , weight matrix W K , and the weight matrix W V is a parameter determined by machine learning.

[0034] The self-attention layer 421 includes a score calculation unit 501. The score calculation unit 501 calculates a score S based on a query Q and a key K. Specifically, the score calculation unit 501 calculates an intermediate matrix by post-multiplying the query Q by a transposed matrix of the key K and dividing each component by a predetermined value (e.g., the square root of the number of columns of the key K). Then, the score calculation unit 501 calculates a score S by applying a Softmax function to each row of the intermediate matrix. Then, the self-attention layer 421 calculates a matrix Y by post-multiplying the score S by a value V. The self-attention layer 421 outputs the matrix Y calculated in this manner. The multiple rows of the matrix Y correspond to the multiple row vectors output from the self-attention layer 421.

[0035] As described above, the feature generator 301R includes the self-attention layer 421, so that the feature generator 301R can accurately extract features from the entire right image 201R. The same applies to the feature generator 301L.

[0036] Next, a configuration example of the map generation unit 302 will be described with reference to Fig. 6. The map generation unit 302 includes image input layers 601 and 602, a cross-attention layer 603, and a transformation layer 605. The map generation unit 302 may include other layers.

[0037] The image input layer 601 is similar to the image input layer 410 in that it calculates the right feature value y R The image input layer 602 converts the left feature value y L The cross attention layer 603 combines the multiple row vectors output from the image input layer 601 into a single two-dimensional input matrix, and generates a weight matrix W K The cross attention layer 603 combines the multiple row vectors output from the image input layer 602 into a single two-dimensional input matrix, and calculates the key K by multiplying the Q The query Q is calculated by multiplying it from the right and the weight matrix W V The value V is calculated by multiplying the weight matrix W Q, weight matrix W K , and the weight matrix W V are parameters determined by machine learning. The parameters of the cross-attention layer 603 may have different values ​​from those of the self-attention layer 421. In the example of FIG. 6, the image input layer 601 is provided with the right feature y R is input, and the left feature value y L Instead, the left feature value y L is input, and the right feature value y R may be input.

[0038] Similar to the score calculation unit 501, the score calculation unit 604 calculates the score S based on the query Q and the key K. Then, the cross attention layer 603 outputs a matrix obtained by post-multiplying the score S by the value V. The transformation layer 605 transforms the output from the cross attention layer 603 into the data structure of the disparity map z.

[0039] The cross attention layer 603 uses the right feature y R The input based on the left feature y L The disparity map z is based on the output from the cross-attention layer 603. This allows the map generation unit 302 to accurately associate pixels in the right image 201R with pixels in the left image 201L. Specifically, the score calculation unit 604 of the cross-attention layer 603 calculates the right feature amount y R The input based on the left feature y L The score between the input based on the input is calculated, and the multiple disparities are weighted based on this score to estimate the disparity. Therefore, the disparity can be estimated with finer granularity than when selecting any one of the multiple disparities.

[0040] In the above example, the output from the map generator 302 is used as the output from the model 300 (i.e., the disparity map z). Alternatively, the model 300 may include a layer after the map generator 302 for fine-tuning the output of the map generator 302. The layer for fine-tuning may be, for example, an existing configuration, such as the configuration used in HITNet.

[0041] A modification of the above-described model 300 will be described with reference to Fig. 7. Model 700 differs from model 300 in that it further includes a correction unit 701 after the map generation unit 302. When model 700 is used, input data 112 may include time-series data of image pairs. Image pairs may be input to model 700 in chronological order (i.e., from older image pairs to newer image pairs).

[0042] The map generation unit 302 generates a disparity map based on the image pair at each time (i.e., the right image 201R and the left image 201L), and outputs the disparity map to the correction unit 701. The correction unit 701 corrects the disparity map generated by the map generation unit 302. Specifically, based on the disparity map generated by the map generation unit 302 for the image pair at a certain time, the correction unit 701 corrects the disparity map generated by the map generation unit 302 for the image pair at a later time. In other words, based on the disparity map generated by the map generation unit 302 for the past image pair, the correction unit 701 corrects the disparity map generated by the map generation unit 302 for the current image pair.

[0043] The correction unit 701 may be configured, for example, by a Gated Recurrent Unit (GRU) or a Convolutional Gated Recurrent Unit (ConvGRU). Specifically, the correction unit 701 may store internal data representing a disparity map generated by the map generation unit 302 for a past image pair, and correct the disparity map generated by the map generation unit 302 for a current image pair based on the internal data.

[0044] An example of a learning method for learning the model 300 will be described with reference to Fig. 8. Each step of the method in Fig. 8 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method in Fig. 8 may be executed by a dedicated circuit such as an application specific integrated circuit (ASIC). At the start of Fig. 8, the parameters of the model 300 may be set to random values.

[0045] In S801, the computer 100 acquires one piece of training data 111. The training data 111 may be read out from the database 110 at this point, or may be stored in advance in the secondary storage device 106. Instead of using each piece of training data 111 one by one, multiple pieces of training data 111 may be used together as a batch.

[0046] In S802, the computer 100 generates output data by inputting the input data 112 included in the training data 111 acquired in S801 into the model 300. As described above, the output data may include a disparity map.

[0047] In S803, the computer 100 updates the parameters of the model 300 so as to reduce the loss obtained by inputting the output data generated in S802 and the correct answer data 113 included in the training data 111 acquired in S801 into the loss function 310. The parameter update may be performed using an existing method such as Adam. The loss function 310 may include, for example, an L1 error of pixel values.

[0048] In S804, the computer 100 determines whether a condition for terminating the parameter update iterations (hereinafter, the termination condition) is satisfied. If it is determined that the termination condition is satisfied ("YES" in S804), the computer terminates the process. Otherwise ("NO" in S804), the computer transitions the process to S801. The termination condition may be that the parameters have been updated a predetermined number of times (i.e., S804 has been executed). After executing the process of FIG. 8, the computer 100 may store the trained model 300 in the secondary storage device 106 for future processing or may transmit it to another device (e.g., the database 110).

[0049] Next, with reference to FIG. 9 , an example of an estimation method for estimating disparity using the model 300 will be described. The estimation method of FIG. 9 may be executed by, for example, the computer 100. Therefore, the computer 100 may be referred to as an estimation device. The computer 100 that executes the estimation method of FIG. 9 may be different from the computer 100 that executes the learning method of FIG. 8. Each step of the method of FIG. 9 may be processed, for example, by the processor 101 of the computer 100 executing a program read into the memory 102. Alternatively, some or all of the steps of the method of FIG. 9 may be executed by a dedicated circuit such as an ASIC. It is assumed that a trained model 300 is available to the computer 100 at the start of FIG. 9 . For example, the trained model 300 may be stored in the secondary storage device 106 of the computer 100.

[0050] In S901, the computer 100 acquires input data to be input to the model 300. The input data may include a right image 201R and a left image 201L. The input data may be images captured by a stereo camera 211 of a moving object such as a vehicle 210.

[0051] In S902, the computer 100 generates output data by inputting the input image acquired in S901 into the model 300. As described above, the output data of the model 300 includes a disparity map. The disparity map indicates an estimate of the disparity between the right image 201R and the left image 201L. Therefore, in S902, the disparity between the right image 201R and the left image 201L is estimated. The computer 100 creates a depth map based on the estimated disparity, and the depth map may be used to control a moving object such as the vehicle 210.

[0052] <Summary of the embodiment> [Item 1] A learning device (100) that performs machine learning, An acquisition means for acquiring teacher data including input data (112) and correct answer data (113), wherein the input data includes a first image (201R) and a second image (201L); generating means for generating output data representing the disparity between the first image and the second image by inputting the input data into a model (300, 700); an update means for updating the parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function (310) is reduced; The model is a feature generating unit (301) that generates a first feature based on the first image and a second feature based on the second image; a map generation unit (302) that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer (603) that receives an input based on the first feature amount and an input based on the second feature amount; The disparity map is based on the output from the cross-attention layer. According to this item, a model can be generated that can accurately identify the correspondence between two images, thereby enabling accurate estimation of the disparity between the two images. [Item 2] The feature generation unit includes a self-attention layer (421), the input to the self-attention layer is based on the first image; Item 1. The learning device according to item 1, wherein the first feature is based on the output of the self-attention layer. This item allows for the generation of a model that can properly extract image features, thereby enabling more accurate estimation of the disparity between two images. [Item 3] 3. The learning device according to item 2, wherein the feature generation unit includes a path (423) that bypasses the self-attention layer. This item allows for the generation of a model that can more appropriately extract image features, thereby enabling more accurate estimation of the disparity between two images. [Item 4] the input data includes time-series data of an image pair of the first image and the second image; The model further includes a correction unit (701) that corrects the disparity map generated by the map generation unit, The learning device according to any one of items 1 to 3, wherein the correction unit corrects the disparity map generated by the map generation unit for the image pair at a second time that is later than the first time, based on the disparity map generated by the map generation unit for the image pair at a first time. According to this item, the disparity between two images can be estimated more accurately using time series data. [Item 5] 5. The learning device according to item 4, wherein the correction unit is configured by a Convolutional Gated Recurrent Unit (ConvGRU). This item allows for more accurate estimation of the disparity between two images. [Item 6] 6. The learning device according to any one of items 1 to 5, wherein the first image and the second image are two images captured by a stereo camera (211) of a moving object (210). This item allows for more accurate estimation of the disparity between two images used in a moving object. [Item 7] A program for causing a computer to function as each means of the learning device described in any one of items 1 to 6. According to this item, the above-mentioned effects can be obtained in the form of a program. [Item 8] An estimation device (100) for estimating disparity, an acquisition means for acquiring input data including a first image (201R) and a second image (201L); an estimation means for estimating the disparity between the first image and the second image by inputting the input data into a model (300, 700); Equipped with The model is a feature generating unit (301) that generates a first feature based on the first image and a second feature based on the second image; a map generation unit (302) that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer (603) that receives an input based on the first feature amount and an input based on the second feature amount; The disparity map is based on the output from the cross-attention layer. According to this item, the disparity between two images can be estimated with high accuracy using a model that can accurately identify the correspondence between the two images. [Item 9] A program for causing a computer to function as each means of the estimation device described in item 8. According to this item, the above-mentioned effects can be obtained in the form of a program. [Item 10] A learning method for machine learning, comprising: an acquisition step (S801) of acquiring training data including input data (112) and correct answer data (113), wherein the input data includes a first image (201R) and a second image (201L); a generating step (S802) of generating output data representing the disparity between the first image and the second image by inputting the input data into a model (300, 700); An updating step (S803) of updating the parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function (310) is reduced, The model is a feature generating unit (301) that generates a first feature based on the first image and a second feature based on the second image; a map generation unit (302) that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer (603) that receives an input based on the first feature amount and an input based on the second feature amount; The learning method, wherein the disparity map is based on the output from the cross-attention layer. According to this item, a model can be generated that can accurately identify the correspondence between two images, thereby enabling accurate estimation of the disparity between the two images. [Item 11] A method for estimating disparity, comprising: An acquisition step (S901) of acquiring input data including a first image (201R) and a second image (201L); an estimation step (S902) of estimating the disparity between the first image and the second image by inputting the input data into a model (300, 700); Equipped with The model is a feature generating unit (301) that generates a first feature based on the first image and a second feature based on the second image; a map generation unit (302) that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer (603) that receives an input based on the first feature amount and an input based on the second feature amount; The estimation method, wherein the disparity map is based on the output from the cross-attention layer. According to this item, the disparity between two images can be estimated with high accuracy using a model that can accurately identify the correspondence between the two images.

[0053] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0054] 100 computer, 112 input data, 113 correct answer data, 300 model

Claims

1. A learning device that performs machine learning, an acquisition means for acquiring teacher data including input data and correct answer data, the input data including a first image and a second image; generating means for generating output data representing the disparity between the first image and the second image by inputting the input data into a model; an update means for updating parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function is reduced; The model is a feature generating unit that generates a first feature based on the first image and a second feature based on the second image; a map generation unit that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer that receives an input based on the first feature amount and an input based on the second feature amount; The disparity map is based on the output from the cross-attention layer.

2. The feature generator includes a self-attention layer, the input to the self-attention layer is based on the first image; The learning device according to claim 1 , wherein the first feature is based on an output of the self-attention layer.

3. The learning device according to claim 2 , wherein the feature generation unit includes a path that bypasses the self-attention layer.

4. the input data includes time-series data of an image pair of the first image and the second image; the model further includes a correction unit that corrects the disparity map generated by the map generation unit; 2. The learning device according to claim 1, wherein the correction unit corrects the disparity map generated by the map generation unit for the image pair at a second time that is later than the first time, based on the disparity map generated by the map generation unit for the image pair at a first time.

5. The learning device according to claim 4 , wherein the correction unit is configured by a ConvGRU (Convolutional Gated Recurrent Unit).

6. The learning device according to claim 1 , wherein the first image and the second image are two images captured by a stereo camera of a moving object.

7. A program for causing a computer to function as each of the means of the learning device according to any one of claims 1 to 6.

8. An estimation device for estimating disparity, comprising: acquiring means for acquiring input data including a first image and a second image; an estimation means for estimating a disparity between the first image and the second image by inputting the input data into a model; Equipped with The model is a feature generating unit that generates a first feature based on the first image and a second feature based on the second image; a map generation unit that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer that receives an input based on the first feature amount and an input based on the second feature amount; The disparity map is based on the output from the cross-attention layer.

9. A program for causing a computer to function as each means of the estimation device according to claim 8.

10. A learning method for machine learning, comprising: an acquisition step of acquiring teacher data including input data and correct answer data, the input data including a first image and a second image; generating output data representing the disparity between the first image and the second image by inputting the input data into a model; an updating step of updating parameters of the model so that a loss obtained by inputting the output data and the correct answer data into a loss function is reduced; The model is a feature generating unit that generates a first feature based on the first image and a second feature based on the second image; a map generation unit that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer that receives an input based on the first feature amount and an input based on the second feature amount; The learning method, wherein the disparity map is based on the output from the cross-attention layer.

11. A method for estimating disparity, comprising: acquiring input data including a first image and a second image; an estimation step of estimating a disparity between the first image and the second image by inputting the input data into a model; Equipped with The model is a feature generating unit that generates a first feature based on the first image and a second feature based on the second image; a map generation unit that generates a disparity map between the first image and the second image based on the first feature amount and the second feature amount, the map generation unit includes a cross-attention layer that receives an input based on the first feature amount and an input based on the second feature amount; The estimation method, wherein the disparity map is based on the output from the cross-attention layer.

Images