Learning model

Abstract

Accurately estimate parallax. [Solution] The learning model is input with a first image, a second image, and a third image. The first image and the second image have parallax in the first direction. The first image and the third image have parallax in the third direction, which is tilted at a predetermined angle from the first direction toward the second direction. The second direction is orthogonal to the first direction. The learning model generates a first parallax map that associates the parallax between the first image and the second image. The learning model generates a second parallax map that associates the parallax between the first image and the third image. The learning model updates the first and second parallax maps by alternately performing a first process using the feature map of the first image and the feature map of the third image, and a second process using the feature map of the first image and the feature map of the second image.

Description

[Technical Field]

[0001] This disclosure relates to a learning model. [Background technology]

[0002] In recent years, with the advancement of deep learning, stereo matching using machine learning has been actively researched. In stereo matching, the disparity between corresponding parts of two images is estimated. Non-patent document 1 describes a method for estimating disparity using a quad-pixel sensor. Non-patent document 2 describes a method for estimating disparity using a three-lens system. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zhuofeng Wu et al., “Disparity Estimation Using a Quad-Pixel Sensor,” arXiv preprint arXiv:2409.00665, 2024. [Non-Patent Document 2] Faranak Shamsafar et al., “riterpene: A Trinocular Framework for Multi-baseline Disparity Estimation,” arXiv preprint arXiv:2111.12502, 2021. [Non-Patent Document 3] Jiankun Li et al., “Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation,” arXiv preprint arXiv:2203.11483, 2022. [Overview of the project] [Problems that the invention aims to solve]

[0004] Conventional methods using machine learning have room for improvement. For example, in conventional methods using machine learning, when the same pattern is repeatedly arranged in the horizontal direction in two images, the accuracy of parallax estimation may deteriorate.

[0005] An object of the present disclosure made in view of such a point is to accurately estimate parallax. [Means for Solving the Problems]

[0006] A learning model according to an embodiment of the present disclosure is a learning model to which a first image, a second image, and a third image are input, wherein the first image and the second image have parallax in a first direction, the first image and the third image have parallax in a third direction inclined at a predetermined angle from the first direction toward a second direction, and the second direction is orthogonal to the first direction, the learning model generates a first parallax map in which parallax between the first image and the second image is associated, generates a second parallax map in which parallax between the first image and the third image is associated, and alternately executes a first process using the feature map of the first image and the feature map of the third image, and a second process using the feature map of the first image and the feature map of the second image, to update the first parallax map and the second parallax map. [Effects of the Invention]

[0007] According to an embodiment of the present disclosure, parallax can be estimated with high accuracy. [Brief Description of the Drawings]

[0008] [Figure 1] This is a block diagram illustrating a configuration example of an imaging system according to an embodiment of the present disclosure. [Figure 2] This is a diagram for explaining an example of the positional relationship of the first camera, the second camera, and the third camera shown in FIG. 1. [Figure 3] This diagram illustrates an example of the positional relationship between the first, second, and third images. [Figure 4] This block diagram shows an example of the configuration of a learning model according to one embodiment of this disclosure. [Figure 5] This block diagram shows an example of the configuration of the first RUM shown in Figure 4. [Figure 6] This block diagram shows an example of the configuration of AGCL, as shown in Figure 5. [Figure 7] This block diagram shows an example of the configuration of the second RUM, as shown in Figure 4. [Figure 8] Figure 1 is a flowchart illustrating an example of the operations performed by the image processing device shown. [Figure 9] This figure shows the experimental results. [Figure 10] This figure shows the experimental results. [Modes for carrying out the invention]

[0009] The embodiments relating to this disclosure will be described below with reference to the drawings.

[0010] [Configuration of the imaging system] The imaging system 1 according to this embodiment, as shown in Figure 1, includes a stereo camera 10 and an image processing device 20. The stereo camera 10 and the image processing device 20 are connected to each other via wired or wireless communication.

[0011] The stereo camera 10 generates a stereo image by capturing images of the same subject from three different viewpoints. The stereo camera 10 includes a first camera 10L, a second camera 10R, and a third camera 10T as the three different viewpoints. The first camera 10L generates a first image I1, as shown in Figure 3 below, by capturing images of the subject. The second camera 10R generates a second image I2, as shown in Figure 3 below, by capturing images of the subject. The third camera 10T generates a third image I3, as shown in Figure 3 below, by capturing images of the subject. The first image I1, the second image I2, and the third image I3 are stereo images generated by capturing images of the same subject from three different viewpoints. The resolution of the first image I1, the second image I2, and the third image I3 may be the same.

[0012] As shown in Figures 1 and 2, the stereo camera 10 includes a first camera 10L, a second camera 10R, and a third camera 10T, each comprising lenses 11L, 11R, and 11T, and an image sensor 12. The first camera 10L is composed of a lens 11L and an image sensor 12. The second camera 10R is composed of a lens 11R and an image sensor 12. The third camera 10T is composed of a lens 11T and an image sensor 12. However, the first camera 10L, the second camera 10R, and the third camera 10T may be independent cameras.

[0013] The optical axes OX1 of lens 11L, OX2 of lens 11R, and OX3 of lens 11T are oriented in directions that allow them to image the same subject. The optical axes OX1 of lens 11L, OX2 of lens 11R, and OX3 of lens 11T may be parallel to each other. The sizes of lens 11L, lens 11R, and lens 11T may be the same. Lenses 11L, 11R, and 11T are positioned apart from each other.

[0014] As shown in Figure 2, the lenses 11L, 11R, and 11T are arranged within the plane of the image sensor 12. The image sensor 12 is, for example, a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor).

[0015] As shown in Figure 2, the first direction D1 corresponds to the horizontal direction. When the stereo camera 10 is imaging a subject, it faces the subject. When the stereo camera 10 is facing the subject, the first direction D1 is the direction from left to right as seen from the stereo camera 10. The second direction D2 is the vertical direction perpendicular to the horizontal direction. In other words, the second direction D2 is perpendicular to the first direction D1. When the stereo camera 10 is facing the subject, the second direction D2 is the direction from bottom to top as seen from the stereo camera 10. The third direction D3 is the direction tilted at a predetermined angle θ from the first direction D1 toward the second direction D2. As will be described later, lenses 11L and 11T are aligned along the third direction D3. The predetermined angle θ may be set based on the size of lenses 11L, 11R, and 11T, etc. The predetermined angle θ may be 0 degrees < θ ≤ 90 degrees. Also, the predetermined angle θ may be 20 degrees ≤ θ ≤ 90 degrees. By setting the predetermined angle θ to 20 degrees ≤ θ, the lenses 11L and 11T, described later, can be positioned with an appropriate angular difference. The first direction D1, the second direction D2, and the third direction D3 are contained in the same plane. The plane containing the first direction D1, the second direction D2, and the third direction D3 is parallel to the image sensor 12. The fourth direction D4, which is orthogonal to the plane containing the first direction D1, the second direction D2, and the third direction D3, corresponds to the depth direction. When the stereo camera 10 is facing the subject, the fourth direction D4 is the direction from the stereo camera 10 toward the subject.

[0016] Lenses 11L and 11R are aligned along the first direction D1. When the stereo camera 10 is facing the subject, lens 11L is located to the left of lens 11R. Lens 11T is located closer to the second direction D2 than lenses 11L and 11R. Lenses 11L and 11T are aligned along the third direction D3. Lens 11T is located closer to the third direction D3 than lens 11L. The optical axis OX1 of lens 11L, the optical axis OX2 of lens 11R, and the optical axis OX3 of lens 11T are aligned along the fourth direction D4.

[0017] As shown in Figure 2, due to the relative positions of lenses 11L and 11T, the first image I1 and the second image I2 have a parallax in the first direction D1, as shown in Figure 3. Here, stereo matching is performed using the first image I1 and the second image I2 by the learning model 2 described later. In this stereo matching, the search direction d1 for searching for corresponding parts in the first image I1 and the second image I2 is along the first direction D1. Also, the first image I1 and the third image I3 have a parallax in the third direction D3. Stereo matching is performed using the first image I1 and the third image I3 by the learning model 2 described later. In this stereo matching, the search direction d2 for searching for corresponding parts in the first image I1 and the third image I3 is along the third direction D3.

[0018] Here, we assume that a bar 100, as shown in Figure 3, is projected onto the first image I1, the second image I2, and the third image I3. The bar 100 is parallel to the first direction D1. The bar 100 has few features that can be extracted as feature quantities. In this case, stereo matching using the first image I1 and the second image I2 may not detect the portion corresponding to the bar 100. Even in this case, in this embodiment, the portion corresponding to the bar 100 can be detected using stereo matching using the first image I1 and the third image I3. In other words, in this embodiment, even if a portion cannot be detected by stereo matching using the first image I1 and the second image I2 in the search direction d1, it can be detected by stereo matching using the first image I1 and the third image I3 in the search direction d2.

[0019] As shown in Figure 1, the image processing device 20 comprises an acquisition unit 21, a storage unit 22, and a control unit 23.

[0020] The acquisition unit 21 is the input interface for the image processing device 20. The acquisition unit 21 is capable of receiving image input from the stereo camera 10. The acquisition unit 21 may be configured to include a physical connector or a wireless communication module, etc.

[0021] The acquisition unit 21 receives data of the first image I1, the second image I2, and the third image I3 from the stereo camera 10. The acquisition unit 21 outputs the input data, such as the first image I1, to the control unit 23.

[0022] The storage unit 22 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The storage unit 22 may function as a main memory, an auxiliary memory, or a cache memory. The storage unit 22 stores data used for the operation of the image processing device 20 and data obtained by the operation of the image processing device 20. The storage unit 22 may also store a program executed by the control unit 23.

[0023] The control unit 23 is configured to include at least one processor. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The control unit 23 controls each part of the image processing device 20 and executes processes related to the operation of the image processing device 20.

[0024] The control unit 23 acquires the first image I1, the second image I2, and the third image I3 from the stereo camera 10 using the acquisition unit 21. The control unit 23 inputs the first image I1, the second image I2, and the third image I3 to the learning model 2 as shown in Figure 4. When these images are input to the learning model 2, the learning model 2 outputs a disparity map M1 and a disparity map M2 as shown in Figure 4. Disparity map M1 is a map that associates the disparity of corresponding parts in the first image I1 and the second image I2. Disparity map M2 is a map that associates the disparity of corresponding parts in the first image I1 and the third image I3.

[0025] [Structure of the learning model] The learning model 2 shown in Figure 4 is a model that uses CREStereo (Cascaded REcurrent Stereo matching network) as machine learning. For more information on CREStereo, please refer to Non-Patent Literature 3. Learning model 2 can perform machine learning using images created with computer graphics (CG) as training data. Learning model 2 is a pre-trained model. The first image I1, the second image I2, and the third image I3 are input to learning model 2.

[0026] The learning model 2 generates a first disparity map m1, which associates the disparity between the first image I1 and the second image I2, using the first RUM40_1, 40_2, etc., described later. In the generation of the first disparity map m1, the first image I1 becomes the reference image, and the second image I2 becomes the reference image.

[0027] The learning model 2 generates a second disparity map m2, which associates the disparity between the first image I1 and the third image I3, using the first RUM40_3, 40_4, etc., described later. In generating the second disparity map m2, the first image I1 becomes the reference image, and the third image I3 becomes the reference image.

[0028] The learning model 2 generates disparity maps M1 and M2 by updating the first disparity map m1 and the second disparity map m2 using the second RUM70 described later. The first disparity map m1 that has been finally updated by the second RUM70 described later becomes disparity map M1. In other words, the learning model 2 may use the first disparity map m1 that has been finally updated by the second RUM70 described later as the final first disparity map m1. Similarly, the second disparity map m2 that has been finally updated by the second RUM70 described later becomes disparity map M2. In other words, the learning model 2 may use the second disparity map m2 that has been finally updated by the second RUM70 described later as the final second disparity map m2.

[0029] Learning model 2 includes rotation units 3 and 4, feature extractors 31, 32, 33, and 34, first RUMs (Recurrent Update Modules) 40_1, 40_2, 40_3, and 40_4, and a second RUM 70.

[0030] Hereafter, unless otherwise specified, feature extractors 31-34 will be referred to as "feature extractor 30". Similarly, unless otherwise specified, first RUM40_1, 40_2, 40_3, and 40_4 will be referred to as "first RUM40".

[0031] The rotation unit 3 receives the first image I1 as input. The rotation unit 3 rotates the first image I1 by a predetermined angle θ around the optical axis OX1 of the lens 11L from the third direction D3 towards the first direction D1, as shown in Figure 3. The first image I1 that has been rotated by the predetermined angle θ from the third direction D3 towards the first direction D1 is labeled as "first image I1r". First image I1r can also be described as the first image I1 rotated by a predetermined angle θ from the third direction D3 towards the first direction D1. The rotation unit 3 outputs first image I1r to the feature extractor 33.

[0032] The rotation unit 4 receives the third image I3 as input. The rotation unit 4 rotates the third image I3 by a predetermined angle θ around the optical axis OX3 of the lens 11T from the third direction D3 toward the first direction D1, as shown in Figure 3. The third image I3, which has been rotated by the predetermined angle θ from the third direction D3 toward the first direction D1, is described as "third image I3r". The third image I3r can also be described as the third image I3 rotated by a predetermined angle θ from the third direction D3 toward the first direction D1. The rotation unit 4 outputs the third image I3r to the feature extractor 34.

[0033] The first image I1r and the third image I3r are used to generate the first disparity map m1. By rotating the first image I1 and the third image I3 by a predetermined angle θ from the third direction D3 toward the first direction D1 by the rotation units 3 and 4, the search direction d2 can be set to the horizontal direction (first direction D1) as shown in Figure 3 when generating the first disparity map m1.

[0034] Images are input to the feature extractor 30. Hereafter, images input to the feature extractor 30 will be referred to as "input images". The first image I1 is input to the feature extractor 31. The second image I2 is input to the feature extractor 32. The first image I1r is input to the feature extractor 33. The third image I3r is input to the feature extractor 34.

[0035] The feature extractor 30 generates a feature map by extracting features from the input image. The feature extractor 30 is composed of, for example, a CNN (Convolutional Neural Network). Feature extractors 31 to 34 share weights optimized by machine learning.

[0036] The feature extractor 30 generates three feature maps with different resolutions from a single input image. These three feature maps with different resolutions construct a three-level feature pyramid. In this embodiment, the feature extractor 30 generates three feature maps with different resolutions: one with a resolution of 1 / 4 of the input image's resolution, one with a resolution of 1 / 8 of the input image's resolution, and one with a resolution of 1 / 16 of the input image's resolution.

[0037] Hereafter, a feature map whose resolution is 1 / 4 of the input image's resolution will be described as a "1 / 4 resolution feature map." Similarly, a feature map whose resolution is 1 / 8 of the input image's resolution will be described as a "1 / 8 resolution feature map." Furthermore, a feature map whose resolution is 1 / 16 of the input image's resolution will be described as a "1 / 16 resolution feature map."

[0038] The feature extractor 30 is configured to include a self-attention layer. The self-attention layer may be provided by a transformer. The self-attention layer calculates self-attention information from the feature map with the lowest resolution among three feature maps, each with a different resolution. The self-attention layer adds the calculated self-attention information to the feature map with the lowest resolution. In this embodiment, the feature map with the lowest resolution is a feature map with 1 / 16 resolution. In other words, in this embodiment, the self-attention layer calculates self-attention information from the feature map with 1 / 16 resolution and adds the calculated self-attention information to the feature map with 1 / 16 resolution. For example, the self-attention layer converts the feature map with 1 / 16 resolution into a one-dimensional vector. The self-attention layer adds position encoding to the converted one-dimensional vector. The self-attention layer calculates self-attention information based on the converted one-dimensional vector to which position encoding has been added.

[0039] The feature extractor 31 generates feature maps I1A, I1B, and I1C with different resolutions by extracting features from the first image I1. Feature map I1A is a feature map with 1 / 16th the resolution. Feature map I1B is a feature map with 1 / 8th the resolution. Feature map I1C is a feature map with 1 / 4th the resolution. The feature extractor 31 calculates self-attention information from feature map I1A and adds the calculated self-attention information to feature map I1A.

[0040] The feature extractor 31 outputs feature map I1A to the first RUM40_1. The feature extractor 31 outputs feature map I1B to the first RUM40_2. The feature extractor 31 outputs feature map I1C to the second RUM70.

[0041] The feature extractor 32 generates feature maps I2A, I2B, and I2C with different resolutions by extracting features from the second image I2. Feature map I2A is a feature map with 1 / 16th the resolution. Feature map I2B is a feature map with 1 / 8th the resolution. Feature map I2C is a feature map with 1 / 4th the resolution. The feature extractor 32 calculates self-attention information from feature map I2A and adds the calculated self-attention information to feature map I2A.

[0042] The feature extractor 32 outputs feature map I2A to the first RUM40_1. The feature extractor 32 outputs feature map I2B to the first RUM40_2. The feature extractor 32 outputs feature map I2C to the second RUM70.

[0043] The feature extractor 33 generates feature maps I1rA, I1rB, and I1rC, each with a different resolution, by extracting features from the first image I1r. Feature map I1rA is a feature map with 1 / 16th the resolution. Feature map I1rB is a feature map with 1 / 8th the resolution. Feature map I1rC is a feature map with 1 / 4th the resolution. The feature extractor 33 calculates self-attention information from feature map I1rA and adds the calculated self-attention information to feature map I1rA.

[0044] The feature extractor 33 outputs feature map I1rA to the first RUM40_3. The feature extractor 33 outputs feature map I1rB to the first RUM40_4. The feature extractor 33 outputs feature map I1rC to the second RUM70.

[0045] The feature extractor 34 generates feature maps I3rA, I3rB, and I3rC, each with a different resolution, by extracting features from the third image I3r. Feature map I3rA is a feature map with 1 / 16th the resolution. Feature map I3rB is a feature map with 1 / 8th the resolution. Feature map I3rC is a feature map with 1 / 4th the resolution. The feature extractor 34 calculates self-attention information from feature map I3rA and adds the calculated self-attention information to feature map I3rA.

[0046] The feature extractor 34 outputs feature map I3rA to the first RUM40_3. The feature extractor 34 outputs feature map I3rB to the first RUM40_4. The feature extractor 34 outputs feature map I3C to the second RUM70.

[0047] The first RUM40 is a Recurrent Update Module. The configuration of the first RUM40 can adopt the RUM configuration described in Non-Patent Document 3.

[0048] The first RUM40 includes input terminals PI1, PI2, and PI3, and an output terminal PO1. The feature map is input to input terminal PI1 of the first RUM40 from the feature extractor 30. The feature map is input to input terminal PI2 of the first RUM40 from the feature extractor 30. The first updated disparity map, which will be described later, is input to input terminal PI3 of the first RUM40.

[0049] As shown in Figure 5 below, the feature map input from input terminal PI1 of the first RUM40 is referred to as the "first feature map." The feature map input from input terminal PI2 of the first RUM40 is referred to as the "second feature map."

[0050] The first RUM40 generates a second updated disparity map by repeatedly updating the first updated disparity map using the AGCL50 described later, based on the first and second feature maps. The second updated disparity map is output from the output terminal PO3 of the first RUM40.

[0051] As shown in Figure 4, the input terminal PI1 of the first RUM40_1 receives feature map I1A as the first feature map from the feature extractor 31. The input terminal PI1 of RUM40_2 receives feature map I1B as the first feature map from the feature extractor 31. The input terminal PI1 of RUM40_3 receives feature map I1rA as the first feature map from the feature extractor 33. The input terminal PI1 of RUM40_4 receives feature map I1rB as the first feature map from the feature extractor 34.

[0052] As shown in Figure 4, the input terminal PI2 of the first RUM40_1 receives feature map I2A as the second feature map from the feature extractor 32. The input terminal PI2 of RUM40_2 receives feature map I2B as the second feature map from the feature extractor 32. The input terminal PI2 of RUM40_3 receives feature map I3rA as the second feature map from the feature extractor 34. The input terminal PI2 of RUM40_4 receives feature map I3rB as the second feature map from the feature extractor 34.

[0053] As shown in Figure 4, the initial disparity map is input to the input terminal PI3 of the first RUM40_1 and 40_3 as the first updated disparity map. The disparity associated with the initial disparity map is zero. The second updated disparity map output from the output terminal PO1 of the first RUM40_1 is input to the input terminal PI3 of the first RUM40_2 as the first updated disparity map. The second updated disparity map output from the output terminal PO1 of the first RUM40_3 is input to the input terminal PI3 of the first RUM40_4 as the first updated disparity map.

[0054] The second updated disparity map output from output terminal PO1 of the first RUM40_1 is input to input terminal PI3 of the first RUM40_2 as the first updated disparity map. The second updated disparity map output from output terminal PO1 of the first RUM40_2 becomes the first disparity map m1. The first disparity map m1 is input to the second RUM70.

[0055] The second updated disparity map output from output terminal PO1 of the first RUM40_3 is input to input terminal PI3 of the first RUM40_4 as the first updated disparity map. The second updated disparity map output from output terminal PO1 of the first RUM40_4 becomes the second disparity map m2. The second disparity map m2 is input to the second RUM70.

[0056] As shown in Figure 5, the first RUM40 includes AGCL (Adaptive Group Correlation Layer) 50_1, ..., 50_i, ..., 50_n. The first RUM40 includes GRU (Gated Recurrent Unit) 60_1, ..., 60_i, ..., 60_n. The first RUM40 includes adders 61_1, ..., 61_i, ..., 61_n. i is an integer satisfying 1 ≤ i ≤ n. n is an integer satisfying 1 ≤ n.

[0057] In the following, unless otherwise specified, AGCL50_1 to 50_n will be referred to as "AGCL50". Unless otherwise specified, GRU60_1 to 60_n will be referred to as "GRU60". Unless otherwise specified, adders 61_1 to 61_n will be referred to as "adder 61".

[0058] AGCL50 is an Adaptive Group Correlation Layer. The configuration of AGCL50 can be the same as that of AGCL described in Non-Patent Document 3.

[0059] AGCL50 includes input terminals PJ1, PJ2, and PJ3, and output terminal PP1. The first feature map is input to input terminal PJ1 of AGCL50. The second feature map is input to input terminal PJ2 of AGCL50. The disparity map d from adder 61_i-1 is input to input terminal PJ3 of AGCL50_i. i The following is input. However, the first updated disparity map is input to AGCL50_1 (i=1) as disparity map d1 from input terminal PI3. The cost volume C_i, described later, is output from output terminal PP1 of AGCL50_i.

[0060] As shown in Figure 6, AGCL50 includes a cross-attention layer 51, grouping sections 52 and 53, a sampler 54, an adaptive local correlation layer 55, and a coupling section 56. However, only AGCL50 that receives a first feature map with 1 / 16 resolution and a second feature map with 1 / 16 resolution include the cross-attention layer 51. In other words, AGCL50 included in the first RUM40_1 and 40_3 include the cross-attention layer 51. AGCL50 included in the first RUM40_2 and 40_4 do not include the cross-attention layer 51.

[0061] The cross-attention layer 51 receives a first feature map with a resolution of 1 / 16 and a second feature map with a resolution of 1 / 16 as inputs. The cross-attention layer 51 may be provided by a Transformer.

[0062] The cross-attention layer 51 calculates cross-attention information to be assigned to the first feature map. The cross-attention layer 51 calculates the cross-attention information to be assigned to the first feature map by calculating the cre (Q) from the first feature map and the key (K) and value (V) from the second feature map. The cross-attention layer 51 assigns the calculated cross-attention information to the first feature map.

[0063] The cross-attention layer 51 calculates cross-attention information to be assigned to the second feature map. The cross-attention layer 51 calculates the cross-attention information to be assigned to the second feature map by calculating the cre (Q) from the second feature map and the key (K) and value (V) from the first feature map. The cross-attention layer 51 assigns the calculated cross-attention information to the second feature map.

[0064] As described above, the 1 / 16 resolution feature map is imbued with self-attention information by the feature extractor 30. This self-attention information, along with the cross-attention information imbued by the cross-attention layer 15 on the 1 / 16 resolution feature map, aggregates the contextual information of the entire image. This contextual information of the entire image includes, for example, information such as the shape, color, or background of objects in the entire image.

[0065] The cross-attention layer 51 outputs a first feature map with cross-attention information to the grouping unit 52. The cross-attention layer 51 outputs a second feature map with cross-attention information to the grouping unit 53.

[0066] If AGCL 50 includes a cross-attention layer 51, the grouping unit 52 receives a first feature map with cross-attention information added from the cross-attention layer 51. If AGCL 50 does not include a cross-attention layer 51, the grouping unit 52 receives a first feature map from input terminal PJ1.

[0067] The grouping unit 52 divides the first feature map into multiple groups, thereby generating multiple group-wise first feature maps F1. The grouping unit 52 outputs the multiple group-wise first feature maps F1 to the adaptive local correlation layer 55.

[0068] If AGCL 50 includes a cross-attention layer 51, the grouping unit 53 receives a second feature map with cross-attention information added from the cross-attention layer 51. If AGCL 50 does not include a cross-attention layer 51, the grouping unit 53 receives a second feature map from input terminal PJ2.

[0069] The grouping unit 53 divides the second feature map into multiple groups, thereby generating multiple group-wise second feature maps D2. The grouping unit 53 outputs the multiple group-wise second feature maps D2 to the sampler 54.

[0070] The sampler 54 receives a second feature map D2 for each of the multiple groups from the grouping unit 53. The sampler 54 also receives a disparity map d from the input terminal PJ3. i The following is input. Sampler 54 is parallax map d iThe coordinate grid is derived. The sampler 54 samples features from the coordinate grid derived from the second feature map D2. The sampler 54 generates the second feature map F2 by sampling features from the second feature map D2. The second feature map F2 can also be called the sampled second feature map. The sampler 54 outputs the second feature map F2 for each of the multiple groups to the adaptive local correlation layer 55.

[0071] The adaptive local correlation layer 55 receives a first feature map F1 for each of several groups from the grouping unit 52. The adaptive local correlation layer 55 also receives a second feature map F2 for each of several groups from the sampler 54.

[0072] The adaptive local correlation layer 55 calculates the local correlation of features between the first feature map F1 and the second feature map F2. The adaptive local correlation layer 55 calculates the local correlation of cost volumes H × W × D by setting a local search window. H is the height of the first feature map F1 and the second feature map F2. W is the width of the first feature map F1 and the second feature map F2. D is the number of correlation pairs for which a local correlation is calculated by the first feature map F1 and the second feature map F2.

[0073] The adaptive local correlation layer 55 calculates the local correlation of features between the first feature map F1 and the second feature map F2 by calculating Corr(x,y,d) using equations (1) and (2) while moving the position (x,y).

number

number

[0074] In equations (1) and (2), f(d) and g(d) are set depending on whether the adaptive local correlation layer 55 calculates local correlation in first-dimensional search mode (1D search mode) or second-dimensional search mode (2D search mode).

[0075] In one-dimensional search mode, g(d) is set to zero (g(d)=0). Also, f(d) is set to [-r,r], where r is, for example, 4 (r=4).

[0076] In 2D search mode, a k×k grid with an expansion rate similar to that of a dilated convolution layer is used. f(d) and g(d) are set appropriately to form a k×k grid.

[0077] Here, the one-dimensional search mode and the two-dimensional search mode should be executed alternately in AGCL50_1 to 50_n. For example, if the identifier i of AGCL50_i is odd (i=2j-1), the adaptive local correlation layer 55 of AGCL50_i calculates the local correlation in the first-dimensional search mode. j is an integer satisfying j≧1. Also, if the identifier i of AGCL50_i is even (i=2j), the adaptive local correlation layer 55 of AGCL50_i calculates the local correlation in the second-dimensional search mode.

[0078] The adaptive local correlation layer 55 calculates local correlations for multiple groups using a first feature map F1 for each of the multiple groups and a second feature map F2 for each of the multiple groups. The adaptive local correlation layer 55 outputs the local correlations for each of the multiple groups to the coupling unit 56.

[0079] The coupling unit 56 receives local correlations for multiple groups from the adaptive local correlation layer 55. The coupling unit 56 generates a cost volume of correlation between the first feature map and the second feature map by coupling the local correlations for multiple groups in the channel direction. The cost volume of correlation generated by the coupling unit 56 of AGCL50_i is denoted as "cost volume C_i". Cost volume C_i is output from the output terminal PP1 of AGCL50_i.

[0080] The GRU60 shown in Figure 5 is a gated regression unit. The GRU60 consists of a reset gate and an update gate.

[0081] The hidden state h_i-1 is input to GRU60_i from GRU60_i-1. Here, to GRU60_1 (i=1), a predetermined feature map generated from the first image I1 or the first image I1r is input as the hidden state h_0 of the initial value. The predetermined feature map may be one generated by the feature extractor 31 or the feature extractor 33, or may be one generated by another feature extractor. The predetermined feature map is obtained by dividing the number of channels of the feature map I1A or the like into halves. The disparity map d from the adder 61_i-1 is input to GRU60_i i . Here, to GRU60_1 (i=1), the first updated disparity map is input as the disparity map d1. The cost volume C_i is input to GRU60_i from AGCL50_i.

[0082] GRU60_i generates the input X_i by concatenating the cost volume C_i and the disparity map d i-1 in the channel direction. GRU60_i updates the hidden state h_i-1 to the hidden state h_i based on the input X(i) using the reset gate and the update gate. GRU60_i generates the update map Δd from the updated hidden state h_i i . The update map Δd i is an update amount for updating the disparity map d by the adder 61_i described later i . For example, GRU60_i generates the update map Δd by applying two convolution layers to the updated hidden state h_i i .

[0083] GRU60_i outputs the updated hidden state h_i to GRU60_i+1. GRU60_i outputs the update map Δd i to the adder 61_i+1.

[0084] The disparity map d is input to the adder 61_i from the adder 61_i-1 i . The update map Δd is input to the adder 61_i from GRU60_i i . The adder 61_i processes the disparity map d i and the update map Δdi By adding these together, the parallax map d i+1 It generates the disparity map d. Adder 61_i generates the disparity map d i+1 Output to AGCL50_i+1 and adder 61_i+1.

[0085] Disparity map d output from adder 61_n n+1 This becomes the second updated disparity map. The second updated disparity map output from adder 61_n of RUM40-2 is the first disparity map m1 as shown in Figure 4. The second updated disparity map output from adder 61_n of RUM40-4 is the second disparity map m2 as shown in Figure 4.

[0086] As shown in Figure 4, the second RUM70 receives the feature map I1C from the feature extractor 31. The second RUM70 receives the feature map I2C from the feature extractor 32. The second RUM70 receives the feature map I1rC from the feature extractor 33. The second RUM70 receives the feature map I3rC from the feature extractor 34. The second RUM70 receives the first disparity map m1 from the first RUM40-2. The second RUM70 receives the second disparity map m2 from the first RUM40-4.

[0087] As shown in Figure 7, the second RUM70 includes rotating parts 80_1, ..., 80_2p-1, 80_2p, ..., 80_q. The second RUM70 includes rotating parts 81_1, ..., 81_2p-1, 81_2p, ..., 81_q. The second RUM70 includes AGCL150_1, ..., 150_2p-1, 150_2p, ..., 150_q. The second RUM70 includes GRU160_1, ..., 160_2p-1, 160_2p, ..., 160_q. The second RUM70 includes adders 161_1, ..., 161_2p-1, 161_2p, ..., 161_q. The second RUM70 includes adders 162_1, ..., 162_2p-1, 162_2p, ..., 162_q. q is an integer satisfying q = 2N-1 or q = 2N. N is an integer satisfying 1 ≤ N. p is an integer satisfying 1 ≤ p ≤ N. Here, q may be even; that is, q may be 2N.

[0088] Hereafter, unless otherwise specified, the rotating parts 80_1 to 80_q will be referred to as "rotating part 80". Similarly, unless otherwise specified, the rotating parts 81_1 to 81_q will be referred to as "rotating part 81".

[0089] Hereafter, unless otherwise specified, AGCL150_1 to AGCL150_q will be referred to as "AGCL50". In other words, the configuration of AGCL150_1 to AGCL150_q is the same as the configuration of AGCL50 shown in Figure 6. Unless otherwise specified, GRU160_1 to 160_q will be referred to as "GRU160".

[0090] Hereafter, unless specifically distinguished, adders 161_1 to 161_q will be referred to as "adder 161". Similarly, unless specifically distinguished, adders 162_1 to 162_q will be referred to as "adder 162".

[0091] The second RUM70 updates the first disparity map m1 and the second disparity map m by alternately executing the first and second processes.

[0092] In the first processing step, the second RUM70 uses the feature map I1C of the first image I1r and the feature map I3rC of the third image I3r. In the first processing step, the second RUM70 updates the first disparity map m1 and the second disparity map m2, respectively, using the feature map I1C of the first image I1r and the feature map I3rC of the third image I3r. In this embodiment, the first processing step is implemented by AGCL150_2p-1, GRU160_2p-1, adder 161_2p-1, and adder 162_2p-1.

[0093] In the second processing step, the second RUM70 uses the feature map I1C of the first image I1 and the feature map I2C of the second image I2. In the second processing step, the second RUM70 updates the first disparity map m1 and the second disparity map m2, respectively, using the feature map I1C of the first image I1 and the feature map I2C of the second image I2. In this embodiment, the first processing step is implemented by AGCL150_2p, GRU160_2p, adder 161_2p, and adder 162_2p.

[0094] In the second RUM70, the first stage of the alternating first and second processes may be designated as the first process. In this embodiment, the second RUM70 executes a first process as the first stage, which includes AGCL150_1, GRU160_1, adder 161_1, and adder 162_1.

[0095] The second RUM70 may generate disparity maps M1 and M2 by repeatedly executing the first and second processes alternately a predetermined number of times. In this case, q is an even number (q=2N). In this embodiment, the predetermined number of times is q / 2 (=N). The predetermined number of times may be set according to the disparity estimation accuracy obtained in the imaging system 1.

[0096] The rotation unit 80_2p-1 receives the disparity map dm1_2p-2, described later, from the adder 161_2p-2. However, the rotation unit 80_1 (p=1) receives the first disparity map m1 as disparity map dm1_0. The rotation unit 80_2p-1 rotates the disparity map dm1_2p-2 by a predetermined angle θ from the first direction D1 to the third direction D3 as shown in Figure 3. The rotation unit 80_2p-1 outputs the rotated disparity map dm1_2p-2 to GRU160_2p-1.

[0097] The rotation unit 80_2p receives the disparity map dm2_2p-1 from the adder 161_2p-1. The rotation unit 80_2p rotates the disparity map dm2_2p-1 by a predetermined angle θ from the third direction D3 to the first direction D1 as shown in Figure 3. The rotation unit 80_2p outputs the rotated disparity map dm2_2p-1 to the GRU 160_2p.

[0098] The rotating section 81_2p-1 receives the updated map Δdd from GRU160_2p-1. 2p-1 The following is input. The rotating part 81_2p-1 updates the map Δdd 2p-1 The rotation is performed by a predetermined angle θ from the third direction D3 to the first direction D1 as shown in Figure 3. The rotating part 81_2p-1 updates the map Δdd after rotation. 2p-1 Output this to adder 162_2p-1.

[0099] The rotating section 81_2p receives the updated map Δdd from GRU160_2p. 2p The input is entered. The rotating part 81_2p updates the map Δdd 2p The rotating part 81_2p rotates the map Δdd after rotation. 2p Output this to adder 162_2p.

[0100] The feature map I1rC is input to input terminal PJ1 of AGCL150_2p-1. The feature map I3rC is input to input terminal PJ2 of AGCL150_2p-1. The disparity map d2m_2p-2 is input to input terminal PJ3 of AGCL150_2p-1 from adder 162_2p-2. However, the second disparity map m2 is input to input terminal PJ3 of AGCL150_1 (p=1). Here, the configuration of AGCL150_2p-1 is the same as AGCL50 shown in Figure 6, as described above. However, AGCL150_2p-1 does not include the cross attention layer 51.

[0101] AGCL150_2p-1 generates a cost volume of the correlation between feature map I3rC and feature map I1rC using feature map I1rC, feature map I3rC, and disparity map d2m_2p-2, in the same or similar configuration as shown in Figure 6. Hereinafter, the cost volume generated by AGCL150_2p-1 will be referred to as "cost volume C_2p-1". AGCL150_2p-1 outputs cost volume C_2p-1 to GRU160_2p-1 via output terminal PP1.

[0102] The feature map I1C is input to input terminal PJ1 of AGCL150_2p. The feature map I2C is input to input terminal PJ2 of AGCL150_2p. The disparity map d2m_2p-1 is input to input terminal PJ3 of AGCL150_2p from adder 162_2p-1. Here, the configuration of AGCL150_2p is the same as that of AGCL50 shown in Figure 6, as described above. However, AGCL150_2p-1 does not include the cross attention layer 51.

[0103] AGCL150_2p generates a cost volume of the correlation between feature map I1C and feature map I2C using feature map I1C, feature map I2C, and disparity map d2m_2p-1, in the same or similar configuration as shown in Figure 6. Hereinafter, the cost volume generated by AGCL150_2p will be referred to as "cost volume C_2p". AGCL150_2p outputs cost volume C_2p to GRU160_2p via output terminal PP1.

[0104] The GRU160 is a gated regression unit. The GRU160 consists of a reset gate and an update gate.

[0105] GRU160_2p-1 receives the hidden state hh_2(p-1)-1 from GRU160_2(p-1)-1. However, GRU160_1(p=1) receives a predetermined feature map generated from the first image I1 or first image I1r as the initial hidden state hh_-1. The predetermined feature map may be generated by feature extractor 31 or feature extractor 33, or by another feature extractor. The predetermined feature map is obtained by dividing the number of channels of feature map I1A, etc., in half. GRU160_2p-1 receives the disparity map dm2_2p-2 from adder 162_2p-2. However, GRU160_1(p=1) receives the second disparity map m2 as the disparity map dm2_-2. The rotational parallax map dm1_2p-2 is input to GRU160_2p-1 from the rotating unit 80_2p-1. The cost volume C_2p-1 is input to GRU160_2p-1 from AGCL150_2p-1.

[0106] GRU160_2p-1 generates input X_2p-1 by connecting cost volume C_2p-1, disparity map dm2_2p-2, and rotated disparity map dm1_2p-2 in the channel direction. GRU160_2p-1 updates the hidden state hh_2(p-1)-1 to the hidden state hh_2p-1 based on input X_2p-1 using a reset gate and an update gate. GRU160_2p-1 then updates the map Δdd from the updated hidden state hh_2p-1. 2p-1 Generates the updated map Δdd. 2p-1 This is the update amount used to update the disparity maps dm1_2p-2 and dm2_2p-2 by adders 162_2p-1 and 161_2p-1, respectively. For example, GRU160_2p-1 updates the map Δdd by applying two convolutional layers to the hidden state hh_2p-1. 2p-1 Generates.

[0107] GRU160_2p-1 outputs the updated hidden state hh_2p-1 to GRU160_2(p+1)-1. GRU160_2p-1 outputs the updated map Δdd 2p-1The output is sent to adder 161_2p-1 and adder 162_2p-1.

[0108] GRU160_2p receives the hidden state hh_2(p-1) from GRU160_2(p-1). However, GRU160_2(p=1) receives a predetermined feature map generated from the first image I1 or first image I1r as the initial hidden state hh_0. The predetermined feature map may be created by feature extractor 31 or feature extractor 33, or it may be generated by another feature extractor. The predetermined feature map is obtained by dividing the number of channels of feature map I1A, etc., in half. GRU160_2p receives the disparity map dm1_2p-1 from adder 162_2p-1. GRU160_2p receives the disparity map dm2_2p-1 after rotation from rotation unit 80_2p. GRU160_2p receives the cost volume C_2p from AGCL150_2p.

[0109] GRU160_2p generates input X_2p by connecting cost volume C_2p, disparity map dm1_2p-1, and rotated disparity map dm2_2p-1 in the channel direction. GRU160_2p updates the hidden state hh_2(p-1) to the hidden state hh_2p based on input X_2p-1 using a reset gate and an update gate. GRU160_2p then updates the map Δdd from the updated hidden state hh_2p. 2p Generates the updated map Δdd. 2p This is the update amount for updating the adders 161_2p and 162_2, and the disparity maps dm1_2p-1 and dm2_2p-1. For example, GRU160_2p updates the map Δdd by applying two convolutional layers to the updated hidden state hh_2p. 2p Generates.

[0110] GRU160_2p outputs the updated hidden state hh_2p to GRU160_2(p+1). GRU160_2p outputs the updated map Δdd 2p The output is sent to adders 161_2p and 162_2p.

[0111] Adder 161_2p-1 receives the disparity map dm2_2p-2 from adder 162_2p-2. However, adder 161_1 (p=1) receives the second disparity map m2 as disparity map dm2_0. Adder 161_2p-1 receives the updated map Δdd from GRU160_2p-1. 2p―1 The following is input. Adder 161_2p-1 receives the disparity map dm2_2p-2 and the update map Δdd. 2p―1 By adding these two values, a disparity map dm2_2p-1 is generated. Adder 161_2p-1 outputs the generated disparity map dm2_2p-1 to adder 162_2p.

[0112] Adder 161_2p receives the disparity map dm1_2p-1 from adder 162_2p-1. Adder 161_2p receives the update map Δdd from GRU160_2p. 2p The following is input. Adder 161_2p receives the disparity map dm1_2p-1 and the update map Δdd. 2p By adding these two values, a disparity map dm1_2p is generated. Adder 161_2p outputs the generated disparity map dm1_2p to adder 162_2p+1.

[0113] The disparity map dm1_q generated by adder 161_q becomes the disparity map M1.

[0114] Adder 162_2p-1 receives the disparity map dm1_2p-2 from adder 161_2p-2. However, adder 162_1 (p=1) receives the first disparity map m1 as disparity map dm1_0. Adder 162_2p-1 receives the updated map Δdd after rotation from the rotation unit 81_2p-1. 2p-1 The following is input. Adder 162_2p-1 receives the disparity map dm1_2p-2 and the update map Δdd. 2p-1 By adding these two values, a disparity map dm1_2p-1 is generated. Adder 162_2p-1 outputs the generated disparity map dm1_2p-1 to adder 161_2p.

[0115] The adder 162_2p receives the disparity map dm2_2p-1 from adder 161_2p-1. The adder 162_2p receives the updated map Δdd after rotation from the rotation unit 81_2p. 2p The following is input. Adder 162_2p uses the disparity map dm2_2p-1 and the update map Δdd. 2p By adding these two values, a disparity map dm2_2p is generated. Adder 162_2p outputs the generated disparity map dm2_2p to adder 161_2p+1.

[0116] The disparity map dm2_q generated by adder 162_q becomes the disparity map M2.

[0117] (Operation of imaging system 1) Figure 8 is a flowchart showing an example of the operation performed by the image processing device 20 shown in Figure 1.

[0118] In step S1, the control unit 23 acquires the first image I1, the second image I2, and the third image I3 from the stereo camera 10 using the acquisition unit 21. The control unit 23 inputs the first image I1, the second image I2, and the third image I3 to the learning model 2.

[0119] In step S2, feature extractor 31 generates feature maps I1A, I1B, and I1C from the first image I1. Feature extractor 32 generates feature maps I2A, I2B, and I2C from the second image I2. Feature extractor 33 generates feature maps I1rA, I1rB, and I1rC from the rotated first image I1, i.e., the first image I1r. Feature extractor 34 generates feature maps I3rA, I3rB, and I3rC from the rotated third image I3, i.e., the third image I3r.

[0120] In step S3, the first RUM40-1 and 40-2 generate the first disparity map m1 using the feature maps I1A and I1B of the first image I1 and the feature maps I2A and I2B of the second image I2. The first RUM40-3 and 40-4 generate the second disparity map m2 using the feature maps I1rA and I1rB of the first image I1r and the feature maps I3rA and I3rB of the third image I3r.

[0121] In step S4, the second RUM70 generates the final first disparity map m1, i.e., disparity map M1, and the final second disparity map m2, i.e., disparity map M2, by repeatedly performing the first and second processes alternately a predetermined number of times.

[0122] <Experiment> The inventors conducted experiments to demonstrate the effectiveness of the method described herein. The experiments will be described with reference to Figures 9 and 10.

[0123] In the experiment, parallax maps were generated using three patterns: "CG horizontal bar, thick bar, no tilt," "CG horizontal bar, thin bar, no tilt," and "live-action horizontal bar."

[0124] In "CG Horizontal Bar, Thick Bar with No Inclination," the first, second, and third images, created using computer graphics, were used. These images show a thick bar positioned horizontally. Figure 9 shows only the first image. The diameter of the thick bar is 2.9 mm.

[0125] In "CG Horizontal Bar, Thin Bar with No Inclination," the first, second, and third images, created using computer graphics, were used. These images show a thin bar positioned horizontally. Figure 9 shows only the first image. The diameter of the thin bar is 0.5 mm.

[0126] In the "Live-Action Horizontal Bar" section, the first, second, and third images, created through live-action photography, were used. These images show a horizontally positioned bar. Figure 9 shows only the first image.

[0127] In the experiment, disparity maps were generated using the method disclosed herein and CREstereo. Figure 9 shows the disparity map generated by CREstereo, the disparity map generated by the method disclosed herein, and the ground truth disparity map. The disparity map shown in Figure 9 corresponds to disparity map M1. From Figure 9, it can be seen that the disparity map generated by the method disclosed herein is closer to the ground truth disparity map than the disparity map generated by CREstereo.

[0128] Figure 10 shows the evaluation accuracy (EPE). Figure 10 shows the error from the correct answer. The unit of the numerical values ​​in Figure 10 is pixels. The inventors used a CG dataset and the publicly available dataset from "Vision Multiscopic" (https: / / sites.google.com / view / multiscopic). The CG dataset includes "thick bar, no tilt," "thick bar, tilted," "thin bar, no tilt," and "thin bar, tilted." "Thick bar, no tilt" is a state where the thick bar is placed horizontally. "Thick bar, tilted" is a state where the thick bar is placed tilted at 54.4 degrees to the horizontal. The diameter of the thick bar is 2.9 mm. "Thin bar, no tilt" is a state where the thin bar is placed horizontally. "Thin bar, tilted" is a state where the thin bar is placed tilted at 43.1 degrees to the horizontal. The diameter of the thin bar is 0.5 mm. In Figure 10, smaller numerical values ​​indicate higher accuracy. The method disclosed herein is more accurate than CREstereo.

[0129] In this embodiment, the learning model 2 receives the first image I1, the second image I2, and the third image I3 as input. The learning model 2 generates a first disparity map m1, which associates the disparity between the first image I1 and the second image I2. For example, the first RUM40-1,40-2 generates the first disparity map m1 using the feature maps 1IA,I1B of the first image I1 and the feature maps I2A,I2B of the second image I2. The learning model 2 also generates a second disparity map m2, which associates the disparity between the first image I1 and the third image I3. For example, the first RUM40-3,40-4 generates the second disparity map m2 using the feature maps Ir1A,Ir1B of the first image Ir1 and the feature maps Ir3A,Ir3B of the third image Ir3. Furthermore, the learning model 2 updates the first disparity map m1 and the second disparity map m2 by alternately executing the first and second processes. By updating the first and second disparity maps m1 and m2, the finally updated first disparity map m1, i.e., disparity map M1, and the finally updated second disparity map m2, i.e., disparity map M2, are generated. With this configuration, disparity maps M1 and M2 can be generated even when the first image I1, the second image I2, and the third image I3 are in the positional relationship shown in Figure 3. In other words, in this embodiment, a disparity map M1 obtained by stereo matching in the search direction d1 and a disparity map M2 obtained by stereo matching in the search direction d2 are generated as shown in Figure 3.

[0130] Here, the method described in Non-Patent Document 1 corresponds to multiple images using a quad-pixel sensor. The method described in Non-Patent Document 2 corresponds to three images generated by three cameras: left, center, and right. Non-Patent Documents 1 and 2 do not correspond to the first image I1, second image I2, and third image I3, which are in the positional relationship shown in Figure 3. Therefore, the methods described in Non-Patent Documents 1 and 2 cannot generate the disparity map M1 by stereo matching in the search direction d1 and the disparity map M2 by stereo matching in the search direction d2, as shown in Figure 3.

[0131] In contrast to Non-Patent Documents 1 and 2, the learning model 2 according to this embodiment can generate disparity maps M1 and M2. These disparity maps M1 and M2 enable accurate estimation of disparity even when the same pattern is repeated in the first direction D1, i.e., horizontally, in a stereo image. Furthermore, these disparity maps M1 and M2 enable accurate estimation of disparity even when a rod 100 without features parallel to the first direction D1, as shown in Figure 3, is captured in the first image I1, etc.

[0132] Furthermore, in the learning model 2 according to this embodiment, in the first process, the first disparity map m1 and the second disparity map m2 may be updated using the feature map I1rC of the first image I1r and the feature map I3rC of the third image I3r, respectively. For example, as described above, the first process is implemented by AGCL150_2p-1, GRU160_2p-1, adder 161_2p-1, and adder 162_2p-1. In this case, AGCL150_2p-1 of the first process generates a cost volume C_2p-1 based on the feature map I1rC of the first image I1r and the feature map I3rC of the third image I3r. GRU160_2p-1 of the first process updates the map Δdd based on the cost volume C_2p-1 and the disparity map dm1_2p-2. 2p-1 The first process adds the disparity map dm2_2p-2 and the update map Δdd. 2p―1 By adding these two values, a disparity map dm2_2p-1 is generated. The generated disparity map dm2_2p-1 becomes the second disparity map m2, which has been updated by the first process. In addition, the adder 162_2p-1 of the first process adds the disparity map dm1_2p-2 and the updated map Δdd. 2p-1 By adding these together, a disparity map dm1_2p-1 is generated. The generated disparity map dm1_2p-1 becomes the first disparity map m1 updated by the first process. In this way, by using the feature map I1rC of the first image I1r and the feature map I3rC of the third image I3r in the first process, the first disparity map m1 and the second disparity map m2 can be updated with greater accuracy.

[0133] Furthermore, in the second processing step of the learning model 2 according to this embodiment, the first disparity map m1 and the second disparity map m2 may be updated using the feature map I1C of the first image I1 and the feature map I2C of the second image I2, respectively. For example, as described above, the second processing step is implemented by AGCL150_2p, GRU160_2p, adder 161_2p, and adder 162_2p. In this case, AGCL150_2p in the second processing step generates a cost volume C_2p based on the feature map I1C of the first image I1 and the feature map I2C of the second image I2. GRU160_2p in the second processing step updates the map Δdd based on the cost volume C_2p and the disparity map dm2_2p-1. 2p The second process adds the disparity map dm1_2p-1 and the update map Δdd. 2p By adding these two values, a disparity map dm1_2p is generated. The generated disparity map dm1_2p becomes the first disparity map m1, which has been updated by the second process. In addition, the adder 162_2p of the second process adds the disparity map dm2_2p-1 and the updated map Δdd. 2p By adding these two values, a disparity map dm2_2p is generated. The generated disparity map dm2_2p becomes the second disparity map m2, which has been updated by the second process. In this way, by using the feature map I1C of the first image I1 and the feature map I2C of the second image I2 in the second process, the first disparity map m1 and the second disparity map m2 can be updated with greater accuracy.

[0134] Furthermore, the learning model 2 according to this embodiment may perform a first process in the first stage and a second process in the final stage. The learning model 2 may use the first disparity map m1 updated in the second process of the final stage as the final first disparity map m1, i.e., disparity map M1. The learning model 2 may also use the second disparity map m2 updated in the second process of the final stage as the final second disparity map m2, i.e., disparity map M2. When the learning model 2 performs a first process in the first stage and a second process in the final stage, q may be an even number. For example, the second process of the final stage includes adders 161_q and 162_q. In this case, the disparity map dm1_q generated by adder 161_q becomes the final first disparity map m1, i.e., disparity map M1. Also, the disparity map dm2_q generated by adder 162_q becomes the final second disparity map m2, i.e., disparity map M2. Here, rotating the image may introduce errors into the rotated image. In the second process, the feature map I1C of the unrotated first image I1 and the feature map I2C of the unrotated second image I2 are used. Therefore, by executing the second process in the final stage to generate disparity maps M1 and M2, it is possible to generate disparity maps M1 and M2 with high accuracy.

[0135] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated.

[0136] For example, an embodiment is also possible in which a general-purpose computer functions as the image processing apparatus 20 according to the above embodiment. Specifically, a program describing the processing content that realizes each function of the image processing apparatus 20 according to the above embodiment is stored in the memory of the general-purpose computer, and the processor reads and executes the program. Therefore, this disclosure can also be realized as a program that can be executed by a processor, or as a non-temporary computer-readable medium that stores the program.

[0137] In one embodiment, (1) the learning model is A learning model in which the first image, the second image, and the third image are input, The first image and the second image have a parallax in the first direction. The first image and the third image have parallax in a third direction that is tilted at a predetermined angle from the first direction toward the second direction, and the second direction is perpendicular to the first direction. The aforementioned learning model, A first disparity map is generated in which the disparity between the first image and the second image is associated. A second disparity map is generated, which associates the disparity between the first image and the third image. The first parallax map and the second parallax map are updated by alternately performing a first process using the feature map of the first image and the feature map of the third image, and a second process using the feature map of the first image and the feature map of the second image.

[0138] (2) In the learning model described in (1) above, The aforementioned learning model, The second disparity map is generated using the first image rotated by a predetermined angle from the third direction toward the first direction, and the third image rotated by a predetermined angle from the third direction toward the first direction.

[0139] (3) In the learning model described in (1) or (2) above, The learning model repeatedly executes the first process and the second process alternately a predetermined number of times.

[0140] (4) In any one of the learning models described in (1) to (3) above, The learning model performs the first process in the first stage and the second process in the final stage, using the first disparity map updated in the second process of the final stage as the final first disparity map, and using the second disparity map updated in the second process of the final stage as the final second disparity map.

[0141] (5) In any one of the learning models described in (1) to (4) above, The learning model, in the first process, The first parallax map and the second parallax map are updated using the feature map of the first image and the feature map of the third image, respectively.

[0142] (6) In any one of the learning models described in (1) to (5) above, In the second process, the learning model described above, The first disparity map and the second disparity map are updated using the feature map of the first image and the feature map of the second image, respectively.

[0143] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, RUM 1 can swap the identifiers "First" and "Second" with RUM 2. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers. [Explanation of Symbols]

[0144] 1: Imaging system, 2: Learning model, 3,4: Rotating unit, 10: Stereo camera, 10L: First camera, 10R: Second camera, 10T: Third camera, 11L,11R,11T: Lens, 12: Image sensor, 20: Image processing unit, 21: Acquisition unit, 22: Memory unit, 23: Control unit, 30,31~34: Feature extractor, 40,40_1~40_4: First RUM, 50,50_1~50_n,150_1~150_q: AGCL, 51: Cross attention layer, 52,53: G Looping section, 54: Sampler, 55: Adaptive local correlation layer, 56: Coupling section, 60, 60_1~60_n, 160, 160_1~160_q: GRU, 61, 61_1~61_n, 161, 161_1~161_q, 162, 162_1~162_q: Adder, 70: Second RUM, 80, 80_1~80_n: Rotation section, 81, 81_1~81_n: Rotation section, I1: First image, I2: Second image, I3: Third image, I1C, I2C, I1rC, I3rC: Feature map

Claims

1. A learning model in which the first image, the second image, and the third image are input, The first image and the second image have a parallax in the first direction. The first image and the third image have parallax in the third direction which is tilted at a predetermined angle from the first direction toward the second direction, and the second direction is perpendicular to the first direction. The aforementioned learning model, A first disparity map is generated in which the disparity between the first image and the second image is associated. A second disparity map is generated, which associates the disparity between the first image and the third image. A learning model that updates the first and second disparity maps by alternately performing a first process using the feature map of the first image and the feature map of the third image, and a second process using the feature map of the first image and the feature map of the second image.

2. The aforementioned learning model, The learning model according to claim 1, wherein the second disparity map is generated using an image obtained by rotating the first image from the third direction toward the first direction by a predetermined angle, and an image obtained by rotating the third image from the third direction toward the first direction by a predetermined angle.

3. The learning model according to claim 1, wherein the first process and the second process are repeatedly executed alternately a predetermined number of times.

4. The learning model according to any one of claims 1 to 3, wherein the learning model performs the first process in the first stage, performs the second process in the final stage, uses the first disparity map updated in the second process of the final stage as the final first disparity map, and uses the second disparity map updated in the second process of the final stage as the final second disparity map.

5. In the first processing, the learning model, The learning model according to claim 1, wherein the first disparity map and the second disparity map are updated using the feature map of the first image and the feature map of the third image, respectively.

6. In the second process, the learning model is configured as follows: The learning model according to claim 1, wherein the first disparity map and the second disparity map are updated using the feature map of the first image and the feature map of the second image, respectively.

Images