A method for producing trinocular stereo matching dataset based on view synthesis

Abstract

The present invention discloses a method for preparing a three-eye stereo matching data set based on perspective synthesis, including obtaining a left-eye image and a right-eye image, obtaining a disparity true value matrix from each pixel in the left-eye image to the right-eye image, and calculating the theoretical disparity from the pixel in the left-eye image to the third-eye image, and then calculating the sub-pixel point of the pixel in the left-eye image in the third-eye image; performing a splash transformation based on disparity comparison on the sub-pixel point to obtain the non-normalized contribution of the sub-pixel point to the integer pixel point; performing weight normalization and summing on the non-normalized contribution mapped to the same integer pixel point to complete the synthesis of the third-eye image. The present invention is applied to the field of visual processing, and performs third-eye synthesis by using a one-dimensional Softmax splash transformation based on disparity comparison, quickly generating a large number of TSM data sets, solving the many-to-one mapping conflict problem, thereby obtaining a good synthesis effect for the occluded area, and also achieving a dense synthesis effect.

Description

Technical Field

[0001] The present invention relates to the field of visual processing technology, and in particular to a method for producing a trinocular stereo matching data set based on viewpoint synthesis. Background Art

[0002] Binocular stereo matching (BSM) aims to estimate the pixel-by-pixel motion between two camera images. In the standard setting, the left and right images are stereo-rectified, so that the motion of each pixel is constrained to lie on a horizontal line. Recently, as with many vision tasks, deep learning has revitalized the field of BSM, shifting research away from traditional methods and towards deep learning approaches. However, the current BSM field faces challenges such as high computational cost and poor estimation accuracy in non-co-viewing areas. One solution is to add a third camera to the binocular system, creating a trinocular stereo matching system. The trinocular setup provides more references for the area to be matched, effectively alleviating the matching estimation problem in non-co-viewing areas.

[0003] Triocular stereo matching (TSM), a special case of multi-baseline stereo matching, was first proposed in 1993. A triocular hardware system with an additional vertically mounted camera was constructed. Its algorithmic principle is an extension of semiglobal matching (SGM) in the triocular setting. Specifically, after separately calculating the matching costs for left and right image pairs and bottom and top image pairs, they are merged using cubic spline interpolation for cost aggregation and disparity estimation. A wide-plus-narrow multi-baseline stereo framework has been introduced, enabling accurate 3D scene reconstruction within a few meters of the vehicle. A special triocular dynamic disparity range (T-DDR) scheme has been designed in the triocular setting to accelerate the stereo matching algorithm. All of these methods are based on traditional binocular stereo matching methods. Currently, the only deep learning approach is the GwcNet-based Tristeronet, which sums aligned 3D cost tensors to predict the disparity map corresponding to the wide baseline.

[0004] Currently, TSM has not been fully studied. One important reason is that it does not have a rich and large dataset resource like BSM. There are few existing TSM datasets. Middlebury2003 is a TSM dataset with practical significance and challenges produced using structured light technology. The dataset consists of 9 rectified views of equidistant points on a straight line, of which only the view Figure 2 Heshi Figure 6There are two real scenes. Middlebury2005 was created using the same technique as Middlebury2003, but provides more samples. The dataset consists of 7 rectified views of equidistant points on a straight line, of which only the view Figure 1 Heshi Figure 5 There are nine real-world scenes, three of which do not provide ground truth. Middlebury2006, released immediately after Middlebury2005, contains 21 real-world scenes. These TSM datasets were collected using a physical trinocular system built in real-world scenes. This requires costly ground truth calculation and calibration processes, resulting in a very limited amount of data. This inadequate data support for deep learning methods has limited the research and application of trinocular stereo matching algorithms. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the existing technology, the present invention provides a method for producing a trinocular stereo matching dataset based on view synthesis, which can quickly generate a large number of TSM datasets at a low cost and can effectively solve the bottleneck problem of the small scale of the current trinocular stereo matching dataset.

[0006] To achieve the above object, the present invention provides a method for producing a trinocular stereo matching dataset based on view synthesis, comprising the following steps:

[0007] Step 1: Obtain an existing binocular stereo matching dataset;

[0008] Step 2: obtaining a set of corresponding left-eye images and right-eye images from the binocular stereo matching dataset, and obtaining a true disparity matrix from each pixel in the left-eye image to the right-eye image;

[0009] Step 3: Based on the disparity true value matrix, calculate the theoretical disparity from each pixel in the left image to the third image;

[0010] Step 4: Based on the theoretical parallax, calculate the sub-pixel point corresponding to each pixel point in the left image in the third image;

[0011] Step 5: performing a splash transform on each of the sub-pixel points in the third image to calculate a non-normalized contribution of each sub-pixel point to each integer pixel point in its left and right neighborhoods;

[0012] Step 6: In the third image, weight normalize and sum all non-normalized contributions mapped to the same integer pixel to obtain the pixel value of each integer pixel in the third image, thereby completing the synthesis of the third image;

[0013] Step 7: Repeat steps 2 to 6 until the binocular stereo matching dataset is traversed, thereby obtaining a trinocular stereo matching dataset.

[0014] Compared with the prior art, the present invention has the following beneficial technical effects:

[0015] 1. Based on the existing binocular stereo matching dataset, this paper uses a one-dimensional Softmax splash transform based on disparity comparison to perform mapping operations, and performs fast and high-quality third-eye synthesis on the existing binocular stereo matching dataset. This can quickly generate a large number of TSM datasets at a low cost.

[0016] 2. In the one-dimensional Softmax splash transformation process based on disparity comparison, the present invention can effectively solve the many-to-one mapping conflict problem when mapping the pixels in the left-eye image to the third-eye image, thereby obtaining a good synthesis effect for the occluded area. At the same time, since the splashing direction is the left and right directions, it can also achieve a dense synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0018] Figure 1 This is a flow chart of a method for producing a trinocular stereo matching dataset based on view synthesis in an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of the one-dimensional Softmax splash transformation principle based on disparity comparison in an embodiment of the present invention;

[0020] Figure 3 Schematic diagram of weight normalization in an embodiment of the present invention;

[0021] Figure 4 Schematic diagrams of the synthesis effect of the method for producing a trinocular stereo matching dataset based on view synthesis in an embodiment of the present invention, wherein: (a) is a schematic diagram of the synthesis effect of the first scene, (b) is a schematic diagram of the synthesis effect of the second scene, (c) is a schematic diagram of the synthesis effect of the third scene, (d) is a schematic diagram of the synthesis effect of the fourth scene, and (e) is a schematic diagram of the synthesis effect of the fifth scene;

[0022] Figure 5Figure 1 shows a comparison of the synthesis effects of the trinocular stereo matching dataset production method based on viewpoint synthesis in an embodiment of the present invention and two other existing synthesis methods, where (a) shows the synthesis effect comparison of the first scene, and (b) shows the synthesis effect comparison of the second scene.

[0023] Figure 6 These are diagrams showing the synthesis effects of the trinocular stereo matching dataset production method based on view synthesis under different baseline ratio conditions in an embodiment of the present invention, where: (a) is a schematic diagram of the synthesis effect of the first scene, and (b) is a schematic diagram of the synthesis effect of the second scene.

[0024] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0027] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0028] This embodiment discloses a method for producing a trinocular stereo matching dataset based on view synthesis. This method primarily utilizes a one-dimensional Softmax splattering transform based on disparity comparison to perform mapping operations, enabling high-quality tertiary view synthesis of existing BSM datasets. During this one-dimensional Softmax splattering transform based on disparity comparison, the many-to-one mapping conflict problem is effectively resolved when mapping pixels in the left-view image to the tertiary image, resulting in excellent synthesis effects even for occluded areas. Furthermore, since the splattering is directed in both left and right directions, dense synthesis effects can be achieved.

[0029] refer to Figure 1In this embodiment, the method for preparing a three-eye stereo matching dataset based on view synthesis specifically includes the following steps:

[0030] Step 1: Obtain an existing binocular stereo matching dataset, where the binocular stereo matching dataset can use a conventional BSM dataset, such as SceneFlow, KITTI, and Middlebury2014;

[0031] Step 2: Obtain a set of corresponding left-eye and right-eye images from the binocular stereo matching dataset, and obtain the true disparity matrix from each pixel in the left-eye image to the right-eye image;

[0032] Step 3: Based on the true disparity matrix, calculate the theoretical disparity from each pixel in the left image to the third image.

[0033] Step 4: Based on the theoretical parallax, calculate the sub-pixel point corresponding to each pixel in the left image in the third image;

[0034] Step 5: Perform a splash transform on each sub-pixel in the third image to calculate the non-normalized contribution of each sub-pixel to each integer pixel in its left and right neighborhoods.

[0035] Step 6: In the third image, all non-normalized contributions mapped to the same integer pixel are weighted normalized and summed to obtain the pixel value of each integer pixel in the third image, thus completing the synthesis of the third image.

[0036] Step 7: Repeat steps 2 to 6 until the binocular stereo matching dataset is traversed, thus obtaining a trinocular stereo matching dataset and outputting it.

[0037] For the existing BSM dataset, it can provide the true disparity value from the left view to the right view, so the true disparity matrix in step 2 can be directly obtained. In the true disparity matrix, the i-th element is the true disparity value d from the i-th pixel in the left image to the right image. lr-i .

[0038] In the specific implementation process of this embodiment, based on the BD constraint, the disparity true value matrix D of each pixel in the left image to the right image is used. lr , calculate the theoretical disparity matrix D from each pixel in the left image to the third image lt Among them, the BD constraint refers to the hard constraint that the baseline length ratio is equal to the negative parallax ratio, that is, Ratio baseline =-Ratio disparity Assuming that the third eye in the three-eye system is to the left of the left eye, the calculation process of the theoretical disparity matrix is:

[0039]

[0040] Among them, b lt is the baseline length of the third-left eye binocular subsystem, d lr is the baseline length of the third-right eye binocular subsystem;

[0041] Therefore, the theoretical disparity from each pixel in the left eye image to the third eye image is:

[0042]

[0043] Among them, d lt-i is the theoretical disparity from the i-th pixel in the left image to the third image.

[0044] After calculating the theoretical disparity matrix from each pixel in the left-eye image to the third-eye image, the sub-pixel position of the original pixel in the left-eye image projected in the third-eye image can be determined according to the theoretical disparity matrix. Specifically,

[0045] x pi =x i -d lt-i

[0046] y pi =y i

[0047] Among them, (x i ,y i ) is the coordinate of the i-th pixel in the left image, (x pi ,y pi ) is the sub-pixel coordinate of the ith pixel in the left image in the third image.

[0048] Splash transform is a forward mapping operation widely used in the field of video interpolation. The original Softmax splash transform is a two-dimensional transform and requires an additional calculation of a source-to-target similarity coefficient Z to alleviate the many-to-one mapping conflict problem. Therefore, this embodiment proposes a one-dimensional Softmax splash transform operation based on disparity comparison. The principle of the splash transform is as follows: Figure 2 In the splash transformation process, we first calculate the left and right neighborhoods of the original pixel in the left image and the corresponding sub-pixel in the third image ( Figure 2 The left and right neighborhoods shown are both 1 pixel distance, which can also be set to more than 2 pixel distances) are the non-normalized contributions of each integer pixel point. In the left image, all pixels V l After all the splashes are applied to the third-view image, the non-normalized contributions of the same integer pixel in the third-view image are weighted normalized and summed to obtain the final pixel value of the third-view image. In addition, all integer pixels in the third-view image that were not splashed are filled with a pixel value of 0.

[0049] In the specific implementation process, for a sub-pixel point on the third-view image, the calculation process of its non-normalized contribution is:

[0050]

[0051]

[0052] in, is the non-normalized contribution of the sub-pixel corresponding to the i-th pixel in the left image to the j1-th integer pixel in its left neighborhood, is the similarity coefficient between the sub-pixel point corresponding to the i-th pixel point in the left image and the j1-th integer pixel point in its left neighborhood, is the splash weight of the sub-pixel corresponding to the i-th pixel in the left image to the j1-th integer pixel in its left neighborhood; is the non-normalized contribution of the sub-pixel corresponding to the i-th pixel in the left image to the j2-th integer pixel in its right neighborhood, is the similarity coefficient between the sub-pixel point corresponding to the i-th pixel point in the left image and the j2-th integer pixel point in its right neighborhood, V is the splash weight of the sub-pixel corresponding to the i-th pixel in the left image to the j2-th integer pixel in its right neighborhood; l-i is the pixel value of the i-th pixel in the left image.

[0053] The left splash weight and the right splash weight are calculated based on the distance between the sub-pixel point and the integer pixel point corresponding to the splash in its neighborhood, specifically:

[0054]

[0055]

[0056] in, is the x-coordinate value of the j1th integer pixel in the left neighborhood of the sub-pixel corresponding to the i-th pixel in the left image, is the x-coordinate value of the j2-th integer pixel point in the right neighborhood of the sub-pixel point corresponding to the i-th pixel point in the left eye image, and δ is the neighborhood range threshold. For example, the neighborhood range threshold can be set to δ=1.

[0057] After all pixels in the left image are splashed into the third image, the non-normalized contributions of the same integer pixel in the third image can be weighted normalized and summed to obtain the pixel value of each integer pixel in the third image. The specific calculation process is:

[0058]

[0059] in, is the pixel value of the kth integer pixel in the third image; N f is the number of sub-pixels in the left neighborhood of the kth integer pixel in the third image, is the kth integer pixel in the left neighborhood of the kth integer pixel in the third image. f The similarity coefficient of sub-pixel points, is the kth integer pixel in the left neighborhood of the kth integer pixel in the third image. f The splash weight of each sub-pixel, is the kth integer pixel in the left neighborhood of the kth integer pixel in the third image. f The pixel value of the pixel in the left image corresponding to the sub-pixel point; N c is the number of sub-pixels in the right neighborhood of the kth integer pixel in the third image, is the kth integer pixel in the right neighborhood of the kth integer pixel in the third image. c The similarity coefficient of sub-pixel points, is the kth integer pixel in the right neighborhood of the kth integer pixel in the third image. c The splash weight of each sub-pixel, is the kth integer pixel in the right neighborhood of the kth integer pixel in the third image. c The pixel value of the pixel in the right image corresponding to each sub-pixel point.

[0060] In the specific implementation process, there may be a many-to-one mapping conflict when mapping the pixels in the left eye image to the third eye image, resulting in the problem that the occluded area cannot be well synthesized. In this embodiment, after all the pixels in the left eye image are splashed to the third eye image, the theoretical disparity values ​​corresponding to multiple non-normalized contributions of the same integer pixel splashed to the third eye image are compared, and the maximum profit disparity value corresponding to Z = Z max , and the other Z=Z min , where Z max , Z min are respectively the preset maximum and minimum values ​​of the similarity coefficient. Specifically, for the similarity coefficient The specific determination process is: compare the theoretical disparities corresponding to all sub-pixel points in the left and right neighborhoods of the kth integer pixel in the third image, and set the similarity coefficient of the sub-pixel point corresponding to the maximum theoretical disparity as Z max , the similarity coefficients of all other sub-pixel points are set to Z min .

[0061] For example Figure 3As shown in the figure, assuming that the left and right neighborhood ranges are both one pixel away, there are integer pixel A and integer pixel B in the same x-direction in the third-view image, sub-pixels 1 to 2 exist in the left neighborhood of integer pixel A, sub-pixels 3 to 5 exist in the right neighborhood of integer pixel A and in the left neighborhood of integer pixel B, and sub-pixel 6 exists in the right neighborhood of integer pixel B.

[0062] In the process of calculating the pixel value of integer pixel point A, it mainly calculates the non-normalized contribution of sub-pixel points 1 to 2 to the right of integer pixel point A. And the non-normalized contribution of sub-pixels 4 to 5 to the left of integer pixel A Normalize the weights and add them up. Assume The corresponding similarity coefficients are And among the sub-pixels 1 to 5, the theoretical parallax corresponding to sub-pixel 4 is the largest, then

[0063] In the process of calculating the pixel value of integer pixel point B, it mainly calculates the non-normalized contribution of sub-pixels 3 to 5 to the right of integer pixel point B. And the non-normalized contribution of sub-pixel 6 to the left at integer pixel B Normalize the weights and add them up. Assume The corresponding similarity coefficients are And among the sub-pixels 3 to 6, the theoretical parallax corresponding to sub-pixel 6 is the largest, then

[0064] The following further illustrates the method for producing a three-view stereo matching dataset based on view synthesis in this embodiment with reference to specific examples.

[0065] The experiments in this example use the BSM datasets to be expanded, including SceneFlow, KITTI2015, and Middlebury2014, and set Z max =20 and Z min =10, using Python and C++ programming languages ​​to implement the trinocular stereo matching dataset production method based on view synthesis in this embodiment.

[0066] refer to Figure 4 It is the synthetic effect diagram of the method proposed in this embodiment, from Figure 4 It can be seen that the method proposed in this embodiment can effectively perform TSM expansion on the existing widely used BSM dataset. At the same time, the almost real-time synthesis speed also demonstrates the ability of synthesizing large-scale datasets proposed in this embodiment.

[0067] refer to Figure 5The figure shows the comparison of the synthesis effect of the method proposed in this embodiment and the other two synthesis methods. The two synthesis methods are the nearest neighbor method and the average method. The nearest neighbor method maps all pixels in the left image to the third image by filling the nearest integer pixels in a covering manner. The average method performs an average normalized summation of the contributions splashed to the same integer pixel. Figure 5 As can be seen, the method proposed in this embodiment can effectively handle occlusion issues and achieve a denser synthesis effect compared to the other two methods. The solution to the occlusion problem is attributed to the parallax comparison, while the dense synthesis effect is attributed to the target sub-pixel's contribution being distributed in both the left and right directions.

[0068] In addition, Figure 6 The synthetic effect diagram of the method proposed in this embodiment under different baseline ratio conditions is shown in Figure 2. x represents the absolute value of the baseline ratio of the third eye-left eye binocular subsystem to the left eye-right eye binocular subsystem. Lx means that the third eye is located to the left of the left eye, while Rx means that both the right eye and the third eye are located to the right of the left eye. Figure 6 The synthetic effect diagrams of three ratios, L1.00, L0.25 and R0.50, are given.

[0069] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for producing a three-eye stereo matching data set based on view synthesis, characterized in that: The steps include: Step 1, obtain the existing binocular stereo matching dataset; Step 2, obtaining a set of corresponding left-eye images and right-eye images in the binocular stereo matching data set, and obtaining a disparity true value matrix from each pixel in the left-eye image to the right-eye image; Step 3, based on the disparity true value matrix, calculate the theoretical disparity from each pixel in the left eye image to the third eye image, which is: in, The left image The theoretical disparity from pixel to the third-eye image is: The left image The true disparity value from the pixel to the right eye image, is the baseline length of the third-left eye binocular subsystem, is the baseline length of the third-right eye binocular subsystem; Step 4, based on the theoretical parallax, calculating the sub-pixel point corresponding to each pixel point in the left eye image in the third eye image; Step 5, performing a splash transformation based on parallax comparison on each of the sub-pixel points in the third eye image, and calculating the non-normalized contribution of each of the sub-pixel points to each integer pixel point in its left and right neighborhoods, which is: in, The left image The sub-pixel corresponding to the pixel point is the first pixel in its left neighborhood. The unnormalized contribution of integer pixels, The left image The sub-pixel corresponding to the pixel and its left neighbor The similarity coefficient of integer pixels, The left image The sub-pixel corresponding to the pixel point is the first pixel in its left neighborhood. The splash weight of integer pixels; The left image The sub-pixel corresponding to the pixel point is the The unnormalized contribution of integer pixels, The left image The sub-pixel corresponding to the pixel and its right neighbor The similarity coefficient of integer pixels, The left image The sub-pixel corresponding to the pixel point is the The splash weight of integer pixels; The left image The pixel value of a pixel; Step 6, in the third image, weight normalize and sum all non-normalized contributions mapped to the same integer pixel point to obtain the pixel value of each integer pixel point in the third image, thus completing the synthesis of the third image; Step 7, repeating steps 2 to 6 until the binocular stereo matching data set is traversed, that is, a trinocular stereo matching data set is obtained.

2. The method for producing a three-eye stereo matching data set based on view synthesis according to claim 1, characterized in that: In step 4, the calculation process of the sub-pixel point is: in, The left image The coordinates of the pixels, The left image The sub-pixel coordinates of the pixels in the third-party image. The left image The theoretical disparity from a pixel to a third-eye image.

3. The method for producing a three-eye stereo matching data set based on view synthesis according to claim 1 or 2, characterized in that: The left splash weight and the right splash weight are specifically: in, The left image The pixel point corresponds to the left neighborhood of the sub-pixel point Integer pixels Coordinate values, The left image The pixel point corresponds to the sub-pixel point in the right neighborhood Integer pixels Coordinate values, is the neighborhood range threshold.

4. The method for producing a three-eye stereo matching data set based on view synthesis according to claim 1 or 2, characterized in that: In step 6, the pixel value of each integer pixel in the third image is specifically: in, The third image The pixel value of an integer pixel; The third image The number of sub-pixels in the left neighborhood of an integer pixel, The third image The integer pixel in the left neighborhood The similarity coefficient of sub-pixels, The third image The integer pixel in the left neighborhood The splash weight of each sub-pixel, The third image The integer pixel in the left neighborhood The pixel value of the pixel in the left image corresponding to the sub-pixel point; The third image The number of sub-pixels in the right neighborhood of an integer pixel, The third image The integer pixel in the right neighborhood The similarity coefficient of sub-pixels, The third image The integer pixel in the right neighborhood The splash weight of each sub-pixel, The third image The integer pixel in the right neighborhood The pixel value of the pixel in the right image corresponding to the sub-pixel point.

5. The method for producing a three-eye stereo matching data set based on view synthesis according to claim 1 or 2, characterized in that: In calculating the third image In the process of calculating the pixel values ​​of integer pixels, the similarity coefficient , The determination process is: Compare the third image The theoretical disparity corresponding to all sub-pixel points in the left and right neighborhoods of integer pixels is set as , the similarity coefficients of all other sub-pixels are set to ,in, , They are respectively the preset maximum and minimum values ​​of the similarity coefficient.

Images