A stereo vision matching method and apparatus

Abstract

This application discloses a stereo vision matching method, comprising the following steps: acquiring a left image and a right image of the same scene taken in parallel with a first size; extracting features from the left image and the right image respectively to obtain two first feature matrices; concatenating the two first feature matrices to obtain a first disparity value for each pixel; constructing a first disparity map based on the first disparity value, and constructing a first cost cube based on the first disparity map and a first feature set; compressing the size of the first disparity map in the first cost cube, then extracting features, and then enlarging the size, wherein the feature set used is the feature set after size compression and feature extraction; repeating the above steps several times, and iterating the feature extraction method sequentially, finally obtaining the output disparity value. This stereo vision matching method and apparatus have a short processing time when processing images.

Description

Technical Field

[0001] This application generally relates to the field of machine vision technology, and more particularly to a stereo vision matching method and apparatus. Background Technology

[0002] With the rapid development of image processing technology, it has become possible for robots to mimic the human visual system. To enable robots to construct 3D scenes, the most commonly used technology at present is binocular stereo vision recognition to obtain depth information of 3D scenes. Existing technologies often use convolutional neural networks to train the above process to obtain more accurate output feature maps. The training process generally involves several rounds of size compression and feature extraction on the cost cube formed by the initial disparity map output by the model, followed by a corresponding number of size enlargements. During this process, the model parameters are continuously adjusted to extract more image features. However, its drawback is that since the modules in the model are connected in series, the features extracted by the nearest feature extraction module are used during the adjustment of model parameters. Because the model itself suffers from the vanishing gradient problem as the number of training layers increases, meaning that the model's ability to adjust the parameters for feature extraction weakens with the increase in processing modules, more modules need to be connected in series to extract features from the cost cube. This results in a long time required for visual matching, and if the equipment fails, the results of the previous long-term calculations will be wasted, leading to a waste of manpower and resources. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a stereo vision matching method and apparatus with shorter image processing time.

[0004] The specific technical solution is as follows:

[0005] First aspect:

[0006] This application provides a stereo vision matching method, including the following steps:

[0007] Acquire a left and a right image of the same scene taken in parallel, each with a first size, wherein the first size is the length and width data of the left or right image, and the corresponding number of pixels is a first value;

[0008] Feature extraction is performed on the left image and the right image respectively, and corresponding first feature matrix of the left image and first feature matrix of the right image are obtained. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set.

[0009] The first feature matrix of the left image and the first feature matrix of the right image are concatenated to obtain the first disparity value of each pixel;

[0010] Construct a first disparity map of size first based on the first disparity value of each pixel;

[0011] Construct a first cost cube based on the first disparity map and the first feature set;

[0012] The size of the first disparity map in the first cost cube is compressed by a first ratio to obtain a second cost cube, and the size of the second disparity map in the second cost cube is the second size.

[0013] Feature extraction is performed on the second disparity map to obtain a third cost cube, the feature set of the third cost cube being the second feature set;

[0014] The size of the disparity map in the third cost cube is enlarged by the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set;

[0015] The feature extraction method is adjusted using the feature set in the fourth cost cube;

[0016] Repeat the above steps several times, and iterate the feature extraction method sequentially to obtain the fifth cost cube;

[0017] The cost amount is obtained by upsampling the fifth cost cube;

[0018] The first disparity value is corrected based on the stated cost amount to obtain the output disparity value.

[0019] Optionally, when performing feature extraction on the left image and the right image respectively, three 3×3 convolutional kernels and several residual modules are used.

[0020] Optionally, when compressing the size of the first disparity map in the first cost cube at a first ratio, convolution operations are performed using convolution kernels with 3×3×3 and 1×1×1.

[0021] Optionally, when correcting the first disparity value based on the stated cost amount, the formula used is:

[0022] Where d represents the value of the disparity, which can take any value from 0 to 255; Indicates the output disparity value; c d This represents the cost amount, and its value corresponds to the disparity value; σ(-c d) represents the probability of the cost amount when the disparity value is d.

[0023] Optionally, before concatenating the first feature matrix of the left image and the first feature matrix of the right image, feature extraction is performed on the edge pixels of the first feature matrix of the left image and the first feature matrix of the right image using dilated convolutions of several sizes.

[0024] The second aspect:

[0025] This application provides a stereo vision matching device, comprising:

[0026] The acquisition module is used to acquire a left image and a right image of the same scene taken in parallel with a first size, wherein the first size is the length and width data of the left image or the right image, and the corresponding number of pixels is a first value;

[0027] The first feature extraction module is used to extract features from the left image and the right image respectively, and obtain the corresponding first feature matrix of the left image and the first feature matrix of the right image. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set.

[0028] A cascade module is used to cascade the first feature matrix of the left image and the first feature matrix of the right image to obtain the first disparity value of each pixel.

[0029] A construction module is configured to construct a first disparity map of a first size based on the first disparity value of each pixel, and to construct a first cost cube based on the first disparity map and the first feature set.

[0030] A first image compression module is used to compress the size of the first disparity map in the first cost cube at a first ratio to obtain a second cost cube, wherein the size of the second disparity map in the second cost cube is a second size;

[0031] The second feature extraction module is used to extract features from the second disparity map and obtain a third cost cube, wherein the feature set of the third cost cube is the second feature set;

[0032] A first image magnification module is used to magnify the size of the disparity map in the third cost cube at the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set.

[0033] The model iteration module is used to adjust the feature extraction method using the feature set in the fourth cost cube, repeat the above steps several times, and iterate the feature extraction method in sequence to obtain the fifth cost cube.

[0034] The calculation output module is used to upsample the fifth cost cube to obtain the cost amount, and correct the first disparity value according to the cost amount to obtain the output disparity value.

[0035] The beneficial effects of this application are:

[0036] The stereo vision matching method described above, when performing size restoration and feature extraction on the fourth cost cube after disparity map size compression and feature extraction, simultaneously adds the first feature set in the cost cube before feature extraction and the second feature set in the cost cube after feature extraction. This way, even if the processing is repeated multiple times, the relatively original features can be used each time, which can effectively alleviate the gradient vanishing defect. This can effectively reduce the number of modules used for image compression, feature extraction, and image enlargement, and thus take less time to obtain disparity maps of the same processing quality. Attached Figure Description

[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 A flowchart of the stereo vision matching method provided in the embodiments of this application. Detailed Implementation

[0039] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] Example 1

[0042] Please refer to Figure 1 The stereo vision matching method provided in this embodiment includes the following steps:

[0043] Acquire a left and a right image of the same scene taken in parallel, each with a first size, wherein the first size is the length and width data of the left or right image, and the corresponding number of pixels is a first value;

[0044] Feature extraction is performed on the left image and the right image respectively, and corresponding first feature matrix of the left image and first feature matrix of the right image are obtained. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set.

[0045] The first feature matrix of the left image and the first feature matrix of the right image are concatenated to obtain the first disparity value of each pixel;

[0046] Construct a first disparity map of size first based on the first disparity value of each pixel;

[0047] Construct a first cost cube based on the first disparity map and the first feature set;

[0048] The size of the first disparity map in the first cost cube is compressed by a first ratio to obtain a second cost cube, and the size of the second disparity map in the second cost cube is the second size.

[0049] Feature extraction is performed on the second disparity map to obtain a third cost cube, the feature set of the third cost cube being the second feature set;

[0050] The size of the disparity map in the third cost cube is enlarged by the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set;

[0051] The feature extraction method is adjusted using the feature set in the fourth cost cube; the above steps are repeated several times, and the feature extraction method is iterated sequentially to obtain the fifth cost cube;

[0052] The cost amount is obtained by upsampling the fifth cost cube;

[0053] The first disparity value is corrected based on the stated cost amount to obtain the output disparity value.

[0054] The stereo vision matching method described above, when performing size restoration and feature extraction on the fourth cost cube after disparity map size compression and feature extraction, simultaneously adds the first feature set in the cost cube before feature extraction and the second feature set in the cost cube after feature extraction. This way, even if the processing is repeated multiple times, the relatively original features can be used each time, which can effectively alleviate the gradient vanishing defect. This can effectively reduce the number of modules used for image compression, feature extraction, and image enlargement, and thus take less time to obtain disparity maps of the same processing quality.

[0055] In a preferred embodiment that further improves the speed of stereo vision matching processing, when extracting features from the left image and the right image respectively, three 3×3 convolutional kernels and several residual modules are used.

[0056] In existing technologies, 7×7 convolutions are generally used to extract features from the left and right images. However, this is computationally complex when the set receptive field is obtained. Replacing it with three 3×3 convolution kernels and several residual modules can achieve the same receptive field, and the computation is simpler, effectively improving the speed of stereo vision matching processing.

[0057] In a preferred embodiment that further improves the speed of stereo vision matching processing, when compressing the size of the first disparity map in the first cost cube at a first ratio, convolution operations are performed using convolution kernels of 3×3×3 and 1×1×1.

[0058] In existing technologies, average pooling layers are generally used to reduce the dimensionality of disparity maps. However, by using 3×3×3 convolutional layers, more feature information can be retained. This eliminates the need for subsequent feature extraction modules, simplifies the model, and improves processing speed.

[0059] In a preferred embodiment of calculating the output disparity value of each pixel in the output disparity map, the formula used when correcting the first disparity value according to the cost amount is:

[0060] Where d represents the value of the disparity, which can take any value from 0 to 255; Indicates the output disparity value; c d This represents the cost amount, and its value corresponds to the disparity value; σ(-c d ) represents the probability of the cost amount when the disparity value is d.

[0061] In a preferred embodiment that further improves the speed of stereo vision matching processing, before concatenating the first feature matrix of the left image and the first feature matrix of the right image, feature extraction is performed on the edge pixels of the first feature matrix of the left image and the first feature matrix of the right image using dilated convolutions of several sizes.

[0062] By employing different dilated convolutions—that is, convolution kernels used to extract disparity map features—with varying dilation rates and kernel sizes, more edge information of objects within the disparity map can be extracted. This results in higher efficiency in feature extraction, which in turn reduces the number of modules and improves processing speed.

[0063] Example 2

[0064] This embodiment provides a stereoscopic vision matching device, comprising:

[0065] The acquisition module is used to acquire a left image and a right image of the same scene taken in parallel with a first size, wherein the first size is the length and width data of the left image or the right image, and the corresponding number of pixels is a first value;

[0066] The first feature extraction module is used to extract features from the left image and the right image respectively, and obtain the corresponding first feature matrix of the left image and the first feature matrix of the right image. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set.

[0067] A cascade module is used to cascade the first feature matrix of the left image and the first feature matrix of the right image to obtain the first disparity value of each pixel.

[0068] A construction module is configured to construct a first disparity map of a first size based on the first disparity value of each pixel, and to construct a first cost cube based on the first disparity map and the first feature set.

[0069] A first image compression module is used to compress the size of the first disparity map in the first cost cube at a first ratio to obtain a second cost cube, wherein the size of the second disparity map in the second cost cube is a second size;

[0070] The second feature extraction module is used to extract features from the second disparity map and obtain a third cost cube, wherein the feature set of the third cost cube is the second feature set;

[0071] A first image magnification module is used to magnify the size of the disparity map in the third cost cube at the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set.

[0072] The model iteration module is used to adjust the feature extraction method using the feature set in the fourth cost cube, repeat the above steps several times, and iterate the feature extraction method in sequence to obtain the fifth cost cube.

[0073] The calculation output module is used to upsample the fifth cost cube to obtain the cost amount, and correct the first disparity value according to the cost amount to obtain the output disparity value.

[0074] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A stereo vision matching method, characterized in that, Includes the following steps: Acquire a left and a right image of the same scene taken in parallel, each with a first size, wherein the first size is the length and width data of the left or right image, and the corresponding number of pixels is a first value; Feature extraction is performed on the left image and the right image respectively, and corresponding first feature matrix of the left image and first feature matrix of the right image are obtained. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set. The first feature matrix of the left image and the first feature matrix of the right image are concatenated to obtain the first disparity value of each pixel; Construct a first disparity map of a first size based on the first disparity value of each pixel; Construct a first cost cube based on the first disparity map and the first feature set; The size of the first disparity map in the first cost cube is compressed by a first ratio to obtain a second cost cube, and the size of the second disparity map in the second cost cube is the second size; Feature extraction is performed on the second disparity map to obtain a third cost cube, the feature set of the third cost cube being the second feature set; The size of the disparity map in the third cost cube is enlarged by the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set; The feature extraction method is adjusted using the feature set in the fourth cost cube; Repeat the above steps several times, and iterate the feature extraction method sequentially to obtain the fifth cost cube; The cost amount is obtained by upsampling the fifth cost cube; The first disparity value is corrected based on the cost amount to obtain the output disparity value; When compressing the size of the first disparity map in the first cost cube at a first ratio, convolution operations are performed using 3×3×3 and 1×1×1 convolution kernels.

2. The stereo vision matching method according to claim 1, characterized in that, When extracting features from the left and right images respectively, three 3×3 convolutional kernels and several residual modules are used.

3. The stereo vision matching method according to claim 1, characterized in that, The formula used when correcting the first disparity value based on the stated cost amount is: , where d represents the value of the disparity, which can take any value from 0 to 255; Indicates the output disparity value; c d This represents the cost amount, and its value corresponds to the disparity value; σ(-c d ) represents the probability of the cost amount when the disparity value is d.

4. The stereo vision matching method according to any one of claims 1-3, characterized in that, Before concatenating the first feature matrix of the left image and the first feature matrix of the right image, feature extraction is performed on the edge pixels of the first feature matrix of the left image and the first feature matrix of the right image using dilated convolutions of several sizes.

5. A device for stereoscopic vision matching, characterized in that, include: The acquisition module is used to acquire a left image and a right image of the same scene taken in parallel with a first size, wherein the first size is the length and width data of the left image or the right image, and the corresponding number of pixels is a first value; The first feature extraction module is used to extract features from the left image and the right image respectively, and obtain the corresponding first feature matrix of the left image and the first feature matrix of the right image. The feature matrix is ​​a three-dimensional vector composed of the size of the image and the types of features extracted from it. The image size of the first feature matrix of the left image and the first feature matrix of the right image is a first size, the number of pixels is a first value, and the feature set is a first feature set. A cascade module is used to cascade the first feature matrix of the left image and the first feature matrix of the right image to obtain the first disparity value of each pixel. A construction module is configured to construct a first disparity map of a first size based on the first disparity value of each pixel, and to construct a first cost cube based on the first disparity map and the first feature set. A first image compression module is used to compress the size of the first disparity map in the first cost cube at a first ratio to obtain a second cost cube, wherein the size of the second disparity map in the second cost cube is a second size; wherein, when compressing the size of the first disparity map in the first cost cube at the first ratio, convolution operations are performed using 3×3×3 and 1×1×1 convolution kernels; The second feature extraction module is used to extract features from the second disparity map and obtain a third cost cube, wherein the feature set of the third cost cube is the second feature set; A first image magnification module is used to magnify the size of the disparity map in the third cost cube at the first ratio to obtain a fourth cost cube, wherein the size of the disparity map in the fourth cost cube is the first size, and the feature set is the first feature set and the second feature set. The model iteration module is used to adjust the feature extraction method using the feature set in the fourth cost cube, repeat the above steps several times, and iterate the feature extraction method in sequence to obtain the fifth cost cube. The calculation output module is used to upsample the fifth cost cube to obtain the cost amount, and correct the first disparity value according to the cost amount to obtain the output disparity value.

Images