A binocular matching method, device and computer readable storage medium

Abstract

Embodiments of the present application provide a binocular matching method and device and a computer readable storage medium, and relate to the field of computer vision. In the binocular matching method and device, a normal map is obtained by using a photometric stereo method, and the normal map is fused with an original image in a channel, so that the photometric image originally having only intensity features becomes a first multi-dimensional image and a second multi-dimensional image containing normal vectors, many binocular matching features can be increased, the matching effect is improved, and the situation of few textures or no textures in three-dimensional reconstruction can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a binocular matching method, device and computer readable storage medium. BACKGROUND

[0002] Binocular matching is a technique for recovering depth from a pair of pictures taken from different angles, usually each pair of pictures is obtained by a pair of cameras placed side by side or above and below. In order to simplify the problem, the pictures taken from different cameras are corrected so that when the cameras are placed side by side, the corresponding pixels are located on the same horizontal line, or when the cameras are placed above and below, the corresponding pixels are located on the same vertical line. However, in the prior art, for objects with little texture, no texture and pure color, such as skin surface and white wall, the feature points obtained after binocular matching are very few, resulting in failure of reconstruction. SUMMARY

[0003] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a binocular matching method, device and computer readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides a binocular matching method, which is applied to a computer device in a binocular matching system, the binocular matching system comprising two cameras distributed along a horizontal direction or a vertical direction, and point light sources distributed around each camera, and the method comprising:

[0005] obtaining a set of photometric images of a target scene from the cameras, the set of photometric images comprising a first set of photometric images and a second set of photometric images;

[0006] calculating a normal map of the target scene according to the set of photometric images;

[0007] preprocessing the set of photometric images to obtain a photometric processed image;

[0008] performing image fusion on the normal map and the photometric processed image to obtain a first multi-dimensional image and a second multi-dimensional image;

[0009] performing binocular matching on the first multi-dimensional image and the second multi-dimensional image to output a binocular matching image.

[0010] In a possible implementation, the step of calculating the normal map of the target scene according to the set of photometric images comprises:

[0011] configuring light source direction and light source intensity parameters in the target scene;

[0012] Perform a pixel-by-pixel normal calculation on the photometric image set based on the light source direction and the light source intensity parameter, and output a normal map of the target scene, wherein the normal map includes a first normal map corresponding to the first photometric image set and a second normal map corresponding to the second photometric image set.

[0013] In a possible implementation, the step of preprocessing the photometric image set to obtain a photometric processed image includes:

[0014] Perform image quality assessment on the photometric image set to obtain a first quality assessment score of the photometric image set.

[0015] Detect a region in the photometric image set in which pixel values are greater than a first preset pixel value or less than a second preset pixel value, and preprocess the region in the photometric image set in which pixel values are greater than the first preset pixel value or less than the second preset pixel value, wherein the first preset pixel value is greater than the second preset pixel value.

[0016] Perform image quality assessment on the preprocessed photometric image set to obtain a second quality assessment score of the photometric image set.

[0017] Compare the first quality assessment score and the second quality assessment score, and output a photometric processed image with the highest image quality assessment score, wherein the photometric processed image with the highest image quality assessment score includes a first photometric processed image and a second photometric processed image.

[0018] In a possible implementation, the step of detecting a region in the photometric image set in which pixel values are greater than a first preset pixel value or less than a second preset pixel value, and preprocessing the region in the photometric image set in which pixel values are greater than the first preset pixel value or less than the second preset pixel value includes:

[0019] Detect a first region in the photometric image set in which all pixel values are greater than the first preset pixel value.

[0020] Perform mean value processing on the first region, and perform filter smoothing processing on an edge of the first region.

[0021] Perform image quality assessment on the photometric image set to obtain a process quality assessment score of each image in the photometric image set.

[0022] Detect a second region in the photometric image with the highest process quality assessment score in which all pixel values are less than the second preset pixel value.

[0023] Perform mean value processing on the second region, and perform filter smoothing processing on an edge of the second region.

[0024] In a possible implementation, the step of comparing the first quality evaluation score and the second quality evaluation score, and outputting a photometrically processed image with the highest image quality evaluation score, wherein the photometrically processed image with the highest image quality evaluation score comprises the first photometrically processed image and the second photometrically processed image, comprises the following steps of:

[0025] performing image quality evaluation on the set of photometric images to obtain a first quality evaluation score of each image in the set of photometric images;

[0026] detecting a first region in which pixel values in the set of photometric images are all greater than the first preset pixel value;

[0027] performing average processing on the region in which pixel values are all greater than the first preset pixel value, and performing filter smoothing processing on an edge of the first region;

[0028] performing image quality evaluation on the preprocessed set of photometric images to obtain a process quality evaluation score of each image in the set of photometric images;

[0029] comparing the process quality evaluation score and the first quality evaluation score, and if the process quality evaluation score of each image in the set of photometric images is less than the first quality evaluation score, returning to the step of performing average processing on the region in which pixel values are all greater than the first preset pixel value, and performing filter smoothing processing on an edge of the first region;

[0030] if the process quality evaluation score of each image in the set of photometric images is greater than the first quality evaluation score, detecting a second region in which pixel values in the set of photometric images are all less than the second preset pixel value;

[0031] performing average processing on the region in which pixel values are all less than the second region, and performing filter smoothing processing on an edge of the second region;

[0032] performing image quality evaluation on the set of photometric images to obtain a second quality evaluation score of each image in the set of photometric images;

[0033] comparing the second quality evaluation score and the process quality evaluation score, and if the second quality evaluation score of each image in the set of photometric images is less than the process quality evaluation score, returning to the step of performing average processing on the second region, and performing filter smoothing processing on an edge of the second region;

[0034] If the second quality evaluation scores of each image in the photometric image set are greater than the process quality evaluation score, output the photometric processing image with the highest image quality evaluation score, the photometric processing image with the highest image quality evaluation score including a first photometric processing image and a second photometric processing image.

[0035] In a possible implementation, the step of performing binocular matching on the first multi-dimensional image and the second multi-dimensional image and outputting a binocular matching map includes:

[0036] performing feature extraction on the first multi-dimensional image and the second multi-dimensional image, and performing matching on the features extracted from the first multi-dimensional image and the second multi-dimensional image to output a binocular matching map.

[0037] In a possible implementation, the step of performing feature extraction on the first multi-dimensional image and the second multi-dimensional image, and performing matching on the features extracted from the first multi-dimensional image and the second multi-dimensional image to output a binocular matching map includes:

[0038] performing gray-scale feature extraction on the first multi-dimensional image and the second multi-dimensional image, and performing feature matching and epipolar matching on the gray-scale features extracted from the first multi-dimensional image and the second multi-dimensional image to obtain a first depth matching result and a first matching region, where the first matching region is a region in which the gray-scale feature matching fails in the first multi-dimensional image and the second multi-dimensional image;

[0039] performing normal feature extraction on the first matching region, and performing feature matching and epipolar matching on the normal features extracted from the first matching region to obtain a second depth matching result;

[0040] performing three-dimensional reconstruction based on the first depth matching result and the second depth matching result to obtain a binocular matching map, and outputting the binocular matching map.

[0041] In a second aspect, the embodiments of the present application further provide a binocular matching system, which includes a computer device, two cameras distributed in a horizontal direction or a vertical direction, and point light sources distributed around each camera.

[0042] The camera is configured to acquire a photometric image set obtained by taking a target scene, the photometric image set including a first photometric image set and a second photometric image set.

[0043] The computer device is configured to perform a binocular matching method, the binocular matching method including:

[0044] The camera is configured to acquire a photometric image set obtained by taking a target scene, the photometric image set including a first photometric image set and a second photometric image set.

[0045] compute a normal map of the target scene according to the set of photometric images;

[0046] pre-process the set of photometric images to obtain a photometric processed image;

[0047] fuse the normal map and the photometric processed image to obtain a first multi-dimensional image and a second multi-dimensional image;

[0048] perform binocular matching on the first multi-dimensional image and the second multi-dimensional image to output a binocular matching map.

[0049] In a possible implementation, the camera includes a first camera and a second camera, and the point light sources include a first point light source surrounding the first camera and a second point light source surrounding the second camera, where the number of the first point light sources and the second point light sources is multiple.

[0050] The first camera is configured to take a view of a target scene when the multiple first point light sources are sequentially lighted to obtain the first set of photometric images, and the first set of photometric images include photometric images when the multiple first point light sources are sequentially lighted. The second camera is configured to take a view of the target scene when the multiple second point light sources are sequentially lighted to obtain the second set of photometric images, and the second set of photometric images include photometric images when the multiple second point light sources are sequentially lighted.

[0051] In a third aspect, the embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the binocular matching method according to any one of the above aspects.

[0052] Based on any one of the above aspects, the binocular matching method, device, and computer readable storage medium provided by the embodiments of the present application use the photometric stereo method to obtain a normal map, and fuse the normal map and the original image in the channel to make the photometric image originally having only intensity features become the first multi-dimensional image and the second multi-dimensional image containing normal vectors, so that many binocular matching features can be increased, the matching effect can be improved, and the case of less texture or no texture in three-dimensional reconstruction can be solved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be called in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0054] Figure 1A possible flowchart of the binocular matching method provided in an embodiment of the present application;

[0055] Figure 2 To achieve Figure 1 A possible flow diagram of step S12;

[0056] Figure 3 To achieve Figure 1 A possible flow diagram of step S13;

[0057] Figure 4 To achieve Figure 3 A possible flow diagram of step S132;

[0058] Figure 5 A schematic diagram of a possible partial structure of the binocular matching device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0062] In the description of this application, it should be noted that the terms "upper" and "lower" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or the orientations or positional relationships in which the product of this application is typically placed when in use. These terms are intended solely to facilitate the description of this application and simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" and the like are used solely for distinction and should not be construed as indicating or implying relative importance.

[0063] It should be noted that, in the description of the present application, unless otherwise explicitly specified and limited, the terms "set", "communication", "connection", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or it can be indirectly connected through an intermediate medium, it can be the internal connection of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0064] It should be noted that, in the description of the present application, unless otherwise explicitly specified and limited, the terms "set", "communication", "connection", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or it can be indirectly connected through an intermediate medium, it can be the internal connection of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0065] Please refer to Figure 1 , Figure 1 The flowchart of the binocular matching method provided by the embodiment of the present application is shown. It should be noted that the binocular matching method provided by the embodiment of the present application is applicable to the computer device in the binocular matching system, the binocular matching system includes two cameras distributed along the horizontal direction or the vertical direction, and point light sources distributed around each camera, and the binocular matching method can be realized by the following method.

[0066] Step S11: obtaining a set of photometric images of the target scene from the camera, the set of photometric images including a first set of photometric images and a second set of photometric images.

[0067] In this step, the set of photometric images is a collection of images with different light and dark distributions in the target scene. Specifically, the first set of photometric images includes photometric images obtained by the first camera when the point light sources distributed around the first camera are sequentially lighted, and the second set of photometric images includes photometric images obtained by the second camera when the point light sources distributed around the second camera are sequentially lighted.

[0068] Step S12: calculating the normal map of the target scene according to the set of photometric images.

[0069] In this step, the normal map is a kind of texture associated with a geometric surface, which represents the normal direction of each pixel by using RGB color components, and can be used to enhance the surface texture of computer graphics, and can be obtained based on the reflection equation set of Lambert reflection principle.

[0070] Step S13: pre-processing the set of photometric images to obtain a set of photometric processing images.

[0071] In this step, the first set of photometric images and the second set of photometric images are preprocessed to synthesize photometric processing images. In the case of little texture or no texture, the texture can be enriched by imaging under different illuminations. In addition, preprocessing the set of photometric images can effectively remove noise in the images, while balancing the brightness difference between the stereo image pairs, so that the balanced stereo image pairs can meet the requirements of subsequent stereo matching.

[0072] Step S14: image fusion of the normal map and the photometric processing image to obtain a first multi-dimensional image and a second multi-dimensional image.

[0073] In this step, image fusion refers to the process of fusing two or more image data obtained under different conditions of sensors, times, angles of view, radiation, etc., to obtain more accurate, comprehensive, and high-quality image information. Image fusion of the normal map and the photometric processing image can obtain more accurate, comprehensive, and high-quality first multi-dimensional image and second multi-dimensional image.

[0074] Step S15: binocular matching of the first multi-dimensional image and the second multi-dimensional image to output a binocular matching map.

[0075] In this step, the depth matching result of the first multi-dimensional image and the second multi-dimensional image can be obtained by feature matching and epipolar matching of the gray features and / or normal features in the first multi-dimensional image and the second multi-dimensional image. Based on the depth matching result, three-dimensional reconstruction is performed to obtain the binocular matching map.

[0076] In this embodiment, the normal map is obtained by using the photometric stereo method, and the normal map is fused with the original image in the channel to make the photometric image originally having only intensity features become the first multi-dimensional image and the second multi-dimensional image containing normal vectors. This can increase many binocular matching features and improve the matching effect, and can solve the problem of little texture or no texture in three-dimensional reconstruction.

[0077] In one possible implementation of this embodiment, please refer to Figure 2 Step S12 can be implemented by the following method.

[0078] Sub-step S121: configuring the light source direction and light source intensity parameters in the target scene.

[0079] In this step, the light source direction and light source intensity in the target scene can be obtained by calibrating the light source in the target scene.

[0080] Sub-step S122: performing pixel-by-pixel normal calculation on the set of photometric images based on the light source direction and light source intensity parameters to output a normal map of the target scene, the normal map including a first normal map corresponding to the first set of photometric images and a second normal map corresponding to the second set of photometric images.

[0081] In this step, the normal map of the target scene can be obtained by pixel-wise normal calculation through the following formula:

[0082] tij = pi (Ni · Li) lj

[0083] In the above formula, tij represents the pixel value generated by light source j irradiating the i-th point on the object surface, pi represents the reflectivity of the i-th point, Ni represents the normal vector of the i-th point, Li represents the unit vector of the light source direction, and lj represents the light source intensity.

[0084] In this embodiment, the reflection equation set based on the Lambert reflection principle is used to perform pixel-wise normal calculation on the photometric image set to obtain the normal map of the target scene. In subsequent steps, many binocular matching features can be added to improve the matching effect and solve the problem of less texture or no texture in three-dimensional reconstruction.

[0085] In a possible implementation manner of the embodiment, referring to Figure 3 , step S13 can be implemented by the following method.

[0086] Sub-step S131: performing image quality assessment on the photometric image set to obtain a first quality assessment score of the photometric image set.

[0087] In this step, the BRISQUE algorithm can be used to perform image quality assessment on the photometric image set, and the first quality assessment score can be used to evaluate the effectiveness of image preprocessing.

[0088] Sub-step S132: detecting a region in the photometric image set with a pixel value greater than a first preset pixel value or less than a second preset pixel value, and preprocessing the region in the photometric image set with a pixel value greater than the first preset pixel value or less than the second preset pixel value, wherein the first preset pixel value is greater than the second preset pixel value.

[0089] In this step, preprocessing the region in the photometric image set with a pixel value greater than the first preset pixel value or less than the second preset pixel value can effectively remove noise in the image and synthesize a photometric processing image with better imaging effect.

[0090] Sub-step S133: performing image quality assessment on the preprocessed photometric image set to obtain a second quality assessment score of the photometric image set.

[0091] In this step, the BRISQUE algorithm can be used to perform image quality assessment on the photometric image set, and the second quality assessment score can be used to evaluate the result of image preprocessing.

[0092] Sub-step S134: comparing the first quality evaluation score and the second quality evaluation score, outputting the photometrically processed image with the highest image quality evaluation score, the photometrically processed image with the highest image quality evaluation score including the first photometrically processed image and the second photometrically processed image.

[0093] In this step, by comparing the first quality evaluation score and the second quality evaluation score, it is ensured that the image preprocessing has a positive effect on the image quality.

[0094] In this embodiment, the first set of photometric images and the second set of photometric images are preprocessed to synthesize photometrically processed images, which can enrich the texture under different illuminations in the case of less texture or no texture. In addition, preprocessing the set of photometric images can effectively remove the noise of the image, while balancing the brightness difference between the stereo image pairs, so that the balanced stereo image pairs can meet the requirements of subsequent stereo matching.

[0095] Further, please refer to Figure 4 , step S132 can be implemented by the following way.

[0096] Sub-step S1321: detecting a first region in which all pixel values in the set of photometric images are greater than a first preset pixel value.

[0097] In this step, the first region in which all pixel values in the set of photometric images are greater than the first preset pixel value can also be referred to as an “overexposure” region.

[0098] Sub-step S1322: performing mean processing on the first region, and performing filter smoothing processing on the edges of the first region.

[0099] In this step, the “overexposure” region in the set of photometric images can be repaired, and the brightness difference between the stereo object pairs can be balanced.

[0100] Sub-step S1323: performing image quality evaluation on the set of photometric images to obtain a process quality evaluation score of each image in the set of photometric images.

[0101] In this step, the BRISQUE algorithm can be used to perform image quality evaluation on the set of photometric images, and the process quality evaluation score can evaluate whether the repair of the “overexposure” region is effective, and the photometric image with the highest process quality evaluation score is selected for the next step.

[0102] Sub-step S1324: detecting the photometric image with the highest process quality evaluation score to obtain a second region in which all pixel values in the photometric image are less than a second preset pixel value.

[0103] In this step, the second region in which all pixel values in the photometric image are less than the second preset pixel value can also be referred to as a “dark” region.

[0104] Sub-step S1325: average processing is performed on the second region, and filter smoothing processing is performed on the edge of the second region.

[0105] In this step, the photometric enhancement can be performed on the "too dark" region in the photometric image set, and the brightness difference between the stereoscopic object pairs is balanced.

[0106] Further, step S134 can be implemented by the following method.

[0107] Firstly, the image quality of the photometric image set is evaluated to obtain the first quality evaluation score of each image in the photometric image set.

[0108] Secondly, a first region in which the pixel value is greater than a first preset pixel value is detected in the photometric image set, average processing is performed on the region in which the pixel value is greater than the first preset pixel value, filter smoothing processing is performed on the edge of the first region, and the image quality of the preprocessed photometric image set is evaluated to obtain the process quality evaluation score of each image in the photometric image set.

[0109] Then, the process quality evaluation score and the first quality evaluation score are compared, if the process quality evaluation score of each image in the photometric image set is less than the first quality evaluation score, the step of performing average processing on the first region and performing filter smoothing processing on the edge of the first region is returned.

[0110] If the process quality evaluation score of each image in the photometric image set is greater than the first quality evaluation score, the photometric image with the highest process quality evaluation score is detected to obtain a second region in which the pixel value is less than a second preset pixel value in the photometric image. Average processing is performed on the second region in which the pixel value is less than the second region, and filter smoothing processing is performed on the edge of the second region. The image quality of the photometric image set is evaluated to obtain the second quality evaluation score of each image in the photometric image set.

[0111] Finally, the second quality evaluation score and the process quality evaluation score are compared, if the second quality evaluation score of each image in the photometric image set is less than the process quality evaluation score, the step of performing average processing on the second region and performing filter smoothing processing on the edge of the second region is returned.

[0112] If the second quality evaluation score of each image in the photometric image set is greater than the process quality evaluation score, the photometric processing image with the highest image quality evaluation score is output, and the photometric processing image with the highest image quality evaluation score includes the first photometric processing image and the second photometric processing image.

[0113] In this embodiment, the quality evaluation scores of two consecutive times are compared to ensure that the image preprocessing has a positive effect on the image quality, and an image with more abundant imaging texture is obtained.

[0114] In a possible implementation of the embodiment, step S15 can be implemented in the following manner.

[0115] The first multi-dimensional image and the second multi-dimensional image are subjected to feature extraction, the features extracted from the first multi-dimensional image and the second multi-dimensional image are matched, and a binocular matching image is output.

[0116] Further, the first multi-dimensional image and the second multi-dimensional image can be subjected to gray feature extraction, the gray features extracted from the first multi-dimensional image and the second multi-dimensional image are subjected to feature matching and epipolar matching, and a first depth matching result and a first matching region are obtained, wherein the first matching region is a region in which the gray feature matching of the first multi-dimensional image and the second multi-dimensional image fails.

[0117] The first matching region is subjected to normal feature extraction, the normal features extracted from the first matching region are subjected to feature matching and epipolar matching, and a second depth matching result is obtained.

[0118] Based on the first depth matching result and the second depth matching result, three-dimensional reconstruction is performed, a binocular matching image is obtained, and the binocular matching image is output.

[0119] In the embodiment, by increasing the feature matching and epipolar matching of the normal features in the first multi-dimensional image and the second multi-dimensional image, the effect of binocular matching can be improved, and the case of less texture or no texture in three-dimensional reconstruction can be solved.

[0120] The application also provides a binocular matching system, which comprises a computer device, two cameras distributed in a horizontal direction or a vertical direction, and point light sources distributed around each camera.

[0121] The camera is configured to acquire a set of light intensity images obtained by taking a target scene, and the set of light intensity images comprises a first set of light intensity images and a second set of light intensity images.

[0122] The computer device is configured to execute a binocular matching method, which will be described again with reference to Figure 1 , the binocular matching method comprising the following steps.

[0123] Step S11: acquiring, from the camera, a set of light intensity images obtained by taking a target scene, and the set of light intensity images comprises a first set of light intensity images and a second set of light intensity images.

[0124] In this step, the set of light intensity images is a collection of images with different light and shade distributions in the target scene.

[0125] Step S12: calculating a normal map of the target scene according to the set of light intensity images.

[0126] In step, the normal map is a kind of texture associated with geometric surface, the normal direction of each pixel is represented by using RGB color component, can be used to enhance the surface texture of computer graphics, can be obtained based on the reflection equation set of Lambert reflection principle.

[0127] Step S13: the photometric image set is preprocessed to obtain a photometric processing image.

[0128] In this step, the first photometric image set and the second photometric image set are preprocessed to synthesize a photometric processing image, which can enrich the texture under different illuminations in the case of less texture or no texture.In addition, preprocessing the photometric image set can effectively remove the noise of the image, and balance the brightness difference between the stereo image pairs, so that the balanced stereo image pairs can meet the requirements of subsequent stereo matching.

[0129] Step S14: image fusion is performed on the normal map and the photometric processing image to obtain a first multi-dimensional image and a second multi-dimensional image.

[0130] In this step, image fusion refers to the fusion processing of image data obtained under different conditions such as different sensors, different times, different angles of view, different radiation, etc., to obtain more accurate, comprehensive and high-quality image information.The image fusion of the normal map and the photometric processing image can obtain more accurate, comprehensive and high-quality first multi-dimensional image and second multi-dimensional image.

[0131] Step S15: binocular matching is performed on the first multi-dimensional image and the second multi-dimensional image to output a binocular matching map.

[0132] In this step, the depth matching result of the first multi-dimensional image and the second multi-dimensional image can be obtained by performing feature matching and epipolar matching on the gray features and / or normal features in the first multi-dimensional image and the second multi-dimensional image, and three-dimensional reconstruction is performed based on the depth matching result to obtain a binocular matching map.

[0133] In this embodiment, the normal map is obtained by using photometric stereo method, and the normal map is fused with the original image in the channel, so that the photometric image with only intensity feature becomes the first multi-dimensional image and the second multi-dimensional image containing normal vector, which can increase many binocular matching features and improve the matching effect, and can solve the problem of less texture or no texture in three-dimensional reconstruction.

[0134] Further, please refer to Figure 5The camera 110 includes a first camera 111 and a second camera 112, and the point light source 120 includes a first point light source 121 surrounding the first camera 111 and a second point light source 122 surrounding the second camera 112, wherein the first point light source 121 and the second point light source 122 are multiple in number. The first camera 111 is configured to take a view of a target scene when the multiple first point light sources 121 are sequentially lighted, to obtain a first set of light intensity images, and the first set of light intensity images includes light intensity images when the multiple first point light sources 121 are sequentially lighted. The second camera 112 is configured to take a view of the target scene when the multiple second point light sources 122 are sequentially lighted, to obtain a second set of light intensity images, and the second set of light intensity images includes light intensity images when the multiple second point light sources 122 are sequentially lighted.

[0135] It should be noted that the number of the first point light source 121 and the second point light source 122 can be the same or different, and the first point light source 121 and the second point light source 122 can be uniformly distributed or not uniformly distributed. For example, the number of the first point light source 121 and the second point light source 122 can be 3, and the first point light source 121 and the second point light source 122 are uniformly distributed around the first camera 111 and the second camera 112.

[0136] In the embodiment, when the relative positions of the first camera, the second camera and the target scene are fixed, the same target scene is irradiated by the point light sources in different directions, and the first camera and the second camera can capture images of the target scene with different light and shade distributions, thereby enriching the texture of the images.

[0137] The application also provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement any one of the binocular matching methods provided in the above embodiments.

[0138] In summary, the embodiments of the application provide a binocular matching method, device and computer readable storage medium. In the binocular matching method and device, the normal map is obtained by using the photometric stereo method, and the normal map and the original image are fused in the channel, so that the original light intensity image with only intensity features becomes the first multi-dimensional image and the second multi-dimensional image containing normal vectors, many binocular matching features can be increased, the matching effect can be improved, and the case of less texture or no texture in three-dimensional reconstruction can be solved.

[0139] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A binocular matching method, characterized in that: A computer device used in a binocular matching system, wherein the binocular matching system includes two cameras distributed in a horizontal direction or a vertical direction and point light sources distributed around each camera, and the method includes: Acquire a photometric image set obtained by framing a target scene from the camera, the photometric image set comprising a first photometric image set and a second photometric image set; Calculating a normal map of the target scene based on the photometric image set; Preprocessing the photometric image set to obtain a photometrically processed image; The step of preprocessing the photometric image set includes: performing image quality assessment on the photometric image set to obtain a first quality assessment score of the photometric image set; Detecting areas in the photometric image set where pixel values ​​are greater than a first preset pixel value or less than a second preset pixel value, and preprocessing areas in the photometric image set where pixel values ​​are greater than the first preset pixel value or less than the second preset pixel value. The step of preprocessing an area in the photometric image set whose pixel value is greater than a first preset pixel value or less than a second preset pixel value comprises: detecting a first region in the photometric image set where all pixel values ​​are greater than the first preset pixel value; performing averaging processing on the first region and performing filtering and smoothing processing on the edge of the first region; Performing image quality assessment on the photometric image set to obtain a process quality assessment score for each image in the photometric image set; Detecting the photometric image with the highest process quality assessment score to obtain a second region in the photometric image where all pixel values ​​are smaller than the second preset pixel value; performing averaging processing on the second region and performing filtering and smoothing processing on the edge of the second region; Wherein, the first preset pixel value is greater than the second preset pixel value; performing image quality assessment on the preprocessed photometric image set to obtain a second quality assessment score of the photometric image set; comparing the first quality assessment score and the second quality assessment score, and outputting a photometrically processed image with a highest image quality assessment score, wherein the photometrically processed image with the highest image quality assessment score includes the first photometrically processed image and the second photometrically processed image; Performing image fusion on the normal map and the photometrically processed image to obtain a first multidimensional image and a second multidimensional image; Perform binocular matching on the first multi-dimensional image and the second multi-dimensional image, and output a binocular matching map.

2. The binocular matching method according to claim 1, wherein The step of calculating the normal map of the target scene based on the photometric image set includes: Configuring the light source direction and light source intensity parameters in the target scene; Based on the light source direction and the light source intensity parameters, pixel-by-pixel normal calculation is performed on the photometric image set, and a normal map of the target scene is output, where the normal map includes a first normal map corresponding to the first photometric image set and a second normal map corresponding to the second photometric image set.

3. The binocular matching method according to claim 1, wherein: The step of comparing the first quality assessment score and the second quality assessment score and outputting the photometrically processed image with the highest image quality assessment score, wherein the photometrically processed image with the highest image quality assessment score includes the first photometrically processed image and the second photometrically processed image, comprises: Performing image quality assessment on the photometric image set to obtain a first quality assessment score for each image in the photometric image set; Detecting a first region in the photometric image set where all pixel values ​​are greater than the first preset pixel value; Performing an averaging process on the first area and performing a filtering and smoothing process on the edge of the first area; Performing image quality assessment on the preprocessed photometric image set to obtain a process quality assessment score for each image in the photometric image set; comparing the process quality assessment score with the first quality assessment score; if the process quality assessment score of each image in the photometric image set is less than the first quality assessment score, returning to the step of performing averaging processing on the first region and performing filtering and smoothing processing on the edge of the first region; If the process quality assessment score of each image in the photometric image set is greater than the first quality assessment score, detecting the photometric image with the highest process quality assessment score to obtain a second region in the photometric image where all pixel values ​​are smaller than the second preset pixel value; Performing averaging processing on all pixel values ​​smaller than the second region, and performing filtering and smoothing processing on the edge of the second region; Performing image quality assessment on the photometric image set to obtain a second quality assessment score for each image in the photometric image set; comparing the second quality assessment score with the process quality assessment score; if the second quality assessment score of each image in the photometric image set is less than the process quality assessment score, returning to the step of performing averaging processing on the second area and performing filtering and smoothing processing on the edge of the second area; If the second quality assessment score of each image in the photometric image set is greater than the process quality assessment score, the photometrically processed image with the highest image quality assessment score is output, and the photometrically processed image with the highest image quality assessment score includes the first photometrically processed image and the second photometrically processed image.

4. The binocular matching method according to claim 1, wherein: The step of performing binocular matching on the first multi-dimensional image and the second multi-dimensional image and outputting a binocular matching map includes: Feature extraction is performed on the first multi-dimensional image and the second multi-dimensional image, the features extracted from the first multi-dimensional image and the second multi-dimensional image are matched, and a binocular matching map is output.

5. The binocular matching method according to claim 4, wherein: The step of extracting features from the first multidimensional image and the second multidimensional image, matching the features extracted from the first multidimensional image and the second multidimensional image, and outputting a binocular matching map includes: performing grayscale feature extraction on the first multidimensional image and the second multidimensional image, and performing feature matching and epipolar matching on the grayscale features extracted from the first multidimensional image and the second multidimensional image to obtain a first depth matching result and a first matching area, wherein the first matching area is an area in the first multidimensional image and the second multidimensional image where grayscale feature matching fails; performing normal feature extraction on the first matching area, and performing feature matching and epipolar matching on the normal features extracted from the first matching area to obtain a second depth matching result; Perform three-dimensional reconstruction based on the first depth matching result and the second depth matching result to obtain a binocular matching map, and output the binocular matching map.

6. A binocular matching system, characterized in that: The system includes a computer device, two cameras distributed in a horizontal direction or a vertical direction, and point light sources distributed around each camera; The camera is used to obtain a photometric image set obtained by framing a target scene, wherein the photometric image set includes a first photometric image set and a second photometric image set; The computer device is used to perform a binocular matching method, and the binocular matching method includes: Acquire a photometric image set obtained by framing a target scene from the camera, the photometric image set comprising a first photometric image set and a second photometric image set; Calculating a normal map of the target scene based on the photometric image set; Preprocessing the photometric image set to obtain a photometrically processed image; The step of preprocessing the photometric image set includes: performing image quality assessment on the photometric image set to obtain a first quality assessment score of the photometric image set; Detecting areas in the photometric image set where pixel values ​​are greater than a first preset pixel value or less than a second preset pixel value, and preprocessing areas in the photometric image set where pixel values ​​are greater than the first preset pixel value or less than the second preset pixel value. The step of preprocessing an area in the photometric image set whose pixel value is greater than a first preset pixel value or less than a second preset pixel value comprises: detecting a first region in the photometric image set where all pixel values ​​are greater than the first preset pixel value; performing averaging processing on the first region and performing filtering and smoothing processing on the edge of the first region; Performing image quality assessment on the photometric image set to obtain a process quality assessment score for each image in the photometric image set; Detecting the photometric image with the highest process quality assessment score to obtain a second region in the photometric image where all pixel values ​​are smaller than the second preset pixel value; performing averaging processing on the second region and performing filtering and smoothing processing on the edge of the second region; Wherein, the first preset pixel value is greater than the second preset pixel value; performing image quality assessment on the preprocessed photometric image set to obtain a second quality assessment score of the photometric image set; Comparing the first quality assessment score and the second quality assessment score, outputting a photometrically processed image with the highest image quality assessment score, wherein the photometrically processed image with the highest image quality assessment score includes the first photometrically processed image and the second photometrically processed image; performing image fusion on the normal map and the photometrically processed image to obtain a first multidimensional image and a second multidimensional image; Perform binocular matching on the first multi-dimensional image and the second multi-dimensional image, and output a binocular matching map.

7. The binocular matching system according to claim 6, characterized in that: The camera includes a first camera and a second camera, the point light source includes a first point light source surrounding the first camera and a second point light source surrounding the second camera, wherein the number of the first point light source and the number of the second point light source are multiple; The first camera is configured to frame a target scene when the plurality of first point light sources are sequentially illuminated, to obtain the first photometric image set, wherein the first photometric image set includes photometric images when the plurality of first point light sources are sequentially illuminated; The second camera is used to frame the target scene when multiple second point light sources are lit in sequence, and obtain the second photometric image set, which includes photometric images when multiple second point light sources are lit in sequence.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the binocular matching method according to any one of claims 1 to 5 is implemented.

Images