Filtering system, method, device and storage medium for reconstructing a view based on a depth map

Abstract

The application provides a view filtering method and system based on depth map reconstruction, a device and a storage medium. The method comprises the following steps: obtaining a depth map based on a left-view original image and a right-view original image; converting each valid depth value in the depth map into disparity information of a corresponding pixel, generating a simulated image of a second view based on the original image of the first view and the disparity information; obtaining an error value of each pixel according to the simulated image of the second view and the original image of the second view; generating an error threshold value according to the error value, collecting pixels with error values less than the error threshold value to generate a pixel set to be filtered, and filtering the valid depth values corresponding to the pixel set to be filtered in the depth map. The application can generate a simulated image of another view based on the original image of one view and the disparity information, compare the simulated image with the corresponding original image, filter out abnormal pixel points, and improve the accuracy of the depth map.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a filtering method, system, device, and storage medium for reconstructing views based on depth maps. Background Technology

[0002] Stereo matching has always been a research hotspot in binocular vision. Binocular cameras capture two viewpoint images of the same scene from the left and right perspectives, and stereo matching algorithms are used to obtain disparity maps, which in turn generate depth maps. Depth maps have a wide range of applications. Because they can record the distance between objects in a scene and the camera, they can be used for measurement, 3D reconstruction, and the synthesis of virtual viewpoints.

[0003] The purpose of stereo matching vision algorithms is to generate depth maps, such as BM (Bidirectional Matching), SGBM (Stereo Processing by Semiglobal Matching and Mutual Information), and GC (Graph cuts), which will not be elaborated on here. Due to inherent limitations in these algorithms, the generated depth maps suffer from discontinuities, noise, and trailing (broom-like pixels at object edges). A typical problem is also the generation of false positives (objects appearing in depth regions where they don't exist). General depth map denoising and enhancement methods employ filtering techniques, such as bilinear filtering and spatiotemporal filtering, but these methods cannot fundamentally solve the problems of false positives and trailing.

[0004] In particular, 3D videos generated based on the aforementioned depth maps are prone to distortion when viewed by users due to issues such as incorrect objects and trailing problems, which greatly reduces the simulated visual effect of 3D videos.

[0005] In view of this, the present invention provides a filtering method, system, device and storage medium for reconstructing views based on depth maps.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] To address the problems in the prior art, the present invention aims to provide a filtering method, system, device, and storage medium for reconstructing views based on depth maps. This invention overcomes the difficulties of the prior art and can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information. By comparing the simulated image with the corresponding native image, abnormal pixels can be filtered out, thereby improving the accuracy of the depth map.

[0008] Embodiments of the present invention provide a filtering method for reconstructing a view based on a depth map, comprising the following steps:

[0009] Depth maps are obtained based on the original images from the left and right viewpoints;

[0010] Each valid depth value in the depth map is converted into disparity information for the corresponding pixel, and a simulated image of the second viewpoint is generated based on the native image of the first viewpoint and the disparity information.

[0011] The error value for each pixel is obtained based on the simulated image from the second viewpoint and the native image from the second viewpoint; and

[0012] An error threshold is generated based on the error value, and pixels with error values ​​less than the error threshold are collected to generate a set of pixels to be filtered. The effective depth values ​​corresponding to the set of pixels to be filtered are then filtered in the depth map.

[0013] Preferably, obtaining the depth map based on the original left-viewpoint image and the original right-viewpoint image includes:

[0014] Obtain the original left-viewpoint and original right-viewpoint images using a binocular device;

[0015] The disparity is obtained using a binocular stereo matching algorithm based on the original left-viewpoint image and the original right-viewpoint image.

[0016] The parallax is converted into a depth map based on the baseline information and focal length information between the two eyes.

[0017] Preferably, the binocular device is a binocular visible light sensor or a binocular infrared sensor.

[0018] Preferably, in the step of converting each effective depth value in the depth map into disparity information of the corresponding pixel, and generating a simulated image of the second viewpoint based on the native image of the first viewpoint and the disparity information, the first viewpoint is a left viewpoint and the second viewpoint is a right viewpoint; or, the first viewpoint is a right viewpoint and the second viewpoint is a left viewpoint.

[0019] Preferably, in the step of obtaining the error value of each pixel based on the simulated image and the original image of the second viewpoint, the formula for obtaining the error value is as follows:

[0020]

[0021] Among them, I i These are the pixel values ​​of the simulated image.

[0022] These are the pixel values ​​of the original image.

[0023] α is a preset variable, and its value ranges from 0.8 to 0.9. In this embodiment, the value of α is 0.85, but it is not limited to this value.

[0024] SSIM stands for Structural Similarity Algorithm, and it performs calculations in a 3x3 domain. SSIM is a metric for measuring the similarity between two images. The SSIM algorithm is primarily used to detect the similarity between two images of the same size, or to detect the degree of image distortion. In the original paper, the SSIM algorithm mainly compares the luminance, contrast, and structure of two images separately, then weights these three factors and represents the result as a product; this will not be elaborated upon here.

[0025] Preferably, the step of generating an error threshold based on the error value, collecting pixels with error values ​​less than the error threshold to generate a set of pixels to be filtered, and filtering the effective depth values ​​corresponding to the set of pixels to be filtered in the depth map includes:

[0026] The median or average error value of all the pixels is obtained as the error threshold;

[0027] Collect pixels whose error values ​​are less than the error threshold to generate a set of pixels to be deleted;

[0028] Delete the depth value corresponding to the set of pixels to be deleted in the depth map.

[0029] Preferably, the step of generating an error threshold based on the error value, collecting pixels with error values ​​less than the error threshold to generate a set of pixels to be filtered, and filtering the effective depth values ​​corresponding to the set of pixels to be filtered in the depth map includes:

[0030] The average error value of all pixels within a preset range centered on each pixel is obtained as the corresponding error threshold for that pixel;

[0031] Collect pixels whose error values ​​are less than the corresponding error threshold to generate a set of pixels to be deleted;

[0032] Delete the depth value corresponding to the set of pixels to be deleted in the depth map.

[0033] Preferably, the method further includes the following steps:

[0034] The effective depth value of the filtered pixel is reconstructed using the effective depth values ​​of the surrounding pixels.

[0035] Embodiments of the present invention also provide a filtering method for reconstructing a view based on a depth map, characterized by comprising the following steps:

[0036] Capture a 3D video using a binoculars device;

[0037] The original left-view image and the original right-view image obtained at the same time stamp in the 3D video are filtered using a depth map-based view reconstruction filtering method; and

[0038] 3D video is generated based on the filtered depth map.

[0039] Embodiments of the present invention also provide a filtering system for reconstructing views based on depth maps, used to implement the above-described filtering method for reconstructing views based on depth maps. The filtering system for reconstructing views based on depth maps includes:

[0040] The viewpoint depth module obtains a depth map based on the original images of the left and right viewpoints;

[0041] The disparity information module converts each effective depth value in the depth map into disparity information for the corresponding pixel, and generates a simulated image of the second viewpoint based on the native image of the first viewpoint and the disparity information.

[0042] The pixel error module obtains the error value for each pixel based on the simulated image and the native image of the second viewpoint; and

[0043] The depth filtering module generates an error threshold based on the error value, collects pixels whose error value is less than the error threshold to generate a set of pixels to be filtered, and filters the effective depth values ​​corresponding to the set of pixels to be filtered in the depth map.

[0044] Embodiments of the present invention also provide a filtering device for reconstructing a view based on a depth map, comprising:

[0045] processor;

[0046] Memory, which stores the processor's executable instructions;

[0047] The processor is configured to execute the steps of the aforementioned filtering method for reconstructing a view based on a depth map by executing executable instructions.

[0048] Embodiments of the present invention also provide a computer-readable storage medium for storing a program that, when executed, implements the steps of the above-described filtering method for reconstructing a view based on a depth map.

[0049] The filtering method, system, device, and storage medium for reconstructing views based on depth maps of the present invention can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map. Attached Figure Description

[0050] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart of the filtering method for reconstructing a view based on a depth map according to the present invention.

[0052] Figure 2 and 3 This is a schematic diagram illustrating the implementation process of the filtering method for reconstructing a view based on a depth map according to the present invention.

[0053] Figure 4 This is a schematic diagram of the filtering system for reconstructing a view based on a depth map according to the present invention.

[0054] Figure 5 This is a schematic diagram of the filtering device for reconstructing a view based on a depth map according to the present invention.

[0055] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0056] The following specific examples illustrate the implementation methods of this application. Those skilled in the art can easily understand the other advantages and effects of this application from the content disclosed herein. This application can also be implemented or applied through other different specific embodiments, and various details in this application can be modified or changed according to different viewpoints and application systems without departing from the spirit of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0057] The embodiments of this application will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the application. This application may be embodied in many different forms and is not limited to the embodiments described herein.

[0058] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics represented in connection with that embodiment or example, which are included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples represented in this application, as well as features of different embodiments or examples.

[0059] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0060] For the purpose of clearly describing this application, devices that are not relevant to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0061] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0062] When we say that a device is "above" another device, this can mean that it is directly above the other device, or it can mean that other devices are present in between. Conversely, when we say that a device is "directly" "above" another device, there are no other devices present in between.

[0063] While the terms first, second, etc., are used in some instances to refer to various components in this invention, these components should not be limited by these terms. These terms are used only to distinguish one component from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of features, steps, operations, components, elements, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, components, elements, kinds, and / or groups. The terms “or” and “and / or” as used herein are to be interpreted inclusively, or mean any one or any combination thereof. Therefore, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will only occur if the combination of components, functions, steps, or operations is inherently mutually exclusive in some way.

[0064] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this application. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in the specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0065] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the content of this present application, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0066] Figure 1 This is a flowchart of the filtering method for reconstructing a view based on a depth map according to the present invention. Figure 1 As shown, an embodiment of the present invention provides a filtering method for reconstructing a view based on a depth map, comprising the following steps:

[0067] S110. Obtain a depth map based on the original images from the left and right viewpoints.

[0068] S120. Convert each valid depth value in the depth map into disparity information of the corresponding pixel, and generate a simulated image of the second viewpoint based on the original image of the first viewpoint and the disparity information.

[0069] S130. Obtain the error value of each pixel based on the simulated image of the second viewpoint and the original image of the second viewpoint.

[0070] S140. Generate an error threshold based on the error value, collect pixels with error values ​​less than the error threshold to generate a set of pixels to be filtered, and filter the effective depth values ​​corresponding to the set of pixels to be filtered in the depth map.

[0071] In a preferred embodiment, step S110 includes:

[0072] S111. Obtain the original image of the left viewpoint and the original image of the right viewpoint based on a binocular device, wherein the binocular device is a binocular visible light sensor or a binocular infrared sensor.

[0073] S112. Based on the original images of the left and right viewpoints, disparity is obtained using a binocular stereo matching algorithm. This invention may employ existing or future methods to obtain disparity, but is not limited to these. For example, binocular stereo matching can be divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization.

[0074] Step 1: Matching Cost Calculation. The purpose of matching cost calculation is to measure the correlation between the pixel to be matched and the candidate pixels. Regardless of whether two pixels are identical points, the matching cost function can be used to calculate the matching cost. The smaller the cost, the greater the correlation and the higher the probability that they are identical points.

[0075] Step 2 Cost Aggregation. The fundamental purpose of cost aggregation is to ensure that the cost value accurately reflects the correlation between pixels. The previous step of matching cost calculation often only considers local information, calculating the cost value through pixel information within a certain size window in the neighborhood of two pixels. This is easily affected by image noise, and when the image is in a weak or repetitive texture area, this cost value may not accurately reflect the correlation between pixels. The direct manifestation is that the cost value of the true corresponding points is not the minimum.

[0076] Step 3: Disparity Calculation. Disparity calculation involves determining the optimal disparity value for each pixel using the cost matrix after cost aggregation. This is typically done using the Winner-Takes-All (WTA) algorithm. Figure 2 As shown, among the costs of all disparities for a given pixel, the disparity corresponding to the minimum cost is selected as the optimal disparity. This step is very simple, which means that the value of the aggregated cost matrix S must accurately reflect the correlation between pixels. It also shows that the previous cost aggregation step is a crucial step in stereo matching, directly determining the accuracy of the algorithm.

[0077] Step 4: Disparity Optimization. The purpose of disparity optimization is to further improve the quality of the disparity map obtained in the previous step. This includes steps such as removing erroneous disparities, appropriate smoothing, and sub-pixel precision optimization. Generally, a left-right consistency check algorithm is used to remove erroneous disparities caused by occlusion and noise; a small connected region removal algorithm is used to remove isolated outliers; and smoothing algorithms such as median filtering and bilateral filtering are used to smooth the disparity map. In addition, other effective methods to improve the quality of the disparity map, such as robust plane fitting, intensity consistency constraints, and locally consistency constraints, are also commonly used.

[0078] S113. The disparity is converted into a depth map based on the baseline information (prior parameters) and focal length information (prior parameters) between the two cameras. In this embodiment, the depth disparity conversion formula is:

[0079] Z = (B*f) / S;

[0080] Where Z is depth, f is the focal length of the stereo camera, B is the optical center distance of the stereo camera, and S is the average parallax, but not limited to these values.

[0081] In a preferred embodiment, in step S120, the first viewpoint is a left viewpoint and the second viewpoint is a right viewpoint. Alternatively, the first viewpoint is a right viewpoint and the second viewpoint is a left viewpoint, but this is not a limitation.

[0082] In a preferred embodiment, the formula for obtaining the error value in step S130 is:

[0083]

[0084] Among them, I i These are the pixel values ​​of the simulated image.

[0085] These are the pixel values ​​of the original image.

[0086] α is a preset variable, and the value of α ranges from 0.8 to 0.9. In this embodiment, the value of α is one of 0.83, 0.84, 0.85, 0.86, and 0.87.

[0087] SSIM stands for Structural Similarity Algorithm, and it is computed in a 3x3 domain, but is not limited thereto.

[0088] In a preferred embodiment, step S140 includes:

[0089] S141. Obtain the median or average error value of all pixels as the error threshold.

[0090] S142. Collect pixels with error values ​​less than the error threshold and generate a set of pixels to be deleted.

[0091] S143. Delete the depth values ​​corresponding to the set of pixels to be deleted in the depth map, but this is not a limitation.

[0092] In a preferred embodiment, step S140 includes:

[0093] S146. Obtain the average error value of all pixels within a preset range centered on each pixel as the corresponding error threshold for that pixel.

[0094] S147. Collect pixels with error values ​​less than the corresponding error threshold and generate a set of pixels to be deleted.

[0095] S148. Delete the depth values ​​corresponding to the set of pixels to be deleted in the depth map, but not limited to this.

[0096] In a preferred embodiment, the following steps are also included:

[0097] S150. Reconstruct the effective depth value of the filtered pixel using the effective depth values ​​of the surrounding pixels, but this is not a limitation. In this embodiment, the effective depth value of the filtered pixel can be simulated using the effective depth values ​​of the surrounding pixels through existing compensation algorithms. The relevant algorithms will not be described in detail here, but are not limited thereto.

[0098] The filtering method for reconstructing a view based on a depth map of the present invention can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0099] The specific implementation process of this invention is as follows:

[0100] Figure 2 and 3 This is a schematic diagram illustrating the implementation process of the filtering method for reconstructing a view based on a depth map according to the present invention. See also... Figure 2As shown, a left-view native image 21 is obtained based on the left viewpoint 11 of a binocular device (where the position of the observed object observed based on the left viewpoint 11 is located at the position of icon 24), and a right-view native image 22 is obtained based on the right viewpoint 12 of the binocular device (where the position of the observed object observed based on the right viewpoint 12 is located at the position of icon 25). Disparity is obtained using a binocular stereo matching algorithm based on the left-view native image 21 and the right-view native image 22. The disparity is converted into a depth map 23 based on the baseline information and focal length information between the two cameras. The input to the binocular stereo matching algorithm is natural headshots or infrared images captured by the left and right cameras. The output of the algorithm is disparity, which is converted into a depth map 23 using the baseline and focal length between the two cameras. Therefore, there is a one-to-one correspondence between disparity and depth map 23, and the conversion formula is shown in (Formula 1). Disparity is the difference in the horizontal direction between corresponding pixels of the same object in the left and right images. In theory, given the left image (or the right image) and the disparity map (corresponding to the depth map), the reconstructed right image can be obtained by shifting the disparity distance of each pixel in the left image. If the disparity map is very accurate, the reconstructed right image and the captured right image will be completely identical.

[0101] Each valid depth value in the depth map 23 is converted into disparity information for the corresponding pixel, and a simulated image 22' for the right viewpoint is generated based on the original image 21 of the left viewpoint and the disparity information.

[0102] Determining whether the disparity of each pixel is reasonable requires first establishing a reconstruction error formula. A typical formula is DSSIM+L1. For example, based on the simulated image 22' of the right viewpoint and the native image 22 of the right viewpoint, the error value of each pixel is obtained using the following formula:

[0103]

[0104] Among them, I i The pixel values ​​of the simulated image 22' from the right viewpoint. represents the pixel value of the original image from the right viewpoint. α is a preset variable with a value of 0.85. SSIM stands for Structural Similarity Algorithm, and it is calculated within a 3x3 neighborhood. DSSIM refers to Differential Structural Similarity (DSSIM).

[0105] Then, the median or average of the error values ​​of all pixels is obtained as the error threshold. Pixels with error values ​​less than the error threshold are collected to generate a set of pixels to be deleted, 26. The depth values ​​corresponding to the set of pixels to be deleted are deleted in the depth map 23, and the rest are retained to obtain the filtered depth map 27. Alternatively, in a variation, the average of the error values ​​of all pixels within a preset range centered on each pixel is obtained as the corresponding error threshold for that pixel. Pixels with error values ​​less than the corresponding error threshold are collected to generate a set of pixels to be deleted. The depth values ​​corresponding to the set of pixels to be deleted are deleted in the depth map. The method of this invention is run only once for each depth map.

[0106] Finally, the effective depth value of the filtered pixel is reconstructed using the effective depth values ​​of the surrounding pixels.

[0107] In this example, by checking the difference between the reconstructed right image and the original right image, it is determined whether the parallax of each pixel is reasonable. If it is unreasonable, it is discarded, thus effectively removing trailing and erroneous objects.

[0108] This invention utilizes reconstruction errors to filter depth maps, where the filtering threshold can be dynamically determined or based on empirical values. The algorithm of this invention can reconstruct a right image from a left image, or vice versa. Alternatively, the algorithm of this invention can be combined with other filtering algorithms.

[0109] The filtering method for reconstructing a view based on a depth map of the present invention can generate a simulated image 22' of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image 22' with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0110] Embodiments of the present invention also provide a filtering method for reconstructing a view based on a depth map, characterized by comprising the following steps:

[0111] S210, Capture a 3D video using a binocular device.

[0112] S220. For the original left-viewpoint and original right-viewpoint images obtained at the same timestamp in the 3D video, a filtering method based on depth map-reconstructed view is used for filtering.

[0113] as well as

[0114] S230. Generate a 3D video based on the filtered depth map. The relevant technical features are as described above and will not be repeated here.

[0115] This invention proposes a novel depth map filtering scheme based on the reconstruction of left and right images. Using the generated depth map, the right image is reconstructed from the left image (or vice versa). By comparing the reconstructed right image with the original right image, abnormal pixels (such as incorrect objects and trailing artifacts) are filtered out. The 3D video generated by this invention ensures the continuity of the depth map, effectively removing noise, incorrect objects, and trailing artifacts (pixels resembling a comet's broom at the edges of objects), making it easier for users to feel immersed and greatly improving the simulated visual effect of the 3D video.

[0116] Figure 4 This is a schematic diagram of the filtering system for reconstructing a view based on a depth map according to the present invention. Figure 4 As shown, the filtering system 5 for reconstructing a view based on a depth map according to the present invention includes:

[0117] The viewpoint depth module 51 obtains a depth map based on the original images of the left and right viewpoints.

[0118] The disparity information module 52 converts each valid depth value in the depth map into disparity information for the corresponding pixel, and generates a simulated image of the second viewpoint based on the original image of the first viewpoint and the disparity information.

[0119] The pixel error module 53 obtains the error value of each pixel based on the simulated image of the second viewpoint and the original image of the second viewpoint.

[0120] The depth filtering module 54 generates an error threshold based on the error value, collects pixels with error values ​​less than the error threshold to generate a set of pixels to be filtered, and filters the effective depth values ​​corresponding to the set of pixels to be filtered in the depth map.

[0121] In a preferred embodiment, the viewpoint depth module 51 is configured to acquire a left-viewpoint native image and a right-viewpoint native image based on a binocular device. The disparity is obtained using a binocular stereo matching algorithm based on the left-viewpoint native image and the right-viewpoint native image. The disparity is then converted into a depth map based on baseline information and focal length information between the two binocular devices.

[0122] In a preferred embodiment, the binocular device is a binocular visible light sensor or a binocular infrared sensor, but is not limited thereto.

[0123] In a preferred embodiment, the parallax information module 52 is configured such that the first viewpoint is a left viewpoint and the second viewpoint is a right viewpoint. Alternatively, the first viewpoint is a right viewpoint and the second viewpoint is a left viewpoint.

[0124] In a preferred embodiment, the formula for obtaining the error value in the pixel error module 53 is:

[0125]

[0126] Among them, I i These are the pixel values ​​of the simulated image.

[0127] These are the pixel values ​​of the original image.

[0128] α is a preset variable, and the value of α ranges from 0.8 to 0.9.

[0129] SSIM stands for Structural Similarity Algorithm, and it is computed in a 3x3 domain, but is not limited thereto.

[0130] In a preferred embodiment, the depth filtering module 54 is configured to obtain the median or average of the error values ​​of all pixels as an error threshold. Pixels with error values ​​less than the error threshold are collected to generate a set of pixels to be deleted. The depth values ​​corresponding to the set of pixels to be deleted are deleted from the depth map, but this is not a limitation.

[0131] In a preferred embodiment, the depth filtering module 54 is configured to obtain the average error value of all pixels within a preset range centered on each pixel as the corresponding error threshold for that pixel. Pixels with error values ​​less than the corresponding error threshold are collected to generate a set of pixels to be deleted. The depth values ​​corresponding to the set of pixels to be deleted are deleted from the depth map, but this is not a limitation.

[0132] In a preferred embodiment, the system further includes a depth compensation module, which reconstructs the effective depth value of the filtered pixel using the effective depth values ​​of the surrounding pixels, but is not limited thereto.

[0133] The filtering system for reconstructing a view based on a depth map of the present invention can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0134] This invention also provides a filtering device for reconstructing a view based on a depth map, including a processor and a memory storing executable instructions for the processor. The processor is configured to execute steps of a filtering method for reconstructing a view based on a depth map by executing the executable instructions.

[0135] As described above, the filtering device for reconstructing a view based on a depth map of the present invention can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0136] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0137] Figure 5 This is a schematic diagram of the filtering device for reconstructing a view based on a depth map according to the present invention. See below for reference. Figure 5 To describe an electronic device 600 according to this embodiment of the present invention. Figure 5 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0138] like Figure 5 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0139] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the above-described section on the electronic prescription transfer processing method according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0140] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0141] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0142] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0143] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0144] This invention also provides a computer-readable storage medium for storing a program that, when executed, implements the steps of a filtering method for reconstructing a view based on a depth map. In some possible implementations, various aspects of the invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described electronic prescription processing method section of this specification according to various exemplary embodiments of the invention.

[0145] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, it can generate a simulated image of another viewpoint from the native image of one viewpoint and the parallax information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0146] Figure 6 This is a schematic diagram of the structure of the computer-readable storage medium of the present invention. (Reference) Figure 6 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0149] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0150] In summary, the filtering method, system, device, and storage medium for reconstructing views based on depth maps of the present invention can generate a simulated image of another viewpoint from the native image of one viewpoint and disparity information, and compare the simulated image with the corresponding native image to filter out abnormal pixels and improve the accuracy of the depth map.

[0151] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A filtering method for reconstructing views based on depth maps, characterized in that, Includes the following steps: A depth map is generated based on the original images from the left and right viewpoints using a stereo matching algorithm. Each valid depth value in the depth map is converted into disparity information for the corresponding pixel, and a simulated image of the second viewpoint is generated based on the native image of the first viewpoint and the disparity information. Based on the simulated image and the original image of the second viewpoint, an error value for each pixel is obtained based on pixel value differences and structural similarity. The formula for obtaining the error value is as follows: （1） in, These are the pixel values ​​of a simulated image. These are the pixel values ​​of the original image; α is a preset variable, and the value of α ranges from 0.8 to 0.9; SSIM stands for Structural Similarity Algorithm, and it is computed within a 3x3 neighborhood; and The average error value of all pixels within a preset range centered on each pixel is obtained as the corresponding error threshold for that pixel; pixels with error values ​​less than the corresponding error threshold are collected to generate a set of pixels to be deleted; the depth values ​​corresponding to the set of pixels to be deleted are deleted from the depth map to achieve filtering of the depth map; The effective depth value of the filtered pixel is reconstructed using the effective depth values ​​of the surrounding pixels.

2. The filtering method for reconstructing a view based on a depth map according to claim 1, characterized in that, The process of generating a depth map based on the original left-viewpoint and right-viewpoint images using a binocular stereo matching algorithm includes: Obtain the original left-viewpoint and original right-viewpoint images using a binocular device; The disparity is obtained using a binocular stereo matching algorithm based on the original left-viewpoint image and the original right-viewpoint image. The parallax is converted into a depth map based on the baseline information and focal length information between the two eyes.

3. The filtering method for reconstructing a view based on a depth map according to claim 2, characterized in that, The binocular device is a binocular visible light sensor or a binocular infrared sensor.

4. The filtering method for reconstructing a view based on a depth map according to claim 1, characterized in that, In the step of converting each effective depth value in the depth map into disparity information of the corresponding pixel, and generating a simulated image of the second viewpoint based on the native image of the first viewpoint and the disparity information, the first viewpoint is a left viewpoint and the second viewpoint is a right viewpoint; or, the first viewpoint is a right viewpoint and the second viewpoint is a left viewpoint.

5. A filtering method for reconstructing a view based on a depth map, characterized in that, Includes the following steps: Capture a 3D video using a binoculars device; The original left-viewpoint image and the original right-viewpoint image obtained at the same timestamp in the three-dimensional video are filtered using the depth map-based view reconstruction filtering method as described in claim 1. as well as 3D video is generated based on the filtered depth map.

6. A filtering system for reconstructing views based on depth maps, characterized in that, The system includes: The viewpoint depth module generates a depth map based on the original images of the left and right viewpoints using a binocular stereo matching algorithm. The disparity information module converts each effective depth value in the depth map into disparity information for the corresponding pixel, and generates a simulated image of the second viewpoint based on the native image of the first viewpoint and the disparity information. The pixel error module obtains the error value for each pixel based on the difference in pixel values ​​and structural similarity between the simulated image and the original image of the second viewpoint. The formula for obtaining the error value is as follows: （1） in, These are the pixel values ​​of a simulated image. These are the pixel values ​​of the original image; α is a preset variable, and the value of α ranges from 0.8 to 0.9; SSIM stands for Structural Similarity Algorithm, and it is computed within a 3x3 neighborhood; and The depth filtering module obtains the average error value of all pixels within a preset range centered on each pixel as the corresponding error threshold for that pixel; collects pixels whose error values ​​are less than the corresponding error threshold to generate a set of pixels to be deleted; and deletes the depth value corresponding to the set of pixels to be deleted from the depth map. The depth compensation module reconstructs the effective depth value of the filtered pixel using the effective depth values ​​of the surrounding pixels.

7. A filtering device for reconstructing views based on depth maps, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the filtering method for reconstructing a view based on a depth map as described in any one of claims 1 to 4 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the filtering method for reconstructing a view based on a depth map as described in any one of claims 1 to 4.

Images