Image processing method and device

Abstract

The present disclosure provides an image processing method and apparatus. The method comprises: obtaining an initial depth image corresponding to an original image, obtaining pixel value similarity between adjacent pixels in the original image, adjusting depth values ​​in the initial depth image based on the pixel value similarity, and obtaining a target depth image corresponding to the original image. Compared to the initial depth image, the original image has characteristics such as clear boundaries of the captured object. By adjusting the depth values ​​in the initial depth image based on the pixel value similarity between adjacent pixels in the original image, the depth information of the boundaries of the captured object in the initial depth image can be improved, thereby obtaining a high-quality target depth image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer communication technology, and in particular to an image processing method and device. Background Art

[0002] With the development of computer vision technology, especially 3D vision, computer vision systems have an increasing demand for obtaining high-quality depth information to achieve functions such as background blur, AR ruler, and AR special effects.

[0003] Currently, depth images are acquired using technologies such as TOF (Time of Flight), structured light, and binocular vision. However, these methods suffer from numerous issues, such as loss of depth information at the object's boundaries, making it difficult to obtain high-quality depth images. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image processing method and apparatus.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0006] Get the initial depth image corresponding to the original image;

[0007] Obtaining pixel value similarity between adjacent pixels in the original image;

[0008] The depth values ​​in the initial depth image are adjusted according to the pixel value similarity to obtain a target depth image corresponding to the original image.

[0009] Optionally, the initial depth image includes a set of pixels with a depth value of zero; and adjusting the depth value in the initial depth image according to the pixel value similarity includes:

[0010] The depth values ​​of the pixels in the pixel set are adjusted according to the pixel value similarity.

[0011] Optionally, adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image includes:

[0012] For a pixel in the initial depth image, determining a weight used by the pixel in the initial depth image based on a similarity in pixel values ​​corresponding to a target pixel in the original image, the target pixel being located at the same pixel position as the pixel;

[0013] Constructing an objective function, the objective function including a first data item, a second data item, and a third data item, wherein the first data item is used to represent a first difference, which is a difference between an optimized depth value and an actual depth value of the pixel in the initial depth image; the second data item is used to represent a second difference, which is a difference between the optimized depth value of the pixel and an optimized depth value of an adjacent pixel in the initial depth image; and the third data item is used to represent a weight used by the pixel in the initial depth image;

[0014] The optimized depth value of each pixel in the objective function is adjusted, and in response to the adjusted objective function satisfying a preset condition, the target depth image is obtained according to the current optimized depth value of each pixel.

[0015] Optionally, the weight used by the pixel point is negatively correlated with the similarity of the pixel value corresponding to the target pixel point.

[0016] Optionally, obtaining the target depth image according to the current optimized depth value of each pixel point includes:

[0017] For each pixel in the initial depth image, determining a deviation between a current optimized depth value of the pixel and an actual depth value;

[0018] In response to the deviation not falling within the preset deviation range, adjusting the current optimized depth value of the pixel point to the actual depth value;

[0019] The target depth image is obtained according to the actual depth value of the pixel point.

[0020] Optionally, adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image includes:

[0021] Adjusting the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image;

[0022] Denoising is performed on the intermediate depth image to obtain the target depth image.

[0023] Optionally, the method further includes:

[0024] After acquiring a group of target depth images corresponding to a group of continuously captured original images, the group of original images including the original image, acquiring, for a first target depth image corresponding to a first original image in the group of original images, a first depth value of a first pixel point used to represent a captured object in the first target depth image, and a second depth value of a second pixel point used to represent the captured object in a second target depth image, where the second target depth image is a target depth image corresponding to a second original image in the group of original images, and the second original image is captured adjacent to the first original image;

[0025] A target depth value of the first pixel in the first target depth image is determined according to the first depth value and the second depth value, or according to the second depth value.

[0026] Optionally, obtaining a second depth value of a second pixel point representing the photographed object in the second target depth image includes:

[0027] Acquire a third pixel point in the first original image and a fourth pixel point in the second original image, where both the third pixel point and the fourth pixel point are used to represent the photographed object;

[0028] Determining a positional relationship between the third pixel point and the fourth pixel point;

[0029] Determining a target pixel position in the second target depth image based on the pixel position of the first pixel in the first target depth image and the positional relationship;

[0030] Determining the second pixel point located at the target pixel point position in the second target depth image;

[0031] Obtain a second depth value of the second pixel.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0033] An image acquisition module is configured to acquire an initial depth image corresponding to the original image;

[0034] A similarity acquisition module is configured to acquire pixel value similarities between adjacent pixels in the original image;

[0035] The depth value adjustment module is configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image.

[0036] Optionally, the initial depth image includes a set of pixels with a depth value of zero;

[0037] The depth value adjustment module is configured to adjust the depth values ​​of the pixel points in the pixel point set according to the pixel value similarity.

[0038] Optionally, the depth value adjustment module includes:

[0039] a weight determination submodule configured to determine, for a pixel in the initial depth image, a weight used by the pixel in the initial depth image based on a similarity in pixel value corresponding to a target pixel in the original image, the target pixel being located at the same pixel position as the pixel;

[0040] a function construction submodule, configured to construct an objective function, the objective function including a first data item, a second data item, and a third data item, wherein the first data item is used to represent a first difference, which is the difference between the optimized depth value and the actual depth value of the pixel in the initial depth image; the second data item is used to represent a second difference, which is the difference between the optimized depth value of the pixel and the optimized depth value of the adjacent pixel in the initial depth image; and the third data item is used to represent the weight used by the pixel in the initial depth image;

[0041] an optimized depth value adjustment submodule, configured to adjust the optimized depth value of each pixel in the objective function;

[0042] The image acquisition submodule is configured to obtain the target depth image according to the current optimized depth value of each pixel point in response to the adjusted objective function satisfying a preset condition.

[0043] Optionally, the weight used by the pixel point is negatively correlated with the similarity of the pixel value corresponding to the target pixel point.

[0044] Optionally, the image acquisition submodule includes:

[0045] a deviation determining unit configured to determine, for each pixel in the initial depth image, a deviation between a current optimized depth value of the pixel and an actual depth value;

[0046] a depth value adjusting unit, configured to adjust the current optimized depth value of the pixel point to the actual depth value in response to the deviation not falling within a preset deviation range;

[0047] The image obtaining unit is configured to obtain the target depth image according to the actual depth value of the pixel point.

[0048] Optionally, the depth value adjustment module includes:

[0049] a depth value adjustment submodule, configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image;

[0050] The image denoising module is configured to denoise the intermediate depth image to obtain the target depth image.

[0051] Optionally, the device further comprises:

[0052] a depth value acquisition module configured to, after acquiring a set of target depth images corresponding to a set of continuously captured original images, the set of original images including the original image, acquire, for a first target depth image corresponding to a first original image in the set of original images, a first depth value of a first pixel point used to represent a captured object in the first target depth image, and a second depth value of a second pixel point used to represent the captured object in a second target depth image, wherein the second target depth image is a target depth image corresponding to a second original image in the set of original images, and the second original image is captured adjacent to the first original image;

[0053] The depth value determination module is configured to determine a target depth value of the first pixel in the first target depth image according to the first depth value and the second depth value, or according to the second depth value.

[0054] Optionally, the depth value acquisition module includes:

[0055] a pixel acquisition submodule, configured to acquire a third pixel in the first original image and a fourth pixel in the second original image, wherein the third pixel and the fourth pixel are both used to represent the photographed object;

[0056] a position relationship determination submodule, configured to determine a position relationship between the third pixel point and the fourth pixel point;

[0057] a pixel position determination submodule, configured to determine a target pixel position in the second target depth image based on the pixel position of the first pixel in the first target depth image and the position relationship;

[0058] a pixel point determination submodule, configured to determine the second pixel point located at the target pixel point position in the second target depth image;

[0059] The second depth value acquisition submodule is configured to acquire a second depth value of the second pixel point.

[0060] According to a third aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.

[0061] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0062] processor;

[0063] a memory for storing instructions executable by the processor;

[0064] The processor is configured to execute the instructions to implement the method described in any one of the first aspects above.

[0065] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0066] The embodiments of the present disclosure provide an image processing method. Compared with the initial depth image, the original image has characteristics such as clear boundaries of the photographed object. The depth value in the initial depth image is adjusted by utilizing the similarity of pixel values ​​between adjacent pixels in the original image, which can improve the depth information of the boundary of the photographed object in the initial depth image and obtain a high-quality target depth image.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of an image processing method according to an exemplary embodiment;

[0069] Figure 2 is a flow chart of another image processing method according to an exemplary embodiment;

[0070] Figure 3 is a block diagram of an image processing apparatus according to an exemplary embodiment;

[0071] Figure 4 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0073] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0074] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0075] The image processing method provided by the present disclosure can be applied to electronic devices. For example, the electronic device is equipped with a camera module. After the camera module captures an image, the image processing method provided by the present disclosure is used to obtain a depth image corresponding to the image.

[0076] The image processing method provided by this disclosure can be applied to a server. For example, the server can obtain an image uploaded by an electronic device and use the image processing method provided by this disclosure to obtain a depth image corresponding to the image. Furthermore, the server can send the obtained depth image to the electronic device for use by the electronic device.

[0077] The image processing method provided by the present disclosure is described below by taking the application of the image processing method to an electronic device as an example.

[0078] Figure 1 is a flow chart of an image processing method according to an exemplary embodiment. Figure 1 The methods shown include:

[0079] In step 101, an initial depth image corresponding to an original image is obtained.

[0080] The original image is the image to be processed. The original image needs to be processed to obtain the depth image corresponding to the original image. The original image can be an RGB image or other applicable type of image.

[0081] The method in the related art may be used to obtain the initial depth image corresponding to the original image.

[0082] In one embodiment, the electronic device is provided with specific functions, such as AR ruler, AR special effects, etc. When the specific function is turned on, the electronic device starts to execute the image processing method provided in this embodiment.

[0083] In step 102, the pixel value similarity between adjacent pixels in the original image is obtained.

[0084] For a pixel in the original image, there are multiple adjacent pixels, for example, there are eight adjacent pixels. In this step, the pixel value similarity between the pixel and the multiple adjacent pixels can be calculated, for example, the pixel value similarity between the pixel and each of the adjacent pixels can be calculated.

[0085] There are various methods for calculating the pixel value similarity between a pixel and each adjacent pixel. For example, the pixel value of the pixel can be divided by the pixel value of each adjacent pixel to obtain the pixel value similarity between the pixel and each adjacent pixel. Another example is to divide the pixel value of the pixel by the pixel value of each adjacent pixel, then adjust the above division result based on the pixel value differences between the pixel and multiple adjacent pixels to obtain the pixel value similarity between the pixel and each adjacent pixel. Pixel value similarity can be determined using methods described in related art.

[0086] In step 103, the depth values ​​in the initial depth image are adjusted according to the pixel value similarity to obtain a target depth image corresponding to the original image.

[0087] In one embodiment, depth images obtained using existing methods suffer from varying degrees of localized loss of depth information, resulting in the depth image including areas resembling holes. In this case, the initial depth image includes a set of pixels with a depth value of zero, the depth values ​​of the pixels in the set are zero, and the set includes abnormal pixels, representing objects whose actual distance from the capture module is not zero.

[0088] To solve the above problem, when executing this step, the electronic device may adjust the depth values ​​of the pixel points in the pixel point set included in the initial depth image according to the pixel value similarity obtained in step 102 .

[0089] By adopting the above method, the number of pixels with a depth value of zero in the depth image is reduced, and the number of pixels with a depth value not equal to zero in the depth image is increased, thereby obtaining a dense depth image.

[0090] In one embodiment, step 103 can be implemented as follows:

[0091] The first step is to determine the weights used by the pixels in the initial depth image based on the similarity of the pixel values ​​corresponding to the target pixels in the original image. The target pixels are located at the same pixel position as the pixels in the initial depth image.

[0092] The pixel value similarity corresponding to the target pixel in the original image can be understood as the pixel value similarity between the target pixel and adjacent pixels in the original image. The method described in step 102 can be used to obtain the pixel value similarity corresponding to the target pixel in the original image.

[0093] For example: the initial depth image includes adjacent pixel 1 and pixel 2, the coordinates of pixel 1 are (x1, y1), and the coordinates of pixel 2 are (x2, y2). The original image includes pixel 3 and pixel 4, the coordinates of pixel 3 are (x1, y1), and the coordinates of pixel 4 are (x2, y2).

[0094] According to the pixel value similarity between pixel 3 and adjacent pixel 4, a weight suitable for pixel 2 used by pixel 1 in the initial depth image is determined.

[0095] One case: the weight used for the pixel point in the initial depth image is negatively correlated with the similarity of the pixel value corresponding to the target pixel point in the original image.

[0096] The specific relationship between the weight and the pixel value similarity can be set as needed. For example, the weight is equal to one-half of the pixel value similarity.

[0097] Step 2: Construct the objective function.

[0098] The objective function includes a first data item, a second data item, and a third data item, wherein the first data item is used to characterize a first difference, wherein the first difference is the difference between the optimized depth value and the actual depth value of the pixel point in the initial depth image, the second data item is used to characterize a second difference, and the second difference is the difference between the optimized depth value of the pixel point in the initial depth image and the optimized depth value of the adjacent pixel point, and the third data item is used to characterize the weight used by the pixel point in the initial depth image.

[0099] The depth value of a pixel in the initial depth image is called the actual depth value.

[0100] The optimized depth value of a pixel in the initial depth image is a depth value obtained by optimizing the actual depth value of the pixel in the initial depth image. The optimized depth value is the depth value to be determined.

[0101] On the basis that the above data items have specific characterization functions, the specific forms of the first data item, the second data item and the third data item can be set according to needs and experience.

[0102] For example, the objective function is expressed as follows:

[0103]

[0104] Among them, λ is the penalty factor, A is the set of pixel coordinates in the initial depth image, i is the pixel coordinates in the initial depth image, x i is the optimized depth value of pixel a at position i in the initial depth image, d i is the actual depth value of pixel a at position i in the initial depth image, N(i) is the coordinate set of all pixels adjacent to pixel a in the initial depth image, j is the coordinate of a pixel adjacent to pixel a in the initial depth image, x j is the optimized depth value of pixel b at position j in the initial depth image, W ij The weight appropriate for pixel b used for pixel a in the initial depth image.

[0105] There is a pixel value similarity between the pixel at position i and the pixel at position j in the original image, W ij It is negatively correlated with the above pixel similarity.

[0106] Usually λ is a fixed value, such as one thousandth or one ten-thousandth, and can be set according to needs and experience.

[0107] Typically, the initial depth image and the original image have the same pixel distribution, and pixels at the same pixel coordinates are used to represent the same object. For example, pixel a at position i in the initial depth image and pixel i in the original image are used to represent the same object, and pixel b at position j in the initial depth image and pixel j in the original image are used to represent the same object.

[0108] When using the above objective function, relevant calculations are performed on each j in N(i), and the calculation results related to each j are summed up. Relevant calculations are performed on each i in A, and the calculation results related to each i are summed up.

[0109] In the above expression of the objective function, the first data item includes (x i -d i ) 2 , the second data item includes (x i -x j ) 2 , the third data item includes W ij .

[0110] On the basis of the above expression of the objective function, non-global terms, first-order derivative terms of the optimized depth value of the pixel point, first-order derivative terms of the actual depth value of the pixel point, etc. can also be added to the expression.

[0111] This example only introduces the expression of the objective function. Any applicable expression can be used.

[0112] The third step: adjusting the optimized depth value of each pixel in the objective function, and in response to the adjusted objective function satisfying a preset condition, obtaining a target depth image according to the current optimized depth value of each pixel.

[0113] For the objective function in the above example, in the process of adjusting the optimized depth value of each pixel point in the objective function, after completing one adjustment, the function value of the objective function is calculated based on the current optimized depth value of each pixel point. If the function value is the smallest, it is determined that the adjusted objective function meets the preset conditions.

[0114] In the above example, the objective function includes W ij , W ij It is determined based on the similarity of pixel values ​​between adjacent pixels in the original image, using W ij The purpose is to introduce the pixel value information in the original image. Compared with the initial depth image, the original image has the characteristics of clear boundaries of the photographed object. ij Adjusting the depth value in the initial depth image can improve the depth information of the boundary of the photographed object in the initial depth image and obtain a high-quality target depth image.

[0115] Using the objective function in the above example, the depth values ​​of each pixel in the initial depth image are adjusted to achieve global optimization. The objective function is used to adjust the zero depth values ​​in the initial depth image to non-zero depth values. This removes the hole-like regions in the initial depth image composed of multiple zero depth values, increases the number of non-zero depth values, and ultimately obtains a dense target depth image.

[0116] For the third step, after using the objective function to determine the current optimized depth value of the pixel point, due to the influence of some factors, there is a deviation between the current optimized depth value of the pixel point and the actual depth value. If the deviation is too large, it will affect the quality of the target depth image finally obtained.

[0117] Based on this, the electronic device can determine the deviation between the current optimized depth value of the pixel and the actual depth value for each pixel in the initial depth image. In response to the deviation not falling within the preset deviation range, the electronic device adjusts the current optimized depth value of the pixel to the actual depth value, and obtains the target depth image based on the actual depth value of the pixel.

[0118] For example, in response to the deviation being greater than a preset deviation, the current optimized depth value of the pixel is adjusted to the actual depth value, and a target depth image including the actual depth value of the pixel is obtained.

[0119] In the above process, the constraints of the actual depth values ​​of the pixels are added to remove the depth values ​​with large deviations and obtain an accurate target depth image.

[0120] In one embodiment, step 103 may be implemented in the following manner: first, adjusting the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image; second, denoising the intermediate depth image to obtain a target depth image.

[0121] A median filtering method may be used for denoising, for example, a 3×3 median filtering method may be used for denoising to reduce discrete noise points of the initial depth image.

[0122] In this embodiment, a denoising operation is added to obtain a low-noise target depth image.

[0123] In one embodiment, if the initial depth image is noisy, the initial depth image can be denoised first. For example, a median filter can be used to remove discrete noise from the initial depth image. The depth values ​​in the denoised initial depth image are then adjusted based on pixel value similarity to obtain a target depth image corresponding to the original image.

[0124] The embodiments of the present disclosure provide an image processing method. Compared with the initial depth image, the original image has characteristics such as clear boundaries of the photographed object. The depth value in the initial depth image is adjusted by utilizing the similarity of pixel values ​​between adjacent pixels in the original image, which can improve the depth information of the boundary of the photographed object in the initial depth image and obtain a high-quality target depth image.

[0125] In one embodiment, using Figure 1 The method shown in FIG. 4 is used to obtain a set of target depth images corresponding to a set of continuously captured original images. In this case, the electronic device may further perform the following operations.

[0126] Figure 2 is a flow chart of another image processing method according to an exemplary embodiment. Figure 2 The methods shown include:

[0127] In step 201, for a first target depth image corresponding to a first original image in a group of original images, a first depth value of a first pixel point used to characterize the photographed object in the first target depth image and a second depth value of a second pixel point used to characterize the photographed object in the second target depth image are obtained, where the second target depth image is a target depth image corresponding to the second original image in the group of original images, and the second original image is shot adjacent to the first original image.

[0128] In step 202 , a target depth value of a first pixel in a first target depth image is determined according to the first depth value and the second depth value, or according to the second depth value.

[0129] In this embodiment, the depth value in the second target depth image corresponding to the second original image taken adjacently is used to constrain the depth value in the first target depth image corresponding to the first original image, so as to ensure that the depth value at the same pixel position in the two target depth images corresponding to the two adjacent original images is relatively stable, thereby avoiding the occurrence of visual flickering due to large differences in depth values ​​at the same pixel position, and achieving stability in the time domain.

[0130] Step 201 can be implemented as follows:

[0131] Step 1: Acquire a third pixel point in the first original image and a fourth pixel point in the second original image, where both the third pixel point and the fourth pixel point are used to represent the photographed object.

[0132] The first pixel point, the second pixel point, the third pixel point, and the fourth pixel point are used to represent the same photographed object.

[0133] Step 2: Determine the positional relationship between the third pixel point and the fourth pixel point.

[0134] Step 3: Determine the target pixel position in the second target depth image according to the pixel position of the first pixel in the first target depth image and the positional relationship determined in step 2.

[0135] Step 4: Determine a second pixel point located at the target pixel point position in the second target depth image.

[0136] Step 5: Obtain a second depth value of a second pixel in the second target depth image.

[0137] For example, during the process of shooting a video, a group of original images shot continuously are obtained, and a group of target depth images corresponding to the group of original images are obtained.

[0138] First, one frame of original image in a group of original images is used as a benchmark image, and at least one frame of original image taken adjacent to the benchmark image is used as a reference image.

[0139] There are various ways to select reference images. For example, a set of original images can be arranged in chronological order, and m frames preceding the reference image and n frames following the reference image can be used as reference images, where m and n are equal or unequal. For example, m is an integer between 3 and 5, and n is an integer between 3 and 5.

[0140] For the original image at the front, such as the original image at the top, a certain number of original images that are arranged behind it can be used as reference images. For the original image at the back, such as the original image at the end, a certain number of original images that are arranged before it can be used as reference images.

[0141] Next, a first positional relationship between pixel points representing the same photographed object in the benchmark image and the reference image is determined.

[0142] The base image corresponds to target depth image 1, and the reference image corresponds to target depth image 2. For the same photographed object, the pixels representing the same photographed object in the base image and the reference image have a first positional relationship, and the pixels representing the same photographed object in the target depth image 1 and the target depth image 2 should also have a first positional relationship.

[0143] Thirdly, the positions of the pixels used to represent the same object in the target depth image 2 are determined based on the positions of the pixels used to represent the same object in the target depth image 1 and the above-mentioned positional relationship.

[0144] In an application, the reference image includes multiple objects, and the above method is used to obtain a first positional relationship corresponding to each object. The multiple first positional relationships can be arranged into a matrix based on the positions of multiple pixels representing the multiple objects in the reference image to obtain a homography matrix. The homography matrix is ​​used to determine the positions of the pixels representing different objects in the target depth image 2.

[0145] With respect to step 202 , in one case, statistics are performed on the first depth value and the second depth value, and the statistical result is determined as the target depth value of the first pixel.

[0146] The first original image can be understood as a base image, and the second original image can be understood as a reference image. When there are multiple second original images, there are multiple second depth values. Statistics are performed on the first depth value and the multiple second depth values, and the statistical result is determined as the target depth value of the first pixel.

[0147] In another case, the target depth value of the first pixel in the first target depth image is determined according to the second depth value.

[0148] When the number of the second original image is one, the number of the second depth value is one. In this case, the second depth value can be determined as the target depth value of the first pixel point, or the second depth value can be processed, such as multiplied by a coefficient, and the processing result is determined as the target depth value of the first pixel point.

[0149] When there are multiple second original images, there are also multiple second depth values. In this case, statistics may be performed on the multiple second depth values, and the statistical result may be determined as the target depth value of the first pixel.

[0150] The statistics of the multiple second depth values ​​may be calculated by weighting and summing the multiple second depth values. For example, for the first pixel, the second pixel, the third pixel, and the fourth pixel, the pixel value similarity between the fourth pixel and the third pixel may be determined, and based on the pixel value similarity, the weight used for the second depth value of the second pixel is determined. In this example, the weight is positively correlated with the pixel value similarity.

[0151] In an application, a weighted median filtering algorithm may be used to calculate multiple depth values ​​to obtain an optimal depth value of a first pixel in a first target depth image.

[0152] For a video stream with a high input frame rate, a median filtering algorithm may be used to perform statistics on multiple depth values ​​to obtain an optimal depth value for the first pixel in the first target depth image.

[0153] The method provided in this embodiment can be used to take each frame of the continuously captured original image as a reference image, and optimize the target depth image corresponding to each frame of the original image to obtain a high-quality depth image.

[0154] In the application, each frame of a group of original images can be used as a reference image in sequence according to the shooting time from early to late.

[0155] For the sake of simplicity, the aforementioned method embodiments are all expressed as a series of action combinations. However, those skilled in the art should know that the present disclosure is not limited to the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0156] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0157] Corresponding to the aforementioned application function implementation method embodiments, the present disclosure also provides application function implementation devices and corresponding embodiments.

[0158] Figure 3is a block diagram of an image processing device according to an exemplary embodiment. Figure 3 , the device comprises:

[0159] The image acquisition module 31 is configured to acquire an initial depth image corresponding to the original image;

[0160] A similarity acquisition module 32 is configured to acquire pixel value similarities between adjacent pixels in the original image;

[0161] The depth value adjustment module 33 is configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image.

[0162] In an optional embodiment, Figure 3 Based on the image processing device shown, the initial depth image includes a set of pixels with a depth value of zero;

[0163] The depth value adjustment module 33 may be configured to adjust the depth values ​​of the pixels in the pixel set according to the pixel value similarity.

[0164] In an optional embodiment, Figure 3 Based on the image processing apparatus shown, the depth value adjustment module 33 may include:

[0165] a weight determination submodule configured to determine, for a pixel in the initial depth image, a weight used by the pixel in the initial depth image based on a similarity in pixel value corresponding to a target pixel in the original image, the target pixel being located at the same pixel position as the pixel;

[0166] a function construction submodule, configured to construct an objective function, the objective function including a first data item, a second data item, and a third data item, wherein the first data item is used to represent a first difference, which is the difference between the optimized depth value and the actual depth value of the pixel in the initial depth image; the second data item is used to represent a second difference, which is the difference between the optimized depth value of the pixel and the optimized depth value of the adjacent pixel in the initial depth image; and the third data item is used to represent the weight used by the pixel in the initial depth image;

[0167] an optimized depth value adjustment submodule, configured to adjust the optimized depth value of each pixel in the objective function;

[0168] The image acquisition submodule is configured to obtain the target depth image according to the current optimized depth value of each pixel point in response to the adjusted objective function satisfying a preset condition.

[0169] In an optional embodiment, the weight used by the pixel point is negatively correlated with the similarity of the pixel value corresponding to the target pixel point.

[0170] In an optional embodiment, the image acquisition submodule includes:

[0171] a deviation determining unit configured to determine, for each pixel in the initial depth image, a deviation between a current optimized depth value of the pixel and an actual depth value;

[0172] a depth value adjusting unit, configured to adjust the current optimized depth value of the pixel point to the actual depth value in response to the deviation not falling within a preset deviation range;

[0173] The image obtaining unit is configured to obtain the target depth image according to the actual depth value of the pixel point.

[0174] In an optional embodiment, Figure 3 Based on the image processing apparatus shown, the depth value adjustment module 33 may include:

[0175] a depth value adjustment submodule, configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image;

[0176] The image denoising module is configured to denoise the intermediate depth image to obtain the target depth image.

[0177] In an optional embodiment, Figure 3 Based on the image processing device shown in the figure, the device may further include:

[0178] a depth value acquisition module configured to, after acquiring a set of target depth images corresponding to a set of continuously captured original images, the set of original images including the original image, acquire, for a first target depth image corresponding to a first original image in the set of original images, a first depth value of a first pixel point used to represent a captured object in the first target depth image, and a second depth value of a second pixel point used to represent the captured object in a second target depth image, wherein the second target depth image is a target depth image corresponding to a second original image in the set of original images, and the second original image is captured adjacent to the first original image;

[0179] The depth value determination module is configured to determine a target depth value of the first pixel in the first target depth image according to the first depth value and the second depth value, or according to the second depth value.

[0180] In an optional embodiment, the depth value acquisition module may include:

[0181] a pixel acquisition submodule, configured to acquire a third pixel in the first original image and a fourth pixel in the second original image, wherein the third pixel and the fourth pixel are both used to represent the photographed object;

[0182] a position relationship determination submodule, configured to determine a position relationship between the third pixel point and the fourth pixel point;

[0183] a pixel position determination submodule, configured to determine a target pixel position in the second target depth image based on the pixel position of the first pixel in the first target depth image and the position relationship;

[0184] a pixel point determination submodule, configured to determine the second pixel point located at the target pixel point position in the second target depth image;

[0185] The second depth value acquisition submodule is configured to acquire a second depth value of the second pixel point.

[0186] Figure 4 1 is a schematic diagram illustrating the structure of an electronic device 1600 according to an exemplary embodiment. For example, electronic device 1600 may be a user device, specifically a mobile phone, a computer, a digital radio, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or a wearable device such as a smart watch, smart glasses, a smart bracelet, or a smart running shoe.

[0187] Reference Figure 4 , the electronic device 1600 may include one or more of the following components: a processing component 1602 , a memory 1604 , a power component 1606 , a multimedia component 1608 , an audio component 1610 , an input / output (I / O) interface 1612 , a sensor component 1614 , and a communication component 1616 .

[0188] The processing component 1602 generally controls the overall operation of the electronic device 1600, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1602 may include one or more processors 1620 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1602 may include one or more modules to facilitate interaction between the processing component 1602 and other components. For example, the processing component 1602 may include a multimedia module to facilitate interaction between the multimedia component 1608 and the processing component 1602.

[0189] The memory 1604 is configured to store various types of data to support the operations of the device 1600. Examples of such data include instructions for any application or method operating on the electronic device 1600, contact data, phone book data, messages, pictures, videos, etc. The memory 1604 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0190] The power supply component 1606 provides power to the various components of the electronic device 1600. The power supply component 1606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1600.

[0191] The multimedia component 1608 includes a screen that provides an output interface between the electronic device 1600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 1608 includes a front camera and / or a rear camera. When the device 1600 is in an operating mode, such as adjustment mode or video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0192] The audio component 1610 is configured to output and / or input audio signals. For example, the audio component 1610 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 1600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1604 or transmitted via the communication component 1616. In some embodiments, the audio component 1610 also includes a speaker for outputting audio signals.

[0193] I / O interface 1612 provides an interface between processing component 1602 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0194] The sensor assembly 1614 includes one or more sensors for providing various aspects of the status assessment of the electronic device 1600. For example, the sensor assembly 1614 can detect the open / closed state of the device 1600, the relative positioning of components, such as the display and keypad of the electronic device 1600. The sensor assembly 1614 can also detect changes in the position of the electronic device 1600 or a component of the electronic device 1600, the presence or absence of user contact with the electronic device 1600, the orientation or acceleration / deceleration of the electronic device 1600, and changes in the temperature of the electronic device 1600. The sensor assembly 1614 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1614 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1614 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0195] The communication component 1616 is configured to facilitate wired or wireless communication between the electronic device 1600 and other devices. The electronic device 1600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the above-mentioned communication component 1616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0196] In an exemplary embodiment, the electronic device 1600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0197] In an exemplary embodiment, a non-temporary computer-readable storage medium is also provided, such as a memory 1604 including instructions. When the instructions in the storage medium are executed by the processor 1620 of the electronic device 1600, the electronic device 1600 is enabled to perform an image processing method, which includes: obtaining an initial depth image corresponding to the original image; obtaining the pixel value similarity between adjacent pixel points in the original image; adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image.

[0198] The non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0199] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0200] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the initial depth image corresponding to the original image; Obtaining pixel value similarity between adjacent pixels in the original image; Adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image; The method further comprises: After acquiring a group of target depth images corresponding to a group of continuously captured original images, the group of original images including the original image, acquiring, for a first target depth image corresponding to a first original image in the group of original images, a first depth value of a first pixel point used to represent a captured object in the first target depth image, and a second depth value of a second pixel point used to represent the captured object in a second target depth image, where the second target depth image is a target depth image corresponding to a second original image in the group of original images, and the second original image is captured adjacent to the first original image; A target depth value of the first pixel in the first target depth image is determined according to the first depth value and the second depth value, or according to the second depth value.

2. The method according to claim 1, characterized in that The initial depth image includes a set of pixels with a depth value of zero; The adjusting the depth value in the initial depth image according to the pixel value similarity includes: The depth values ​​of the pixels in the pixel set are adjusted according to the pixel value similarity.

3. The method according to claim 1, characterized in that The adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image includes: For a pixel in the initial depth image, determining a weight used by the pixel in the initial depth image based on a similarity in pixel values ​​corresponding to a target pixel in the original image, the target pixel being located at the same pixel position as the pixel; Constructing an objective function, the objective function including a first data item, a second data item, and a third data item, wherein the first data item is used to represent a first difference, which is a difference between an optimized depth value and an actual depth value of the pixel in the initial depth image; the second data item is used to represent a second difference, which is a difference between the optimized depth value of the pixel and an optimized depth value of an adjacent pixel in the initial depth image; and the third data item is used to represent a weight used by the pixel in the initial depth image; The optimized depth value of each pixel in the objective function is adjusted, and in response to the adjusted objective function satisfying a preset condition, the target depth image is obtained according to the current optimized depth value of each pixel.

4. The method according to claim 3, characterized in that The weight used by the pixel point is negatively correlated with the similarity of the pixel value corresponding to the target pixel point.

5. The method according to claim 3 or 4, characterized in that The obtaining of the target depth image according to the current optimized depth value of each pixel point includes: For each pixel in the initial depth image, determining a deviation between a current optimized depth value of the pixel and an actual depth value; In response to the deviation not falling within the preset deviation range, adjusting the current optimized depth value of the pixel point to the actual depth value; The target depth image is obtained according to the actual depth value of the pixel point.

6. The method according to claim 1, characterized in that The adjusting the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image includes: Adjusting the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image; Denoising is performed on the intermediate depth image to obtain the target depth image.

7. The method according to claim 1, characterized in that The obtaining of a second depth value of a second pixel point representing the photographed object in the second target depth image includes: Acquire a third pixel point in the first original image and a fourth pixel point in the second original image, where both the third pixel point and the fourth pixel point are used to represent the photographed object; Determining a positional relationship between the third pixel point and the fourth pixel point; Determining a target pixel position in the second target depth image based on the pixel position of the first pixel in the first target depth image and the positional relationship; Determining the second pixel point located at the target pixel point position in the second target depth image; Obtain a second depth value of the second pixel.

8. An image processing device, characterized in that: The device comprises: An image acquisition module is configured to acquire an initial depth image corresponding to the original image; A similarity acquisition module is configured to acquire pixel value similarities between adjacent pixels in the original image; a depth value adjustment module configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain a target depth image corresponding to the original image; a depth value acquisition module configured to, after acquiring a set of target depth images corresponding to a set of continuously captured original images, the set of original images including the original image, acquire, for a first target depth image corresponding to a first original image in the set of original images, a first depth value of a first pixel point used to represent a captured object in the first target depth image, and a second depth value of a second pixel point used to represent the captured object in a second target depth image, wherein the second target depth image is a target depth image corresponding to a second original image in the set of original images, and the second original image is captured adjacent to the first original image; The depth value determination module is configured to determine a target depth value of the first pixel in the first target depth image according to the first depth value and the second depth value, or according to the second depth value.

9. The device according to claim 8, characterized in that The initial depth image includes a set of pixels with a depth value of zero; The depth value adjustment module is configured to adjust the depth values ​​of the pixel points in the pixel point set according to the pixel value similarity.

10. The device according to claim 8, characterized in that The depth value adjustment module includes: a weight determination submodule configured to determine, for a pixel in the initial depth image, a weight used by the pixel in the initial depth image based on a similarity in pixel value corresponding to a target pixel in the original image, the target pixel being located at the same pixel position as the pixel; a function construction submodule, configured to construct an objective function, the objective function including a first data item, a second data item, and a third data item, wherein the first data item is used to represent a first difference, which is the difference between the optimized depth value and the actual depth value of the pixel in the initial depth image; the second data item is used to represent a second difference, which is the difference between the optimized depth value of the pixel and the optimized depth value of the adjacent pixel in the initial depth image; and the third data item is used to represent the weight used by the pixel in the initial depth image; an optimized depth value adjustment submodule, configured to adjust the optimized depth value of each pixel in the objective function; The image acquisition submodule is configured to obtain the target depth image according to the current optimized depth value of each pixel point in response to the adjusted objective function satisfying a preset condition.

11. The device according to claim 10, characterized in that The weight used by the pixel point is negatively correlated with the similarity of the pixel value corresponding to the target pixel point.

12. The device according to claim 10 or 11, characterized in that The image acquisition submodule includes: a deviation determining unit configured to determine, for each pixel in the initial depth image, a deviation between a current optimized depth value of the pixel and an actual depth value; a depth value adjusting unit, configured to adjust the current optimized depth value of the pixel point to the actual depth value in response to the deviation not falling within a preset deviation range; The image obtaining unit is configured to obtain the target depth image according to the actual depth value of the pixel point.

13. The device according to claim 8, characterized in that The depth value adjustment module includes: a depth value adjustment submodule, configured to adjust the depth value in the initial depth image according to the pixel value similarity to obtain an intermediate depth image; The image denoising module is configured to denoise the intermediate depth image to obtain the target depth image.

14. The device according to claim 8, characterized in that The depth value acquisition module includes: a pixel acquisition submodule, configured to acquire a third pixel in the first original image and a fourth pixel in the second original image, wherein the third pixel and the fourth pixel are both used to represent the photographed object; a position relationship determination submodule, configured to determine a position relationship between the third pixel point and the fourth pixel point; a pixel position determination submodule, configured to determine a target pixel position in the second target depth image based on the pixel position of the first pixel in the first target depth image and the position relationship; a pixel point determination submodule, configured to determine the second pixel point located at the target pixel point position in the second target depth image; The second depth value acquisition submodule is configured to acquire a second depth value of the second pixel point.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

16. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

Images