Image processing method, image processing device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method, an image processing device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring depth features of a to-be-processed image; and converting the depth feature into a depth feature in a polyhedral lattice space according to a preset depth feature conversion relationship. And performing depth feature completion according to the depth features in the polyhedral lattice space to obtain the depth features after completion. And according to the preset depth feature conversion relationship, inversely converting the complemented depth features in the polyhedral lattice space into target depth features in the to-be-processed image to obtain an inversely converted to-be-processed image, and determining the inversely converted to-be-processed image as a target image. According to the invention, in the process of sparse depth completion and dense depth obtaining, the calculation amount is reduced, the overall operation speed is improved, sparse depth completion is carried out based on the depth features in the polyhedral lattice space, and the depth completion effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to an image processing method, an image processing apparatus, an electronic device, and a storage medium. Background Art

[0002] Some existing image processing methods require the use of depth data of an image. A large number of invalid disparities exist in the depth map obtained by traditional image depth acquisition methods, that is, the obtained depth map is sparse, which will interfere with many algorithms and applications that rely on depth information and affect the processing effect. Therefore, completing a sparse depth map into a dense depth map is an important step in obtaining an image depth map.

[0003] Related technologies include traditional methods based on bilateral filtering and methods based on deep learning. In traditional methods, an image is transformed from an image space to a bilateral space, and iterative optimization and sparse depth completion are performed in the bilateral space to complete the sparse depth into a dense depth, with a large amount of calculation and a poor completion effect. In deep learning methods, a deep learning model is used to input an RGB image and a sparse depth map to obtain a dense depth map. The amount of calculation is even larger, and it is difficult to obtain training data for the model. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides an image processing method, an image processing apparatus, an electronic device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, characterized by including: obtaining depth features of an image to be processed; converting the depth features into depth features in a polyhedral lattice space according to a preset depth feature conversion relationship, where the preset depth feature conversion relationship is a correspondence between the depth features in the image to be processed and the depth features in the polyhedral lattice space, and the coordinates in the polyhedral lattice space are obtained by projecting a preset dimensional integer space onto a hyperplane with a normal vector being a vector of all ones; performing depth feature completion according to the depth features in the polyhedral lattice space to obtain completed depth features; and inversely converting the completed depth features in the polyhedral lattice space into target depth features in the image to be processed according to the preset depth feature conversion relationship to obtain the image to be processed after inverse conversion processing, and determining the image to be processed obtained by inverse conversion processing as a target image.

[0006] In one implementation, it is characterized in that the preset depth feature conversion relationship is determined in the following manner: Obtain the bilateral spatial coordinates of each pixel in the image to be processed. The sub - coordinates that make up the bilateral spatial coordinates include the spatial sub - coordinates and color sub - coordinates of the pixel; Determine the mapping coordinates of the bilateral spatial coordinates mapped to the polyhedral lattice space, and determine the vertex coordinate group and the weights corresponding to the vertex coordinate group in the polyhedral lattice space according to the mapping coordinates. The vertex coordinate group includes a preset number of vertex coordinates closest to the mapping coordinates. The weights corresponding to the vertex coordinate group include the weights corresponding to each vertex coordinate in the preset number of vertex coordinates. The preset number corresponds to the number of sub - coordinates that make up the bilateral spatial coordinates. The weight corresponding to the vertex coordinate represents the distance between the vertex coordinate and the corresponding mapping coordinate; Determine the corresponding relationship between the bilateral spatial coordinates of each pixel in the image to be processed, the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel, and the weights of the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel as the preset coordinate conversion relationship; Determine the preset depth feature conversion relationship according to the preset coordinate conversion relationship.

[0007] In one implementation, the polyhedral lattice space includes a plurality of regular polygons. The vertex coordinates in the polyhedral lattice space are located at the vertices of the regular polygons. The number of sides of the regular polygon is a preset number. For each regular polygon among the plurality of regular polygons, the vertex position numbers are set for the plurality of vertices corresponding to the regular polygon according to a unified rule; Determining the preset depth feature conversion relationship according to the preset coordinate conversion relationship includes: Determining all vertex coordinate groups in the polyhedral lattice space that have a corresponding relationship with all bilateral spatial coordinates, sorting and numbering all vertex coordinates included in all vertex coordinate groups, encoding the coordinate values of all vertex coordinates as hash values, and storing all vertex coordinate numbers and hash values in the form of key - value pairs; Sorting and numbering all pixels in the image to be processed; For the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel among all pixels, associating the pixel number, the vertex coordinate number, and the vertex position number with the preset number of vertex coordinates included in the vertex coordinate group, and associating the pixel number, the vertex coordinate number, and the vertex position number with the weights corresponding to the preset number of vertex coordinates, to obtain the preset depth feature conversion relationship.

[0008] In one implementation, the depth features of the image to be processed include the depth value and the depth value confidence of each pixel in the image to be processed; according to a preset depth feature conversion relationship, converting the depth features into depth features in a polyhedral lattice space includes: for each pixel in the image to be processed, according to the preset depth feature conversion relationship, converting the depth value of the pixel into a first depth corresponding value in the polyhedral lattice space, and converting the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space; according to the first depth corresponding value in the polyhedral lattice space and a preset vector conversion method, determining a first vector corresponding to the pixel depth value, and according to the confidence corresponding value of the depth value and the preset vector conversion method, determining a second vector corresponding to the pixel depth value confidence, where the first vector and the second vector represent the depth features in the polyhedral lattice space.

[0009] In one implementation, the step of converting the depth value of a pixel into a first depth corresponding value in the polyhedral lattice space according to the preset depth feature conversion relationship includes: converting the depth value of each pixel in the image to be processed into a first depth corresponding value of a corresponding vertex coordinate group in the polyhedral lattice space, where the first depth corresponding value of the vertex coordinate group includes the first depth corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate, the pixel depth value, and the pixel depth value confidence; the step of converting the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space includes: converting the depth value of each pixel in the image to be processed into a confidence corresponding value of a corresponding vertex coordinate group in the polyhedral lattice space, where the confidence corresponding value of the vertex coordinate group includes the confidence corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate and the pixel depth value confidence.

[0010] In one implementation, the step of performing depth feature completion according to the depth features in the polyhedral lattice space to obtain the completed depth features includes: traversing all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space, assigning corresponding distance weights to the distances between every two vertices among all the vertices, and constructing a metric matrix for measuring the distances between every two vertices among all the vertices; performing depth completion according to the metric matrix, the first vector, and the second vector to obtain the completed depth features, where the first vector and the second vector represent the depth features in the polyhedral lattice space.

[0011] In one implementation, performing depth completion based on the metric matrix, the first vector, and the second vector to obtain a completed depth feature includes: substituting the metric matrix, the first vector, and the second vector into a preset solution equation to obtain a third vector, where the third vector represents the completed depth feature.

[0012] In one implementation, the preset solution equation is solved based on the stabilized bi-conjugate gradient.

[0013] In one implementation, inverse-converting the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship includes: converting the third vector into a second depth corresponding value corresponding to each vertex coordinate in the vertex coordinate set, where the vertex coordinate set is a set of all vertices in the polyhedral lattice space corresponding to the pixels of the image to be processed; and converting the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed according to the preset depth feature conversion relationship, where the target depth value represents the target depth feature.

[0014] In one implementation, converting the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed according to the preset depth feature conversion relationship includes: for each pixel in the image to be processed, determining a preset number of vertex coordinates corresponding to the pixel, and determining the second depth corresponding value and weight corresponding to each vertex coordinate among the preset number of vertex coordinates; and weighted-summing the second depth corresponding values according to the weights, and determining the value obtained by the weighted summation as the target depth value.

[0015] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including: an acquisition unit configured to acquire the depth feature of an image to be processed; convert the depth feature into a depth feature in a polyhedral lattice space according to a preset depth feature conversion relationship, where the preset depth feature conversion relationship is a corresponding relationship between the depth feature in the image to be processed and the depth feature in the polyhedral lattice space, and the coordinates in the polyhedral lattice space are obtained by projecting a preset-dimensional integer space onto a hyperplane with a normal vector of all-one vector; perform depth feature completion based on the depth feature in the polyhedral lattice space to obtain a completed depth feature; inverse-convert the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship to obtain the image to be processed after inverse-conversion processing, and determine the image to be processed obtained by the inverse-conversion processing as the target image.

[0016] In one implementation, the preset depth feature conversion relationship is determined by a conversion unit in the following manner: Obtain the bilateral spatial coordinates of each pixel in the image to be processed. The sub - coordinates constituting the bilateral spatial coordinates include the spatial sub - coordinates and color sub - coordinates of the pixel; Determine the mapping coordinates of the bilateral spatial coordinates mapped to the polyhedral lattice space, and determine the vertex coordinate group and the weights corresponding to the vertex coordinate group in the polyhedral lattice space according to the mapping coordinates. The vertex coordinate group includes a preset number of vertex coordinates closest to the mapping coordinates. The weights corresponding to the vertex coordinate group include the weights corresponding to each vertex coordinate in the preset number of vertex coordinates. The preset number corresponds to the number of sub - coordinates constituting the bilateral spatial coordinates. The weight corresponding to the vertex coordinate represents the distance between the vertex coordinate and the corresponding mapping coordinate; Determine the corresponding relationship between the bilateral spatial coordinates of each pixel in the image to be processed, the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel, and the weights of the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel as the preset coordinate conversion relationship; Determine the preset depth feature conversion relationship according to the preset coordinate conversion relationship.

[0017] In one implementation, the polyhedral lattice space includes a plurality of regular polygons. The vertex coordinates in the polyhedral lattice space are located at the vertices of the regular polygons. The number of sides of the regular polygon is a preset number. For each of the plurality of regular polygons, a vertex position number is set for the plurality of vertices corresponding to the regular polygon according to a unified rule; The conversion unit determines the preset depth feature conversion relationship according to the preset coordinate conversion relationship in the following manner: Determine all vertex coordinate groups in the polyhedral lattice space that have a corresponding relationship with all bilateral spatial coordinates, sort and number all vertex coordinates included in all vertex coordinate groups, and encode the coordinate values of all vertex coordinates as hash values. Store all vertex coordinate numbers and hash values in the form of key - value pairs; Sort and number all pixels in the image to be processed; For the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel among all pixels, associate the pixel number, vertex coordinate number, and the vertex position number with the preset number of vertex coordinates included in the vertex coordinate group, and associate the pixel number, the vertex coordinate number, and the vertex position number with the weights corresponding to the preset number of vertex coordinates, to obtain the preset depth feature conversion relationship.

[0018] In one implementation, the depth features of the image to be processed include the depth value and the depth value confidence of each pixel in the image to be processed; the conversion unit converts the depth features into depth features in the polyhedral lattice space according to a preset depth feature conversion relationship in the following manner: for each pixel in the image to be processed, according to the preset depth feature conversion relationship, the depth value of the pixel is converted into a first depth corresponding value in the polyhedral lattice space, and the depth value confidence of the pixel is converted into a confidence corresponding value in the polyhedral lattice space; according to the first depth corresponding value in the polyhedral lattice space and a preset vector conversion method, a first vector corresponding to the pixel depth value is determined, and according to the depth value confidence corresponding value and the preset vector conversion method, a second vector corresponding to the pixel depth value confidence is determined, where the first vector and the second vector represent the depth features in the polyhedral lattice space.

[0019] In one implementation, the conversion unit converts the depth value of a pixel into a first depth corresponding value in the polyhedral lattice space according to the preset depth feature conversion relationship in the following manner: the depth value of each pixel in the image to be processed is converted into a first depth corresponding value of a corresponding vertex coordinate group in the polyhedral lattice space, where the first depth corresponding value of the vertex coordinate group includes the first depth corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate, the pixel depth value, and the pixel depth value confidence; the conversion of the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space includes: the depth value of each pixel in the image to be processed is converted into a confidence corresponding value of a corresponding vertex coordinate group in the polyhedral lattice space, where the confidence corresponding value of the vertex coordinate group includes the confidence corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate and the pixel depth value confidence.

[0020] In one implementation, the completion unit completes the depth features according to the depth features in the polyhedral lattice space to obtain the completed depth features in the following manner: traverse all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space, assign corresponding distance weights to the distances between every two vertices among all the vertices, construct a metric matrix, where the metric matrix is used to measure the distances between every two vertices among all the vertices; perform depth completion according to the metric matrix, the first vector, and the second vector to obtain the completed depth features, where the first vector and the second vector represent the depth features in the polyhedral lattice space.

[0021] In one implementation, the completion unit performs depth completion according to the metric matrix, the first vector, and the second vector in the following manner to obtain the completed depth feature: substituting the metric matrix, the first vector, and the second vector into a preset solution equation to obtain a third vector, where the third vector represents the completed depth feature.

[0022] In one implementation, the preset solution equation is solved based on the stabilized bi-conjugate gradient.

[0023] In one implementation, the inverse transformation unit inversely transforms the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature transformation relationship in the following manner: converting the third vector into the second depth corresponding value corresponding to each vertex coordinate in the vertex coordinate set, where the vertex coordinate set is the set of all vertices corresponding to the pixels in the image to be processed in the polyhedral lattice space; according to the preset depth feature transformation relationship, converting the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed, where the target depth value represents the target depth feature.

[0024] In one implementation, the inverse transformation unit converts the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed according to the preset depth feature transformation relationship in the following manner: for each pixel in the image to be processed, determining the preset number of vertex coordinates corresponding to the pixel, and determining the second depth corresponding value and weight corresponding to each vertex coordinate among the predetermined number of vertex coordinates; weighted summing the second depth corresponding values according to the weights, and determining the value obtained by the weighted summing as the target depth value.

[0025] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to: execute the image processing method described in the first aspect or any one of the implementations of the first aspect.

[0026] According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium, in which instructions are stored, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image processing method described in the first aspect or any one of the implementations of the first aspect.

[0027] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: For the image to be processed, its depth features are obtained. And the depth features to be processed are converted into depth features in the polyhedral lattice space according to a preset depth feature conversion relationship. The depth features in the polyhedral lattice space are complemented to obtain the complemented depth features. According to the preset depth feature conversion relationship, the complemented depth features in the polyhedral lattice space are inversely converted into the target depth features in the image to be processed, and the image to be processed after the inverse conversion process (i.e., the target image) is obtained. Through the present disclosure, during the process of complementing sparse depth to obtain dense depth, the amount of calculation is reduced, the overall operation speed is improved, and the sparse depth is complemented based on the depth features in the polyhedral lattice space, improving the depth complementation effect.

[0028] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0030] Figure 1 It is a schematic diagram of an application scenario of an image processing method shown according to an exemplary embodiment of the present disclosure.

[0031] Figure 2 It is a flowchart of an image processing method shown according to an exemplary embodiment.

[0032] Figure 3A and Figure 3B It is a schematic diagram of an image space conversion shown according to an exemplary embodiment of the present disclosure.

[0033] Figure 4 It is a flowchart of a method for determining a preset depth feature conversion relationship shown according to an exemplary embodiment.

[0034] Figure 5 It is a flowchart of a method for determining a preset depth feature conversion relationship shown according to an exemplary embodiment.

[0035] Figure 6 It is a schematic diagram of a storage method of a preset depth feature conversion relationship shown according to an exemplary embodiment of the present disclosure.

[0036] Figure 7 It is a schematic diagram of a storage method of a preset depth feature conversion relationship shown according to another exemplary embodiment of the present disclosure.

[0037] Figure 8It is a flowchart of a method for converting depth features into depth features in a polyhedral lattice space according to an exemplary embodiment.

[0038] Figure 9 It is a schematic diagram of a depth feature conversion method according to an exemplary embodiment of the present disclosure.

[0039] Figure 10 It is a flowchart of a method for performing depth feature completion according to an exemplary embodiment.

[0040] Figure 11 It is a flowchart of a method for performing depth feature completion according to an exemplary embodiment.

[0041] Figure 12 It is a flowchart of a method for inversely converting the completed depth features in the polyhedral lattice space into the target depth features in the image to be processed according to an exemplary embodiment.

[0042] Figure 13 It is a flowchart of a method for converting the second depth corresponding value corresponding to the vertex coordinates in the polyhedral lattice space into the target depth value in the image to be processed according to an exemplary embodiment.

[0043] Figure 14 It is a schematic diagram of a depth feature inverse conversion method according to an exemplary embodiment of the present disclosure.

[0044] Figure 15A and Figure 15B It is a schematic diagram of the processing results of different image processing methods according to an exemplary embodiment of the present disclosure.

[0045] Figure 16A and Figure 16B It is a schematic diagram of the processing results of different image processing methods according to an exemplary embodiment of the present disclosure.

[0046] Figure 17A It is an image to be processed according to an exemplary embodiment of the present disclosure.

[0047] Figure 17B and Figure 17C It is a schematic diagram of the processing results of different image processing methods according to an exemplary embodiment of the present disclosure.

[0048] Figure 18A It is an image to be processed according to an exemplary embodiment of the present disclosure.

[0049] Figure 18B and Figure 18C It is a schematic diagram of the processing results of different image processing methods according to an exemplary embodiment of the present disclosure.

[0050] Figure 19A is an image to be processed shown according to an exemplary embodiment of the present disclosure.

[0051] Figure 19B and Figure 19C are schematic diagrams of the processing results of different image processing methods shown according to an exemplary embodiment of the present disclosure.

[0052] Figure 20 is a block diagram of an image processing apparatus shown according to an exemplary embodiment.

[0053] Figure 21 is a block diagram of an apparatus for image processing shown according to an exemplary embodiment. Detailed implementation manners

[0054] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure.

[0055] The image processing method provided by the embodiments of the present disclosure is applied to a scenario of performing depth completion on a sparse depth image to obtain a dense depth image.

[0056] Some existing image processing methods require the use of depth data of images. The traditional method for obtaining a depth image is a binocular depth estimation method that calculates the disparity by matching the features of the left and right views. The depth information of the image can be further obtained based on the original image, and then the depth image corresponding to the original image can be obtained. However, there will be a large number of invalid disparities in the depth map obtained by the above binocular depth estimation method, that is, the obtained depth map image is sparse, which will interfere with many algorithms and applications that rely on depth information and affect the graphics processing effect. Therefore, after obtaining the sparse depth map corresponding to the original image by the binocular depth estimation method, it is a necessary technology to complete the sparse depth map to obtain a dense depth map.

[0057] In the related art, there are traditional depth completion methods based on bilateral filtering and depth completion methods based on deep learning.

[0058] In traditional depth completion methods, based on the idea of fast bilateral filtering, the vertices of the RGB image are segmented, enabling the RGB image to be transformed from the image space to the bilateral space, and the transformation relationship for mutual conversion between the image space and the bilateral space is obtained. Then, using this conversion relationship, the sparse depth corresponding to the RGB image is transformed from the image space to the bilateral space. Subsequently, iterative optimization is carried out in the bilateral space to complete the sparse depth completion. After the optimization result is transformed back to the image space in the bilateral space, the final dense depth is obtained. The vertex segmentation method used in the above traditional depth completion method can lead to severe block effects in the dense depth, with poor depth transition effects. Moreover, the obtained dense effect is relatively poor, showing phenomena such as blurred depth edges, incorrect depths at cutout areas, and incomplete completion. The computational cost in the iterative optimization stage is very high, resulting in low efficiency.

[0059] In depth completion based on deep learning methods, a deep learning model is obtained by pre-training with a large amount of training data (including a large number of corresponding RGB images, sparse depth maps, and dense depth maps). During the sparse depth completion process, the RGB image and the corresponding sparse depth map are input into the deep learning model to obtain the dense depth map. In the depth completion based on the above deep learning method, the computational cost is even higher, with high requirements for hardware. It relies on hardware platforms such as general-purpose digital signal processors (DSPs) and graphics processing units (GPUs) to achieve real-time computing and obtain dense depth images. Moreover, the training data required for training the deep learning model is difficult to obtain.

[0060] In view of this, the present disclosure proposes an image processing method. After obtaining the image to be processed, the depth features of the image to be processed are extracted. And according to the preset depth feature conversion relationship, the depth features in the image to be processed are converted into depth features in the polyhedral lattice space. Based on the depth features in the polyhedral lattice space, iterative optimization and depth feature completion are carried out to obtain the completed depth features. According to the preset depth feature conversion relationship, the completed depth features in the polyhedral lattice space are inversely converted into the target depth features in the image to be processed, and the image to be processed after the inverse conversion process (i.e., the target image) is obtained. Through the present disclosure, during the completion of sparse depth to obtain dense depth, the computational cost is reduced, the overall operation speed is improved, and the sparse depth is completed based on the depth features in the polyhedral lattice space, improving the depth completion effect.

[0061] The image processing proposed by the present disclosure is applied to the completion of sparse depth in a binocular stereo vision system to obtain dense depth. As Figure 1As shown in the schematic diagram of the application scenario of the image processing method. After obtaining views from different angles through a binocular camera, based on the binocular depth estimation method, features of views from different angles are matched through a stereo matching algorithm to obtain a sparse depth image. Furthermore, the sparse depth image obtained is subjected to sparse depth densification (i.e., sparse depth completion) through the image processing method of the present disclosure to obtain a dense depth image.

[0062] Figure 2 It is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 2 shown, the method includes steps S101 to S104.

[0063] In step S101, depth features of the image to be processed are obtained.

[0064] In step S102, according to a preset depth feature conversion relationship, the depth features of the image to be processed are converted into depth features in a polyhedral lattice space.

[0065] Among them, the preset depth feature conversion relationship is the correspondence between the depth features in the image to be processed and the depth features in the polyhedral lattice space, and the coordinates in the polyhedral lattice space are obtained by projecting a preset dimensional integer space onto a hyperplane with a normal vector of all-one vector.

[0066] In step S103, according to the depth features in the polyhedral lattice space, depth feature completion is performed to obtain the completed depth features.

[0067] In step S104, according to the preset depth feature conversion relationship, the completed depth features in the polyhedral lattice space are inversely converted into the target depth features in the image to be processed, and the image to be processed after the inverse conversion process is obtained, and the image to be processed obtained through the inverse conversion process is determined as the target image.

[0068] In the implementation of the present disclosure, the coordinates in the polyhedral lattice space are obtained by projecting a multi-dimensional integer space onto a hyperplane with a normal vector of all-one vector. The above preset dimension is the dimension of the polyhedral lattice space plus 1, where the dimension of the polyhedral lattice space is determined based on the number of dimensions of the coordinates to be mapped into the polyhedral lattice space. As Figure 3A and Figure 3B shown in the schematic diagram of image space conversion in, taking a 2D polyhedral lattice space as an example, the coordinates in its space are obtained by projecting the cube-shaped coordinates in a three-dimensional space onto a plane with a normal vector of [1, 1, 1] T The bilateral space coordinates [x i , y i , r i , g i , b iIt includes five dimensions. Therefore, the polyhedral lattice space in the present disclosure is a five-dimensional polyhedral lattice space, and the corresponding multi-dimensional integer space is a six-dimensional integer space.

[0069] In an embodiment of the present disclosure, after obtaining the image to be processed, the depth feature of the image to be processed is extracted, and the depth feature in the image to be processed is converted into the depth feature in the polyhedral lattice space. Depth feature completion is performed in the polyhedral lattice space to obtain the completed depth feature. After depth completion, the completed depth feature in the polyhedral lattice space is inversely converted into the target depth feature in the image to be processed, and the image to be processed after the inverse conversion process (i.e., the target image) is obtained.

[0070] In an embodiment of the present disclosure, the bilateral space coordinates of the image to be processed are converted into vertex coordinates in the polyhedral lattice space. Compared with the traditional sparse depth completion method, the number of vertices used for iterative optimization after vertex segmentation is smaller. Therefore, during the sparse depth completion to obtain the dense depth, the computational amount is reduced and the overall operation speed is improved. Moreover, the present disclosure performs sparse depth completion based on the vertex coordinates in the polyhedral lattice space, and the vertex segmentation is more refined, ensuring the depth completion effect.

[0071] In an embodiment of the present disclosure, each pixel in the image to be processed respectively has a corresponding image depth feature and bilateral coordinates, and the depth feature of the pixel is associated with the bilateral coordinates. Based on the correspondence between the bilateral coordinates in the image to be processed and the vertex coordinates in the polyhedral lattice space, and combined with the correspondence between the pixel, the corresponding pixel bilateral coordinates, and the pixel depth feature in the image to be processed, the depth feature of the pixel in the image to be processed can be converted into the depth feature in the polyhedral lattice space. The following embodiments of the present disclosure illustrate the method for determining the preset depth feature conversion relationship.

[0072] Figure 4 is a flowchart of a method for determining a preset depth feature conversion relationship shown according to an exemplary embodiment. As Figure 4 shown, the method includes steps S201 to S204.

[0073] In step S201, the bilateral space coordinates of each pixel in the image to be processed are obtained.

[0074] Among them, the sub-coordinates constituting the bilateral space coordinates include the spatial sub-coordinates and color sub-coordinates of the pixel.

[0075] In step S202, the mapping coordinates of the bilateral space coordinates mapped to the polyhedral lattice space are determined, and the vertex coordinate group and the weight corresponding to the vertex coordinate group in the polyhedral lattice space are determined according to the mapping coordinates.

[0076] Among them, the vertex coordinate group includes a preset number of vertex coordinates closest to the mapping coordinates. The weights corresponding to the vertex coordinate group include the weights corresponding to each vertex coordinate in the preset number of vertex coordinates. The preset number corresponds to the number of sub-coordinates that make up the bilateral space coordinates. The weight corresponding to the vertex coordinate represents the distance between the vertex coordinate and the corresponding mapping coordinate.

[0077] In step S203, determine a preset coordinate conversion relationship among the bilateral space coordinates of each pixel in the image to be processed, the vertex coordinate group corresponding to the bilateral space coordinates of each pixel, and the weights of the vertex coordinate group corresponding to the bilateral space coordinates of each pixel.

[0078] In step S204, determine a preset depth feature conversion relationship according to the preset coordinate conversion relationship.

[0079] In the embodiments of the present disclosure, the image space of the RGB image is a two-dimensional space. Each pixel of the RGB image has corresponding pixel space coordinates (including the abscissa x i and the ordinate y i ), and corresponding pixel color coordinates (including the red channel value r i , the green channel value g i and the blue channel value b i ). The mapping relationship between the pixel space coordinates and the pixel color coordinates satisfies f(x i , y i ) = [r i , g i , b i , where i represents the serial number of the pixel in the RGB image. The bilateral space refers to the space jointly composed of the position space (spatial space) and the color space (range space). Among them, the position space refers to the space composed of the abscissa and the ordinate, and the color space refers to the space composed of the color values of the RGB three channels. The bilateral space is a five-dimensional space, and its coordinates are expressed as [x i , y i , r i , g i , b i . Therefore, the bilateral space coordinates in the image to be processed in the present disclosure are [x i , y i , r i , g i , b i .

[0080] In the embodiments of the present disclosure, the polyhedral lattice space has the following properties: (1) The vertex coordinates in a d-dimensional (where d is the number of dimensions of the polyhedral lattice space) polyhedral lattice space are (d + 1)-dimensional. This is caused by the definition of the polyhedral lattice space itself. Therefore, the vertex coordinates of the polyhedral lattice space are over-complete, and the sum of all coordinate values of each vertex coordinate is 0. (2) After the coordinate points of a d-dimensional bilateral space are mapped to a d-dimensional polyhedral lattice space, there are exactly (d + 1) polyhedral lattice vertices closest to this point. The (d + 1) vertices are connected to form a regular polygon, which is called a "simplex". For example Figure 3B As shown in the polyhedral lattice space after the image space conversion in, the simplex is an equilateral triangle. (3) The coordinate of each vertex in the d-dimensional polyhedral lattice space has the same remainder when divided by (d + 1). The remainders of different vertices may be different. When the remainder is k, it is called a k-congruent point. (4) Each simplex in the d-dimensional polyhedral lattice space contains (d + 1) vertices, which are the 0-congruent point to the d-congruent point, a total of (d + 1) congruent points. (5) There are 2*(d + 1) closest vertices with the same distance around the origin of the d-dimensional polyhedral lattice space. The convex polygon formed by these vertices is called a meta-form. For example Figure 3B As shown in, in the 2-dimensional polyhedral lattice space, the meta-form is a regular hexagon. The simplex has translational invariance in the polyhedral lattice space, that is, any simplex can be translated into the meta-form. This property provides convenience for calculating the vertex coordinates corresponding to the bilateral space coordinate points. (6) The meta-form of the polyhedral lattice space contains multiple simplices. According to the sorting of the coordinates in the simplex, the position of this point in the simplex it belongs to can be determined, as Figure 3B shown. If the sorting of the coordinates itself is in descending order, the simplex it belongs to is called the base simplex.

[0081] In the embodiments of the present disclosure, for the above-mentioned inherent properties of the polyhedral lattice space, vertex splitting is performed on the mapping coordinates obtained by mapping the bilateral space coordinates to the polyhedral lattice space to obtain multiple vertex coordinates (vertex coordinate groups) corresponding to the bilateral space coordinates. The basic process of vertex splitting of the mapping coordinates is as follows: Map the bilateral space coordinates in the to-be-processed to the mapping coordinates of the polyhedral lattice space, calculate several vertex coordinates closest to the mapping coordinates, and solve the weight of each vertex with respect to the space mapping coordinates. This weight represents the distance from the space mapping coordinates to the above-mentioned several vertices. In the present disclosure, multiple mapping coordinates may correspond to the same set of vertex coordinates in the polyhedral lattice space. The purpose of vertex splitting in the present disclosure is to divide the pixel points with relatively close spatial distances and similar colors in the image space into the same vertex coordinate groups in the polyhedral lattice space.

[0082] In the embodiments of the present disclosure, the specific steps for performing vertex splitting on the mapping coordinates obtained by mapping the bilateral space coordinates to the polyhedral lattice space to obtain multiple vertex coordinates corresponding to the bilateral space coordinates are as follows:

[0083] 1. Downsample the bilateral spatial coordinates to obtain the downsampled bilateral spatial coordinates p i , and the expression for downsampling the bilateral spatial coordinates is: where σ s represents the downsampling rate of the position space, and σ r represents the downsampling rate of the color space, where i represents the i-th pixel point.

[0084] 2. Map p i to the polytope lattice space according to the following manner based on the preset parameter E to obtain the mapped coordinates after mapping.

[0085] Among them, the expression of the preset parameter E is:

[0086] The expression for obtaining the mapped coordinates after mapping is: y i = Ep i

[0087] where d represents the dimension of the polytope lattice space, and in this disclosure, d = 5.

[0088] 3. Calculate the vertex coordinates of the 0-congruent point in the simplex where the mapped coordinate y i is located

[0089] Among them, the expression of the vertex coordinates of the 0-congruent point is:

[0090] where is the vertex coordinate of the 0-congruent point including the subscript j (i.e., ), is the residual vector, representing the distance from the current point to the 0-congruent point. The subscript i represents the i-th pixel point, and the subscript j represents the j-th element in the vector. Using the translation invariance of the vertex (i.e., any simplex can be translated to the meta-simplex), the residual vector is equivalent to translating the current point to the meta-simplex, which is convenient for subsequent solving of the vertex coordinates of other k-congruent points. Among them, k = 0,..., d.

[0091] 4. Calculate the vertex coordinates of other k-congruent points in the simplex where the mapped coordinate y i is located. Since the residual vector in the above step is equivalent to translating y i to the meta-simplex, it is only necessary to find the vertex coordinates of other k-congruent points in the prototype and then translate them back to obtain the vertex coordinates of other k-congruent points.

[0092] The process for obtaining the vertex coordinates of other k-congruent points is:

[0093] (1) Sort the coordinate values in the residual vector in descending order, and record the index of each element in the vector after sorting;

[0094] (2) Calculate the coordinates of k congruent points in the basic simplex, and the calculation formula is as follows:

[0095] where s k corresponds to multiple congruent points of the basic simplex in the meta-simplex. When d = 5, the 6 congruent points corresponding to the basic simplex in the meta-simplex are:

[0096] s0 = [0,0,0,0,0,0]

[0097] s1 = [1,1,1,1,1,-5]

[0098] s2 = [2,2,2,2,-4,-4]

[0099] s3 = [3,3,3,-3,-3,-3]

[0100] s4 = [4,4,-2,-2,-2,-2]

[0101] s5 = [5,-1,-1,-1,-1,-1]

[0102] (3) Rearrange the coordinates of the 6 congruent points according to the index of each element after sorting in (1) to obtain all the congruent points corresponding to the simplex where the residual vector is located, denoted as s k '.

[0103] (4) Using the translation invariance, the vertex coordinates of all k-congruent points corresponding to the mapping coordinate y i can be obtained as

[0104] 5. Calculate the weight b i between the point y i,k (k = 0,..., d).

[0105] i represents the i-th pixel point.

[0106] 6. Obtain the d + 1 vertex coordinates corresponding to each pixel point in the image to be processed and their corresponding d + 1 weights.

[0107] 7. Traverse each pixel point in the image to solve the corresponding vertex coordinates and weights.

[0108] In the disclosed embodiments, a correspondence relationship is established between the bilateral spatial coordinates in the image to be processed and the vertex coordinates in the polyhedral lattice space through the vertex segmentation method. By vertex segmentation, pixel points with similar distances and colors in the image space are grouped into the same vertex coordinate group in the polyhedral lattice space, so that the number of vertices after division is much smaller than the number of pixel points. Since each pixel point is independent during calculation, the present disclosure can be highly parallelized to improve the calculation speed and reduce the calculation amount of subsequent iterative optimization and image depth completion steps.

[0109] In the embodiments of the present disclosure, there is an association relationship between pixels, the depth features of pixels, and the coordinates of pixels. Combining the above-determined preset coordinate conversion relationship (i.e., the bilateral coordinates in the image to be processed and the vertex coordinates in the polyhedral lattice space), the depth features corresponding to the vertices in the polyhedral lattice space can be obtained based on the depth features of the pixels in the original image, and then the correspondence relationship between the depth features in the original image and the depth features in the polyhedral lattice space, that is, the preset depth conversion relationship, can be obtained. The following embodiments of the present disclosure further illustrate the method for determining the preset depth feature conversion relationship.

[0110] Figure 5 is a flowchart of a method for determining a preset depth feature conversion relationship shown according to an exemplary embodiment. As Figure 5 shown, the method includes steps S301 to S303.

[0111] In step S301, all vertex coordinate groups corresponding to all bilateral spatial coordinates in the polyhedral lattice space are determined according to the preset coordinate conversion relationship, all the vertex coordinates included in all the vertex coordinate groups are sorted and numbered, and the coordinate values of all the vertex coordinates are encoded as hash values, and all the vertex coordinate numbers and hash values are stored in the form of key-value pairs.

[0112] In step S302, all the pixels in the image to be processed are sorted and numbered.

[0113] In step S303, for each vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel among all the pixels, the pixel number, the vertex coordinate number, and the vertex position number are associated with a preset number of vertex coordinates included in the vertex coordinate group, and the pixel number, the vertex coordinate number, and the vertex position number are associated with the weights corresponding to the preset number of vertex coordinates, so as to obtain the preset depth feature conversion relationship.

[0114] In the embodiments of the present disclosure, each coordinate of the vertices of the polyhedral lattice is recorded with an 8-bit width. When d = 5, a total of 48 bits [(5 + 1) * 8 bits] of width are required. Hash encoding is performed on the vertices, and the long long type in C / C++ is used to store the hash values. When the coordinates of the polyhedral lattice vertices are negative, their complements are used as the corresponding hash values. As Figure 6 shown in the schematic diagram of the storage method of the preset depth feature conversion relationship in . The storage method of the present disclosure for a set of vertex coordinates is: in the form of store the multiple vertex coordinates included in a single set of vertices one by one from the high bit to the low bit: where j represents the sorting of the vertex coordinates in the same group, such as represents the j-th element in the vertex coordinate vector

[0115] In the embodiments of the present disclosure, after obtaining the vertex coordinates corresponding to each pixel and performing hash encoding on the vertices, the hash encodings of all vertices are stored in an unordered dictionary. The key-value pairs in the dictionary are hash value - vertex number (vertex coordinate number). Among them, the vertex number is the label of all vertices in the polyhedral lattice space that have a corresponding relationship with the pixels of the image to be processed, including 0 to N - 1, where N represents the number of vertices. According to the above dictionary, record the numbers and weights of all vertices that have a corresponding relationship with the pixels of the image to be processed, and based on the pixel number division, include multiple groups of d + 1 vertex numbers and the corresponding d + 1 weights. In the present disclosure, the correspondence relationship between the pixel number, vertex coordinate number, vertex position number, and weight represents the transformation relationship between the bilateral space and the polyhedral lattice space. As Figure 7 shown in the schematic diagram of the storage method of the preset depth feature conversion relationship in

[0116] Vertex numbers [VIdx0, VIdx1, VIdx2, VIdx3, VIdx4, VIdx5],

[0117] Vertex weights [w i,0 , w i,1 , w i,2 , w i,3 , w i,4 , w i,5

[0118] where VIdx0, VIdx1, VIdx2, VIdx3, VIdx4, VIdx5 are vertex numbers (i.e., vertex coordinate numbers), i represents the i-th pixel point, and the subscripts 1 - 5 in the vertex weights are vertex position numbers.

[0119] ​In the embodiments of the present disclosure, a preset depth feature conversion relationship is obtained based on the determined preset coordinate conversion relationship, and the preset depth conversion relationship is stored, which is convenient for quickly obtaining the parameters required for operations based on the stored fast preset depth feature conversion relationship during subsequent operation processing, thereby improving the image processing efficiency.

[0120] In the embodiments of the present disclosure, after obtaining the preset depth feature conversion relationship, the depth features of the image to be processed are converted into depth features in the polyhedral lattice space, and then depth completion is performed based on the depth features in the polyhedral lattice space. The following embodiments of the present disclosure illustrate the conversion process of the depth features in the image to be processed.

[0121] Figure 8 It is a flowchart of a method for converting depth features into depth features in a polyhedral lattice space shown according to an exemplary embodiment. As Figure 8 shown, the method includes steps S401 to S402.

[0122] In step S401, for each pixel in the image to be processed, according to the preset depth feature conversion relationship, the depth value of the pixel is converted into a first depth corresponding value in the polyhedral lattice space, and the confidence of the depth value of the pixel is converted into a confidence corresponding value in the polyhedral lattice space.

[0123] In step S402, according to the first depth corresponding value in the polyhedral lattice space and the preset vector conversion method, a first vector corresponding to the pixel depth value is determined, and according to the confidence corresponding value of the depth value and the preset vector conversion method, a second vector corresponding to the confidence of the pixel depth value is determined. The first vector and the second vector represent the depth features in the polyhedral lattice space.

[0124] In the embodiments of the present disclosure, the depth features of the image to be processed include the depth value and the confidence of the depth value of each pixel in the image to be processed. After obtaining the depth value and the corresponding confidence of the depth value of each pixel in the image to be processed, according to the preset depth feature conversion relationship, the depth value and the corresponding confidence of the depth value of each pixel in the image to be processed are converted into corresponding values in the polyhedral lattice space (i.e., the first depth corresponding value and the confidence corresponding value). Then, further conversion processing is performed on the corresponding values in the polyhedral lattice space to obtain a first vector and a second vector representing the depth features in the polyhedral lattice space. Among them, the first vector corresponds to the first depth corresponding value in the polyhedral lattice space and indirectly corresponds to the pixel value in the image to be processed; the second vector corresponds to the confidence corresponding value in the polyhedral lattice space and indirectly corresponds to the confidence of the pixel value in the image to be processed.

[0125] In the disclosed embodiment, according to a preset depth feature conversion relationship and a preset vector conversion method, the depth value and the corresponding depth value confidence in the image to be processed are converted into corresponding values in the polyhedron lattice space (i.e., a first depth corresponding value and a confidence corresponding value), and the corresponding value in the polyhedron lattice space is converted into a vector (including a first vector and a second vector) representing the depth feature in the polyhedron lattice space, so as to facilitate the subsequent processing process to substitute the calculation formula for image depth completion, perform optimization iteration, complete image depth completion, and obtain the target image.

[0126] In the embodiment of the present disclosure, a single pixel in the image to be processed corresponds to a plurality of vertices grouped in the polyhedron lattice space, so the depth feature (including the depth value and the depth value confidence) of the pixel in the image to be processed can be apportioned to the corresponding plurality of vertices in the polyhedron lattice space. The following embodiment of the present disclosure further describes the depth feature conversion method.

[0127] In one implementation of the disclosed embodiment, according to a preset depth feature conversion relationship, the depth value of a pixel is converted into a first depth corresponding value in a polyhedron lattice space, including: converting the depth value of each pixel in the image to be processed into a first depth corresponding value of a corresponding vertex coordinate group in the polyhedron lattice space, the first depth corresponding value of the vertex coordinate group includes the first depth corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate, the pixel depth value, and the pixel depth value confidence. Converting the pixel depth value confidence into a confidence corresponding value in a polyhedron lattice space includes: converting the depth value of each pixel in the image to be processed into a confidence corresponding value of a corresponding vertex coordinate group in the polyhedron lattice space, the confidence corresponding value of the vertex coordinate group includes the confidence corresponding value of each vertex coordinate in the vertex coordinate group, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate and the pixel depth value confidence.

[0128] In the embodiment of the present disclosure, the present disclosure uses the following method to preliminarily convert the depth features in the image to be processed into the polyhedron lattice space, that is, to obtain the depth corresponding value corresponding to the depth value, and to obtain the confidence corresponding value corresponding to the depth value confidence:

[0129] 1. For a pixel point (such as the i-th pixel point) in the sparse depth image (i.e., the image to be processed), obtain its corresponding sparse depth (depth value) (such as the depth value d corresponding to the i-th pixel point) i ) and confidence (depth value confidence) (such as the depth value confidence c corresponding to the i-th pixel i ) According to the correspondence between the pixel points and vertex numbers and weights stored in the above unordered dictionary, the value allocated to each vertex (such as the d corresponding to the i-th pixel point) is calculated. i *c i)。

[0130] 2. Traverse all the pixel points in the image to be processed. It can be understood that a vertex in the polyhedral lattice space corresponds to multiple pixel points. Therefore, when traversing all the pixel points, for each pixel, obtain the product of the pixel depth value, the confidence of the pixel depth value, and the vertex weight (the first depth corresponding value), and repeatedly accumulate the values allocated to the corresponding vertex by the pixel points (the first depth corresponding value). That is, for a single vertex, when corresponding to different pixels, the value of the single vertex will be repeatedly accumulated. Finally, the sparse depth image (i.e., the image to be processed) is converted into a vector (the first vector) of length N, denoted as b, where N represents the number of vertices. Each element in this vector represents the sum of the sparse depths allocated to the vertex with the corresponding serial number.

[0131] 3. Traverse all the pixel points in the image to be processed. When traversing all the pixel points, for each pixel, obtain the product of the confidence of the pixel depth value and the confidence of the vertex (the confidence corresponding value), and repeatedly accumulate the values allocated to the corresponding vertex by the pixel points (the confidence corresponding value). That is, for a single vertex, when corresponding to different pixels, the value of the single vertex will be repeatedly accumulated. Finally, the sparse depth image (i.e., the image to be processed) is converted into a vector (the second vector) of length N, denoted as c, where N represents the number of vertices. Each element in this vector represents the sum of the depth confidences allocated to the vertex with the corresponding serial number.

[0132] In an exemplary embodiment of the present disclosure, the pixels participating in the calculation in the image to be processed include the pixel with serial number i and the pixel with serial number j. As Figure 9 shown in the schematic diagram of the depth feature conversion method, the depth conversion relationships of the two pixels are respectively:

[0133] Pixel with serial number i:

[0134] Vertex serial numbers [VIdx0, VIdx1, VIdx2, VIdx3, VIdx4, VIdx5]

[0135] Conversion result [w i，0 d i c i ,w i，1 d i c i ,w i，2 d i c i ,w i，3 d i c i ,w i，4 d i c i ,w i，5 d i c i]

[0136] Pixel numbered j:

[0137] Vertex number [VIdx1, VIdx2, VIdx6, VIdx7, VIdx8, VIdx9]

[0138] Conversion result [w j，0 d j c j , w j，1 d j c j , w j，2 d j c j , w j，3 d j c j , w j，4 d j c j , w j，5 d j c j ]

[0139] Among them, the pixel with serial number i and the pixel with serial number j correspond to the same vertex, namely VIdx1 and VIdx2. It can be understood that the above-mentioned pixel with serial number i and the pixel with serial number j are only two of all the pixels involved in the calculation. All pixels and parameters corresponding to the pixels in the image to be processed in the present disclosure need to participate in the calculation.

[0140] For the above embodiment, when converting the sparse depth feature to the polyhedral lattice space, all the pixels are traversed and the values of the corresponding vertices of the pixels (i.e., depth value * vertex weight * depth value confidence) are accumulated. Finally, a grayscale image is converted into a vector with a length of N (N represents the number of vertices), i.e., the first vector b, to complete the conversion of the sparse depth feature. Among them, the way to obtain the first vector b is as follows:

[0141] c=[w i，0 d i c i , w i，1 d i c i +w j，0 d j c j , w i，2 d i c i +w j，1 d j c j , w i，3 d i c i ,......], where wi，0 is the vertex weight, d i is the depth value, c i is the depth value confidence.

[0142] Similarly, when converting the sparse depth value confidence to the polyhedral lattice space, all pixel points are traversed, and the values allocated to the corresponding vertices of the pixel points (i.e., vertex weight * depth value confidence) are accumulated. Finally, a vector of length N (where N represents the number of vertices) is obtained, which is the second vector c, completing the conversion of the depth value confidence. Among them, the method for obtaining the second vector c is as follows:

[0143] c = [w i，0 c i , w i，1 c i + w j，0 c j , w i，2 c i + w j，1 c j , w i，3 c i ,......], where w i，0 is the vertex weight, c i is the depth value confidence.

[0144] In the embodiments of the present disclosure, after obtaining the first vector and the second vector representing the depth features in the polyhedral lattice space, it is also necessary to further obtain the distance relationship between all vertices (vertices corresponding to the pixels of the image to be processed) in the polyhedral lattice space. Then, based on the first vector, the second vector, and the distance relationship between all vertices in the polyhedral lattice space, depth completion is performed. The following embodiments of the present disclosure illustrate the method for depth feature completion.

[0145] Figure 10 is a flowchart of a method for depth feature completion shown according to an exemplary embodiment. As Figure 10 shown, the method includes steps S501 to S502.

[0146] In step S501, all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space are traversed, and corresponding distance weights are assigned to the distances between every two vertices among all vertices to construct a metric matrix.

[0147] Among them, the metric matrix is used to measure the distances between every two vertices among all vertices.

[0148] In step S502, depth completion is performed according to the metric matrix, the first vector, and the second vector to obtain the completed depth features. The first vector and the second vector represent the depth features in the polyhedral lattice space.

[0149] In the embodiments of the present disclosure, a metric matrix is used to measure the distance between any two vertices. The construction method of the metric matrix is as follows:

[0150] 1. Construct a sparse matrix of size N*N, where the subscripts of rows and columns correspond to the serial numbers of vertices, and each element of the matrix represents the correlation between the corresponding two vertices. Here, N represents the number of vertices. For vertex j, its coordinates are stored in an unordered dictionary, denoted as vertex coordinates v j =[v j,0 ,v j,1 ,v j,2 ,v j,3 ,v j,4 ,v j,5 . Given that the polyhedral lattice space in the present disclosure is a five-dimensional polyhedral lattice space, this coordinate has a total of 6 (i.e., 5 + 1) dimensions. For the first dimension of the coordinate, calculate the coordinates of vertices with a distance of 1, such as [v j,0 +1,v j,1 ,v j,2 ,v j,3 ,v j,4 ,v j,5 and [v j,0 -1,v j,1 ,v j,2 ,v j,3 ,v j,4 ,v j,5 . Then, look up the serial numbers of the vertices corresponding to the coordinates of the vertices with a distance of 1 in the obtained unordered dictionary, denoted as n0 and n1, and assign corresponding weights to the elements at positions (j, n0), (n0, j), (j, n1), and (n1, j) in the N*N sparse matrix. In one example, the weight is defaulted to 1. Also assign corresponding weights to the elements at position (j, j) in the sparse matrix. In one example, the weight is set to 2. The values of the above two weights can be changed based on the operation requirements. Given that the elements on the diagonal represent the weights of vertices to themselves, they are usually greater than other weights. Further, using the same method for vertex j, traverse all vertices to complete the construction of the sparse matrix.

[0151] 3. In the same way as in step 1, create a corresponding sparse matrix for each dimension, that is, a total of 6 sparse matrices. Multiply the 6 sparse matrices to obtain the final metric matrix, denoted as B.

[0152] In the embodiments of the present disclosure, the above-mentioned first vector, second vector, and metric matrix are necessary parameters for sparse depth completion. After obtaining the above-mentioned first vector, second vector, and metric matrix, the first vector, second vector, and metric matrix are input into a preset solution equation for depth completion to perform depth completion and obtain the completed depth features. The following embodiments of the present disclosure further illustrate the process of depth completion.

[0153] Figure 11 It is a flowchart of a method for depth feature completion shown according to an exemplary embodiment. As Figure 11 shown, the method includes steps S601 to S602.

[0154] In step S601, the metric matrix, the first vector, and the second vector are substituted into the preset solution equation.

[0155] In step S602, a third vector is obtained, and the third vector represents the completed depth features.

[0156] In the embodiments of the present disclosure, the metric matrix represents the distances between pairs of vertices (all vertices corresponding to the pixels of the image to be processed) in the polyhedral lattice space, and the first vector and the second vector represent the depth features in the polyhedral lattice space, indirectly representing the depth features in the image to be processed. Substituting the above-mentioned metric matrix, first vector, and second vector into the preset solution equation can perform depth completion on the depth features and obtain the completed depth features (i.e., the third vector). It can be understood that the third vector is essentially the sum of multiple parameters, and the multiple parameters constituting the third vector correspond to the vertices (all vertices corresponding to the pixels of the image to be processed) in the polyhedral lattice space.

[0157] The following embodiments of the present disclosure further illustrate the preset solution equation.

[0158] In one implementation manner of the embodiments of the present disclosure, the preset solution equation is: [λB + diag(c)] * x = b; where B is the metric matrix, b is the first vector, and c is the second vector. Among them, diag() is used for extracting the diagonal elements of the matrix and creating a diagonal matrix.

[0159] In the embodiments of the present disclosure, the first vector, the second vector, and the metric matrix are substituted into the linear equation system (preset solution equation) [λB + diag(c)] * x = b. Where λ represents the smoothing term coefficient, and the smoothing term coefficient can be set artificially based on requirements. The larger the smoothing term coefficient, the smoother the obtained dense depth map. x is a vector (third vector) with a length of N (N represents the number of vertices), which is the target to be solved by the preset solution equation of the present disclosure.

[0160] The following embodiments of the present disclosure illustrate the solution method of the preset solution equation.

[0161] In one implementation of the embodiments of the present disclosure, the preset solution equation is solved based on the stabilized bi-conjugate gradient.

[0162] In the embodiments of the present disclosure, the coefficient matrix of the above linear equation system (preset solution equation) is an asymmetric sparse matrix. The present disclosure uses the stabilized bi-conjugate gradient method for iterative optimization, and the convergence rate of the stabilized bi-conjugate gradient method is faster than that of the iterative optimization method used in the traditional method solution. Therefore, the solution method based on the stabilized bi-conjugate gradient method in the present disclosure can improve the speed of iterative optimization and depth completion, and thus improve the efficiency of image processing.

[0163] In the embodiments of the present disclosure, the obtained third vector is the depth feature in the polyhedral lattice space. Based on the obtained depth feature conversion relationship, the completed depth feature in the polyhedral lattice space can be inversely converted into the depth feature in the image to be processed. In the following embodiments of the present disclosure, the method for obtaining the target depth feature in the image to be processed will be further described.

[0164] Figure 12 is a flowchart of a method for inversely converting the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to an exemplary embodiment. As Figure 12 shown, the method includes steps S701 to S702.

[0165] In step S701, the third vector is converted into the corresponding second depth corresponding value of each vertex coordinate in the vertex coordinate set, and the vertex coordinate set is the set of all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space.

[0166] In step S702, according to the preset depth feature conversion relationship, the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space is converted into the target depth value in the image to be processed, and the target depth value represents the target depth feature.

[0167] In the embodiments of the present disclosure, the above-mentioned third vector representing the polyhedral lattice space is essentially a vector obtained by adding multiple parameters, and each parameter contains the corresponding pixel and vertex numbers such as the preset pixel number, vertex coordinate number, and vertex position number. Through the above numbers, the third vector can be divided into multiple parameters (second depth corresponding values) corresponding to different vertices. Furthermore, based on the preset depth feature conversion relationship, the multiple parameters corresponding to the vertices in the polyhedral lattice space are inversely converted into the pixels of the image to be processed, and the target depth value corresponding to each pixel in the image to be processed is obtained, and the depth feature of the image to be processed is completed into the target depth feature. That is, the sparse depth image is completed into a dense depth image.

[0168] The following embodiments of the present disclosure illustrate a method for obtaining a target depth value.

[0169] Figure 13 It is a flowchart of a method for converting a second depth corresponding value corresponding to vertex coordinates in a polyhedral lattice space into a target depth value in a to-be-processed image shown according to an exemplary embodiment. As Figure 13 shown, the method includes steps S801 to S802.

[0170] In step S801, for each pixel in the to-be-processed image, a preset number of vertex coordinates corresponding to the pixel are determined, and a second depth corresponding value and a weight corresponding to each vertex coordinate among the predetermined number of vertex coordinates are determined.

[0171] In step S802, the second depth corresponding values are weighted and summed according to the weights, and the value obtained by the weighted sum is determined as the target depth value.

[0172] In the embodiments of the present disclosure, the third vector x obtained by iterative solution based on a preset solution equation is transformed back from the polyhedral lattice space to the image space. As Figure 14 shown in the schematic diagram of the depth feature inverse conversion method. The specific steps of this solution are as follows:

[0173] 1. For the i-th pixel point, the element corresponding to the position in the vector x is found according to the corresponding vertex number, and then weighted summation is performed with the corresponding weight.

[0174] where the vector x is denoted as: x = [x0, x1,..., x N-1 .

[0175] The weighted summation formula is

[0176] where x0 to x N- 1 are parameters (second depth corresponding values) corresponding to each vertex in the polyhedral lattice space, is corresponding to x0 to x N-1 , VIdx k corresponds to 0 to N - 1, W i，k corresponds to the vertex weight, i corresponds to the pixel number, and k corresponds to the vertex position number.

[0177] 2. Traverse all pixel points, and obtain the target depth value corresponding to each vertex in the manner of step 1 above, then the final dense depth can be obtained, and the image obtained after the inverse conversion is determined as the target image (i.e., the dense depth image).

[0178] In the embodiments of the present disclosure, the following method is adopted to perform depth completion processing on the image to be processed to obtain the target image; Vertex segmentation: Perform polyhedral lattice vertex segmentation on the image to be processed (sparse depth image), and calculate the coordinates and weights of the polyhedral lattice vertices corresponding to each pixel point. Hash encoding: Perform hash encoding on all polyhedral lattice vertex coordinates, and calculate the transformation relationship between the bilateral space vertices and the polyhedral lattice vertices. Depth transformation: According to the transformation relationship between the bilateral space vertices and the polyhedral lattice vertices, transform the sparse depth from the bilateral space to the polyhedral lattice space. Construct a metric matrix: Traverse all vertices, and assign corresponding weights according to the distance between each two vertices, thereby constructing a metric matrix. Iterative optimization: Based on the metric matrix, perform iterative optimization on the transformed sparse depth to perform depth filling. Depth inverse transformation: Inverse transform the result of iterative optimization from the polyhedral lattice space back to the bilateral space to obtain the filled depth map, that is, the dense depth image. Through the present disclosure, the obtained target image has a good depth completion effect, that is, the vertex segmentation is finer, the depth calculation result in the hollow area is more accurate, and the edge of the obtained dense depth is more accurate. Further, in the present disclosure, less time is consumed in the operation process of iterative optimization. Most of the operation time consumed by the present disclosure is concentrated on the first step (polyhedral lattice vertex segmentation), and the first step can be parallelized, greatly improving the operation speed.

[0179] In the embodiments of the present disclosure, the above image processing method is applied to an electronic device supporting image processing. The electronic devices in the present disclosure include but are not limited to mobile phones, tablets, computers, and other electronic devices supporting image processing.

[0180] In an exemplary embodiment of the present disclosure, as Figure 15A and Figure 15B shown in the schematic diagrams of the processing results of different image processing methods. Among them Figure 15A is the target image obtained by the traditional depth completion method, Figure 15B is the target image obtained by the image processing method of the present disclosure. Comparing 15A and Figure 15B it can be seen that compared with the traditional method solution, the effect of the technical solution of the present disclosure is that the vertex segmentation of the processing result is finer.

[0181] In an exemplary embodiment of the present disclosure, as Figure 16A and Figure 16B shown in the schematic diagrams of the processing results of different image processing methods. Among them Figure 16A is the target image obtained by the traditional depth completion method, Figure 16B is the target image obtained by the image processing method of the present disclosure. Comparing 16A and Figure 16B it can be seen that compared with the traditional method solution, the effect of the technical solution of the present disclosure is that there is no block effect in the image vertices of the processing result.

[0182] In an exemplary embodiment of the present disclosure, as Figure 17A the image to be processed in Figure 17B and Figure 17C shown in the schematic diagrams of the processing results of different image processing methods. Among them Figure 17B is the target image obtained by the traditional depth completion method, Figure 17C is the target image obtained by the image processing method of the present disclosure. Comparing 17B and Figure 17C it can be seen that compared with the solution of the traditional method, the effect of the technical solution of the present disclosure is that the depth calculation result in the hollowed-out area of the processing result is more accurate.

[0183] In an exemplary embodiment of the present disclosure, as Figure 18A the image to be processed in Figure 18B and Figure 18C shown in the schematic diagrams of the processing results of different image processing methods. Among them Figure 18B is the target image obtained by the traditional depth completion method, Figure 18C is the target image obtained by the image processing method of the present disclosure. Comparing 18B and Figure 18C it can be seen that compared with the solution of the traditional method, the effect of the technical solution of the present disclosure is that the edge of the obtained dense depth is more accurate.

[0184] In an exemplary embodiment of the present disclosure, as Figure 19A the image to be processed in Figure 19B and Figure 19C shown in the schematic diagrams of the processing results of different image processing methods. The edge of the obtained dense depth is more accurate. Among them Figure 19B is the target image obtained by the traditional depth completion method, Figure 19C is the target image obtained by the image processing method of the present disclosure. Comparing 19B and Figure 19C it can be seen that compared with the solution of the traditional method, the effect of the technical solution of the present disclosure is that the dense depth will not have the phenomenon of incomplete filling.

[0185] In an exemplary embodiment of the present disclosure, compared with the solution of the traditional method, the vertex segmentation method used in the technical solution of the present disclosure has fewer vertices, which is 1 / 3 of it. Therefore, the computational complexity of the technical solution in the iterative optimization step is less.

[0186] Based on the same concept, the embodiment of the present disclosure also provides an image processing apparatus 100.

[0187] It can be understood that, in order to implement the above functions, the image processing apparatus 100 provided in the embodiments of the present disclosure includes the corresponding hardware structures and / or software modules for executing each function. Combining the units and algorithm steps of the examples disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solutions of the embodiments of the present disclosure.

[0188] Figure 20 is a block diagram of an image processing apparatus 100 shown according to an exemplary embodiment. Referring to Figure 20 this, the apparatus includes an acquisition unit 101, a conversion unit 102, a completion unit 103, and an inverse conversion unit 104.

[0189] The acquisition unit 101 is configured to acquire the depth feature of the image to be processed.

[0190] The conversion unit 102 is configured to convert the depth feature into a depth feature in the polyhedral lattice space according to a preset depth feature conversion relationship.

[0191] Wherein, the preset depth feature conversion relationship is the corresponding relationship between the depth feature in the image to be processed and the depth feature in the polyhedral lattice space, and the coordinates in the polyhedral lattice space are obtained by projecting the preset dimensional integer space onto a hyperplane with a normal vector of all-one vector.

[0192] The completion unit 103 is configured to perform depth feature completion according to the depth feature in the polyhedral lattice space to obtain the completed depth feature.

[0193] The inverse conversion unit 104 is configured to inversely convert the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship, obtain the image to be processed after the inverse conversion process, and determine the image to be processed obtained by the inverse conversion process as the target image.

[0194] In one implementation, the preset depth feature conversion relationship is determined by the conversion unit 102 in the following manner: Obtain the bilateral spatial coordinates of each pixel in the image to be processed. The sub - coordinates that make up the bilateral spatial coordinates include the spatial sub - coordinates and color sub - coordinates of the pixel. Determine the mapping coordinates of the bilateral spatial coordinates in the polyhedral lattice space, and determine the vertex coordinate group and the weights corresponding to the vertex coordinate group in the polyhedral lattice space according to the mapping coordinates. The vertex coordinate group includes a preset number of vertex coordinates closest to the mapping coordinates. The weights corresponding to the vertex coordinate group include the weights corresponding to each vertex coordinate in the preset number of vertex coordinates. The preset number corresponds to the number of sub - coordinates that make up the bilateral spatial coordinates. The weight corresponding to the vertex coordinate represents the distance between the vertex coordinate and the corresponding mapping coordinate. Determine the corresponding relationship between the bilateral spatial coordinates of each pixel in the image to be processed, the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel, and the weights of the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel as the preset coordinate conversion relationship. Determine the preset depth feature conversion relationship according to the preset coordinate conversion relationship.

[0195] In one implementation, the polyhedral lattice space includes a plurality of regular polygons. The vertex coordinates in the polyhedral lattice space are located at the vertices of the regular polygons. The number of sides of the regular polygon is a preset number. For each of the plurality of regular polygons, the plurality of vertices corresponding to the regular polygon are set with vertex position numbers in a unified rule. The conversion unit 102 determines the preset depth feature conversion relationship according to the preset coordinate conversion relationship in the following manner: Determine all vertex coordinate groups in the polyhedral lattice space that have a corresponding relationship with all bilateral spatial coordinates, sort and number all vertex coordinates included in all vertex coordinate groups, and encode the coordinate values of all vertex coordinates as hash values. Store all vertex coordinate numbers and hash values in the form of key - value pairs. Sort and number all pixels in the image to be processed. For the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel among all pixels, associate the pixel number, vertex coordinate number, and vertex position number with the preset number of vertex coordinates included in the vertex coordinate group, and associate the pixel number, vertex coordinate number, and vertex position number with the weights corresponding to the preset number of vertex coordinates, to obtain the preset depth feature conversion relationship.

[0196] In one implementation, the depth feature of the image to be processed includes the depth value and depth value confidence of each pixel in the image to be processed. The conversion unit 102 converts the depth feature into a depth feature in the polyhedral lattice space according to the preset depth feature conversion relationship in the following manner: for each pixel in the image to be processed, according to the preset depth feature conversion relationship, the depth value of the pixel is converted into a first depth corresponding value in the polyhedral lattice space, and the depth value confidence of the pixel is converted into a confidence corresponding value in the polyhedral lattice space. According to the first depth corresponding value in the polyhedral lattice space and the preset vector conversion method, a first vector corresponding to the pixel depth value is determined, and according to the confidence corresponding value of the depth value and the preset vector conversion method, a second vector corresponding to the pixel depth value confidence is determined. The first vector and the second vector represent the depth feature in the polyhedral lattice space.

[0197] In one implementation, the conversion unit 102 converts the depth value of the pixel into a first depth corresponding value in the polyhedral lattice space according to the preset depth feature conversion relationship in the following manner: the depth value of each pixel in the image to be processed is converted into a first depth corresponding value of the corresponding vertex coordinate group in the polyhedral lattice space. The first depth corresponding value of the vertex coordinate group includes the first depth corresponding value of each vertex coordinate in the vertex coordinate group. The first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate, the pixel depth value, and the pixel depth value confidence. Converting the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space includes: converting the depth value of each pixel in the image to be processed into a confidence corresponding value of the corresponding vertex coordinate group in the polyhedral lattice space. The confidence corresponding value of the vertex coordinate group includes the confidence corresponding value of each vertex coordinate in the vertex coordinate group. The first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate and the pixel depth value confidence.

[0198] In one implementation, the completion unit 103 performs depth feature completion according to the depth feature in the polyhedral lattice space in the following manner to obtain the completed depth feature: traverse all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space, assign corresponding distance weights to the distances between every two vertices among all vertices, and construct a metric matrix. The metric matrix is used to measure the distances between every two vertices among all vertices. According to the metric matrix, the first vector, and the second vector, depth completion is performed to obtain the completed depth feature. The first vector and the second vector represent the depth feature in the polyhedral lattice space.

[0199] In one implementation, the completion unit 103 performs depth completion according to the metric matrix, the first vector, and the second vector in the following manner to obtain the completed depth feature: substitute the metric matrix, the first vector, and the second vector into a preset solution equation to obtain a third vector. The third vector represents the completed depth feature.

[0200] In one implementation, the preset solution equation is solved based on the stabilized bi-conjugate gradient.

[0201] In one implementation, the inverse conversion unit 104 inversely converts the complemented depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship in the following manner: converting the third vector into the second depth corresponding value corresponding to each vertex coordinate in the vertex coordinate set, where the vertex coordinate set is the set of all vertices corresponding to the pixels in the image to be processed in the polyhedral lattice space. According to the preset depth feature conversion relationship, the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space is converted into the target depth value in the image to be processed, and the target depth value represents the target depth feature.

[0202] In one implementation, the inverse conversion unit 104 converts the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed according to the preset depth feature conversion relationship in the following manner: for each pixel in the image to be processed, determining the preset number of vertex coordinates corresponding to the pixel, and determining the second depth corresponding value and weight corresponding to each vertex coordinate among the predetermined number of vertex coordinates. The second depth corresponding values are weighted and summed according to the weights, and the value obtained by the weighted sum is determined as the target depth value.

[0203] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0204] Figure 21 FIG. is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. The apparatus 200 may be provided as a terminal. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0205] Referring to Figure 21 , the apparatus 200 may include one or more of the following components: a processing component 202, a memory 204, a power component 206, a multimedia component 208, an audio component 210, an input / output (I / O) interface 212, a sensor component 214, and a communication component 216.

[0206] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to complete all or part of the steps of the above - mentioned methods. In addition, the processing component 202 may include one or more modules to facilitate the interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0207] The memory 204 is configured to store various types of data to support the operation of the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non - volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read - only memory (EEPROM), erasable programmable read - only memory (EPROM), programmable read - only memory (PROM), read - only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0208] The power component 206 provides power for various components of the device 200. The power component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 200.

[0209] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 208 includes a front - facing camera and / or a rear - facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front - facing camera and / or the rear - facing camera can receive external multimedia data. Each front - facing camera and rear - facing camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0210] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) that is configured to receive external audio signals when the device 200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 further includes a speaker for outputting audio signals.

[0211] The I / O interface 212 provides an interface between the processing component 202 and peripheral interface modules, and the peripheral interface modules may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0212] The sensor component 214 includes one or more sensors for providing an assessment of various aspects of the state of the device 200. For example, the sensor component 214 can detect the on / off state of the device 200, the relative positioning of components, such as the display and keypad of the device 200. The sensor component 214 can also detect a change in the position of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200, and a change in the temperature of the device 200. The sensor component 214 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 214 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 214 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0213] The communication component 216 is configured to facilitate communication between the device 200 and other devices in a wired or wireless manner. The device 200 can access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0214] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0215] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the above instructions can be executed by a processor 220 of the apparatus 200 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0216] It can be understood that "a plurality of" in the present disclosure means two or more, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The singular forms of "a", "the", and "said" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0217] Furthermore, it can be understood that the terms "first", "second", etc. are used to describe various information, but this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and do not represent a specific order or importance. In fact, the expressions such as "first" and "second" can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0218] Furthermore, it can be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "front", "rear", "upper", "lower", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present embodiment and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0219] Furthermore, it can be understood that unless otherwise specified, "connection" includes direct connection without other components between the two, and also includes indirect connection with other elements between the two.

[0220] It can be further understood that although the operations in the embodiments of the present disclosure are described in a specific order in the drawings, it should not be construed as requiring these operations to be performed in the specific order shown or in a serial order, or requiring all the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0221] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of this solution, which follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure.

[0222] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Obtain the depth features of the image to be processed; According to a preset depth feature conversion relationship, convert the depth features into depth features in a polyhedral lattice space. The preset depth feature conversion relationship is the corresponding relationship between the depth features in the image to be processed and the depth features in the polyhedral lattice space. The coordinates in the polyhedral lattice space are obtained by projecting a preset dimensional integer space onto a hyperplane with a normal vector of all-one vector; According to the depth features in the polyhedral lattice space, perform depth feature completion to obtain the completed depth features; According to the preset depth feature conversion relationship, inversely convert the completed depth features in the polyhedral lattice space into the target depth features in the image to be processed, obtain the image to be processed after inverse conversion processing, and determine the image to be processed obtained by inverse conversion processing as the target image.

2. The method according to claim 1, wherein The preset depth feature conversion relationship is determined in the following manner: Obtain the bilateral spatial coordinates of each pixel in the image to be processed. The sub-coordinates constituting the bilateral spatial coordinates include the spatial sub-coordinates and color sub-coordinates of the pixel; Determine the mapping coordinates of the bilateral spatial coordinates mapped to the polyhedral lattice space, and determine the vertex coordinate group and the weights corresponding to the vertex coordinate group in the polyhedral lattice space according to the mapping coordinates. The vertex coordinate group includes a preset number of vertex coordinates closest to the mapping coordinates. The weights corresponding to the vertex coordinate group include the weights corresponding to each vertex coordinate in the preset number of vertex coordinates. The preset number corresponds to the number of sub-coordinates constituting the bilateral spatial coordinates. The weight corresponding to the vertex coordinate represents the distance between the vertex coordinate and the corresponding mapping coordinate; Determine the corresponding relationship between the bilateral spatial coordinates of each pixel in the image to be processed, the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel, and the weights of the vertex coordinate group corresponding to the bilateral spatial coordinates of each pixel as the preset coordinate conversion relationship; Determine the preset depth feature conversion relationship according to the preset coordinate conversion relationship.

3. The method according to any one of claims 1 to 2, characterized in that, The polyhedral lattice space includes a plurality of regular polygons. The vertex coordinates in the polyhedral lattice space are located at the vertices of the regular polygons. The number of sides of the regular polygons is a preset number. For each of the plurality of regular polygons, the plurality of vertices corresponding to the regular polygon are provided with vertex position numbers in a unified rule; Determining the preset depth feature conversion relationship according to the preset coordinate conversion relationship includes: Determine all vertex coordinate groups in the polyhedral lattice space that have a corresponding relationship with all bilateral spatial coordinates according to the preset coordinate conversion relationship, sort and number all vertex coordinates included in all vertex coordinate groups, encode the coordinate values of all vertex coordinates as hash values, and store all vertex coordinate numbers and hash values in the form of key-value pairs; Sort and number all pixels in the image to be processed; For each set of vertex coordinates corresponding to the bilateral spatial coordinates of each pixel among all the pixels, associate the pixel number, the vertex coordinate number, and the vertex position number with a preset number of vertex coordinates included in the set of vertex coordinates, and associate the pixel number, the vertex coordinate number, and the vertex position number with the weights corresponding to the preset number of vertex coordinates, to obtain the preset depth feature conversion relationship.

4. The method according to claim 1, wherein The depth feature of the image to be processed includes the depth value and the depth value confidence of each pixel in the image to be processed; According to the preset depth feature conversion relationship, convert the depth feature into a depth feature in the polyhedral lattice space, including: For each pixel in the image to be processed, according to the preset depth feature conversion relationship, convert the depth value of the pixel into a first depth corresponding value in the polyhedral lattice space, and convert the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space; According to the first depth corresponding value in the polyhedral lattice space and a preset vector conversion method, determine a first vector corresponding to the pixel depth value, and according to the confidence corresponding value of the depth value and the preset vector conversion method, determine a second vector corresponding to the pixel depth value confidence. The first vector and the second vector represent the depth feature in the polyhedral lattice space.

5. The method according to claim 4, wherein The step of converting the depth value of the pixel into a first depth corresponding value in the polyhedral lattice space according to the preset depth feature conversion relationship includes: Convert the depth value of each pixel in the image to be processed into a first depth corresponding value of the corresponding set of vertex coordinates in the polyhedral lattice space. The first depth corresponding value of the set of vertex coordinates includes the first depth corresponding value of each vertex coordinate in the set of vertex coordinates, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate, the pixel depth value, and the pixel depth value confidence; The step of converting the depth value confidence of the pixel into a confidence corresponding value in the polyhedral lattice space includes: Convert the depth value of each pixel in the image to be processed into a confidence corresponding value of the corresponding set of vertex coordinates in the polyhedral lattice space. The confidence corresponding value of the set of vertex coordinates includes the confidence corresponding value of each vertex coordinate in the set of vertex coordinates, and the first depth corresponding value of the vertex coordinate is the product of the weight corresponding to the vertex coordinate and the pixel depth value confidence.

6. The method according to claim 1, characterized in that The step of performing depth feature completion according to the depth feature in the polyhedral lattice space to obtain the completed depth feature includes: Traverse all vertices corresponding to the pixels of the image to be processed in the polyhedral lattice space, assign corresponding distance weights to the distances between every two vertices among all the vertices, and construct a metric matrix, where the metric matrix is used to measure the distances between every two vertices among all the vertices; According to the metric matrix, the first vector, and the second vector, perform depth completion to obtain the completed depth feature, where the first vector and the second vector represent the depth feature in the polyhedral lattice space.

7. The method according to claim 6, characterized in that, The step of performing depth completion according to the metric matrix, the first vector, and the second vector to obtain the completed depth feature includes: Substitute the metric matrix, the first vector, and the second vector into a preset solution equation to obtain a third vector, where the third vector represents the completed depth feature.

8. The method according to claim 7, characterized in that, The preset solution equation is solved based on the stabilized bi-conjugate gradient.

9. The method according to claim 1, wherein The inverse conversion of the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship includes: Convert the third vector into a second depth corresponding value corresponding to each vertex coordinate in the vertex coordinate set, where the vertex coordinate set is the set of all vertices in the polyhedral lattice space corresponding to the pixels of the image to be processed; According to the preset depth feature conversion relationship, convert the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed, where the target depth value represents the target depth feature.

10. The method according to claim 9, characterized in that The conversion of the second depth corresponding value corresponding to the vertex coordinate in the polyhedral lattice space into the target depth value in the image to be processed according to the preset depth feature conversion relationship includes: For each pixel in the image to be processed, determine a preset number of vertex coordinates corresponding to the pixel, and determine the second depth corresponding value and weight corresponding to each vertex coordinate among the predetermined number of vertex coordinates; Weighted sum the second depth corresponding values according to the weights, and determine the value obtained by the weighted sum as the target depth value.

11. An image processing apparatus, characterized in that, including: An acquisition unit for acquiring the depth feature of the image to be processed; A conversion unit for converting the depth feature into a depth feature in a polyhedral lattice space according to a preset depth feature conversion relationship, where the preset depth feature conversion relationship is the correspondence between the depth feature in the image to be processed and the depth feature in the polyhedral lattice space, and the coordinates in the polyhedral lattice space are obtained by projecting a preset-dimensional integer space onto a hyperplane with a normal vector of all-one vector; A completion unit for performing depth feature completion according to the depth feature in the polyhedral lattice space to obtain a completed depth feature; An inverse conversion unit for inversely converting the completed depth feature in the polyhedral lattice space into the target depth feature in the image to be processed according to the preset depth feature conversion relationship to obtain the image to be processed after inverse conversion processing, and determining the image to be processed obtained by inverse conversion processing as the target image.

12. An electronic device, characterized in that, including: A processor: A memory for storing processor-executable instructions; wherein the processor is configured to execute the image processing method according to any one of claims 1 to 10.

13. A storage medium, characterized in that, Instructions are stored in the storage medium, and when the instructions in the storage medium are executed by the processor, the processor is enabled to execute the image processing method according to any one of claims 1 to 10.

Images