Image processing method, apparatus, device, medium, and program product

Abstract

The present disclosure relates to an image processing method, device, equipment, medium and program product. The image processing method comprises: acquiring a color image and a sparse depth image corresponding to the color image; inputting the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image. By acquiring the color image and the sparse depth image corresponding to the color image, and inputting the color image and the sparse depth image into the pre-trained multi-input absolute depth model, an absolute depth image with absolute depth information can be obtained, and the determination of the absolute depth is realized. The color image and the sparse depth image are jointly used as the input of the multi-input absolute depth model, which provides a multi-modal basis for the generation of the absolute depth image, can fully utilize the color and texture details provided by the color image and the partial depth information provided by the sparse depth image, ensures the applicability of the method and the accuracy of the result, and improves the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, specifically to an image processing method, apparatus, device, medium, and program product. Background Technology

[0002] In recent years, absolute depth calculation has become increasingly crucial in the field of computer vision, playing a significant role in 3D scene reconstruction, occlusion and lighting processing in augmented reality, and has been widely applied in electronic devices. An absolute depth image can be generated from a captured image to determine the absolute depth.

[0003] However, the use of related technologies to calculate absolute depth in image processing is subject to many interference factors due to the influence of a single image input, resulting in problems such as insufficient applicability and accuracy of results, and a poor user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, device, medium, and program product.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, the image processing method comprising:

[0006] Obtain a color image and a sparse depth image corresponding to the color image;

[0007] The color image and the sparse depth image are input into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

[0008] In some embodiments of this disclosure, the step of inputting the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image includes:

[0009] The multi-input absolute depth model fuses the color image and the sparse depth image to obtain a first fused image;

[0010] The encoder in the multi-input absolute depth model extracts image features from the first fused image;

[0011] The decoder in the multi-input absolute depth model decodes the image features and outputs the decoding result.

[0012] Based on the decoding results, the absolute depth image is determined.

[0013] In some embodiments of this disclosure, the encoder includes a first branch and a second branch, wherein the first branch is used to extract local features of the first fused image and the second branch is used to extract overall features of the first fused image.

[0014] Before the decoder in the multi-input absolute depth model decodes the image features and outputs the decoding result, the step of inputting the color image and the sparse depth image into the pre-trained multi-input absolute depth model to obtain the absolute depth image further includes:

[0015] The extraction results from the first branch and the second branch are fused together to serve as the input to the decoder.

[0016] In some embodiments of this disclosure, the first branch includes a convolutional neural network branch; and / or,

[0017] The second branch includes a self-attention branch.

[0018] In some embodiments of this disclosure, determining the absolute depth image based on the decoding result includes:

[0019] The decoding result and the sparse depth image are fused to obtain a second fused image;

[0020] The second fused image is depth-corrected by a correction network module to obtain the absolute depth image.

[0021] In some embodiments of this disclosure, the corrected network module includes at least one convolutional layer and at least one dilated convolutional layer, wherein the convolutional layer is used to extract feature information and the dilated convolutional layer is used to adjust the receptive field.

[0022] In some embodiments of this disclosure, there are multiple convolutional layers, with different convolutional layers corresponding to different convolutional kernel sizes. The corrected network module further includes a first fusion layer, which is used to fuse the outputs of each convolutional layer as input to the dilated convolutional layer; and / or,

[0023] The dilated convolutional layers are multiple, and different dilated convolutional layers correspond to different expansion ratios. The correction network module also includes a second fusion layer, which is used to fuse the output results of each dilated convolutional layer.

[0024] In some embodiments of this disclosure, the image processing method further includes:

[0025] Based on the sparse depth image and the absolute depth image, the target correction factor is determined;

[0026] Based on the target correction factor, the absolute depth image is corrected to obtain a corrected image.

[0027] In some embodiments of this disclosure, determining the target correction factor based on the sparse depth image and the absolute depth image includes:

[0028] Divide the depth value of each sparse point in the sparse depth image by the depth value of the corresponding pixel in the absolute depth image to obtain the initial correction factor for each sparse point. The sparse point is a pixel in the sparse depth image with a depth value greater than 0.

[0029] Calculate the mean and standard deviation of each of the initial correction factors;

[0030] For each of the initial correction factors, if the difference between the initial correction factor and the mean is greater than the standard deviation, the initial correction factor is determined as an abnormal correction factor; if the difference between the initial correction factor and the mean is less than or equal to the standard deviation, the initial correction factor is determined as a normal correction factor.

[0031] Calculate the average value of each of the normal correction factors, and use it as the target correction factor.

[0032] In some embodiments of this disclosure, the step of correcting the absolute depth image based on the target correction factor includes:

[0033] For each pixel in the absolute depth image, the depth value of the pixel is multiplied by the target correction factor to obtain the corrected depth value, which is used as the depth value of the pixel in the corrected image.

[0034] In some embodiments of this disclosure, the training process of the multi-input absolute depth model includes:

[0035] Generate training samples, which include color sample images, sparse depth sample images, and absolute depth sample images;

[0036] The color sample image and the corresponding sparse depth sample image are used as inputs, and the absolute depth sample image is used as a label. The initial model is iteratively trained to obtain the multi-input absolute depth model.

[0037] In some embodiments of this disclosure, generating training samples includes:

[0038] A color simulation image and the corresponding absolute depth sample image are generated through simulation rendering.

[0039] The sparse depth sample image is generated based on the absolute depth sample image.

[0040] In some embodiments of this disclosure, the generation of training samples further includes:

[0041] Use the color simulation image as the color sample image; or...

[0042] The color simulation image is input into a pre-trained generative adversarial network model, and the output image is used as the color sample image.

[0043] According to a second aspect of the present disclosure, an image processing apparatus is provided, the image processing apparatus comprising:

[0044] The acquisition module is used to acquire a color image and a sparse depth image corresponding to the color image;

[0045] The generation module is used to input the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

[0046] In some embodiments of this disclosure, the multi-input absolute depth model includes:

[0047] A first fusion unit is configured to fuse the color image and the sparse depth image to obtain a first fused image;

[0048] An encoder, wherein the encoder is used to extract image features from the first fused image;

[0049] A decoder, which is used to decode the image features and output the decoding result.

[0050] In some embodiments of this disclosure, the encoder includes a first branch and a second branch, wherein the first branch is used to extract local features of the first fused image and the second branch is used to extract overall features of the first fused image.

[0051] The multi-input absolute depth model further includes a second fusion unit, which is used to fuse the extraction results of the first branch and the second branch as input to the decoder.

[0052] In some embodiments of this disclosure, the first branch includes a convolutional neural network branch; and / or,

[0053] The second branch includes a self-attention branch.

[0054] In some embodiments of this disclosure, the multi-input absolute depth model further includes:

[0055] The third fusion unit is used to fuse the decoding result and the sparse depth image to obtain a second fused image;

[0056] A correction network module is used to perform depth correction on the second fused image to obtain the absolute depth image.

[0057] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0058] processor;

[0059] Memory used to store processor-executable instructions;

[0060] The processor is configured to perform the image processing method as described in the first aspect.

[0061] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in the first aspect.

[0062] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method as described in the first aspect.

[0063] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: using color images and sparse depth images together as inputs to a multi-input absolute depth model provides a multimodal basis for the generation of absolute depth images, which can make full use of the color and texture details provided by the color images and the partial depth information provided by the sparse depth images, ensuring the applicability of the method and the accuracy of the results, and improving the user experience.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0066] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0067] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, how to input a color image and a sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

[0068] Figure 3 This is a schematic diagram of the network structure of a multi-input absolute depth model according to an exemplary embodiment.

[0069] Figure 4This is a flowchart illustrating, according to an exemplary embodiment, the determination of absolute depth of an image based on decoding results.

[0070] Figure 5 This is a schematic diagram of the network structure of a modified network module according to an exemplary embodiment.

[0071] Figure 6 This is a flowchart illustrating an image processing method according to another exemplary embodiment.

[0072] Figure 7 This is a flowchart illustrating, according to an exemplary embodiment, the determination of a target correction factor based on a sparse depth image and an absolute depth image.

[0073] Figure 8 This is a flowchart illustrating the training process of a multi-input absolute depth model according to an exemplary embodiment.

[0074] Figure 9 This is a flowchart illustrating the generation of training samples according to an exemplary embodiment.

[0075] Figure 10 This is a flowchart illustrating an image processing method according to another exemplary embodiment.

[0076] Figure 11 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0077] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment.

[0078] In the picture:

[0079] 10-First fusion unit; 20-Encoder; 21-First branch; 22-Second branch; 30-Decoder; 40-Second fusion unit; 50-Third fusion unit; 60-Correction network module; 61-Convolutional layer; 62-Dilated convolutional layer; 63-First fusion layer; 64-Second fusion layer; 70-Acquisition module; 80-Generation module; 101-Processing component; 102-Memory; 103-Power component; 104-Multimedia component; 105-Audio component; 106-Input / output interface; 107-Sensor component; 108-Communication component; 109-Processor. Detailed Implementation

[0080] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0081] In recent years, with the rapid development of absolute depth algorithms, absolute depth estimation algorithms and techniques have become increasingly crucial in the field of computer vision. They are of great significance for 3D scene reconstruction, occlusion and lighting processing in augmented reality, and have been widely applied in electronic devices. In related technologies, the acquired RGB image is input into the algorithm model, which then calculates the depth and generates the corresponding depth image.

[0082] However, when using related technologies to calculate absolute depth in image processing, RGB images are greatly affected by factors such as shooting angle or light intensity. Using RGB images as the sole input to the algorithm model leads to many interference factors. Furthermore, due to the limitations of the algorithm model, there are problems such as low algorithm generalization and insufficient accuracy of calculation results, which are not conducive to the functions implemented based on absolute depth and result in a poor user experience.

[0083] Based on this, an exemplary embodiment of this disclosure provides an image processing method that, by acquiring a color image and its corresponding sparse depth image, and inputting the color image and sparse depth image into a pre-trained multi-input absolute depth model, can obtain an absolute depth image with absolute depth information, thus achieving the determination of absolute depth. Using both the color image and the sparse depth image as input to the multi-input absolute depth model provides a multimodal basis for the generation of the absolute depth image, fully utilizing the color and texture details provided by the color image and the partial depth information provided by the sparse depth image, ensuring the applicability of the method and the accuracy of the results, and improving the user experience.

[0084] In one exemplary embodiment, an image processing method is provided, applied to an electronic device, which may include, for example, a mobile phone, camera, tablet computer, or other device capable of having a built-in network model. (Reference) Figure 1 As shown, the image processing methods include:

[0085] S100: Obtain the color image and the sparse depth image corresponding to the color image.

[0086] In step S100, the electronic device acquires a color image and a corresponding sparse depth image. The color image has multiple color channels, and the sparse depth image is a depth image that represents depth using a small number of sampling points; that is, some pixels in the sparse depth image have depth values ​​in the depth channels. For example, the color image is an RGB image. The electronic device can acquire the color image through its camera function and generate the corresponding sparse depth image based on the color image using Time of Flight (TOF) or a dual-camera system.

[0087] S200. Input the color image and sparse depth image into the pre-trained multi-input absolute depth model to obtain the absolute depth image.

[0088] In step S200, the pre-trained multi-input absolute depth model has multiple input terminals and has been iteratively trained based on a specific dataset. It is used to generate corresponding absolute depth images based on various input information or data. An absolute depth image is a depth image that represents the scene depth through all pixels; that is, all pixels in an absolute depth image have depth values ​​in the depth channel. The acquired color image and sparse depth image are input into the pre-trained multi-input absolute depth model, causing the model to output the corresponding absolute depth image, thus realizing the calculation and representation of absolute depth. After obtaining the absolute depth image, it can be used in computer vision for 3D reconstruction, object detection and recognition, and scene analysis; or in augmented reality and virtual reality for placing virtual objects and improving interactive experiences; and also in surveying and mapping for terrain mapping and maintenance.

[0089] It's important to note that using color images and sparse depth images as input to the multi-input absolute depth model (MID) requires a certain number of color sample images and sparse depth sample images as training samples during the DMD's training process. This ensures that the DMD has the ability to output absolute depth images based on the color and sparse depth images. Using color and sparse depth images as the basis for generating absolute depth images provides multimodal information to the DMD, fully utilizing the color and texture details provided by the color images, and the depth values ​​of some pixels provided by the sparse depth images. This guarantees the accuracy of the DMD's output, ensuring the accuracy of the absolute depth and making the method applicable to different shooting scenarios and conditions.

[0090] In this embodiment, by acquiring a color image and its corresponding sparse depth image, and inputting both the color image and the sparse depth image into a pre-trained multi-input absolute depth model, an absolute depth image with absolute depth information can be obtained, thus determining the absolute depth. Using both the color image and the sparse depth image as input to the multi-input absolute depth model provides a multimodal basis for generating the absolute depth image. This fully utilizes the color and texture details provided by the color image and the partial depth information provided by the sparse depth image, ensuring the applicability of the method and the accuracy of the results, and improving the user experience.

[0091] In some embodiments, reference Figure 2 As shown, the color image and sparse depth image are input into a pre-trained multi-input absolute depth model to obtain an absolute depth image, including:

[0092] S210. The multi-input absolute depth model fuses the color image and the sparse depth image to obtain the first fused image.

[0093] In step S210, as Figure 3 As shown, the multi-input absolute depth model can perform image fusion on the color image and the sparse depth image through its first fusion unit 10, so as to fuse the color image with color channels and the sparse depth image with depth channels into a first fused image that simultaneously has color channels and depth channels. Exemplarily, the color image and the sparse depth image can be fused by interpolation, filtering or deep learning.

[0094] S220, The encoder in the multi-input absolute depth model extracts image features from the first fused image.

[0095] In step S220, the encoder 20 extracts features by sequentially performing preprocessing, convolution, downsampling, feature determination, and representation on the input image. For example... Figure 3 As shown, the multi-input absolute depth model can extract features from the first fused image through its encoder 20 to obtain the image features of the first fused image.

[0096] The decoder in the S230 multi-input absolute depth model decodes image features and outputs the decoding result.

[0097] In step S230, the decoder 30 receives the features extracted by the encoder 20 and sequentially performs decoding processes such as convolution, nonlinear activation function, upsampling, depth prediction, and post-processing to output the decoding result. Figure 3As shown, the multi-input absolute depth model can decode the image features extracted by the encoder 20 through its decoder 30 to obtain the decoding result corresponding to the image features. For example, the decoding result can be a preliminarily generated absolute depth image.

[0098] S240. Based on the decoding results, determine the absolute depth image.

[0099] In step S240, the multi-input absolute depth model can generate the final absolute depth image based on the decoding result output by the decoder 30, thereby determining the absolute depth image. For example, when the decoding result is a preliminarily generated absolute depth image, this preliminarily generated absolute depth image can be directly used as the final absolute depth image output by the multi-input absolute depth model. Figure 3 As shown, the initially generated absolute depth image can also be corrected, and the corrected image can be used as the final absolute depth image output by the multi-input absolute depth model.

[0100] In this embodiment, by fusing a color image and a sparse depth image, extracting image features from the first fused image, and then decoding the image features to output a decoding result, the absolute depth image can be determined based on the decoding result, thus realizing the generation of the absolute depth image. The image fusion process can merge multimodal image information, and through encoder 20 and decoder 30, feature extraction and image reconstruction are achieved, ensuring the efficiency and accuracy of the output decoding result, thereby ensuring the generation accuracy of the absolute depth image and improving the user experience.

[0101] In some embodiments, such as Figure 3 As shown, the encoder 20 includes a first branch 21 and a second branch 22. The first branch 21 is used to extract local features of the first fused image, and the second branch 22 is used to extract overall features of the first fused image.

[0102] The encoder 20 of the multi-input absolute depth model includes a first branch 21 and a second branch 22. When extracting image features of the first fused image through the encoder 20, the first fused image is simultaneously input to the first branch 21 and the second branch 22 of the encoder 20 so that the first fused image can be extracted through the first branch 21 and the second branch 22 respectively.

[0103] The first branch 21 is used to extract local features of the first fused image, and the second branch 22 is used to extract the overall features of the first fused image. This enables the encoder 20 to perform multi-branch feature extraction, so as to simultaneously take into account the extraction accuracy of both local and overall features of the image. This provides a more comprehensive and accurate basis for the decoder 30 to output the decoding results, and effectively improves the detail capture and global modeling capabilities of the multi-input absolute depth model.

[0104] Before the decoder 30 in the multi-input absolute depth model decodes the image features and outputs the decoding result, the color image and sparse depth image are input into the pre-trained multi-input absolute depth model to obtain the absolute depth image. This also includes fusing the extraction results of the first branch 21 and the second branch 22 as input to the decoder 30.

[0105] like Figure 3 As shown, when the encoder 20 includes a first branch 21 and a second branch 22, the first branch 21 and the second branch 22 can output different feature extraction results respectively. In order to ensure the decoding efficiency and accuracy of the decoder 30, the extraction results of the first branch 21 and the second branch 22 need to be fused by the second fusion unit 40 of the multi-input absolute depth model, and the fused image features are used as the input of the decoder 30, so that the decoder 30 can decode the image features formed by the fusion of local features and global features to output the decoding result.

[0106] In this embodiment, the encoder 20 is configured to include a first branch 21 and a second branch 22, and the extraction results of the first branch 21 and the second branch 22 are fused to serve as the input of the decoder 30. This provides a decoding object for the decoding process of the decoder 30, enabling the extraction of local and global features of the first fused image through the first branch 21 and the second branch 22 respectively. This approach balances the extraction accuracy of both local and global features, providing a more comprehensive and accurate basis for the decoder 30 to output the decoding results. This effectively enhances the detail capture and global modeling capabilities of the multi-input absolute depth model, giving it a multi-branch characteristic and further ensuring the accuracy of the absolute depth results, thus improving the user experience.

[0107] In some embodiments, the first branch 21 includes a convolutional neural network branch, or the second branch 22 includes a self-attention branch. Alternatively, the first branch 21 may include a convolutional neural network branch and the second branch 22 may include a self-attention branch.

[0108] As mentioned earlier, the first branch 21 is used to extract local features of the first fused image. A Convolutional Neural Network (CNN) branch can be used as the first branch 21. The CNN branch can capture local information in the image through convolution operations, such as recognizing the edges and textures of objects in the image. The second branch 22 is used to extract overall features of the first fused image. A self-attention branch, i.e., a Transformer branch, can be used as the second branch 22. The self-attention mechanism of the Transformer branch processes the relationships between image elements globally, thereby integrating global information in the image.

[0109] The encoder 20, composed of convolutional neural network branches and self-attention branches, can extract image features from different scales. It is sensitive to both local and global features, which improves the quality of feature extraction and meets more complex feature extraction requirements. This further improves the overall performance and generalization ability of the multi-input absolute depth model.

[0110] In this embodiment, the convolutional neural network branch and the self-attention branch are used as the first branch 21 and the second branch 22 of the encoder 20, respectively. The local and global features can be accurately extracted through the convolutional neural network branch and the self-attention branch, respectively. This enables the encoder 20 to have sensitivity to both local and global features, further improving the overall performance and generalization ability of the multi-input absolute depth model and enhancing the user experience.

[0111] In some embodiments, reference Figure 4 As shown, based on the decoding results, the absolute depth image is determined, including:

[0112] S241. The decoding result and the sparse depth image are fused to obtain the second fused image.

[0113] In step S241, as Figure 3 As shown, when determining the absolute depth image based on the decoding result of the decoder 30, the third fusion unit 50 of the multi-input absolute depth model first fuses the decoding result, i.e., the initially generated absolute depth image and the sparse depth image, to fuse the decoding result and the sparse depth image, which each have a single depth channel, into a second fused image with two depth channels. The second fused image is also a depth image with depth information.

[0114] S242. The second fused image is depth-corrected by the correction network module to obtain an absolute depth image.

[0115] In step S242, as Figure 3 As shown, the multi-input absolute depth model also includes a correction network module 60. After obtaining the second fused image, the multi-input absolute depth model performs depth correction on the second fused image through its correction network module 60 to correct the depth information included in the second fused image, making it closer to the real result, and obtaining the final absolute depth image output by the multi-input absolute depth model, thereby further improving the performance of the multi-input absolute depth model and obtaining an absolute depth result with higher accuracy.

[0116] In this embodiment, a second fused image is obtained by fusing the decoding result and the sparse depth image. The second fused image is then depth-corrected by the correction network module 60 to obtain an absolute depth image. This achieves the determination of the absolute depth image and enables the multi-input absolute depth model to have a depth self-correction function, further ensuring the accuracy of the absolute depth result and improving the representation ability of the multi-input absolute depth model.

[0117] In some embodiments, such as Figure 5 As shown, the corrected network module 60 includes at least one convolutional layer 61 and at least one dilated convolutional layer 62. The convolutional layer 61 is used to extract feature information, and the dilated convolutional layer 62 is used to adjust the receptive field.

[0118] The correction network module 60 includes convolutional layers 61 and dilated convolutional layers 62. The number of convolutional layers 61 and dilated convolutional layers 62 can be one or more. The second fused image input to the correction network module 60 first passes through each convolutional layer 61, and the result is input to each dilated convolutional layer 62 to determine the final absolute depth image output by the correction network module 60 based on the output result of the dilated convolutional layer 62.

[0119] Convolutional layer 61 is used to extract feature information by sliding the convolutional kernel across the input data. This information can include details such as edges, corners, and textures, enabling fine-grained classification and understanding of complex features. Dilated convolutional layer 62 increases the effective size of the convolutional kernel by inserting "holes" into it, thereby adjusting the receptive field. This allows it to capture a wider range of input data while maintaining resolution and without increasing computational cost, thus obtaining multi-scale information.

[0120] In this embodiment, a correction network module 60 is formed by convolutional layer 61 and dilated convolutional layer 62. The correction network module 60 can extract feature information and adjust the receptive field through convolutional layer 61 and dilated convolutional layer 62 respectively, so as to achieve depth correction of the second fused image, further ensuring the accuracy of the absolute depth result and improving the representation ability of the multi-input absolute depth model.

[0121] In some embodiments, there are multiple convolutional layers 61, with different convolutional layers 61 corresponding to different convolutional kernel sizes. The corrected network module 60 further includes a first fusion layer 63, which is used to fuse the outputs of each convolutional layer 61 as input to the dilated convolutional layer 62. There are multiple dilated convolutional layers 62, with different dilated convolutional layers 62 corresponding to different expansion ratios. The corrected network module 60 further includes a second fusion layer 64, which is used to fuse the outputs of each dilated convolutional layer 62.

[0122] like Figure 5As shown, multiple convolutional layers 61 are configured, with different kernel sizes corresponding to different convolutional layers 61. This allows for the extraction of feature information at multiple scales using convolutional kernels of different sizes. The outputs of each convolutional layer 61 are then fused using a first fusion layer 63, and the fused result is used as the input to a dilated convolutional layer 62. For example, four convolutional layers 61 are configured, with kernel sizes of 1*1, 3*3, 5*5, and 7*7, respectively. The feature information is a feature image, and the first fusion layer 63 is used to fuse the feature images output by each convolutional layer 61.

[0123] Multiple dilated convolutional layers 62 are configured, each corresponding to a different expansion ratio. This allows for varying degrees of receptive field adjustment and extraction of multi-scale features from different receptive fields. The outputs of each dilated convolutional layer 62 are then fused using a second fusion layer 64 to generate the final absolute depth image output by the corrected network module 60 and the multi-input absolute depth model. For example, four dilated convolutional layers 62 are configured, each with a 3*3 kernel size and expansion ratios of 1, 2, 4, and 8, resulting in receptive fields of 3*3, 5*5, 9*9, and 17*17, respectively. The output of each dilated convolutional layer 62 is a feature image, and the second fusion layer 64 performs image fusion on these feature images.

[0124] In this embodiment, by setting multiple convolutional layers 61 and multiple dilated convolutional layers 62 in the correction network module 60, feature extraction of multiple scales and multiple receptive fields can be achieved through multiple convolutional layers 61 and multiple dilated convolutional layers 62. The extraction results are then fused through the first fusion layer 63 and the second fusion layer 64, thereby realizing depth correction of the second fused image, further ensuring the accuracy of the absolute depth result and improving the representation ability of the multi-input absolute depth model.

[0125] In some embodiments, reference Figure 6 As shown, the image processing method also includes:

[0126] S310. Determine the target correction factor based on the sparse depth image and the absolute depth image.

[0127] In step S310, the sparse depth image and the absolute depth image have partial and complete depth information, respectively. The depth information of the absolute depth image can be corrected based on the partial depth information of the sparse depth image. First, the target correction factor can be determined by an absolute depth correction module independent of the multi-input absolute depth model, based on the sparse depth image and the absolute depth image, which serve as the input and output of the multi-input absolute depth model, respectively.

[0128] S320. Based on the target correction factor, the absolute depth image is corrected to obtain the corrected image.

[0129] In step S320, after determining the target correction factor, the absolute depth image can be corrected according to the target correction factor to obtain the corrected image. This achieves a further correction of the absolute depth result, making the absolute depth represented by the corrected image more accurate than the absolute depth represented by the absolute depth image.

[0130] In this embodiment, a target correction factor is determined based on the sparse depth image and the absolute depth image, and the absolute depth image is corrected according to the target correction factor to obtain a corrected image. This achieves a further correction of the absolute depth result, making the corrected image have higher accuracy and image quality than the absolute depth image, and further improving the accuracy of the absolute depth result.

[0131] In some embodiments, reference Figure 7 As shown, based on the sparse depth image and the absolute depth image, the target correction factor is determined, including:

[0132] S311. Divide the depth value of each sparse point in the sparse depth image by the depth value of the corresponding pixel in the absolute depth image to obtain the initial correction factor for each sparse point. The sparse point is the pixel in the sparse depth image with a depth value greater than 0.

[0133] In step S311, pixels with depth values ​​greater than 0 in the sparse depth image are designated as sparse points, or discrete points, in the sparse depth image. Each discrete point has a corresponding pixel in the absolute depth image, and this pixel also has a corresponding depth value. The depth value of each sparse point in the sparse depth image can be divided by the depth value of its corresponding pixel in the absolute depth image, and the quotient is used as an initial correction factor for each sparse point. This initial correction factor reflects the depth difference between the sparse depth image and the absolute depth image at the same location.

[0134] S312. Calculate the mean and standard deviation of each initial correction factor.

[0135] In step S312, the mean and standard deviation of all initial correction factors are calculated to reflect the average size of the initial correction factors and the degree of difference from the average size, i.e., the dispersion, so as to serve as the basis for subsequent screening of abnormal correction factors and normal correction factors.

[0136] S313. For each initial correction factor, if the difference between the initial correction factor and the mean is greater than the standard deviation, the initial correction factor is determined as an abnormal correction factor; if the difference between the initial correction factor and the mean is less than or equal to the standard deviation, the initial correction factor is determined as a normal correction factor.

[0137] In step S313, for each initial correction factor, the difference is taken from the mean, and the difference is compared with the standard deviation. If the difference is greater than the standard deviation, it indicates that the initial correction factor has a high degree of dispersion, and the initial correction factor is identified as an abnormal correction factor and removed. If the difference is less than or equal to the standard deviation, it indicates that the initial correction factor has a low degree of dispersion, and the initial correction factor is identified as a normal correction factor and retained.

[0138] S314. Calculate the average value of each normal correction factor as the target correction factor.

[0139] In step S314, the average value of all normal correction factors retained after removing abnormal correction factors from the initial correction factor is calculated, and the calculated average value is used as the target correction factor, which provides a basis for the correction of the absolute depth image, improves the consistency between the absolute depth image and the sparse depth image, and makes the absolute depth represented by the corrected image more accurate than the absolute depth represented by the absolute depth image.

[0140] For example, a sparse depth image has N discrete points, where N is a positive integer greater than 1. The depth value of each sparse point in the sparse depth image can be divided by the depth value of the corresponding pixel in the absolute depth image to obtain initial correction factors scale1, scale2, ..., scaleN for each sparse point. The mean (mean1) and standard deviation (std) of the initial correction factors scale1, scale2, ..., scaleN are then calculated. The difference between each initial correction factor scalei and the mean (mean1), scalei-mean1, is then compared with the standard deviation (std) to eliminate abnormal correction factors from the initial correction factors, resulting in the retained normal correction factors scale1', scale2', ..., scaleM', where M is a positive integer less than or equal to N. Finally, the mean (mean2) of each normal correction factor scale1', scale2', ..., scaleM' is calculated as the final target correction factor scale_end.

[0141] In this embodiment, the initial correction factor for each sparse point is obtained by dividing the depth value of each sparse point in the sparse depth image by the depth value of the corresponding pixel in the absolute depth image. The mean and standard deviation of each initial correction factor are then calculated. The normal correction factor is determined by comparing the difference between each initial correction factor and the mean with the standard deviation. The mean of each normal correction factor is then used as the target correction factor, thus determining the target correction factor. This provides a basis for the correction of the absolute depth image, enabling the correction of the absolute depth image to improve its consistency with the sparse depth image, thereby further improving the accuracy of the absolute depth results.

[0142] In some embodiments, the absolute depth image is corrected based on the target correction factor, including: for each pixel in the absolute depth image, multiplying the depth value of the pixel by the target correction factor to obtain a corrected depth value, which is used as the depth value of the pixel in the corrected image.

[0143] When correcting an absolute depth image based on a target correction factor, for each pixel in the absolute depth image, the depth value of that pixel is multiplied by the target correction factor to obtain the corresponding corrected depth value, thus achieving depth correction for that pixel. The pixels with corrected depth values ​​together form the corrected image.

[0144] In this embodiment, the corrected depth value is obtained by multiplying the depth value of each pixel by the target correction factor, and then used as the depth value in the corrected image. This achieves the correction of the absolute depth image based on the target correction factor, making the corrected depth value more similar to the depth value in the sparse depth image, improving the consistency with the sparse depth image, and thus further improving the accuracy of the absolute depth result.

[0145] In some embodiments, reference Figure 8 As shown, the training process of a multi-input absolute depth model includes:

[0146] S410. Generate training samples, which include color sample images, sparse depth sample images, and absolute depth sample images.

[0147] In step S410, training samples required for training the multi-input absolute depth model are generated. The training samples include color sample images that meet a certain number of requirements, as well as sparse depth sample images and absolute depth sample images corresponding to each color sample image, so as to ensure that the pre-trained multi-input absolute depth model can output the corresponding absolute depth image based on the color image and sparse depth image.

[0148] S420. Using color sample images and corresponding sparse depth sample images as inputs, and absolute depth sample images as labels, iteratively train the initial model to obtain a multi-input absolute depth model.

[0149] In step S420, the color sample image and the corresponding sparse depth sample image in the training samples are used as the input of the initial model, and the absolute depth sample image is used as the training label of the initial model. The initial model is iteratively trained to obtain a multi-input absolute depth model, so that the multi-input absolute depth model can output an absolute depth image with the required accuracy.

[0150] For example, color sample images and corresponding sparse depth sample images can be input into the initial model, and the difference between the model output and the training label, i.e., the absolute depth sample image, can be calculated as the loss function. The model parameters of the initial model can be changed by backpropagating the loss function so that the model output gradually tends to the absolute depth sample image. When the number of iterations reaches a preset number or the output meets the preset consistency condition, a multi-input absolute depth model can be obtained.

[0151] In this embodiment, training samples are generated, and color sample images and corresponding sparse depth sample images are used as inputs, while absolute depth sample images are used as labels. The initial model is iteratively trained to obtain a multi-input absolute depth model, which realizes the pre-training of the multi-input absolute depth model. This enables the multi-input absolute depth model to output an absolute depth image with the required accuracy based on the color image and sparse depth image, providing a basis for the generation of absolute depth images.

[0152] In some embodiments, reference Figure 9 As shown, training samples are generated, including:

[0153] S411. Generate a color simulation image and an absolute depth sample image corresponding to the color simulation image through simulation rendering.

[0154] In step S411, a color simulation image and a corresponding absolute depth sample image can be generated using a simulation rendering engine to achieve the acquisition of the absolute depth sample image. For example, a virtual environment can be created in the Unity simulation rendering engine, and objects of specific materials or types can be added to the scene to generate a color simulation image, i.e., a simulated RGB image, ensuring the realism of the color simulation image. Then, the depth value corresponding to each pixel is obtained through the depth buffer of the camera component of the Unity simulation rendering engine to obtain the corresponding absolute depth sample image.

[0155] S412. Generate sparse depth sample images based on absolute depth sample images.

[0156] In step S412, a sparse depth sample image can be generated based on the absolute depth sample image to obtain the sparse depth sample image. For example, discrete points of 1 / 10000 to 1 / 000 of the corresponding number of pixels in the absolute depth sample image can be randomly collected to form the corresponding sparse depth sample image.

[0157] In this embodiment, a color simulation image and a corresponding absolute depth sample image are generated through simulation rendering. Based on the absolute depth sample image, a sparse depth sample image is generated, thereby realizing the acquisition of absolute depth sample images and sparse depth sample images. This provides high-quality simulation images as training samples for the multi-input absolute depth model, avoiding the impact of sample data quality on the accuracy of the multi-input absolute depth model, further ensuring the accuracy of the absolute depth results, and improving the user experience.

[0158] In some embodiments, generating training samples further includes: using a color simulation image as a color sample image, or inputting a color simulation image into a pre-trained generative adversarial network model and using the output image as a color sample image.

[0159] When generating training samples, the color simulation image generated by simulation rendering can be directly used as the color sample image, thus realizing the acquisition of color sample images, so that the color sample image, absolute depth sample image and sparse depth sample image can be used as complete training samples.

[0160] The color simulation image can also be input into a pre-trained Generative Adversarial Network (GAN) model to further improve the realism of the color simulation image through the GAN model, and the output image is used as a color sample image, thus realizing the acquisition of color sample images, and using color sample images, absolute depth sample images and sparse depth sample images as complete training samples.

[0161] In this embodiment, a color simulation image is used as a color sample image, or the color simulation image is input into a pre-trained generative adversarial network model, and the output image is used as a color sample image. This achieves the acquisition of color sample images, providing complete and high-quality training samples for the multi-input absolute depth model. By inputting the color simulation image into the pre-trained generative adversarial network model, the realism of the color simulation image can be improved, ensuring that the color sample image can meet the realism requirements of the training samples, further improving the accuracy of the multi-input absolute depth model, ensuring the accuracy of the absolute depth results, and enhancing the user experience.

[0162] In one exemplary embodiment, an image processing method is provided, applied to an electronic device, with reference to... Figure 10 As shown, the image processing methods include:

[0163] S1. Generate a color simulation image and an absolute depth sample image corresponding to the color simulation image through simulation rendering;

[0164] S2. Input the color simulation image into the pre-trained generative adversarial network model, and use the output image as the color sample image;

[0165] S3. Generate sparse depth sample images based on absolute depth sample images;

[0166] S4. Using the color sample image and the corresponding sparse depth sample image as input, and the absolute depth sample image as the label, iteratively train the initial model to obtain the multi-input absolute depth model.

[0167] S5. Obtain the color image and the corresponding sparse depth image;

[0168] S6. The multi-input absolute depth model fuses the color image and the sparse depth image to obtain the first fused image;

[0169] S7. The first and second branches of the encoder in the multi-input absolute depth model extract the image features of the first fused image, respectively.

[0170] S8. The extraction results of the first branch and the second branch are fused together and used as the input of the decoder;

[0171] S9. The decoder in the multi-input absolute depth model decodes image features and outputs the decoding result;

[0172] S10. The decoding result and the sparse depth image are fused to obtain the second fused image;

[0173] S11. The second fused image is depth-corrected by the correction network module to obtain an absolute depth image;

[0174] S12. Determine the target correction factor based on the sparse depth image and the absolute depth image;

[0175] S13. Based on the target correction factor, the absolute depth image is corrected to obtain the corrected image.

[0176] In this embodiment, by acquiring a color image and its corresponding sparse depth image, and inputting both the color image and the sparse depth image into a pre-trained multi-input absolute depth model, an absolute depth image with absolute depth information can be obtained, thus determining the absolute depth. Using both the color image and the sparse depth image as input to the multi-input absolute depth model provides a multimodal basis for generating the absolute depth image. This fully utilizes the color and texture details provided by the color image and the partial depth information provided by the sparse depth image, ensuring the applicability of the method and the accuracy of the results. Furthermore, through multi-branch settings, multi-scale correction, and correction based on a target correction factor in the multi-input absolute depth model, the accuracy of the absolute depth results is further improved, enhancing the user experience.

[0177] In one exemplary embodiment, an image processing apparatus is provided, with reference to Figure 11 As shown, the image processing device includes an acquisition module 70 and a generation module 80. The acquisition module 70 is used to acquire a color image and a corresponding sparse depth image. The generation module 80 is used to input the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

[0178] In this embodiment, the acquisition module 70 acquires a color image and its corresponding sparse depth image, and the generation module 80 inputs the color image and sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image with absolute depth information, thus determining the absolute depth. Using both the color image and the sparse depth image as input to the multi-input absolute depth model provides a multimodal basis for generating the absolute depth image. This fully utilizes the color and texture details provided by the color image and the partial depth information provided by the sparse depth image, ensuring the applicability of the method and the accuracy of the results, and improving the user experience.

[0179] In one embodiment, the multi-input absolute depth model includes a first fusion unit 10, an encoder 20, and a decoder 30. The first fusion unit 10 fuses a color image and a sparse depth image to obtain a first fused image. The encoder 20 extracts image features from the first fused image. The decoder 30 decodes the image features and outputs a decoding result.

[0180] In one embodiment, the encoder 20 includes a first branch 21 and a second branch 22. The first branch 21 is used to extract local features of the first fused image, and the second branch 22 is used to extract global features of the first fused image. The multi-input absolute depth model also includes a second fusion unit 40, which is used to fuse the extraction results of the first branch 21 and the second branch 22 as input to the decoder 30.

[0181] In one embodiment, the first branch 21 includes a convolutional neural network branch; and / or, the second branch 22 includes a self-attention branch.

[0182] In one embodiment, the multi-input absolute depth model further includes a third fusion unit 50 and a correction network module 60. The third fusion unit 50 is used to fuse the decoded result and the sparse depth image to obtain a second fused image. The correction network module 60 is used to perform depth correction on the second fused image to obtain an absolute depth image.

[0183] In one embodiment, the correction network module 60 includes at least one convolutional layer 61 and at least one dilated convolutional layer 62, wherein the convolutional layer 61 is used to extract feature information and the dilated convolutional layer 62 is used to adjust the receptive field.

[0184] In one embodiment, there are multiple convolutional layers 61, with different convolutional layers 61 corresponding to different convolutional kernel sizes. The correction network module 60 also includes a first fusion layer 63, which is used to fuse the output results of each convolutional layer 61 as the input of the dilated convolutional layer 62; and / or, there are multiple dilated convolutional layers 62, with different dilated convolutional layers 62 corresponding to different expansion ratios. The correction network module 60 also includes a second fusion layer 64, which is used to fuse the output results of each dilated convolutional layer 62.

[0185] In one embodiment, the generation module 80 is further configured to: determine a target correction factor based on the sparse depth image and the absolute depth image; and correct the absolute depth image based on the target correction factor to obtain a corrected image.

[0186] In one embodiment, the generation module 80 is further configured to: divide the depth value of each sparse point in the sparse depth image by the depth value of the corresponding pixel in the absolute depth image to obtain an initial correction factor for each sparse point, wherein the sparse point is a pixel in the sparse depth image with a depth value greater than 0; calculate the mean and standard deviation of each initial correction factor; for each initial correction factor, if the difference between the initial correction factor and the mean is greater than the standard deviation, the initial correction factor is determined as an abnormal correction factor; if the difference between the initial correction factor and the mean is less than or equal to the standard deviation, the initial correction factor is determined as a normal correction factor; calculate the mean of each normal correction factor as the target correction factor.

[0187] In one embodiment, the generation module 80 is further configured to: for each pixel in the absolute depth image, multiply the depth value of the pixel by the target correction factor to obtain a corrected depth value, which is used as the depth value of the pixel in the corrected image.

[0188] In one embodiment, the image processing apparatus further includes a training module, which is used to: generate training samples, including color sample images, sparse depth sample images, and absolute depth sample images; and iteratively train an initial model using the color sample images and the corresponding sparse depth sample images as inputs and the absolute depth sample images as labels to obtain a multi-input absolute depth model.

[0189] In one embodiment, the training module is further configured to: generate a color simulation image and an absolute depth sample image corresponding to the color simulation image through simulation rendering; and generate a sparse depth sample image based on the absolute depth sample image.

[0190] In one embodiment, the training module is further configured to: use the color simulation image as a color sample image; or, input the color simulation image into a pre-trained generative adversarial network model and use the output image as a color sample image.

[0191] In one exemplary embodiment, an electronic device is provided, which may include, for example, a mobile phone, a camera, a tablet computer, or other devices capable of having a built-in network model.

[0192] refer to Figure 12 As shown, the electronic device may include one or more of the following components: processing component 101, memory 102, power component 103, multimedia component 104, audio component 105, input / output (I / O) interface 106, sensor component 107, and communication component 108.

[0193] Processing component 101 typically controls the overall operation of an electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 101 may include one or more processors 109 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 101 may include one or more modules to facilitate interaction between processing component 101 and other components. For example, processing component 101 may include a multimedia module to facilitate interaction between multimedia component 104 and processing component 101.

[0194] Memory 102 is configured to store various types of data to support the operation of the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0195] Power component 103 provides power to various components of the electronic device. Power component 103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0196] Multimedia component 104 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 104 includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0197] Audio component 105 is configured to output and / or input audio signals. For example, audio component 105 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 102 or transmitted via communication component 108. In some embodiments, audio component 105 also includes a speaker for outputting audio signals.

[0198] I / O interface 106 provides an interface between processing component 101 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0199] Sensor assembly 107 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 107 can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 107 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 107 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 107 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0200] Communication component 108 is configured to facilitate wired or wireless communication between electronic devices and other devices. Devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 108 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 108 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0201] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method applied to the electronic device described above.

[0202] In one exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 102 including instructions, which can be executed by a processor 109 of an electronic device to perform the image processing method applied to the electronic device described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the storage medium are executed by the processor 109 of the electronic device, the electronic device is able to perform the image processing method shown in the above embodiments.

[0203] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by processor 109, implements the image processing method shown in the above embodiments.

[0204] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0205] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The image processing method includes: Obtain a color image and a sparse depth image corresponding to the color image; The color image and the sparse depth image are input into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

2. The image processing method according to claim 1, characterized in that, The step of inputting the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image includes: The multi-input absolute depth model fuses the color image and the sparse depth image to obtain a first fused image; The encoder in the multi-input absolute depth model extracts image features from the first fused image; The decoder in the multi-input absolute depth model decodes the image features and outputs the decoding result. Based on the decoding results, the absolute depth image is determined.

3. The image processing method according to claim 2, characterized in that, The encoder includes a first branch and a second branch. The first branch is used to extract local features of the first fused image, and the second branch is used to extract overall features of the first fused image. Before the decoder in the multi-input absolute depth model decodes the image features and outputs the decoding result, the step of inputting the color image and the sparse depth image into the pre-trained multi-input absolute depth model to obtain the absolute depth image further includes: The extraction results from the first branch and the second branch are fused together to serve as the input to the decoder.

4. The image processing method according to claim 3, characterized in that, The first branch includes a convolutional neural network branch; and / or, The second branch includes a self-attention branch.

5. The image processing method according to claim 2, characterized in that, Determining the absolute depth image based on the decoding result includes: The decoding result and the sparse depth image are fused to obtain a second fused image; The second fused image is depth-corrected by a correction network module to obtain the absolute depth image.

6. The image processing method according to claim 5, characterized in that, The corrected network module includes at least one convolutional layer and at least one dilated convolutional layer. The convolutional layer is used to extract feature information, and the dilated convolutional layer is used to adjust the receptive field.

7. The image processing method according to claim 6, characterized in that, The convolutional layers are multiple, with different kernel sizes corresponding to different convolutional layers. The corrected network module further includes a first fusion layer, which is used to fuse the outputs of each convolutional layer as the input to the dilated convolutional layer; and / or, The dilated convolutional layers are multiple, and different dilated convolutional layers correspond to different expansion ratios. The correction network module also includes a second fusion layer, which is used to fuse the output results of each dilated convolutional layer.

8. The image processing method according to any one of claims 1 to 7, characterized in that, The image processing method further includes: Based on the sparse depth image and the absolute depth image, the target correction factor is determined; Based on the target correction factor, the absolute depth image is corrected to obtain a corrected image.

9. The image processing method according to claim 8, characterized in that, The step of determining the target correction factor based on the sparse depth image and the absolute depth image includes: Divide the depth value of each sparse point in the sparse depth image by the depth value of the corresponding pixel in the absolute depth image to obtain the initial correction factor for each sparse point. The sparse point is a pixel in the sparse depth image with a depth value greater than 0. Calculate the mean and standard deviation of each of the initial correction factors; For each of the initial correction factors, if the difference between the initial correction factor and the mean is greater than the standard deviation, the initial correction factor is determined as an abnormal correction factor; if the difference between the initial correction factor and the mean is less than or equal to the standard deviation, the initial correction factor is determined as a normal correction factor. Calculate the average value of each of the normal correction factors, and use it as the target correction factor.

10. The image processing method according to claim 8, characterized in that, The step of correcting the absolute depth image based on the target correction factor includes: For each pixel in the absolute depth image, the depth value of the pixel is multiplied by the target correction factor to obtain the corrected depth value, which is used as the depth value of the pixel in the corrected image.

11. The image processing method according to any one of claims 1 to 7, characterized in that, The training process of the multi-input absolute depth model includes: Generate training samples, which include color sample images, sparse depth sample images, and absolute depth sample images; The color sample image and the corresponding sparse depth sample image are used as inputs, and the absolute depth sample image is used as a label. The initial model is iteratively trained to obtain the multi-input absolute depth model.

12. The image processing method according to claim 11, characterized in that, The generation of training samples includes: A color simulation image and the corresponding absolute depth sample image are generated through simulation rendering. The sparse depth sample image is generated based on the absolute depth sample image.

13. The image processing method according to claim 12, characterized in that, The generation of training samples also includes: Use the color simulation image as the color sample image; or... The color simulation image is input into a pre-trained generative adversarial network model, and the output image is used as the color sample image.

14. An image processing apparatus, characterized in that, The image processing device includes: The acquisition module is used to acquire a color image and a sparse depth image corresponding to the color image; The generation module is used to input the color image and the sparse depth image into a pre-trained multi-input absolute depth model to obtain an absolute depth image.

15. The image processing apparatus according to claim 14, characterized in that, The multi-input absolute depth model includes: A first fusion unit is configured to fuse the color image and the sparse depth image to obtain a first fused image; An encoder, wherein the encoder is used to extract image features from the first fused image; A decoder, which is used to decode the image features and output the decoding result.

16. The image processing apparatus according to claim 15, characterized in that, The encoder includes a first branch and a second branch. The first branch is used to extract local features of the first fused image, and the second branch is used to extract overall features of the first fused image. The multi-input absolute depth model further includes a second fusion unit, which is used to fuse the extraction results of the first branch and the second branch as input to the decoder.

17. The image processing apparatus according to claim 16, characterized in that, The first branch includes a convolutional neural network branch; and / or, The second branch includes a self-attention branch.

18. The image processing apparatus according to claim 15, characterized in that, The multi-input absolute depth model also includes: The third fusion unit is used to fuse the decoding result and the sparse depth image to obtain a second fused image; A correction network module is used to perform depth correction on the second fused image to obtain the absolute depth image.

19. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image processing method as described in any one of claims 1 to 13.

20. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1 to 13.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 13.

Images