Method for training a depth optimization model and method for optimizing a depth estimation image

Abstract

The application provides a training method of a depth optimization model and an optimization method of a depth estimation image, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a sample monocular RGB image, a corresponding sample depth estimation image and a sample matting image of the sample monocular RGB image, and the sample monocular RGB image has a depth optimization label image; inputting the sample monocular RGB image, the sample depth estimation image and the sample matting image into a depth optimization model to output a main body depth prediction image and a depth optimization prediction image; training the depth optimization model based on the main body depth prediction image, the sample matting image, the depth optimization label image, the depth optimization prediction image and a total target loss function to obtain a trained depth optimization model. The application provides a training method of a depth optimization model, and the depth optimization image optimized by the depth optimization model trained by the method has high precision.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for training a depth optimization model and an optimization method for depth estimation images. Background Technology

[0002] Monocular depth estimation algorithms are algorithms that estimate the depth value of each pixel in a given monocular RGB image. Depth-estimated images generated using monocular depth estimation algorithms can be applied in fields such as autonomous driving, augmented reality and virtual reality, and robot navigation. However, due to limitations of the sensor itself, the accuracy of depth-estimated images generated using monocular depth estimation algorithms is not high. To improve the accuracy of depth-estimated images, optimization is necessary.

[0003] Currently, methods such as Conditional Random Fields (CRF), Operator-based methods, and optimization-based methods are mainly used to optimize depth estimation images. However, these methods have certain limitations. For example, CRF methods require multiple iterations, Operator-based methods are prone to introducing double boundaries and generating new errors, and optimization-based methods require a lot of auxiliary information. As a result, the accuracy of depth-optimized images is still not high and cannot meet the requirements of downstream tasks.

[0004] With the development of artificial intelligence technology, models trained using deep learning methods can be applied to various scenarios to complete tasks in different situations. Therefore, how to train a deep optimization model using deep learning methods has become the key to improving the accuracy of deep-optimized images. Summary of the Invention

[0005] This application provides a method for training a depth optimization model and optimizing a depth estimation image. It enables the training of a depth optimization model and the optimization of the depth estimation image based on the trained model, resulting in a high-accuracy depth-optimized image. The technical solution is as follows:

[0006] Firstly, a method for training a deep optimization model is provided, the method comprising:

[0007] Acquire a sample monocular RGB image, a sample depth estimation image, and a sample matted image. The sample depth estimation image is an image obtained by performing depth estimation on the sample monocular RGB image. The sample matted image is an image obtained by matting out the foreground subject in the sample monocular RGB image. The sample monocular RGB image has a depth label image.

[0008] The sample monocular RGB image, the sample depth estimation image, and the sample matted image are input into the depth optimization model, and the main body depth prediction image and the depth prediction image are output.

[0009] Based on the subject depth prediction image, the sample matted image, the depth label image, the depth prediction image, and the overall objective loss function, the depth optimization model is trained to obtain a trained depth optimization model. The trained depth optimization model is used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

[0010] Secondly, an optimization method for depth estimation images is provided, the method applying the trained depth optimization model trained in the first aspect, the method comprising:

[0011] Acquire a monocular RGB image and its corresponding depth estimation image and matted image;

[0012] The monocular RGB image, the depth estimation image, and the matted image are input into the trained depth optimization model, and the depth optimization image is output.

[0013] Thirdly, a training device for a deep optimization model is provided, the device comprising:

[0014] The acquisition module is used to acquire a sample monocular RGB image, a sample depth estimation image, and a sample matted image. The sample depth estimation image is an image obtained by performing depth estimation on the sample monocular RGB image. The sample matted image is an image obtained by matting out the foreground subject in the sample monocular RGB image. The sample monocular RGB image has a depth label image.

[0015] The input / output module is used to input the sample monocular RGB image, the sample depth estimation image, and the sample matting image into the depth optimization model, and output the subject depth prediction image and the depth prediction image.

[0016] The training module is used to train the depth optimization model based on the subject depth prediction image, the sample matted image, the depth label image, the depth prediction image and the total objective loss function to obtain a trained depth optimization model. The trained depth optimization model is used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

[0017] Fourthly, an optimization apparatus for depth estimation images is provided, wherein the trained depth optimization model trained in the first aspect is installed within the apparatus, and the apparatus includes:

[0018] The acquisition module is used to acquire a monocular RGB image and its corresponding depth estimation image and matted image;

[0019] The input / output module is used to input the monocular RGB image, the depth estimation image, and the matted image into the trained depth optimization model and output the depth optimization image.

[0020] Fifthly, an electronic device is provided, including a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is called and executed by the processor to implement the training method of the depth optimization model as described in the first aspect, or the optimization method of the depth estimation image as described in the second aspect.

[0021] In a sixth aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, and when executed by a processor, the at least one computer program is capable of implementing the training method for the depth optimization model as described in the first aspect, or the optimization method for the depth estimation image as described in the second aspect.

[0022] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, can implement the training method for the depth optimization model as described in the first aspect, or the optimization method for the depth estimation image as described in the second aspect.

[0023] The beneficial effects of the technical solutions provided in this application are:

[0024] This application provides a training method for a depth optimization model. The method acquires sample monocular RGB images and their corresponding sample depth estimation images and sample matted images. These images are then input into the depth optimization model for training. Since the training samples include multiple images, including the original image, the model can acquire rich image features. Simultaneously, the auxiliary sample matted images enable the model to selectively learn image features corresponding to the foreground subject and the original image. This results in the output of a subject depth prediction image corresponding to the foreground subject and a depth prediction image corresponding to the original image. Then, a supervised learning method is used to continuously refine the model parameters based on the subject depth prediction image, the depth prediction image, the sample matted image, the depth label image, and the overall objective loss function, ultimately obtaining a well-trained depth optimization model with the expected accuracy. When this trained depth optimization model is used to optimize depth estimation images, it can output highly accurate depth-optimized images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the network structure of a deep optimization model provided by related technologies;

[0027] Figure 2 This is a schematic diagram illustrating the specific structure of the Transformer module and the fusion module in related technologies;

[0028] Figure 3 This is a schematic diagram illustrating the two training stages of training a deep optimization model using related technologies;

[0029] Figure 4 It is a depth-optimized image obtained by a depth optimization model trained using relevant technologies;

[0030] Figure 5 This is a schematic diagram of the implementation environment involved in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram of the network structure of a deep optimization model provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of a fusion upsampling module provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a low-scale information enhancement module provided in an embodiment of this application;

[0034] Figure 9 This is a schematic diagram of the structure of a subject depth prediction module provided in an embodiment of this application;

[0035] Figure 10 This is a schematic diagram of the structure of a main body depth enhancement module provided in an embodiment of this application;

[0036] Figure 11 This is a schematic diagram of the structure of a depth prediction module provided in an embodiment of this application;

[0037] Figure 12 This is a flowchart of a training method for a deep optimization model provided in an embodiment of this application;

[0038] Figure 13 This is a flowchart of an optimization method for depth estimation images provided in an embodiment of this application;

[0039] Figure 14 It is an image of the deep optimization model trained in the embodiments of this application;

[0040] Figure 15 It is the deep optimization model trained using the embodiments of this application that... Figure 14 The depth-optimized image obtained by processing the image shown;

[0041] Figure 16 This is a schematic diagram of a training device structure for a deep optimization model provided in an embodiment of this application;

[0042] Figure 17 This is a schematic diagram of the structure of an optimization device for depth estimation images provided in an embodiment of this application;

[0043] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] It is understood that the terms "each," "multiple," and "any" used in the embodiments of this application, etc., mean that "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the corresponding multiples. For example, multiple words include 10 words, and "each word" refers to each of the 10 words, while "any word" refers to any one of the 10 words.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0047] Monocular depth estimation algorithms generally refer to algorithms that estimate the depth value of each pixel in a given monocular RGB image. Due to limitations of the sensor itself, monocular RGB images lack sufficient sensor type information or viewpoint information, leading to persistently erroneous estimations in these algorithms. Consequently, the accuracy of depth estimation images generated using monocular depth estimation algorithms is not high enough.

[0048] To improve the accuracy of depth estimation images and achieve a better experience in downstream tasks, current methods mainly employ random field methods, operator methods, and optimization-based methods. However, these methods all have certain limitations. Specifically, random field methods require multiple iterations, resulting in high time consumption and generally poor optimization performance; operator methods are prone to introducing double boundaries, leading to new errors in depth estimation; optimization-based methods require additional auxiliary information, and their effectiveness is directly proportional to the amount of auxiliary information. If only basic and easily estimated information such as image matting is available, the optimization effect is poor. To achieve better optimization results, less common auxiliary information such as surface orientation and heatmaps is needed, and estimating this information is no less difficult than depth estimation itself.

[0049] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is being researched and applied in many fields, such as smart homes, wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. With the research and advancement of AI technology, deep learning-based optimization methods have emerged. Models trained using deep learning methods require very little auxiliary information, such as image matting, to achieve good optimization results.

[0050] Figure 1 This diagram illustrates the network structure of a deep optimization model trained on image matting, related to the technology. (See attached image) Figure 1 The network takes as input the depth estimation image P(D') to be optimized, the monocular RGB image I' corresponding to the depth estimation image P(D'), and the foreground subject matting image M corresponding to the monocular RGB image I'. The output is the depth-optimized image. The network consists of a T, a backbone network, and a low-level encoder. The backbone network includes multiple Ts and multiple Fs. Here, T represents a Transformer module (the low-level encoder can also be a Transformer module), F represents a fusion module, the Head can consist of multiple convolutional layers, and + indicates a connection. Figure 2 The specific structures of the Transformer module and the fusion module are shown, in which Figure 2 The left image shows the specific structure of the Transformer module. Figure 2The right figure shows the specific structure of the fusion module. See also... Figure 2 The left image shows that the Transformer module can consist of residual convolutional layers, resampling layers, and mapping layers, etc. (See also...) Figure 2 As shown in the right figure, the fusion module can be composed of convolutional layers, resampling layers, and convolutional and activation function layers. The model has three inputs: one is the input of the individual Transformer module included in the network, another is the input of the backbone network included in the network, and the third is the input of the low-level encoder included in the network.

[0051] Figure 1 The depth optimization model shown employs a self-supervised learning method, requiring no additional labels for training. Training requires a depth estimation dataset, which includes a monocular RGB image I and its corresponding depth estimation image D, as well as a foreground subject matted image M (M may or may not correspond to the monocular RGB image I). See [link to documentation]. Figure 3 The training process can be divided into two stages.

[0052] The first phase of training is as follows:

[0053] The first step is to acquire a monocular RGB image I and its corresponding depth estimation image D. Then, from the monocular RGB image I, foreground image I1 and background image I2 are obtained. Based on the foreground image I1, a foreground depth estimation image D1 is obtained from the depth estimation image D, and based on the background image I2, a background depth estimation image D2 is obtained from the depth estimation image D. Next, a foreground subject cutout image M is randomly acquired, and a new monocular RGB image I is synthesized using the following formula. ’ and the new depth estimation image D ’ :

[0054]

[0055]

[0056] The second step is to use the foreground depth estimation image D1 as the supervision label, and then... ’ I ’ And in the input model M (Rm shown in the figure), the depth-optimized image corresponding to the main part of the foreground subject matting image M output by the model. Then optimize the image based on foreground depth. The model is trained using the foreground depth estimation image D1.

[0057] The third step is to obtain the inverse code of the foreground subject matting image M, which is the background subject matting image 1-M. Using the background depth estimation image D2 as the supervision label, D... ’ I ’And in the 1-M input model (Rm shown in the figure), the background depth optimized image corresponding to the main body of the background subject matting image 1-M output by the model. Then optimize the image based on background depth. The model is trained using the background depth estimation image D2.

[0058] After the first stage of training, the model is able to recover a depth-optimized image of the corresponding part of the subject based on the subject-masked image.

[0059] The second phase of training is as follows:

[0060] The first step is to acquire a monocular RGB image I and its corresponding depth estimation image D. Then, from the monocular RGB image I, obtain the foreground image I1 and the background image I2. Based on the foreground image I1, obtain the foreground depth estimation image D1 from the depth estimation image D, and based on the background image I2, obtain the background depth estimation image D2 from the depth estimation image D. Next, randomly acquire a foreground subject cutout image M, and synthesize a new monocular RGB image I using the following formula. ’ and the new depth estimation image D ’ :

[0061]

[0062]

[0063] The second step involves artificially adding random holes, random dilation or erosion, and random filtering to the new depth estimation image D. ’ Perform data augmentation to obtain P(D').

[0064] The third step is to use the foreground depth estimation image D1 as the supervision label, and to assign P(D') and I... ’ And M is input into the model trained in the first stage (Rm shown in the figure), and the depth-optimized image corresponding to the main part of the foreground subject matting image M is output by the model. And using the background depth estimated image D2 as the supervision label, P(D') and I ’ And in the model trained in the first stage, the 1-M inputs are the background depth optimized images corresponding to the main body parts of the background-machining images 1-M output by the model. Then, based on the foreground subject matted image M and its corresponding depth-optimized image... Background and subject cutout images 1-M and their corresponding depth-optimized images The following formula is used to merge the entire depth-optimized image. :

[0065]

[0066] Next, the new depth estimation image D ’ For supervision of labels, based on D ’ and The model trained in the first stage is then retrained to obtain a well-trained deep optimization model.

[0067] Based on the trained depth optimization model, foreground depth-optimized images and background depth-optimized images can be predicted separately during the inference stage, and then the depth-optimized image can be synthesized. Specifically, the monocular RGB image I and its corresponding depth estimation image, as well as the foreground subject matting image M corresponding to the monocular RGB image I, can be input into the trained depth optimization model to output the foreground depth-optimized image. Then, the monocular RGB image I and its corresponding depth estimation image, as well as the background-subject matting image (1-M) corresponding to the monocular RGB image I, are input into the trained depth optimization model, and the background depth-optimized image is output. Finally, the depth-optimized image is calculated using the following formula. :

[0068]

[0069] Although the above method is superior to traditional deep optimization methods, it still has many problems:

[0070] First, although a self-supervised learning method was used and no additional labels were required, synthetic monocular RGB images and synthetic depth estimation images were used during training. The entire training process did not use real monocular RGB images and their corresponding depth estimation images, which damaged the semantic integrity of the images and the network's ability to identify the depth to be optimized, resulting in low accuracy of the trained depth optimization model.

[0071] Secondly, the Transformer architecture guarantees network performance, but its large size makes the size of the deep optimization network larger than that of the deep estimation network, making it difficult to deploy on mobile devices and limiting its application scenarios.

[0072] Thirdly, the depth-optimized image in the inference stage is obtained by merging the foreground subject matting image and the foreground depth-optimized image, as well as the background subject matting image and the background depth-optimized image. On the one hand, this method itself will bring a heavy jagged edge, resulting in poor quality of the synthesized depth-optimized image. For example, Figure 4 To obtain a depth-optimized image using relevant techniques, by observation Figure 4As can be seen from the depth-optimized image shown, it has obvious jagged edges. On the other hand, the quality of the synthesized depth-optimized image directly depends on the quality of the foreground and background subject matting. If the foreground and background subject matting are of poor quality, the quality of the synthesized depth-optimized image will also be poor.

[0073] To improve the accuracy of the depth optimization model, achieve lighter and stronger mobile deployment performance, and provide more robust optimization capabilities for image matting quality, this application presents a depth optimization model, a training method for the depth optimization model, and an optimization method for depth-estimated images based on the trained depth optimization model. Please refer to... Figure 5 It illustrates the implementation environment involved in the embodiments of this application, see [link to relevant documentation]. Figure 5 The implementation environment includes a server 501 and a terminal 502. The server 501 and the terminal 502 can communicate via a network 503, which can be a wired network or a wireless network.

[0074] Server 501 possesses strong computing power and can be used to train a depth optimization model. Server 501 can be a standalone physical server, a server cluster, or a distributed system composed of multiple physical servers. Once server 501 has trained the depth optimization model, it can send the trained model to terminal 502 via network 503. Terminal 502 then installs and deploys the trained depth optimization model on a mobile device. Terminal 502 can be a smartphone, tablet, robot, vehicle, etc., and can optimize the depth estimation image based on the installed trained depth optimization model.

[0075] Figure 6 A schematic diagram of the network structure of the deep optimization model designed in the embodiments of this application is shown. See also: Figure 6The input information of this depth optimization model consists of a sample monocular RGB image, a sample depth estimation image corresponding to the sample monocular RGB image, and a sample matted image corresponding to the sample monocular RGB image. The output information consists of a subject depth prediction image and a depth prediction image. This depth optimization model includes an encoding module, a fusion upsampling module, a low-scale information enhancement module, a subject depth prediction module, a subject depth enhancement module, and a depth prediction module. The output of the encoding module is connected to the input of the fusion upsampling module; the output of the fusion upsampling module is connected to the inputs of the subject depth prediction module and the subject depth enhancement module; the output of the low-scale information enhancement module is connected to the inputs of the subject depth prediction module and the subject depth enhancement module; the output of the subject depth prediction module is connected to the input of the subject depth enhancement module; and the output of the subject depth enhancement module is connected to the input of the depth prediction module. The following section uses the model training process as an example to explain the various functional modules included in the depth optimization model.

[0076] 1. Encoding module

[0077] The encoding module, also known as the backbone network, is used to encode the input feature images. The input to the encoding module is a first sample stitched feature image obtained by concatenating the feature images of the sample monocular RGB image, the sample depth estimation image, and the sample matted image. The output is multiple first sample feature images of different sizes and different numbers of feature channels. In this embodiment, the sample monocular RGB image, the sample depth estimation image, and the sample matted image are generally the same size. When stitching the feature images of the sample monocular RGB image, the sample depth estimation image, and the sample matted image, the feature channels of pixels at the same position in these three images can be stitched together. For example, the size of the sample monocular RGB image, the sample depth estimation image, and the sample matted image is... The feature image of the sample monocular RGB image has 3 feature channels per pixel. In the feature image of the sample depth image, each pixel has 1 feature channel. The feature image of the sample matted image has 1 feature channel per pixel. The feature images of the three images are concatenated, that is... The first sample connection feature image that can be obtained .

[0078] In this embodiment, the encoding module can use RegNetX-3.2GF, or it can use MobileNet, ResNet, EfficientNet, etc. The encoding module can include multiple stages of convolutional sub-modules. Each stage of the convolutional sub-module includes at least one convolutional layer, which can encode the input feature image to obtain a first sample feature image corresponding to each stage of the convolutional sub-module. It should be noted that although the figure shows an encoding module including four stages of convolutional sub-modules—stage 1, stage 2, stage 3, and stage 4—the encoding module can include more or fewer stages of convolutional sub-modules depending on the training requirements of the model. In this embodiment, the encoding module includes multiple stages of convolutional sub-modules connected sequentially. Specifically, the input of the first-stage convolutional sub-module is used as the input of the encoding module. The output of the first-stage convolutional sub-module is connected to the input of the second-stage convolutional sub-module, and the output of the second-stage convolutional sub-module is connected to the input of the third-stage convolutional sub-module, and so on, until the last stage of the convolutional sub-module. The output of this last stage of the convolutional sub-module can be used as the output of the encoding module. Based on the above connection relationship, the output of the previous stage of the convolutional sub-module can be used as the input of the next stage of the convolutional sub-module. After convolution operations of multiple stages of convolutional sub-modules, multiple first sample feature images of different sizes and different numbers of feature channels can be obtained. The size and number of feature channels of each first sample feature image are related to the stage of the corresponding convolutional sub-module. Assuming the first sample feature image... The first sample feature image corresponding to the convolutional submodule in the i-th stage, then the first sample feature image This can be configured as needed to ensure that as the number of stages increases, the size of the output first sample feature image decreases while the number of feature channels increases. Figure 6 Taking the network structure shown as an example, the encoding module includes four stages of convolutional sub-modules, namely i c1 can be 96, c2 can be 92, c3 can be 432, and c4 can be 1008.

[0079] The encoding module of this application uses a fully convolutional layer, which is easier to deploy on mobile devices. It solves the problems of related technologies that use a large number of Transformer structures, resulting in a large network that is not conducive to quantization and deployment on mobile devices, and the excessive number of parameters in the network is not conducive to application.

[0080] 2. Fusion upsampling module

[0081] The fusion upsampling module is used to decode multiple first-sample feature images input from the encoding module to obtain second-sample feature images. The fusion upsampling module includes multiple fusion upsampling sub-modules, each including an upsampling layer, a convolutional layer, and an activation function layer. The upsampling layer performs an upsampling operation on the input feature image; this upsampling operation can be a 2x bilinear upsampling operation, a transposed convolutional upsampling operation, a nearest neighbor upsampling operation, etc. The convolutional layer performs a convolution operation on the input feature image; this convolutional layer is a 3x3 convolutional layer, but other sizes of convolutional layers can also be used, or various sizes of convolutional layers can be stacked; this embodiment does not specifically limit this. The activation function layer performs a mapping operation on the input feature image using an activation function; this activation function can be the ReLU function, or tanh, etc. Furthermore, although... Figure 7 The diagram shows the sequential connection of the upsampling layer, convolutional layer, and activation function layer in the fusion upsampling submodule, but the connection order of the upsampling layer, convolutional layer, and activation function layer can be changed as needed.

[0082] In this embodiment, multiple fusion upsampling submodules can be connected sequentially. Specifically, the input of the first fusion upsampling submodule is used as the input of the fusion upsampling module. The output of the first fusion upsampling submodule is connected to the input of the second fusion upsampling submodule, the output of the second fusion upsampling submodule is connected to the input of the third fusion upsampling submodule, and so on, until the last fusion upsampling submodule. The output of the last fusion upsampling submodule can be used as the output of the fusion upsampling module. Based on the above connection relationship, the output of the previous fusion upsampling submodule can be used as the input of the next fusion upsampling submodule. After processing by multiple fusion upsampling submodules, the second sample feature image can finally be obtained.

[0083] In the embodiments of this application, some or all of the multiple fusion upsampling modules correspond to the multiple stages of convolutional sub-modules included in the encoding module, and the correspondence is in reverse order. Assuming the encoding module includes M stages of convolutional sub-modules, and the fusion upsampling module includes N fusion upsampling sub-modules (generally, N is greater than or equal to M), then the Mth stage convolutional sub-module corresponds to the first fusion upsampling sub-module, the (M-1)th stage convolutional sub-module corresponds to the second fusion upsampling sub-module, and so on, with the first stage convolutional sub-module corresponding to the Mth fusion upsampling sub-module. Figure 6For example, the encoding module includes convolutional submodules for stage 1, stage 2, stage 3, and stage 4. The fusion upsampling module includes fusion upsampling submodule 1, fusion upsampling submodule 2, fusion upsampling submodule 3, fusion upsampling submodule 4, and fusion upsampling submodule 5. Specifically, the convolutional submodule for stage 4 corresponds to fusion upsampling submodule 1, the convolutional submodule for stage 3 corresponds to fusion upsampling submodule 2, the convolutional submodule for stage 2 corresponds to fusion upsampling submodule 3, the convolutional submodule for stage 1 corresponds to fusion upsampling submodule 2, and fusion upsampling submodule 5 has no corresponding convolutional submodule. In this embodiment, the fusion upsampling submodule is connected to the convolutional submodule of the corresponding stage and uses the convolutional submodule of the corresponding stage as input. Thus, the input of each fusion upsampling module can include two parts: the output of the convolutional submodule of the corresponding stage and the output of the previous fusion upsampling submodule. Of course, if any fusion upsampling submodule does not have a corresponding stage convolutional submodule, the input of that fusion upsampling submodule is the output of the previous fusion upsampling submodule; for the first fusion upsampling submodule, if it does not have a connected previous fusion upsampling submodule, then the input of the first fusion upsampling submodule is the output of the last stage convolutional submodule. For example, Figure 6 The input information of the fusion upsampling submodule 1 is the first sample feature image output by the convolution submodule of stage 4. The input information of the fusion upsampling submodule 2 includes the output information of the fusion upsampling submodule 1 and the first sample feature image output by the convolution submodule of stage 3. The input information of the fusion upsampling submodule 3 includes the output information of the fusion upsampling submodule 2 and the first sample feature image output by the convolution submodule of stage 2. The input information of the fusion upsampling submodule 4 includes the output information of the fusion upsampling submodule 3 and the first sample feature image output by the convolution submodule of stage 1. The input information of the fusion upsampling submodule 5 is the output information of the fusion upsampling submodule 4.

[0084] For the input and output of each fusion upsampling module, the following will combine... Figure 6 Quantify it. See also Figure 6 The encoding module includes four convolutional sub-modules, and the fusion upsampling module includes five fusion upsampling sub-modules. The j-th ( The feature image input to each fusion upsampling submodule is s. j The output feature image is t j For the first fusion upsampling submodule, its input feature image For the last fusion upsampling submodule, its input feature image is: For other intermediate fusion upsampling submodules, their input feature image

[0085] Assuming each fusion upsampling module includes an upsampling layer capable of performing a 2x bilinear upsampling operation, the size of the feature image will double after being input into the fusion upsampling submodule. For any fusion upsampling submodule, assuming its input feature image is a×b, the output feature image will become 2a×2b after the upsampling operation of this submodule. This change can be expressed as:

[0086]

[0087] in, This represents the number of feature channels in the feature image input to the j-th fusion module. This represents the number of feature channels in the feature image output by the j-th fusion model module. In the actual training process of this application embodiment, cout1 can be selected as c2, cout2 as c3, cout3 as c4, cout4 as c4, and cout5 as c4 / 2.

[0088] 3. Low-scale information enhancement module

[0089] The low-scale information enhancement module enhances the depth features corresponding to the foreground subject in the sample matted image. The input to this module is the sample depth estimation image and the sample matted image, and the output is a third sample feature image. The low-scale information enhancement module includes channel connection layers, multiple convolutional layers, multiple batch regularization layers, and multiple convolutional function layers. Specifically, the channel connection layers connect the feature channels of the feature images of the sample depth estimation image and the feature images of the sample matted image. Batch regularization normalizes the input data, mapping it to a normal distribution with a mean of 0 and a variance of 1. In the embodiments of this application, the convolutional layers, batch regularization layers, and activation function layers included in the low-scale information enhancement module can be stacked to form units with different functions. For example, if C represents a convolutional layer, B represents a batch regularization layer, and R represents a ReLU activation function layer, then stacking the convolutional layer and activation function layer can form a convolutional and activation function unit, which can be represented as CR. When a feature image is input into the convolutional and activation function unit, the convolutional and activation function unit first performs a convolution operation on the input feature image, and then performs a mapping operation using an activation function. Similarly, stacking the convolutional layer, batch regularization layer, and activation function layer can form a convolutional, batch regularization, and activation function unit, which can be represented as CBR. When a feature image is input into the convolutional, batch regularization, and activation function unit, the convolutional, batch regularization, and activation function unit first performs a convolution operation on the input feature image, then performs a batch regularization operation, and finally performs a mapping operation using an activation function. Figure 8 A schematic diagram of a low-scale information enhancement module provided in an embodiment of this application is shown below. Figure 8The low-scale information enhancement module includes a channel connection layer, a convolution and activation function unit, a convolution, batch regularization and activation function unit 1, a convolution, batch regularization and activation function unit 2, a convolution, batch regularization and activation function unit 3, a convolution, batch regularization and activation function unit 4, a convolution, batch regularization and activation function unit 5, and a convolution, batch regularization and activation function unit 6. In this unit, the convolutional kernel of the convolutional layer in the convolution and activation function unit can be 3*3, and the convolution and activation function unit is represented as 3*3CR; the convolutional kernel of the convolutional layer in convolution, batch regularization, and activation function unit 1 can be 1*1, and the convolution, batch regularization, and activation function unit 1 is represented as 1*1CBR; the convolutional kernel of the convolutional layer in convolution, batch regularization, and activation function unit 2 can be 3*3, and the convolution, batch regularization, and activation function unit 2 is represented as 3*3CBR; the convolutional kernel of the convolutional layer in convolution, batch regularization, and activation function unit 3 can be 1*1, and the convolutional kernel of the convolutional layer in convolution, batch regularization, and activation function unit 3 can be 1*1. The batch regularization and activation function unit 3 is represented as a 1*1 CBR; the convolution kernel of the convolutional layer in the convolution, batch regularization and activation function unit 4 can be 1*1, and this convolution, batch regularization and activation function unit 4 is represented as a 1*1 CBR; the convolution kernel of the convolutional layer in the convolution, batch regularization and activation function unit 5 can be 3*3, and this convolution, batch regularization and activation function unit 5 is represented as a 3*3 CBR; the convolution kernel of the convolutional layer in the convolution, batch regularization and activation function unit 6 can be 3*3, and this convolution, batch regularization and activation function unit 6 is represented as a 3*3 CBR. The feature channels of the feature image output by each convolutional layer of the above low-scale information enhancement module can be c4 / 2.

[0090] 4. Main body depth prediction module

[0091] The subject depth prediction module predicts the depth image corresponding to the foreground subject. The input to the subject depth prediction module consists of two parts: a second sample feature image output by the upsampling fusion module and a third sample feature image output by the low-scale information enhancement module. The output of the subject depth prediction module is a fourth sample feature image, which is used to generate the subject depth prediction image. For example, the fourth sample feature image is... The sample image is Then, by multiplying the pixel value of each pixel in the sample matted image M with the pixel value of the pixel at the same position in the fourth sample feature image, the subject depth prediction image can be obtained. ,Right now The subject depth prediction image is used to calculate the prediction loss for the depth image corresponding to the foreground subject. This subject depth prediction module may include channel connection layers, activation function layers, and convolutional layers, etc. Figure 9 The structure of the main body depth prediction module is shown in the figure. Figure 9The main body depth prediction module includes a channel connection layer, an activation function layer, convolutional and activation function units, and a convolutional layer. The channel connection layer connects the feature channels of the second and third sample feature images. The first convolutional layer in the main body depth prediction module, i.e., the convolutional layer in the convolutional and activation function unit, has a 3x3 kernel and outputs c4 / 2 feature channels. The second convolutional layer in the main body depth prediction module outputs the fourth sample feature image. It should be noted that, Figure 9 The activation function in the activation function layer of the main body depth prediction module shown can be the ReLU activation function or other activation functions, and the number of convolution kernels and output feature channels of the two convolutional layers can be set according to requirements.

[0092] 5. Main Body Depth Enhancement Module

[0093] The subject depth enhancement module enhances the image features of the foreground subject in the depth image. The input to the subject depth enhancement module consists of three parts: a second sample feature image output by the fusion upsampling module, a third sample feature image output by the low-scale information enhancement module, and a fourth sample feature image output by the subject depth prediction module. The output of the subject depth enhancement module is a fifth sample feature image, which is used to predict the depth-optimized image. The subject depth enhancement module may include channel connection layers, multiple convolutional layers, multiple activation function layers, multiple batch regularization layers, and multiple activation function layers. In this embodiment, the convolutional layers, batch regularization layers, and activation function layers included in the subject depth enhancement module can be stacked to form units with different functions. For example, let C represent a convolutional layer, B represent a batch regularization layer, and R represent a ReLU activation function layer. Stacking convolutional layers and activation function layers can form a convolutional and activation function unit, denoted as CR. When a feature image is input into the convolutional and activation function unit, the unit first performs a convolution operation on the input feature image and then performs a mapping operation using the activation function. Stacking convolutional layers and batch regularization layers can form a convolutional and batch regularization unit, denoted as CB. When a feature image is input into the convolutional and batch regularization unit, the unit first performs a convolution operation on the input feature image and then performs a mapping operation using the activation function. When a feature image is input to a convolution and batch regularization unit (CBR), the CBR first performs a convolution operation on the input feature image, then a batch regularization operation, and finally performs a mapping operation using an activation function. Figure 10A schematic diagram of the structure of a body depth enhancement module provided in an embodiment of this application is shown below. Figure 10 The main deep enhancement module may include a channel connection layer, a convolution and activation function unit, a convolution and batch regularization unit 1, a convolution, batch regularization and activation function unit 1, a convolution, batch regularization and activation function unit 2, a convolution and batch regularization unit 2, an addition and activation function layer, a convolution, batch regularization and activation function unit 3, a convolution, batch regularization and activation function unit 4, and a convolution, batch regularization and activation function unit 5. In the convolution and activation function unit, the convolution kernel of the convolutional layer can be 3*3, and the output feature channels are c4, thus the convolution and activation function unit is represented as 3*3*c4 CR; the convolution kernel of the convolutional layer in convolution and batch regularization unit 1 can be 1*1, and the output feature channels are c4 / 2, thus the convolution and batch regularization unit 1 is represented as 3*3*(c4 / 2). CB; In Convolution, Batch Regularization, and Activation Function Unit 1, the convolutional kernel of the convolutional layer can be 1*1, and the output feature channels are c4 / 2. This convolution, batch regularization, and activation function unit 1 is represented as 1*1*(c4 / 2)CBR; In Convolution, Batch Regularization, and Activation Function Unit 2, the convolutional kernel of the convolutional layer can be 3*3, and the output feature channels are c4 / 2. This convolution, batch regularization, and activation function unit 2 is represented as 3*3*(c4 / 2)CBR; In Convolution and Batch Regularization Unit 2, the convolutional kernel of the convolutional layer can be 1*1, and the output feature channels are c4 / 2. This convolution and batch regularization unit 2 is represented as 1*1*(c4 / 2)CB; Convolution, Batch The convolutional kernel of the convolutional layer in regularization and activation function unit 3 can be 1*1, and the output feature channels are c4 / 2. This convolutional, batch regularization and activation function unit 3 is represented as 1*1*(c4 / 2)CBR; the convolutional kernel of the convolutional layer in convolutional, batch regularization and activation function unit 4 can be 3*3, and the output feature channels are c4 / 2. This convolutional, batch regularization and activation function unit 4 is represented as 3*3*(c4 / 2)CBR; the convolutional kernel of the convolutional layer in convolutional, batch regularization and activation function unit 5 can be 3*3, and the output feature channels are c4 / 2. This convolutional, batch regularization and activation function unit 5 is represented as 3*3*(c4 / 2)CBR.

[0094] 6. Depth Prediction Module

[0095] The depth prediction module predicts the depth-optimized image corresponding to the sample monocular RGB image. The input to the depth prediction module is the fifth sample feature image output by the main depth enhancement module, and the output is the depth prediction image. See also... Figure 11The depth prediction module includes activation function layers and convolutional layers. The activation function in the activation function layer can be ReLU, and the convolutional kernel in the convolutional layer can be 1x1. When the feature image of the fifth sample is input into the depth prediction module, after the mapping operation of the activation function layer and the convolution operation of the convolutional layer, the final output is the depth prediction image corresponding to the sample monocular RGB image.

[0096] The backbone network structure of this embodiment adopts a fully convolutional layer structure, which is easier to deploy on the edge. The input information is relatively rich, including a monocular RGB image and its corresponding depth estimation image and matted image. The network has two input points: one is the input of the encoding module, where the monocular RGB image and its corresponding depth estimation image and matted image are input into the encoding module. After encoding by multiple stages of convolutional sub-modules, multiple first sample feature images of different sizes and feature channel numbers are obtained; the second is the input of the low-scale information enhancement module, where the depth estimation image and matted image are input into the low-scale information enhancement module. After processing by the low-scale information enhancement module, a third sample feature image is obtained. Compared to the three input points in related technologies, the network architecture of this embodiment reduces network branches, thus making it easier to deploy on the edge. The first sample feature image output by the corresponding convolutional sub-module and the output result of the previous fusion upsampling sub-module are input into the corresponding fusion upsampling sub-module. After processing by the corresponding fusion upsampling sub-module, the output result can be used as the input of the next fusion upsampling module. Next, the outputs of the low-scale information enhancement module and the fusion upsampling module are used as inputs to the subject depth prediction module to generate a subject depth prediction image corresponding to the foreground subject. This subject depth prediction image can provide auxiliary loss for model training, thereby improving the final optimization effect. Then, the fourth sample feature image output by the subject depth prediction module, the third sample feature image output by the low-scale information enhancement module, and the second sample feature image output by the fusion upsampling module are input into the subject depth enhancement module. The subject depth enhancement module fuses and enhances the depth prediction result corresponding to the foreground subject with the aforementioned information, outputting a fifth sample feature image. This fifth sample feature image is then input into the depth prediction module, ultimately outputting the depth prediction image.

[0097] The network using this embodiment can directly output a complete depth-optimized image without the need for image matting and synthesis, thus avoiding the jagged edges of the depth-optimized image. Furthermore, the input during model training is a real monocular RGB image and its corresponding depth estimation image, maintaining the semantic integrity of the image and the network's ability to identify the depth location to be optimized, thereby improving the model's accuracy. Additionally, the backbone network uses fully convolutional layers, making the network lighter and easier to deploy on mobile devices.

[0098] The network structure of this application embodiment has better performance and a wider range of applications. It can generate depth-optimized images not only for high-resolution images, such as medical images, but also for low-resolution images.

[0099] Regarding the network designed in the embodiments of this application, the following points need to be explained:

[0100] First, if you want to run the network with less computational cost and achieve better speed, you can remove the scale information enhancement module, the subject depth prediction module, and the subject depth enhancement module from the network, but the depth optimization effect will also be worse.

[0101] Secondly, the above Figure 6 The network structure shown may not include the main depth enhancement module. Accordingly, the network structure of the depth prediction module will change. In this case, the outputs of the fusion upsampling module, the low-scale information enhancement module, and the main depth prediction module will be input into the depth prediction module.

[0102] Thirdly, the network structure provided in this application embodiment can be used for depth estimation instead of depth optimization. When the network structure is used for depth estimation, the input information of the network needs to be changed, such as removing the depth estimation image. At the same time, the input channels of some parts of the network need to be changed. For example, the input of some modules needs to be changed from five channels to four channels. Specifically, the input of the low-scale information enhancement module and the encoding module needs to be changed from five channels to four channels.

[0103] Fourthly, the network structure provided in this application embodiment can be used for obtaining matted images instead of depth optimization. When this network structure is used to obtain matted images, the network input information needs to be changed, such as removing the matted image and removing auxiliary losses related to subject matting. Simultaneously, the input channels of some network components need to be changed; for example, the input of some modules can be changed from five channels to four channels. Specifically, the inputs of the low-scale information enhancement module and the encoding module need to be changed to four channels.

[0104] This application provides a training method for a deep optimization model, the network structure of which is as follows: Figure 6 As shown, with Figure 5 The server execution application embodiment shown is an example, see [link / reference]. Figure 12 The method flow provided in this application embodiment includes:

[0105] 1201. Obtain the sample monocular RGB image, the sample depth estimation image, and the sample matted image.

[0106] The sample depth estimation image is obtained by performing depth estimation on the sample monocular RGB image, which can be obtained using existing monocular depth estimation algorithms. The sample matted image is obtained by matting out the foreground subject from the sample monocular RGB image, which can be manually labeled or generated using matting methods. This sample monocular RGB image has a correct label, i.e., a depth label image. This depth label image can be obtained through sensor acquisition or by estimating the sample monocular RGB image using a large-scale depth estimation network (such as MIDAS or Depth-Anything). This depth label image can be represented as... .

[0107] If the number of acquired sample monocular RGB images and their corresponding sample depth estimation images and sample matting images is sufficient, step 1202 can be executed directly. If the number of acquired sample monocular RGB images and their corresponding sample depth estimation images and sample matting images is insufficient, this embodiment of the application can synthesize samples before executing step 1202. The sample synthesis process is as follows: acquire any sample monocular RGB image I and its corresponding sample depth estimation image D, and randomly acquire a sample matting image M. Then, acquire the foreground image I1 and the background image I2 from the sample monocular RGB image. Based on the foreground image I1, acquire the foreground depth estimation image D1 from the depth estimation image D, and based on the background image I2, acquire the background depth estimation image D2 from the depth estimation image D. Then, use the following formula to synthesize a new monocular RGB image I. ’ and the new depth estimation image D ’ :

[0108]

[0109]

[0110] The synthesized new monocular RGB image is then used as the supervised label for the subsequent optimization process, with the sample depth estimation image D corresponding to the sample monocular RGB image as the supervised label.

[0111] 1202. Input the sample monocular RGB image, sample depth estimation image and sample matted image into the depth optimization model, and output the subject depth prediction image and depth prediction image.

[0112] Specifically, the sample monocular RGB image, sample depth estimation image, and sample matted image are input into the depth optimization model, and the output is the subject depth prediction image and the depth prediction image, including the following steps:

[0113] 12021. Connect the feature channels of pixels at the same position in the sample monocular RGB image, the sample depth estimation image, and the sample matted image to obtain the first sample connected feature image.

[0114] Generally, the sample monocular RGB image I, the sample depth estimation image D, and the sample matted image M have the same size, which can be set to h*w. Each pixel in the sample monocular RGB image M typically includes three feature channels: R / G / B. Since the depth value of each pixel in the sample depth estimation image D represents the distance between the object containing that pixel and the monocular camera, it can be known that each pixel in the sample depth estimation image D has a feature channel. Since the pixel value of each pixel in the sample matted image M represents the probability that the subject containing that pixel belongs to the foreground subject, it can be known that the sample matted image M has a feature channel. When the feature channels of pixels at the same location in the sample monocular RGB image, the sample depth estimation image, and the sample matted image are concatenated, a five-channel first sample concatenated feature image can be obtained. ,Should .

[0115] 12022. Input the first sample concatenation feature image into the encoding module and output multiple first sample feature images.

[0116] The encoding module comprises multiple stages of convolutional sub-modules, each containing a convolutional layer. Multiple first-sample feature images have different sizes and numbers of feature channels, and each first-sample feature image fuses image features from a sample monocular RGB image, a sample depth estimation image, and a sample matted image. Each first-sample feature image corresponds to a stage of the convolutional sub-module. As the number of stages increases, the size of the first-sample feature image corresponding to the convolutional sub-module continuously decreases, and is always smaller than the size of the first-sample connected feature image, while the number of feature channels increases, and is always greater than the number of feature channels in the first-sample connected feature image.

[0117] Specifically, the first sample concatenation feature image is input into the encoding module, and multiple first sample feature images are output. This includes: inputting the first sample concatenation feature image into the first-stage convolutional submodule, where the convolutional layers included in the first-stage convolutional submodule perform convolution operations on the first sample concatenation feature image to obtain the first sample feature image corresponding to the first-stage convolutional submodule; inputting the first sample feature image corresponding to the previous-stage convolutional submodule into the next-stage convolutional submodule for operation, until the last-stage convolutional submodule; and obtaining the first sample feature image output by each stage convolutional submodule to obtain multiple first sample feature images.

[0118] 12023. Input multiple first sample feature images into the fusion upsampling module and output the second sample feature image.

[0119] The fusion upsampling module comprises multiple fusion upsampling sub-modules. Each fusion upsampling module includes an upsampling layer, a convolutional layer, and an activation function layer. Each stage of the convolutional sub-module corresponds to one fusion upsampling sub-module. The second sample feature image also fuses the sample monocular RGB image, the sample depth estimation image, and the sample matting image. After processing by each fusion upsampling sub-module, the size of the second sample image feature can be h*w.

[0120] Specifically, multiple first sample feature images are input into the fusion upsampling module to output a second sample feature image. This includes: inputting the first sample feature image of the last stage convolutional submodule into the corresponding first fusion upsampling submodule; the first fusion upsampling submodule, including its upsampling layer, convolutional layer, and activation function layer, performs upsampling, convolution, and activation function mapping operations on the input first sample feature image to obtain the output of the first fusion upsampling submodule; inputting the output of the previous fusion upsampling submodule and the first sample feature image of the corresponding convolutional submodule into the next fusion upsampling submodule for upsampling, convolution, and activation function mapping operations, until the last fusion upsampling submodule; and using the output of the last fusion upsampling submodule as the second sample feature image.

[0121] 12024. Input the sample depth estimation image and the sample matting image into the low-scale information enhancement module, and output the third sample feature image.

[0122] The low-scale information enhancement module includes channel connection layers, convolutional layers, batch regularization layers, and activation function layers. The third sample feature image integrates image features from the sample depth estimation image and the sample matted image, which is used to enhance the depth features of the corresponding part of the foreground subject.

[0123] Specifically, the sample depth estimation image and the sample matted image are input into the low-scale information enhancement module to output a third sample feature image. This includes: inputting the sample depth estimation image and the sample matted image into the low-scale information enhancement module; the channel connection layer included in the low-scale information enhancement module connects the feature channels of pixels at the same position in the sample depth estimation image and the sample matted image to obtain a second sample connection feature image; and then the convolutional layer, batch regularization layer and activation function layer included in the low-scale information enhancement module perform convolution operation, batch regularization operation and activation function mapping operation on the second sample connection feature image to obtain the third sample feature image.

[0124] 12025. Input the second and third sample feature images into the subject depth prediction module and output the fourth sample feature image.

[0125] The main subject depth prediction module includes channel connection layers, convolutional layers, and activation function layers. The fourth sample feature image integrates image features from the sample monocular RGB image, the sample depth estimation image, and the sample matted image, and the depth features corresponding to the foreground main subject are enhanced.

[0126] Specifically, the second and third sample feature images are input into the subject depth prediction module to output a fourth sample feature image. This includes: inputting the second and third sample feature images into the subject depth prediction module; connecting the feature channels of pixels at the same position in the second and third sample feature images using the channel connection layer included in the subject depth prediction module to obtain a third sample connected feature image; and then performing convolution operations and activation function mapping operations on the third sample connected feature image using the convolution layer and activation function layer included in the subject depth prediction module to obtain the fourth sample feature image.

[0127] 12026. Generate a subject depth prediction image based on the fourth sample feature image and the sample matted image.

[0128] Based on the acquired fourth sample feature map and sample matted image, the subject depth prediction image can be obtained by multiplying the pixel value of each pixel in the fourth sample feature map with the pixel value of the pixel at the same position in the sample matted image. This subject depth prediction image can be represented as follows: .

[0129] 12027. Input the second, third, and fourth sample feature images into the subject depth enhancement module and output the fifth sample feature image.

[0130] The main subject depth enhancement module includes channel connection layers, convolutional layers, batch regularization layers, and activation function layers. The fifth sample feature image integrates image features from the sample monocular RGB image, the sample depth estimation image, and the sample matted image, and the depth features corresponding to the foreground subject are enhanced.

[0131] Specifically, the second, third, and fourth sample feature images are input into the main body depth enhancement module to output the fifth sample feature image. This includes: inputting the second, third, and fourth sample feature images into the main body depth enhancement module; connecting the feature channels of pixels at the same position in the second, third, and fourth sample feature images using the channel connection layer included in the main body depth enhancement module to obtain the fourth sample connection feature image; and then performing convolution, batch regularization, and activation function mapping operations on the fourth sample connection feature image using the convolution layer, batch regularization layer, and activation function layer included in the main body depth enhancement module to obtain the fifth sample feature image.

[0132] 12028. Input the feature image of the fifth sample into the depth prediction module and output the depth prediction image.

[0133] The depth prediction module includes activation function layers and convolutional layers. Specifically, the process of inputting the fifth sample feature image into the depth prediction module and outputting a depth prediction image includes: inputting the fifth sample feature image into the depth prediction module, and having the activation function layer and convolutional layer of the depth prediction module perform activation function mapping operations and convolution operations on the fifth sample feature image, respectively, to obtain the depth prediction image. This depth prediction image can be represented as follows:

[0134] 1203. Based on the subject depth prediction image, sample matted image, depth label image, depth prediction image and total objective loss function, the depth optimization model is trained to obtain the trained depth optimization model.

[0135] The overall objective loss function includes a first objective loss function and a second objective loss function. The first objective loss function calculates the prediction loss for the subject depth image, and the second objective loss function calculates the prediction loss for the depth-optimized image. Both the first and second objective loss functions can be functions composed of L1 loss, L2 loss, and gradient loss functions. The L1 loss function is also called the Mean Absolute Error (MAE), and the L2 loss function is also called the Mean Squared Error (MSE). The L1, L2, and gradient loss functions can all be used to measure the degree of loss between the predicted and labeled values.

[0136] In this embodiment, the first target loss function and the second target loss function, which consist of the L1 loss function, the L2 loss function, and the gradient loss function, can both be expressed as:

[0137]

[0138] In the embodiments of the application, the first objective loss function is used to calculate the loss between the predicted subject depth image and the subject depth label image. The first objective loss function can be expressed as follows: The second objective loss function is used to calculate the loss between the depth prediction image and the depth label image. The second objective loss function can be expressed as follows: The overall objective loss function is obtained by weighting the first objective loss function and the second objective loss function, and can be expressed as:

[0139]

[0140] Wherein, α, β, γ, θ, and λ are weight values. In the embodiments of this application, α can be selected as 10, β as 100, γ as 10, θ as 2, and λ as 1. It is worth noting that other common regression loss functions, such as Smooth L1, can also be used to construct the first objective loss function and the second objective loss function, and the values ​​of α, β, γ, θ, and λ do not necessarily have to be the values ​​mentioned above; other values ​​can also be selected.

[0141] After completing the above settings, the depth optimization model can be trained based on the subject depth prediction image, sample matted image, depth label image, depth prediction image, and overall objective loss function. Specifically, training the depth optimization model based on the subject depth prediction image, sample matted image, depth label image, depth prediction image, and overall objective loss function to obtain the trained depth optimization model includes the following steps:

[0142] 12031. Multiply the pixel value of each pixel in the depth label image with the pixel value of the pixel at the same position in the sample cutout image to obtain the main body depth label image.

[0143] 12032. Input the predicted depth image and the labeled depth image of the subject into the first target loss function to obtain the value of the first target loss function.

[0144] 12033. Input the depth prediction image and the depth label image into the second target loss function to obtain the value of the second target loss function.

[0145] 12034. Based on the first objective loss function value and the second objective loss function value, calculate the total objective loss function value.

[0146] 12035. Based on the target loss function value, adjust the model parameters of the deep optimization model to obtain the trained deep optimization model.

[0147] During model training, the optimizedr selected can be the Adam optimizer, although other optimizers such as SGD and RMSprop can also be chosen. The initial learning rate can be set to lr=1e-5 and eps=1e-8. During model training, the number of iterations (epochs) can be set to 100, with the learning rate decaying by 0.1 every 30 epochs. Of course, other common decay strategies and combinations of epochs can also be used in the training process of this embodiment. When the number of iterations for the model parameters reaches the set number, model training can be stopped, and the model parameters at that iteration point can be obtained. The depth optimization model corresponding to these model parameters is then used as the trained depth optimization model. The trained depth optimization model using the method provided in this embodiment can be used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

[0148] In this embodiment, the input during model training is a real monocular RGB image and its corresponding depth estimation image. Compared to related technologies that employ self-supervised learning methods, this embodiment uses synthesized monocular RGB images and synthesized depth estimation images throughout the entire network training process, without using real monocular RGB images and their corresponding depth estimation images. This embodiment maintains the semantic integrity of the images and the network's ability to identify the depth location to be optimized, thus improving the model's accuracy. Furthermore, the backbone network of this embodiment uses fully convolutional layers, resulting in a lighter network architecture. Therefore, it can be deployed on mobile devices using common machine learning tools such as SNPE, QNN, TFLite, and CoreML through quantization methods.

[0149] In summary, the advantages of the model training method provided in this application are as follows:

[0150] From the perspective of input, the embodiments of this application use a monocular RGB original image as input, which is an additional method compared with the methods of related technologies. For erroneous depth images, more clues from the original image can be used for correction, thereby improving the depth optimization effect.

[0151] From the perspective of the loss function, the embodiments of this application add auxiliary loss calculation for the depth part of the main body, which can enhance the depth image of the matted part separately and correct the depth image of the matted / segmented part.

[0152] From the output perspective, the network in this embodiment can directly output a complete depth-optimized image without using image matting for synthesis, thus avoiding the jagged edges of the depth-optimized image.

[0153] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0154] This application provides an optimization method for depth estimation images, in order to Figure 5 Taking the implementation of this application embodiment in a terminal as an example, the terminal is equipped with... Figure 12 The trained deep optimization model shown in the example is referred to in [reference]. Figure 13 The method flow provided in this application embodiment includes:

[0155] 1301. Obtain the monocular RGB image and its corresponding depth estimation image and matted image.

[0156] In practical applications, when downstream business demands require high-precision depth-optimized images corresponding to monocular RGB images, a monocular camera can be used to acquire monocular RGB images. These acquired images are then input into a depth estimation network to obtain the corresponding depth-estimated images. Simultaneously, a matting algorithm is needed to perform matting processing on the monocular RGB images to obtain the matted images.

[0157] 1302. Input the monocular RGB image, the depth estimation image, and the matted image into the trained depth optimization model, and output the depth optimization image.

[0158] Once you have a monocular RGB image, a depth estimation image, and a matted image, you can call a pre-trained depth optimization model. Input the monocular RGB image, the depth estimation image, and the matted image into the pre-trained depth optimization model. After processing by the pre-trained depth optimization model, the output is a depth-optimized image corresponding to the monocular RGB image. This depth-optimized image has a better effect and will not have a jagged edge.

[0159] For example, Figure 14 Three images are shown, among which, Figure 14 Figure (a) is a monocular RGB image. Figure 14 Figure (b) in the image is the depth estimation image corresponding to the monocular RGB image. Figure 14 Image (c) in the image is the cutout image corresponding to the monocular RGB image. Figure 14 The three images shown are input into a trained deep learning optimization model. After processing by the trained deep learning optimization model, the final output is... Figure 15 The depth-optimized image is shown. By observing... Figure 15 As can be seen from the depth-optimized image shown, the effect of this image is better than... Figure 14 The depth estimation image in (b) is better and has no jagged edges.

[0160] Please refer to Figure 16The diagram illustrates a structural schematic of a training device for a deep optimization model, provided in an embodiment of this application. This device can be implemented through software, hardware, or a combination of both, and can be all or part of an electronic device. The device includes:

[0161] The acquisition module 1601 is used to acquire a sample monocular RGB image, a sample depth estimation image, and a sample matted image. The sample depth estimation image is an image obtained by performing depth estimation on the sample monocular RGB image. The sample matted image is an image obtained by matting the foreground subject in the sample monocular RGB image. The sample monocular RGB image has a depth label image.

[0162] The input / output module 1602 is used to input the sample monocular RGB image, the sample depth estimation image and the sample matting image into the depth optimization model, and output the subject depth prediction image and the depth prediction image.

[0163] Training module 1603 is used to train a depth optimization model based on the subject depth prediction image, sample matted image, depth label image, depth prediction image and total target loss function to obtain a trained depth optimization model. This trained depth optimization model is used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

[0164] In another embodiment of this application, the depth optimization model includes an encoding module, a fusion upsampling module, a low-scale information enhancement module, a subject depth prediction module, a subject depth enhancement module, and a depth prediction module. An input / output module 1602 is used to connect the feature channels of pixels at the same location in a sample monocular RGB image, a sample depth estimation image, and a sample matted image to obtain a first sample connected feature image; input the first sample connected feature image into the encoding module to output multiple first sample feature images, which have different sizes and number of feature channels; input the multiple first sample feature images into the fusion upsampling module to output a second sample feature image; input the sample depth estimation image and the sample matted image into the low-scale information enhancement module to output a third sample feature image; input the second and third sample feature images into the subject depth prediction module to output a fourth sample feature image; generate a subject depth prediction image based on the fourth sample feature image and the sample matted image; input the second, third, and fourth sample feature images into the subject depth enhancement module to output a fifth sample feature image; and input the fifth sample feature image into the depth prediction module to output a depth prediction image.

[0165] In another embodiment of this application, the encoding module includes multiple stages of convolutional sub-modules, each stage of the convolutional sub-module including a convolutional layer. The input / output module 1602 is used to input the first sample concatenation feature image into the first stage of the convolutional sub-module, and the convolutional layer included in the first stage of the convolutional sub-module performs a convolution operation on the first sample concatenation feature image to obtain the first sample feature image corresponding to the first stage of the convolutional sub-module; input the first sample feature image corresponding to the previous stage of the convolutional sub-module into the next stage of the convolutional sub-module for operation, until the last stage of the convolutional sub-module; obtain the first sample feature image output by each stage of the convolutional sub-module to obtain multiple first sample feature images.

[0166] In another embodiment of this application, the fusion upsampling module includes multiple fusion upsampling sub-modules. Each fusion upsampling module includes an upsampling layer, a convolutional layer, and an activation function layer. Each stage of the convolutional sub-module corresponds to one fusion upsampling sub-module. The input-output module 1602 is used to input the first sample feature image of the last stage of the convolutional sub-module into the corresponding first fusion upsampling sub-module. The upsampling layer, convolutional layer, and activation function layer of the first fusion upsampling sub-module perform upsampling operations, convolution operations, and activation function mapping operations on the input first sample feature image, respectively, to obtain the output result of the first fusion upsampling sub-module. The output result of the previous fusion upsampling sub-module and the first sample feature image of the corresponding stage of the convolutional module are input into the next fusion upsampling sub-module to perform upsampling operations, convolution operations, and activation function mapping operations, until the last fusion upsampling sub-module. The output result of the last fusion upsampling sub-module is used as the second sample feature image.

[0167] In another embodiment of this application, the low-scale information enhancement module includes a channel connection layer, a convolutional layer, a batch regularization layer, and an activation function layer. The input / output module 1602 is used to input the sample depth estimation image and the sample matting image into the low-scale information enhancement module. The channel connection layer included in the low-scale information enhancement module connects the feature channels of pixels at the same position in the sample depth estimation image and the sample matting image to obtain a second sample connection feature image. Then, the convolutional layer, batch regularization layer, and activation function layer included in the low-scale information enhancement module perform convolution operation, batch regularization operation, and activation function mapping operation on the second sample connection feature image, respectively, to obtain a third sample feature image.

[0168] In another embodiment of this application, the subject depth prediction module includes a channel connection layer, a convolutional layer, and an activation function layer. The input / output module 1602 is used to input the second sample feature image and the third sample feature image into the subject depth prediction module. The channel connection layer included in the subject depth prediction module connects the feature channels of pixels at the same position in the second sample feature image and the third sample feature image to obtain the third sample connected feature image. Then, the convolutional layer and the activation function layer included in the subject depth prediction module perform convolution operation and activation function mapping operation on the third sample connected feature image to obtain the fourth sample feature image.

[0169] In another embodiment of this application, the input / output module 1602 is used to multiply the pixel value of each pixel in the fourth sample feature image with the pixel value of the pixel at the same position in the sample matting image to obtain the subject depth prediction image.

[0170] In another embodiment of this application, the main depth enhancement module includes a channel connection layer, a convolutional layer, a batch regularization layer, and an activation function layer. The input / output module 1602 is used to input the second sample feature image, the third sample feature image, and the fourth sample feature image into the main depth enhancement module. The channel connection layer included in the main depth enhancement module connects the feature channels of pixels at the same position in the second sample feature image, the third sample feature image, and the fourth sample feature image to obtain the fourth sample connection feature image. Then, the convolutional layer, the batch regularization layer, and the activation function layer included in the main depth enhancement module perform convolution operations, batch regularization operations, and activation function mapping operations on the fourth sample connection feature image, respectively, to obtain the fifth sample feature image.

[0171] In another embodiment of this application, the depth prediction module includes an activation function layer and a convolutional layer. The input-output module 1602 is used to input the fifth sample feature image into the depth prediction module. The activation function layer and the convolutional layer included in the depth prediction module perform activation function mapping operation and convolution operation on the fifth sample feature image, respectively, to obtain the depth prediction image.

[0172] In another embodiment of this application, the total objective loss function includes a first objective loss function and a second objective loss function. The first objective loss function is used to calculate the prediction loss for the subject depth image, and the second objective loss function is used to calculate the prediction loss for the depth-optimized image. The training module 1603 is used to multiply the pixel value of each pixel in the depth label image with the pixel value of the pixel at the same position in the sample matted image to obtain the subject depth label image; input the subject depth prediction image and the subject depth label image into the first objective loss function to obtain the first objective loss function value; input the depth prediction image and the depth label image into the second objective loss function to obtain the second objective loss function value; calculate the total objective loss function value based on the first objective loss function value and the second objective loss function value; and adjust the model parameters of the depth optimization model based on the objective loss function value to obtain the trained depth optimization model.

[0173] Please refer to Figure 17 The diagram illustrates a structural schematic of an optimization device for depth estimation images provided in an embodiment of this application, which includes a... Figure 13 The illustrated embodiment shows a trained deep optimization model. This device can be implemented through software, hardware, or a combination of both, and can be all or part of an electronic device. The device includes:

[0174] The acquisition module 1701 is used to acquire a monocular RGB image and its corresponding depth estimation image and matted image;

[0175] The input / output module 1702 is used to input the monocular RGB image, the depth estimation image, and the matted image into the trained depth optimization model and output the depth optimization image.

[0176] Figure 18 A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 can be a mobile phone, personal computer (PC), tablet computer, AR (Augmented Reality) device, VR (Virtual Reality) device, in-vehicle computer, wearable device, smart home device, etc.

[0177] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, Universal Serial Bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0178] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0179] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0180] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0181] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0182] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a User Identity Module (UIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0183] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0184] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0185] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0186] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0187] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0188] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0189] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, to transfer data between electronic device 100 and peripheral devices, and to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0190] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0191] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0192] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0193] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0194] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0195] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0196] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0197] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0198] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GNASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0199] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0200] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0201] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0202] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0203] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0204] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0205] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0206] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0207] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0208] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phone book, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0209] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, such as music playback and recording.

[0210] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0211] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (e.g., taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (e.g., time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0212] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0213] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0214] This application provides a computer-readable storage medium storing at least one computer program. When executed by a processor, the at least one computer program can implement the training method of the depth optimization model or the optimization method of the depth estimation image described above.

[0215] This application provides a computer program product, which includes a computer program that, when executed by a processor, can implement the training method of the aforementioned depth optimization model or the optimization method of the aforementioned depth estimation image.

[0216] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0217] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a deep optimization model, characterized in that, The method includes: Acquire a sample monocular RGB image, a sample depth estimation image, and a sample matted image. The sample depth estimation image is an image obtained by performing depth estimation on the sample monocular RGB image. The sample matted image is an image obtained by matting out the foreground subject in the sample monocular RGB image. The sample monocular RGB image has a depth label image. The sample monocular RGB image, the sample depth estimation image, and the sample matted image are input into the depth optimization model, and the main body depth prediction image and the depth prediction image are output. Based on the subject depth prediction image, the sample matted image, the depth label image, the depth prediction image, and the overall objective loss function, the depth optimization model is trained to obtain a trained depth optimization model. The trained depth optimization model is used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

2. The method according to claim 1, characterized in that, The depth optimization model includes an encoding module, a fusion upsampling module, a low-scale information enhancement module, a subject depth prediction module, a subject depth enhancement module, and a depth prediction module. The step of inputting the sample monocular RGB image, the sample depth estimation image, and the sample matted image into the depth optimization model, and outputting a subject depth prediction image and a depth prediction image, includes: The feature channels of pixels at the same position in the sample monocular RGB image, the sample depth estimation image, and the sample matted image are connected to obtain the first sample connection feature image; The first sample concatenation feature image is input into the encoding module, and multiple first sample feature images are output, the multiple first sample feature images having different sizes and feature channels; The plurality of first sample feature images are input into the fusion upsampling module, and the second sample feature image is output. The sample depth estimation image and the sample matting image are input into the low-scale information enhancement module, and the third sample feature image is output. The second sample feature image and the third sample feature image are input into the subject depth prediction module, and the fourth sample feature image is output. The subject depth prediction image is generated based on the fourth sample feature image and the sample matted image; The second sample feature image, the third sample feature image, and the fourth sample feature image are input into the subject depth enhancement module, and the fifth sample feature image is output. The fifth sample feature image is input into the depth prediction module, and the depth prediction image is output.

3. The method according to claim 2, characterized in that, The encoding module includes multiple convolutional sub-modules, each of which includes a convolutional layer. The first sample concatenated feature image is input into the encoding module, and multiple first sample feature images are output, including: The first sample concatenation feature image is input into the first stage convolution sub-module, and the convolution layer included in the first stage convolution sub-module performs a convolution operation on the first sample concatenation feature image to obtain the first sample feature image corresponding to the first stage convolution sub-module. The first sample feature image corresponding to the convolutional submodule of the previous stage is input into the convolutional submodule of the next stage for operation, until the last convolutional submodule. Obtain the first sample feature image output by the convolutional submodule at each stage to obtain the plurality of first sample feature images.

4. The method according to claim 3, characterized in that, The fusion upsampling module includes multiple fusion upsampling sub-modules. Each fusion upsampling module includes an upsampling layer, a convolutional layer, and an activation function layer. Each stage of the convolutional sub-module corresponds to one fusion upsampling sub-module. The step of inputting the multiple first sample feature images into the fusion upsampling module and outputting second sample feature images includes: The first sample feature image of the convolutional submodule in the last stage is input into the corresponding first fusion upsampling submodule. The upsampling layer, convolutional layer and activation function layer of the first fusion upsampling submodule perform upsampling operation, convolution operation and activation function mapping operation on the input first sample feature image respectively to obtain the output result of the first fusion upsampling submodule. The output of the previous fusion upsampling submodule and the first sample feature image of the corresponding stage convolution submodule are input into the next fusion upsampling submodule to perform upsampling, convolution and activation function mapping operations, until the last fusion upsampling submodule. The output of the last fusion upsampling submodule is used as the second sample feature image.

5. The method according to claim 2, characterized in that, The low-scale information enhancement module includes a channel connection layer, a convolutional layer, a batch regularization layer, and an activation function layer. The step of inputting the sample depth estimation image and the sample matting image into the low-scale information enhancement module and outputting a third sample feature image includes: The sample depth estimation image and the sample matting image are input into the low-scale information enhancement module. The channel connection layer included in the low-scale information enhancement module connects the feature channels of pixels at the same position in the sample depth estimation image and the sample matting image to obtain a second sample connection feature image. Then, the convolutional layer, batch regularization layer and activation function layer included in the low-scale information enhancement module perform convolution operation, batch regularization operation and activation function mapping operation on the second sample connection feature image to obtain the third sample feature image.

6. The method according to claim 2, characterized in that, The subject depth prediction module includes a channel connection layer, a convolutional layer, and an activation function layer. The step of inputting the second sample feature image and the third sample feature image into the subject depth prediction module and outputting a fourth sample feature image includes: The second sample feature image and the third sample feature image are input into the subject depth prediction module. The channel connection layer included in the subject depth prediction module connects the feature channels of pixels at the same position in the second sample feature image and the third sample feature image to obtain the third sample connection feature image. Then, the convolutional layer and activation function layer included in the subject depth prediction module perform convolution operation and activation function mapping operation on the third sample connection feature image to obtain the fourth sample feature image.

7. The method according to claim 2, characterized in that, The step of generating the subject depth prediction image based on the fourth sample feature image and the sample matted image includes: The pixel value of each pixel in the fourth sample feature image is multiplied by the pixel value of the pixel at the same position in the sample matted image to obtain the subject depth prediction image.

8. The method according to claim 2, characterized in that, The main body depth enhancement module includes a channel connection layer, a convolutional layer, a batch regularization layer, and an activation function layer. The step of inputting the second sample feature image, the third sample feature image, and the fourth sample feature image into the main body depth enhancement module and outputting a fifth sample feature image includes: The second, third, and fourth sample feature images are input into the subject depth enhancement module. The channel connection layer of the subject depth enhancement module connects the feature channels of pixels at the same position in the second, third, and fourth sample feature images to obtain the fourth sample connection feature image. Then, the convolutional layer, batch regularization layer, and activation function layer of the subject depth enhancement module perform convolution operations, batch regularization operations, and activation function mapping operations on the fourth sample connection feature image to obtain the fifth sample feature image.

9. The method according to claim 2, characterized in that, The depth prediction module includes an activation function layer and a convolutional layer. The step of inputting the fifth sample feature image into the depth prediction module and outputting the depth prediction image includes: The fifth sample feature image is input into the depth prediction module, and the activation function layer and convolution layer included in the depth prediction module perform activation function mapping operation and convolution operation on the fifth sample feature image respectively to obtain the depth prediction image.

10. The method according to claim 1, characterized in that, The overall objective loss function includes a first objective loss function and a second objective loss function. The first objective loss function is used to calculate the prediction loss for the subject depth image, and the second objective loss function is used to calculate the prediction loss for the depth-optimized image. The step of training the depth optimization model based on the subject depth prediction image, the sample matted image, the depth label image, the depth prediction image, and the overall objective loss function to obtain a trained depth optimization model includes: Multiply the pixel value of each pixel in the depth label image with the pixel value of the pixel at the same position in the sample cutout image to obtain the main body depth label image; The predicted depth image of the subject and the labeled depth image of the subject are input into the first target loss function to obtain the value of the first target loss function; The depth prediction image and the depth label image are input into the second target loss function to obtain the value of the second target loss function; Calculate the total target loss function value based on the first target loss function value and the second target loss function value; Based on the target loss function value, the model parameters of the deep optimization model are adjusted to obtain the trained deep optimization model.

11. An optimization method for depth estimation images, characterized in that, The method applies the trained deep optimization model trained according to any one of claims 1 to 10, and the method includes: Acquire a monocular RGB image and its corresponding depth estimation image and matted image; The monocular RGB image, the depth estimation image, and the matted image are input into the trained depth optimization model, and the depth optimization image is output.

12. A training device for a deep optimization model, characterized in that, The device includes: The acquisition module is used to acquire a sample monocular RGB image, a sample depth estimation image, and a sample matted image. The sample depth estimation image is an image obtained by performing depth estimation on the sample monocular RGB image. The sample matted image is an image obtained by matting out the foreground subject in the sample monocular RGB image. The sample monocular RGB image has a depth label image. The input / output module is used to input the sample monocular RGB image, the sample depth estimation image, and the sample matting image into the depth optimization model, and output the subject depth prediction image and the depth prediction image. The training module is used to train the depth optimization model based on the subject depth prediction image, the sample matted image, the depth label image, the depth prediction image and the total objective loss function to obtain a trained depth optimization model. The trained depth optimization model is used to optimize the depth estimation image based on a monocular RGB image and its corresponding depth estimation image and matted image.

13. An optimization apparatus for depth estimation images, characterized in that, The device is equipped with the trained deep optimization model trained according to any one of claims 1 to 10, and the device comprises: The acquisition module is used to acquire a monocular RGB image and its corresponding depth estimation image and matted image; The input / output module is used to input the monocular RGB image, the depth estimation image, and the matted image into the trained depth optimization model and output the depth optimization image.

14. An electronic device, characterized in that, It includes a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is called and executed by the processor to implement the training method of the depth optimization model as described in any one of claims 1 to 10, or the optimization method of the depth estimation image as described in claim 11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, is capable of implementing the training method for the depth optimization model as described in any one of claims 1 to 10, or the optimization method for the depth estimation image as described in claim 11.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, can implement the training method of the depth optimization model as described in any one of claims 1 to 10, or the optimization method of the depth estimation image as described in claim 11.

Images