Depth estimation model training method and device, electronic equipment and storage medium
By continuously updating the reference model and the reference depth image of the second image during training, the accuracy and generalization ability of the depth estimation model are ensured, which solves the problem of insufficient generalization and robustness in the existing technology and achieves better zero-shot depth estimation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING XIAOMI MOBILE SOFTWARE CO LTD
- Filing Date
- 2024-10-29
- Publication Date
- 2026-05-01
AI Technical Summary
Existing depth estimation models have poor generalization and robustness in real-world environments, resulting in poor recognition of some objects and missing content recognition in some scenarios.
A first initial model is trained by acquiring a first image and its reference depth image, a reference model is determined, and a reference depth image of the second image is obtained based on the second image and the reference model. Then, a second initial model is trained, and finally, the target depth estimation model is determined when the conditions are met, ensuring that the depth label of the second image used in each round of training is obtained by the reference model with the best performance.
It improves the zero-sample depth estimation capability and generalization ability of the depth estimation model, reduces the generalization error of the model, and enhances robustness.
Smart Images
Figure CN121962787A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for training a depth estimation model. Background Technology
[0002] Depth estimation is a widely used technique in fields such as robotics, autonomous driving, and virtual reality. It involves using a depth estimation model to predict the depth of an image, essentially predicting the monocular depth information. However, constructing a dataset with tens of millions of depth labels is difficult. Influenced by the amount of training data, depth estimation models exhibit poor generalization and robustness in real-world applications, leading to limitations in zero-shot scenarios, such as poor object recognition and missing content recognition in certain situations. Summary of the Invention
[0003] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, electronic device and storage medium for training a depth estimation model.
[0004] According to a first aspect of the present disclosure, a method for training a depth estimation model is provided, the method comprising:
[0005] Acquire a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image;
[0006] A first initial model is trained based on the first image and a reference depth image of the first image to obtain a first target model;
[0007] Based on the first target model obtained during training, a reference model is determined;
[0008] Based on the second image and the reference model, a reference depth image of the second image is obtained;
[0009] A second initial model is trained based on the first image, the reference depth image of the first image, the second image, and the reference depth image of the second image to obtain a second target model;
[0010] In response to the failure to meet the training termination condition, the process switches to training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model;
[0011] In response to the completion of the training, a target depth estimation model is determined based on the second target model, wherein the target depth estimation model is used to obtain the depth image of the input image.
[0012] In an exemplary embodiment, training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model includes:
[0013] The first image is input into the first initial model to obtain the predicted depth image of the first image;
[0014] Based on the reference depth image and the predicted depth image of the first image, the model parameters of the first initial model are adjusted to obtain the first target model.
[0015] In an exemplary embodiment, the step of inputting the first image into the first initial model to obtain a predicted depth image of the first image includes:
[0016] First information is obtained by extracting features from the first image using a first initial encoder.
[0017] The first information is compressed using the first initial module to obtain the second information;
[0018] The predicted depth image of the first image is obtained by performing feature mapping on the second information using a first initial decoder and a pre-trained second encoder.
[0019] In an exemplary embodiment, before extracting features from the first image using the first initial encoder to obtain the first information, the method further includes:
[0020] The first image is image-recognized by a preset recognition module to obtain summary information of the first image;
[0021] First text information is obtained by extracting text features from the summary information of the first image using a preset language module;
[0022] The first text information is used as the key value of the self-attention layer of the first initial encoder.
[0023] In one exemplary embodiment, the first initial decoder includes a convolutional layer and an upsampling layer, wherein the upsampling layer includes a self-attention layer.
[0024] In an exemplary embodiment, determining the reference model based on the first target model obtained during training includes:
[0025] Based on the model parameters corresponding to the first target model, the model parameters of the reference model are determined, wherein the model structure of the reference model is the same as that of the first target model.
[0026] In an exemplary embodiment, training a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model includes:
[0027] The first image and the second image are input into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image;
[0028] Based on the reference depth image and the predicted depth image of the first image, and the reference depth image and the predicted depth image of the second image, the model parameters of the second initial model are adjusted to obtain the second target model.
[0029] In an exemplary embodiment, the step of inputting the first image and the second image into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image includes:
[0030] The first image is processed by a first initial encoder to extract features and obtain first information, and the second image is processed by a second initial encoder to extract features and obtain second information.
[0031] The first initialization module performs feature compression on the first information to obtain the third information, and the second initialization module performs feature compression on the second information to obtain the fourth information;
[0032] The first image's predicted depth image is obtained by performing feature mapping on the third information using a first initial decoder and a pre-trained second encoder, and the second image's predicted depth image is obtained by performing feature mapping on the fourth information using the first initial decoder and the pre-trained second encoder.
[0033] In an exemplary embodiment, before extracting features from the first image using a first initial encoder to obtain first information, and extracting features from the second image using a second initial encoder to obtain second information, the method further includes:
[0034] The first image is image-recognized using a preset recognition module to obtain summary information of the first image, and the second image is image-recognized using the same preset recognition module to obtain summary information of the second image.
[0035] The first image's summary information is extracted using a preset language module to obtain first text information, and the second image's summary information is extracted using the same preset language module to obtain second text information.
[0036] The first text information is used as the key value of the self-attention layer of the first initial encoder, and the second text information is used as the key value of the self-attention layer of the second initial encoder.
[0037] In one exemplary embodiment, adjusting the model parameters of the second initial model includes:
[0038] Keep the parameters of the second initial encoder unchanged, and simultaneously adjust the parameters of the first initial module and the second initial module so that the adjusted parameters of the first initial module and the second initial module are the same.
[0039] In an exemplary embodiment, training a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model includes:
[0040] The second image is subjected to image processing to obtain a processed second image, wherein the image processing includes at least one of color distortion processing and spatial distortion processing;
[0041] A second initial model is trained based on the first image and its labeled depth image, as well as the processed second image and its reference depth image, to obtain the second target model.
[0042] According to a second aspect of the present disclosure, a depth estimation model training apparatus is provided, the apparatus comprising:
[0043] The acquisition module is configured to acquire a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image;
[0044] The first training module is configured to train a first initial model based on the first image and a reference depth image of the first image to obtain a first target model;
[0045] The determination module is configured to determine a reference model based on the first target model obtained during training.
[0046] The first processing module is configured to obtain a reference depth image of the second image based on the second image and the reference model;
[0047] The second training module is configured to train a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model.
[0048] The second processing module is configured to, in response to the failure to meet the training termination condition, switch to training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model; and in response to meeting the training termination condition, determine a target depth estimation model based on the second target model, wherein the target depth estimation model is used to obtain a depth image of the input image.
[0049] According to a third aspect of the present disclosure, an electronic device is provided, comprising:
[0050] processor;
[0051] Memory used to store processor-executable instructions;
[0052] The processor is configured to perform the method described in the first aspect of the embodiments of this disclosure.
[0053] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in the first aspect of the present disclosure.
[0054] The method described above has the following advantages: by updating the reference model in each training round, and thus updating the reference depth image of the second image, this method can ensure that the depth labels of the second image used in each training round are obtained by the best-performing reference model. Therefore, it can ensure the accuracy of the reference depth image of the second image, thereby ensuring the zero-sample depth estimation capability of the depth estimation model obtained from the reference depth image of the second image. Furthermore, since the second image used in training is arbitrary, it can reduce the generalization error of the model and improve the generalization ability and robustness of the model.
[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0056] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0057] Figure 1 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment;
[0058] Figure 2 This is a flowchart illustrating the process of obtaining a first target model according to an exemplary embodiment;
[0059] Figure 3This is a schematic diagram of the model structure of a first initial model according to an exemplary embodiment;
[0060] Figure 4 This is a schematic diagram of the model structure of a first initial model according to an exemplary embodiment;
[0061] Figure 5 This is a flowchart illustrating the process of obtaining a second target model according to an exemplary embodiment;
[0062] Figure 6 This is a schematic diagram of the model structure of a second initial model according to an exemplary embodiment;
[0063] Figure 7 This is a schematic diagram of the model structure of a second initial model according to an exemplary embodiment;
[0064] Figure 8 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment;
[0065] Figure 9 This is a block diagram illustrating a depth estimation model training apparatus according to an exemplary embodiment;
[0066] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0068] In related technologies, depth estimation models can be trained using large training datasets with depth labels. These labeled datasets represent training images with accurate depth information. However, since depth information is primarily obtained through sensors or stereo matching, the acquisition process is costly, time-consuming, and has limited applicability. Therefore, the limited size of the training dataset results in poor generalization and robustness of the depth estimation model. While introducing more prior knowledge or a better objective function can enhance the robustness of the depth estimation model, this approach is not suitable for zero-shot depth estimation in real-time environments. Therefore, using limited training images or prior knowledge to train depth estimation models in these technologies cannot fully meet the needs of real-world applications, leading to poor generalization and robustness in practical applications. This results in problems such as poor object recognition and missing content recognition in certain scenarios.
[0069] In an exemplary embodiment of this disclosure, to overcome the problem of poor generalization and robustness of depth estimation models trained in related technologies, a depth estimation model training method is provided, comprising: acquiring a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image; training a first initial model based on the first image and the reference depth image of the first image to obtain a first target model; determining a reference model based on the first target model obtained during training; obtaining a reference depth image of the second image based on the second image and the reference model; training a second initial model based on the first image, the reference depth image of the first image, the second image, and the reference depth image of the second image to obtain a second target model; in response to not meeting the training termination condition, switching to training the first initial model based on the first image and the reference depth image of the first image to obtain the first target model; and in response to meeting the training termination condition, determining a target depth estimation model based on the second target model, wherein the target depth estimation model is used to acquire the depth image of the input image. This method can ensure the depth estimation capability of the obtained depth estimation model, and since the second image used in training is arbitrary, it can reduce the generalization error of the model and improve the generalization capability and robustness of the model.
[0070] In an exemplary embodiment of this disclosure, a method for training a depth estimation model is provided, which is applied to electronic devices, including smartphones, tablets, personal computers, and other devices. Figure 1 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment, such as... Figure 1 As shown, the process includes the following steps S101-S107:
[0071] Step S101: Obtain a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image.
[0072] Depth images are used to represent the distance information of each pixel in an image to the camera; they can also be called the depth information of each pixel in the image. The reference depth image of the first image is obtained through methods such as sensors or stereo matching, and can accurately represent the depth information of each pixel in the first image. The first image and its reference depth image can be obtained from existing public datasets, such as the KITTI depth dataset. The first image and its reference depth image correspond to each other, with the same number of images; for example, 3 million first images and their corresponding 3 million reference depth images can be obtained from a public dataset. The reference depth image of the first image can also be called the depth label of the first image. The second image differs from the first image and can be any image, such as images captured by a mobile phone camera, images captured by IoT devices, network images, or any other readily available image. The number of second images can be set according to actual needs; the more second images, the better the generalization ability of the trained depth estimation model. For example, the number of second images can be twice the number of first images.
[0073] In some implementations, the images in the training dataset are normalized images. After obtaining the initial image, the pixel value of each pixel in the initial image is transformed by d = 1 / t, where d represents the transformed pixel value and t represents the pixel value before transformation. This can normalize the pixel value of each pixel in the initial image to the range of 0 to 1, so that the final depth estimation model is suitable for small mobile terminals.
[0074] Step S102: Train a first initial model based on the first image and a reference depth image of the first image to obtain a first target model.
[0075] The first initial model is a monocular depth estimation (MDE) model used to predict the depth image of the input image. The model structure of the first initial model can be any encoder-decoder structure. The initial model parameters of the first initial model are the parameters initialized. The model parameters used in each subsequent training round are the model parameters obtained in the previous training round. In each training round, the first image is used as input, and the reference depth image of the first image is used as the depth label of the first image. The first initial model is trained through supervised learning to obtain the first target model after each training round. Therefore, the model structure of the first target model is the same as that of the first initial model, and the model parameters of the first target model are the model parameters trained using the first image and the reference depth image of the first image.
[0076] Step S103: Determine the reference model based on the first target model obtained during training.
[0077] The first target model obtained during training can be a first target model obtained in one training round, or multiple first target models obtained over multiple training rounds. It can be a first target model obtained in a previous training round, or a first target model obtained in the current training round, or a combination of both. The specific number of first target models can be set according to actual needs. In one example, the reference model for each training round is determined based on the first target models obtained in five consecutive training rounds. Therefore, the reference model for the sixth training round is determined based on the first target models obtained in rounds 1 to 5, the reference model for the seventh training round is determined based on the first target models obtained in rounds 2 to 6, and so on. When determining the reference model based on the first target model, the best-performing first target model can be selected from multiple first target models as the reference model. Optionally, the first target model with the smallest loss value among multiple first target models can be selected as the reference model.
[0078] In some implementations, model parameters of a reference model are determined based on the model parameters corresponding to the first target model, wherein the model structure of the reference model is the same as that of the first target model. Optionally, each first target model corresponds to a set of model parameters, and the average or median value of the multiple sets of model parameters corresponding to multiple first target models is used as the model parameters of the reference model, or the model parameters of the first target model with the smallest loss value among multiple first target models are used as the model parameters of the reference model. Since the reference model is updated in each round of training, the performance of the reference model in each round of training can be ensured.
[0079] Step S104: Based on the second image and the reference model, obtain the reference depth image of the second image.
[0080] The second image is input into the first target model, and the predicted reference depth image of the second image is output. Since the reference model is determined by the best-performing first target model, or by multiple first target models, the performance of the reference model can be ensured. The depth image of the second image predicted by the reference model is recorded as the reference depth image of the second image, so that it can be used as the depth pseudo-label of the second image, thus ensuring the accuracy of the depth label of the second image.
[0081] Step S105: Train a second initial model based on the first image, the reference depth image of the first image, the second image, and the reference depth image of the second image to obtain a second target model.
[0082] The second initial model is used to predict the depth image of the input image. The model structure of the second initial model can be any encoder-decoder structure. The initial model parameters of the second initial model are the initialized model parameters. The model parameters of the second initial model used in each subsequent training round are the model parameters obtained in the previous training round. In each training round, the first image and the second image are simultaneously used as input. The reference depth image of the first image is used as the depth label of the first image, and the reference depth image of the second image is used as the depth label of the second image. The second initial model is trained through supervised learning to obtain the second target model after each training round. Therefore, the model structure of the second target model and the second initial model is the same, and the model parameters of the second target model are the trained model parameters. Optionally, the second initial model includes two encoders, used for inputting the first image and the second image respectively.
[0083] Step S106: In response to the failure to meet the training termination condition, proceed to train the first initial model based on the first image and the reference depth image of the first image to obtain the first target model.
[0084] Whether the training termination condition is met indicates whether the training iteration round has been completed. The number of training iteration rounds is a preset value. Steps S102-S105 are executed in each training round. After obtaining the second target model in each training round, it is determined whether the number of training iteration rounds has reached the preset value. If the number of training iteration rounds is less than the preset value, it is determined that the training termination condition is not met. At this time, the next round of iteration training begins, and steps S102-S105 are executed again.
[0085] Step S107: In response to the completion of training, a target depth estimation model is determined based on the second target model, wherein the target depth estimation model is used to obtain the depth image of the input image.
[0086] After obtaining the second target model in each training round, it is determined whether the number of training iterations has reached a preset value. If the number of training iterations equals the preset value, the training termination condition is met, and training ends. Based on the second target model obtained in the last round, the target depth estimation model is determined. Optionally, since only one image needs to be input when using the depth estimation model to estimate the depth of any image, the part of the second target model obtained in the last training round that has the same model structure as the first target model is determined as the target depth estimation model. The model parameters of this part are still the model parameters of that part in the second target model. For example, if the second target model includes a first input branch and a second input branch, and the structure of the first input branch is the same as the model structure of the first target model, then the first input branch in the second target model is used as the target depth estimation model. The target depth estimation model can be applied to any scenario that requires obtaining image depth information, such as 3D reconstruction, pose estimation, intelligent vehicle detection and tracking of other vehicles and pedestrians, virtual makeup, etc. In practical applications, the image to be processed is input into the target depth estimation model, and the depth image of the image to be processed is output. It should be noted that if the training data is a normalized image, the image to be processed also needs to be normalized before being input into the target depth estimation model when applying the target depth estimation model.
[0087] In this embodiment, in each training round, a first initial model is trained based on the first image and a reference depth image of the first image to obtain a first target model. Based on the first target model obtained during training, a reference model is determined. A depth label is added to the second image without a depth label using the reference model, that is, the second image is input into the reference model to obtain a reference depth image of the second image. Then, a second initial model is trained based on the first image and the reference depth image of the first image, as well as the reference depth images of the second image and the second image to obtain a trained second target model. This process is repeated iteratively. At the end of the iterative training, a target depth estimation model is determined based on the second target model obtained in the last training round. In this method, since the reference model is updated in each training round, the reference depth image of the second image is also updated in each training round. Thus, the depth label of the second image used in each training round is obtained from the best-performing reference model, which ensures the accuracy of the reference depth image of the second image. This, in turn, ensures the zero-sample depth estimation capability of the depth estimation model obtained from the reference depth image of the second image. Furthermore, since the second image used in training is arbitrary, it can greatly improve the quality and scope of training data, thereby reducing the generalization error of the model and improving the generalization ability and robustness of the model.
[0088] In some embodiments, Figure 2 This is a flowchart illustrating the process of obtaining a first target model according to an exemplary embodiment, such as... Figure 2 As shown, in each round of training, the first target model is obtained through steps S201-S202:
[0089] Step S201: Input the first image into the first initial model to obtain the predicted depth image of the first image.
[0090] The first initial model has an encoder-decoder structure and the model parameters are initialization parameters. The first image is input into the first initial model. The encoder extracts the depth features of the first image, and the decoder performs feature mapping on the extracted depth features. The depth image of the first image can be predicted by the first initial model, which is denoted as the predicted depth image of the first image.
[0091] In some implementations, the predicted depth image of the first image is obtained through the following steps:
[0092] First information is obtained by extracting features from the first image using a first initial encoder;
[0093] The first information is compressed using the first initial module to obtain the second information;
[0094] The predicted depth image of the first image is obtained by performing feature mapping on the second information using a first initial decoder and a pre-trained second encoder.
[0095] Figure 3 This is a schematic diagram of the model structure of a first initial model according to an exemplary embodiment, such as... Figure 3As shown, the first initial model includes a first initial encoder (Encoder-L), a first initial module (Compression Func), a first initial decoder (Decoder), and a pre-trained second decoder (Pre-Decoder). The first initial encoder can employ any convolutional structure, such as the encoder part of a Variational Auto-Encoder (VAE) model. It extracts depth features from the first image to obtain the first information of the first image. The first initial module includes a max-pooling layer and a linear layer for feature compression. It compresses the first information to obtain the compressed second information. The first initial decoder can employ any deconvolutional or upsampling structure, such as the decoder part of a VAE model. The pre-trained second encoder can have the same or different structure as the first initial encoder; for example, it can also use the decoder part of a VAE model. The first initial decoder and the pre-trained second encoder perform feature mapping on the second information to obtain the predicted depth image of the first image. The parameters of the first initial encoder, the first initial module, and the first initial decoder are all randomly initialized. The base learning rate of the pre-trained second encoder is set to 0.0005 for example, and the learning rate of the first initial decoder is set to 10 for example. The AdamW optimizer is used to decay the learning rate through a linear program.
[0096] In some implementations, the first initial decoder includes convolutional layers and upsampling layers, with the upsampling layers including self-attention layers. The first initial decoder uses the decoder portion of a VAE model, replacing the deconvolutional layers with convolutional and upsampling layers to adapt to the model computation on the edge GPU. Additionally, a self-attention layer is added after each upsampling layer to improve the modeling of local depth features, for example, by adding a 4-head self-attention layer.
[0097] In some implementations, before extracting features from the first image using a first initial encoder to obtain the first information, the following steps are also included:
[0098] The first image is image-recognized by a preset recognition module to obtain summary information of the first image;
[0099] The first text information is obtained by extracting text features from the summary information of the first image using a preset language module;
[0100] The first text information is used as the key value of the self-attention layer of the first initial encoder.
[0101] Figure 4 This is a schematic diagram of the model structure of a first initial model according to an exemplary embodiment, such as... Figure 4 As shown, the first initial model includes a first initial encoder (Encoder-L), a first initial module (Compression Func), a first initial decoder (Decoder), and a pre-trained second decoder (Pre-Decoder), as well as a preset recognition module and a preset language module. The preset recognition module can adopt the modality recognition capability part of Large Language Models (LLM), such as the MiLM-6B model. The preset recognition module performs image recognition on the first image to obtain the summary information of the first image. Optionally, the first image is first initially recognized through question guidance, such as the guiding text: "Please describe the content of the picture in one sentence", "Describe the relationship between objects in the picture in 3-5 words", "Please describe the hierarchical relationship of the content in the picture below", "Please describe the objects from near to far, and the positional relationship between them", etc. Then the image is summarized, for example, the guiding text is "Summarize the input content. The summary content mainly includes the hierarchical and deep hierarchical relationship of the described content, described in 30-50 words, not exceeding 50 words". The preset language module is used to extract text features from the summary information. For example, the preset language module uses the T5-small model to extract text features from the summary information of the first image to obtain the first text information. If the first initial encoder includes a self-attention layer, the text features of the first image, i.e., the first text information, are used as the key value of the self-attention layer of the first initial encoder.
[0102] In this embodiment, by introducing textual feature information from the first image, richer semantic information is used as auxiliary supervision for the depth estimation task, which can improve the model's ability to understand image depth information.
[0103] Step S202: Based on the reference depth image and the predicted depth image of the first image, adjust the model parameters of the first initial model to obtain the first target model.
[0104] The difference between the predicted depth image and the reference depth image of the first image is calculated using a preset loss function. The model parameters of the initial model are adjusted, and the predicted depth image of the first image is re-predicted. If the difference between the predicted depth image and the reference depth image decreases, the model parameters of the initial model are adjusted further. If the difference increases, the model parameters of the initial model are adjusted in reverse, and so on, until the difference between the predicted depth image and the reference depth image of the first image reaches its minimum. At this point, the first target model is obtained. The model parameters of the initial model when the difference between the predicted depth image and the reference depth image of the first image is minimized are the model parameters of the first target model. The preset loss function can be any loss function, such as mean squared error loss or cross-entropy loss. The difference between the predicted depth image and the reference depth image of the first image represents the difference between the pixel value of each pixel in the predicted depth image and the pixel value of the corresponding pixel in the reference depth image. It should be noted that the model training process does not change the model structure; the model structure of the first target model is the same as that of the first initial model.
[0105] In some embodiments, Figure 5 This is a flowchart illustrating the process of obtaining a second target model according to an exemplary embodiment, such as... Figure 5 As shown, in each round of training, the second target model is obtained through steps S501-S502:
[0106] Step S501: Input the first image and the second image into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image.
[0107] The second initial model has an encoder-decoder structure, which includes two encoders. The model parameters are initialization parameters. The first image and the second image are respectively input into one encoder. The encoder extracts the depth features of the image, and the decoder performs feature mapping on the extracted depth features. The second initial model can then predict the depth images of the first image and the second image, which are denoted as the predicted depth images of the first image and the second image, respectively.
[0108] In some implementations, the predicted depth image of the first image and the predicted depth image of the second image are obtained through the following steps:
[0109] First information is obtained by extracting features from the first image using a first initial encoder, and second information is obtained by extracting features from the second image using a second initial encoder.
[0110] The first initial module performs feature compression on the first information to obtain the third information, and the second initial module performs feature compression on the second information to obtain the fourth information.
[0111] The first image's predicted depth image is obtained by performing feature mapping on the third information using the first initial decoder and the pre-trained second encoder, and the second image's predicted depth image is obtained by performing feature mapping on the fourth information using the first initial decoder and the pre-trained second encoder.
[0112] Figure 6 This is a schematic diagram of the model structure of the second initial model according to an exemplary embodiment, such as... Figure 6 As shown, the second initial model includes a first initial encoder (Encoder-L), a second initial encoder (Encoder-U), a first initial module (Compression Func), a second initial module (Compression Func), a first initial decoder (Decoder), and a pre-trained second decoder (Pre-Decoder). The first initial encoder, first initial module, first initial decoder, and pre-trained second decoder are all modules within the first initial model. See [link to documentation]. Figure 3 The relevant descriptions will not be repeated here. The structure of the second initial encoder is the same as that of the first initial encoder, and the structure of the second initial module is the same as that of the first initial module. The first initial encoder extracts features from the first image to obtain first information, and then the first initial module compresses the features of the first information to obtain third information. At the same time, the second initial encoder extracts features from the second image to obtain second information, and then the second initial module compresses the features of the second information to obtain fourth information. After obtaining the third and fourth information, the third information is first feature-mapped using the first initial decoder and the pre-trained second encoder to obtain the predicted depth image of the first image, and then the fourth information is feature-mapped using the first initial decoder and the pre-trained second encoder to obtain the predicted depth image of the second image. The order of feature mapping for the third and fourth information is not limited.
[0113] In some implementations, before extracting features from the first image using a first initial encoder to obtain first information, and extracting features from the second image using a second initial encoder to obtain second information, the following steps are further included:
[0114] The first image is image recognized by a preset recognition module to obtain summary information of the first image, and the second image is image recognized by the preset recognition module to obtain summary information of the second image.
[0115] The first text information is obtained by extracting text features from the summary information of the first image using a preset language module, and the second text information is obtained by extracting text features from the summary information of the second image using the same preset language module.
[0116] The first text information is used as the key value of the self-attention layer of the first initial encoder, and the second text information is used as the key value of the self-attention layer of the second initial encoder.
[0117] Figure 7 This is a schematic diagram of the model structure of the second initial model according to an exemplary embodiment, such as... Figure 7 As shown, the second initial model includes, in addition to the first initial encoder (Encoder-L), the second initial encoder (Encoder-U), the first initial module (Compression Func), the second initial module (Compression Func), the first initial decoder (Decoder), and the pre-trained second decoder (Pre-Decoder), a preset recognition module and a preset language module. The preset recognition module and preset language module in the second initial model are... Figure 4 The preset recognition module and preset language module in the first initial model shown are the same, and will not be described again here. If the first initial encoder and the second initial encoder include a self-attention layer, the text features of the first image, i.e., the first text information, are used as the key values of the self-attention layer of the first initial encoder, and the text features of the second image, i.e., the second text information, are used as the key values of the self-attention layer of the second initial encoder. The key values of the self-attention layer can also be represented as Key values.
[0118] Step S502: Based on the reference depth image and predicted depth image of the first image, and the reference depth image and predicted depth image of the second image, adjust the model parameters of the second initial model to obtain the second target model.
[0119] The difference between the predicted depth image and the reference depth image of the first image, and the difference between the predicted depth image and the reference depth image of the second image, are calculated using a preset loss function. The model parameters of the second initial model are adjusted, and the predicted depth images of the first and second images are re-predicted. If the value of the loss function decreases, the model parameters of the second initial model are adjusted further; if the value of the loss function increases, the model parameters of the second initial model are adjusted in reverse, and so on, until the value of the loss function reaches its minimum. At this point, the second target model is obtained. The model parameters of the second initial model when the value of the loss function is minimum are the model parameters of the second target model. The preset loss function can be any loss function, such as the affine invariant loss shown in the following formula:
[0120]
[0121] Where L1 represents the affine invariant loss, W represents the width of the image, H represents the length of the image, and y i y' represents the pixel value of the i-th pixel in the reference depth image. i α(y) represents the pixel value of the i-th pixel in the predicted depth image. i ,y' i As shown below:
[0122]
[0123] Where Mean(y) represents the average pixel value of all pixels in the reference depth image, and Mean(y') represents the average pixel value of all pixels in the predicted depth image.
[0124] In some implementations, when adjusting the model parameters of the second initial model, the parameters of the second initial encoder are kept unchanged, and the parameters of the first initial module and the second initial module are adjusted simultaneously, so that the adjusted parameters of the first initial module and the second initial module are the same.
[0125] The parameters of the second initial encoder are randomly initialized and then frozen. This ensures that the parameters of the second initial encoder remain unchanged during training, thus preventing it from participating in training. Furthermore, the parameters of both the first and second initial modules are adjusted simultaneously to ensure that the adjusted parameters of the first and second initial modules are identical, guaranteeing that the parameters of both modules remain the same throughout training. This parameter adjustment method ensures that, except for modules in the second target model with the same model structure as the first target model, they do not participate in training. It also ensures that the parameters of the second target model with the same model structure as the first target model—that is, the model parameters of the target depth estimation model—are obtained by training with the depth features of the second image incorporated, thereby improving the generalization ability of the target depth estimation model.
[0126] In an exemplary embodiment of this disclosure, a method for training a depth estimation model is provided. Figure 8 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment, such as... Figure 8 As shown, the process includes the following steps S801-S806:
[0127] Step S801: Obtain a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image.
[0128] For a detailed implementation of step S801, please refer to step S101, which will not be repeated here.
[0129] Step S802: Train a first initial model based on the first image and a reference depth image of the first image to obtain a first target model.
[0130] For the specific implementation of step S802, please refer to step S102 or steps S201-S202, which will not be repeated here.
[0131] Step S803: Determine the reference model based on the first target model obtained during training.
[0132] For a detailed implementation of step S803, please refer to step S103, which will not be repeated here.
[0133] Step S804: Based on the second image and the reference model, obtain the reference depth image of the second image.
[0134] For a detailed implementation of step S804, please refer to step S104, which will not be repeated here.
[0135] Step S805: Perform image processing on the second image to obtain a processed second image. The image processing includes at least one of color distortion processing and spatial distortion processing.
[0136] Color distortion processing, such as color dithering and Gaussian blurring, and spatial distortion processing, such as image rotation and magnification, are used to process a second image without depth labels. This processing involves color distortion, spatial distortion, or both simultaneously, to obtain a processed second image. By adding distortion perturbations to the second image, the model is forced to actively seek additional visual knowledge to obtain invariant depth feature representations from the second image, thus improving the model's robustness.
[0137] Step S806: Train a second initial model based on the first image and its label depth image, as well as the processed second image and its reference depth image, to obtain a second target model.
[0138] For a detailed implementation of step S806, please refer to step S105 or steps S501-S502, which will not be repeated here.
[0139] In step S807, in response to the failure to meet the training termination condition, the process proceeds to train the first initial model based on the first image and a reference depth image of the first image to obtain the first target model.
[0140] For a detailed implementation of step S807, please refer to step S106, which will not be repeated here.
[0141] Step S808: In response to the completion of training, a target depth estimation model is determined based on the second target model, wherein the target depth estimation model is used to obtain the depth image of the input image.
[0142] The specific implementation of step S808 is described in step S107, and will not be repeated here. In some implementations, since the first target model is the model trained from the first initial model and the second target model is the model trained from the second initial model, the model structure of the first target model is also as follows. Figure 4 The model structure shown is the same for the second target model. Figure 7 The model structure shown will, in response to the satisfaction of the training termination condition, be as follows: Figure 7 The model structure shown is similar to... Figure 4 The part of the model structure that is the same as the model structure shown is determined as the model structure of the target depth estimation model. The model parameters of this part in the second target model remain unchanged, that is, the model parameters of this part are determined as the model parameters of the target depth estimation model.
[0143] In an exemplary embodiment of this disclosure, a depth estimation model training apparatus is provided. Figure 9 This is a block diagram illustrating a depth estimation model training apparatus according to an exemplary embodiment, such as... Figure 9 As shown, it includes:
[0144] The acquisition module 901 is configured to acquire a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image;
[0145] The first training module 902 is configured to train a first initial model based on the first image and a reference depth image of the first image to obtain a first target model;
[0146] Module 903 is configured to determine a reference model based on the first target model obtained during training.
[0147] The first processing module 904 is configured to obtain a reference depth image of the second image based on the second image and the reference model;
[0148] The second training module 905 is configured to train a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image, to obtain a second target model;
[0149] The second processing module 906 is configured to, in response to the failure to meet the training termination condition, switch to training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model; and in response to the fulfillment of the training termination condition, determine a target depth estimation model based on the second target model, wherein the target depth estimation model is used to obtain the depth image of the input image.
[0150] In one exemplary embodiment, the first training module 902 is further configured to:
[0151] Input the first image into the first initial model to obtain the predicted depth image of the first image;
[0152] Based on the reference depth image and the predicted depth image of the first image, the model parameters of the first initial model are adjusted to obtain the first target model.
[0153] In one exemplary embodiment, the first training module 902 is further configured to:
[0154] First information is obtained by extracting features from the first image using a first initial encoder;
[0155] The first information is compressed using the first initial module to obtain the second information;
[0156] The predicted depth image of the first image is obtained by performing feature mapping on the second information using a first initial decoder and a pre-trained second encoder.
[0157] In one exemplary embodiment, the first training module 902 is further configured to:
[0158] The first image is image-recognized by a preset recognition module to obtain summary information of the first image;
[0159] The first text information is obtained by extracting text features from the summary information of the first image using a preset language module;
[0160] The first text information is used as the key value of the self-attention layer of the first initial encoder.
[0161] In one exemplary embodiment, the first initial decoder includes a convolutional layer and an upsampling layer, and the upsampling layer includes a self-attention layer.
[0162] In one exemplary embodiment, the determining module 903 is further configured to:
[0163] Based on the model parameters corresponding to the first target model, the model parameters of the reference model are determined, wherein the model structure of the reference model is the same as that of the first target model.
[0164] In one exemplary embodiment, the second training module 905 is further configured to:
[0165] The first image and the second image are input into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image;
[0166] Based on the reference depth image and predicted depth image of the first image, and the reference depth image and predicted depth image of the second image, the model parameters of the second initial model are adjusted to obtain the second target model.
[0167] In one exemplary embodiment, the second training module 905 is further configured to:
[0168] First information is obtained by extracting features from the first image using a first initial encoder, and second information is obtained by extracting features from the second image using a second initial encoder.
[0169] The first initial module performs feature compression on the first information to obtain the third information, and the second initial module performs feature compression on the second information to obtain the fourth information.
[0170] The first image's predicted depth image is obtained by performing feature mapping on the third information using the first initial decoder and the pre-trained second encoder, and the second image's predicted depth image is obtained by performing feature mapping on the fourth information using the first initial decoder and the pre-trained second encoder.
[0171] In one exemplary embodiment, the second training module 905 is further configured to:
[0172] The first image is image recognized by a preset recognition module to obtain summary information of the first image, and the second image is image recognized by the preset recognition module to obtain summary information of the second image.
[0173] The first text information is obtained by extracting text features from the summary information of the first image using a preset language module, and the second text information is obtained by extracting text features from the summary information of the second image using the same preset language module.
[0174] The first text information is used as the key value of the self-attention layer of the first initial encoder, and the second text information is used as the key value of the self-attention layer of the second initial encoder.
[0175] In one exemplary embodiment, the second training module 905 is further configured to:
[0176] Keep the parameters of the second initial encoder unchanged, and simultaneously adjust the parameters of the first initial module and the second initial module so that the adjusted parameters of the first initial module and the second initial module are the same.
[0177] In one exemplary embodiment, the second training module 905 is further configured to:
[0178] The second image is processed to obtain a processed second image. The image processing includes at least one of color distortion processing and spatial distortion processing.
[0179] A second initial model is trained based on the first image and its labeled depth image, as well as the processed second image and its reference depth image, to obtain the second target model.
[0180] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0181] Figure 10 This is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment.
[0182] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.
[0183] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.
[0184] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of this data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0185] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.
[0186] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0187] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.
[0188] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0189] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.
[0190] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0191] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0192] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0193] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a depth estimation model training method, including any of the methods described above.
[0194] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0195] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A method for training a depth estimation model, characterized in that, The method includes: Acquire a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image; A first initial model is trained based on the first image and a reference depth image of the first image to obtain a first target model; Based on the first target model obtained during training, a reference model is determined; Based on the second image and the reference model, a reference depth image of the second image is obtained; A second initial model is trained based on the first image, the reference depth image of the first image, the second image, and the reference depth image of the second image to obtain a second target model; In response to the failure to meet the training termination condition, the process switches to training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model; In response to the completion of the training, a target depth estimation model is determined based on the second target model, wherein the target depth estimation model is used to obtain the depth image of the input image.
2. The depth estimation model training method according to claim 1, characterized in that, The step of training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model includes: The first image is input into the first initial model to obtain the predicted depth image of the first image; Based on the reference depth image and the predicted depth image of the first image, the model parameters of the first initial model are adjusted to obtain the first target model.
3. The depth estimation model training method according to claim 2, characterized in that, The step of inputting the first image into the first initial model to obtain a predicted depth image of the first image includes: First information is obtained by extracting features from the first image using a first initial encoder. The first information is compressed using the first initial module to obtain the second information; The predicted depth image of the first image is obtained by performing feature mapping on the second information using a first initial decoder and a pre-trained second encoder.
4. The depth estimation model training method according to claim 3, characterized in that, Before extracting features from the first image using the first initial encoder to obtain the first information, the method further includes: The first image is image-recognized by a preset recognition module to obtain summary information of the first image; First text information is obtained by extracting text features from the summary information of the first image using a preset language module; The first text information is used as the key value of the self-attention layer of the first initial encoder.
5. The depth estimation model training method according to claim 3 or 4, characterized in that, The first initial decoder includes convolutional layers and upsampling layers, and the upsampling layers include self-attention layers.
6. The depth estimation model training method according to claim 1, characterized in that, The step of determining a reference model based on the first target model obtained during training includes: Based on the model parameters corresponding to the first target model, the model parameters of the reference model are determined, wherein the model structure of the reference model is the same as that of the first target model.
7. The depth estimation model training method according to claim 1, characterized in that, The step of training a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model includes: The first image and the second image are input into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image; Based on the reference depth image and the predicted depth image of the first image, and the reference depth image and the predicted depth image of the second image, the model parameters of the second initial model are adjusted to obtain the second target model.
8. The depth estimation model training method according to claim 7, characterized in that, The step of inputting the first image and the second image into the second initial model to obtain the predicted depth image of the first image and the predicted depth image of the second image includes: The first image is processed by a first initial encoder to extract features and obtain first information, and the second image is processed by a second initial encoder to extract features and obtain second information. The first initialization module performs feature compression on the first information to obtain the third information, and the second initialization module performs feature compression on the second information to obtain the fourth information; The first image's predicted depth image is obtained by performing feature mapping on the third information using a first initial decoder and a pre-trained second encoder, and the second image's predicted depth image is obtained by performing feature mapping on the fourth information using the first initial decoder and the pre-trained second encoder.
9. The depth estimation model training method according to claim 8, characterized in that, Before extracting features from the first image using a first initial encoder to obtain first information, and extracting features from the second image using a second initial encoder to obtain second information, the method further includes: The first image is image-recognized using a preset recognition module to obtain summary information of the first image, and the second image is image-recognized using the same preset recognition module to obtain summary information of the second image. The first image's summary information is extracted using a preset language module to obtain first text information, and the second image's summary information is extracted using the same preset language module to obtain second text information. The first text information is used as the key value of the self-attention layer of the first initial encoder, and the second text information is used as the key value of the self-attention layer of the second initial encoder.
10. The depth estimation model training method according to claim 8, characterized in that, The adjustment of the model parameters of the second initial model includes: Keep the parameters of the second initial encoder unchanged, and simultaneously adjust the parameters of the first initial module and the second initial module so that the adjusted parameters of the first initial module and the second initial module are the same.
11. The depth estimation model training method according to claim 1, characterized in that, The step of training a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model includes: The second image is subjected to image processing to obtain a processed second image, wherein the image processing includes at least one of color distortion processing and spatial distortion processing; A second initial model is trained based on the first image and its reference depth image, as well as the processed second image and its reference depth image, to obtain the second target model.
12. A depth estimation model training device, characterized in that, The device includes: The acquisition module is configured to acquire a first image, a reference depth image of the first image, and a second image, wherein the second image is different from the first image; The first training module is configured to train a first initial model based on the first image and a reference depth image of the first image to obtain a first target model; The determination module is configured to determine a reference model based on the first target model obtained during training. The first processing module is configured to obtain a reference depth image of the second image based on the second image and the reference model; The second training module is configured to train a second initial model based on the first image, a reference depth image of the first image, the second image, and a reference depth image of the second image to obtain a second target model. The second processing module is configured to, in response to the failure to meet the training termination condition, switch to training a first initial model based on the first image and a reference depth image of the first image to obtain a first target model; and in response to meeting the training termination condition, determine a target depth estimation model based on the second target model, wherein the target depth estimation model is used to obtain a depth image of the input image.
13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method as described in any one of claims 1-11.
14. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-11.