Model training method and device, electronic equipment and computer program product

Abstract

The invention discloses a model training method, a model training device, electronic equipment and a computer program product. The model training method comprises the steps that based on image priori knowledge of a pre-trained first depth estimation model, first-stage knowledge migration training is carried out on the to-be-trained model, a first target model is obtained, the first depth estimation model is constructed based on a diffusion model, and depth estimation is carried out in a generative mode; based on depth priori knowledge of a pre-trained second depth estimation model, second-stage knowledge migration training is carried out on the first target model to obtain a second target model, the second depth estimation model carries out depth estimation in a discriminant mode, and the second target model is obtained; the second target model is used for realizing monocular depth estimation in combination with image knowledge and depth knowledge. Through the scheme of the invention, the model used for monocular depth estimation can be used for realizing the balance between high detail fidelity and fast reasoning capability.

Description

Technical Field

[0001] This application belongs to the field of depth estimation technology, and in particular relates to a model training method, a model training device, an electronic device, and a computer program product. Background Technology

[0002] Monocular depth estimation is a key technology in 3D scene understanding, with important applications in autonomous driving, robot manipulation, and 3D reconstruction. Existing depth estimation methods mainly fall into two categories: discriminative learning-based models and generative learning-based models. While these methods generally have high estimation efficiency, they often exhibit blurring and artifacts in detailed areas such as object edges, making it difficult to guarantee detail fidelity. Generative learning-based models, particularly diffusion models, can output detailed depth maps, but their multi-step iterative inference mechanism leads to high computational overhead and slow inference speed, making them unsuitable for real-time scenarios. Therefore, there is currently a lack of monocular depth estimation methods that can maintain the high detail fidelity of generative methods while achieving fast inference. Summary of the Invention

[0003] This application provides a model training method, a model training device, an electronic device, and a computer program product that can be used in monocular depth estimation models to achieve a balance between high detail fidelity and fast reasoning ability.

[0004] Firstly, this application provides a model training method, including: Based on the prior image knowledge of the pre-trained first depth estimation model, the first stage of knowledge transfer training is performed on the model to be trained to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach for depth estimation. Based on the depth prior knowledge of the pre-trained second depth estimation model, the first target model is trained in the second stage of knowledge transfer to obtain the second target model. The second depth estimation model uses a discriminative approach for depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

[0005] Secondly, this application provides a model training apparatus, comprising: The first transfer training module is used to perform the first stage of knowledge transfer training on the model to be trained based on the prior image knowledge of the pre-trained first depth estimation model, so as to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach to perform depth estimation. The second transfer training module is used to perform a second-stage knowledge transfer training on the first target model based on the depth prior knowledge of the pre-trained second depth estimation model, so as to obtain the second target model. The second depth estimation model uses a discriminative approach to perform depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

[0006] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0008] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0009] The advantages of this application compared to existing technologies are as follows: This application employs a generative depth estimation model based on a diffusion model for the first stage of knowledge transfer training. This allows the model to learn its prior image knowledge, ensuring that the depth map predicted by the model has excellent detail fidelity and edge sharpness, effectively overcoming the blurring and artifact problems in detail regions encountered by discriminative methods. Furthermore, this application introduces a pre-trained discriminative depth estimation model for the second stage of knowledge transfer training. Utilizing its efficient reasoning ability and generalization priors, the model undergoes effective knowledge transfer and acceleration guidance, enabling it to learn its depth priors. This significantly improves the model's inference speed without sacrificing generation quality, achieving a balance between high detail fidelity and fast inference capability.

[0010] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a schematic diagram illustrating the implementation process of the model training method provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the relationship between the model to be trained, the first target model, and the second target model provided in an embodiment of this application. Figure 3 This is a schematic diagram illustrating the implementation process of the first-stage knowledge transfer training provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the implementation process of the second-stage knowledge transfer training provided in an embodiment of this application; Figure 5 This is a structural block diagram of the model training device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0015] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly indicating the number, specific order, or primary and secondary relationship of the indicated technical features.

[0016] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0017] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0018] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), unless otherwise expressly and specifically defined.

[0019] This application proposes a model training method. This model training method can be applied to electronic devices. Please refer to... Figure 1 , Figure 1 The implementation flow of the model training method applied to this electronic device is given, and the details are as follows: Step 101: Based on the prior image knowledge of the pre-trained first depth estimation model, perform the first stage of knowledge transfer training on the model to be trained to obtain the first target model.

[0020] The first depth estimation model is built on the Diffusion Model architecture and uses a generative approach for depth estimation. This model has already been pre-trained. The generative approach to depth estimation means that the model predicts depth through a reverse process of gradually denoising random noise and iteratively generating a depth map. This method typically has high detail fidelity, especially performing well in high-frequency information regions such as object edges.

[0021] Since a large-scale diffusion model capable of generating high-quality images necessarily encodes a deep understanding of the physical laws governing the scene's 3D structure, lighting, and occlusion in its parameters, constituting a powerful image prior, electronic devices can select a pre-trained generative depth estimation diffusion model on a large dataset as the first depth estimation model. Based on its image prior knowledge, the model to be trained undergoes a first-stage knowledge transfer training, thereby completing the image prior transfer. In some examples, the first depth estimation model may be a Stable Diffusion model, etc., but this application does not limit this.

[0022] It is understood that the model to be trained is a novel depth estimation model, which, after training, will be used for the final monocular image depth estimation task. This allows the resulting first target model to initially possess strong visual prior knowledge, without needing to learn basic image features from scratch.

[0023] Step 102: Based on the deep prior knowledge of the pre-trained second depth estimation model, perform a second-stage knowledge transfer training on the first target model to obtain the second target model.

[0024] The second depth estimation model employs a discriminative approach for depth estimation, predicting depth by directly learning a mapping function from the input image to the depth map. It typically features fast forward inference speed, but its predictions may be somewhat smooth in detail. In some examples, this second depth estimation model may be a Metric3D v2 model, etc., but this application does not limit this approach.

[0025] Based on this, in this embodiment, the pre-trained second depth estimation model can be defined as the teacher model, and the first target model as the student model. The teacher model provides supervision signals or knowledge guidance to the student model for the second stage of knowledge transfer training, thereby completing the deep prior transfer. It can be understood that this process allows the first target model to focus on learning a specific mapping from image to depth during training, significantly accelerating its convergence. This allows the advantages of the second depth estimation model, such as its fast inference ability and strong generalization, to be transferred to the generative first target model. In this way, the final trained model, i.e., the second target model, can retain the high fidelity advantage of generative models while improving its training efficiency and the inference speed of the final model, solving the problem of slow inference in generative models.

[0026] It can be understood that the model to be trained, the first target model, and the second target model actually refer to the same model at different stages. Please refer to [link / reference]. Figure 2 , Figure 2 The relationship between the model to be trained, the first target model, and the second target model is given.

[0027] In some embodiments, the first-stage knowledge transfer training and / or the second-stage knowledge transfer training can be implemented in various ways. For example, it can be achieved through model parameter transfer, that is, initializing or updating the trained model using some or all of the network parameters of the pre-trained model, thereby enabling the trained model to inherit the prior knowledge learned in the pre-trained model. Another example is knowledge transfer training through intermediate feature representation transfer, that is, during training, constraints are imposed on the intermediate feature representations of the trained model to ensure they are consistent with or similar to the feature representations of the corresponding layers of the pre-trained model, thus achieving the transfer of prior knowledge. Yet another example is knowledge transfer training through output result constraints, that is, using the output results generated by the pre-trained model to constrain or guide the prediction results of the trained model, thereby enabling the trained model to learn the prior knowledge contained in the pre-trained model.

[0028] It should be noted that the different knowledge transfer methods mentioned above can be used individually or in any combination. The specific method used can be selected according to the actual application requirements, and this application embodiment does not limit this.

[0029] In some embodiments, please refer to Figure 3 Electronic devices can achieve the first stage of knowledge transfer training in the following ways: A1 is a backbone network based on the first depth estimation model, which is used to construct the network architecture of the model to be trained.

[0030] The backbone network refers to the core part of the first depth estimation model that undertakes the functions of core feature extraction and processing. For example, in the case of the first depth estimation model being Stable Diffusion 2.1, its backbone network is specifically the U-Net backbone network, which includes a complete encoder, intermediate layers, and decoder structure. It can be understood that this backbone network actually learns prior image knowledge.

[0031] The electronic device can analyze the overall structure of the first depth estimation model, identify and extract the architecture definition of its backbone network, including but not limited to the number of layers, the type of each layer (such as convolutional layers and attention layers), the inter-layer connection method (such as skip connections), and the hyperparameters of each layer (such as the number of filters and kernel size). Subsequently, according to this architecture definition, a new neural network with the same structure is created as the network architecture of the model to be trained.

[0032] A2 initializes the network parameters of the model to be trained based on the network parameters of the backbone network, thus obtaining the first target model.

[0033] Network parameters include, but are not limited to, the weights and biases in the neural network that need to be learned through training. The electronic device can first acquire all the network parameters saved by the backbone network of the first depth estimation model after pre-training. Then, when initializing a newly created model to be trained, the electronic device does not use random initialization; instead, it directly loads all the saved network parameters into each layer of the model to be trained, ensuring that the internal state of the model to be trained is highly consistent with the backbone of the fully pre-trained first depth estimation model. The model to be trained obtained after the above processing can be designated as the first target model, which has completed the first stage of knowledge transfer training.

[0034] In summary, the pre-trained first depth estimation model already possesses powerful image feature extraction and generation capabilities; therefore, by directly reusing the network parameters of the pre-trained first depth estimation model, the first target model possesses strong visual prior knowledge from the beginning of training, without needing to learn basic image features from scratch.

[0035] The training methods proposed above can be classified as transfer training based on model parameters.

[0036] In some embodiments, please refer to Figure 4 Electronic devices can achieve the second stage of knowledge transfer training in the following ways: B1, based on the second depth estimation model, processes the general samples in the general dataset that do not carry real labels, to obtain the first sample data with pseudo labels.

[0037] Pseudo-labels refer to label data that is not precisely labeled manually, but is automatically generated by another model (such as the second depth estimation model proposed in this application) and used as a supervision signal. In this application embodiment, an electronic device can input a batch of original images without real labels (real depth labels) into the second depth estimation model; the second depth estimation model can output a corresponding predicted depth map for each original image, thereby forming a data pair of (original image, predicted depth map), which constitutes the first sample data carrying pseudo-labels; that is, pseudo-labels are also depth labels, specifically: predicted depth data generated for the original image based on the depth prior knowledge learned by the second depth estimation model. It can be understood that although both pseudo-labels and real labels are depth labels and express depth data, their difference lies in that one expresses predicted depth data (generated by the model, which may not be accurate), and the other expresses real depth data (manually labeled, which is usually accurate).

[0038] In practical applications, a second depth estimation model can be used to perform the inference described above on general samples in a general dataset that do not carry true labels, thereby obtaining predicted depth maps for each general sample. The resulting general samples and their corresponding predicted depth maps constitute the first sample data carrying pseudo-labels. This process is typically completed in batches offline. The general dataset can be a large-scale dataset, and the images in this dataset do not carry true depth labels. In some examples, the general dataset can be the Unsplash dataset, other publicly available datasets, or a self-built dataset; this application does not limit this.

[0039] Through this step, the electronic device can use the second depth estimation model (specifically, an efficient discriminative depth estimation model) to automatically generate large-scale sample data, thereby solving the problem of the scarcity and high cost of obtaining high-quality real depth annotation data, and providing rich data resources for subsequent training of the first target model.

[0040] B2, based on the first sample data, trains the first target model to obtain the second target model.

[0041] After obtaining the first sample data, the first target model can be optimized based on it. In some examples, the first target model can be trained solely on this first sample data. In other examples, the first target model can be trained primarily on the first sample data, supplemented by other sample data. The process can be specifically described as follows: First, obtain a second set of sample data carrying the true labels. The number of these second set of sample data is significantly less than the number of the first set of sample data. In some examples, this second set of sample data may come from datasets such as Hypersim or Virtual KITTI.

[0042] Then, a sample set is constructed based on the first and second sample data. It's important to note that this process is not a simple merging of the two types of sample data. Rather, during training, samples are dynamically or statically extracted from both data sources (the first and second sample data) according to a specific strategy or proportion, collectively forming the sample set used for a single training batch. That is, this sample set actually contains two subsets: a pseudo-label subset including the first sample data, and a real-label subset including the second sample data.

[0043] In some examples, before each training epoch or each parameter update, a certain number of samples are randomly sampled from the first and second sample data, respectively, to form a sample set including a pseudo-labeled subset and a real-labeled subset. This sample set is then used to train the first target model for that training epoch. This process allows the first target model to learn generalization knowledge from the second deep estimation model, as well as precise geometric constraints from the real data, simultaneously during each parameter update.

[0044] Finally, the first target model is trained using the sample set to obtain the second target model. Each forward propagation of the first target model processes all sample data in a training batch and calculates the loss for that batch. Since the first target model is optimized under the joint guidance of supervisory signals from two data sources with different characteristics and confidence levels, it can simultaneously balance the breadth of learning (thanks to the large-scale first sample data) and accuracy (thanks to the small amount of second sample data). Thus, under limited, high-cost labeled data conditions, a high-performance model with strong generalization ability—the second target model—is ultimately trained.

[0045] The training methods proposed above can be classified as transfer training based on output constraints.

[0046] In some examples, to enable training of the first target model based on mixed data, the following loss function is proposed: L total =L gt +λL pseudo L in the loss function pseudo The first loss term is used specifically in the pseudo-labeled subset D. u The teacher guidance loss is calculated based on the pseudo-labeled subset D.u The first sample data and the corresponding prediction results are used to calculate the value, which measures the prediction result d of the first target model. s (Specifically, the predicted depth) and the pseudo-label d of the first sample data t The difference between them. In some examples, the first loss term L pseudo It can be calculated in the following way:

[0047] Where E represents the expected value (mean), (d s ,d t )~D u This indicates that the data pairs originate from the pseudo-labeled subset D. u .

[0048] In the loss function The second loss term is specifically defined in the true labeled subset D. gt The true supervised loss is calculated based on the real labeled subset D. gt The second sample data and the corresponding prediction results are used to calculate the result d of the first target model. s (Specifically, the predicted depth) and the true label d of the second sample data gt The difference between them. In some examples, the second loss term... It can be calculated using L1 loss (Mean Absolute Error), as shown below:

[0049] Where E represents the expected value (mean), (d s ,d gt )~D gt This indicates that the data pairs originate from a real-label subset D. gt .

[0050] In the loss function, λ represents the weight of the first loss term, which is a hyperparameter used to balance the contributions of the first and second sample data during training. By adjusting the weight of this first loss term, the dependence of the first target model on (potentially noisy) pseudo-label data can be controlled, thereby optimizing the learning effect.

[0051] In this embodiment of the application, several possible weighting strategies are proposed for setting the weight of the first loss term: a weighting strategy based on a preset value, a weighting strategy based on data quality, a weighting strategy based on the training phase, a weighting strategy based on the loss ratio, and a weighting strategy based on the validation set. The electronic device can determine the weighting strategy to be adopted according to the user's selection or the needs of the actual application scenario.

[0052] The weighting strategy based on preset values ​​is as follows: before training begins, a fixed value is assigned to the weight of the first loss term based on prior experience or preliminary experiments; for example, λ = 0.5. This weight remains unchanged throughout the entire training process.

[0053] The data quality-based weighting strategy specifically involves dynamically adjusting weights based on the quality of the pseudo-labels. In some examples, the confidence level of each first sample data point (output when the second depth estimation model generates the first sample data) can be used as its quality score; subsequently, when calculating the batch loss, the contribution of each first sample data point in the first loss term is no longer equal, but is associated with its quality score (e.g., weight λ). i = λ base * Confidence i In this way, high-quality (i.e., high-confidence) first sample data can have a greater weight in training, while the contribution of low-quality first sample data can be suppressed.

[0054] The weighting strategy based on the training phase specifically involves changing the weights as the training progresses (e.g., training epochs). In some examples, in the early stages of training, the first target model is more sensitive to noise from false labels, so the electronic device can set the weight of the first loss term to a smaller value. As training progresses and the first target model gradually stabilizes, the electronic device can linearly or according to a predetermined curve increase the weight of the first loss term to a larger value. Of course, a decay strategy from large to small can also be implemented, which will not be elaborated here.

[0055] The weighting strategy based on loss ratio specifically involves calculating the ratio between the first and second loss terms over a recent period before each parameter update (or after each training epoch), and dynamically adjusting the weight of the first loss term based on this ratio. This automatically balances the contributions of the two losses. For example, the weight of the first loss term can be set to (L...) gt / L pseudo The two losses are proportional to each other, so that the contribution of the two losses to gradient updates is kept within a preset balance range.

[0056] The weighting strategy based on the validation set is as follows: a separate validation dataset with real labels is prepared in advance; then, during the training process, the performance (e.g., depth estimation accuracy) of the first target model is evaluated on the validation set at multiple candidate fixed weight values ​​(e.g., λ=0.1, 0.3, 0.5, 0.7, 0.9); finally, the weight value corresponding to the optimal performance on the validation set is selected as the fixed weight used for subsequent training of the first target model.

[0057] As can be seen from the above, this embodiment employs a generative depth estimation model based on a diffusion model for the first stage of knowledge transfer training. This allows the model to learn its prior image knowledge, ensuring that the depth map predicted by the model has excellent detail fidelity and edge sharpness, effectively overcoming the blurring and artifact problems in detail regions of discriminative methods. Building upon this, this embodiment also introduces a pre-trained discriminative depth estimation model for the second stage of knowledge transfer training. Utilizing its efficient reasoning ability and generalization prior, the model undergoes effective knowledge transfer and acceleration guidance, enabling it to learn its depth prior knowledge. This significantly improves the model's inference speed without sacrificing generation quality. In summary, this embodiment achieves a balance between high detail fidelity and fast inference ability through a dual knowledge transfer strategy, leveraging existing image and depth priors.

[0058] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0059] Corresponding to the model training method provided above, this application also provides a model training apparatus. Please refer to... Figure 5 The model training device 5 in this embodiment includes: The first transfer training module 501 is used to perform a first-stage knowledge transfer training on the model to be trained based on the prior image knowledge of the pre-trained first depth estimation model, so as to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach to perform depth estimation. The second transfer training module 502 is used to perform a second-stage knowledge transfer training on the first target model based on the depth prior knowledge of the pre-trained second depth estimation model to obtain the second target model. The second depth estimation model uses a discriminative approach for depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

[0060] In some embodiments, the first-stage knowledge transfer training and / or the second-stage knowledge transfer training are implemented in any of the following ways: transfer training based on model parameters, transfer training based on intermediate feature representations, transfer training based on output result constraints, or a combination of at least two of the above methods.

[0061] In some embodiments, the first transfer training module 501 includes: The building unit is used to construct the network architecture of the model to be trained based on the backbone network of the first depth estimation model, wherein the backbone network learns prior knowledge of the image. The initialization unit is used to initialize the network parameters of the model to be trained based on the network parameters of the backbone network, so as to obtain the first target model.

[0062] In some embodiments, the second transfer training module 502 includes: The processing unit is used to process the general samples in the general dataset that do not carry real labels based on the second depth estimation model to obtain the first sample data with pseudo labels. The pseudo labels are: the predicted depth data generated for the corresponding general samples based on the depth prior knowledge learned by the second depth estimation model. The training unit is used to train the first target model based on the first sample data to obtain the second target model.

[0063] In some embodiments, the training unit includes: The acquisition sub-unit is used to acquire second sample data carrying the real label. The number of second sample data is less than the number of first sample data. Construct sub-units to build sample sets based on the first sample data and the second sample data; The training subunit is used to train the first target model using a sample set to obtain the second target model.

[0064] In some embodiments, during the training process of training the first target model using a sample set to obtain the second target model, the total loss function used includes a first loss term, a second loss term, and preset weights; wherein, the first loss term is calculated based on the first sample data and the corresponding prediction results, and is used to describe the difference between the pseudo label of the first sample data and the corresponding prediction results; the second loss term is calculated based on the second sample data and the corresponding prediction results, and is used to describe the difference between the true label of the second sample data and the corresponding prediction results.

[0065] In some embodiments, the weight of the first loss term is determined based on a preset weighting strategy; the weighting strategy includes any of the following: a weighting strategy based on a preset value, a weighting strategy based on data quality, a weighting strategy based on the training phase, a weighting strategy based on the loss ratio, and a weighting strategy based on the validation set.

[0066] As can be seen from the above, this application embodiment employs a generative depth estimation model based on a diffusion model for the first stage of knowledge transfer training. This allows the model to learn its prior image knowledge, ensuring that the depth map predicted by the model has excellent detail fidelity and edge sharpness, effectively overcoming the blurring and artifact problems in detail regions of discriminative methods. Building upon this, this application embodiment also introduces a pre-trained discriminative depth estimation model for the second stage of knowledge transfer training. Utilizing its efficient reasoning ability and generalization prior, the model learns its depth prior knowledge, effectively transferring and accelerating the model's knowledge. This significantly improves the model's inference speed without sacrificing generation quality. In summary, this application implementation achieves a balance between high detail fidelity and fast inference ability through a dual knowledge transfer strategy, utilizing existing image and depth priors.

[0067] Corresponding to the model training method provided above, this application also provides an electronic device. Please refer to... Figure 6 The electronic device 6 in this embodiment includes: a memory 601, and one or more processors 602. Figure 6 (Only one is shown in the image) and a computer program stored in memory 601 and executable on the processor. Specifically, the processor 602 performs the following steps by running the aforementioned computer program stored in memory 601: Based on the prior image knowledge of the pre-trained first depth estimation model, the first stage of knowledge transfer training is performed on the model to be trained to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach for depth estimation. Based on the depth prior knowledge of the pre-trained second depth estimation model, the first target model is trained in the second stage of knowledge transfer to obtain the second target model. The second depth estimation model uses a discriminative approach for depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

[0068] Assuming the above is the first possible implementation, in the second possible implementation provided based on the first possible implementation, the first stage knowledge transfer training and / or the second stage knowledge transfer training are implemented in any of the following ways: transfer training based on model parameters, transfer training based on intermediate feature representations, transfer training based on output result constraints, or a combination of at least two of the above methods.

[0069] Based on the first possible implementation described above, or based on the second possible implementation described above, in a third possible implementation, the image prior knowledge of the pre-trained first depth estimation model is used to perform a first-stage knowledge transfer training on the model to be trained to obtain a first target model, including: Based on the backbone network of the first depth estimation model, the network architecture of the model to be trained is constructed, wherein the backbone network learns prior knowledge of the image. The network parameters of the model to be trained are initialized based on the network parameters of the backbone network to obtain the first target model.

[0070] Based on the first possible implementation described above, or based on the second possible implementation described above, in a fourth possible implementation, the first target model is trained using the deep prior knowledge of the pre-trained second depth estimation model to obtain the second target model, including: Based on the second depth estimation model, the general samples in the general dataset that do not carry real labels are processed to obtain the first sample data with pseudo labels. The pseudo labels are: the predicted depth data generated for the corresponding general samples based on the depth prior knowledge learned by the second depth estimation model. The first target model is trained based on the first sample data to obtain the second target model.

[0071] In a fifth possible implementation based on the fourth possible implementation described above, the second target model is obtained by training the first target model based on the first sample data, including: Obtain second sample data carrying the real labels; the number of second sample data is less than the number of first sample data. A sample set is constructed based on the first sample data and the second sample data; The second target model is obtained by training the first target model using the sample set.

[0072] In the sixth possible implementation based on the fifth possible implementation described above, the processor 602 further implements the following steps when running the computer program stored in the memory 601: during the training process of training the first target model with a sample set to obtain the second target model, the total loss function used includes a first loss term, a second loss term, and preset weights; wherein, the first loss term is calculated based on the first sample data and the corresponding prediction result, and is used to describe the difference between the pseudo label of the first sample data and the corresponding prediction result; the second loss term is calculated based on the second sample data and the corresponding prediction result, and is used to describe the difference between the true label of the second sample data and the corresponding prediction result.

[0073] In the seventh possible implementation provided based on the sixth possible implementation described above, the weight of the first loss term is determined based on a preset weighting strategy; the weighting strategy includes any of the following: a weighting strategy based on a preset value, a weighting strategy based on data quality, a weighting strategy based on the training phase, a weighting strategy based on the loss ratio, and a weighting strategy based on the validation set.

[0074] It should be understood that, in the embodiments of this application, the processor 602 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0075] Memory 601 may include read-only memory and random access memory, and provides instructions and data to processor 602. Some or all of memory 601 may also include non-volatile random access memory. For example, memory 601 may also store device type information.

[0076] As can be seen from the above, this embodiment employs a generative depth estimation model based on a diffusion model for the first stage of knowledge transfer training. This allows the model to learn its prior image knowledge, ensuring that the depth map predicted by the model has excellent detail fidelity and edge sharpness, effectively overcoming the blurring and artifact problems in detail regions of discriminative methods. Building upon this, this embodiment also introduces a pre-trained discriminative depth estimation model for the second stage of knowledge transfer training. Utilizing its efficient reasoning ability and generalization prior, the model undergoes effective knowledge transfer and acceleration guidance, enabling it to learn its depth prior knowledge. This significantly improves the model's inference speed without sacrificing generation quality. In summary, this embodiment achieves a balance between high detail fidelity and fast inference ability through a dual knowledge transfer strategy, leveraging existing image and depth priors.

[0077] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0082] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.

[0084] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Based on the prior image knowledge of the pre-trained first depth estimation model, the first stage of knowledge transfer training is performed on the model to be trained to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach for depth estimation. Based on the depth prior knowledge of the pre-trained second depth estimation model, the first target model is subjected to a second stage of knowledge transfer training to obtain the second target model. The second depth estimation model uses a discriminative approach for depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

2. The model training method as described in claim 1, characterized in that, The first stage of knowledge transfer training and / or the second stage of knowledge transfer training are implemented in any of the following ways: transfer training based on model parameters, transfer training based on intermediate feature representations, transfer training based on output result constraints, or a combination of at least two of the above methods.

3. The model training method as described in claim 1 or 2, characterized in that, The image prior knowledge based on the pre-trained first depth estimation model is used to perform a first-stage knowledge transfer training on the model to be trained, resulting in a first target model, including: Based on the backbone network of the first depth estimation model, the network architecture of the model to be trained is constructed, wherein the backbone network has learned prior knowledge of the image. The network parameters of the model to be trained are initialized based on the network parameters of the backbone network to obtain the first target model.

4. The model training method as described in claim 1 or 2, characterized in that, The second target model is obtained by performing a second-stage knowledge transfer training on the first target model based on the deep prior knowledge of the pre-trained second depth estimation model, including: Based on the second depth estimation model, the general samples in the general dataset that do not carry real labels are processed to obtain the first sample data with pseudo labels. The real label is: real depth data, and the pseudo label is: predicted depth data generated for the corresponding general samples based on the depth prior knowledge learned by the second depth estimation model. The first target model is trained based on the first sample data to obtain the second target model.

5. The model training method as described in claim 4, characterized in that, The step of training the first target model based on the first sample data to obtain the second target model includes: Obtain second sample data carrying real labels, wherein the number of second sample data is less than the number of first sample data. A sample set is constructed based on the first sample data and the second sample data; The second target model is obtained by training the first target model using the sample set.

6. The model training method as described in claim 5, characterized in that, In the process of training the first target model using the sample set to obtain the second target model, the total loss function used includes a first loss term, a second loss term, and preset weights; wherein, the first loss term is calculated based on the first sample data and the corresponding prediction result, and is used to describe the difference between the pseudo label of the first sample data and the corresponding prediction result; the second loss term is calculated based on the second sample data and the corresponding prediction result, and is used to describe the difference between the true label of the second sample data and the corresponding prediction result.

7. The model training method as described in claim 6, characterized in that, The weight of the first loss term is determined based on a preset weighting strategy; the weighting strategy includes any one of the following: a weighting strategy based on a preset value, a weighting strategy based on data quality, a weighting strategy based on the training phase, a weighting strategy based on the loss ratio, and a weighting strategy based on the validation set.

8. A model training device, characterized in that, include: The first transfer training module is used to perform a first-stage knowledge transfer training on the model to be trained based on the prior image knowledge of the pre-trained first depth estimation model to obtain the first target model. The first depth estimation model is constructed based on the diffusion model and uses a generative approach for depth estimation. The second transfer training module is used to perform a second-stage knowledge transfer training on the first target model based on the depth prior knowledge of the pre-trained second depth estimation model to obtain the second target model. The second depth estimation model uses a discriminative approach for depth estimation, and the second target model is used to combine image knowledge and depth knowledge to achieve monocular depth estimation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1 to 7.

Images