Training method of depth map generation model, and depth map generation method and device

Abstract

The invention provides a training method of a depth map generation model, and a depth map generation method and apparatus. The method comprises the steps of obtaining a target coding model; obtaining first training data; coding the first sample image based on the target coding model to obtain a first sample potential mark vector; performing attention calculation on the first sample potential marker vector and the second sample potential marker vector based on a to-be-trained depth map generation model to obtain an attention matrix; performing nonlinear processing on the attention matrix based on a to-be-trained depth map generation model to obtain an output matrix; decoding the output matrix based on the target coding model, and outputting a predicted depth map; and calculating the predicted depth map and the first sample depth map to obtain a first loss function value. According to the technical scheme, the problem that the generated depth map is discontinuous can be solved, and then the precision of the target depth map is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a depth map generation model, a depth map generation method and a device. Background Art

[0002] Depth estimation is a key task in computer vision, which aims to infer the depth information of a scene from a single image or a sequence of images. This depth information is usually presented in the form of a depth map, which contains the distance information from each point in the image to the camera. In traditional depth estimation methods, it mainly relies on geometric and texture information. Traditional depth estimation methods include stereo vision. Stereo vision infers depth information by comparing images from two or more perspectives. This method uses the principle of parallax, that is, the difference in pixel coordinates of the same object in two images to convert the distance between the object and the shooting point.

[0003] However, in the above method, the binocular baseline determines the accuracy of the two cameras, and the difference in viewing angles of the two cameras will cause the two cameras to be mismatched, resulting in holes and discontinuities in the estimated depth map, thereby reducing the accuracy of the depth map. Summary of the invention

[0004] The embodiment of the present application provides a training method for a depth map generation model, a depth map generation method and a device, which can solve the problem of discontinuity of the generated depth map, thereby improving the accuracy of the target depth map. The technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, a method for training a depth map generation model is provided, the method comprising:

[0006] Obtain target encoding model;

[0007] Acquire first training data; the first training data includes a first sample image, a first sample depth map and a first noise set; the first sample image is matched with the first sample depth map; the first sample image is obtained based on a monocular camera; the first noise set includes a plurality of first noises;

[0008] Encoding the first sample image based on the target coding model to obtain a first sample potential label vector; and encoding the first sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector;

[0009] Performing attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix;

[0010] Based on the depth map generation model to be trained, the attention matrix is ​​nonlinearly processed to obtain an output matrix;

[0011] Decoding the output matrix based on the target coding model and outputting a predicted depth map;

[0012] The predicted depth map and the first sample depth map are calculated to obtain a first loss function value; the first loss function value is used to update a first weight parameter of the depth map generation model to be trained.

[0013] In a possible implementation manner, obtaining the target coding model includes:

[0014] Acquire second training data; the second training data includes a second sample image; the second sample image is obtained based on the monocular camera;

[0015] Encoding the second sample image based on a first coding model to be trained to obtain a plurality of first training coding block vectors;

[0016] Based on the first coding model to be trained, embedding processing is performed on the plurality of first training coding block vectors to obtain a first training potential label vector;

[0017] Decoding the first training potential label vectors based on the first encoding model to be trained to obtain an output image;

[0018] Calculating the output image and the second sample image to obtain a second loss function value;

[0019] Based on the second loss function value, the second weight parameter of the first coding model to be trained is updated to obtain a first target coding model.

[0020] In a possible implementation, the acquiring the target coding model further includes:

[0021] Acquire third training data; the third training data includes a second sample depth map and a second noise set; the third noise set includes the second noise;

[0022] Encoding the second sample depth map based on a second coding model to be trained to obtain a plurality of second training coding block vectors;

[0023] Embedding a plurality of third training coding block vectors based on the second coding model to be trained to obtain a second training potential label vector; the third training coding block vector is the second training coding block vector to which the second noise is added;

[0024] Decoding the second training potential label vector based on the second encoding model to be trained to obtain an output depth map;

[0025] Calculating the second sample depth map and the output depth map to obtain a third loss function value;

[0026] Based on the third loss function value, the third weight parameter of the second coding model to be trained is updated to obtain a second target coding model.

[0027] In a possible implementation, encoding the first sample image based on the target coding model to obtain a first sample potential label vector; and encoding the second sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector, includes:

[0028] Encoding the first sample image based on the first target coding model to obtain a plurality of first sample coding block vectors; and encoding the first sample depth map based on the second target coding model to obtain a plurality of second sample coding block vectors;

[0029] Based on the first target coding model, multiple first sample coding block vectors are embedded to obtain the first sample potential label vector; and based on the second target coding model, multiple third sample coding block vectors are embedded to obtain the second sample potential label vector; the third sample coding block vector is the second sample coding block vector with the first noise added.

[0030] In a possible implementation, the depth map generation model to be trained performs attention calculation on the first sample potential label vector and the second sample potential label vector to obtain an attention matrix, including:

[0031] Based on the depth map generation model to be trained, performing self-attention processing on the first sample potential label vector and the second sample potential label vector to obtain a self-attention matrix;

[0032] Based on the depth map generation model to be trained, cross-attention processing is performed on the self-attention matrix to obtain a cross-attention matrix.

[0033] In a possible implementation, the self-attention matrix is ​​subjected to cross-attention processing based on the depth map generation model to be trained to obtain a cross-attention matrix;

[0034] Get the third noise set;

[0035] Based on the depth map generation model to be trained, cross-attention processing is performed on the third noise set and the self-attention matrix to obtain the cross-attention matrix.

[0036] In a possible implementation, the nonlinear processing of the attention matrix based on the depth map generation model to be trained to obtain an output matrix includes:

[0037] Based on the depth map generation model to be trained, the cross attention matrix is ​​nonlinearly transformed to obtain the output matrix.

[0038] According to a second aspect of an embodiment of the present application, a depth map generation method is provided, the method comprising:

[0039] Acquire a target image; the target image is obtained based on a monocular camera;

[0040] Encoding the target image based on a first target encoding model to obtain a target latent label vector;

[0041] Obtaining a preset parameter vector; the preset parameter vector includes the size of the target depth map and the accuracy of the target depth map;

[0042] Performing attention processing on the target potential tag vector and the preset parameter vector based on the target depth map generation model to obtain a target attention matrix;

[0043] Based on the target depth map generation model, the target attention matrix is ​​nonlinearly transformed to output a target matrix;

[0044] The target matrix is ​​decoded based on the target coding model to obtain a target depth map.

[0045] According to a third aspect of an embodiment of the present application, a training device for a depth map generation model is provided, the device comprising:

[0046] A first acquisition module, used for acquiring a target coding model;

[0047] A second acquisition module is used to acquire first training data; the first training data includes a first sample image, a first sample depth map and a first noise set; the first sample image matches the first sample depth map; the first sample image is obtained based on a monocular camera; the first noise set includes a plurality of first noises;

[0048] an encoding module, configured to encode the first sample image based on the target encoding model to obtain a first sample potential label vector; and to encode the first sample depth map and the first noise set based on the target encoding model to obtain a second sample potential label vector;

[0049] An attention module, configured to perform attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix;

[0050] A nonlinear calculation module, used for performing nonlinear processing on the attention matrix based on the depth map generation model to be trained to obtain an output matrix;

[0051] A decoding module, configured to decode the output matrix based on the target coding model and output a predicted depth map;

[0052] A loss calculation module is used to calculate the predicted depth map and the first sample depth map to obtain a first loss function value; the first loss function value is used to update a first weight parameter of the depth map generation model to be trained.

[0053] According to a fourth aspect of an embodiment of the present application, a depth map generating device is provided, the device comprising:

[0054] A first acquisition unit is used to acquire a target image; the target image is obtained based on a monocular camera;

[0055] An encoding unit, configured to encode the target image based on a first target encoding model to obtain a target latent marker vector;

[0056] A second acquisition unit, configured to acquire a preset parameter vector; the preset parameter vector includes a size of a target depth map and an accuracy of the target depth map;

[0057] An attention unit performs attention processing on the target potential tag vector and the preset parameter vector based on a target depth map generation model to obtain a target attention matrix;

[0058] A nonlinear transformation unit, used to perform a nonlinear transformation on the target attention matrix based on the target depth map generation model, and output a target matrix;

[0059] A decoding unit is used to decode the target matrix based on the target coding model to obtain a target depth map.

[0060] According to a fifth aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes a training method for a depth map generation model or a depth map generation method.

[0061] According to a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement a training method for a depth map generation model or a depth map generation method.

[0062] In an embodiment of the present application, an embodiment of the present application provides a training method for a depth map generation model, wherein a depth map generation model to be trained is trained through a first sample image, a matching first depth map, and a first noise set to obtain a target depth map generation model. The first sample image is obtained based on a monocular camera. In the technical solution of the present application, a target image is obtained through a monocular camera, and a target depth map generation model estimates a target depth map based on a single target image. Therefore, the problem of mismatch between the two cameras of the binocular camera is avoided, so that there will be no holes in the target depth map, and a continuous target depth map is obtained, thereby improving the accuracy of the target depth map. In addition, multiple first noises are gradually added to the first sample depth map, and different first noises can simulate different scenes, thereby improving the robustness of the target depth map generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;

[0065] Figure 2 It is a flowchart of a training method for a depth map generation model provided in an embodiment of the present application;

[0066] Figure 3 It is a first flow chart of step 201 provided according to an embodiment of the present application;

[0067] Figure 4 is a second flow chart of step 201 provided according to an embodiment of the present application;

[0068] Figure 5 is a flow chart of step 203 provided according to an embodiment of the present application;

[0069] Figure 6 is a schematic diagram of an example flow of step 203 provided according to an embodiment of the present application;

[0070] Figure 7is a flow chart of step 204 provided according to an embodiment of the present application;

[0071] Figure 8 is a flowchart of step 2042 provided according to an embodiment of the present application;

[0072] Fig. 9 is an example flow chart of step 204 to step 205 provided according to an embodiment of the present application;

[0073] Fig.10 is a flowchart of a depth map generation method provided according to an embodiment of the present application;

[0074] Fig.11 It is a structural schematic diagram of a training device for a depth map generation model provided according to an embodiment of the present application;

[0075] Fig.12 is a structural schematic diagram of a depth map generating device provided according to an embodiment of the present application;

[0076] Fig.13 is a schematic diagram of the structure of a terminal provided according to an embodiment of the present application;

[0077] Fig.14 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0079] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0080] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms.

[0081] These terms are only used to distinguish one element from another element. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.

[0082] Here, at least one means one or more than one, for example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple means two or more than two, for example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.

[0083] Figure 1 It is a schematic diagram of an implementation environment provided according to an embodiment of the present application, and the implementation environment may include a terminal 101 and a server 102.

[0084] A depth map generation model is provided in the terminal 101. In addition, a monocular camera required for acquiring sample images is also provided on the terminal 101.

[0085] In some embodiments, a depth map generation model is trained based on sample images and sample depth maps to obtain a target depth map generation model.

[0086] The terminal 101 may be a smart phone with a depth map generation model, a wearable device, a personal computer, a laptop computer, a tablet computer, a smart TV, a car terminal, etc.

[0087] The server 102 may be a single server, a server cluster consisting of multiple servers, or a cloud processing center.

[0088] The terminal 101 is connected to the server 102 via a wired or wireless network.

[0089] In some embodiments, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including HyperText Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above data communication technology.

[0090] In related technologies, depth estimation is performed through methods such as stereo vision, structured light, and time of flight.

[0091] In the first example, stereo vision infers depth information by comparing images from two or more perspectives. This method uses the principle of parallax, that is, the difference in pixel coordinates of the same object in two images to convert the distance between the object and the shooting point. Stereo vision methods can be implemented using a binocular camera. A binocular camera captures images through two cameras to simulate the binocular vision of the human eye. It uses the parallax between the two cameras and calculates the difference between the two images through advanced algorithms to derive the depth information of objects in the scene. This imaging method enables the binocular camera to provide images with depth and color information, and the imaging effect is relatively more realistic and clear. However, in this method, the binocular baseline determines the accuracy of the dual camera, and the difference in binocular perspective will cause the two cameras to be mismatched, resulting in discontinuity and holes in the generated depth map, which in turn reduces the accuracy of the depth map.

[0092] In the second example, structured light calculates the distance of an object by measuring the propagation time or phase difference of light. This method usually uses a specific light pattern to project onto an object and observes the deformation of the light pattern on the surface of the object to infer the depth. However, since it is judged by the flight speed of photons, the material of the reflective surface will affect the accuracy of the judgment.

[0093] In the third example, time of flight calculates the distance of an object by emitting a light pulse to the object and measuring the time it takes for the light pulse to be reflected back. However, due to the inconsistent angles between the projection and the camera, some of the projected textures cannot be captured by the camera.

[0094] Figure 2 is a flow chart of a training method for a depth map generation model provided in an embodiment of the present application, such as Figure 2 As shown, in the embodiment of the present application, the application is described by taking the application on a terminal with a depth map generation model as an example. The method comprises the following steps:

[0095] In step 201, the terminal obtains a target coding model.

[0096] In some embodiments, the target coding model includes an encoder and a decoder, wherein the encoder is used to encode data to obtain an encoding result, and the decoder is used to decode the encoding result to obtain a decoding result.

[0097] Figure 3 It is a first flow chart of step 201 provided according to an embodiment of the present application.

[0098] Combine the following Figure 3 Step 201 is exemplarily described.

[0099] In some embodiments, the above step 201 may include the following steps 2011a to 2016a:

[0100] In step 2011a, the terminal obtains second training data; the second training data includes a second sample image.

[0101] In one example, the second sample image is a single image obtained based on a monocular camera. The monocular camera captures an image from a single perspective to obtain a two-dimensional image. The two-dimensional image does not include depth information.

[0102] In step 2012a, the terminal encodes the second sample image based on the first coding model to be trained to obtain a plurality of first training coding block vectors.

[0103] In one example, the first coding model to be trained performs feature extraction and dimensionality reduction processing on the second sample image based on the set feature extraction layer. For example, the feature extraction layer is a convolution layer, etc. These feature extraction layers can capture local information and global information in the second sample image, thereby representing the second sample image as a low-dimensional first training coding vector. The first training coding vector includes the main features and structural information of the second sample image. Optionally, the first training coding vector is divided into a plurality of first training coding block vectors. For example, the first training coding vector is divided according to a preset size. That is, a plurality of first training coding block vectors of a preset size are obtained. In an embodiment of the present application, by dividing the first training coding vector into a plurality of first training coding block vectors, each first training coding block vector represents a local area or feature of the second sample image. Therefore, the block processing helps to capture the local features of the second sample image, thereby improving the detail capture capability of the first target coding model. In addition, the first coding model to be trained performs feature extraction and dimensionality reduction processing on the second sample image based on the set feature extraction layer, which reduces the amount of data calculation compared to directly processing the second sample image.

[0104] In step 2013a, the terminal embeds a plurality of first training coding block vectors based on a first coding model to be trained to obtain a first training potential label vector.

[0105] In one example, after obtaining multiple first training coding block vectors, the first coding model to be trained embeds the multiple first training coding blocks based on the embedding layer, thereby mapping the multiple first training coding block vectors to a low-dimensional latent space to obtain a low-dimensional first training latent label vector. It should be understood that the latent space is a continuous, low-dimensional space. Each point in this space corresponds to a latent vector representation. In an embodiment of the present application, the embedding process helps the first coding model to be trained to capture the global features and structural information of the second sample image, thereby improving the generalization ability and robustness of the first coding model to be trained. In addition, the first training latent label vector is a more compact and easy-to-process vector representation, which facilitates subsequent decoding.

[0106] In step 2014a, the terminal decodes a plurality of first training potential label vectors based on a first coding model to be trained to obtain an output image.

[0107] In one example, the first encoding model to be trained decodes the first training potential label vector, that is, reconstructs the second sample image from the first training potential label vector. Optionally, the first encoding model to be trained reconstructs the features of the first training potential label vector through a neural network layer to obtain an output image with the same dimension as the second sample image. For example, the neural network layer is a fully connected layer or a deconvolution layer.

[0108] In step 2015a, the terminal calculates the output image and the second sample image to obtain a second loss function value.

[0109] In step 2016a, the terminal updates the second weight parameter of the first coding model to be trained based on the second loss function value to obtain a first target coding model.

[0110] In the above steps 2015a to 2016a, the second weight parameter is updated based on the second loss function value, so that the first coding model to be trained can learn how to reconstruct an output image that is increasingly similar to the second sample image from the potential representation until the similarity reaches the expected level, thereby obtaining the first target coding model. For example, the first target coding model is implemented based on a variational autoencoder. Similarly, the second target coding model can be obtained.

[0111] For example, the second loss function value can be obtained based on the reconstruction loss and the KL divergence regularization term. It should be noted that the reconstruction loss and the KL divergence regularization term can be calculated based on relevant technologies, and the embodiments of this application will not be described in detail.

[0112] Through the above analysis of steps 2011a to 2016a, it can be seen that the first coding model to be trained encodes the first sample image to obtain a low-dimensional first training coding vector. The first coding model to be trained embeds the first training coding vector to further obtain a low-dimensional first training potential label vector. Therefore, the second sample image of the first coding model to be trained undergoes data compression and dimensionality reduction processing twice, thereby reducing the amount of data for subsequent parameter calculations, thereby saving computing power.

[0113] Figure 4 It is a second flow chart of step 201 provided according to an embodiment of the present application.

[0114] Combine the following Figure 4 Step 201 is exemplarily described.

[0115] In some embodiments, the above step 201 may further include the following steps 2011b to 2016b:

[0116] In step 2011b, the terminal obtains third training data; the third training data includes a second sample depth map and a second noise set.

[0117] It should be noted that the second sample depth map matched with each second sample image can be obtained based on relevant technologies, and the embodiments of the present application will not be described in detail.

[0118] In step 2012b, the terminal encodes the second sample depth map based on the second coding model to be trained to obtain multiple second training coding block vectors.

[0119] It should be noted that step 2012b is similar to step 2012a and will not be described in detail in the embodiments of the present application.

[0120] In step 2013b, the terminal embeds multiple third training coding block vectors based on the second coding model to be trained to obtain a second training potential label vector; the third training coding block vector is the second training coding block vector with the second noise added.

[0121] In one example, a second noise set is obtained from the relevant technology. The second noise set includes multiple second noises. The second noise can also be understood as Gaussian noise or random noise. Different second noises correspond to different noise scenes. Multiple random noises are added to multiple first training potential marker vectors. Optionally, the second coding model to be trained gradually diffuses multiple noise vectors into multiple second training coding block vectors based on the first preset time step to obtain multiple third training coding block vectors. Among them, the second training coding block vector corresponds one-to-one to the second noise vector. The first preset time step can be represented by t1. t1 is an integer greater than or equal to 1.

[0122] In step 2014b, the terminal decodes the second training potential marker vector based on the second coding model to be trained to obtain an output depth map.

[0123] Step 2014b is similar to step 2014a. The embodiments of the present application will not be described in detail.

[0124] In step 2015b, the terminal calculates the second sample depth map and the output depth map to obtain a third loss function value.

[0125] In step 2016b, the terminal updates the third weight parameter of the second coding model to be trained based on the third loss function value to obtain a second target coding model.

[0126] In the above steps 2015b to 2016b, the third weight parameter is updated based on the third loss function value, so that the second coding model to be trained can learn how to reconstruct an output depth map that is increasingly similar to the second sample depth map from the second training potential marker vector until the similarity reaches the expected level, thereby obtaining a second target coding model. It should be noted that the calculation method of the third loss function value is the same as the calculation method of the second loss function value. The embodiments of the present application will not be described in detail.

[0127] Through the above analysis of steps 2011b to 2016b, it can be known that the second coding model to be trained encodes the second sample depth map to obtain a low-dimensional second training coding vector. The second coding model to be trained embeds the second training coding vector to obtain a low-dimensional second training potential label vector. Therefore, the second sample depth map of the second coding model to be trained undergoes data compression and dimensionality reduction processing twice, thereby reducing the amount of data for subsequent parameter calculations and saving computing power. In addition, in an embodiment of the present application, based on a first preset time step, multiple second noises in the second noise set are diffused into the second training coding vector, simulating a variety of noise scenarios and improving the robustness of the second coding model to be trained.

[0128] In step 202, the terminal obtains first training data.

[0129] In some embodiments, the first training data includes a first sample image, a first sample depth map, and a first noise set; the first sample image matches the first sample depth map; the first sample image is obtained based on a monocular camera. The first noise set includes multiple first noises. Among them, the first noise is random noise, which can also be understood as Gaussian noise. Different first noises correspond to different noise scenes. Each first noise corresponds to a pixel point in the first sample depth map. The pixel point represents the depth value of the point from the monocular camera or other reference plane. It should be noted that the first sample depth map can be obtained based on related technologies, and the embodiments of the present application will not be repeated.

[0130] It can be seen from step 202 that the embodiment of the present application is applied to a scene in which the first sample image is obtained based on a monocular camera. That is, the depth map is estimated based on a single image. Therefore, the embodiment of the present application does not have the problem of the two cameras of the binocular camera not being able to match due to the binocular parallax problem, thereby avoiding the problem of holes in the estimated depth map, and ensuring the continuity and accuracy of the estimated depth map.

[0131] In step 203, the terminal encodes the first sample image based on the target coding model to obtain a first sample potential label vector.

[0132] In some embodiments, the terminal encodes the first sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector.

[0133] Figure 5 It is a flowchart of step 203 provided according to an embodiment of the present application.

[0134] Combine the following Figure 5 Step 203 is exemplarily described.

[0135] In some embodiments, the above step 203 includes the following steps 2031 and 2032:

[0136] In step 2031, the terminal encodes the first sample image based on the first target coding model to obtain multiple first sample coding block vectors.

[0137] In some embodiments, the first sample depth map is encoded based on a second target coding model to obtain a plurality of second sample coding block vectors.

[0138] In step 2032, the terminal performs embedding processing on a plurality of first sample coding block vectors based on the first target coding model to obtain a first sample potential tag vector, wherein the first sample potential tag vector includes a plurality of word vectors (token).

[0139] In some embodiments, embedding processing is performed on multiple third sample coding block vectors based on the second target coding model to obtain a second sample potential label vector; the third sample coding block vector is the second sample coding block vector with the first noise added.

[0140] Figure 6 It is an example flow chart of step 203 provided according to an embodiment of the present application.

[0141] Combine the following Figure 6 Step 203 is exemplarily described.

[0142] In one example, the first target coding model extracts features from the first sample image based on the feature extraction layer to achieve coding. These feature extraction layers can capture local information and global information in the first sample image, thereby representing the first sample image as a low-dimensional first sample coding vector. The first sample coding vector is processed in blocks to obtain multiple first sample coding block vectors. Similarly, the second sample coding block vector can be obtained.

[0143] In one example, after obtaining the plurality of first sample coding block vectors, the first target coding model performs compression processing on the first sample coding block vectors. The compression processing can also be understood as embedding processing. Optionally, the first target coding model maps the plurality of first sample coding block vectors to a low-dimensional latent space to obtain a low-dimensional first sample latent label vector. Similarly, the second sample latent label vector can be obtained.

[0144] In one example, although there may be noise information in the first sample image, the noise type included in the noise information in the first sample image is relatively single, which is not conducive to obtaining a depth map generation model to be trained with good robustness. In an embodiment of the present application, a first noise set is obtained, and a first noise set vector is obtained based on relevant technologies. Assume that each vector in the second sample coding block vector corresponds to a pixel point. Then the terminal diffuses the first noise set vector to multiple second sample coding block vectors based on the second preset time step. That is, a first noise vector is added to each vector in the second sample coding block vector, thereby obtaining a third sample coding block vector. In an embodiment of the present application, by adding the first noise vector to the second sample coding block vector, in the process of training the depth map generation model to be trained, the depth map generation model to be trained learns to extract useful information from a variety of noise data, which is crucial for the depth map generation model to be trained in processing changes and noise in the image to be processed, and also improves the robustness of the depth map generation model to be trained. In addition, the first noise set vector is diffused to multiple second sample coding block vectors. It also enriches the diversity of the second sample coding block vector, which can simulate more real noise scenes, thereby helping to improve the accuracy and applicability of the depth map generation model to be trained in actual applications. It should be understood that the more the second preset time step, the better the performance of the depth map generation model to be trained, but the relative cost is also higher. Therefore, in the actual training process of the depth map generation model to be trained, the number of the second preset time step is determined according to actual needs. The second preset time step can be represented by t2. t2 is an integer greater than or equal to 1.

[0145] In some embodiments, after obtaining the first sample potential label vector and the second sample potential label vector, the terminal splices the first sample potential label vector and the second sample potential label vector together, and then uses the spliced ​​first sample potential label vector and the second sample potential label vector as inputs of the depth map generation model to be trained. Splicing can be understood as superimposing the first sample potential label vector and the second sample potential label vector together.

[0146] In step 204, the terminal performs attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix.

[0147] Figure 7 It is a flowchart of step 204 provided according to an embodiment of the present application.

[0148] Combine the following Figure 7 Step 204 is exemplarily described.

[0149] In some embodiments, the above step 204 includes the following steps 2041 and 2042:

[0150] In step 2041, the terminal performs self-attention processing on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain a self-attention matrix.

[0151] In step 2042, the terminal performs cross-attention processing on the self-attention matrix based on the depth map generation model to be trained to obtain a cross-attention matrix.

[0152] Figure 8 It is a flowchart of step 2042 provided according to an embodiment of the present application.

[0153] Combine the following Figure 8 Step 2042 is exemplarily described.

[0154] In some embodiments, the above step 2042 may include the following steps 20421 to 20422:

[0155] In step 20421, the terminal obtains a third noise set.

[0156] In one example, the third noise set includes a plurality of third noises. Different third noises correspond to different noise scenes. Each third noise is a random noise, which can also be understood as a Gaussian noise.

[0157] In step 20422, the terminal performs cross-attention processing on the third noise set and the self-attention matrix based on the depth map generation model to be trained to obtain a cross-attention matrix.

[0158] In step 205, the terminal performs nonlinear processing on the attention matrix based on the depth map generation model to be trained to obtain an output matrix.

[0159] In some embodiments, the above step 205 can be implemented based on the following implementation:

[0160] Based on the deep graph generation model to be trained, the cross attention matrix is ​​nonlinearly transformed to obtain the output matrix.

[0161] Fig. 9 It is an example flowchart of step 204 to step 205 provided according to an embodiment of the present application.

[0162] Combine the following Fig. 9 Step 204 to step 205 are exemplarily described.

[0163] In one example, after obtaining the first sample potential label vector and the second sample potential label vector, the depth map generation model to be trained first normalizes the first sample potential label vector and the second sample potential label vector. The normalization process can scale the numerical range of the first sample potential label vector and the second sample potential label vector to a relatively stable range, thereby avoiding numerical overflow or underflow problems during the calculation process.

[0164] In one example, the deep map generation model to be trained divides each word vector in the first sample latent tag vector and the second sample latent tag vector into multiple heads. Each head performs self-attention calculation independently. For example: if the dimension of the input word vector is 512, you can choose to create 8 heads, and the dimension of each head is 64 (512 / 8=64). For each head, a set of query, key, and value vectors are generated. These vectors are obtained from the original word vectors through linear transformation. For each head, the attention score of each word vector with other word vectors is calculated. This is achieved by the dot product of the query vector and the key vector. Then, the Softmax function is applied to the attention score of each head to obtain weights. These weights represent the importance of each word in the current head. Among them, the Softmax function is a normalized exponential function. Then, the value vector is weighted and summed according to these weights to generate the output of each head, thereby realizing the extraction and aggregation of important information. The outputs of all heads are spliced ​​together to form a new representation. The dimension of this representation is the same as the original input, but it contains the information that multiple heads pay attention to. Finally, the concatenated representation is linearly transformed through an output weight matrix to obtain the self-attention intermediate matrix. In addition, since the first sample potential label vector and the second sample potential label vector that have not been processed by self-attention include more detailed information, and the self-attention matrix only includes the information that the multi-head self-attention mechanism pays attention to, in order to ensure the integrity of the information, the first sample potential label vector and the second sample potential label vector that have not been processed by self-attention are superimposed on the self-attention intermediate matrix to obtain the self-attention matrix, so that the self-attention matrix includes more detailed information. For example: texture and other information.

[0165] In one example, since the multi-head self-attention mechanism generates a self-attention matrix by calculating multiple self-attention heads in parallel and fusing their information, if the output data distribution of each head is quite different, the fused data distribution may be unstable. Therefore, after obtaining the self-attention matrix, the self-attention intermediate matrix, the first sample potential label vector, and the second sample potential label vector are superimposed together, and the superimposed self-attention intermediate matrix, the first sample potential label vector, and the second sample potential label vector are normalized to obtain the self-attention normalization result. In this way, the output data of each head can be kept within a relatively stable distribution range, which is beneficial to the processing of subsequent network layers.

[0166] In one example, the multi-head cross attention mechanism obtains the self-attention normalization result and the third noise set, and fuses the self-attention normalization result and the third noise set to obtain a cross-attention intermediate matrix. Optionally, multiple third noises are fused into the self-attention normalization result according to the third preset time step. The third preset time step can be represented by t3. t3 is an integer greater than or equal to 1. Optionally, the self-attention matrix is ​​fused with the cross-attention intermediate matrix to obtain a cross-attention matrix. The cross-attention matrix is ​​normalized to obtain a cross-normalized matrix. It should be understood that continuing to add noise to the depth map generation model to be trained does not mean adding noise again to the sample data to which noise has been added. Instead, in the reverse diffusion process of the depth map generation model to be trained, that is, in the process of generating data from noise, the simulated noise is gradually reduced, so that the depth map generation model to be trained needs to learn to gradually remove noise in the reverse diffusion process to generate clear and high-quality data, that is, the cross-attention matrix.

[0167] In one example, the depth map generation model to be trained is based on multiple layers of neurons in a feedforward neural network, and the cross-normalized results are forward propagated. Each layer of neurons receives input from the neurons in the previous layer and performs nonlinear transformation through the activation function. In the last layer of the feedforward neural network (i.e., the output layer), the feedforward neural network generates one or more continuous values ​​as the output of the feedforward neural network. The output of the feedforward neural network is superimposed with the cross-attention matrix to obtain the output matrix, so that the output matrix also retains more detailed information in the image.

[0168] In some embodiments, after step 205, the terminal performs a linear transformation on the output matrix to obtain a linear matrix, and then changes the dimension of the linear matrix to obtain multiple sample dimension vectors. For example, the terminal performs a linear transformation on the output matrix based on a linear decoder to obtain a linear matrix. The linear matrix is ​​then reshaped to obtain multiple sample dimension vectors.

[0169] For example, the depth map generation model to be trained involved in the present application may be a latent diffusion transformer model. Transformer is a neural network architecture that relies on an attention mechanism to establish a global dependency relationship between input and output.

[0170] In step 206, the terminal decodes the output matrix based on the target coding model and outputs a predicted depth map.

[0171] In one example, the terminal maps each sample dimension vector back to the original data space or a form close to the original data based on the target coding model, wherein the original data is the first sample depth map.

[0172] In step 207, the terminal calculates the predicted depth map and the first sample depth map to obtain a first loss function value.

[0173] In some embodiments, the first loss function value is used to update the first weight parameter of the depth map generation model to be trained. For example, the predicted depth map and the first sample depth map are calculated based on the mean square error (MSE) loss function to obtain the first loss function value. The mean square error can also be expressed as L2.

[0174] It should be noted that the above steps 201 to 207 are one implementation. In addition, in another implementation, in step 202, the first training data includes a plurality of first sample images, a plurality of first depth maps, and a first noise set. Figure 1 One match.

[0175] In an embodiment of the present application, a depth map generation model to be trained is trained by a first sample image, a matching first depth map, and a first noise set to obtain a target depth map generation model. The first sample image is obtained based on a monocular camera. In the technical solution of the present application, a target image is obtained by a monocular camera, and a target depth map generation model estimates a target depth map based on a single target image. Therefore, the problem of mismatch between the two cameras of the binocular camera is avoided, so that there will be no holes in the target depth map, and a continuous target depth map is obtained, thereby improving the accuracy of the target depth map. In addition, multiple first noises are gradually added to the first sample depth map, and different first noises can simulate different scenes, thereby improving the robustness of the target depth map generation model.

[0176] Fig.10 1 is a flow chart of a depth map generation method provided in an embodiment of the present application. In the embodiment of the present application, the method is described by taking application on a terminal having a depth map generation device as an example. The method comprises the following steps:

[0177] In step 1001, the terminal obtains a target image; the target image is obtained based on a monocular camera.

[0178] In one example, in actual use of the target depth map generation model, the input depth map is no longer required.

[0179] In step 1002, the terminal encodes the target image based on the first target coding model to obtain a target latent marker vector.

[0180] It should be noted that step 1002 is similar to steps 2012a to 2013a, and the embodiments of the present application will not be described in detail.

[0181] In step 1003, the terminal obtains a preset parameter vector; the preset parameter vector includes the size of the target depth map and the accuracy of the target depth map.

[0182] In one example, since in actual applications, the required size and accuracy of the target depth map may not be fixed. Therefore, by inputting preset parameters such as the size of the target depth map and the accuracy of the target depth map into the target depth map generation model, the target depth map generation model can dynamically adjust the internal calculation process to ensure that the target depth map output by the target depth generation model matches the preset parameters, thereby optimizing the performance of the target depth map generation model, and then outputting a target depth map that meets expectations. That is, the size of the target depth map and the accuracy of the target depth map will affect the calculation process of the target depth map generation model. For example: the target depth map generation model dynamically adjusts the calculation process of the multi-head self-attention mechanism, the multi-head cross-self-attention mechanism, and the feedforward neural network.

[0183] In step 1004, the terminal performs attention processing on the target potential tag vector and the preset parameter vector based on the target depth map generation model to obtain a target attention matrix.

[0184] In step 1005, the terminal performs a nonlinear transformation on the target attention matrix based on the target depth map generation model and outputs the target matrix.

[0185] In step 1006, the terminal decodes the target matrix based on the target coding model to obtain a target depth map.

[0186] The above steps 1004 to 1006 are similar to steps 204 to 206, and the embodiments of the present application will not be described in detail.

[0187] In the embodiment of the present application, a target image is obtained by a monocular camera, and a target depth map generation model estimates a target depth map based on the target image. Therefore, the problem of mismatch between the two cameras of the binocular camera is avoided, and the problem of holes does not occur, thereby obtaining a continuous and highly accurate target depth map.

[0188] Fig.11 1 is a schematic diagram of a structure of a training device 1100 for a depth map generation model provided in an embodiment of the present application, the device comprising:

[0189] A first acquisition module 1101 is used to acquire a target coding model;

[0190] The second acquisition module 1102 is used to acquire first training data; the first training data includes a first sample image, a first sample depth map and a first noise set; the first sample image matches the first sample depth map; the first sample image is obtained based on a monocular camera; the first noise set includes a plurality of first noises;

[0191] The encoding module 1103 is used to encode the first sample image based on the target encoding model to obtain a first sample potential label vector; and encode the first sample depth map and the first noise set based on the target encoding model to obtain a second sample potential label vector;

[0192] An attention module 1104 is used to perform attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix;

[0193] A nonlinear calculation module 1105, used for performing nonlinear processing on the attention matrix based on the depth map generation model to be trained to obtain an output matrix;

[0194] A decoding module 1106, configured to decode the output matrix based on a target coding model and output a predicted depth map;

[0195] The loss calculation module 1107 is used to calculate the predicted depth map and the first sample depth map to obtain a first loss function value; the first loss function value is used to update the first weight parameter of the depth map generation model to be trained.

[0196] In some embodiments, obtaining a target coding model includes:

[0197] Acquire second training data; the second training data includes a second sample image; the second sample image is obtained based on a monocular camera;

[0198] Encoding the second sample image based on the first coding model to be trained to obtain a plurality of first training coding block vectors;

[0199] Embedding a plurality of first training coding block vectors based on a first coding model to be trained to obtain a first training potential label vector;

[0200] Decoding a plurality of first training potential label vectors based on a first encoding model to be trained to obtain an output image;

[0201] Calculating the output image and the second sample image to obtain a second loss function value;

[0202] The second weight parameter of the first coding model to be trained is updated based on the second loss function value to obtain a first target coding model.

[0203] In some embodiments, obtaining the target coding model further includes:

[0204] Acquire third training data; the third training data includes a second sample depth map and a second noise set; the second noise set includes a plurality of second noises;

[0205] Encoding the second sample depth map based on a second coding model to be trained to obtain a plurality of second training coding block vectors;

[0206] Embedding a plurality of third training coding block vectors based on a second coding model to be trained to obtain a second training potential label vector; the third training coding block vector is the second training coding block vector with a second noise added thereto;

[0207] Decoding the second training latent marker vector based on the second encoding model to be trained to obtain an output depth map;

[0208] Calculating the second sample depth map and the output depth map to obtain a third loss function value;

[0209] Based on the third loss function value, the third weight parameter of the second coding model to be trained is updated to obtain a second target coding model.

[0210] In some embodiments, encoding a first sample image based on a target coding model to obtain a first sample potential label vector; and encoding a second sample depth map and a first noise set based on the target coding model to obtain a second sample potential label vector, including:

[0211] Encoding the first sample image based on the first target coding model to obtain a plurality of first sample coding block vectors; and encoding the first sample depth map based on the second target coding model to obtain a plurality of second sample coding block vectors;

[0212] Based on the first target coding model, multiple first sample coding block vectors are embedded to obtain first sample potential label vectors; and based on the second target coding model, multiple third sample coding block vectors are embedded to obtain second sample potential label vectors; the third sample coding block vector is the second sample coding block vector with the first noise added.

[0213] In some embodiments, an attention calculation is performed on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix, including:

[0214] Based on the depth map generation model to be trained, self-attention is performed on the first sample potential label vector and the second sample potential label vector to obtain a self-attention matrix;

[0215] Based on the deep graph generation model to be trained, the self-attention matrix is ​​cross-attention processed to obtain a cross-attention matrix.

[0216] In some embodiments, cross-attention processing is performed on the self-attention matrix based on the depth map generation model to be trained to obtain a cross-attention matrix;

[0217] Get the third noise set;

[0218] Based on the depth map generation model to be trained, cross attention processing is performed on the third noise set and the self-attention matrix to obtain a cross attention matrix.

[0219] In some embodiments, the attention matrix is ​​nonlinearly processed based on the depth map generation model to be trained to obtain an output matrix, including:

[0220] Based on the deep graph generation model to be trained, the cross attention matrix is ​​nonlinearly transformed to obtain the output matrix.

[0221] It should be noted that: when executing the corresponding steps, the training device for the depth map generation model provided in the above embodiment only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the depth map generation model provided in the above embodiment and the training method embodiment of the depth map generation model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0222] In an embodiment of the present application, a depth map generation model to be trained is trained by a first sample image, a matching first depth map, and a first noise set to obtain a target depth map generation model. The first sample image is obtained based on a monocular camera. In the technical solution of the present application, a target image is obtained by a monocular camera, and a target depth map generation model estimates a target depth map based on a single target image. Therefore, the problem of mismatch between the two cameras of the binocular camera is avoided, so that there will be no holes in the target depth map, and a continuous target depth map is obtained, thereby improving the accuracy of the target depth map. In addition, multiple first noises are gradually added to the first sample depth map, and different first noises can simulate different scenes, thereby improving the robustness of the target depth map generation model.

[0223] Fig.12 1 is a schematic diagram of a depth map generating device 1200 provided according to an embodiment of the present application, the device comprising:

[0224] The first acquisition unit 1201 is used to acquire a target image; the target image is obtained based on a monocular camera;

[0225] The encoding unit 1202 is used to encode the target image based on the first target encoding model to obtain a target latent label vector;

[0226] The second acquisition unit 1203 is used to acquire a preset parameter vector; the preset parameter vector includes the size of the target depth map and the accuracy of the target depth map;

[0227] Attention unit 1204 performs attention processing on the target potential tag vector and the preset parameter vector based on the target depth map generation model to obtain a target attention matrix;

[0228] A nonlinear transformation unit 1205 is used to perform a nonlinear transformation on the target attention matrix based on the target depth map generation model and output a target matrix;

[0229] The decoding unit 1206 is used to decode the target matrix based on the target coding model to obtain a target depth map.

[0230] It should be noted that: when executing the corresponding steps, the training device for the depth map generation model provided in the above embodiment only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the depth map generation device provided in the above embodiment belongs to the same concept as the depth map generation method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0231] In the embodiment of the present application, a target image is obtained by a monocular camera, and a target depth map generation model estimates a target depth map based on the target image. Therefore, the problem of mismatch between the two cameras of the binocular camera is avoided, and the problem of holes does not occur, thereby obtaining a continuous and highly accurate target depth map.

[0232] An embodiment of the present application further provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above method when executing the computer program.

[0233] Taking computer equipment as the terminal as an example, Fig.13 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application, see Fig.13 The terminal 1300 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The terminal 1300 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0234] Typically, the terminal 1300 includes a processor 1301 and a memory 1302 .

[0235] The processor 1301 may include one or more processing cores, such as a 4-core processor, a 5-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0236] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one program code, which is used to be executed by the processor 1301 to implement the process provided by the method embodiment of the present application for the terminal execution in the above method.

[0237] In some embodiments, the terminal 1300 may further optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302 and the peripheral device interface 1303 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a display screen 1304, a camera assembly 1305, an audio circuit 1306 and a power supply 1307.

[0238] The peripheral device interface 1303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0239] The display screen 1304 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1304 is a touch display screen, the display screen 1304 also has the ability to collect touch signals on the surface or above the surface of the display screen 1304. The touch signal can be input to the processor 1301 as a control signal for processing. At this time, the display screen 1304 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1304 can be one, set on the front panel of the terminal 1300; in other embodiments, the display screen 1304 can be at least two, respectively set on different surfaces of the terminal 1300 or in a folding design; in other embodiments, the display screen 1304 can be a flexible display screen, set on the curved surface or folding surface of the terminal 1300. Even, the display screen 1304 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1304 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0240] The camera assembly 1305 is used to capture images or videos. In some embodiments, the camera assembly 1305 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1305 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0241] The audio circuit 1306 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1301 for processing. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1300. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1301 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1306 may also include a headphone jack.

[0242] The power supply 1307 is used to power various components in the terminal 1300. The power supply 1307 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1307 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0243] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0244] Take the computer device as a server as an example. Fig.14 14 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, wherein the one or more memories 1402 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1401 to implement the above-mentioned background removal method. Of course, the server 1400 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1400 may also include other components for implementing device functions, which will not be described in detail here.

[0245] The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating an image processing model. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0246] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0247] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a depth map generation model, characterized in that: include: Obtain target encoding model; Acquire first training data; The first training data includes a first sample image, a first sample depth map, and a first noise set; The first sample image is matched with the first sample depth map; The first sample image is obtained based on a monocular camera; The first noise set includes a plurality of first noises; Encode the first sample image based on the target coding model to obtain a first sample potential label vector; and encoding the first sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector; Performing attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix; Based on the depth map generation model to be trained, the attention matrix is ​​nonlinearly processed to obtain an output matrix; Decoding the output matrix based on the target coding model and outputting a predicted depth map; The predicted depth map and the first sample depth map are calculated to obtain a first loss function value; the first loss function value is used to update a first weight parameter of the depth map generation model to be trained.

2. The method according to claim 1, characterized in that The obtaining of the target coding model comprises: Acquire second training data; the second training data includes a second sample image; the second sample image is obtained based on the monocular camera; Encoding the second sample image based on a first coding model to be trained to obtain a plurality of first training coding block vectors; Based on the first coding model to be trained, embedding processing is performed on the plurality of first training coding block vectors to obtain a first training potential label vector; Decoding the first training potential label vectors based on the first encoding model to be trained to obtain an output image; Calculating the output image and the second sample image to obtain a second loss function value; Based on the second loss function value, the second weight parameter of the first coding model to be trained is updated to obtain a first target coding model.

3. The method according to claim 2, characterized in that The acquiring of the target coding model further includes: Acquire third training data; the third training data includes a second sample depth map and a second noise set; the second noise set includes a second noise; Encoding the second sample depth map based on a second coding model to be trained to obtain a plurality of second training coding block vectors; Based on the second coding model to be trained, embedding processing is performed on a plurality of third training coding block vectors to obtain a second training potential label vector; the third training coding block vector is the second training coding block vector to which the second noise is added; Decoding the second training potential label vector based on the second encoding model to be trained to obtain an output depth map; Calculating the second sample depth map and the output depth map to obtain a third loss function value; Based on the third loss function value, the third weight parameter of the second coding model to be trained is updated to obtain a second target coding model.

4. The method according to claim 3, characterized in that encoding the first sample image based on the target coding model to obtain a first sample potential label vector; And encoding the second sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector, including: Encoding the first sample image based on the first target coding model to obtain a plurality of first sample coding block vectors; and encoding the first sample depth map based on the second target coding model to obtain a plurality of second sample coding block vectors; Based on the first target coding model, multiple first sample coding block vectors are embedded to obtain the first sample potential label vector; and based on the second target coding model, multiple third sample coding block vectors are embedded to obtain the second sample potential label vector; the third sample coding block vector is the second sample coding block vector with the first noise added.

5. The method according to claim 1, characterized in that: The depth map generation model to be trained is based on which attention is calculated on the first sample potential label vector and the second sample potential label vector to obtain an attention matrix, including: Based on the depth map generation model to be trained, performing self-attention processing on the first sample potential label vector and the second sample potential label vector to obtain a self-attention matrix; Based on the depth map generation model to be trained, cross-attention processing is performed on the self-attention matrix to obtain a cross-attention matrix.

6. The method according to claim 5, characterized in that The cross-attention processing is performed on the self-attention matrix based on the depth map generation model to be trained to obtain a cross-attention matrix; Get the third noise set; Based on the depth map generation model to be trained, cross-attention processing is performed on the third noise set and the self-attention matrix to obtain the cross-attention matrix.

7. The method according to claim 6, characterized in that The nonlinear processing of the attention matrix based on the depth map generation model to be trained to obtain an output matrix includes: Based on the depth map generation model to be trained, the cross attention matrix is ​​nonlinearly transformed to obtain the output matrix.

8. A method for generating a depth map, characterized in that: include: Get the target image; The target image is obtained based on a monocular camera; Encoding the target image based on a first target encoding model to obtain a target latent label vector; Obtaining a preset parameter vector; the preset parameter vector includes the size of the target depth map and the accuracy of the target depth map; Performing attention processing on the target potential tag vector and the preset parameter vector based on the target depth map generation model to obtain a target attention matrix; Based on the target depth map generation model, the target attention matrix is ​​nonlinearly transformed to output a target matrix; The target matrix is ​​decoded based on the target coding model to obtain a target depth map.

9. A training device for a depth map generation model, characterized in that: include: A first acquisition module, used for acquiring a target coding model; A second acquisition module, used to acquire first training data; The first training data includes a first sample image, a first sample depth map, and a first noise set; the first sample image matches the first sample depth map; The first sample image is obtained based on a monocular camera; The first noise set includes a plurality of first noises; An encoding module, configured to encode the first sample image based on the target encoding model to obtain a first sample potential label vector; and encoding the first sample depth map and the first noise set based on the target coding model to obtain a second sample potential label vector; An attention module, configured to perform attention calculation on the first sample potential label vector and the second sample potential label vector based on the depth map generation model to be trained to obtain an attention matrix; A nonlinear calculation module, used for performing nonlinear processing on the attention matrix based on the depth map generation model to be trained to obtain an output matrix; A decoding module, configured to decode the output matrix based on the target coding model and output a predicted depth map; A loss calculation module is used to calculate the predicted depth map and the first sample depth map to obtain a first loss function value; the first loss function value is used to update a first weight parameter of the depth map generation model to be trained.

10. A depth map generating device, characterized in that: include: A first acquisition unit, used for acquiring a target image; The target image is obtained based on a monocular camera; An encoding unit, configured to encode the target image based on a first target encoding model to obtain a target latent marker vector; A second acquisition unit, configured to acquire a preset parameter vector; the preset parameter vector includes a size of a target depth map and an accuracy of the target depth map; An attention unit performs attention processing on the target potential tag vector and the preset parameter vector based on a target depth map generation model to obtain a target attention matrix; A nonlinear transformation unit, used to perform a nonlinear transformation on the target attention matrix based on the target depth map generation model, and output a target matrix; A decoding unit is used to decode the target matrix based on the target coding model to obtain a target depth map.

11. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the training method of the depth map generation model according to any one of claims 1 to 7 or the depth map generation method according to claim 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the depth map generation model according to any one of claims 1 to 7 or the depth map generation method according to claim 8.

Images