Training method and device of depth estimation model, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a method and device for training a depth estimation model, electronic equipment and a storage medium. The method comprises obtaining a first image and a second image; inputting the first image into the depth estimation model to output a first depth image based on parameters of the depth estimation model; inputting the first image and the second image into a pose estimation model to extract a pose conversion relationship between the first image and the second image; generating a reconstructed image according to the first depth image, the pose conversion relationship and pre-acquired camera parameters; calculating a similarity between the reconstructed image and the first image to obtain a two-dimensional loss map to extract a weak texture region in the first image; determining a first similarity value of a pixel point in the weak texture region according to the two-dimensional loss map; reducing a proportion of the first similarity value to obtain a loss value; and adjusting the parameters of the depth estimation model based on the loss value. The influence of the weak texture region on the depth estimation can be reduced.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a training method, apparatus, electronic device, and storage medium for a depth estimation model. Background Technology

[0002] Currently, one deep learning method for monocular depth estimation utilizes the SFM (Structure From Motion) principle. By providing the model with images from different times and perspectives (such as target frames and reference frames), the model infers the depth value of the target frame and uses the reference frame to reconstruct the target frame, thus obtaining a reconstructed frame. Depth estimation with lower error will result in a reconstructed frame that is closer to the target frame.

[0003] However, the reconstructed frame similarity cannot accurately express the degree of depth error in the following situations: low-texture images can be reconstructed very closely to the target frame even if the depth error is large, causing the network to learn incorrect target depth estimates. Summary of the Invention

[0004] In view of the above, it is necessary to propose a training method, device, electronic device and storage medium for a depth estimation model that can reduce the impact of weak texture images on depth estimation.

[0005] The training method for the depth estimation model includes: acquiring a first image and a second image; inputting the first image into the depth estimation model to output a first depth image based on the parameters of the depth estimation model; inputting the first image and the second image into a pose estimation model to extract the pose transformation relationship between the first image and the second image, wherein the first image and the second image are images from different viewpoints; generating a reconstructed image based on the first depth image, the pose transformation relationship, and pre-acquired camera parameters; calculating the similarity between the reconstructed image and the first image to obtain a two-dimensional loss map and extracting weak texture regions in the first image; determining a first similarity value of the pixels in the weak texture regions based on the two-dimensional loss map; reducing the weight of the first similarity value to obtain a loss value; and adjusting the parameters of the depth estimation model based on the loss value.

[0006] Compared to existing technologies, the training method, apparatus, electronic device, and storage medium for the depth estimation model provided by this invention generally have low loss values ​​based on weak texture regions. Regardless of whether the model has learned the correct depth inference, the loss value of this region will not differ much. Therefore, if the proportion of weak texture regions is too high, it may even drag down the overall training. Therefore, here we reduce the proportion of weak texture regions, that is, reduce the proportion of loss value contribution of weak texture regions, so as to reduce the adverse effect of weak texture regions on model training. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the training method for the depth estimation model provided in the embodiments of this application.

[0008] Figure 2 This is a schematic diagram illustrating the principle of the synthesized and reconstructed image provided in the embodiments of this application.

[0009] Figure 3 This is a schematic diagram of another deep image model training method provided in an embodiment of this application.

[0010] Figure 4 This is a schematic diagram of a training device structure provided in an embodiment of this application.

[0011] Figure 5 A schematic diagram of the structure of the electronic device provided in the application embodiment.

[0012] Explanation of main component symbols

[0013]

[0014]

[0015] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] Before providing a detailed explanation of the embodiments of this application, the application scenarios involved in the embodiments of this application will be introduced first.

[0018] Depth information from images is crucial for supporting computer vision technology and is essential for 3D reconstruction. It can be applied in fields such as autonomous driving, scene understanding, robotics, 3D reconstruction, photography, intelligent medicine, intelligent human-computer interaction, spatial mapping, and augmented reality. For example, in autonomous driving, depth information from images can be used to assist in sensor fusion, drivable space exploration, and navigation.

[0019] The following description, with reference to the accompanying drawings, illustrates a method, apparatus, and electronic device for training a depth estimation model according to embodiments of this application.

[0020] Figure 1This is a schematic flowchart illustrating the training method for the depth estimation model provided in this embodiment. It should be noted that the execution entity of the training method for the depth estimation model in this embodiment can be a training device, which can specifically be a hardware device or software within a hardware device. The hardware device can be, for example, a terminal device or a server. Figure 1 As shown, the training method for the depth estimation model proposed in this embodiment includes the following steps:

[0021] Step S10: Obtain the first image and the second image.

[0022] In this embodiment of the application, training the depth estimation model requires acquiring training image data, which includes a first image and a second image. Both the first and second images are images of the same target object, but taken from different perspectives. For example, the first image is a frontal view image, and the second image is a non-frontal view image. The target object can be, for example, a person, an animal, a vehicle, or a building.

[0023] In this embodiment, the first image and the second image can be captured by a monocular camera or a binocular stereo camera. In some embodiments, the first image and the second image can be video frames extracted from videos captured by a monocular camera or a binocular stereo camera. In some embodiments, the training device can store training image data in various data repositories, for example, storing monocular images in a monocular image repository and storing stereo image pairs in a stereo image data repository.

[0024] Step S11: Input the first image into the depth estimation model and output the first depth image based on the parameters of the depth estimation model.

[0025] In this embodiment, a first image is input into a depth estimation model, which outputs a first depth image corresponding to the first image. The pixel value (or grayscale value) of each pixel in the first depth image can represent either relative depth or depth value. Relative depth can be understood as the logical relationship between individual pixels. The distance between the real object corresponding to a pixel and the camera device or reference plane is called the depth value of that pixel; that is, the depth value of a pixel is the vertical distance from the real object corresponding to that point to the aforementioned camera device.

[0026] The depth estimation model can be an autoencoder (AE) network.

[0027] Autoencoders are a class of artificial neural networks (ANNs) used in semi-supervised and unsupervised learning. Their function is to learn representations of the input information by using the input information as the learning target. An autoencoder consists of an encoder and a decoder. Based on learning paradigm, autoencoders can be divided into contractive autoencoders, regularized autoencoders, and variational autoencoders (VAEs), where the first two are discriminative models and the latter is a generative model. Based on architecture, autoencoders can be feedforward or recursive neural networks.

[0028] In some embodiments, the depth estimation model includes a depth encoder and a depth decoder. The depth estimation model feeds a first image through the depth encoder to extract abstract depth features. The depth encoder can use different machine learning algorithms and techniques to extract these features.

[0029] For example, a deep encoder is a convolutional neural network consisting of multiple layers, where each subsequent layer reduces the dimensionality of the extracted features. For instance, after the first layer, an input image with an order of magnitude of 10^6 pixels or data points is reduced to a set of features on the order of 10^5. Through the last layer in a deep encoder, the abstracted depth features can be on the order of 10^4 or smaller. These numbers are purely illustrative. In practice, deep encoders can have different numbers of layers, and the number of pixels and depth features can vary.

[0030] In contrast, deep decoders include multiple layers to increase the dimensionality of abstract features.

[0031] As in the example above, the depth decoder can employ abstract depth features on the order of 10^4 and progressively derive the depth (relative depth or depth value) at each pixel of the input image across multiple layers. The depth decoder then outputs a first depth image, where each pixel in the depth map corresponds to the distance of the nearest object in the scene projected onto that pixel.

[0032] In some embodiments, the depth decoder outputs a disparity map, where each pixel on the disparity map corresponds to the reciprocal of the distance. In embodiments of this application, the principles described with reference to the depth map are readily applied to implementations with disparity maps. For example, a first image has already captured a tree at a given pixel at an unknown distance from the camera. The depth decoder outputs a depth value corresponding to the distance from the camera to the block at that pixel.

[0033] In some embodiments, the output depth value may be relative to another depth value. In other embodiments, the output depth value is scaled to provide a realistic measurement of objects in the scene, such as a city block being 10 meters away or a building being 25 meters away.

[0034] Step S12: Input the first image and the second image into the pose estimation model to extract the pose transformation relationship between the first image and the second image, wherein the first image and the second image are images from different viewpoints.

[0035] In embodiments of this application, the pose estimation model receives multiple cascaded images (such as a first image and a second image) to determine the pose of each input image. Pose generally refers to a mathematical transformation between perspectives of two images. In some embodiments, pose more generally describes the perspective of an image, where the perspective can be used to define the transformation between two images.

[0036] In this embodiment, the pose estimation model may include a pose encoder and a pose decoy. A first image and a second image are fed into the pose encoder, which extracts abstract pose features from the first and second images. These abstract pose features are then input to a pose decoder, which determines the pose transformation relationship between the first and second images. The pose encoder may be configured as a convolutional neural network comprising multiple layers to extract the abstract pose features and then derive the pose of the first and second images.

[0037] Step S13: Generate a reconstructed image based on the first depth image, pose transformation relationship, and pre-acquired camera parameters.

[0038] In this embodiment of the application, after acquiring the first depth image, a reconstructed image can be generated based on the depth values ​​of each pixel in the first depth image, the pose transformation relationship between the first image and the second image, and the camera parameters.

[0039] Specifically, the depth value of each pixel in the first depth image, camera parameters, pose transformation relationship, and position coordinates of each pixel in the first image are used as input to the position transformation algorithm to obtain the position coordinates of each pixel in the reconstructed image; the pixel value of each pixel in the first depth image is determined, and the reconstructed image is generated based on the pixel value and position coordinates.

[0040] Specifically, the formula for the position transformation algorithm is shown in Equation (1). By inputting the depth value of each pixel in the first depth image, camera parameters, pose transformation relationship, and position coordinates of each pixel in the first image into Equation (1), the position of each pixel in the first image in the reconstructed image can be obtained.

[0041] B2 = KTD1K -1 Formula B1 (1)

[0042] Where B1 is the position of each pixel in the first image, represented by (x1, y1), B2 is the position of each pixel in the first image in the reconstructed image, represented by (x2, y2), K is a matrix composed of camera parameters, and T is the pose transformation relationship. The number of columns in K is equal to the number of rows in T.

[0043] For example, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of the synthesized and reconstructed image provided in the embodiments of this application.

[0044] like Figure 2 As shown, the first image 100 is obtained by capturing the target point P from the target viewpoint O1, and the target point P is P1 in the first image 100. The second image 200 is obtained by capturing the target point P from the reference viewpoint O2, and the target point P is P2 in the second image 200. Reconstruction process overview: The pixels under the reference viewpoint O2 are remapped to the pixel positions under the target viewpoint O1, thus reconstructing a reconstructed image under the target viewpoint O1. Mapping principle: Pixel P1 under the target viewpoint O1 corresponds to the target point P in 3D coordinates with the correct depth value, and a unique P2 can be found under the reference viewpoint O2. Mapping calculation: Given any P1, to find P2, firstly, the position of the target point P in the target viewpoint O1 coordinate system can be obtained through the inferred depth (i.e., depth value) of P1 and the camera intrinsic parameter K, i.e., P1 in the first image. The position of the target point P in the reference viewpoint O2 coordinate system is obtained through pose transformation, i.e., P2 in the second image. Given the position of the target point P in the reference viewpoint O2 coordinate system and the camera intrinsic parameter K, the corresponding mapped pixel can be calculated. After calculating the corresponding P2 for all pixels P1 in the first image, the color of P1 is replaced with the color of P2 to obtain the reconstructed image.

[0045] Step S14: Calculate the similarity between the reconstructed image and the first image to obtain a two-dimensional loss map.

[0046] In this embodiment, the reconstructed image and the first image are two-dimensional images of the same size. The similarity between the reconstructed image and the first image is calculated, that is, the degree of consistency between the first image and the reconstructed image is estimated, to obtain a similarity value. A relevant similarity calculation function can be used to calculate the similarity value.

[0047] In some embodiments, the similarity calculation function can be implemented to calculate the color difference between two pixels corresponding to the same target point in the reconstructed image and the first image. For example, in the example above, both the first image and the reconstructed image are RGB images. The target point P corresponds to pixel P3 in the reconstructed image, and the color vector of pixel P3 is (r3, g3, b3). The target point P corresponds to pixel P1 in the first image, and the color vector of pixel P3 is (r1, g1, b1). Then, the difference between the color vectors of pixel P3 and pixel P1 is calculated. For example, the similarity between these two pixels is |r1-r3|+|g1-g3|+|b1-b3|. Another example is...

[0048] In other embodiments, the similarity calculation function can be implemented to calculate the structural similarity (SSIM) between the reconstructed image and the first image. For example, it can calculate the SSIM difference between windows at the same location in the reconstructed image and the first image. A window is a set of pixels within an N*N range around a specified center pixel location.

[0049] In this embodiment of the application, a two-dimensional loss map is obtained based on the calculated similarity value. The two-dimensional loss map is a single-channel two-dimensional map, and the pixel value on the two-dimensional loss map is the similarity value between the two pixels (or two windows) at the same position.

[0050] In this embodiment, the reconstructed image is obtained by calculating the pixel mapping relationship based on the first depth image and the pose transformation relationship, and then pasting the colors from the second image back to the target viewpoint. Ideally (with minimal depth and pose errors), the reconstructed image is almost identical to the first image. However, even if the depth estimation model derives a large depth error, it can still reconstruct a very close image to the first image, but the depth estimation model will learn an incorrect target depth estimate.

[0051] For example, such as Figure 2 As shown, if the depth estimation model has a large error in inferring the depth of pixel P1, it will calculate P'. Following the aforementioned mapping process, P3 is calculated from P', resulting in an erroneous reconstructed image. In reality, the pixel P2 and the pixel P3 derived under a large error, when pixel P2 and pixel P3 have similar textures, cannot reflect the positional error between P and P'. This means that even with a large depth error, a weakly textured image can be reconstructed very closely to the original image, causing the network to learn incorrect target depth estimates.

[0052] Therefore, by reducing the weight of the loss value in weak texture regions and increasing the weight of the loss value in non-weak texture regions (i.e., more obvious regions), the weight of weak texture regions in the training stage of the depth estimation model is reduced, thereby reducing the extent to which weak textures mislead the correction of the depth estimation model during the training stage and reducing the error impact of weak texture regions on the learning of the depth estimation model.

[0053] Step S15: Extract the weak texture region from the first image.

[0054] In this embodiment, color and brightness information of a first image can be obtained, and the first image can be segmented into several regions based on the color and brightness information. Then, gradient information corresponding to the first image is calculated, and weak texture regions are selected from the several regions based on the gradient information. The weak texture regions are regions where the average gradient statistical value is within a preset range.

[0055] Specifically, region-based segmentation methods, such as region growing, can be applied to segment images. The basic idea of ​​region growing is to group pixels with similar properties together to form regions. First, a seed pixel is selected as the starting point for each region to be segmented. Then, pixels around the seed pixel that have the same or similar properties (in this example, pixels with similar color and brightness information) are merged into the region containing the seed pixel. These new pixels are used as new seed pixels, and the above process continues until no more pixels meeting the conditions can be included, thus dividing the image into several regions. Image segmentation is then performed based on the color and brightness information of the image. The pixels within the segmented regions have similar color and brightness characteristics. The image can be viewed as a two-dimensional discrete function I(i,j), where (i,j) are the coordinates of a pixel in the image, and I(i,j) is the pixel value (e.g., RGB value) of pixel (i,j). The gradient information of the image is essentially the derivative of this two-dimensional discrete function, which can be calculated using the gradient formula. For example, operators such as Sobel, Roberts, Kirsch, Laplace, Piewitt, and Robinson. The gradient magnitude of an image reflects the magnitude of the brightness and color frequency changes of its pixels. For weakly textured regions, the brightness and color of the pixels within them are relatively similar with small variations, resulting in smaller gradient values. According to this principle, for a segmented region, the region with the smaller average gradient statistical value is the weakly textured region.

[0056] In some embodiments, the regions opposite to weak texture areas are non-weak texture areas. Locations with significant color gradient changes, i.e., regions with more pronounced visual features (non-weak texture areas), such as object edges, can also be identified. The first image, excluding non-weak texture areas, contains only weak texture areas.

[0057] Step S16: Determine the first similarity value of pixels in the weak texture region based on the two-dimensional loss map.

[0058] In this embodiment, the size of the two-dimensional loss map is the same as that of the first image. There is a natural one-to-one relationship between the pixels of the two-dimensional loss map and the first image; weak texture regions found in the first image can correspond to the two-dimensional loss map. For example, N pixels in a weak texture region found in the first image can correspond to N first similarity values ​​in the two-dimensional loss map.

[0059] Step S17: Reduce the weight of the first similarity value to obtain the loss value.

[0060] In this embodiment, the loss value is obtained by processing each pixel in the two-dimensional loss map. For example, the loss value is obtained by weighted summation of each pixel in the two-dimensional loss map. The loss value can also indicate whether the depth estimation model has been sufficiently trained and / or whether it is accurate enough in estimating depth information.

[0061] In this embodiment, after finding the first similarity value of the pixels in the weak texture region, a hyperparameter can be set to specify the reduction ratio of the first similarity value. For example, if the reduction ratio is set to S, the first similarity value in the two-dimensional loss map is multiplied by the reduction ratio S. Then, the pixel values ​​of the N pixels corresponding to the weak texture region in the two-dimensional loss map are all multiplied by the reduction ratio S. After that, each pixel in the two-dimensional loss map is processed to obtain the loss value. The reduction ratio can be set according to actual experimental data.

[0062] In the embodiments of this application, as mentioned above, the loss values ​​provided by weak texture regions are generally low. Regardless of whether the model has learned the correct depth inference, the loss value of this region will not be significantly different. Therefore, if the proportion of weak texture regions is too high, it may even drag down the overall training. Therefore, the proportion of weak texture regions is reduced here, that is, the proportion of loss value contribution of weak texture regions is reduced, so as to reduce the adverse effect of weak texture regions on model training.

[0063] Step S18: Adjust the parameters of the depth estimation model based on the loss value.

[0064] In this embodiment of the application, the parameters of the depth estimation model are adjusted based on the loss value obtained in step S17, so that the loss value is minimized during the adjustment of the parameters of the depth estimation model.

[0065] In some embodiments, please refer to Figure 3 The training method for the aforementioned depth estimation model also includes the following steps:

[0066] Step S31: Extract the non-weak texture regions from the first image.

[0067] In this embodiment, the first image includes weak texture regions and non-weak texture regions. Specifically, referring to step S15, the color and brightness information of the first image can be obtained, and the first image is divided into several regions based on the color and brightness information. Then, the gradient information corresponding to the first image is calculated, and non-weak texture regions are selected from the several regions based on the gradient information. Weak texture regions are regions where the average gradient value is outside a preset range. The difference between steps S31 and S15 is that step S31 identifies regions with larger average gradient values, which are the non-weak texture regions.

[0068] In some embodiments, locations with small color gradient changes can be directly identified, which are areas with indistinct visual features (weak texture areas). The weak texture areas are the regions in the first image excluding the non-weak texture areas.

[0069] In some embodiments, step S31 can be performed together with step S15 after the weak texture region in the first image is extracted in step S15.

[0070] Step S32: Determine the second similarity value of pixels in the non-weak texture region based on the two-dimensional loss map.

[0071] In this embodiment, the size of the two-dimensional loss map is the same as that of the first image. There is a natural one-to-one relationship between the pixels of the two-dimensional loss map and the first image; the non-weak texture regions found in the first image can correspond to the two-dimensional loss map. For example, M pixels in a non-weak texture region found in the first image can correspond to M second similarity values ​​in the two-dimensional loss map.

[0072] Step S33: Increase the weight of the second similarity value to obtain the loss value.

[0073] In this embodiment, after finding the second similarity value of pixels in the non-weak texture region, a hyperparameter can be set to specify the amplification ratio of the second similarity value. For example, if the amplification ratio is set to Q, the second similarity value in the two-dimensional loss map is multiplied by the amplification ratio Q. Then, the pixel values ​​of the M pixels corresponding to the non-weak texture region in the two-dimensional loss map are all multiplied by the amplification ratio Q. After that, each pixel in the two-dimensional loss map is processed to obtain the loss value. The amplification ratio can be set according to actual experimental data.

[0074] Step S34: Adjust the parameters of the depth estimation model based on the loss value.

[0075] In this embodiment of the application, the parameters of the depth estimation model are adjusted based on the loss value obtained in step S33, so that the loss value is minimized during the adjustment of the parameters of the depth estimation model.

[0076] In some embodiments, the weight of the first similarity value can be reduced and the weight of the second similarity value can be increased simultaneously to obtain a loss value, and the parameters of the depth estimation model can be adjusted according to the loss value.

[0077] In this embodiment, as mentioned above, the loss values ​​provided by weak texture regions are generally low. Regardless of whether the model has learned the correct depth inference, the loss value of this region will not differ significantly. If the proportion of weak texture regions is too high, it may even drag down the overall training. Therefore, this paper uses the method of finding non-weak texture region pixels to identify non-weak texture regions and increases the proportion of loss value contribution from non-weak texture regions. This can reduce the adverse effect of weak texture regions on model training.

[0078] Please see Figure 4 , Figure 4 This is a schematic diagram of a training device structure provided in an embodiment of this application.

[0079] In this embodiment of the application, the training device 41 includes an acquisition module 410, a first input module 411, a second input module 412, a generation module 413, a calculation module 414, an extraction module 415, a determination module 416, a reduction module 417, and an adjustment module 418.

[0080] The acquisition module 410 is used to acquire the first image and the second image;

[0081] The first input module 411 is used to input the first image into the depth estimation model and output the first depth image based on the parameters of the depth estimation model;

[0082] The second input module 412 is used to input the first image and the second image into the pose estimation model to extract the pose transformation relationship between the first image and the second image, wherein the first image and the second image are images from different perspectives.

[0083] The generation module 413 is used to generate a reconstructed image based on the first depth image, the pose transformation relationship and the pre-acquired camera parameters.

[0084] The calculation module 414 is used to calculate the similarity between the reconstructed image and the first image to obtain a two-dimensional loss map.

[0085] Extraction module 415 is used to extract weak texture regions in the first image.

[0086] The determination module 416 is used to determine the first similarity value of pixels in the weak texture region based on the two-dimensional loss map.

[0087] The reduction module 417 is used to reduce the weight of the first similarity value to obtain the loss value.

[0088] The adjustment module 418 is used to adjust the parameters of the depth estimation model based on the loss value.

[0089] See Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the application. In a preferred embodiment of the present invention, the electronic device 50 includes a memory 51 and at least one processor 52. Those skilled in the art should understand that... Figure 5 The structure of the computer device shown does not constitute a limitation of the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 50 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0090] In some embodiments, the electronic device 50 includes a terminal capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices.

[0091] It should be noted that electronic device 50 is only an example. Other existing or future electronic products that are suitable for this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0092] In some embodiments, memory 51 is used to store program code and various data, such as training device 41 installed in electronic device 50, and to enable high-speed, automatic access to programs or data during the operation of electronic device 50. Memory 51 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable storage medium capable of carrying or storing data.

[0093] In some embodiments, at least one processor 52 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. At least one processor 52 is the control unit of the electronic device 50, connecting various components of the entire electronic device 50 via various interfaces and lines. It performs various functions of the electronic device 50 and processes data by running or executing programs or modules stored in the memory 51 and calling data stored in the memory 51, such as training a depth estimation model.

[0094] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0095] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a server, personal computer, etc.) or processor to execute the methods of the various embodiments of the present invention.

[0096] In a further embodiment, combined with Figure 1 At least one processor 52 can execute the operating device of the electronic device 50 and various installed applications, program code, etc., such as the modules mentioned above.

[0097] The memory 51 stores program code, and at least one processor 52 can call the program code stored in the memory 51 to perform related functions. For example, Figure 4 Each module is program code stored in memory 51 and executed by at least one processor 52, thereby realizing the function of each module to achieve the purpose of training the depth estimation model.

[0098] In one embodiment of the invention, memory 51 stores one or more instructions (i.e., at least one instruction), the at least one instruction being executed by at least processor 52 to implement... Figure 1 The purpose of training the depth estimation model shown is...

[0099] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0100] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in the apparatus claims may also be implemented by a single element or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A training method for a depth estimation model, applied to electronic devices, characterized in that, The method includes: Acquire the first image and the second image; The first image is input into the depth estimation model to output a first depth image based on the parameters of the depth estimation model; The first image and the second image are input into the pose estimation model to extract the pose transformation relationship between the first image and the second image, wherein the first image and the second image are images from different viewpoints; A reconstructed image is generated based on the first depth image, the pose transformation relationship, and the pre-acquired camera parameters; Calculate the similarity between the reconstructed image and the first image to obtain a two-dimensional loss map; Extracting weak texture regions from the first image includes: acquiring color and brightness information of the first image; dividing the first image into regions based on the color and brightness information; calculating gradient information corresponding to the first image; and selecting weak texture regions from the regions based on the gradient information, wherein the weak texture regions are regions where the gradient statistical average value is within a preset range. The first similarity value of the pixels in the weak texture region is determined based on the two-dimensional loss map. Reducing the weight of the first similarity value to obtain a loss value includes: processing each pixel in the two-dimensional loss map to obtain the loss value; Based on the loss value, the parameters of the depth estimation model are adjusted.

2. The training method for the depth estimation model as described in claim 1, characterized in that, The method further includes: Extract the non-weak texture regions from the first image; The second similarity value of the pixels in the non-weak texture region is determined based on the two-dimensional loss map. The weight of the second similarity value is increased to obtain the loss value.

3. The training method for the depth estimation model as described in claim 1, characterized in that, The reduction of the weight of the first similarity value to obtain the loss value includes: Obtain the reduction ratio of the first similarity value; The weight of the first similarity value is adjusted based on the reduction ratio to obtain the loss value.

4. The training method for the depth estimation model as described in claim 2, characterized in that, The step of increasing the weight of the second similarity value to obtain the loss value includes: Obtain the magnification ratio of the second similarity value; The weight of the second similarity value is adjusted based on the magnification ratio to obtain the loss value.

5. The training method for the depth estimation model as described in claim 2 or 4, characterized in that, The step of extracting the non-weak texture region in the first image includes: Obtain the color and brightness information of the first image; The first image is divided into regions based on the color and brightness information; The gradient information corresponding to the first image is calculated; Based on the gradient information, non-weak texture regions are selected from the region, wherein the non-weak texture regions are regions where the average gradient statistical value is outside a preset range.

6. The training method for the depth estimation model as described in claim 2 or 4, characterized in that, The non-weak texture region includes the object edge region.

7. A training apparatus, said training apparatus being used to implement the training method of the depth estimation model as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is used to acquire the first image and the second image; The first input module is used to input the first image into the depth estimation model, so as to output the first depth image based on the parameters of the depth estimation model; The second input module is used to input the first image and the second image into the pose estimation model to extract the pose transformation relationship between the first image and the second image, wherein the first image and the second image are images from different viewpoints; The generation module is used to generate a reconstructed image based on the first depth image, the pose transformation relationship, and pre-acquired camera parameters; The calculation module is used to calculate the similarity between the reconstructed image and the first image, and obtain a two-dimensional loss map; The extraction module is used to extract weak texture regions in the first image; The determination module is used to determine the first similarity value of the pixels in the weak texture region based on the two-dimensional loss map; The reduction module is used to reduce the weight of the first similarity value to obtain a loss value; An adjustment module is used to adjust the parameters of the depth estimation model based on the loss value.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store at least one instruction, and the processor being used to implement the training method of the depth estimation model as described in any one of claims 1 to 6 when executing the at least one instruction.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the training method for the depth estimation model as described in any one of claims 1 to 6.

Images