Image estimation method, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and discloses an image estimation method, an electronic device and a storage medium. When training an initial neural network model, a second loss function is added on the basis of a first loss function, wherein the second loss function is the product of a regression loss function and a weight coefficient, and the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of the depth of a pixel point in a training sample image during loss calculation, so that the initial network model can pay attention to the pixel points with larger depth values ​​in the training sample image when performing loss calculation, thereby being able to more accurately estimate the depth values ​​of the pixel points with larger depth values ​​in the training sample image, and then when estimating an image to be identified according to a preset convolutional neural network model after training, being able to accurately output a depth map of the image to be identified, thereby improving the accuracy of depth value prediction of background pixel points in the image to be identified.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image estimation method, electronic equipment and storage medium. Background Art

[0002] Image depth estimation is a technology that estimates the physical depth value in an image based on the image information on a plane. Depth estimation can be used in 3D modeling, depth perception, scene understanding and other scenarios. For example, on a mobile terminal, depth estimation can be used to estimate the degree relationship between a portrait, an object or a background during the shooting process, so that different levels of blur can be used according to different depths, thereby achieving a more realistic and natural blur gradient effect.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] In the depth value data of some captured images, most of the depth value data are close to the camera, and a few of the depth value data are far away from the camera, which makes the depth value distribution of the captured image have a certain "long tail" phenomenon. When calculating the loss, the loss function will mainly perform back propagation based on the pixels with lower depth values, and thus cannot accurately predict the depth values ​​of background pixels with higher depth values. Summary of the invention

[0005] The embodiments of the present invention aim to provide an image estimation method, an electronic device and a storage medium, which can improve the accuracy of depth value prediction of background pixels.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an image estimation method, the method comprising:

[0008] Obtain an image to be recognized;

[0009] Acquire a depth map of the image to be identified by using a preset convolutional neural network model;

[0010] Among them, the loss function of the preset convolutional neural network model includes a first loss function and a second loss function, the first loss function is a scale invariance loss function, the second loss function is the product of a regression loss function and a weight coefficient, the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of the depth of the pixel point in the training sample image during the loss calculation.

[0011] Optionally, the attention weight is an actual depth value of the training sample image.

[0012] Optionally, the weight coefficient also includes a regularization coefficient, and the weight coefficient is the sum of the attention weight and the regularization coefficient.

[0013] Optionally, the first loss function is:

[0014]

[0015] Among them, g i =logd i -logd i gt , λ∈[0,1], N is the number of pixels in the training sample image, d i is the predicted depth value of the i-th pixel of the training sample image, d i gt is the actual depth value of the i-th pixel of the training sample image.

[0016] Optionally, the first loss function is:

[0017] g i =logd i -logd i gt

[0018] Among them, g i =logd i -logd i gt , λ∈[0,1], N is the number of pixels in the training sample image, d i is the predicted depth value of the i-th pixel of the training sample image, d i gt is the actual depth value of the i-th pixel of the training sample image.

[0019] Optionally, the second loss function is:

[0020]

[0021] Among them, α D is the attention weight, β D is the regularization coefficient, l(d i ,d i gt ) is the BerHu loss function.

[0022] Optionally, the loss function is:

[0023] L=γl1+αl2,

[0024] Among them, γ and α are coefficients of the first loss function L1 and the second loss function L2 respectively.

[0025] Optionally, the convolutional neural network model includes an encoder and a decoder having multiple layers, and the step of inputting the image to be recognized into a preset convolutional neural network model to obtain a depth map of the image to be recognized includes:

[0026] Downsampling the input image to be recognized in turn through multiple levels of downsampling convolutional layers in the encoder, and outputting feature map data obtained by the processing to the decoder;

[0027] The feature map data is upsampled sequentially through multiple levels of upsampling convolution layers in the decoder, and local depth data is added to the feature map data output by at least one of the upsampling convolution layers through a plane fitting layer, wherein the local depth data is obtained by fitting the feature map data input to the upsampling convolution layer to a plane through the plane fitting layer;

[0028] Obtain a depth map of the image to be recognized output by the decoder.

[0029] Optionally, at least one level of the decoder includes an upsampling convolution layer and a plane fitting layer; adding local depth data to feature map data output by at least one of the upsampling convolution layers through the plane fitting layer comprises:

[0030] For a layer having an upsampling convolution layer and a plane fitting layer in the decoder, upsampling the input feature map data is performed through the upsampling convolution layer in the layer;

[0031] The input feature map data is fitted to a plane through the plane fitting layer pair in the level, and the local depth data of the input feature map data under the geometric constraints of the plane is obtained; wherein the size of the feature map data output by the plane fitting layer of the level is the same as the size of the feature map data output by the upsampling convolution layer of the level;

[0032] The feature map data output by the upsampling convolution layer and the plane fitting layer of this level are fused and input into the next level of the decoder.

[0033] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer-executable program. When the computer-executable program is executed by a processor, the computer executes the image estimation method as described in any one of the above items.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0035] Processor; and

[0036] A memory, configured to store executable instructions of the processor;

[0037] The processor is configured to perform the image estimation method as described in any one of the above by executing the executable instructions.

[0038] Compared with traditional technologies, the image estimation method, electronic device and storage medium provided in each embodiment of the present invention, when generating the preset convolutional neural network model, according to the depth value characteristics of the pixels in the image, a second loss function is added to the first loss function, wherein the second loss function is the product of the regression loss function and the weight coefficient, and the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of the depth of the pixels in the training sample image during the loss calculation, so that the initial network model can pay attention to the pixels with larger depth values ​​in the training sample image when performing loss calculation, thereby more accurately estimating the depth values ​​of the pixels with larger depth values ​​in the training sample image, and then, when estimating the image to be identified according to the trained preset convolutional neural network model, the depth map of the image to be identified can be accurately output, thereby improving the accuracy of the depth value prediction of the background pixels in the image to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0040] Figure 1-Figure 3 A schematic diagram of the model structure of a preset convolutional neural network model provided in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a flow chart of an image estimation method provided by an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be arbitrarily combined with each other. The steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. Moreover, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0045] Please refer to Figure 1 , Figure 1 A schematic diagram of the model structure of a preset convolutional neural network model provided for an embodiment of the present invention, wherein the preset convolutional neural network model is a U-Net structure, and the model includes an encoder and a decoder, and the encoder and the decoder each include n layers.

[0046] Specifically, any level of the encoder includes a downsampling convolution layer, and corresponding to the encoder, any level of the decoder includes an upsampling convolution layer. The output data of the downsampling convolution layer at any level in the encoder is passed to the downsampling convolution layer at the next level and the upsampling convolution layer at the same level in the decoder.

[0047] The output data of each downsampling convolution layer in the encoder is passed to the downsampling convolution layer of the next level and the upsampling convolution layer of the same level in the decoder.

[0048] For example, in Figure 1 In the model structure shown, for the encoder, the second level is the upper level of the third level, and the data output by the downsampling convolution layer of the second level of the encoder is passed to the downsampling convolution layer of the third level of the encoder. At the same time, the data output by the downsampling convolution layer of the second level of the encoder is also passed to the upsampling convolution layer of the second level in the decoder.

[0049] The downsampling convolution layer in the encoder is used to downsample the input data through the convolution kernel. Figure 1 As shown in the figure, the downsampling convolution layer is 2 times downsampling. For the first level of the encoder, the input data is an image to be processed with a size of H / 2. After being processed by the downsampling convolution layer of the first level, the output result is a feature map of the image processed to a size of H / 4. After the H / 4 feature map is input to the downsampling convolution layer of the second level for processing, the output result is a feature map of size H / 8.

[0050] The input data of each upsampling convolution layer in the decoder includes the output data of the upsampling convolution layer of the previous level of the decoder and the output data of the downsampling convolution layer of the same level in the encoder. The upsampling convolution layer in the decoder is used to upsample the input data through the convolution kernel.

[0051] For example, in Figure 1In the model structure shown, for the decoder, the third layer is the upper layer of the second layer. The data output by the upsampling convolution layer of the third layer of the decoder and the downsampling convolution layer of the second layer of the encoder are fused and input into the upsampling convolution layer of the second layer of the decoder.

[0052] In some embodiments, at least one level of the decoder includes not only an upsampling convolution layer but also a plane fitting layer, and the upsampling convolution layer and the plane fitting layer process the data output by the previous level of the decoder respectively, and then fuse the processing results and output them to the next level of the decoder. In other words, the data input to this level is processed by the upsampling convolution layer and the plane fitting layer respectively, and then the processing results are fused and output to the next level of the decoder.

[0053] For example, in Figure 1 In the model structure shown, the second layer of the decoder includes an upsampling convolution layer and a plane fitting layer. The data output from the second layer of the encoder and the data output from the third layer of the decoder are fused and input to the second layer of the decoder. The data input to the second layer of the decoder are processed by the upsampling convolution layer and the plane fitting layer respectively, and the processing results are fused and input to the first layer of the decoder.

[0054] It should be noted that Figure 1 This is only a schematic diagram for illustrating the location of the plane fitting layer. In this embodiment, the plane fitting layer is not limited to being located at Figure 1 In the second level shown, the plane fitting layer can also be set in one or more other levels of the decoder, and the setting position of the plane fitting layer is not limited here.

[0055] In this embodiment, the plane fitting layer is used to fit the input feature map data to a fitting plane representing depth information, and obtain the local depth data of the input feature map data under the geometric constraints of the plane as output data. The size of the feature map data output by the plane fitting layer is the same as the size of the feature map data output by the upsampling convolution layer at the same level.

[0056] Specifically, in this embodiment, the plane fitting layer can estimate a four-dimensional vector composed of four parameters for each pixel position of the input feature map, and these four parameters can define a plane in a spatial rectangular coordinate system. That is, Ax+By+Cz+D=0, and the size of the plane can be any N times the size of the input feature map. That is, through the plane fitting layer, each pixel position in the input feature map can be converted into an (N*N) size area on a plane that is N times the size of the input image, and each (N*N) size area corresponds to the local depth of each pixel position of the input image.

[0057] Afterwards, the plane can be fused with the feature maps output by other upsampling convolution layers, and the geometric constraints brought by the fitting plane can be added to the result of the upsampling convolution processing, thereby constraining the predicted depth and making the prediction result have a more stable numerical range.

[0058] In some possible implementations, in order to reduce the number of model parameters and inference time and meet the requirement of lightweight network, the n value of the plane fitting layer is 1. Figure 1 In the model structure shown, the feature map size of the second level of the input decoder is H / 8. After the upsampling convolution layer of this layer is upsampled, the feature map is stored as H / 4, and the size of the fitting plane generated by the plane fitting layer of this level is also H / 4. Because of the same size, the data output by the upsampling convolution layer and the plane fitting layer of this level can be directly fused and input to the next level. In addition to the spatial features extracted by the upsampling convolution layer, the fused output data also adds the geometric constraints brought by the plane fitting layer, which can make the results obtained when the fused features are used for further feature extraction or depth prediction more accurately reflect the actual depth of the image.

[0059] In some embodiments, see Figure 2 The plane fitting layer may include a 1*1 convolution layer and a numerical constraint layer, i.e., a geometric mapping layer.

[0060] Among them, the 1*1 collection layer is used to compress the input feature map data into four data channels, and the four data channels include a first data channel, a second data channel, a third data channel and a fourth data channel.

[0061] The numerical constraint layer is used to compress the values ​​of the data of the first data channel and the second data channel to -0.5 to 0.5, the values ​​of the data of the third data channel to 0.50 to 1.0, and the values ​​of the data of the fourth data channel to 0 to 1.0 using a preset activation function. The compressed data of the first data channel, the second data channel, and the third data channel are subjected to L2 regularization and normalization. The preset activation function may be a Sigmoid function.

[0062] The geometric mapping layer is used to fit the regularized and normalized data of the first data channel, the second data channel, the third data channel, and the compressed data of the fourth data channel to a plane to obtain four fitting coefficients; and obtain the local depth data of each position on the input feature map based on the four fitting coefficients.

[0063] In the geometric mapping layer, the following formula is used to convert the four constructed fitting coefficients into a local depth of (N*N):

[0064]

[0065] Among them C i Represents the depth value of each pixel in the local depth of (N*N) size, (n1,n2,n3,n4) are the four coefficients predicted to characterize the fitting plane, (u i ,v i ) represents the coordinate value of pixel position i on the image. That is, through the above formula, each pixel position on the input feature map can be corresponded to the local depth value of the (N*N) size area in the fitted plane.

[0066] In some embodiments, at least one level in the decoder having the plane fitting layer also includes a dilated convolution layer, which performs dilated convolution processing on the feature map output by the previous level of the decoder, and outputs the processing results to the upsampling convolution layer and the plane fitting layer of the level respectively.

[0067] Please continue reading Figure 1 In the second level of the decoder, in addition to an upsampling convolution layer and a plane fitting layer, there is also a dilated convolution layer. The data input to this level is first processed by the dilated convolution layer. The dilated convolution layer uses a dilated convolution kernel to process the input feature map, and can obtain receptive fields of different sizes.

[0068] In some embodiments, the dilated convolution layer includes three types of stored dilated convolution kernels with 3, 6 and 9 zero-padding numbers.

[0069] In some other embodiments, the structure of any upsampling convolution layer of the decoder is a depth-separable convolution. In the embodiment of the present invention, the depth-separable convolution can be connected across layers. For details, please refer to Figure 3 , any upsampling convolution layer of the decoder includes a global pooling module, a 1*1 convolution module, a batch normalization module and an activation function module, and the activation function module can be a Sigmoid function. Any upsampling convolution layer of the decoder processes the input feature map data in sequence through the global pooling module, the 1*1 convolution module, the batch normalization module and the activation function module, and then multiplies the input feature map and outputs it.

[0070] It should be noted that, in this embodiment, H / 2 images to be identified are used as input of the preset convolutional neural network model.

[0071] In the embodiment of the present invention, a plane fitting layer is added to the convolutional neural network model of U-Net, and local depth data obtained by fitting the feature map data input to the upsampling convolution layer to a plane is added to the feature map data output by at least one upsampling convolution layer through the plane fitting layer. In this way, by adding the geometric constraints brought by the fitting plane to the features, the estimated result can have a more stable numerical range, thereby making the depth estimation result more accurate.

[0072] See also Figure 4 The present invention provides an image estimation method, which is applied to a preset convolutional neural network model having the above structure, and the method comprises:

[0073] S41, obtaining an image to be identified;

[0074] In this embodiment, the H / 2 images to be recognized are used as input of the preset convolutional neural network model.

[0075] S42. Obtain a depth map of the image to be identified through a preset convolutional neural network model.

[0076] The preset convolutional neural network model includes an encoder and a decoder with multiple layers. The image to be identified is down-sampled in turn by the downsampling convolution layers of multiple layers in the encoder of the preset convolutional neural network model, and the feature map data obtained by the processing is output to the decoder of the preset convolutional neural network model; the input feature map data is up-sampled in turn by the upsampling convolution layers of multiple layers in the decoder, and local depth data is added to the feature map data output by at least one of the upsampling convolution layers through a plane fitting layer. The local depth data is obtained by fitting the data input to the upsampling convolution layer to a plane by the plane fitting layer, thereby obtaining the depth map of the image to be identified output by the decoder.

[0077] In this embodiment, by adding geometric constraints brought by the fitting plane to the result of the upsampling convolution process, the estimated depth value is constrained, so that the estimated result can have a more stable value range, thereby making the depth estimation result more accurate.

[0078] It can be understood that the preset convolutional neural network model is a stable convolutional neural network model obtained after the initial neural network model is trained with a large amount of training sample image data. When defining the initial neural network model, the loss function of the model must be clarified first. The loss function is used to indicate the difference between the predicted depth value and the actual depth value of the sample image, and the model parameters of the initial neural network model are adjusted by reducing the difference until the convergence of the initial neural network reaches the preset standard, and the training of the initial neural network is completed.

[0079] Specifically, the actual sample depth map of the training sample image and the predicted depth map predicted by the initial neural network model are used to calculate the loss through the loss function. When the loss calculated by the loss function is less than the preset loss threshold, the model parameters of the initial neural network model enter a stable state, thereby obtaining a convolutional neural network model with a higher degree of fit.

[0080] During the loss calculation process, the size of the predicted depth map and the actual depth map of the sample image used for loss calculation are the same, wherein the input data of the initial neural network model is an H / 2 sample image. Therefore, the loss calculation of the initial neural network model is set to be executed in the H / 2 upsampling convolution layer in the decoder of the initial neural network model.

[0081] In an embodiment of the present invention, the loss function of the preset convolutional neural network model includes a first loss function and a second loss function, wherein the first loss function is a scale invariance loss function, and the second loss function is the product of a regression loss function and a weight coefficient, and the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of pixels with larger depth values ​​in the training sample image during loss calculation.

[0082] On the basis of the scale invariance error, the scale invariance error is introduced into the loss function to obtain the first loss function. Specifically, the formula of the first loss function is:

[0083]

[0084] Among them, g i =logd i -logd i gt , λ∈[0,1], N is the number of pixels in the training sample image, d i is the predicted depth value of the i-th pixel of the training sample image, d i gtis the actual depth value of the ith pixel of the training sample image. When λ=0, the first loss function is in the same form as the L2 norm, and when λ=1, the first loss function is in the form of a size invariant error.

[0085] In order to more accurately focus on the composition of the loss function, in some other embodiments, after the scale invariance loss function is deformed, the first loss function is obtained as follows:

[0086]

[0087] Among them, g i =logd i -logd i gt , λ∈[0,1], N is the number of pixels in the training sample image, d i is the predicted depth value of the i-th pixel of the training sample image, d i gt is the actual depth value of the i-th pixel of the training sample image.

[0088] From the formula of the first loss function obtained after the deformation, it can be known that the first loss function is composed of variance loss and weighted square loss. Therefore, when adjusting the loss calculation of the initial neural network model, the focus of the loss function can be adjusted by setting the value of λ. For example, if the loss function needs to increase the focus on the variance in the loss, a larger λ value can be set to increase the focus on the variance in the loss. Preferably, the value of λ is 0.8.

[0089] In the depth value data of some captured images, most of the depth value data are close to the camera, and a few of the depth value data are far from the camera, which makes the depth value distribution of the captured image have a certain "long tail" phenomenon. When calculating the loss, the loss function will mainly perform back propagation based on the pixels with lower depth values, thereby reducing the weight of the pixels with higher depth values ​​when calculating the loss, and will also affect the accuracy of the neural network model in estimating the depth of the pixels with higher depth values. Therefore, in an embodiment of the present invention, the loss function of the initial neural network model is based on the first loss function, and a second loss function is added to increase the weight of the depth value of the pixel in the sample image.

[0090] Specifically, the second loss function is the product of the regression loss function and the weight coefficient, and the weight coefficient includes the attention weight, and the attention weight is used to adjust the weight of the depth of the pixel points in the training sample image during the loss calculation, so that the initial neural network model can pay attention to the pixels with larger depth values.

[0091] In order to avoid the gradient disappearance or explosion during the training process of the training sample images, in some further embodiments, the weight coefficient also includes a regularization coefficient, and the weight coefficient is the sum of the attention weight and the regularization coefficient.

[0092] Specifically, the second loss function is:

[0093]

[0094] Among them, α D is the attention weight, β D is the regularization coefficient, l(d i ,d i gt ) is the BerHu loss function.

[0095] In some embodiments, the attention weight α D is the actual depth value of the training sample image. It can be understood that when the distance between the pixel point in the training sample image and the camera device is relatively close, the attention weight α D Approaching 0.

[0096] Regularization coefficient β D The calculation formula is: Among them, the regularization coefficient is used to indicate that if the predicted depth value of the i-th pixel is close to the actual depth value, then β D approaches 0; if the predicted depth value of the initial neural network is inaccurate, then β D is close to 1. Therefore, through this loss function, for pixels with smaller depth values, i.e., closer distances (α D ->0) is inaccurate, it can still be updated according to the direction of gradient descent, thereby continuously optimizing the initial neural network model.

[0097] l(d i ,d i gt ) is the BerHu loss function, and its calculation formula is:

[0098]

[0099] Among them, L1(d i -d i gt ) is the L1 norm. Specifically, L2(d i -d i gt ) is the L2 norm. Specifically, t is a hyperparameter of the initial neural network model. In the embodiment of the present invention, t is set to 15% of the maximum loss value calculated in each batch of training sample images, that is, t = 0.15*max(|d i -d i gt |).

[0100] In some other embodiments, in order to balance the numerical relationship between the first loss function and the second loss function to be at the same level, the loss function is:

[0101] L=γl1+αl2,

[0102] Among them, γ and α are coefficients of the first loss function l1 and the second loss function l2 respectively.

[0103] In an embodiment of the present invention, when generating the preset convolutional neural network model, according to the depth value characteristics of the pixels in the image, a second loss function is added to the first loss function, wherein the second loss function is the product of the regression loss function and the weight coefficient, and the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of the pixels with larger depth values ​​in the training sample image during the loss calculation, so that the initial network model can pay attention to the pixels with larger depth values ​​in the training sample image when performing the loss calculation, thereby more accurately estimating the depth values ​​of the pixels with larger depth values ​​in the training sample image, and then, when estimating the image to be identified according to the trained preset convolutional neural network model, the depth map of the image to be identified can be accurately output, thereby improving the accuracy of the depth value prediction of the background pixels in the image to be identified.

[0104] It should be noted that the image estimation method and loss function of the embodiment of the present invention are used for Figure 1 or Figure 2 In addition to the convolutional neural network model shown, it can also be used for any other suitable convolutional neural network model.

[0105] See also Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device 500 provided for this embodiment. The electronic device 500 may include a processor 510 and a memory 520. The processor 510 and the memory 520 may communicate via a system bus. In addition, the memory 520 stores machine executable instructions, which may include instructions corresponding to the preset convolutional neural network model described in this embodiment, and instructions corresponding to training and using the preset convolutional neural network model. By reading and executing the machine executable instructions corresponding to the logic of the image estimation method in the memory 520, the processor 510 may execute the image estimation method described above.

[0106] The memory 520 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the memory 520 may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage media, or a combination thereof.

[0107] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0109] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0110] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0111] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image estimation method, characterized in that: The method comprises: Obtain an image to be recognized; Downsampling the input image to be recognized in sequence through multiple levels of downsampling convolutional layers in the encoder included in the preset convolutional neural network model, and outputting the feature map data obtained by the processing to the decoder included in the preset convolutional neural network model; The feature map data are sequentially upsampled by upsampling convolution layers of multiple levels in the decoder, and for a level in the decoder having an upsampling convolution layer and a plane fitting layer, the input feature map data are upsampled by the upsampling convolution layer in the level; the input feature map data are fitted to a plane by a plane fitting layer in the level to obtain local depth data of the input feature map data under the geometric constraints of the plane; wherein the size of the feature map data output by the plane fitting layer of the level is the same as the size of the feature map data output by the upsampling convolution layer of the level; the feature map data output by the upsampling convolution layer and the plane fitting layer of the level are fused and input to the next level of the decoder, wherein at least one level of the decoder includes an upsampling convolution layer and a plane fitting layer; Obtaining a depth map of the image to be recognized output by the decoder; Among them, the loss function of the preset convolutional neural network model includes a first loss function and a second loss function, the first loss function is a scale invariance loss function, the second loss function is the product of a regression loss function and a weight coefficient, the weight coefficient includes an attention weight, and the attention weight is used to adjust the weight of the depth of the pixel point in the training sample image during the loss calculation.

2. The method according to claim 1, characterized in that The attention weight is the actual depth value of the training sample image.

3. The method according to claim 2, characterized in that The weight coefficient also includes a regularization coefficient, and the weight coefficient is the sum of the attention weight and the regularization coefficient.

4. The method according to claim 3, characterized in that The first loss function is: ， in, , , N is the number of pixels in the training sample image, is the predicted depth value of the i-th pixel point of the training sample image, is the actual depth value of the i-th pixel of the training sample image.

5. The method according to claim 3, characterized in that: The first loss function is: in, , , N is the number of pixels in the training sample image, is the predicted depth value of the i-th pixel point of the training sample image, is the actual depth value of the i-th pixel of the training sample image.

6. The method according to claim 4 or 5, characterized in that: The second loss function is: ， in, is the attention weight, is the regularization coefficient, is the BerHu loss function.

7. The method according to claim 6, characterized in that The loss function is: ， in, , They are the first loss functions And the second loss function The coefficient of .

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer-executable program, and when the computer-executable program is executed by a processor, the computer is caused to execute the image estimation method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the image estimation method according to any one of claims 1 to 7 by executing the executable instructions.

Images