Training method and device for depth prediction model, medium and electronic device

Abstract

The present disclosure provides a training method for a depth prediction model, a training device for a depth prediction model, a computer-readable medium, and an electronic device, and relates to the field of image processing technology. The method includes: inputting an image sequence into a model to be trained to obtain a set of depth prediction images corresponding to the image sequence; obtaining a first depth prediction image corresponding to the current image and a second depth prediction image corresponding to the previous image of the current image in the image sequence from the depth prediction image set; calculating a target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; and updating the weights of the model to be trained based on the target loss function to obtain a depth prediction model. The present disclosure can optimize the continuity and consistency of the prediction results of the depth prediction model, so that the depth prediction results of the depth prediction model for stationary objects in the image sequence remain consistent, while making the transition of the prediction results for moving objects in the image sequence smooth.

Description

Background Art

[0002] In related technologies, there are usually two methods to build a deep learning model for depth prediction of images: one is to train the convolutional neural network based on binocular images, and the other is to train the network based on monocular images. Among them, the first method requires two cameras with fixed relative positions to simultaneously capture images of the scene to obtain binocular images. When training the network and making predictions based on the network, two pairs of frames of images need to be input; the second method only requires a camera with one perspective to capture images of the scene. When training the network and making predictions based on the network, only a single image needs to be input.

[0003] For monocular depth prediction models, two methods can usually be used to optimize the model: one is to optimize the monocular depth prediction model through a purely supervised method, that is, a single-frame annotation method; the other is to optimize the monocular depth prediction model based on the spatial consistency of image features between binocular images.

[0004] However, the monocular depth prediction model obtained based on the above optimization method is prone to discontinuous prediction results between frames in the depth prediction of videos with complex backgrounds or uneven lighting, which may cause the prediction results to flicker and jump in the video. Summary of the invention

[0005] The purpose of the present disclosure is to provide a training method for a depth prediction model, a training device for a depth prediction model, a computer-readable medium and an electronic device, thereby improving the continuity and consistency of a monocular depth prediction result to at least a certain extent, so that the depth prediction result of the model for a stationary object in a sequence remains unchanged, while ensuring a smooth transition of the depth prediction result for a moving object.

[0006] According to a first aspect of the present disclosure, a training method for a depth prediction model is provided, including: inputting an image sequence into a model to be trained to obtain a set of depth prediction images corresponding to the image sequence; the image sequence includes at least two images corresponding to a target scene acquired from the same perspective; obtaining, from the depth prediction image set, a first depth prediction image corresponding to a current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence; calculating a target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; and updating the weights of the model to be trained based on the target loss function to obtain a depth prediction model.

[0007] According to a second aspect of the present disclosure, a training device for a depth prediction model is provided, comprising: a sequence processing module, used to input an image sequence into a model to be trained to obtain a set of depth prediction images corresponding to the image sequence; the image sequence comprises at least two images corresponding to a target scene acquired from the same viewing angle; an image acquisition module, used to obtain, from the depth prediction image set, a first depth prediction image corresponding to a current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence; a loss calculation module, used to calculate a target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; and a weight updating module, used to perform weight updating on the model to be trained based on the target loss function to obtain a depth prediction model.

[0008] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0010] A training method for a depth prediction model provided by an embodiment of the present disclosure obtains a set of depth prediction images corresponding to the image sequence by inputting an image sequence into a model to be trained, and then, for a current image, obtains a first depth prediction image corresponding to the current image and a second depth prediction image corresponding to a migration image of the current image in the image sequence from the depth prediction image set, and calculates a target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image, and then updates the weights of the model to be trained based on the target loss function to obtain a depth prediction model. The present disclosure combines the depth prediction images corresponding to the image sequence obtained by inputting an image sequence into a model to be trained, and then utilizes the consistency of features and structures in each image in the image sequence to optimize the continuity and consistency of the prediction results of the depth prediction model, so that the depth prediction results of the depth prediction model for stationary objects in the image sequence remain consistent, and at the same time, the prediction results for moving objects in the image sequence are smoothly transitioned.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0015] Figure 3 A flowchart schematically illustrates a method for training a depth prediction model in an exemplary embodiment of the present disclosure;

[0016] Figure 4 A flowchart schematically illustrates a method for calculating a target loss function in an exemplary embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of the structure of a model to be trained in an exemplary embodiment of the present disclosure is schematically shown;

[0018] Figure 6 A flowchart schematically illustrates another method for training a depth prediction model in an exemplary embodiment of the present disclosure;

[0019] Figure 7 An image corresponding to a portrait scene containing a portrait in the image sequence is shown;

[0020] Figure 8 The depth prediction model obtained by the training method of the depth prediction model disclosed in the present invention is shown. Figure 7 The depth prediction image obtained by processing the image shown;

[0021] Fig. 9 The depth prediction model based on the traditional training method is shown Figure 7 The depth prediction image obtained by processing the image shown;

[0022] Fig.10 A schematic diagram schematically illustrates the composition of a training device for a depth prediction model in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0025] Figure 1 A schematic diagram of the system architecture of an exemplary application environment in which a method and apparatus for training a depth prediction model according to an embodiment of the present disclosure can be applied is shown.

[0026] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0027] The training method of the depth prediction model provided in the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the training device of the depth prediction model is generally arranged in the server 105. However, it is easy for those skilled in the art to understand that the training method of the depth prediction model provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, and 103, and accordingly, the training device of the depth prediction model can also be arranged in the terminal devices 101, 102, and 103, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, a user may capture an image sequence through a camera module included in the terminal devices 101, 102, and 103, and then send the image sequence to the server 105, and the server 105 trains the model to be trained through the training method of the depth prediction model provided in the embodiment of the present disclosure to obtain a depth prediction model.

[0028] An exemplary embodiment of the present disclosure provides an electronic device for implementing a training method for a depth prediction model, which may be Figure 1 The terminal device 101, 102, 103 or server 105 in the embodiment. The electronic device includes at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the training method of the depth prediction model by executing the executable instructions.

[0029] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0030] like Figure 2As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0031] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0032] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can be used to realize applications such as intelligent cognition of the mobile terminal 200, such as image recognition, face recognition, speech recognition, text understanding, etc. In an exemplary embodiment, a training method for a depth prediction model can be performed based on NPU. For example, a depth prediction image can be obtained by predicting an image sequence based on NPU.

[0033] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0034] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 200; modulation and demodulation processor can include modulator and demodulator; wireless communication module 260 can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 200. In some embodiments, antenna 1 of mobile terminal 200 is coupled with mobile communication module 250, and antenna 2 is coupled with wireless communication module 260, so that mobile terminal 200 can communicate with network and other devices through wireless communication technology.

[0035] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information. In an exemplary embodiment, the format of the images in the image sequence can be changed by the GPU, the display screen 290, and the application processor to obtain images that meet the format conditions.

[0036] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs. In an exemplary embodiment, images can be captured through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor to obtain an image sequence.

[0037] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be set in the camera module 291, and then the depth data corresponding to the target scene is collected based on the camera module equipped with the depth sensor to generate a depth annotation image corresponding to the current image.

[0038] The pressure sensor 2802 is used to sense the pressure signal and convert the pressure signal into an electrical signal. The gyroscope sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In addition, sensors with other functions can be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0039] For the monocular depth prediction model obtained by the relevant optimization method, when processing some image sequences with complex backgrounds, uneven lighting and other problems, it is easy to have the problem of discontinuous prediction results of each image in the image sequence when predicting the sequence. The main reason for these problems is that when training the relevant deep network model, the depth annotation image of a single image is usually used as a reference. In this case, since the invariance characteristics of image translation, rotation and other perspective transformations are not taken into account, it is easy for the monocular depth prediction model to have problems such as inconsistent prediction results of the previous image and the next image when predicting the image sequence, which leads to flickering and jumping phenomena.

[0040] Based on one or more of the above problems, this exemplary embodiment provides a method for training a depth prediction model. The method for training a depth prediction model can be applied to the above server 105, or to one or more of the above terminal devices 101, 102, and 103, which is not particularly limited in this exemplary embodiment. Figure 3 As shown, the training method of the depth prediction model may include the following steps S310 to S340:

[0041] In step S310, the image sequence is input into the model to be trained to obtain a set of depth prediction images corresponding to the image sequence.

[0042] The image sequence includes at least two images corresponding to the target scene acquired from the same viewing angle, and the image sequence generally refers to a plurality of images with continuity, and the target scene generally refers to a scene including a portrait. For example, it can be a video clip, or an image sequence with continuity of content extracted from a video. It should be noted that in order to obtain the image sequence, the sample data used to train the model to be trained can be read in batches. For example, a preset number of video frames with unchanged viewing angles can be read as an image sequence.

[0043] In an exemplary embodiment, the model to be trained may include a monocular depth prediction network such as BTS, MIDAS, or a monocular depth prediction network of other structures, and the present disclosure does not specifically limit this. Based on this, when the image sequence is input into the model to be trained, although the image sequence is input once, when performing depth prediction, the prediction is performed based on each image in the image sequence in turn to obtain the depth prediction image corresponding to each image, and then obtain the depth prediction image set corresponding to the image sequence.

[0044] It should be noted that, since different models to be trained support different image formats, the image sequence can be converted before being input into the model to be trained. For example, if the model to be trained supports images with a resolution of 480×640, each image in the image sequence can be scaled to obtain an image with a resolution of 480×640 before being input into the model to be trained, and then the obtained image sequence can be input into the model to be trained.

[0045] In an exemplary embodiment, in order to facilitate rapid convergence during training of the model to be trained, each sample image in the image sequence may be normalized before inputting the image sequence into the model to be trained. For example, the pixel values ​​corresponding to all pixels may be subtracted from 127.5 and then divided by 127.5 to normalize the pixel values ​​corresponding to all pixels to the range of [-1, 1].

[0046] In step S320, a first depth prediction image corresponding to the current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence are obtained from the depth prediction image set.

[0047] The current image refers to an image that is being calculated at the current moment during the training process of the model to be trained so as to update the weights of the model to be trained.

[0048] In an exemplary embodiment, after obtaining a set of depth prediction images corresponding to an image sequence, based on the consistency of features and structures in the image sequence, when training based on the current image, a first depth prediction image corresponding to the current image and a second depth prediction image corresponding to the previous image of the current image in the image sequence can be simultaneously obtained.

[0049] In step S330 , a target loss function corresponding to the current image is calculated based on the first depth prediction image and the second depth prediction image.

[0050] In an exemplary embodiment, after obtaining the first depth prediction image and the second depth prediction image, the target loss function can be calculated based on the consistency of features and structures in the current image and the previous image, and then the continuity and consistency of the prediction results of the model to be trained can be optimized according to the target loss function.

[0051] In an exemplary embodiment, when calculating the target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image, referring to Figure 4 As shown, the following steps S410 to S430 may be included:

[0052] In step S410, an absolute loss function corresponding to the current image is calculated based on the first depth prediction image.

[0053] Among them, the absolute loss function refers to a loss function calculated based on the current image itself or the prediction results, annotation results, etc. corresponding to the current image. The absolute loss function may include all types of loss functions determined according to the depth characteristics of the current image itself. For example, the absolute loss function may include a depth loss function determined based on the first depth prediction image corresponding to the current image and the depth annotation image corresponding to the current image; for another example, the absolute loss function may include a smoothing loss function determined based on the first depth prediction image corresponding to the current image. In addition, the absolute loss function may also include other loss functions determined according to the depth characteristics of the current image itself, and the present disclosure does not make any special limitations on this.

[0054] In an exemplary embodiment, when the absolute loss function includes a depth loss function, the depth loss function may be calculated based on a first depth prediction image obtained by predicting the current image using the to-be-trained model and a depth annotated image obtained by annotating the current image in advance. Specifically, the logarithmic difference between the depth values ​​corresponding to the first depth prediction image and the depth annotated image may be calculated for each pixel, and then the depth loss function corresponding to the current image may be calculated based on the logarithmic difference. The logarithmic difference g between the depth values ​​corresponding to each pixel of the first depth prediction image and the depth annotated image is i It can be calculated based on the following formula (1):

[0055]

[0056] Among them, d i represents the depth prediction value of the i-th pixel, Indicates the labeled depth value of the i-th pixel.

[0057] In an exemplary embodiment, when calculating the depth loss function corresponding to the current image based on the logarithmic difference, the depth loss function L d It can be calculated based on the following formula (2):

[0058]

[0059] Among them, g i represents the logarithmic difference of the depth value corresponding to the i-th pixel, T represents the number of pixels in the current image; α and λ are parameters set in advance. For example, α can be 10 and λ can be 0.5.

[0060] In an exemplary embodiment, the absolute loss function may also include a smooth loss function. In this case, the absolute loss function corresponding to the current image may be directly calculated based on the first depth prediction image. Specifically, based on the first depth prediction image, the lateral partial derivative and the longitudinal partial derivative of each pixel on the first depth prediction image may be calculated, and then the smooth loss function corresponding to the current image may be calculated based on the lateral partial derivative and the longitudinal partial derivative corresponding to each pixel.

[0061] In an exemplary embodiment, when the smoothing loss function corresponding to the current image is calculated based on the lateral partial derivative and the longitudinal partial derivative corresponding to each pixel, the smoothing loss function L smooth It can be calculated based on the following formula (3):

[0062]

[0063] Among them, d i,j Indicates the depth prediction value corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ x d i,j Represents the lateral partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ y d i,j Represents the longitudinal partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image.

[0064] In step S420, a relative loss function corresponding to the current image is calculated based on the first depth prediction image and the second depth prediction image.

[0065] Among them, the relative loss function value is based on the current image and the previous image of the current image in the image sequence or the prediction results, annotation results, etc. corresponding to the current image and the previous image, and the loss function is calculated. The relative loss function may include a loss function determined according to the depth characteristics of the current image itself and the depth characteristics of the previous image, which can characterize the depth change of the current image relative to the previous image.

[0066] It should be noted that the relative loss function includes all types of loss functions that can characterize the depth change of the current image relative to the previous image, and the present disclosure does not make any special limitation on this.

[0067] In an exemplary embodiment, when calculating the relative loss function, it can be calculated based on similarity. Specifically, the similarity can be calculated based on the first depth prediction image and the second depth prediction image, and then the relative loss function of the current image pair is calculated based on the similarity.

[0068] Among them, when calculating the similarity based on the first depth prediction image and the second depth prediction image, the image similarity between the first depth prediction image and the second depth prediction image can be directly calculated; the feature similarity of the feature layer output corresponding to the first depth prediction image and the second prediction image can also be calculated; the image similarity and the feature similarity can also be calculated at the same time. It should be noted that when calculating the similarity, if only one similarity is calculated, the similarity can be directly used as the relative loss function; if multiple similarities are calculated at the same time, the relative loss function can be calculated based on multiple similarities. During the specific calculation, different calculation formulas can be designed to calculate the relative loss function according to different application scenarios. For example, the multiple similarities calculated can be directly added to obtain the relative loss function.

[0069] In an exemplary embodiment, when the similarity includes both image similarity and feature similarity, the image similarity of the first depth prediction image and the second depth prediction image can be calculated first; then, when the model to be trained predicts the current image and the previous image, the first output corresponding to the current image output by the target feature layer of the model to be trained and the second output corresponding to the previous image output by the target feature layer of the model to be trained are respectively obtained, and then the feature similarity is calculated based on the first output and the second output.

[0070] Specifically, when calculating the image similarity between the first depth prediction image and the second depth prediction image, the first depth prediction image and the second depth prediction image may be downsampled first, and then the mean square error of all pixels after downsampling may be calculated. The above process of calculating the image similarity S may be expressed by the following formula (4):

[0071] S = MSE(w 0 , w 1 ), formula (4)

[0072] Among them, MSE means calculating the mean square error of all pixels in the image; w 0 Indicates the downsampling result corresponding to the current image; w 1 Indicates the downsampling result corresponding to the previous image.

[0073] Specifically, when calculating the feature similarity E based on the first output and the second output, it can be calculated by the following formula (5):

[0074] E = MSE(X 0 , X 1 ), formula (5)

[0075] Among them, MSE means calculating the mean square error of all pixels in the image; X 0 represents the first output of the target feature layer corresponding to the first depth prediction image; X 1 Represents the second output of the target feature layer corresponding to the second depth prediction image.

[0076] The following takes the target feature layer as the x2 feature layer and the x4 feature layer of the model to be trained as an example to describe in detail the process of calculating the relative loss function.

[0077] Reference Figure 5 As shown, in the model to be trained for realizing depth prediction of different regions by fusion of hierarchical features in the scene, an encoding branch and a decoding branch may be included. When performing depth prediction based on such a model to be trained, the output may be designed to include the following three parts: a depth prediction image, output 1 of the feature output layer (x2 feature layer) with a resolution of 2, and output 2 of the feature output layer (x4 feature layer) with a resolution of 4.

[0078] At this time, the image similarity and feature similarity can be calculated separately. For image similarity, the first depth prediction image and the second depth prediction image can be directly downsampled, and then the image similarity S can be calculated based on formula (4); for feature similarity, Figure 5 In the x2 feature layer, the first output a when the model to be trained processes the current image, and the second output b when the model to be trained processes the previous image, then based on the first output a and the second output b, the first feature similarity E is calculated by formula (5) 1 At the same time, for feature similarity, it is also necessary to obtain Figure 5 In the x4 feature layer, the first output c when the model to be trained processes the current image and the second output d when the model to be trained processes the previous image are calculated, and then based on the first output c and the second output d, the second feature similarity E is calculated by formula (5) 2 .

[0079] It should be noted that the feature output resolutions of the x2 feature layer and the x4 feature layer are both [batch_size, height, width, channel_number]. Among them, batch_size refers to the n value corresponding to the model to be trained, that is, the number of stacked input images; channel_number refers to the c value corresponding to the model to be trained, that is, the number of channels.

[0080] When the image similarity, the first feature similarity, and the second feature similarity are obtained, the image similarity, the first feature similarity, and the second feature similarity are added together to obtain a relative loss function.

[0081] In step S430, a target loss function corresponding to the current image is calculated based on the absolute loss function and the relative loss function.

[0082] In an exemplary embodiment, after obtaining the absolute loss function and the relative loss function, the target loss function corresponding to the current image can be calculated based on the absolute loss function and the relative loss function. It should be noted that, under the premise of different requirements, when calculating the target loss function, different weights can be set for different loss functions based on the impact of different loss functions on the requirements. In addition, when setting weights for the absolute loss function or the relative loss function, different weights can be set for different types of loss functions, or the same weights can be set. For example, when calculating the target loss function L, the weighted sum of the absolute loss function and the relative loss function can be calculated. For example, it can be calculated by the following formula (6):

[0083] L=α s (S+E 1 +E 2 )+α d *L d +α smooth *L smooth Formula (6)

[0084] Among them, for the depth loss function L in the absolute loss function d And the smooth loss function L smooth Different weights α were used d and α smooth ; For the relative loss function S+E 1 +E 2 The weight α is used s .

[0085] In step S340, the weight of the model to be trained is updated based on the target loss function to obtain a depth prediction model.

[0086] In an exemplary embodiment, after obtaining the target loss function, the weight of the training model can be updated based on the target loss function to obtain a depth prediction model. Figure 6As shown, the following steps are included: step S601, reading sample data in batches to obtain an image sequence; step S603, normalizing each image in the image sequence to obtain a normalized image sequence; step S605, inputting the normalized image sequence into the model to be trained for processing, and obtaining a set of depth prediction images corresponding to the image sequence; step S607, calculating a target loss function based on a first depth prediction image corresponding to a current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence; step S609, calculating a gradient based on the target loss function to update the weight of the model to be trained, and after multiple epochs, the target loss function converges to a certain range, thereby obtaining a depth prediction model.

[0087] In summary, in this exemplary embodiment, the consistency of features and structures in the image sequence can be used to effectively optimize the continuity and consistency of depth prediction, so that the depth prediction model can basically ensure that the depth prediction of stationary objects in the sequence remains unchanged, and the transition of depth prediction for moving objects can be smooth.

[0088] In addition, when the depth prediction model obtained based on the present disclosure is applied in the video blurring function, the problem of inaccurate or uneven blurring strength can be effectively reduced. Figure 7 The portrait scene including a portrait shown in FIG. 1 can be processed by the depth prediction model obtained by the depth prediction model training method disclosed in the present invention to obtain the following Figure 8 The depth prediction image 1 shown in FIG. 1 can be obtained by processing the depth prediction model obtained by using the traditional depth annotation image based on a single image as a reference. Fig. 9 The depth prediction image 2 shown. By comparing the depth prediction image 1 with the depth prediction image 2, it can be found that the problem of inaccurate or uneven blur strength has been improved.

[0089] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0090] For further reference, Fig.10 As shown, in the embodiment of this example, a training device 1000 for a depth prediction model is also provided, comprising a sequence processing module 1010, an image acquisition module 1020, a loss calculation module 1030 and a weight update module 1040. Among them:

[0091] The sequence processing module 1010 can be used to input the image sequence into the model to be trained to obtain a set of depth prediction images corresponding to the image sequence; the image sequence includes at least two images corresponding to the target scene acquired from the same viewing angle.

[0092] The image acquisition module 1020 may be configured to acquire, from the depth prediction image set, a first depth prediction image corresponding to the current image and a second depth prediction image corresponding to the previous image of the current image in the image sequence.

[0093] The loss calculation module 1030 may be configured to calculate a target loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image.

[0094] The weight updating module 1040 may be used to update the weights of the model to be trained based on the target loss function to obtain a depth prediction model.

[0095] In an exemplary embodiment, the loss calculation module 1030 can be used to calculate an absolute loss function corresponding to a current image based on a first depth prediction image; calculate a relative loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; and calculate a target loss function corresponding to the current image based on the absolute loss function and the relative loss function.

[0096] In an exemplary embodiment, the absolute loss function includes a depth loss function, and the loss calculation module 1030 can be used to calculate, for each pixel, the logarithmic difference between the depth value of the first depth prediction image and the depth value of the depth annotation image corresponding to the current image; and calculate the depth loss function corresponding to the current image based on the logarithmic difference.

[0097] In an exemplary embodiment, the absolute loss function also includes a smoothing loss function, and the loss calculation module 1030 can be used to calculate the lateral partial derivative and longitudinal partial derivative of each pixel on the first depth prediction image; based on the lateral partial derivative and longitudinal partial derivative corresponding to each pixel, the absolute loss function corresponding to the current image is calculated.

[0098] In an exemplary embodiment, the loss calculation module 1030 may be used to calculate a similarity based on the first depth prediction image and the second depth prediction image; and calculate a relative loss function corresponding to the current image based on the similarity.

[0099] In an exemplary embodiment, the similarity includes image similarity and feature similarity, and the loss calculation module 1030 can be used to calculate the image similarity of the first depth prediction image and the second depth prediction image; obtain the first output of the target feature layer corresponding to the first depth prediction image and the second output of the target feature layer corresponding to the second depth prediction image, and calculate the feature similarity based on the first output and the second output.

[0100] In an exemplary embodiment, the sequence processing module 1010 may be used to normalize all images included in the image sequence to obtain a normalized image sequence.

[0101] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0102] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0103] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figure 3 , Figure 4 , Figure 6 Any one or more steps in .

[0104] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0105] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0106] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0107] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0108] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a depth prediction model, characterized in that: include: Inputting an image sequence into a model to be trained to obtain a set of depth prediction images corresponding to the image sequence; The image sequence includes at least two images corresponding to the target scene acquired from the same viewing angle; Acquire, from the depth prediction image set, a first depth prediction image corresponding to a current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence; Calculate an absolute loss function corresponding to the current image based on the first depth prediction image; calculate a relative loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; calculate a target loss function corresponding to the current image based on the absolute loss function and the relative loss function; The weight of the model to be trained is updated based on the target loss function to obtain a depth prediction model.

2. The method according to claim 1, characterized in that The absolute loss function includes a depth loss function; The calculating the absolute loss function corresponding to the current image based on the first depth prediction image includes: For each pixel, calculating a logarithmic difference between a depth value of the first depth prediction image and a depth value of the depth annotation image corresponding to the current image; A depth loss function corresponding to the current image is calculated based on the logarithmic difference.

3. The method according to claim 2, characterized in that The absolute loss function also includes a smoothing loss function; The calculating the absolute loss function corresponding to the current image based on the first depth prediction image also includes: Calculating a lateral partial derivative and a longitudinal partial derivative of each pixel on the first depth prediction image; A smoothing loss function corresponding to the current image is calculated based on the lateral partial derivative and the longitudinal partial derivative corresponding to each pixel.

4. The method according to claim 1, characterized in that: The calculating the relative loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image includes: Calculating a similarity based on the first depth prediction image and the second depth prediction image; A relative loss function corresponding to the current image is calculated based on the similarity.

5. The method according to claim 4, characterized in that The similarity includes image similarity and feature similarity; The calculating feature similarity based on the first depth prediction image and the second depth prediction image includes: Calculating image similarity between the first depth prediction image and the second depth prediction image; A first output of a target feature layer corresponding to the first depth prediction image and a second output of the target feature layer corresponding to the second depth prediction image are obtained, and feature similarity is calculated based on the first output and the second output.

6. The method according to claim 1, characterized in that Before inputting the image sequence into the model to be trained, the method further includes: All images contained in the image sequence are normalized to obtain a normalized image sequence.

7. A training device for a depth prediction model, characterized in that: include: A sequence processing module, used to input an image sequence into a model to be trained to obtain a set of depth prediction images corresponding to the image sequence; the image sequence includes at least two images corresponding to the target scene acquired from the same viewing angle; An image acquisition module, configured to acquire, from the depth prediction image set, a first depth prediction image corresponding to a current image and a second depth prediction image corresponding to a previous image of the current image in the image sequence; A loss calculation module, configured to calculate an absolute loss function corresponding to the current image based on the first depth prediction image; calculate a relative loss function corresponding to the current image based on the first depth prediction image and the second depth prediction image; and calculate a target loss function corresponding to the current image based on the absolute loss function and the relative loss function; A weight updating module is used to update the weight of the model to be trained based on the target loss function to obtain a depth prediction model.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

Images