Training method and device for depth prediction model, medium and electronic device

Abstract

The present disclosure provides a depth prediction model training method, a depth prediction model training device, a computer-readable medium, and an electronic device, relating to the field of image processing technology. The method comprises: obtaining an object mask image corresponding to a sample image, inputting the sample image into a model to be trained to obtain a depth prediction image; calculating a target loss function corresponding to the sample image based on the object mask image and the depth prediction image; and updating the weights of the model to be trained based on the target loss function to obtain a depth prediction model. The present disclosure can effectively optimize depth prediction at object edges, resulting in more accurate depth prediction results at object edges.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a depth prediction model training method, a depth prediction model training device, a computer-readable medium, and an electronic device. Background Art

[0002] In related technologies, there are generally two approaches to building deep learning models for depth prediction from images: one is to train a convolutional neural network based on binocular images, and the other is to train the network based on monocular images. The first approach requires simultaneously capturing binocular images of the scene using two cameras with fixed relative viewpoints. This requires input of two pairs of images for both network training and network-based prediction. The second approach only requires capturing images of the scene using a camera with a single viewpoint. Only a single image is required for both network training and network-based prediction.

[0003] For monocular depth prediction models, the following methods can usually be used to optimize the model: first, optimize the monocular depth prediction model through a purely supervised method, that is, single-frame annotation; second, optimize the monocular depth prediction model based on the spatial consistency of image features between binocular images.

[0004] However, depth prediction based on the monocular depth prediction model obtained by the above optimization method may result in inaccurate prediction of the edges of objects contained in the scene. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a training method for a depth prediction model, a training device for a depth prediction model, a computer-readable medium, and an electronic device, thereby improving the accuracy of the depth prediction results of the object edge at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a method for training a depth prediction model is provided, comprising: obtaining an object mask image corresponding to a sample image, and inputting the sample image into a model to be trained to obtain a depth prediction image; calculating a target loss function corresponding to the sample image based on the object mask image and the depth prediction image; and updating the weights of the model to be trained based on the target loss function to obtain a depth prediction model.

[0007] According to a second aspect of the present disclosure, a training device for a depth prediction model is provided, comprising: an image acquisition module for acquiring an object mask image corresponding to a sample image, and inputting the sample image into a model to be trained to obtain a depth prediction image; a loss calculation module for calculating a target loss function corresponding to the sample image based on the object mask image and the depth prediction image; and a weight update module for updating the weights of the model to be trained based on the target loss function to obtain a depth prediction model.

[0008] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which, when the one or more programs are executed by one or more processors, enables the one or more processors to implement the above-mentioned method.

[0010] An embodiment of the present disclosure provides a method for training a depth prediction model. This method obtains an object mask image corresponding to a sample image and inputs the sample image into a model to be trained to obtain a depth prediction image. The method then calculates a target loss function corresponding to the sample image based on the object mask image and the depth prediction image. The method then updates the weights of the model to be trained based on the target loss function to obtain a depth prediction model. The present disclosure utilizes the object mask image to effectively optimize depth prediction at object edges, resulting in more accurate depth prediction results at object edges.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0013] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure may be applied;

[0015] Figure 3 A flowchart schematically illustrates a method for training a depth prediction model in an exemplary embodiment of the present disclosure;

[0016] Figure 4 A sample image corresponding to a portrait scene containing a portrait is shown;

[0017] Figure 5 Shown Figure 4 The object mask image corresponding to the sample image shown;

[0018] Figure 6 A schematic diagram of the structure of a model to be trained in an exemplary embodiment of the present disclosure is schematically shown;

[0019] Figure 7 A flowchart schematically illustrates a method for calculating a target loss function in an exemplary embodiment of the present disclosure;

[0020] Figure 8 A flowchart schematically illustrates another method for training a depth prediction model in an exemplary embodiment of the present disclosure;

[0021] Figure 9 Shows the Figure 4 An object segmentation image obtained by performing image segmentation on the sample image shown;

[0022] Figure 10 The depth prediction model obtained by the training method of the depth prediction model disclosed in the present invention is shown. Figure 4 The depth prediction image obtained by processing the sample image shown;

[0023] Figure 11 The depth prediction model based on the traditional training method is shown. Figure 4 The depth prediction image obtained by processing the sample image shown;

[0024] Figure 12 The following is a schematic diagram schematically illustrating the composition of a training device for a depth prediction model in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] Figure 1A schematic diagram of the system architecture of an exemplary application environment in which a method and apparatus for training a depth prediction model according to an embodiment of the present disclosure can be applied is shown.

[0028] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as needed. For example, the server 105 may be a server cluster consisting of multiple servers.

[0029] The training method of the depth prediction model provided in the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the training device of the depth prediction model is generally provided in the server 105. However, it is easy for those skilled in the art to understand that the training method of the depth prediction model provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, and 103, and accordingly, the training device of the depth prediction model can also be provided in the terminal devices 101, 102, and 103, and this is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, the user may collect sample images through the camera module included in the terminal devices 101, 102, and 103, and then send the sample images to the server 105. The server 105 trains the model to be trained through the training method of the depth prediction model provided in the embodiment of the present disclosure to obtain a depth prediction model.

[0030] An exemplary embodiment of the present disclosure provides an electronic device for implementing a training method for a depth prediction model, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory being used to store executable instructions of the processor, and the processor being configured to execute the training method of the depth prediction model by executing the executable instructions.

[0031] Below is Figure 2 The structure of the electronic device is exemplarily described by taking the mobile terminal 200 in FIG. 1 as an example. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2The structure in FIG. 2 can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0032] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, and the like.

[0033] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0034] The NPU is a neural network (NN) computing processor that rapidly processes input information by drawing on biological neural network structures, such as the transmission patterns between neurons in the human brain, and can also continuously self-learn. The NPU can be used to implement intelligent cognition applications of the mobile terminal 200, such as image recognition, face recognition, speech recognition, and text understanding. In an exemplary embodiment, a depth prediction model training method can be performed based on the NPU. For example, a sample image can be predicted based on the NPU to obtain a depth prediction image.

[0035] The processor 210 is provided with a memory that can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution of the instructions is controlled by the processor 210.

[0036] The charging management module 240 is configured to receive charging input from a charger. The power management module 241 is configured to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display 290, the camera module 291, and the wireless communication module 260.

[0037] The wireless communication functions of mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide wireless communication solutions for mobile terminal 200, including 2G / 3G / 4G / 5G; the modem processor can include a modulator and a demodulator; and wireless communication module 260 can provide wireless communication solutions for mobile terminal 200, including wireless local area networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks) and Bluetooth (BT). In some embodiments, antenna 1 of mobile terminal 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling mobile terminal 200 to communicate with networks and other devices via wireless communication technologies.

[0038] Mobile terminal 200 implements display functions through a GPU, display screen 290, and an application processor. The GPU is a microprocessor for image processing and is connected to the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or change display information. In an exemplary embodiment, the GPU, display screen 290, and application processor can be used to enhance the sample image to obtain an enhanced sample image.

[0039] Mobile terminal 200 can implement a camera function through an ISP, camera module 291, video codec, GPU, display 290, and application processor. The ISP processes data fed back by camera module 291; camera module 291 captures still images or video; the digital signal processor processes digital signals, including digital image signals and other digital signals; and the video codec compresses or decompresses digital video. Mobile terminal 200 may also support one or more video codecs. In an exemplary embodiment, the ISP, camera module 291, video codec, GPU, display 290, and application processor can be used to capture images and obtain sample images.

[0040] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be set in the camera module 291, and then based on the camera module equipped with the depth sensor, the depth data corresponding to the target scene is collected to generate a depth-annotated image corresponding to the current image.

[0041] Pressure sensor 2802 is used to sense pressure signals and convert them into electrical signals. Gyroscope sensor 2803 can be used to determine the motion posture of mobile terminal 200. Furthermore, other functional sensors can be provided in sensor module 280 as needed, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0042] For the monocular depth prediction model obtained by the relevant optimization method, when processing some images containing specific objects with problems such as simple texture or uneven lighting, the loss function used in the training of the depth prediction model is usually based on a single depth-labeled image as a reference, and the information reflected by the depth-labeled image is mainly the hierarchical relationship of the scene. The characteristics of this hierarchical relationship make the feature layer in the trained depth prediction model more biased towards the hierarchical characteristics of the scene, which will cause some local texture, color and other features in the image to be lost, and then lead to problems such as the depth prediction results being not smooth enough in places where the color and texture are relatively consistent, and the edge depth prediction being not accurate enough.

[0043] Based on one or more of the above problems, this exemplary embodiment provides a method for training a depth prediction model. The method for training a depth prediction model can be applied to the above server 105, or to one or more of the above terminal devices 101, 102, and 103, and this exemplary embodiment does not specifically limit this. Figure 3 As shown, the training method of the depth prediction model may include the following steps S310 to S330:

[0044] In step S310, an object mask image corresponding to the sample image is obtained, and the sample image is input into the model to be trained to obtain a depth prediction image.

[0045] The object mask image corresponding to the sample image may include a mask image for representing the object in the area where the sample image is located. Figure 4 As shown, the sample image contains human body. If the object is human body, the following can be obtained: Figure 5 The object mask image shown. The model to be trained may include a monocular depth prediction network such as BTS, MIDAS, or a monocular depth prediction network of other structures, and this disclosure does not specifically limit this. It should be noted that when inputting sample images into the model to be trained, they can be input in batches.

[0046] For example, refer to Figure 6 As shown, the model to be trained for implementing depth prediction of different regions by fusion of hierarchical features in a scene may include an encoding branch and a decoding branch.

[0047] In addition, since different models to be trained support different image formats, the sample images can be converted before being input into the model to be trained. For example, if the model to be trained supports images with a resolution of 480×640, the sample images can be scaled to obtain a sample image with a resolution of 480×640 before being input into the model to be trained.

[0048] In an exemplary embodiment, to facilitate rapid convergence during training of the model to be trained, the sample image may be normalized before being input into the model to be trained. For example, the pixel values ​​corresponding to all pixels may be subtracted from 127.5 and then divided by 127.5 to normalize the pixel values ​​corresponding to all pixels to the range of [-1, 1].

[0049] In an exemplary embodiment, to increase the diversity of sample images, the sample images may be enhanced before being input into the training model to obtain enhanced sample images. Image enhancement may include performing geometric transformations, color transformations, and other enhancement processes on the sample images. For example, the sample images may be subjected to a random 90-degree counterclockwise rotation.

[0050] In step S320 , a target loss function corresponding to the sample image is calculated based on the object mask image and the depth prediction image.

[0051] In an exemplary embodiment, after obtaining the object mask image and the depth prediction image, the object edge information and the depth prediction image contained in the object mask image can be used to calculate the target loss function, and then the accuracy of the depth prediction results of the model to be trained at the object edge can be optimized according to the target loss function.

[0052] In an exemplary embodiment, when calculating the target loss function corresponding to the sample image based on the object mask image and the depth prediction image, referring to Figure 7 As shown, the following steps S710 to S730 may be included:

[0053] In step S710 , a first loss function corresponding to the sample image is calculated based on the depth prediction image.

[0054] Among them, the first loss function refers to a loss function calculated based on data containing depth information such as a depth prediction image and a depth annotation image. The first loss function may include all types of loss functions determined according to the depth characteristics corresponding to the sample image. For example, the first loss function may include a depth loss function calculated based on the depth prediction image and the depth annotation image corresponding to the sample image; for example, the first loss function may also include a smoothing loss function determined based on the depth prediction image corresponding to the sample image itself. In addition, the first loss function may also include other loss functions determined based on the depth characteristics of the sample image itself (predicted depth data or annotated depth data, etc.), and the present disclosure does not make any special restrictions on this.

[0055] In an exemplary embodiment, when the first loss function includes a depth loss function, the depth loss function can be calculated based on a depth prediction image obtained by predicting the sample image by the to-be-trained model and a depth annotation image obtained by depth-annotating the sample image in advance. Specifically, the logarithmic difference between the depth values ​​corresponding to the depth prediction image and the depth annotation image can be calculated for each pixel, and then the depth loss function corresponding to the sample image can be calculated based on the logarithmic difference. The logarithmic difference g between the depth values ​​corresponding to each pixel of the depth prediction image and the depth annotation image is i It can be calculated based on the following formula (1):

[0056]

[0057] Among them, d i represents the depth prediction value of the i-th pixel, Indicates the labeled depth value of the i-th pixel.

[0058] In an exemplary embodiment, when calculating the depth loss function corresponding to the sample image based on the logarithmic difference, the depth loss function L d It can be calculated based on the following formula (2):

[0059]

[0060] Among them, g i represents the logarithmic difference of the depth value corresponding to the i-th pixel, T represents the number of pixels in the current image, and α and λ are pre-set parameters. For example, α can be 10 and λ can be 0.5.

[0061] In an exemplary embodiment, the first loss function may further include a smoothing loss function. In this case, the first loss function corresponding to the sample image may be calculated directly based on the depth prediction image. Specifically, based on the depth prediction image, the first horizontal partial derivative and the first vertical partial derivative of each pixel on the depth prediction image may be calculated, and then the smoothing loss function corresponding to the sample image may be calculated based on the first horizontal partial derivative and the first vertical partial derivative corresponding to each pixel.

[0062] In an exemplary embodiment, when the smoothing loss function corresponding to the sample image is calculated based on the first transverse partial derivative and the first longitudinal partial derivative corresponding to each pixel, the smoothing loss function L smooth It can be calculated based on the following formula (3):

[0063]

[0064] Among them, d i，j Indicates the depth prediction value corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ x di，j Represents the lateral partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ y d i，j Represents the longitudinal partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image.

[0065] In step S720 , a second loss function corresponding to the sample image is calculated based on the depth prediction image and the object mask image.

[0066] Among them, the second loss function includes a loss function calculated based on data containing depth information such as a depth prediction image and a depth annotation image, and an object mask image. The second loss function may include various types of loss functions determined based on the depth characteristics of the sample image itself and the data contained in the object mask image.

[0067] In an exemplary embodiment, the second loss function may include an edge loss function. In this case, the second loss function corresponding to the sample image may be directly calculated based on the depth prediction image and the object mask image. Specifically, the first lateral partial derivative and the first longitudinal partial derivative of each pixel on the depth prediction image may be calculated first, and then the second lateral partial derivative and the second longitudinal partial derivative of each pixel on the object mask image may be calculated. Thereafter, the edge loss function is calculated based on the obtained first lateral partial derivative, first longitudinal partial derivative, second lateral partial derivative, and second longitudinal partial derivative.

[0068] In an exemplary embodiment, when calculating the edge loss function based on the first transverse partial derivative, the first longitudinal partial derivative, the second transverse partial derivative, and the second longitudinal partial derivative, the edge loss function may be calculated based on the following formula (4):

[0069]

[0070] Among them, d i，j Indicates the depth prediction value corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ x d i，j Represents the first lateral partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image; δ y d i，j Represents the first longitudinal partial derivative corresponding to the pixel with coordinates (i, j) in the depth prediction image; seg i，j Indicates the pixel value corresponding to the pixel with coordinates (i, j) in the object mask image; δ x seg i，j represents the second lateral partial derivative corresponding to the pixel with coordinates (i, j) in the object mask image; δ y seg i，jRepresents the second longitudinal partial derivative corresponding to the pixel with coordinates (i, j) in the object mask image; N is the number of pixels.

[0071] In step S730 , a target loss function corresponding to the sample image is calculated according to the first loss function and the second loss function.

[0072] In an exemplary embodiment, after obtaining the first loss function and the second loss function, the target loss function corresponding to the sample image can be calculated based on the first loss function and the second loss function. It should be noted that, under the premise of different requirements, when calculating the target loss function, different weights can be set for different loss functions based on the impact of different loss functions on the requirements. In addition, when setting weights for absolute loss functions or relative loss functions, different weights can be set for different types of loss functions, or the same weights can be set. For example, when calculating the target loss function L, the weighted sum of the absolute loss function and the relative loss function can be calculated. For example, it can be calculated by the following formula (5):

[0073] L=α e *L e +α d *L d +α smoom *L smooth Formula (5) Among them, the depth loss function L in the first loss function d and smooth loss function L smooth Different weights α are used d and α smooth ; Weight α is used for the second loss function e .

[0074] In step S330, the weights of the model to be trained are updated based on the target loss function to obtain a depth prediction model.

[0075] In an exemplary embodiment, after obtaining the target loss function, the weight of the training model can be updated based on the target loss function to obtain a depth prediction model. Figure 8As shown, the following steps are included: step S801, reading sample images in batches; step S803, normalizing the sample images to obtain normalized sample images; step S805, obtaining the object mask image corresponding to the sample image; step S807, inputting the normalized sample image into the model to be trained for processing to obtain the depth prediction image corresponding to the sample image; step S809, calculating the target loss function based on the depth prediction image corresponding to the sample image and the object mask image corresponding to the sample image; step S811, updating the weight of the model to be trained based on the target calculation gradient. After multiple epochs, the target loss function converges to a certain range, and then the depth prediction model is obtained.

[0076] In addition, when obtaining the object mask image corresponding to the sample image, the sample image can be segmented first to obtain the object segmentation image, and then different values ​​are assigned to different areas in the object segmentation image based on a preset threshold to obtain the object mask image. For example, when the object is a portrait, for example Figure 4 The sample image shown in can be processed by the portrait segmentation model to obtain a segmented image, such as Figure 9 As shown in the figure. In the segmented image, each pixel value is in the range of 0-255. At this time, the image can be assigned a value based on the preset threshold value of 127.5. Specifically, the pixels with a value less than or equal to 127.5 can be reassigned to 0, and the pixels with a value greater than 127.5 can be reassigned to 255. The resulting binary image is the object mask image, as shown in the figure. Figure 5 shown.

[0077] It should be noted that, in addition to the above methods, methods such as manual annotation may also be used to obtain the object mask corresponding to the sample image, and the present disclosure does not impose any special limitation on this.

[0078] In summary, in this exemplary embodiment, utilizing the smoothing information of edge or region segments can effectively optimize depth prediction at the edges or regions of a portrait, resulting in smoother depth prediction in areas with relatively consistent color and texture, and more accurate depth prediction at edges. Furthermore, this optimization method can effectively reduce the issue of inaccurate or uneven blurring in the blurring function.

[0079] For example, for Figure 4 The sample images and Figure 5 The object mask image shown in FIG. 1 can be processed by the depth prediction model obtained by the depth prediction model training method disclosed in the present invention to obtain the following: Figure 10 The depth prediction image 1 shown in FIG. 1 is processed according to the depth prediction model obtained by using the traditional depth annotation image based on a single image as a reference to obtain the following Figure 11Depth prediction image 2 is shown. Comparing depth prediction image 1 and depth prediction image 2, it can be seen that the depth prediction is smoother in areas where the color and texture are relatively consistent, and the depth prediction is more accurate at the edges, and the problem of inaccurate or uneven blurring is improved.

[0080] It should be noted that the above figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0081] For further reference, Figure 12 As shown, in the embodiment of this example, a training device 1200 for a depth prediction model is also provided, comprising an image acquisition module 1210, a loss calculation module 1220, and a weight update module 1230. Among them:

[0082] The image acquisition module 1210 can be used to obtain an object mask image corresponding to a sample image, and input the sample image into the model to be trained to obtain a depth prediction image.

[0083] The loss calculation module 1220 may be configured to calculate a target loss function corresponding to the sample image based on the object mask image and the depth prediction image.

[0084] The weight updating module 1230 may be used to update the weights of the model to be trained based on the target loss function to obtain a depth prediction model.

[0085] In an exemplary embodiment, the loss calculation module 1220 can be used to calculate a first loss function corresponding to the sample image based on the depth prediction image; calculate a second loss function corresponding to the sample image based on the depth prediction image and the object mask image; and calculate a target loss function corresponding to the sample image based on the first loss function and the second loss function.

[0086] In an exemplary embodiment, the first loss function includes a depth loss function, and the loss calculation module 1220 can be used to calculate the logarithmic difference between the depth value of the depth prediction image and the depth value of the depth annotation image corresponding to the sample image for each pixel; and calculate the depth loss function corresponding to the sample image based on the logarithmic difference.

[0087] In an exemplary embodiment, the first loss function also includes a smoothing loss function, and the loss calculation module 1220 can be used to calculate the first horizontal partial derivative and the first vertical partial derivative of each pixel on the depth prediction image; based on the first horizontal partial derivative and the first vertical partial derivative corresponding to each pixel, the smoothing loss function corresponding to the sample image is calculated.

[0088] In an exemplary embodiment, the second loss function includes an edge loss function, and the loss calculation module 1220 can be used to calculate the first lateral partial derivative and the first longitudinal partial derivative of each pixel on the depth prediction image; calculate the second lateral partial derivative and the second longitudinal partial derivative of each pixel on the object mask image; and calculate the edge loss function based on the first lateral partial derivative, the first longitudinal partial derivative, the second lateral partial derivative, and the second longitudinal partial derivative.

[0089] In an exemplary embodiment, the image acquisition module 1210 may be configured to perform image segmentation on the sample image to obtain an object segmentation image; and assign a value to the object segmentation image based on a preset threshold to obtain an object mask image.

[0090] In an exemplary embodiment, the image acquisition module 1210 may be configured to perform image enhancement on the sample image to obtain an enhanced sample image.

[0091] The specific details of each module in the above device have been described in detail in the implementation method part. The undisclosed details can be found in the implementation method part, so they will not be repeated here.

[0092] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0093] The exemplary embodiments of the present disclosure further provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present disclosure, for example, Figure 3 、 Figure 7 and Figure 8 Any one or more steps in .

[0094] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0095] In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0096] In addition, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0097] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0098] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a depth prediction model, characterized in that: include: Performing image segmentation on the sample image to obtain an object segmentation image, assigning a value to the object segmentation image based on a preset threshold to obtain an object mask image, and inputting the sample image into the model to be trained to obtain a depth prediction image; Calculating a first loss function corresponding to the sample image based on the depth prediction image; calculating a second loss function corresponding to the sample image based on the depth prediction image and the object mask image; and calculating a target loss function corresponding to the sample image based on the first loss function and the second loss function; Performing weight updates on the model to be trained based on the target loss function to obtain a depth prediction model; The second loss function includes an edge loss function; and the step of calculating the second loss function corresponding to the sample image based on the depth prediction image and the object mask image includes: Calculating a first horizontal partial derivative and a first vertical partial derivative of each pixel on the depth prediction image; Calculating a second transverse partial derivative and a second longitudinal partial derivative of each pixel on the object mask image; The edge loss function is calculated based on the first transverse partial derivative, the first longitudinal partial derivative, the second transverse partial derivative, and the second longitudinal partial derivative.

2. The method according to claim 1, characterized in that The first loss function includes a depth loss function; The calculating a first loss function corresponding to the sample image based on the depth prediction image includes: For each pixel, calculating the logarithmic difference between the depth value of the depth prediction image and the depth value of the depth annotation image corresponding to the sample image; A depth loss function corresponding to the sample image is calculated based on the logarithmic difference.

3. The method according to claim 2, characterized in that The first loss function also includes a smoothing loss function; The calculating of the first loss function corresponding to the sample image based on the depth prediction image further includes: Calculating a first horizontal partial derivative and a first vertical partial derivative of each pixel on the depth prediction image; A smoothing loss function corresponding to the sample image is calculated based on the first transverse partial derivative and the first longitudinal partial derivative corresponding to each pixel.

4. The method according to claim 1, wherein Before inputting the sample image into the to-be-trained model to obtain the depth prediction image, the method further includes: Perform image enhancement on the sample image to obtain an enhanced sample image.

5. A training device for a depth prediction model, characterized in that: include: An image acquisition module is configured to perform image segmentation on a sample image to obtain an object segmentation image, assign a value to the object segmentation image based on a preset threshold to obtain an object mask image, and input the sample image into a model to be trained to obtain a depth prediction image; a loss calculation module, configured to calculate a first loss function corresponding to the sample image based on the depth prediction image; calculate a second loss function corresponding to the sample image based on the depth prediction image and the object mask image; and calculate a target loss function corresponding to the sample image based on the first loss function and the second loss function; A weight updating module, configured to update the weight of the model to be trained based on the target loss function to obtain a depth prediction model; Among them, the second loss function includes an edge loss function; the loss calculation module is used to calculate the first horizontal partial derivative and the first vertical partial derivative of each pixel on the depth prediction image; calculate the second horizontal partial derivative and the second vertical partial derivative of each pixel on the object mask image; and calculate the edge loss function based on the first horizontal partial derivative, the first vertical partial derivative, the second horizontal partial derivative and the second vertical partial derivative.

6. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 4 by executing the executable instructions.

Images