Model training method, image depth prediction method and device, equipment, and medium
By obtaining the training labels and auxiliary information of the reference image, and using the auxiliary encoder and the actual encoder to alternate training, the instability problem of monocular depth estimation task is solved, the stability and efficiency of model training are improved, and the model is effectively converged.
Patent Information
- Application Number
- CN202111052247.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-08
AI Technical Summary
In the prior art, the monocular depth estimation task is unstable, which leads to long-term training, low training efficiency and may lead to non-convergence of training.
By obtaining the training labels and auxiliary information of the reference image, the auxiliary encoder and the actual encoder are used to train alternately to obtain the depth estimation model, and improve the stability and efficiency of model training.
Improve the stability and efficiency of model training, ensure effective convergence of the model, and improve model performance.
Smart Images

Figure CN113743517B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular, to a model training method, a model training device, an image depth prediction method, an image depth prediction device, an electronic device, and a computer-readable storage medium. Background Art
[0002] Depth estimation is widely used in fields such as autonomous driving and augmented reality. In order to avoid the high cost of using professional equipment to collect depth data, related technologies use the depth estimated by traditional methods as the training target of convolutional neural networks to perform depth estimation.
[0003] In the above method, due to the instability of the monocular depth estimation task, the related model training process takes a long time, the training efficiency is low, and it may cause the training to not converge.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide a model training method, an image depth prediction method and device, an electronic device, and a storage medium, thereby overcoming the problem of low model training efficiency caused by the limitations and defects of related technologies to at least a certain extent.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, a model training method is provided, including: obtaining a training label corresponding to a reference image, and obtaining auxiliary information of the reference image; performing feature extraction on the reference image and the auxiliary information using an auxiliary encoder to determine an auxiliary prediction result, and performing feature extraction on the reference image based on an actual encoder to determine an actual prediction result; alternately training the auxiliary encoder and the actual encoder according to the auxiliary prediction result, the actual prediction result and the training label, and training the decoder to obtain a depth estimation model.
[0008] According to one aspect of the present disclosure, a method for predicting image depth is provided, comprising: inputting an image to be processed into an actual encoder in a depth estimation model to obtain actual features; decoding and predicting the actual features according to a decoder in the depth estimation model to determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to any one of the model training methods described above.
[0009] According to one aspect of the present disclosure, a model training device is provided, including: an auxiliary information acquisition module, used to obtain a training label corresponding to a reference image, and obtain auxiliary information of the reference image; a feature extraction module, used to perform feature extraction on the reference image and the auxiliary information using an auxiliary encoder to determine an auxiliary prediction result, and perform feature extraction on the reference image based on an actual encoder to determine an actual prediction result; an alternating training module, used to perform alternating training on the auxiliary encoder and the actual encoder according to the auxiliary prediction result, the actual prediction result and the training label, and train the decoder to obtain a depth estimation model.
[0010] According to one aspect of the present disclosure, there is provided an image depth prediction device, comprising: a feature acquisition module, used to input an image to be processed into an actual encoder in a depth estimation model to obtain actual features; a depth estimation module, used to decode and predict the actual features according to a decoder in the depth estimation model, and determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to any one of the model training methods described above.
[0011] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned model training methods by executing the executable instructions.
[0012] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model training method described in any one of the above is implemented.
[0013] In the model training method, model training device, image depth prediction method, image depth prediction device, electronic device and computer-readable storage medium provided in the embodiments of the present disclosure, on the one hand, in the training process, auxiliary information is added as guiding information, and the auxiliary encoder is trained according to the auxiliary information, and then the actual encoder is guided to be trained according to the auxiliary encoder, thereby avoiding the limitation of requiring training skills, reducing the difficulty of model training, and improving the stability of model training. On the other hand, during training, the auxiliary encoder and the actual encoder are trained alternately, and the actual encoder is trained according to the auxiliary trainer with a similar structure to the actual encoder, so that the actual encoder is prompted to learn the feature extraction mode of the auxiliary information, thereby enhancing the feature extraction capability of the actual encoder, improving the training efficiency of the actual encoder, and being able to ensure the effective convergence of the model and improve the model performance.
[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0016] Figure 1 A schematic diagram shows an application scenario in which the model training method or image depth prediction method of an embodiment of the present disclosure can be applied.
[0017] Figure 2 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown.
[0018] Figure 3 A schematic diagram schematically illustrates a model training method in an embodiment of the present disclosure.
[0019] Figure 4 A schematic diagram schematically illustrates the processing process of the encoder in an embodiment of the present disclosure.
[0020] Figure 5 The flowchart of the model training in the embodiment of the present disclosure is schematically shown.
[0021] Figure 6 A schematic diagram for obtaining a depth estimation model in an embodiment of the present disclosure is schematically shown.
[0022] Figure 7 A flowchart of an image depth prediction method in an embodiment of the present disclosure is schematically shown.
[0023] Figure 8 A block diagram of a model training device in an embodiment of the present disclosure is schematically shown.
[0024] Fig. 9 A block diagram of an image depth prediction device in an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0027] In related technologies, depth estimation can be accomplished by the following method: take a group of photos of a scene or an object, select one of them as the target frame, and the others as reference frames. First, restore the camera parameters from multiple photos through the camera posture estimation algorithm, and then use three-dimensional reconstruction or key point matching methods to infer the depth spectrum of the target frame from the reference frame information. The advantage of this type of traditional method is high accuracy, but it is generally time-consuming and has requirements for the subject, number of shots, and shooting equipment, making it difficult to be universal.
[0028] With the increasing maturity of deep learning algorithms represented by convolutional neural network models, the industry already has a large number of depth estimation algorithms based on convolutional neural network models. This type of algorithm generally uses operations such as convolution to directly fit the depth value after inputting the image, which is simple and efficient overall. However, in order for the convolutional neural network model to have better performance, a large amount of training data must be collected in advance, that is, a set of images with known depth. At present, there are generally two methods for constructing such training sets. One is to directly use special equipment such as lidar to simultaneously obtain the original image and the corresponding depth. The other is to use traditional methods to predict the depth values of a large number of images in advance, and then let the convolutional neural network model fit the results of the traditional method. However, the cost of using special equipment to collect depth is high, and the difficulty of training the depth estimation convolutional neural network model is also high.
[0029] In order to solve the above technical problems, a model training method is provided in an embodiment of the present disclosure, which can be applied to scenes requiring high-precision depth estimation, such as light and shadow effects.
[0030] Figure 1 A schematic diagram shows an application scenario in which the model training method and device or the image depth prediction method and device of the embodiments of the present disclosure can be applied.
[0031] This model training method can be applied to the training scenario of the model for depth estimation of images. Figure 1 As shown in , it can be specifically applied to the process of using the client 101 to perform depth estimation on the target object 102 received on the client. Among them, the client can be various types of devices with computing functions, such as smart phones, tablet computers, desktop computers, vehicle-mounted devices, wearable devices, etc. The target object 102 can be any type of image in various scenes, it can be an image obtained by shooting, or it can be an image obtained from the network or other terminals, and the type of image is not limited. The client 101 can use the depth spectrum data of the target image as a training target, and add auxiliary information as guiding information during the training process, alternately train the actual encoder and the auxiliary encoder, and update the model parameters of the decoder at the same time, so as to improve the training speed of the actual encoder and the decoder. When the client 101 detects the object to be processed 103 that needs to be estimated in depth, the trained depth estimation model can be used to determine the depth value of the object to be processed 103.
[0032] It should be noted that the model training method and image depth prediction method provided in the embodiments of the present disclosure can be completely executed by the client. Accordingly, the model training device and image depth prediction device can be set in the client.
[0033] Figure 2 Schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. The first terminal or the second terminal of the present disclosure may be configured as follows Figure 2 The form of the electronic device shown, however, needs to be explained. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0034] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the method of the exemplary embodiment of the present disclosure.
[0035] Specifically, Figure 2As shown, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.
[0036] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0037] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.
[0038] The USB interface 230 is an interface that complies with the USB standard specification, and specifically can be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the electronic device 200, and can also be used to transmit data between the electronic device 200 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, etc.
[0039] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.
[0040] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, the baseband processor, and the like.
[0041] The mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 200 .
[0042] The wireless communication module 260 can provide wireless communication solutions for application in the electronic device 200, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc.
[0043] The electronic device 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image blurring, connecting the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.
[0044] The electronic device 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor. In some embodiments, the electronic device 200 may include 1 or N camera modules 291, N is a positive integer greater than 1, if the electronic device 200 includes N cameras, one of the N cameras is a main camera, and the others may be secondary cameras, such as telephoto cameras.
[0045] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.
[0046] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the headphone jack 274 and the application processor.
[0047] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.
[0048] The speaker 271 is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or listen to hands-free calls through the speaker 271. The receiver 272, also called the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 answers a call or voice message, the voice can be answered by placing the receiver 272 close to the human ear. The microphone 273, also called the "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by placing the human mouth close to the microphone 273 to input the sound signal into the microphone 273. The electronic device 200 can be provided with at least one microphone 273. The headphone jack 274 is used to connect wired headphones.
[0049] Regarding the sensors included in the electronic device 200, the depth sensor is used to obtain the depth information of the scene. The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor can be used to determine the motion posture of the electronic device 200. The air pressure sensor is used to measure the air pressure. The magnetic sensor includes a Hall sensor. The electronic device 200 can use the magnetic sensor to detect the opening and closing of the flip leather case. The acceleration sensor can detect the magnitude of the acceleration of the electronic device 200 in various directions (generally three axes). The distance sensor is used to measure the distance. The proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The fingerprint sensor is used to collect fingerprints. The temperature sensor is used to detect the temperature. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation can be provided through the display screen 290. The ambient light sensor is used to sense the ambient light brightness. The bone conduction sensor can obtain a vibration signal.
[0050] The button 294 includes a power button, a volume button, etc. The button 294 can be a mechanical button. It can also be a touch button. The motor 293 can generate a vibration prompt. The motor 293 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The electronic device 200 interacts with the network through the SIM card to realize functions such as calls and data communications.
[0051] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.
[0052] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0053] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0054] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.
[0055] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0056] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0057] Next, the model training method in the embodiment of the present disclosure is described in detail with reference to the accompanying drawings.
[0058] In step S310, a training label corresponding to a reference image is obtained, and auxiliary information of the reference image is obtained.
[0059] In the disclosed embodiment, the reference image refers to an image used to train the model. The training label refers to the real label of the reference image. The training label can be obtained by algorithms such as colmap, MVS, and SFM.
[0060] In order to simplify the process, after acquiring the video, the target frame can be determined by extracting the frame, and the depth of the target frame can be estimated to obtain a large number of image-depth spectrum pairs as training data, and the training label of the reference image can be further determined based on the training data. It should be noted that in order to improve the accuracy, the collected video should ensure that the picture is clear and has fewer black edges. The specific requirements depend on the scope of application of the selected traditional algorithm, and this disclosure does not limit this.
[0061] First, the training data can be obtained, and the training labels corresponding to the reference images can be obtained from the training data. The training data refers to the training set, which includes multiple image-depth spectrum data pairs. A pair of image-depth spectrum data pairs can be selected from the image-depth spectrum data pairs included in the training set, and the selection can be random or according to the selection criteria. The selected image-depth spectrum data pairs can be denoted as I and D, and the image-depth spectrum data pairs can be data enhanced. Specifically, data enhancement can be performed by random cropping or random flipping. By performing data enhancement on the image, the amount of training data can be increased, the generalization ability of the model can be improved, and noise data can be increased to improve the robustness of the model. The training label refers to the label that needs to be fitted during the training process, which can be specifically represented by the depth spectrum data in the acquired image-depth spectrum data. Among them, the depth spectrum data is used to represent the depth value of the image.
[0062] Auxiliary information is feature information used to guide model training. In the disclosed embodiment, multiple feature information corresponding to the original image can be obtained during training, and the multiple feature information can be processed into a feature spectrum form of the same size as the original image, and then the multiple feature information can be used as a training guide to train the model. Auxiliary information may include but is not limited to optical flow, segmentation spectrum, and the like. Optical flow refers to the displacement of each pixel on the image in all directions during the movement of the picture. When the auxiliary information is optical flow, the optical flow can be determined based on the adjacent frames of the reference image in the original video and the reference image. The adjacent frame can be the previous or next frame of the reference image. The segmentation spectrum is generated by segmenting the image using an image segmentation method, and the segmentation spectrum obtained by different image segmentation methods may be different.
[0063] In step S320, an auxiliary encoder is used to perform feature extraction on the reference image and the auxiliary information to determine an auxiliary prediction result, and an actual prediction result is determined based on feature extraction on the reference image by an actual encoder.
[0064] In the embodiment of the present disclosure, the structure of the depth estimation model belongs to a conventional encoder-decoder structure. Among them, the encoder is responsible for extracting image features from the input reference image, and the image features can be in the form of a feature spectrum. The structure of the encoder can be a plurality of cascaded convolutional layers and downsampling layers. The decoder is responsible for predicting the depth value from the image features, and the structure of the decoder is generally a plurality of cascaded convolutional layers and upsampling layers. It should be noted that the depth estimation model in the embodiment of the present disclosure includes two encoders, namely an actual encoder and an auxiliary encoder. Among them, the auxiliary encoder is used to guide the training of the actual encoder. The actual encoder is used to process the image. Reference Figure 4 As shown in , the structure of the actual encoder, i.e., encoder 1, is consistent with that of a conventional encoder. The first layer of the auxiliary encoder, i.e., encoder 2, is a cascade layer, which is responsible for receiving multiple inputs and cascading them in the channel dimension, and the rest is consistent with that of a conventional encoder. In the disclosed embodiment, by setting the parameters of the encoder, the size of the feature spectrum output by the auxiliary encoder is completely consistent with the size of the feature spectrum output by the actual encoder. Among them, setting the parameters of the encoder refers to setting the convolution parameters of the encoder to the same parameters, so that the structures and parameters of other modules of the encoder are the same. Specifically, assuming that the first calculation modules of the two encoders are both convolutional layers, their parameters such as output channels, convolution kernel size, convolution step size, etc. are set to the same values, and the number of input channels is set to the same as the actual input of each. For example, the actual input of encoder 1 may be 3, then the number of input channels of encoder 1 is set to 3; the actual input of encoder 2 may be 10, then the number of input channels of encoder 2 is set to 10. And keep the structures and parameters of the other modules of the two encoders exactly the same. In this way, the size of the feature spectrum output by the auxiliary encoder can be completely consistent with the size of the feature spectrum output by the actual encoder.
[0065] Based on the structure of the above depth estimation model, the reference image and the auxiliary information can be input into an auxiliary encoder for feature extraction to obtain an auxiliary feature spectrum, and the auxiliary feature spectrum can be decoded to perform depth prediction to obtain the auxiliary prediction result. Figure 4 As shown in , the reference image and the auxiliary information can be simultaneously input into the auxiliary encoder for feature extraction to obtain the auxiliary feature spectrum F2. Specifically, the reference image and the auxiliary information can be cascaded to obtain a cascade result, which belongs to the whole formed by the reference image and the auxiliary information. The cascade result is further repeatedly convolved and downsampled by the auxiliary encoder to obtain the auxiliary feature spectrum.
[0066] Furthermore, the auxiliary feature spectrum may be input into a decoder so as to perform depth value prediction on the auxiliary feature spectrum through the decoder to obtain an auxiliary predicted depth spectrum as an auxiliary prediction result. The auxiliary prediction result may be represented by P2.
[0067] At the same time, reference Figure 4 As shown in , the reference image can be input into the actual encoder, i.e., encoder 1, for feature extraction to obtain the actual feature spectrum F1. Specifically, the reference image can be convolved and downsampled by the actual encoder to obtain the actual feature spectrum F1. Further, the actual feature spectrum F1 can be input into the decoder to perform depth value prediction on the actual feature spectrum through the decoder to obtain the actual predicted depth spectrum as the actual prediction result, and the actual prediction result can be represented by P1. It should be noted that the size of the auxiliary feature spectrum is consistent with the size of the actual feature spectrum.
[0068] In step S330, the auxiliary encoder and the actual encoder are alternately trained according to the auxiliary prediction result, the actual prediction result and the training label, and the decoder is trained to obtain a depth estimation model.
[0069] In the disclosed embodiment, the two encoders may be alternately trained according to the prediction results corresponding to the two encoders and the training labels of the reference image. Alternating training refers to training the auxiliary encoder and training the actual encoder under the guidance of the auxiliary encoder, that is, training the actual encoder in combination with the auxiliary features corresponding to the auxiliary encoder.
[0070] Figure 5 The flowchart of model training is shown schematically in Figure 5 As shown in , it mainly includes the following steps:
[0071] In step S510, an auxiliary loss function is calculated based on the auxiliary prediction results and the training labels, and the model parameters of the auxiliary encoder and the model parameters of the decoder are adjusted by the auxiliary loss function to train the auxiliary encoder and the decoder.
[0072] In this step, the logarithmic loss between the auxiliary prediction result P2 and the depth spectrum data D can be calculated according to formula (1), where D represents the depth spectrum data and P represents the auxiliary prediction result.
[0073] loss1=||log(D)-log(P)||2 Formula (1)
[0074] The auxiliary loss function may be a logarithmic L2 loss function. After calculating the auxiliary loss function between the auxiliary prediction result P2 and the depth spectrum data D, the model parameters of the auxiliary encoder and the model parameters of the decoder may be adjusted according to the auxiliary loss function. Specifically, the minimum auxiliary loss function may be used as the training goal, and the auxiliary loss function may be back-propagated to adjust the model parameters of the auxiliary encoder and the model parameters of the decoder until the training goal is reached and the training process of the auxiliary encoder is terminated. After the training is completed, a trained auxiliary encoder and a reference decoder are obtained. The reference decoder may be called an intermediate decoder, and the model parameters of the reference decoder may be the first training parameters.
[0075] In step S520, an actual loss function is calculated in combination with the actual prediction result, the training label and the auxiliary feature, and the model parameters of the actual encoder and the model parameters of the decoder are adjusted by the actual loss function to train the actual encoder and the decoder.
[0076] In this step, the actual loss function can be determined by the actual prediction result, the depth spectrum data, the auxiliary features and the actual features. Specifically, the first loss function is first determined according to the actual prediction result and the training label, and can be calculated by referring to formula (1). Wherein, D represents the depth spectrum data, and P represents the actual prediction result.
[0077] Furthermore, a second loss function may be calculated based on the auxiliary feature and the actual feature, for example, as shown in formula (2). Wherein, F1 represents the auxiliary feature, and F2 represents the actual feature.
[0078] loss2=||F1-F2||2 Formula (2)
[0079] Next, the first loss function and the second loss function are combined to obtain the actual loss function. Specifically, the first loss function and the second loss function can be calculated as a weighted sum, and the weighted sum can be used as the actual loss function, which is calculated specifically according to formula (3):
[0080] loss=loss1+αloss2 formula (3)
[0081] The actual loss function may also be a logarithmic L2 loss. After calculating the actual loss function corresponding to the actual prediction result P1, the model parameters of the actual encoder and the model parameters of the decoder may be adjusted according to the actual loss function. Specifically, the minimum actual loss function may be used as the training target, and the actual loss function may be back-propagated to adjust the model parameters of the actual encoder and the model parameters of the decoder until the training target is reached, and the training process of the actual encoder is terminated.
[0082] It should be noted that the auxiliary encoder and the actual encoder are trained alternately, that is, not only the auxiliary encoder is trained, but also the actual encoder is trained based on the auxiliary features calculated by the auxiliary encoder.
[0083] After the training of the actual encoder is completed, a trained actual encoder is obtained, and the parameters of the reference decoder are updated according to the actual loss function to obtain a trained decoder, and the parameters of the trained decoder may be final parameters.
[0084] In step S530, the trained auxiliary encoder is ignored, and the depth estimation model is determined according to the trained actual encoder and the trained decoder.
[0085] In this step, after the actual encoder and the auxiliary encoder are trained, the trained auxiliary encoder can be ignored, and the depth estimation model is generated only based on the trained actual encoder and the trained decoder. Since the auxiliary encoder has auxiliary information, it can converge faster, but since the auxiliary information required by the auxiliary encoder is not available or the acquisition cost is high in actual application, the auxiliary encoder cannot be used directly, that is, the auxiliary encoder does not play any role in the use stage, so the auxiliary encoder can be ignored. Ignoring the trained auxiliary encoder here can be deleting the trained auxiliary encoder. That is to say, after the training is completed, only the actual encoder and decoder need to be retained, and the auxiliary encoder can be directly discarded. Based on this, the auxiliary encoder is only used in the model training process in the depth estimation model, and only serves as a guide to constrain the actual encoder, i.e., encoder 1, from the intermediate output and the final prediction, thereby accelerating the convergence of the actual encoder and improving the training efficiency of the actual encoder.
[0086] The decoder updates parameters during the training of the auxiliary encoder and the actual encoder, so the decoder parameters can be determined by the two encoders. The training process of the decoder can be: update the model parameters of the decoder according to the auxiliary loss function to obtain a reference decoder whose parameters are the training parameters, and further update the model parameters of the reference decoder according to the actual loss function to obtain a trained decoder whose parameters are the final parameters. Determining the decoder through the training process of the auxiliary loss function and the actual loss function can improve the accuracy of the decoder.
[0087] In the embodiments of the present disclosure, by providing an auxiliary encoder with a structure similar to that of the actual encoder and guiding the training of the actual encoder by obtaining a number of auxiliary information during the training process, the network convergence can be accelerated, and the conventional encoder can be prompted to learn the feature extraction mode of the auxiliary information, thereby enhancing the feature extraction capability of the encoder. This can solve the problems of slow model training speed and high training difficulty in conventional methods, improve model performance, and improve model training efficiency.
[0088] refer to Figure 6 As shown in , the reference image 601 and the auxiliary information 602 are input to the auxiliary encoder 603 for feature extraction to obtain auxiliary features 604, and the auxiliary features are input to the decoder 605 to obtain the auxiliary predicted depth spectrum 606. The auxiliary loss function 608 is calculated according to the auxiliary predicted depth and depth spectrum data 607. The auxiliary loss function is back-transferred to the auxiliary encoder for training to obtain the trained auxiliary encoder and decoder.
[0089] The reference image 601 is input to the actual encoder 611 for feature extraction to obtain the actual feature 612, and the actual feature is input to the decoder 605 to obtain the actual predicted depth spectrum 613. The actual loss function 614 is calculated based on the actual predicted depth spectrum and the depth spectrum data 607, the auxiliary features and the actual features. The actual loss function is reversely transmitted to the actual encoder for training to obtain the trained actual encoder and decoder. The depth estimation model 600 is generated based on the trained actual encoder 611 and the trained decoder 605.
[0090] In the embodiment of the present disclosure, a method for predicting image depth is also provided. Figure 7 As shown in , it mainly includes the following steps:
[0091] In step S710, the image to be processed is input to an actual encoder in the depth estimation model to obtain actual features;
[0092] In step S720, the actual features are decoded and predicted according to the decoder in the depth estimation model to determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to any one of the model training methods described above.
[0093] In the embodiments of the present disclosure, the image to be processed may be any type of image, specifically an image whose depth is to be predicted. The depth estimation model refers to a model for performing depth estimation on the image to be processed, which may be any type of model capable of performing predictive estimation, for example, it may be in the form of an encoder-decoder. In actual use, the depth estimation model may include only one actual encoder, which refers to an encoder in the depth estimation model for extracting features from the image. In addition, the depth estimation model may also include a decoder. The decoder is used to decode the result output by the actual encoder to perform depth estimation, thereby obtaining a corresponding depth estimation result.
[0094] In the disclosed embodiments, depth estimation refers to monocular depth estimation. Its task objective is: input a depth image to be estimated, and output a single-channel depth spectrum of the same size. The value of each point on the depth spectrum represents the depth value of the corresponding pixel in the input image. The depth value can have a clear physical dimension or only represent relative distance.
[0095] Based on this, in actual application, it is only necessary to input the image to be processed into the actual encoder for convolution and downsampling operations to extract features and obtain the corresponding actual features, that is, the actual feature spectrum; further, the actual feature spectrum obtained by the actual encoder is input into the decoder for decoding prediction, and the output is the predicted depth of the image to be processed.
[0096] Through the actual encoder and decoder in the trained depth estimation model, feature extraction and depth estimation are performed on the processed image. The whole process is simple and efficient without the need for other auxiliary information or equipment. It simplifies the operational steps for depth estimation of the image, improves the efficiency of depth estimation, and avoids the limitation of needing to rely on other equipment.
[0097] The present disclosure provides a model training device, referring to Figure 8 As shown in , the model training device 800 may include:
[0098] Auxiliary information acquisition module 801, used to acquire a training label corresponding to a reference image and acquire auxiliary information of the reference image;
[0099] A feature extraction module 802 is used to perform feature extraction on the reference image and the auxiliary information using an auxiliary encoder to determine an auxiliary prediction result, and to perform feature extraction on the reference image based on an actual encoder to determine an actual prediction result;
[0100] The alternating training module 803 is used to alternately train the auxiliary encoder and the actual encoder according to the auxiliary prediction result, the actual prediction result and the training label, and train the decoder to obtain a depth estimation model.
[0101] In an exemplary embodiment of the present disclosure, the feature extraction module includes: an auxiliary feature extraction module, which is used to input the reference image and the auxiliary information into an auxiliary encoder for feature extraction to obtain an auxiliary feature spectrum, and decode the auxiliary feature spectrum to perform depth prediction to obtain the auxiliary prediction result; an actual feature extraction module, which is used to input the reference image into an actual encoder for feature extraction to obtain an actual feature spectrum, and decode the actual feature spectrum to obtain the actual prediction result.
[0102] In an exemplary embodiment of the present disclosure, the alternating training module includes: an auxiliary training module, which is used to calculate an auxiliary loss function based on the auxiliary prediction results and training labels, and adjust the model parameters of the auxiliary encoder and the model parameters of the decoder through the auxiliary loss function to train the auxiliary encoder and the decoder; an actual training module, which is used to calculate the actual loss function in combination with the actual prediction results, training labels and auxiliary features, and adjust the model parameters of the actual encoder and the model parameters of the decoder through the actual loss function to train the actual encoder and the decoder; a model combination module, which is used to ignore the trained auxiliary encoder and determine the depth estimation model based on the trained actual encoder and the trained decoder.
[0103] In an exemplary embodiment of the present disclosure, the actual training module is configured to: determine a first loss function based on the actual prediction result and the training label; calculate a second loss function based on the auxiliary feature and the actual feature; and merge the first loss function and the second loss function to obtain the actual loss function.
[0104] In an exemplary embodiment of the present disclosure, the auxiliary feature spectrum corresponding to the auxiliary encoder has the same size as the actual feature spectrum corresponding to the actual encoder.
[0105] It should be noted that the specific details of each module in the above-mentioned model training device have been described in detail in the corresponding model training method, so they will not be repeated here.
[0106] The present disclosure also provides an image depth prediction device, referring to Fig. 9 As shown in , the image depth prediction device 900 may include:
[0107] A feature acquisition module 901 is used to input the image to be processed into an actual encoder in the depth estimation model to obtain actual features;
[0108] The depth estimation module 902 is used to decode and predict the actual features according to the decoder in the depth estimation model to determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to any one of the model training methods described above.
[0109] It should be noted that the specific details of each module in the above-mentioned depth prediction device have been described in detail in the corresponding depth prediction method, so they will not be repeated here.
[0110] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0111] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0112] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0113] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.
[0114] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A model training method, characterized in that: include: Obtaining a training label corresponding to a reference image and obtaining auxiliary information of the reference image; Using an auxiliary encoder to extract features from the reference image and the auxiliary information to determine an auxiliary prediction result, and based on an actual encoder to extract features from the reference image to determine an actual prediction result; Alternately training the auxiliary encoder and the actual encoder according to the auxiliary prediction result, the actual prediction result and the training label, and training the decoder to obtain a depth estimation model; The auxiliary encoder and the actual encoder are alternately trained according to the auxiliary prediction result, the actual prediction result and the training label, and the decoder is trained to obtain a depth estimation model, including: Calculating an auxiliary loss function based on the auxiliary prediction result and the training label, and adjusting the model parameters of the auxiliary encoder and the model parameters of the decoder by the auxiliary loss function to train the auxiliary encoder and the decoder; Calculating an actual loss function in combination with the actual prediction result, the training label, and the auxiliary features of the auxiliary encoder, and adjusting the model parameters of the actual encoder and the model parameters of the decoder by using the actual loss function to train the actual encoder and the decoder; The trained auxiliary encoder is ignored, and the depth estimation model is determined based on the trained actual encoder and the trained decoder.
2. The model training method according to claim 1, characterized in that: The step of extracting features from the reference image and the auxiliary information using an auxiliary encoder to determine an auxiliary prediction result, and extracting features from the reference image based on an actual encoder to determine an actual prediction result includes: Inputting the reference image and the auxiliary information into an auxiliary encoder for feature extraction to obtain an auxiliary feature spectrum, and decoding the auxiliary feature spectrum to perform depth prediction to obtain the auxiliary prediction result; The reference image is input into an actual encoder for feature extraction to obtain an actual feature spectrum, and the actual feature spectrum is decoded to obtain the actual prediction result.
3. The model training method according to claim 1, characterized in that: The calculating the actual loss function by combining the actual prediction result, the training label and the auxiliary feature includes: Determine a first loss function according to the actual prediction result and the training label; Calculate a second loss function according to the auxiliary feature and the actual feature; The first loss function and the second loss function are combined to obtain the actual loss function.
4. The model training method according to claim 2, characterized in that: The auxiliary feature spectrum corresponding to the auxiliary encoder has the same size as the actual feature spectrum corresponding to the actual encoder.
5. A method for predicting image depth, characterized in that: include: Inputting the image to be processed into the actual encoder in the depth estimation model to obtain actual features; The actual features are decoded and predicted according to the decoder in the depth estimation model to determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to the model training method according to any one of claims 1-4.
6. A model training device, characterized in that: include: An auxiliary information acquisition module, used to acquire a training label corresponding to a reference image and acquire auxiliary information of the reference image; A feature extraction module, configured to perform feature extraction on the reference image and the auxiliary information using an auxiliary encoder to determine an auxiliary prediction result, and perform feature extraction on the reference image based on an actual encoder to determine an actual prediction result; An alternating training module, used to perform alternating training on the auxiliary encoder and the actual encoder according to the auxiliary prediction result, the actual prediction result and the training label, and to train the decoder to obtain a depth estimation model; The auxiliary encoder and the actual encoder are alternately trained according to the auxiliary prediction result, the actual prediction result and the training label, and the decoder is trained to obtain a depth estimation model, including: Calculating an auxiliary loss function based on the auxiliary prediction result and the training label, and adjusting the model parameters of the auxiliary encoder and the model parameters of the decoder by the auxiliary loss function to train the auxiliary encoder and the decoder; Calculating an actual loss function in combination with the actual prediction result, the training label, and the auxiliary features of the auxiliary encoder, and adjusting the model parameters of the actual encoder and the model parameters of the decoder by using the actual loss function to train the actual encoder and the decoder; The trained auxiliary encoder is ignored, and the depth estimation model is determined based on the trained actual encoder and the trained decoder.
7. An image depth prediction device, characterized in that: include: A feature acquisition module, used for inputting the image to be processed into an actual encoder in the depth estimation model to obtain actual features; A depth estimation module, used to decode and predict the actual features according to the decoder in the depth estimation model to determine the predicted depth of the image to be processed; wherein the actual encoder and the decoder in the depth estimation model are trained according to the model training method according to any one of claims 1-4.
8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the model training method described in any one of claims 1-4 or the image depth prediction method described in claim 5 by executing the executable instructions.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method described in any one of claims 1 to 4 or the image depth prediction method described in claim 5.
Citation Information
Patent Citations
Image identification method and device and storage medium
CN108304846A
Method for determining to-be-labeled image, and method and device for model training
CN110517759A
3D depth-of-field estimation method and device and terminal equipment
CN110969651A
Method and system for image analysis
US20210042930A1
