Network model knowledge distillation method, image depth estimation method and electronic equipment

Abstract

The embodiment of the invention provides a network model knowledge distillation method, an image depth estimation method and electronic equipment, and the method comprises the steps: obtaining training data and a depth label corresponding to the training data, and the training data comprises a plurality of pieces of image data; inputting the training data into a first model to obtain a first intermediate output feature and a first depth estimation result, the first model being a trained convergence network model; inputting the training data into a second model to obtain a second intermediate output feature and a second depth estimation result, the network complexity of the second model being lower than the network complexity of the first model; calculating loss corresponding to the second model according to the first intermediate output feature, the first depth estimation result, the second intermediate output feature, the second depth estimation result and a depth label; and training the second model based on the loss to obtain a training convergent target network model. Therefore, the result precision when the electronic equipment executes the image depth estimation task can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a network model knowledge distillation method, an image depth estimation method and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning network models (such as neural network models) have been widely used, especially in the field of computer vision, providing effective solutions for image depth estimation tasks.

[0003] At present, network models used for image depth estimation tasks mostly appear in the form of encoder-decoder structures. Since the encoder has a large number of calculation parameters and requires high computing power of the processor, such network models usually run on personal computers (PCs) to perform tasks. For embedded devices, due to the limited computing power and memory capacity of the processors they carry, network models with fewer calculation parameters and lower computing power requirements are needed to perform tasks.

[0004] For the existing network models running on embedded devices for image depth estimation tasks, although the amount of calculation parameters and the requirements for computing power are reduced, the accuracy of the results when performing image depth estimation tasks is also reduced, and a better image depth estimation effect is not achieved. Summary of the invention

[0005] The present application provides a network model knowledge distillation method, an image depth estimation method and an electronic device, which can improve the accuracy of results when the electronic device performs an image depth estimation task.

[0006] In a first aspect, the present application provides a network model knowledge distillation method, which is performed by an electronic device and includes:

[0007] Obtain training data and depth labels corresponding to the training data, where the training data includes multiple image data;

[0008] Input the training data into the first model to obtain a first intermediate output feature and a first depth estimation result, wherein the first model is a trained and converged network model, the first intermediate output feature is feature data obtained when multiple network layers in the first model process the training data, and the first depth estimation result is image depth information corresponding to the training data predicted by the first model;

[0009] Input the training data into the second model to obtain a second intermediate output feature and a second depth estimation result, wherein the second intermediate output feature is feature data obtained when multiple network layers in the second model process the training data, and the second depth estimation result is image depth information corresponding to the training data predicted by the second model, and the network complexity of the second model is lower than the network complexity of the first model;

[0010] Calculate the loss corresponding to the second model according to the first intermediate output feature, the first depth estimation result, the second intermediate output feature, the second depth estimation result, and the depth label;

[0011] The second model is trained based on the loss to obtain a target network model with converged training.

[0012] Among them, the network model knowledge distillation method provided in this application aims to adopt a teacher-student learning mechanism, with the first model as the teacher model and the second model as the student model. Through the intermediate output features of the first model, the depth estimation results and the depth labels corresponding to the training data, the second model is supervised and trained from multiple scales to obtain a lightweight and high-precision target network model. The target network model can run on embedded devices and has high processing performance, thereby improving the accuracy of the results when performing image depth estimation tasks on embedded devices.

[0013] In some implementations, the first model and the second model may have the structure of a MiDaS deep estimation network model, and the network complexity of the second model is lower than that of the first model. For example, a MiDaS large backbone network model is used as the first model and a MiDaS lightweight backbone network model is used as the second model.

[0014] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned calculation of the loss corresponding to the second model based on the first intermediate output feature, the first depth estimation result, the second intermediate output feature, the second depth estimation result and the depth label includes: calculating the first loss based on the first intermediate output feature and the second intermediate output feature; calculating the second loss based on the first depth estimation result and the second depth estimation result; calculating the third loss based on the second depth estimation result and the depth label; calculating the loss corresponding to the second model based on the first loss, the second loss and the third loss.

[0015] Among them, when calculating the loss corresponding to the second model, the electronic device can respectively calculate the first loss between the first intermediate output feature and the second intermediate output feature, the second loss between the first depth estimation result and the second depth estimation result, and the third loss between the second depth estimation result and the depth label, and then these three losses together constitute the total loss.

[0016] In some implementations, since the first model may include multiple network layers, for example, the first model may include an encoder network and a decoder network, the encoder network and the decoder network may each include a 4-layer structure, and the second model may also include multiple network layers, the electronic device may respectively calculate the sub-losses between the output features of each network layer in the first model and the output features of each network layer in the second model, and then add up the sub-losses to obtain the above-mentioned first loss.

[0017] That is, the first intermediate output feature may include multiple first sub-features respectively output by multiple network layers in the first model, and the second intermediate output feature may include multiple second sub-features respectively output by multiple network layers in the second model. The first loss is calculated based on the first intermediate output feature and the second intermediate output feature, including:

[0018] For the first network layer and the second network layer, the first sub-loss between the first sub-feature corresponding to the first network layer and the second sub-feature corresponding to the second network layer is calculated, the first network layer is any one of the multiple network layers in the first model, the second network layer is any one of the multiple network layers in the second model, and the hierarchical position of the first network layer in the first model is the same as the hierarchical position of the second network layer in the second model; based on the multiple first sub-losses between the multiple first sub-features and the multiple second sub-features, the first loss is calculated.

[0019] Assume that the first sub-features output by the multiple network layers in the first model are Fe t 1. Fe t 2. Fe t 3 and Fe t 4. And Fd t 4. Fd t 3. Fd t 2 and Fd t 1. The multiple second sub-features output by the multiple network layers in the second model are Fe s 1. Fe s 2. Fe s 3 and Fe s 4. And Fd s 4. Fd s 3. Fd s 2 and Fd s 1, then the electronic equipment calculates Fe t 1 and Fe s 1, Fe t 2 and Fe s 2, Fe t 3 and Fe s 3, Fe t 4 and Fe s 4 between, Fdt 4 and Fd s 4 between, Fd t 3 and Fd s 3 between, Fd t 2 and Fd s 2, Fd t 1 and Fd s 1 (i.e., multiple first sub-losses are obtained), and then multiple first sub-losses can be accumulated to obtain the first loss. Thus, by calculating the loss between the output features of each network layer in the first model and the second model, the dimension of the calculated loss is increased, and the second model can be better supervised for training.

[0020] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned calculation of the first sub-loss between the first sub-feature corresponding to the first network layer and the second sub-feature corresponding to the second network layer includes: determining the probability distribution of the single-channel feature corresponding to the first sub-feature according to a preset distillation temperature and the feature corresponding to the single channel in the first sub-feature; determining the probability distribution of the single-channel feature corresponding to the second sub-feature according to the distillation temperature and the feature corresponding to the single channel in the second sub-feature; and calculating the first sub-loss between the first sub-feature and the second sub-feature based on the distillation temperature, the probability distribution of the single-channel feature corresponding to the first sub-feature and the probability distribution of the single-channel feature corresponding to the second sub-feature.

[0021] In this implementation, the electronic device may calculate the first loss by using a channel-by-channel normalized loss, that is, determine the corresponding single-channel feature probability distribution according to the feature corresponding to a single channel in each sub-feature, and then calculate the first sub-loss between the sub-features according to the single-channel feature probability distribution, and then calculate the first loss based on multiple first sub-losses. The relationship between the first sub-feature and the single-channel feature probability distribution corresponding to the first sub-feature is determined by T, which is the distillation temperature, Fe t 1(c) is the feature corresponding to a single channel in the first sub-feature. Similarly, the probability distribution of the single channel feature corresponding to the second sub-feature can also be calculated.

[0022] In combination with the first aspect, in some implementations of the first aspect, calculating the first sub-loss between the first sub-feature and the second sub-feature based on the distillation temperature, the single-channel feature probability distribution corresponding to the first sub-feature, and the single-channel feature probability distribution corresponding to the second sub-feature includes:

[0023] Based on inclusion The first sub-loss between the first sub-feature and the second sub-feature is calculated using the relationship between represents the single-channel feature probability distribution corresponding to the first sub-feature, represents the probability distribution of the single channel feature corresponding to the second sub-feature, c t represents the number of channels corresponding to the first sub-feature and the second sub-feature, h1×w1 represents the feature dimension corresponding to the first sub-feature and the second sub-feature, and Γ represents the distillation temperature.

[0024] It can be understood that based on the above relationship principle, the electronic device can calculate the first sub-loss between each first sub-feature and the corresponding second sub-feature, that is, Fe t 1 and Fe s 1, Fe t 2 and Fe s 2, Fe t 3 and Fe s 3, Fe t 4 and Fe s 4 between, Fd t 4 and Fd s 4 between, Fd t 3 and Fd s 3 between, Fd t 2 and Fd s 2, Fd t 1 and Fd s 1, then the first loss can be the sum of these losses.

[0025] In combination with the first aspect, in some implementations of the first aspect, before calculating the first sub-loss between the first sub-feature corresponding to the first network layer and the second sub-feature corresponding to the second network layer, the method further includes: if the number of channels corresponding to the second sub-feature is not equal to the number of channels corresponding to the first sub-feature, performing a convolution operation on the second sub-feature so that the number of channels of the second sub-feature is equal to the number of channels corresponding to the first sub-feature.

[0026] Among them, since the network structure of the second model is different from that of the first model, the number of channels corresponding to the second sub-feature output by the second model may not be equal to the number of channels corresponding to the first sub-feature output by the first model. Then, the electronic device can perform a convolution operation on the second sub-feature (such as a 1×1 convolution operation) to align the number of channels of the second sub-feature output by the second model with the number of channels of the first sub-feature output by the first model, thereby improving data uniformity and providing a data basis for the subsequent loss calculation process.

[0027] In combination with the first aspect, in some implementations of the first aspect, after calculating the second loss based on the first depth estimation result and the second depth estimation result, the method further includes: calculating a second gradient matching item loss based on the first depth estimation result, the second depth estimation result, and the pixel change gradient corresponding to the first depth estimation result, and adding the second gradient matching item loss to the second loss.

[0028] In combination with the first aspect, in some implementations of the first aspect, after calculating the third loss based on the second depth estimation result and the depth label, the method further includes: calculating a third gradient matching item loss based on the second depth estimation result, the depth label, and the pixel change gradient corresponding to the depth label, and adding the third gradient matching item loss to the third loss.

[0029] In the above two implementation methods, after calculating the second loss and the third loss, the electronic device may further add a gradient matching term loss after the second loss and the third loss respectively, which can be understood as a smaller gradient regularization term for controlling the gradient smoothness of the final depth estimation result to further improve the training accuracy of the second model.

[0030] In combination with the first aspect, in some implementations of the first aspect, calculating the loss corresponding to the second model based on the first loss, the second loss and the third loss includes: performing a weighted summation of the first loss, the second loss and the third loss to obtain the loss corresponding to the second model.

[0031] That is, the electronic device can add corresponding weight coefficients to the first loss, the second loss and the third loss obtained by the above calculation, respectively, and obtain the total loss corresponding to the second model after weighted summation.

[0032] In combination with the first aspect, in some implementations of the first aspect, after obtaining the training data and the depth labels corresponding to the training data, the method further includes: normalizing and resolution converting the training data to obtain training data with a target resolution; converting the depth label with a depth dimension to a disparity dimension, and performing scale normalization and resolution conversion on the converted depth label to obtain a depth label with a target resolution.

[0033] Because the acquired training data and the data dimensions of the depth label, such as the size and format, may not be uniform, the electronic device may first perform data preprocessing on the training data and the depth label corresponding to the training data to achieve data uniformity. For example, the electronic device may first scale the pixel values ​​of the training data to normalize the pixel values ​​to the interval [0, 1]; then scale the depth label, unify the depth label of the depth dimension to the parallax dimension and then scale it, such as given a valid label range interval [min, max], the depth labels that are not in the range interval are cropped, and the remaining ones are valid labels, and then scale the valid depth labels to the interval [0, 1], etc. Then, in order to ensure that the resolution of the training data processed by the network model is a more appropriate resolution, so that the network model can obtain more detailed information and more comprehensive global information in the image, and to make the same batch of training data processed by the network model have the same resolution, the electronic device can also perform resolution conversion on the above-mentioned transformed training data and depth labels, and convert them to the target resolution. Thus, the uniformity of the training data is improved.

[0034] In a second aspect, the present application provides an image depth estimation method, which is executed by an electronic device, including: acquiring a first image; inputting the first image into a target network model, and outputting an image depth estimation result corresponding to the first image, wherein the target network model is a model trained according to the method in the first aspect and any possible implementation method thereof.

[0035] Among them, the first image can be any image obtained by an electronic device (such as an embedded device), and then the electronic device inputs the first image into the target network model for processing. For example, through the processing process of the encoder network and the decoder network, the image depth estimation result corresponding to the first image can be output. Since the target network model is trained using the above-mentioned network model knowledge distillation method, the model accuracy is relatively high. Therefore, the image depth estimation result output by the above-mentioned image depth estimation method also has a relatively high accuracy.

[0036] In a third aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect, and the behavior of the electronic device in the implementation of the second aspect. The function can be implemented by hardware, or by hardware executing the corresponding software implementation. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.

[0037] In a fourth aspect, the present application provides an electronic device, the electronic device comprising: one or more processors, and a memory;

[0038] The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute any one of the methods in the technical solution of the first aspect, as well as the method for executing the technical solution of the second aspect.

[0039] In a fifth aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method in the first aspect and any possible implementation thereof, and executes the method in the implementation of the second aspect.

[0040] Optionally, the chip system also includes a memory, and the memory is connected to the processor via circuits or wires.

[0041] Further optionally, the chip system also includes a communication interface.

[0042] In a sixth aspect, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect, as well as the method of executing the technical solution of the second aspect.

[0043] In the seventh aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solution of the first aspect, as well as the method for executing the technical solution of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of an effect of performing depth estimation on an image to obtain a depth map provided in an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of the overall process of a network model knowledge distillation method provided in an embodiment of the present application;

[0047] Figure 4 It is a flowchart of a network model knowledge distillation method provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of a basic network structure of a teacher model provided in an embodiment of the present application;

[0049] Figure 6is a flowchart of an image depth estimation method provided in an embodiment of the present application;

[0050] Figure 7 This is a schematic diagram of the comparison of depth maps output by a target network model, a teacher model, and a student pre-trained model respectively provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0052] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0053] With the rapid development of artificial intelligence technology, deep learning network models (such as neural network large models) have been widely used, especially in the field of computer vision, providing effective solutions for image depth estimation tasks. Among them, computer vision (CV) is an important branch of deep learning technology, mainly including image recognition, image depth estimation, object detection, image segmentation, style transfer, optical character recognition, picture completion, three-dimensional vision, image super-resolution and other research directions.

[0054] Image depth estimation is to obtain the distance information from each pixel in the image scene to the camera. We call this distance information a depth map. Image depth estimation includes monocular depth estimation (MDE). Monocular depth estimation is a task that studies the pixel information in a single image and then derives the distance relationship between objects in the image scene. The core concept of monocular depth estimation is disparity. Disparity refers to the pixel difference between corresponding points in the left and right images. It can also be understood as the offset of the same object under two perspectives. The larger the disparity, the closer the object is to the camera, and the smaller the disparity, the farther the object is from the camera.

[0055] At present, image depth estimation is widely used in autonomous driving, robot navigation and augmented reality. The network model used to perform image depth estimation tasks mostly appears in the form of an encoder-decoder structure. After the encoder extracts features from the input image, the decoder analyzes the feature information provided by the encoder to calculate the depth map of the input image. For example, the depth map obtained by processing an image using the image depth estimation method can be as follows: Figure 1 As shown in the figure, it can be seen that different pixels in the image correspond to different brightness in the depth map. The brighter the brightness, the smaller the depth of the pixel, that is, the closer it is to the camera, and the darker the brightness, the greater the depth of the pixel, that is, the farther it is from the camera. However, since large neural network models usually have high requirements for calculation parameters and computing power, in the above solutions, the encoder generally has a large amount of calculation parameters, and some operators (such as Transformer structure) have high requirements for the computing power of the processor. Therefore, this type of network model can usually only perform tasks under the support of a high-configuration graphics processing unit (GPU) of a PC. For embedded devices (such as smart terminals, cars, etc.), due to the limited computing power and memory capacity of the processors they carry, it is necessary to use a network model with fewer calculation parameters and lower computing power requirements to perform tasks. Therefore, the task of applying large deep learning neural network models to embedded devices is very challenging.

[0056] In order to support the application of large neural network models in embedded devices, the related technology proposes a lightweight encoder solution, using relatively simple operators and smaller calculation parameters to implement the entire image depth estimation task on the central processing unit (CPU) of the embedded device. In the process of training the network model, the related technology generally uses the loss between the output of the network model and the training label to optimize and adjust the model parameters. However, although the network model trained by this method reduces the amount of calculation parameters and the requirements for computing power, it also reduces the accuracy of the results when performing the image depth estimation task, and does not achieve a good image depth estimation effect.

[0057] In view of this, an embodiment of the present application provides a network model knowledge distillation method, which is based on the learning method of the teacher-student model, and obtains a network model (i.e., a student model) for performing image depth estimation tasks from supervised training at multiple scales. The network model has a lightweight network structure, can run on embedded devices, and has high processing performance. It can better handle image depth estimation tasks and improve the accuracy of results when performing image depth estimation tasks.

[0058] It can be understood that the network model knowledge distillation method provided in the embodiment of the present application can be applied to devices such as PCs, servers or server clusters, and a network model (which can be called a target network model) is trained through the knowledge distillation process of a large amount of training data. Preferably, the specific knowledge distillation process can be executed by the GPU of the device to improve processing efficiency.

[0059] It can also be understood that after the target network model is trained, the target network model can be transplanted into an embedded device, and the embedded device can perform image depth estimation based on the target network model. Therefore, the embodiment of the present application also provides an image depth estimation method, which can be applied to embedded devices (or other mobile devices). For example, in the field of autonomous driving, the target network model is transplanted to the on-board processor, and the on-board processor can perform image depth estimation on the acquired map data through the target network model, obtain more accurate road condition information in real time, and realize safe autonomous driving.

[0060] Next, we first introduce the network model knowledge distillation method provided in the embodiment of the present application. The network model knowledge distillation method can be executed by an electronic device. For example, the electronic device 100 is a server. Figure 2 It is a structural diagram of an example of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a wireless communication module 150, a display screen 160, etc.

[0061] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0062] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0063] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0064] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface, etc.

[0065] It is understandable that the interface connection relationship between the modules shown in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0066] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0067] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 160, and the wireless communication module 150. In some embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0068] The wireless communication module 150 can provide wireless communication solutions including WLAN (such as Wi-Fi), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. For example, in the embodiment of the present application, the electronic device 100 can establish a Bluetooth connection with a terminal device (such as a wireless headset 100) through the wireless communication module 150.

[0069] The wireless communication module 150 may be one or more devices integrating at least one communication processing module. The wireless communication module 150 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 150 may also receive a signal to be sent from the processor 110, modulate the frequency of the signal, amplify the signal, and convert it into electromagnetic waves for radiation via the antenna.

[0070] The electronic device 100 implements the display function through a GPU, a display screen 160, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 160 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0071] The display screen 160 is used to display images, videos, etc. The display screen 160 includes a display panel.

[0072] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0073] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in an embodiment of the present application, the processor 110 may execute the instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area.

[0074] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0075] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0076] In having Figure 2 Based on the electronic device of the structure, the embodiment of the present application can execute the following network model knowledge distillation method. Knowledge distillation is the process of transferring knowledge from a large model to a smaller training model without causing significant performance loss. In knowledge distillation, a small student model can learn to imitate a large teacher model and use the knowledge of the "teacher" to obtain similar or higher accuracy.

[0077] First, we select a basic depth estimation network model and perform knowledge distillation of a lightweight model on the depth estimation network model. For example, since the current MiDaS depth estimation network model can realize mixed training of multiple data sets and can complete image depth estimation tasks with high precision, the embodiment of the present application adopts the MiDaS depth estimation network model for knowledge distillation. In the process of knowledge distillation, we use the teacher-student learning mechanism, using the MiDaS large backbone network model as the teacher model and the MiDaS lightweight backbone network model as the student model, where the teacher model is a trained network model with high precision. Then, through the learning and training of the teacher model by the student model, a lightweight student model with high precision is obtained as the target network model to complete the knowledge distillation process of the network model.

[0078] On the whole, if Figure 3 As shown in FIG. 1 , the above-mentioned knowledge distillation process may include the following steps: after selecting the above-mentioned MiDaS large backbone network model as the teacher model and the MiDaS lightweight backbone network model as the student model, sufficient training data (such as sufficient image data) and depth labels corresponding to the training data are prepared at the same time. Next, all the training data and depth labels are preprocessed, and the preprocessed training data are respectively input into the teacher model and the student model. After the network model is calculated, the depth estimation results of the teacher model and the depth estimation results of the student model can be obtained respectively. After that, the depth estimation results of the two network models and the intermediate output features of the network models (including the intermediate output features of the encoder and the intermediate output features of the decoder) are extracted at the same time, and the preprocessed depth labels, the depth estimation results of the teacher model, and the intermediate output features of the teacher model are respectively used to supervise the student model at different scales, and different loss functions are used to calculate the losses under supervision at different scales, such as channel-by-channel normalization loss, scale-translation invariance soft loss, and scale-translation invariance hard loss. Finally, different weights are assigned to the losses under supervision at different scales to obtain the final loss. A backpropagation operation is performed based on the final loss to realize end-to-end training of the student model and obtain the trained student model (i.e., the target network model).

[0079] Specifically, Figure 4 As shown, it is a flow chart of a network model knowledge distillation method provided in an embodiment of the present application, including:

[0080] S101, obtaining training data and depth labels corresponding to the training data.

[0081] The training data (input) may be image data, for example, a red, green, and blue (RGB) image. Optionally, the selected RGB image may be an 8-bit 3-channel image, i.e., input∈Rh×w×3 , h and w represent the height and width of the RGB image respectively, and the pixel value of a single channel pixel in the RGB image is in the range [0, 255].

[0082] The depth label (GT) corresponding to the training data can include two categories, one is the ground truth label of the depth dimension, and the other is the ground truth label of the disparity dimension, that is, the depth label of some training data is the depth dimension, and the depth label of some training data is the disparity dimension. Here, the relationship between the ground truth label of the depth dimension and the ground truth label of the disparity dimension can be Depth = 1 / Disparity. It can be understood that the depth label corresponding to the training data is the depth map of the training data, which is used to supervise the training process of the network model, and the depth label can be a two-dimensional grayscale image.

[0083] S102: Perform data preprocessing on the training data and the depth labels corresponding to the training data.

[0084] Because the acquired training data and depth labels may not have uniform data dimensions such as size and format, the electronic device may first perform data preprocessing on the training data and the depth labels corresponding to the training data to achieve data uniformity.

[0085] In one implementation, when the training data (input) is an RGB image, since the pixel values ​​of its pixels are in the interval [0, 255], the interval range is relatively large, the electronic device can first scale the pixel values ​​of the training data, even if Normalize the range to the interval [0, 1].

[0086] Then, the electronic device normalizes the depth label (GT) because the depth label includes the label of the depth dimension and the label of the disparity dimension, and the MiDaS depth estimation network model used above estimates the image depth in the disparity space. Therefore, it is necessary to normalize the depth dimension label to the disparity dimension. The specific process may include: first converting the depth dimension label to the disparity space, for example, The depth dimension labels are converted according to the relationship; then for the depth labels that are already in the disparity space, a valid label range interval [min, max] is given, and the depth labels that are not in the range interval are cropped, that is, the depth labels that are not in the range interval are removed, and the remaining ones are valid labels. At the same time, the valid pixel binary mask = GT [GT ≥ min, GT ≤ max] can also be set for use in the subsequent loss calculation process; Next, the valid depth labels are scaled to the [0, 1] interval, for example, by Finally, a threshold ε is given, and the depth labels greater than or equal to the threshold are assigned the value of ε to avoid the appearance of 0 value, that is, GT[GT≥ε]=ε, for example, ε=10 -4 Thus, the scale normalization of the depth labels is completed.

[0087] Next, in order to ensure that the resolution of the training data processed by the network model is a more appropriate resolution, so that the network model can obtain more detailed information and more comprehensive global information in the image, and that the same batch of training data processed by the network model has the same resolution, the electronic device can also perform resolution conversion on the above-mentioned transformed training data and depth labels, and convert them to the target resolution. For example, the electronic device can first perform a proportional downsampling operation on the training data, the corresponding depth label, and the mask, and then perform a short-edge padding operation on the downsampled data to convert it to the target resolution. Exemplarily, assuming that the target resolution is height × width = 384 × 384, the training data, the corresponding depth label and the original resolution of the mask have a height / width = 4 / 3. Taking the training data as an example, when performing the downsampling operation, the height / width ratio needs to be kept unchanged at 4 / 3. If the height (i.e., the long side) is downsampled to 384, the width should be 384 × 3 / 4 = 288, that is, the resolution of the training data after the downsampling operation is height × width = 384 × 288; if it is to be converted to the target resolution of 384 × 384, the width (i.e., 288) also needs to be padded, that is, padded to 384. For example, a black border of (384-288) / 2 width can be padded on both sides of the width to convert the resolution of the training data to the target resolution. At this time, the size of the training data input∈R 384×384×3 The resolution conversion process for the depth label and mask is similar to that for the training data, but when performing the short edge padding operation, the value of the corresponding padding position in the mask can be assigned an invalid value. At this time, the size of the depth label and mask GT, mask∈R 384×384×1 .

[0088] To further improve the uniformity of the training data, the electronic device may further normalize the training data after the resolution conversion. For example, the electronic device may normalize the training data based on a fixed mean and a fixed standard deviation. Assuming that the fixed mean is a and the fixed standard deviation is b, the electronic device may normalize the training data based on The training data is normalized by the relationship of , optionally, a = [0.485, 0.456, 0.406], b = [0.229, 0.224, 0.225]. Finally, the electronic device can perform a dimensionality conversion operation on the training data, the depth label and the mask so that input∈R 3×h′×w′,GT,mask∈R 1×h′×w′ , for example, input∈R 3×384×384 ,GT,mask∈R 1×384×384 .

[0089] S103, input the training data into the teacher model to obtain a first intermediate output feature and a first depth estimation result.

[0090] S104, input the training data into the student model to obtain a second intermediate output feature and a second depth estimation result.

[0091] That is to say, after the electronic device performs data preprocessing on the above-mentioned training data and the depth labels corresponding to the training data, it can input the training data into the selected teacher model and student model respectively for depth estimation. In addition to obtaining the first depth estimation result output by the teacher model and the second depth estimation result output by the student model, the electronic device in the embodiment of the present application also obtains the first intermediate output feature output by the feature layer of the teacher model and the second intermediate output feature output by the feature layer of the student model.

[0092] Based on the MiDaS depth estimation network model, the embodiment of the present application can select the ResNext101 structure as the backbone network of the teacher model and the Efficient-lite3 structure as the backbone network of the student model, use the convolution layer as the basic component to construct the connection network, and use the residual block as the basic component to construct the encoder network and the decoder network. Before each residual block, a convolution layer of size N×N can be used to adjust the number of channels of the feature map, and then the residual block can be used to reduce or increase the dimension of the feature map. Several feature fusion modules can also be connected after the residual block to fuse the feature map output by the residual block with the output of the feature fusion module of the previous layer. For example, taking the teacher model as an example, the basic network structure of the teacher model can be as follows Figure 5 As shown (the basic network structure of the student model is similar and is not shown in the figure), it includes an encoder network and a decoder network. Both the encoder network and the decoder network can be divided into 4 layers. The layers of the encoder network can be represented as: Layer1, Layer2, Layer3 and Layer4. The layers of the decoder network can be represented as: Path1, Path2, Path3 and Path4. The convolution kernels used by the encoder network and the decoder network can be 1×1 or 3×3.

[0093] Then, after the electronic device inputs the training data into the teacher model, it can output the feature Fe after being processed by Layer 1 of the encoder network. t 1∈R c1×h1×w1 , and then after being processed by Layer2, the feature Fe can be output t 2∈R c2×h2×w2, and then after being processed by Layer3, the feature Fe can be output t 3∈R c3×h3×w3 , and then after being processed by Layer4, the feature Fe can be output t 4∈R c4×h4×w4 For example, Fe t 1∈R 256×96×96 , Fe t 2∈R 512×48×48 , Fe t 3∈R 1024×24×24 , Fe t 4∈R 2048×12×12 The output of the encoder network can be processed by Path4 of the decoder network to output the feature Fd t 4∈R c5 ×h5×w5 , and then after processing by Path3, the feature Fd can be output t 3∈R c6×h6×w6 , and then after processing by Path2, the feature Fd can be output t 2∈R c7×h7×w7 , and then after processing by Path1, the feature Fd can be output t 1∈R c8×h8×w8 , where the input data of each layer in the decoder may include the output of the previous layer as well as the output of the corresponding layer in the encoder. t 4∈R 256×24×24 , Fd t 3∈R 256×48×48 , Fd t 2∈R 256×96×96 , Fd t 1∈R 256×192×192 The output result of the decoder network can be processed by the convolution layer and the Bottleneck structure to output the final depth estimation result (i.e., the first depth estimation result). The Bottleneck structure is a network design strategy in deep learning. In the convolutional neural network, the number of input channels of the network is reduced and then increased back to the original size, thereby achieving information compression and decompression. For example, the first depth estimation result Map1∈R 1×384×384 .

[0094] It can be understood that for the teacher model, the first intermediate output feature can include the above Fe t 1. Fe t 2. Fe t 3 and Fe t 4. And Fd t 4. Fd t 3. Fd t 2 and Fdt 1.

[0095] It can also be understood that the processing process of the student model on the training data is similar to that of the teacher model, so the second intermediate output features corresponding to the student model can include Fe s 1. Fe s 2. Fe s 3 and Fe s 4. And Fd s 4. Fd s 3. Fd s 2 and Fd s 1, the corresponding second depth estimation result is Map2. s 1∈R 32×96×96 , Fe s 2∈R 48×48×48 , Fe s 3∈R 136×24×24 , Fe s 4∈R 384×12×12 , Fd s 4∈R 256×24×24 , Fd s 3∈R 128×48×48 , Fd s 2∈R 64×96×96 , Fd s 1∈R 64×192×192 , Map2∈R 1×384×384 It can be seen that the dimension of the features output by a certain layer in the student model is the same as the dimension of the features output by the corresponding layer in the teacher model, but the number of channels may be different. For example, Fe s 1 and Fe t 1 has a dimension of 96×96, but Fe s 1 corresponds to 32 channels, Fe t The number of channels corresponding to 1 is 256, and the dimensions of the first depth estimation result output by the teacher model and the second depth estimation result output by the student model are the same as the dimensions of the preprocessed depth labels above, so that the loss is calculated on the same dimension.

[0096] It can also be understood that when the teacher model or the student model processes the training data, it can receive the input of the training data in batches in sequence. For example, if there are 1,000 training data, these training data are divided into 10 batches, that is, 10 batches, and the size of each batch is 100, that is, batch size = 100.

[0097] S105, respectively calculating a first loss between the first intermediate output feature and the second intermediate output feature, a second loss between the first depth estimation result and the second depth estimation result, and a third loss between the second depth estimation result and the depth label.

[0098] That is to say, when calculating the loss, the embodiment of the present application not only uses the depth labels corresponding to the training data and the output results of the teacher model to supervise the student model, but also adds the intermediate output features of the teacher model to supervise the student model, calculates the corresponding loss functions from multiple different scales, and jointly supervises the learning and training of the student model. Therefore, the electronic device needs to calculate the above-mentioned first loss, second loss and third loss respectively.

[0099] In one implementation, the electronic device may calculate the first loss, the second loss, and the third loss respectively in the following manner:

[0100] First, calculate the first loss, which is the loss between the first intermediate output feature and the second intermediate output feature. Here we can use channel-by-channel normalization loss to calculate the first loss. Channel-by-channel normalization loss is a method of using the knowledge in each channel of the intermediate features of the network to activate the alignment features of the teacher model and the student model through channels, and convert the activated results into soft probability distributions.

[0101] Exemplarily, since both the first intermediate output feature and the second intermediate output feature include output features of each layer of the encoder network and the decoder network, the electronic device can respectively calculate the loss between the output features of each layer included in the first intermediate output feature and the output features of each layer included in the second intermediate output feature, that is, respectively calculate Fe t 1 and Fe s 1, Fe t 2 and Fe s 2, Fe t 3 and Fe s 3, Fe t 4 and Fe s 4 between, Fd t 4 and Fd s 4 between, Fd t 3 and Fd s 3 between, Fd t 2 and Fd s 2, Fd t 1 and Fd s 1. Since the calculation method of the loss between the output features of each layer is the same, the following takes the calculation process of the loss of one layer as an example to illustrate.

[0102] For example, to calculate Fe t 1 and Fes As an example, assume that the loss between From the above description, we can see that the number of channels c of the output features of the student model is s The number of channels c that may be related to the output features of the teacher model t Different, that is Then the electronic device needs to align the number of channels of the two first, and optionally, the output feature Fe of the student model can be s 1 Perform a convolution operation (such as a 1×1 convolution operation) to align the number of channels of the student model output feature with the number of channels of the teacher model output feature, then It can be understood that when the number of channels of the output features of the student model is the same as the number of channels of the output features of the teacher model, there is no need to perform a convolution operation on the student model. t 1 and Fe s 1′ performs channel normalization operation, for example, Fe t 1 and Fe s 1′ performs the Reshape+Softmax operation, where the Reshape function can be used to adjust the shape of the input tensor, and the Softmax function can be used to map the output value to between 0 and 1. First convert to Then perform the Softmax operation to make Same reason Next, the electronic device can be adjusted according to the distillation temperature. Calculate Fe t 1 and Fe s 1′ corresponds to the feature probability score

[0103] Cloth, such as the teacher model Fe t 1 Corresponding single channel feature probability distribution Fe t 1(c) is Fe t 1, the corresponding characteristics of a single channel, then Fe t 1 corresponds to the total channel feature probability distribution Ψ t ∈R 1×h1×w1 Similarly, the Fe of the student model s 1′ corresponding single channel feature probability distribution Fe s 1′ corresponding to the total channel feature probability distribution Finally, the electronic device can be based on the Fe t 1 Corresponding characteristic probability distribution and Fe s 1′ corresponding to the characteristic probability distribution, calculate Fe t 1 and Fe s1 per channel normalized loss L(Ψ t ,Ψ s ),For example

[0104] Based on the above calculation of Fe t 1 and Fe s 1, the electronic equipment can also calculate Fe t 2 and Fe s 2, Fe t 3 and Fe s 3, Fe t 4 and Fe s 4 between, Fd t 4 and Fd s 4 between, Fd t 3 and Fd s 3 between, Fd t 2 and Fd s 2, Fd t 1 and Fd s 1, then the first loss can be the sum of these losses. Optionally, the electronic device can directly sum the loss corresponding to the feature output by the encoder network and the loss corresponding to the feature output by the decoder network to obtain the first loss, then the first loss L1 = ∑ F＝Fe1,Fe2,Fe3,Fe4 L(Ψ t (F),Ψ s (F))+∑ F＝Fd1,Fd2,Fd3,Fd4 L(Ψ t (F),Ψ s (F)).

[0105] Then, the second loss is calculated, which is the loss between the first depth estimation result and the second depth estimation result. Assume that the first depth estimation result d t ∈R 1×h′×w′ , the second depth estimation result d s ∈R 1×h′×w′ , for example, h′×wτ=384×384, the electronic device may calculate the scale-translation invariant soft loss between the first depth estimation result and the second depth estimation result.

[0106] Exemplarily, the electronic device may calculate the second loss L2 according to the following method: Among them, the function ρ(.) = MSELoss(.) represents the mean square error loss, which measures the degree of fit of the model by calculating the sum of the squares of the differences between the predicted value and the true value. and The electronic device can respectively t and d sPerform a linear transformation to obtain, optionally, As for the values ​​of parameters s and t, optionally, they can be determined according to The relationship between s and t is used to determine the values ​​of s and t, that is, there is a set of parameters (s, t) that can make When the value of is the smallest, substitute the value of the group of parameters (s, t) into Get It can be understood that the parameter i represents each pixel in the image. Thus, the electronic device calculates the second loss L2.

[0107] Next, the third loss is calculated, which is the loss between the second depth estimation result and the depth label, that is, the depth label after the above preprocessing is used to supervise the student model. Assume that the depth label GT∈R 1×h′×w′ , the second depth estimation result d s ∈R 1×h′×w′ , the electronic device can calculate the scale-translation invariance hard loss between the second depth estimation result and the depth label.

[0108] Exemplarily, the electronic device may calculate the third loss L3 according to the following method: Among them, the function ρ(.) = L1Loss(.) is used to measure the mean absolute error between the model prediction result and the true label, and M = sum(mask) represents the number of valid pixels in the deep label. and Electronic devices can respectively detect GT and d s Transform to obtain, optionally, Among them, t(GT) = median(GT), that is, the median of GT, t(d s )=median(d s ), that is, d s the median of t(GT) and s(GT), t(d s ) and s(d s ) are substituted into the above formulas, we can get and Thereby, the electronic device calculates the third loss L3.

[0109] S106, determining the total loss according to the first loss, the second loss and the third loss, and adjusting the model parameters of the student model according to the total loss to obtain the target network model.

[0110] Since the embodiment of the present application uses multiple different scales to calculate the loss function, after the electronic device calculates the loss corresponding to each scale respectively, the total loss can be determined according to the loss corresponding to each scale.

[0111] As can be seen from the above S105, the losses corresponding to each scale are the first loss L1, the second loss L2 and the third loss L3. In one implementation, the electronic device can determine the sum of L1, L2 and L3 as the total loss L, that is, L = L1 + L2 + L3. In another implementation, the electronic device can also perform a weighted summation on L1, L2 and L3 to obtain the total loss L, that is: L = λ1L1 + λ2L2 + λ3L3 = λ1(∑ F＝Fe1,Fe2,Fe3,De4 L(Ψ t (F),Ψ s (F))+∑ F＝Fd1,Fd2,Fd3,Fd4 L(Ψ t (F),Ψ s (F)))+λ2L(d t ,d s )+λ3L(GT,d s In another implementation, since the first loss L1 includes the loss corresponding to the feature output by the encoder network and the loss corresponding to the feature output by the decoder network, the electronic device can assign different weights to the two losses, then L = λ4 (∑ F＝Fe1,Fe2,Fe3,Fe4 L(Ψ t (F),Ψ s (F)))+λ5(∑ F＝Fd1,Fd2,Fd3,Fd4 L(Ψ t (F),Ψ s (F)))+λ6L(d t ,d s )+λ7L(GT,d s ).

[0112] In another implementation, after calculating the second loss and the third loss, the electronic device may further add a gradient matching loss to the second loss and the third loss, respectively, which can be understood as a smaller gradient regularization term used to control the gradient smoothness of the final depth estimation result. Assuming that the gradient matching loss is calculated from K resolution scales, for example, the K resolution scales are the original resolution scale (such as 384×384), 1 / 2 of the original resolution scale, 1 / 4 of the original resolution scale, and 1 / 8 of the original resolution scale, the gradient matching loss added to the second loss is in, Represents the horizontal gradient, that is, the pixel change of adjacent pixels in the same row, Represents the vertical gradient, that is, the pixel change of adjacent pixels in the same column; the third loss is the additional gradient matching loss in, On this basis, if the electronic device wants to calculate the above total loss L, it is necessary to superimpose the gradient matching term loss on the corresponding loss. Optionally, the total loss L = λ4(∑ F＝Fe1,Fe2,Fe3,Fe4 L(Ψ t (F),Ψ s (F)))+λ5(∑ F＝Fd1,Fd2,Fd3,Fd4 L(Ψ t (F),Ψ s (F)))+λ6(L(d t ,d s )+αL reg (d t ,d s ))+λ7(L(GT,d s )+αL reg (GT,d s )).

[0113] After determining the total loss, the electronic device can adjust the model parameters of the student model according to the total loss. Here, the model parameters of the student model can be adjusted by back propagation weight update. It can be understood that since the teacher model is a trained model, the teacher model does not perform model parameter update operations.

[0114] It can be understood that the amount of data of the above training data is huge, for example, it can include 1.5 million images, so the Adam optimizer can be selected in the training process, and the initial learning rate (that is, the step size when updating the weights each iteration when training the neural network model) is set to lr=1e-4, and a total of 50 epochs are trained (an epoch is a process of training once using all the samples in the training data), and the learning rate decays to 0.1 of the current learning rate every 10 epochs, thereby training the student model until the training converges, completing the training process of the student model.

[0115] It can also be understood that in the process of calculating the above loss, one or more of the first loss, the second loss and the third loss can be selected according to actual needs. In addition, the weights corresponding to the above first loss, the second loss and the third loss can also be changed according to actual needs, and the embodiment of the present application does not limit this.

[0116] It can also be understood that the resolution of the training data can also be selected according to actual needs. For example, low-resolution training data can be selected to increase the training speed, and high-resolution training data can be selected to improve the training accuracy.

[0117] It can also be understood that in addition to using the MiDaS depth estimation network model for training, other depth estimation network models that can implement the above training process can also be selected as the backbone network of the teacher model and the student model, and the above training process can also be applied to the knowledge distillation task of non-depth estimation regression problems. As long as the loss function supervised at the corresponding scale can be selected according to the task completed by the specific network model, the embodiments of the present application do not limit this.

[0118] The above-mentioned network model knowledge distillation method is based on the learning method of the teacher-student model, and supervises the training of the student model from multiple scales. In particular, it adopts a channel-by-channel normalized loss function based on feature variable supervision, which improves the accuracy of the trained target network model. At the same time, the target network model has a lightweight network structure and can be well deployed and run on embedded devices, and has a wide range of application scenarios.

[0119] Based on the above network model knowledge distillation method, if the trained target network model is transplanted into an embedded device, the embedded device can perform image depth estimation based on the target network model, that is, the embedded device can execute the following image depth estimation method, such as Figure 6 As shown, it is a flow chart of an example of an image depth estimation method provided in an embodiment of the present application, including:

[0120] S201, acquiring a first image.

[0121] Among them, the first image can be any image acquired by the embedded device. For example, in an autonomous driving scenario, it can be a map image (or real-time road condition image) acquired in real time; in an augmented reality scenario, it can be a scene image to be enhanced, and so on.

[0122] In one implementation, the embedded device may also perform a preprocessing operation on the first image, for example, preprocessing the first image using the above-mentioned preprocessing method for training data, so as to convert the first image to a resolution scale suitable for processing by the target network model.

[0123] S202, input the first image into the target network model, and output the image depth estimation result.

[0124] That is to say, after the embedded device obtains the first image, it can input the first image into the target network model for processing. For example, it can output the image depth estimation result corresponding to the first image through the processing of the encoder network and the decoder network.

[0125] Optionally, the image depth estimation result may be a depth map corresponding to the first image. In the case where the image depth estimation result is a depth map, we also conducted a performance comparison experiment on the target network model, the teacher model, and the student pre-trained model of the embodiment of the present application, that is, the first image is respectively input into the target network model, the teacher model, and the student pre-trained model, and the depth maps corresponding to each model are output. For example, the depth maps output by the target network model, the teacher model, and the student pre-trained model may be as follows: Figure 7 As shown, it can be seen that the depth estimation result of the target network model provided by the embodiment of the present application is significantly better than that of other models.

[0126] It can be understood that since the target network model is trained using the above-mentioned network model knowledge distillation method, the model accuracy is relatively high. Therefore, the image depth estimation result output by the above-mentioned image depth estimation method also has relatively high accuracy.

[0127] The above describes in detail the examples of the network model knowledge distillation method and the image depth estimation method provided in the embodiments of the present application. It is understandable that in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0128] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0129] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0130] The electronic device provided in this embodiment is used to execute the above-mentioned network model knowledge distillation method and image depth estimation method, and thus can achieve the same effect as the above-mentioned implementation method.

[0131] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.

[0132] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0133] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 2 Device of the structure shown.

[0134] The embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the network model knowledge distillation method and image depth estimation method of any of the above embodiments. The storage medium may include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0135] An embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the network model knowledge distillation method and image depth estimation method in the above-mentioned embodiments.

[0136] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the network model knowledge distillation method and the image depth estimation method in the above-mentioned method embodiments.

[0137] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0138] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A network model knowledge distillation method, the method being executed by an electronic device, characterized in that: The method comprises: Acquire training data and a depth label corresponding to the training data, wherein the training data includes a plurality of image data; Input the training data into a first model to obtain a first intermediate output feature and a first depth estimation result, wherein the first model is a trained and converged network model, the first intermediate output feature is feature data obtained when multiple network layers in the first model process the training data, and the first depth estimation result is image depth information corresponding to the training data predicted by the first model; Inputting the training data into a second model to obtain a second intermediate output feature and a second depth estimation result, wherein the second intermediate output feature is feature data obtained when multiple network layers in the second model process the training data, and the second depth estimation result is image depth information corresponding to the training data predicted by the second model, and the network complexity of the second model is lower than the network complexity of the first model; Calculating a loss corresponding to the second model according to the first intermediate output feature, the first depth estimation result, the second intermediate output feature, the second depth estimation result, and the depth label; The second model is trained based on the loss to obtain a target network model with converged training.

2. The method according to claim 1, characterized in that The calculating the loss corresponding to the second model according to the first intermediate output feature, the first depth estimation result, the second intermediate output feature, the second depth estimation result and the depth label includes: Calculating a first loss according to the first intermediate output feature and the second intermediate output feature; Calculating a second loss according to the first depth estimation result and the second depth estimation result; Calculating a third loss according to the second depth estimation result and the depth label; The loss corresponding to the second model is calculated according to the first loss, the second loss and the third loss.

3. The method according to claim 2, characterized in that The first intermediate output feature includes a plurality of first sub-features respectively output by a plurality of network layers in the first model, the second intermediate output feature includes a plurality of second sub-features respectively output by a plurality of network layers in the second model, and the first loss is calculated according to the first intermediate output feature and the second intermediate output feature, comprising: For a first network layer and a second network layer, calculating a first sub-loss between a first sub-feature corresponding to the first network layer and a second sub-feature corresponding to the second network layer, wherein the first network layer is any one of the multiple network layers in the first model, the second network layer is any one of the multiple network layers in the second model, and the hierarchical position of the first network layer in the first model is the same as the hierarchical position of the second network layer in the second model; The first loss is calculated based on a plurality of first sub-losses between the plurality of first sub-features and the plurality of second sub-features.

4. The method according to claim 3, characterized in that: The calculating a first sub-loss between a first sub-feature corresponding to the first network layer and a second sub-feature corresponding to the second network layer includes: Determine a single channel feature probability distribution corresponding to the first sub-feature according to a preset distillation temperature and a feature corresponding to a single channel in the first sub-feature; Determine a single channel feature probability distribution corresponding to the second sub-feature according to the distillation temperature and the feature corresponding to the single channel in the second sub-feature; A first sub-loss between the first sub-feature and the second sub-feature is calculated based on the distillation temperature, the single-channel feature probability distribution corresponding to the first sub-feature, and the single-channel feature probability distribution corresponding to the second sub-feature.

5. The method according to claim 4, characterized in that The calculating the first sub-loss between the first sub-feature and the second sub-feature based on the distillation temperature, the single-channel feature probability distribution corresponding to the first sub-feature, and the single-channel feature probability distribution corresponding to the second sub-feature includes: Based on inclusion , calculating the first sub-loss between the first sub-feature and the second sub-feature, wherein, represents the single-channel feature probability distribution corresponding to the first sub-feature, represents the probability distribution of the single channel feature corresponding to the second sub-feature, c t represents the number of channels corresponding to the first sub-feature and the second sub-feature, h1×w1 represents the feature dimension corresponding to the first sub-feature and the second sub-feature, represents the distillation temperature.

6. The method according to any one of claims 3 to 5, characterized in that Before calculating the first sub-loss between the first sub-feature corresponding to the first network layer and the second sub-feature corresponding to the second network layer, the method further includes: If the number of channels corresponding to the second sub-feature is not equal to the number of channels corresponding to the first sub-feature, a convolution operation is performed on the second sub-feature to make the number of channels of the second sub-feature equal to the number of channels corresponding to the first sub-feature.

7. The method according to any one of claims 2 to 6, characterized in that After calculating the second loss according to the first depth estimation result and the second depth estimation result, the method further includes: Based on the first depth estimation result, the second depth estimation result, and the pixel change gradient corresponding to the first depth estimation result, a second gradient matching term loss is calculated, and the second gradient matching term loss is added to the second loss.

8. The method according to any one of claims 2 to 6, characterized in that After calculating the third loss according to the second depth estimation result and the depth label, the method further includes: Based on the second depth estimation result, the depth label, and the pixel change gradient corresponding to the depth label, a third gradient matching item loss is calculated, and the third gradient matching item loss is added to the third loss.

9. The method according to any one of claims 2 to 8, characterized in that The calculating the loss corresponding to the second model according to the first loss, the second loss and the third loss includes: A weighted sum is taken of the first loss, the second loss and the third loss to obtain a loss corresponding to the second model.

10. The method according to any one of claims 1 to 9, characterized in that After acquiring the training data and the depth labels corresponding to the training data, the method further includes: Normalizing and converting the resolution of the training data to obtain training data with a target resolution; The depth label with the depth dimension is converted to the disparity dimension, and the converted depth label is scaled and resolution converted to obtain a depth label with the target resolution.

11. The method according to any one of claims 1 to 10, characterized in that The first model and the second model have the structure of a MiDaS depth estimation network model.

12. A method for estimating image depth, the method being performed by an electronic device, characterized in that: The method comprises: acquiring a first image; The first image is input into a target network model, and an image depth estimation result corresponding to the first image is output, wherein the target network model is a model trained according to the method described in any one of claims 1 to 11.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors, and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 11, or execute the method as claimed in claim 12.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 11, or executes the method as described in claim 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11, or executes the method according to claim 12.

Images