Lightweight student network model training method and monocular depth estimation method for drones

Abstract

This invention discloses a training method for a lightweight student network model and a monocular depth estimation method for UAVs. The training method includes: extracting edge detection images from the original image; inputting the edge detection images and the original image into a teacher network for inference; fusing the initial depth map and edge depth map obtained from the inference, and using the fused depth map as a soft label to guide the training of the student network; inputting the original image into the student network, allowing the student network to perform self-supervised training under the guidance of the soft label using a teacher-student network paradigm distillation method, outputting a predicted depth map, and calculating the overall loss function of the student network. When the overall loss function is within a preset range, the student network converges, obtaining a lightweight student network model. This application can reduce the number of parameters, computational load, and storage requirements, achieving model lightweighting, and enhancing monocular depth estimation capabilities when deployed on UAVs.

Description

Technical Field

[0001] This invention relates to the field of aircraft technology, and in particular to a training method for a lightweight student network model and a monocular depth estimation method for unmanned aerial vehicles. Background Technology

[0002] With the rapid development of autonomous driving, augmented reality, and robot navigation, real-time and accurate depth estimation technology has become one of the key technologies in these application areas. Depth estimation technology provides crucial data support for drones in depth perception, path planning, obstacle detection, and obstacle avoidance decisions. It is a core component of drones' perception and understanding of their surroundings, playing a vital role in their autonomous navigation and environmental interaction. Although depth sensors such as LiDAR, time-of-flight cameras, and binocular cameras can provide relatively accurate depth information, these sensors are typically bulky, expensive, and power-hungry, making it difficult to meet the stringent requirements of drones in terms of weight, size, dimensions, and power consumption.

[0003] For example, in high-risk scenarios of post-disaster search and rescue using drones, characterized by damaged environmental structures and frequent secondary disasters, close-range manual reconnaissance carries extremely high risks. Therefore, it is crucial to prioritize the rapid environmental perception and life-orientation of micro-drones to improve search and rescue efficiency and reduce risks. In such tasks, accurate and real-time depth information is paramount, as drones rely on depth perception to identify obstacles, analyze terrain undulations, and determine the spatial location of trapped individuals. This allows them to plan safe flight paths, support rescue decisions, and provide critical data support for rapidly commencing search and rescue operations.

[0004] However, due to limitations in payload capacity, flight time, and onboard computing power, micro-drones are unable to carry traditional depth sensors such as LiDAR. These traditional devices are often bulky, heavy, power-consuming, and expensive, making them unsuitable for lightweight drone platforms.

[0005] In contrast, a monocular camera, as a sensor that is low-cost, lightweight, small in size, low in power consumption, highly flexible, adaptable, and easy to install and configure, only requires a single RGB camera to estimate the depth of the surrounding environment and infer the depth information between objects in the scene and the camera. It is especially suitable for resource-constrained devices to perform efficient environmental perception.

[0006] There are already some studies on monocular depth estimation algorithms, such as Monodepth2, GeoNet, and CADepth. These algorithms have shown excellent depth estimation accuracy. However, due to the large number of algorithm model parameters and excessive computational resource consumption, the models are large in size and the inference speed is slow. They also have high requirements for the computing power of the hardware platform. This makes it difficult to effectively deploy these algorithms on UAV edge computing platforms with limited computing resources, thus limiting the widespread application of this technology on resource-constrained platforms. Summary of the Invention

[0007] The purpose of this application is to provide a lightweight training method for a student network model and a monocular depth estimation method for unmanned aerial vehicles (UAVs), so as to at least solve the aforementioned technical problems. The various technical effects produced by the optional technical solutions among the many technical solutions provided by this invention are detailed below.

[0008] To achieve the above objectives, firstly, this application provides a method for training a lightweight student network model, comprising:

[0009] The edge detection image of the original image is extracted, and the edge detection image and the original image are respectively input into the pre-trained teacher network for inference to obtain the edge depth map and the initial depth map;

[0010] The edge depth map is used as a guide map to fuse the initial depth map and the edge depth map to obtain a fused depth map, which is then used as a soft label to guide student network training.

[0011] The original image is input into the student network so that the student network can be self-supervised and trained using the teacher-student network paradigm distillation method under the guidance of the soft label, and output a predicted depth map.

[0012] The original image is stitched together with the adjacent frame images and then input into the pose network to obtain the pose transformation matrix. Based on the pose transformation matrix and the predicted depth map, image reconstruction is performed to obtain the reconstructed image.

[0013] Calculate the overall loss function of the student network. When the overall loss function is within a preset range, the student network converges, and a lightweight student network model is obtained.

[0014] The overall loss function is the sum of the pairwise loss function, the Laplacian distillation loss function, the pixel-by-pixel distillation loss function, and the base loss function. The base loss function is the sum of the photometric reprojection loss function and the smoothing loss function between the reconstructed image and the original image. The pixel-by-pixel distillation loss function is the pixel-by-pixel distillation loss function between the fused depth map and the predicted depth map. The pairwise loss function and the Laplacian distillation loss function are determined based on the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network.

[0015] In some embodiments, the edge depth map is used as a guide map to fuse the initial depth map and the edge depth map to obtain a fused depth map, which is expressed by the following formula:

[0016] ;

[0017] in, This represents the fused depth map, where GF represents the guided filtering operation. The initial depth map represents the overall depth information and spatial structure information of the original image; The edge depth map, including edge detail information, serves as a guide map; r represents the radius of the guide filter window, used to control the local filtering range. This represents the regularization parameter, used to prevent overfitting and ensure the smoothness of the filtering.

[0018] In some embodiments, the guided filtering operation includes:

[0019] Using the initial depth map as the input image and the edge depth map as the guide image, guided filtering is performed to output an initial fused image that satisfies the local linear model.

[0020] When the filtering window is located in the edge region of the guide image, the gradient change of the initial fused image in the edge direction is enhanced, highlighting edge details;

[0021] When the filtering window is located in a flat area of ​​the guide image, the corresponding flat area is subjected to local mean filtering.

[0022] In some embodiments, the step of concatenating the original image with adjacent frame images, inputting it into a pose network to obtain a pose transformation matrix, and then performing image reconstruction based on the pose transformation matrix and the predicted depth map to obtain a reconstructed image includes:

[0023] The original image is then stitched together with adjacent frame images to obtain the stitched image.

[0024] The stitched image is input into the pose network to obtain the pose transformation matrix;

[0025] Based on the camera intrinsic parameter matrix, the predicted depth map, and the pose transformation matrix, the pixels on the original image are mapped to the corresponding pixels in the coordinate system of the adjacent frame images through spatial geometric transformation.

[0026] The pixel values ​​from the adjacent frame images are sampled and applied to the viewpoint of the original image to obtain the reconstructed image.

[0027] In some embodiments, calculating the overall loss function of the student network includes:

[0028] Calculate the base loss function, which is the photometric reprojection loss function and the smoothing loss function between the reconstructed image and the original image;

[0029] Calculate the pixel-by-pixel distillation loss function between the fused depth map and the predicted depth map;

[0030] Calculate the pairwise loss function between the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network, and the Laplace distillation loss function;

[0031] Based on the photometric reprojection loss function, smoothing loss function, pixel-wise distillation loss function, pairwise loss function, and Laplace distillation loss function, the overall loss function is calculated using the following formula;

[0032] ;

[0033] in, Represents the overall loss function. Let represent the base loss function, which is the sum of the photometric reprojection loss function and the smoothing loss function. This represents the pairwise affinity loss function between the teacher intermediate feature maps of the teacher network and the student intermediate feature maps of the student network. This represents the Laplace distillation loss function. This represents the pixel-by-pixel distillation loss function.

[0034] In some embodiments, calculating the pairwise loss function between the teacher intermediate feature maps of the teacher network and the student intermediate feature maps of the student network includes:

[0035] Based on the teacher intermediate feature map and the student intermediate feature map, an affinity map is constructed; wherein, nodes in the affinity map represent different spatial locations, and the connection relationship between two nodes represents similarity; the size of the teacher intermediate feature map or the student intermediate feature map is [size missing]. The affinity map includes There are n nodes, and each node is connected to n edges. ; Indicates the range of connections; Indicates granularity;

[0036] Based on the affinity map, the formula for calculating the pairwise loss function is as follows:

[0037] ;

[0038] in, , ;

[0039] in, and These represent the width and height of the teacher's or student's intermediate feature map, respectively. Represents all nodes. This represents the similarity between node i and node j in the affinity graph corresponding to the student network. This represents the similarity between node i and node j in the affinity graph corresponding to the teacher network. and These represent the aggregated features obtained through average pooling. The aggregation operation means performing average pooling on the features of all channels within a node, compressing them into a fixed-length feature vector, thereby extracting the global representation information of the node.

[0040] The Laplace distillation loss function between the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network is calculated using the following formula:

[0041] ;

[0042] Among them, C i is the total number of units contained in the features obtained by the i-th decoder layer of the teacher network or the student network, N represents the total number of decoder layers of the teacher network or the student network, and x is the spatial location index on the teacher's intermediate feature map. This represents the absolute difference of the residuals of the first layer of features in the Laplace pyramid. The absolute difference represents the residuals of the second layer of the Laplace pyramid. It is the squared difference between the features of the teacher network and the student network after downsampling to a 1 / 4 scale.

[0043] In some embodiments, the pixel-by-pixel distillation loss function for calculating the fused depth map and the predicted depth map is expressed by the following formula:

[0044] ;

[0045] in, Let N represent the pixel-wise distillation loss function, where C represents the number of scales predicted by the decoder of the teacher network or the decoder of the student network, and N represents the number of scales predicted. i This represents the number of pixels at the i-th scale. This represents the depth prediction value of the teacher network at the j-th pixel at the i-th scale. This represents the depth prediction value of the student network at the j-th pixel at the i-th scale.

[0046] In some embodiments, calculating the basic loss function includes:

[0047] The photometric reprojection loss function is calculated using the photometric reprojection loss formula, which is as follows:

[0048] ;

[0049] in, represents the photometric reprojection loss function, pe represents the photometric reprojection error, and SSIM represents the structural similarity index; Represents the first-order norm of an image and calculates pixel-level differences; Indicates the weighting coefficient. This refers to the original image. This represents the reconstructed image;

[0050] The smoothing loss function is calculated using the smoothing loss formula, which is:

[0051] ;

[0052] in, Represents the smoothing loss function. Indicates the original image Predicted depth map Depth values ​​after mean normalization This indicates the original image Input the predicted depth map obtained from the student network prediction. Represents the predicted depth map The mean, and These represent the gradient calculations in the x and y directions, respectively. and Represents the image gradient, used as a weighting factor;

[0053] Based on the photometric reprojection loss function and the smoothing loss function, the fundamental loss function is calculated and expressed by the following formula:

[0054] ;

[0055] in, Let C represent the basic loss function, and let C represent the number of scales predicted by the decoder of the teacher network or the decoder of the student network. The weighting coefficients representing the smoothing loss. Represents a depth map related to the predicted depth map A scale-consistent binary matrix, where pixels are occluded or outside the field of view during reprojection, and the corresponding position... The value is 0, and the value is 1 otherwise.

[0056] In some embodiments, after obtaining the lightweight student network model, the training method of the lightweight student network model further includes:

[0057] The monocular depth estimation performance of the lightweight student network model was verified.

[0058] If the monocular depth estimation performance verification passes, the lightweight student network model will be deployed to the drone platform.

[0059] If the monocular depth estimation performance verification fails, the training parameters of the student network are adjusted until the monocular depth estimation performance verification passes.

[0060] Secondly, this application provides a monocular depth estimation method for unmanned aerial vehicles (UAVs), comprising:

[0061] The image acquired by the monocular camera of the UAV at the current moment is input into the trained lightweight student network model to obtain the depth estimate of the UAV; the lightweight student network model is trained by the training method of the lightweight student network model described in any one of the first aspects.

[0062] Thirdly, this application also provides a processing apparatus, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are configured to execute the one or more computer programs stored in the memory to cause the one or more processors to perform a training method for a lightweight student network model as described in any of the first aspects and a monocular depth estimation method for a UAV as described in the second aspect.

[0063] Fourthly, this application provides an aircraft that includes the processing apparatus described in the third aspect.

[0064] Implementing one of the above-described technical solutions of this application has the following advantages or beneficial effects:

[0065] This application presents a lightweight student network model training method and a monocular depth estimation method for UAVs. First, a self-supervised learning method is used to train the student network model, eliminating the need for labeled data and effectively reducing the cost and time of manual annotation, thus better adapting to real-world application environments. Second, a knowledge distillation method based on the teacher-student network paradigm is employed. Knowledge from a teacher network with a large number of parameters and high performance is transferred to a student network model with fewer parameters. This ensures that the student network model maintains high performance while reducing the number of parameters, computational load, and storage requirements, significantly decreasing reliance on hardware computing resources, thereby achieving model lightweighting. Furthermore, a deep learning-based edge extraction algorithm is introduced into the teacher network. By acquiring the edge detection map corresponding to the original image, the teacher network receives both the original image and its corresponding edge detection image as input to predict the edge depth map and the initial depth map. Subsequently, guided filtering fusion technology is used to fuse the edge depth map and the initial depth map to fully combine the overall depth layout and edge detail information, thereby improving the overall depth estimation accuracy. Finally, the fused depth map is used as a guidance signal for the student network to further enhance its depth estimation capability. Ultimately, efficient and real-time depth perception was achieved on a drone platform with limited computing resources, demonstrating significant engineering application value and promising prospects.

[0066] This application proposes a monocular depth estimation method for unmanned aerial vehicles (UAVs). While ensuring the accuracy of monocular depth estimation, it effectively reduces the computational complexity of the algorithm model, improves inference speed, and significantly reduces dependence on hardware computing resources. This invention enables the algorithm to be successfully deployed on edge devices with limited computing resources, such as embedded platforms like UAVs, thereby enhancing the application potential of monocular depth estimation algorithms in real-time perception and environmental understanding. Compared with existing technologies, this invention further enhances the monocular depth estimation capability of UAVs, providing a solid technical foundation for achieving efficient and real-time visual navigation, and has significant engineering application value. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0068] Figure 1 This is a flowchart illustrating a training method for a lightweight student network model according to an embodiment of this application.

[0069] Figure 2 This is a schematic block diagram illustrating a training method for a lightweight student network model according to an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of the structure of the teacher network in this application;

[0071] Figure 4 This is a schematic diagram of the student network structure for this application;

[0072] Figure 5 These are images of post-disaster search and rescue scenes captured by a monocular camera of an unmanned aerial vehicle (UAV) in an embodiment of this application.

[0073] Figure 6 This is a monocular depth estimation map of a UAV disaster search and rescue scenario before the adoption of a lightweight student network model in the embodiments of this application;

[0074] Figure 7 This is a monocular depth estimation map of a UAV disaster search and rescue scenario after adopting a lightweight student network model in an embodiment of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, various exemplary embodiments described below can be referenced to the corresponding drawings, which constitute a part of the exemplary embodiments and illustrate various exemplary embodiments that may be adopted to implement this application. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of this application disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of this application.

[0076] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can refer to fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can refer to the internal connection of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0077] To illustrate the technical solutions described in this application, specific embodiments are provided below, showing only the parts related to the embodiments of this application.

[0078] like Figure 1-2As shown, this application provides a training method for a lightweight student network model, which includes the following steps S10 to S50.

[0079] S10: Extract the edge detection image from the original image, and input the edge detection image and the original image into the pre-trained teacher network for inference to obtain the edge depth map and the initial depth map.

[0080] The pre-trained teacher network is a teacher network model that has been trained in advance. It is a monocular depth estimation teacher model. In some embodiments, the training method of the lightweight student network model may further include:

[0081] Based on a dataset containing deep information, a teacher network with a large number of parameters is trained to obtain a pre-trained teacher network.

[0082] Specifically, the KITTI dataset is one of the most important publicly available datasets in the fields of computer vision and autonomous driving. It contains image data from real-world driving environments and is widely used for validating depth estimation algorithms. In the monocular depth estimation task, the images in the KITTI dataset are taken from a car's front-facing camera, covering a variety of complex urban street, suburban, and highway scenes. For each image, the KITTI dataset provides a corresponding depth map, which is acquired using a laser scanner and therefore has high-precision true depth values. The dataset has a resolution of 1242×375 and contains 44,931 precisely labeled high-resolution images, including 39,810 training images, 4,424 validation images, and 697 test images. Therefore, the dataset in this application embodiment can be the KITTI dataset, which is a publicly available dataset of sequential images containing depth information.

[0083] The collected dataset was used to verify the accuracy and real-time performance of the monocular depth estimation algorithm.

[0084] Then, a teacher network T with a large number of parameters is constructed and trained based on the collected dataset to obtain a pre-trained teacher network, which has the characteristics of high depth accuracy.

[0085] like Figure 3 As shown, Figure 3This is a schematic diagram of the ResNeXt module structure of the teacher network in this embodiment of the application. In this embodiment, the teacher network T uses ResNeXt101 as the feature extraction backbone network during training. The feature decoder consists of 5 layers. Each layer recovers depth information layer by layer by fusing encoder features through progressive upsampling and skip connections, thereby constructing a teacher network with a large number of parameters. The schematic diagram of the module structure in ResNeXt101 is shown below. Figure 3 As shown, this teacher network effectively reduces computation while improving feature representation by replacing standard convolutions with grouped convolutions. Simultaneously, ResNeXt introduces a new hyperparameter called cardinality, which represents the number of grouped convolutions in the network. A larger cardinality increases the network's learning capacity without adding extra computational cost. In ResNeXt, each module does not simply stack multiple convolutional layers; instead, it processes different features using multiple parallel paths. Each path processes different features through grouped convolutions, thereby increasing the network's expressive power. Therefore, increasing cardinality is a more effective way to improve accuracy; compared to simply deepening or widening the network, it is more beneficial for optimizing network performance.

[0086] This embodiment employs a self-supervised approach to train the ResNeXt101 teacher network. During training, the Adam optimizer is used to optimize the parameters of the teacher model, with an initial learning rate of 0.0001, a first-order moment estimation coefficient of 0.9, and a second-order moment estimation coefficient of 0.999. The loss function is gradually minimized by calculating gradients and updating the weights of convolutional kernels and fully connected layers in the network. To improve training efficiency and model convergence speed, this embodiment introduces a learning rate decay strategy to dynamically reduce the learning rate in the later stages of training, thereby avoiding gradient oscillations. Batch normalization is also employed to standardize the data distribution of each batch, effectively enhancing the stability of the training process. After 20 rounds of iterative training, a teacher network with optimal depth estimation accuracy is obtained, laying a solid foundation for the subsequent knowledge distillation process.

[0087] After obtaining the pre-trained teacher network, inference is performed on the original image using the pre-trained teacher network to obtain an initial depth map. The original image can be any image taken by a monocular camera, for example, such as... Figure 2 As shown, Figure 2 The original image is a scene captured by a monocular camera, for example, such as... Figure 5 As shown, Figure 5 Images of post-disaster search and rescue operations captured by a drone's monocular camera.

[0088] Furthermore, edge detection can be performed on the original image, and the edge-detected image can be input into a pre-trained teacher network for inference to obtain an edge depth map.

[0089] In some embodiments, a deep learning-based pixel difference network (PiDiNet) can be used to achieve efficient edge detection of the original image, thereby obtaining an edge-detected image.

[0090] The PiDiNet network structure mainly consists of an efficient backbone and an efficient side structure. The backbone employs a concatenated approach of depthwise separable convolutions and residual blocks, progressively extracting multi-scale features through four stages. Each stage contains four residual blocks, and downsampling is performed between stages using max pooling layers, with the number of channels doubling with each stage. In this embodiment, the number of channels for these four stages are 60, 120, 240, and 240, respectively. Furthermore, each residual block includes a depthwise convolutional layer, a ReLU activation layer, and a pointwise convolutional layer to ensure effective extraction of multi-scale edge features while maintaining a lightweight model.

[0091] The edge feature extraction structure achieves depth supervision by extracting edge feature maps from each stage and calculating edge loss. To further optimize edge feature extraction, a compact dilated convolutional module is introduced after the output of each stage of the pixel difference network to enrich multi-scale edge information, and a compact spatial attention module is used to suppress background noise. Subsequently, the multi-channel feature maps are compressed into single-channel edge maps through 1×1 convolutional layers, and interpolation is performed to restore them to the original image size. Finally, a sigmoid activation function is applied to generate the edge detection image.

[0092] Edge maps not only provide structural information and object boundary features in images, but also play a crucial role in depth estimation tasks. In this embodiment, the edge detection map is input into a pre-trained teacher network. The edge depth map generated after inference by the pre-trained teacher network not only contains global depth information but also retains rich object edge depth details. By leveraging the edge information in the edge depth map, the depth estimation network can better understand edge contours and regions of significant change in the image, thereby improving the depth estimation accuracy of object boundaries and edge regions and alleviating the boundary blurring problem commonly found in traditional monocular depth estimation.

[0093] S20: Using the edge depth map as a guide map, the initial depth map and the edge depth map are fused to obtain a fused depth map, and the fused depth map is used as a soft label to guide the student's network training.

[0094] In some embodiments, step S20, which uses the edge depth map as a guide map to fuse the initial depth map and the edge depth map to obtain a fused depth map, can be expressed by the following formula:

[0095] ;

[0096] in, This represents the fused depth map, where GF represents the guided filtering operation. The initial depth map represents the overall depth information and spatial structure information of the original image; The edge depth map, including edge detail information, serves as a guide map; r represents the radius of the guide filter window, used to control the local filtering range. This represents the regularization parameter, used to prevent overfitting and ensure the smoothness of the filtering.

[0097] Specifically, the embodiments of this application employ a depth map fusion method based on guided filtering, which uses the initial depth map predicted by a pre-trained teacher network from the original image. Edge depth maps predicted by a pre-trained teacher network based on edge detection images Perform adaptive fusion.

[0098] Unlike traditional weighted fusion, guided filtering can adjust based on the edge depth map. The gradient changes dynamically adjust the fusion results, maintaining the smoothness of depth in flat areas and highlighting details in edge areas. While enhancing the edges, the guided filtering ensures the continuity of the overall depth structure, so that the resulting fused depth map maintains spatial consistency in both edge and non-edge areas.

[0099] By using the initial depth map corresponding to the original image As input image, the edge depth map corresponding to the edge detection image As a guiding map, it can fully leverage the guiding role of edge information, directing the edge optimization direction of the input image. While ensuring the consistency of the overall depth structure, it enhances gradient changes in edge regions, resulting in clearer edges and more accurate details in the fused depth map. Ultimately, the fused depth map... The output of the pre-trained teacher network will be used as soft labels to guide the distillation training of the student network.

[0100] Furthermore, in the depth map fusion processing using guided filtering, the initial depth map is used as the input image p, and the edge depth map is used as the guiding image I. Guided filtering is then applied, and the output initial fused image q of the guided filter satisfies a local linear model. for:

[0101] ;

[0102] Where i is the pixel index. This represents a local window centered at pixel k. and Indicated in the image window The linear coefficients within the range are obtained by minimizing the loss function, which is expressed as:

[0103] ;

[0104] in, This represents the regularization parameter, used to suppress overfitting caused by noise.

[0105] linear coefficients and The closed-form solutions are expressed by the following formulas:

[0106] ;

[0107] ;

[0108] in, and These represent the guide diagram I in the window. Mean and variance within, This indicates that the input image p is in the window. The mean within, the final output (Fused depth map) is represented by the following formula, which is the average of the results from all overlapping windows:

[0109] ;

[0110] in, Display window Calculated from all pixels within the range The mean, Display window Calculated from all pixels within the range The mean.

[0111] When the filter window When located in the edge region of the guiding image I, the linear coefficient is... Significantly increase, then enhance the initial fused image Gradient changes in the edge direction highlight edge details; within the filtering window When located in a flat region of the guiding image I, the linear coefficients Approaching 0, at which point the initial fused image Local mean filtering is used in flat regions to achieve smoothing. This mechanism allows for the preservation of the global structure while enhancing local edges.

[0112] As can be understood, edge regions refer to locations in an image where pixel values ​​change drastically, typically corresponding to object outlines, texture details, or structural boundaries. Flat regions refer to areas in an image where pixel values ​​change slowly and with small differences, typically corresponding to uniform parts such as object surfaces and backgrounds.

[0113] S30: Input the original image into the student network so that the student network can perform self-supervised training under the guidance of the soft label using the teacher-student network paradigm distillation method, and output a predicted depth map;

[0114] S40: After stitching the original image with the adjacent frame image, input it into the pose network to obtain the pose transformation matrix, and perform image reconstruction based on the pose transformation matrix and the predicted depth map to obtain the reconstructed image.

[0115] In some implementations, step S30 may include:

[0116] The original image is then stitched together with adjacent frame images to obtain the stitched image.

[0117] The stitched image is input into the pose network to obtain the pose transformation matrix;

[0118] Based on the camera intrinsic parameter matrix, the predicted depth map of the original image predicted by the student network, and the pose transformation matrix, the pixels on the original image are mapped to the corresponding pixels in the coordinate system of the adjacent frame image through spatial geometric transformation.

[0119] The pixel values ​​from the adjacent frame images are sampled and applied to the viewpoint of the original image to obtain the reconstructed image.

[0120] Specifically, the original image is used as the target frame I. t The adjacent frame image adjacent to the original image is called the adjacent frame I. s After stitching, a stitched image is obtained and then fed into a pose network to estimate adjacent frames I. s Relative to target frame I t The camera pose relationship is determined to obtain the predicted pose transformation matrix. Then, combined with the known camera intrinsic parameter matrix K, the predicted depth map, and the pose transformation matrix, spatial geometric transformation is used to transform the target frame I. t Pixel p on the original image t Mapped to adjacent frame I s The corresponding pixel p in the coordinate system of (adjacent frame images) s The mapping transformation formula is expressed as follows:

[0121] ;

[0122] Where K is the camera intrinsic parameter matrix, which includes camera parameters such as focal length and optical center coordinates; This is the camera pose transformation matrix from the target frame to the source frame (adjacent frame images), reflecting the translation and rotation relationship between adjacent frames; Target frame I t At pixel p t Depth estimate; p t p represents the pixel coordinates on the target frame. s It is the pixel coordinates obtained from the source frame's viewpoint after mapping transformation. The "~" in the formula refers to the pixel p in the target frame. t The pixel position p of the source frame is obtained after transformation. s It is an approximate relationship, not an absolute equality.

[0123] Furthermore, by sampling pixel values ​​from adjacent frames to the viewpoint of the target frame (original image), the reconstructed image is finally obtained, expressed by the following formula:

[0124] ;

[0125] Here, <·> represents the bilinear interpolation sampling operation, used to handle the issue that the mapped pixel coordinates are continuous values ​​rather than discrete integers, thus ensuring the smoothness and accuracy of the reconstructed image. Finally, by calculating the reconstructed target frame I... s→t (Reconstructed image) and target frame I t The photometric reprojection error between the original images measures the difference between them and forms a self-supervised signal.

[0126] S50: Calculate the overall loss function of the student network. When the overall loss function is within a preset range, the student network converges, and a lightweight student network model is obtained.

[0127] The overall loss function is the sum of the pairwise loss function, the Laplacian distillation loss function, the pixel-by-pixel distillation loss function, and the base loss function. The base loss function is the sum of the photometric reprojection loss function and the smoothing loss function between the reconstructed image and the original image. The pixel-by-pixel distillation loss function is the pixel-by-pixel distillation loss function between the fused depth map and the predicted depth map. The pairwise loss function and the Laplacian distillation loss function are determined based on the intermediate feature maps of the teacher network and the student network.

[0128] Specifically, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the student network structure. The student network in this embodiment of the application adopts MobileViT, which has an efficient model architecture and optimization strategy.

[0129] First, MobileViT employs depthwise separable convolutions, which significantly reduces the number of parameters and computational cost while maintaining the ability to extract local features. Second, the transformer part uses a lightweight self-attention mechanism to reduce the computationally complex fully connected operations found in traditional transformers, significantly improving computational efficiency. Through these designs, MobileViT achieves a substantial reduction in model computational complexity while maintaining performance, making it highly suitable for resource-constrained devices. Furthermore, this student network fully leverages the strengths of both by combining the local perception capabilities of convolutional neural networks with the global modeling advantages of transformers. Convolutional operations excel at capturing information from local regions, particularly in extracting low-level visual features, while transformers are better at handling global information and long-range dependencies, effectively modeling higher-level feature representations. Therefore, the combination of these two approaches allows the student network to maintain strong feature representation capabilities while remaining lightweight.

[0130] Furthermore, this application employs a self-supervised learning mechanism to optimize the student network. Specifically, on a publicly available sequence dataset, it fully utilizes the geometric constraints and camera pose information between adjacent video frames, and completes inter-frame image reconstruction through depth prediction and image reprojection techniques, thereby obtaining self-supervised signals to guide the training process of the student network.

[0131] Based on the structure of the teacher network and student network, to further improve the effect of knowledge distillation and enhance the ability of the student network to fit the feature representation and distribution of the teacher network, the overall loss function of the student network in this embodiment is expressed by the following formula:

[0132] ;

[0133] in, Represents the overall loss function. This represents the basic loss function, which consists of the photometric reprojection loss function and the smoothing loss. The pairwise loss represents the affinity graph of the intermediate feature graphs between the teacher network and the student network. It is the Laplace distillation loss, which represents the loss at three scales when the intermediate feature maps of the teacher network and the student network are decoupled. It is the pixel-by-pixel distillation loss, which represents the logarithmic error loss of the final depth estimates of the teacher network and the student network.

[0134] In some implementations, the basic loss function can be expressed by the following formula:

[0135] ;

[0136] Where C represents the number of scales predicted by the decoder of the teacher network or the decoder of the student network, and Lph L represents the photometric reprojection loss. sm This represents the smoothing loss. λ is the weighting coefficient of the smoothing loss, while μ is not a simple weighting coefficient, but a binary mask matrix generated by the Iverson (Iverson bracket) to ignore occluded areas or pixels outside the field of view. This design ensures that the photometric reprojection loss only applies to the effective area. Iverson is a mathematical notation, represented as [P], where [P] is a Boolean logical expression. The final result depends on the truth value of condition P, and its calculation formula is expressed as:

[0137] ;

[0138] Specifically, μ is a binary mask matrix. When a pixel is not occluded during reprojection and is within the field of view, the value of μ at the corresponding position is 1; conversely, when a pixel is occluded or exceeds the field of view during reprojection, the value of μ at the corresponding position is 0, thus ignoring the reconstruction error of these pixels and only calculating the luminance loss in the visible area within the field of view. The formula for calculating μ is expressed as:

[0139] ;

[0140] in, Indicates target frame I t (Original image) and the image after reprojecting adjacent frame images onto the target frame's viewpoint The photometric reprojection error calculated between (reconstructed images); Indicates target frame I t (Original image) and source frame I s The photometric error is obtained by directly comparing pixels from adjacent frames under their respective original viewpoints. This indicates a minimization operation across all source frames (adjacent frames), meaning selecting the frame with the smallest reprojection error from multiple source frames as the optimal source frame. When the reprojection error... Less than the photometric error without reprojection When μ=1, it indicates that the pixel is valid, and photometric reprojection loss is taken into account; when reprojection error Greater than or equal to the error without reprojection When μ=0, it means that the pixel is invalid and is not included in the loss calculation.

[0141] The formula for the photometric reprojection loss is:

[0142] ;

[0143] in, represents the photometric reprojection loss function, pe represents the photometric reprojection error, and SSIM represents the structural similarity index; Represents the first-order norm of an image and calculates pixel-level differences; Indicates the weighting coefficient. This represents the original image (target frame). Represents the reconstructed image;

[0144] The smoothing loss function is calculated using the smoothing loss formula, which is:

[0145] ;

[0146] in, Represents the smoothing loss function. Indicates the depth map The mean-normalized depth value, This indicates that the predicted depth map is obtained by inputting the original image into the student network for prediction. Represents the predicted depth map The mean, and These represent the gradient calculations in the x and y directions, respectively. and Represents the image gradient, used as a weighting factor;

[0147] Based on the photometric reprojection loss function and the smoothing loss function, the fundamental loss function is calculated and expressed by the following formula:

[0148] ;

[0149] in, Let C represent the basic loss function, and let C represent the number of scales predicted by the decoder of the teacher network or the decoder of the student network. The weighting coefficients representing the smoothing loss. Represents a depth prediction map A scale-consistent binary matrix, where pixels are occluded or outside the field of view during reprojection, and the corresponding position... The value is 0, and the value is 1 otherwise.

[0150] The photometric reprojection loss function and the smoothing loss function are designed to improve the photometric consistency and the smoothness of depth estimation of images.

[0151] In some embodiments, calculating the pairwise loss function may include:

[0152] Obtain the teacher intermediate feature map of the original image through the teacher network inference;

[0153] Obtain the student intermediate feature map of the original image for the student network inference;

[0154] Based on the teacher intermediate feature map and the student intermediate feature map, an affinity map is constructed; wherein, nodes in the affinity map represent different spatial locations, and the connection relationship between two nodes represents similarity; the size of the teacher intermediate feature map or the student intermediate feature map is [size missing]. The affinity map includes There are n nodes, and each node is connected to n edges. ; Indicates the range of connections; Indicates granularity;

[0155] Based on the affinity map, the formula for calculating the pairwise loss function is as follows:

[0156] ;

[0157] in, , ;

[0158] in, and These represent the width and height of the teacher's or student's intermediate feature map, respectively. Represents all nodes. This represents the similarity between node i and node j in the affinity graph corresponding to the student network. This represents the similarity between node i and node j in the affinity graph corresponding to the teacher network. and These represent the aggregated features obtained through average pooling. The aggregation operation means performing average pooling on the features of all channels within a node, compressing them into a fixed-length feature vector, thereby extracting the global representation information of the node.

[0159] This pairwise loss function, by minimizing the affinity graph difference between the teacher and student networks, constrains the student network to learn the similarity structure of spatial locations more accurately from the perspective of pairwise relationships. It is used to guide the student and teacher networks to maintain a consistent feature data distribution between intermediate feature map pairs.

[0160] In some implementations, the Laplace distillation loss function L is calculated. Laplacion The calculation formula can be expressed as:

[0161] ;

[0162] Among them, C iIt is the total number of units contained in the feature obtained by the i-th decoder layer, and N represents the total number of decoder layers. In some embodiments, N can be set to 4, indicating that there are 4 decoder layers.

[0163] x is the spatial location index on the teacher's intermediate feature map or the student's intermediate feature map. For the i-th decoder layer, the Laplace residual can be expressed by the following formula:

[0164] ;

[0165] ;

[0166] ;

[0167] ;

[0168] Where x represents the spatial location index of the decoder feature of the student network or teacher network. It is the absolute difference of the residuals of the first layer of features of the Laplace pyramid. It is the absolute difference of the residuals of the second layer of the Laplace pyramid. It is the squared difference between the features of the teacher network and the student network after downsampling to a 1 / 4 scale. Let F represent the Laplace residual, where i represents the layer index of the decoder, j represents the scale layer index of the Laplace pyramid, and F... i,j Let represent the feature map corresponding to the j-th layer Gaussian pyramid of the i-th decoder. Up(·) represents the upsampling operation, used to adjust the feature map of the (j+1)-th layer to the same spatial resolution as the j-th layer, thereby calculating the inter-layer Laplacian residual. In this embodiment, bilinear interpolation is used for upsampling, and the number of layers in the Laplacian pyramid is set to 3.

[0169] The Laplace distillation loss function, by comparing the Laplace residuals of the teacher network and the student network on the decoder features of each layer, constrains the student network to more accurately capture multi-scale differences in depth features at a fine-grained level, thereby improving the robustness of depth estimation.

[0170] The Laplacian distillation loss function decouples global contextual information and local detail information in a multi-scale manner to capture structural and detail information at different levels of the image. By calculating the error between these corresponding decoupled features, it helps to improve the overall depth layout of the student network learning scenario and thus improve the depth estimation accuracy.

[0171] In some implementations, the pixel-by-pixel distillation loss function is calculated and can be expressed by the following formula:

[0172] ;

[0173] in, Let C represent the pixel-wise distillation loss function, and let N represent the number of scales predicted by the decoder. i This represents the total number of pixels at the corresponding scale i. Let represent the depth prediction value of the teacher network at the j-th pixel position at the i-th scale. This represents the depth prediction value of the student network at the j-th pixel position at the i-th scale.

[0174] The pixel-wise distillation loss function calculates the difference in depth values ​​between the final outputs of the teacher network and the student network, enabling the student network to accurately mimic the output of the teacher network at each pixel.

[0175] Therefore, the final formula for calculating the overall loss function is expressed as:

[0176] ;

[0177] Specifically, the photometric reprojection loss function Based on the principle of photometric consistency, the photometric error between the target frame (original image) and the reprojected frame (reconstructed image) is measured by comparing the pixel differences after reprojection from the source viewpoint to the target viewpoint. This constrains the depth estimation results to maintain consistency with the photometric relationship under the target viewpoint; a smoothing loss function is used. By applying edge-aware weighting to the depth map gradient, the depth estimation results are encouraged to remain consistent in smooth texture regions while varying flexibly in edge regions. This reduces false edges and noise in the depth estimation, ensuring its smoothness. (Pairwise loss function) By calculating the difference in affinity maps between the teacher and student networks at the intermediate feature layers, the student network is guided to learn the feature space structure of the teacher network, thereby achieving knowledge transfer at the feature level and enabling the student network to better fit the intermediate representation of the teacher network; Laplace pyramid loss function. Based on the Laplacian pyramid, the feature map is decoupled into global contextual information and local detail information, and the differences between the teacher network and the student network at different scales are calculated separately. This helps the student network to enhance its ability to capture edges and details while maintaining consistent depth layout; a pixel-wise distillation loss function is used. By comparing the final output of the teacher network with the final output of the student network pixel by pixel, minimizing the pixel-by-pixel depth difference, the output of the student network is constrained at the pixel level to make it as close as possible to the final predicted distribution of the teacher network, thereby achieving end-to-end knowledge transfer.

[0178] In summary, this overall loss function integrates constraints at the pixel, feature, and multi-scale depth estimation levels, effectively guiding the student network to learn the knowledge representations of the teacher network at different levels. The joint design of multiple loss terms proposed in this application not only comprehensively improves the robustness and generalization ability of the model but also alleviates problems such as boundary ambiguity and noise interference in depth estimation, thereby enhancing the overall performance of monocular depth estimation.

[0179] In a pose network, the pose encoder receives two stitched images as input; here, the stitched image refers to the current frame image I. t and adjacent frame images I s The pose network estimates camera motion between two adjacent frames, ultimately outputting a rotation matrix R and a translation vector t, which describes the required rotation and translation relationship between the camera and another image. The rotation matrix R is expressed as a 3×3 matrix, and the translation vector t is expressed as a 3×1 column vector. During training, the pose network iteratively adjusts the rotation matrix R and translation vector t by optimizing the aforementioned loss function, ensuring that the camera motion transformation accurately reprojects scene points from one viewpoint to another, making the projection positions of the same scene point as aligned as possible across different frames. This approach provides geometric constraints for self-supervised monocular depth estimation, enabling accurate modeling of camera motion even without real pose labels.

[0180] By analyzing the reconstructed image I s→t and current frame I t Photometric reprojection error and smoothing loss error are calculated, and pixel-wise distillation loss error, pairwise loss error, and Laplacian distillation loss error are calculated in conjunction with distillation guidance from the teacher network. These loss terms are synthesized into a final overall loss function. The student network optimizes this overall loss function, progressively adjusting and optimizing the model's weight parameters to improve its performance on the training data. This training process continues until the overall loss function value is within a certain preset range, such as stabilizing and converging to an optimal state, ultimately yielding a small, lightweight student network model distilled by the teacher network.

[0181] In some implementations, after step S50, the training method for the lightweight student network model may further include:

[0182] The monocular depth estimation performance of the lightweight student network model was verified.

[0183] If the monocular depth estimation performance verification passes, the lightweight student network model will be deployed to the drone platform.

[0184] If the monocular depth estimation performance verification fails, the training parameters of the student network are adjusted until the monocular depth estimation performance verification passes.

[0185] Specifically, the quantitative metrics for monocular depth estimation performance include absolute relative error (Abs Rel), squared relative error (Sq Rel), root mean square error (RMSE), logarithmic root mean square error (RMSE log), and accuracy.

[0186] The absolute relative error (Abs Rel) can be expressed by the formula:

[0187] ;

[0188] The absolute relative error reflects the average relative error between the predicted depth and the actual depth; the smaller the value, the more accurate the depth estimation.

[0189] The squared relative error (Sq Rel) can be expressed by the formula:

[0190] ;

[0191] The squared relative error focuses more on measuring the impact of larger errors; when the error of some pixels is large, this index value will increase significantly.

[0192] The root mean square error (RMSE) can be expressed by the formula:

[0193] ;

[0194] The root mean square error measures the standard deviation of the error between the predicted depth and the true depth, and is used to assess the magnitude of the overall error.

[0195] The root mean square error (RMSE log) can be expressed by the formula:

[0196] ;

[0197] Log-mean square error is suitable for scenarios with a wide range of depth values ​​or drastic dynamic changes, effectively reducing the impact of distant targets on error calculation.

[0198] Accuracy can be expressed by the formula:

[0199] ;

[0200] The percentage of pixels whose relative error between predicted and true depth is less than a set threshold is used to measure depth estimation accuracy. Typically, three thresholds are used to measure depth estimation accuracy: threshold=1.25, threshold=1.25², and threshold=1.25³. These three thresholds effectively evaluate the performance of the depth estimation model under different error tolerances, providing a comprehensive assessment of the model's depth estimation accuracy.

[0201] In the above formula, d i and These represent the predicted depth map and the ground truth depth map, respectively, with M representing the number of valid pixels in the depth map.

[0202] In summary, these quantitative metrics collectively evaluate the model's performance from multiple dimensions, including absolute error, relative error, overall error distribution, dynamic range adaptability, and accuracy under different error thresholds. This provides a comprehensive and detailed measurement system for deep estimation tasks, thereby ensuring that the proposed method has sufficient validation evidence in terms of robustness and accuracy.

[0203] Once these metrics pass verification, the lightweight student network model can be deployed on the drone platform; if these metrics fail verification, the parameters for training the student network are adjusted until these metrics pass verification, and then the lightweight student network model is deployed on the drone platform.

[0204] The lightweight student network model training method of this application first employs a self-supervised learning method to train the student network model, eliminating the need for labeled data and effectively reducing the cost and time of manual annotation, thus better adapting to real-world application environments. Second, it uses a knowledge distillation method based on the teacher-student network paradigm, transferring knowledge from a teacher network with a large number of parameters and strong performance to a student network model with fewer parameters. This ensures that the student network model maintains high performance while reducing the number of parameters, computational load, and storage requirements, and significantly reduces dependence on hardware computing resources, thereby achieving model lightweighting. Furthermore, a deep learning-based edge extraction algorithm is introduced into the teacher network. By acquiring the edge detection map corresponding to the original image, the teacher network receives the original image and its corresponding edge detection image as input to predict the edge depth map and the initial depth map. Subsequently, guided filtering fusion technology is used to fuse the edge depth map and the initial depth map to fully combine the overall depth layout and edge detail information, thereby improving the overall depth estimation accuracy. Finally, the fused depth map is used as a guidance signal for the student network to further enhance its depth estimation capability. Ultimately, efficient and real-time depth perception was achieved on a drone platform with limited computing resources, demonstrating significant engineering application value and promising prospects.

[0205] This application also provides a monocular depth estimation method for a UAV, which may include:

[0206] The image acquired by the monocular camera of the UAV at the current moment is input into the trained lightweight student network model to obtain the depth estimate of the UAV; the lightweight student network model is trained by the training method of the lightweight student network model.

[0207] This application proposes a monocular depth estimation method for unmanned aerial vehicles (UAVs). While ensuring the accuracy of monocular depth estimation, it effectively reduces the computational complexity of the algorithm model, improves inference speed, and significantly reduces dependence on hardware computing resources. This invention enables the algorithm to be successfully deployed on edge devices with limited computing resources, such as embedded platforms like UAVs, thereby enhancing the application potential of monocular depth estimation algorithms in real-time perception and environmental understanding. Compared with existing technologies, this invention further enhances the monocular depth estimation capability of UAVs, providing a solid technical foundation for achieving efficient and real-time visual navigation, and has significant engineering application value.

[0208] This application's embodiments are experimentally verified on the publicly available KITTI dataset, focusing on quantitative metrics and time consumption parameters. The algorithm model in this application has only 1.3M parameters and a computational cost of only 0.9 GFLOPs. Compared to the classic Monodepth2 method (which has 14.3M parameters and 8.0 GFLOPs), this application's embodiments reduce the number of parameters by 90.91% and the computational cost by 88.75%. At a threshold of 1.25³, the accuracy of this application's embodiments reaches 98.3%, with an absolute relative error of only 0.116. To further verify the algorithm performance of this application's embodiments, time consumption tests were conducted on different hardware platforms. Specifically, on an RTX 3070 Ti computing platform, the inference speed is 652.8 FPS and the processing time is 1.53 milliseconds; on the resource-constrained UAV embedded computing platform Jetson AGXOrin, the inference speed is 183.1 FPS and the processing time is 5.46 milliseconds, fully meeting the real-time requirements.

[0209] To further verify the application of the lightweight student network model proposed in this invention deployed on a drone in a real-world disaster search and rescue scenario, this application embodiment performs depth prediction on disaster search and rescue scene maps captured by the drone in a real-world setting, obtaining a depth estimation map of the real-world scene. For example... Figure 5 The image shown is a real disaster search and rescue scene captured by a drone's monocular camera. Figure 6 This is a monocular depth estimation map obtained before using the lightweight student network model proposed in this invention. Figure 7 This is a monocular depth estimation map obtained by using a lightweight student network model for the UAV proposed in this invention. Figure 6 and Figure 7 As can be seen from the comparison, the present invention significantly improves the overall depth estimation performance. Specifically, for example, in Figure 7 The present invention can more accurately capture the gradient changes of the depth edges in the head area of ​​the rescuers wearing yellow protective suits and the area of ​​the search and rescue dogs, effectively preserving and enhancing the edge details of the area, making the depth estimation results clearer and more detailed in the outline and boundary area of ​​the target object, thereby comprehensively improving the monocular depth estimation accuracy of the student network in the embodiments of this application.

[0210] This application effectively improves the monocular depth estimation accuracy of UAVs in real-world application scenarios such as disaster relief and search and rescue. Especially in complex disaster scenarios, it can accurately capture the edge and depth changes of the injured and trapped, providing a more precise overall depth layout structure, overall depth distribution, and edge information for detecting the location, outline, and spatial relationship of search and rescue personnel with the surrounding environment. This not only helps UAVs quickly locate survivors in complex environments but also provides reliable data support for path planning and obstacle avoidance decisions, thereby improving the efficiency and safety of search and rescue missions. This fully verifies the practical value and engineering significance of this invention in real-world application scenarios.

[0211] Those skilled in the art will understand that all or part of the features / steps of the above-described method embodiments can be implemented by methods, data processing systems, or computer programs. These features may be implemented without hardware, entirely in software, or in a combination of hardware and software. The aforementioned computer program may be stored in one or more computer-readable storage media. When the computer program is executed (e.g., by a processor), it performs the steps of the above-described embodiments of the quaternion-based hybrid signal in-memory computing quaternion attitude control method and the robot motion prediction method for robot motion prediction.

[0212] The aforementioned storage media capable of storing program code include: static disks, solid-state drives, random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), optical storage devices, magnetic storage devices, flash memory, magnetic disks or optical disks, and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.

[0213] This application also provides a processing apparatus, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors are configured to execute the one or more computer programs stored in the memory to enable the one or more processors to perform a training method for a lightweight student network model and a monocular depth estimation method for an unmanned aerial vehicle as described in any one of Embodiment 1.

[0214] The processing device of this invention can be a chip. When the processing device is a chip, each module in the chip can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computing device in hardware form, or stored in the memory of the computing device in software form, so that the processor can call and execute the operations corresponding to each module. This invention can be applied to fields requiring high precision, high reliability, low power consumption, and low area cost, such as artificial intelligence and metaverse; artificial intelligence training or inference chip products; autonomous driving chips; VR chips; robot embedded chips; and deep learning applications using parallel computing. It can be applied to fields requiring high precision, high reliability, low power consumption, and low area cost, such as artificial intelligence training or inference chips, autonomous driving chips, VR chips, and robot embedded chips.

[0215] This application also provides a drone, which may include a processing unit. The drone may be a drone, such as a multi-rotor drone.

[0216] Since the implementation of the aforementioned drones is described in detail in Embodiment 1, it will not be repeated here.

[0217] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A training method for a lightweight student network model, characterized in that, include: The edge detection image of the original image is extracted, and the edge detection image and the original image are respectively input into the pre-trained teacher network for inference to obtain the edge depth map and the initial depth map; The edge depth map is used as a guide map to fuse the initial depth map and the edge depth map to obtain a fused depth map, which is then used as a soft label to guide student network training. The original image is input into the student network so that the student network can be self-supervised and trained using the teacher-student network paradigm distillation method under the guidance of the soft label, and output a predicted depth map. The original image is stitched together with the adjacent frame images and then input into the pose network to obtain the pose transformation matrix. Based on the pose transformation matrix and the predicted depth map, image reconstruction is performed to obtain the reconstructed image. Calculate the overall loss function of the student network. When the overall loss function is within a preset range, the student network converges, and a lightweight student network model is obtained. The overall loss function is the sum of the pairwise loss function, the Laplacian distillation loss function, the pixel-wise distillation loss function, and the basic loss function. The basic loss function is the sum of the photometric reprojection loss function and the smoothing loss function between the reconstructed image and the original image. The pixel-wise distillation loss function is the pixel-wise distillation loss function between the fused depth map and the predicted depth map. The pairwise loss function and the Laplacian distillation loss function are determined based on the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network. The process of concatenating the original image with adjacent frame images, inputting it into a pose network to obtain a pose transformation matrix, and then reconstructing the image based on the pose transformation matrix and the predicted depth map to obtain a reconstructed image includes: The original image is then stitched together with adjacent frame images to obtain the stitched image. The stitched image is input into the pose network to obtain the pose transformation matrix; Based on the camera intrinsic parameter matrix, the predicted depth map, and the pose transformation matrix, the pixels on the original image are mapped to the corresponding pixels in the coordinate system of the adjacent frame images through spatial geometric transformation. The pixel values ​​from the adjacent frame images are sampled and applied to the viewpoint of the original image to obtain the reconstructed image; The calculation of the overall loss function of the student network includes: Calculate the base loss function, which is the photometric reprojection loss function and the smoothing loss function between the reconstructed image and the original image; Calculate the pixel-by-pixel distillation loss function between the fused depth map and the predicted depth map; Calculate the pairwise loss function between the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network, and the Laplace distillation loss function; Based on the photometric reprojection loss function, smoothing loss function, pixel-wise distillation loss function, pairwise loss function, and Laplace distillation loss function, the overall loss function is calculated using the following formula; ； in, Represents the overall loss function. Let represent the base loss function, which is the sum of the photometric reprojection loss function and the smoothing loss function. This represents the pairwise affinity loss function between the teacher intermediate feature maps of the teacher network and the student intermediate feature maps of the student network. This represents the Laplace distillation loss function. This represents the pixel-by-pixel distillation loss function.

2. The training method for the lightweight student network model according to claim 1, characterized in that, The edge depth map is used as a guide map to fuse the initial depth map and the edge depth map to obtain a fused depth map, expressed by the following formula: ； in, This represents the fused depth map, where GF represents the guided filtering operation. The initial depth map represents the overall depth information and spatial structure information of the original image; The edge depth map, including edge detail information, serves as a guide map; r represents the radius of the guide filter window, used to control the local filtering range. This represents the regularization parameter, used to prevent overfitting and ensure the smoothness of the filtering.

3. The training method for the lightweight student network model according to claim 2, characterized in that, The guided filtering operation includes: Using the initial depth map as the input image and the edge depth map as the guide image, guided filtering is performed to output an initial fused image that satisfies the local linear model. When the filtering window is located in the edge region of the guide image, the gradient change of the initial fused image in the edge direction is enhanced, highlighting edge details; When the filtering window is located in a flat area of ​​the guide image, the corresponding flat area is subjected to local mean filtering.

4. The training method for the lightweight student network model according to claim 1, characterized in that, The calculation of the pairwise loss function between the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network includes: Based on the teacher intermediate feature map and the student intermediate feature map, an affinity map is constructed; wherein, nodes in the affinity map represent different spatial locations, and the connection relationship between two nodes represents similarity; the size of the teacher intermediate feature map or the student intermediate feature map is [size missing]. The affinity map includes There are n nodes, and each node is connected to n edges. ; Indicates the range of connections; Indicates granularity; Based on the affinity map, the formula for calculating the pairwise loss function is as follows: ； in, , ; in, and These represent the width and height of the teacher's or student's intermediate feature map, respectively. Represents all nodes. This represents the similarity between node i and node j in the affinity graph corresponding to the student network. This represents the similarity between node i and node j in the affinity graph corresponding to the teacher network. and These represent the aggregated features obtained through average pooling. The aggregation operation means performing average pooling on the features of all channels within a node, compressing them into a fixed-length feature vector, thereby extracting the global representation information of the node. The Laplace distillation loss function between the teacher intermediate feature map of the teacher network and the student intermediate feature map of the student network is calculated using the following formula: ； Among them, C i is the total number of units contained in the features obtained by the i-th decoder layer of the teacher network or the student network, N represents the total number of decoder layers of the teacher network or the student network, and x is the spatial location index on the teacher's intermediate feature map. This represents the absolute difference of the residuals of the first layer of features in the Laplace pyramid. The absolute difference represents the residuals of the second layer of the Laplace pyramid. It is the squared difference between the features of the teacher network and the student network after downsampling to a 1 / 4 scale.

5. The training method for the lightweight student network model according to claim 1, characterized in that, The pixel-by-pixel distillation loss function for calculating the fused depth map and the predicted depth map is expressed by the following formula: ； in, Let N represent the pixel-wise distillation loss function, where C represents the number of scales predicted by the decoder of the teacher network or the decoder of the student network, and N represents the number of scales predicted. i This represents the number of pixels at the i-th scale. This represents the depth prediction value of the teacher network at the j-th pixel at the i-th scale. This represents the depth prediction value of the student network at the j-th pixel at the i-th scale.

6. The training method for the lightweight student network model according to claim 1, characterized in that, The calculation of the basic loss function includes: The photometric reprojection loss function is calculated using the photometric reprojection loss formula, which is as follows: ； in, represents the photometric reprojection loss function, pe represents the photometric reprojection error, and SSIM represents the structural similarity index; Represents the first-order norm of an image and calculates pixel-level differences; Indicates the weighting coefficient. This refers to the original image. This represents the reconstructed image; The smoothing loss function is calculated using the smoothing loss formula, which is: ； in, Represents the smoothing loss function. Indicates the original image Predicted depth map Depth values ​​after mean normalization This indicates the original image Input the predicted depth map obtained from the student network prediction. Represents the predicted depth map The mean, and These represent the gradient calculations in the x and y directions, respectively. and Represents the image gradient, used as a weighting factor; Based on the photometric reprojection loss function and the smoothing loss function, the fundamental loss function is calculated and expressed by the following formula: ； in, Let C represent the basic loss function, and let C represent the number of scales predicted by the decoder of the teacher network or the decoder of the student network. The weighting coefficients represent the smoothing loss. Represents a depth map related to the predicted depth map A scale-consistent binary matrix, where pixels are occluded or outside the field of view during reprojection, and the corresponding position... The value is 0, and the value is 1 otherwise.

7. The training method for the lightweight student network model according to claim 1, characterized in that, After obtaining the lightweight student network model, the training method for the lightweight student network model further includes: The monocular depth estimation performance of the lightweight student network model was verified. If the monocular depth estimation performance verification passes, the lightweight student network model will be deployed to the drone platform. If the monocular depth estimation performance verification fails, the training parameters of the student network are adjusted until the monocular depth estimation performance verification passes.

8. A monocular depth estimation method for unmanned aerial vehicles (UAVs), characterized in that, include: The image acquired by the drone's monocular camera at the current moment is input into the trained lightweight student network model to obtain the drone's depth estimate. The lightweight student network model is obtained by training using the training method for the lightweight student network model according to any one of claims 1 to 7.

Images