Relative depth estimation method and system for unmanned aerial vehicle obstacle avoidance control, electronic equipment and storage medium
Patent Information
- Application Number
- CN202411216470.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-21
Smart Images
Figure CN119338876A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of unmanned aerial vehicle obstacle avoidance control, and in particular, to a relative depth estimation method and system, electronic device and storage medium for unmanned aerial vehicle obstacle avoidance control. Background Art
[0002] Micro-nano UAVs are small aircraft with the characteristics of small size, light weight, and low power consumption. Due to their unique size advantages, they can be flexibly used in a variety of restricted environments (narrow indoor, pipeline, jungle and other operating environments).
[0003] Therefore, the flight environment of micro-nano UAVs is generally dense and complex, which places high demands on the environmental perception capabilities of micro-nano UAVs. The path planning and autonomous obstacle avoidance functions of micro-nano UAVs all require the scene depth information of the operating environment obtained by the sensor components (such as the distance information between the objects in the scene and the observer).
[0004] However, the inventors of this application found that due to the size limitations of micro-nano UAVs (such as the need for lightweight and portable design requirements), the baseline of micro-nano UAVs is relatively short and it is difficult to equip them with binocular cameras; and current micro-nano UAVs can generally only obtain a small amount of depth information in the scene through sparse dot matrix TOF (Time of Flight) ranging sensors, which has the problem of limited detection range and resolution.
[0005] The inventors believe that how to enhance the sensor's ability to obtain depth information through algorithmic technology (such as neural network algorithms) is a technical problem that needs to be solved urgently. Summary of the invention
[0006] According to one aspect of the present application, the present application provides a relative depth estimation method for unmanned aerial vehicle obstacle avoidance control, including: determining a test data set and a training data set of a preset relative depth estimation model; training an initial baseline relative depth estimation model based on the training data set to obtain a baseline relative depth estimation model; performing model compression on the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model; performing model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model; performing model testing on the lightweight relative depth estimation model according to the test data set; and performing model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
[0007] According to some embodiments of the present application, training an initial baseline relative depth estimation model based on a training data set to obtain a baseline relative depth estimation model includes: enhancing the training data set to obtain an enhanced training data set; using the first preset image in the enhanced training data set as input, and sending it to the initial baseline relative depth estimation model for training until the preset relative depth estimation loss function reaches a convergence state, so as to obtain a baseline relative depth estimation model.
[0008] According to some embodiments of the present application, the preset relative depth estimation loss function is:
[0009]
[0010] in, is the preset relative depth estimation loss function, N l is the size of the training dataset, is the scale-shift invariant loss function, d is the normalized relative depth truth value, d * is the predicted value of the relative depth after linear transformation of scale and displacement, α is the weight of gradient loss, is the gradient loss function;
[0011] The scale-shift invariant loss function is:
[0012]
[0013] Wherein, M is the number of pixels in the first preset image, and ρ is a specific type of scale-shift invariant loss function;
[0014] The gradient loss function is:
[0015]
[0016] Among them, K is the scale of the feature map, is the difference between the relative depth prediction maps at scale k.
[0017] According to some embodiments of the present application, the first preset image in the enhanced training data set is taken as input and sent to the initial baseline relative depth estimation model for training, including: determining the encoder of the initial baseline relative depth estimation model based on the StarNet network structure, and determining a lightweight decoder; inputting the first preset image into the encoder to obtain a downsampled feature map of a first preset multiple; decoding the downsampled feature map of the first preset multiple and adding the encoder features corresponding to the size of the downsampled feature map through the lightweight decoder to obtain a processed downsampled feature map; upsampling the processed downsampled feature map to obtain a downsampled fused feature map of a second preset multiple; and refining the downsampled fused feature map to obtain a relative depth prediction map.
[0018] According to some embodiments of the present application, performing model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model includes: performing model distillation on the initial lightweight relative depth estimation model based on a preset distillation loss function to obtain a lightweight relative depth estimation model; the preset distillation loss function is:
[0019]
[0020] Among them, the baseline relative depth estimation model is the teacher model, the initial lightweight relative depth estimation model is the student model, and R s is the relative depth predicted by the student model, R t is the relative depth predicted by the teacher model, y is the true relative depth in the training dataset, For R t The value after linear transformation of scale and displacement, m is the loss interval where the student model is better than the teacher model, and v is the supervision weight of the teacher model.
[0021] According to some embodiments of the present application, model testing of a lightweight relative depth estimation model based on a test data set includes: inputting a second preset image in the test data set into the lightweight relative depth estimation model to obtain a dense relative depth prediction map; determining the scale and displacement of the relative depth prediction map based on the relative depth prediction map and the true depth value of the second preset image; performing a linear transformation of the scale and displacement of the relative depth prediction map based on the scale and displacement to obtain an absolute depth prediction map; and evaluating whether the lightweight relative depth estimation model meets preset requirements based on the calculation of the depth true value of the absolute depth prediction map and the second preset image.
[0022] According to some embodiments of the present application, model post-processing is performed on a lightweight relative depth estimation model that has passed the model test to determine whether a preset divided area of a relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area, including: dividing the relative depth prediction map into a first number of rectangular areas in the horizontal direction and a second number in the vertical direction; traversing the relative depth values of all pixels in all rectangular areas; determining the pixel ratio of pixels in each rectangular area whose relative depth value is less than a first threshold among all pixels in the current rectangular area; marking a rectangular area whose pixel ratio is greater than a second threshold as an impassable area, and marking a rectangular area whose pixel ratio is less than or equal to the second threshold as a passable area.
[0023] According to another aspect of the present application, a relative depth estimation system for obstacle avoidance control of unmanned aerial vehicles is provided, including a data construction module, a model training module, a model compression module, a model testing module and a model processing module. The data construction module determines a test data set and a training data set of a preset relative depth estimation model; the model training module trains an initial baseline relative depth estimation model based on the training data set to obtain a baseline relative depth estimation model; the model compression module performs model compression on the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model, and performs model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model; the model testing module performs model testing on the lightweight relative depth estimation model according to the test data set; the model processing module performs model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether the preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
[0024] According to another aspect of the present application, an electronic device is also provided. The electronic device includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the relative depth estimation method described above.
[0025] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the relative depth estimation method described above can be implemented.
[0026] This application obtains a baseline relative depth estimation model by training the initial baseline relative depth estimation model based on the training data set, and obtains a lightweight relative depth estimation model by performing model compression and model distillation on the baseline relative depth estimation model. In addition, the lightweight relative depth estimation model is tested and post-processed by the test data set, and the passable area can be judged only by using the relative depth prediction map, and the algorithm control of the drone flight obstacle avoidance control can be realized without distance information.
[0027] The lightweight relative depth estimation model and model post-processing algorithm provided in this application have the characteristics of reducing the amount of calculation, easy training, fast calculation speed, high precision and high generalization, and can achieve high real-time relative depth estimation on the system of low-power micro-nano UAV. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0029] Figure 1 A flow chart showing a relative depth estimation method according to an embodiment of the present application is shown;
[0030] Figure 2 Another flow chart showing the relative depth estimation method according to an embodiment of the present application;
[0031] Figure 3 Another flow chart showing the relative depth estimation method according to an embodiment of the present application;
[0032] Figure 4 A schematic diagram showing a relative depth prediction map generation process according to an embodiment of the present application;
[0033] Figure 5 A schematic diagram showing a relative depth prediction map according to an embodiment of the present application;
[0034] Figure 6 Another flow chart showing the relative depth estimation method according to an embodiment of the present application;
[0035] Figure 7 Another flow chart showing the relative depth estimation method according to an embodiment of the present application;
[0036] Figure 8 A schematic diagram showing a traffic area according to an embodiment of the present application;
[0037] Fig. 9 A schematic structural diagram of a relative depth estimation system according to an embodiment of the present application is shown.
[0038] Description of reference numerals:
[0039] Relative depth estimation system 1; data construction module 10; model training module 20; model compression module 30; model testing module 40; model processing module 50. DETAILED DESCRIPTION
[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0041] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.
[0042] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0043] The terms "first", "second" and the like in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order.
[0044] The following is a clear and complete description of the technical solution of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0045] At present, depth information estimation of micro-nano UAVs is generally performed through a single camera, that is, monocular depth estimation. Monocular depth estimation can include absolute depth estimation algorithms and relative depth estimation algorithms. The absolute depth estimation algorithm estimates the real physical distance of each pixel in the monocular image from the camera plane; the relative depth estimation algorithm estimates the relative depth difference between each pixel and other pixels in the monocular image, that is, the relative depth distribution between each pixel is obtained, rather than the actual value of the physical distance.
[0046] The inventors found that the absolute depth estimation algorithm cannot learn the scene scale information based on a single visible light camera image. It can only be trained to an overfitting state in a fixed image training set, which has poor generalization and cannot meet the deployment and engineering implementation requirements. The relative depth estimation algorithm can be trained on a variety of relative depth data sets based on scale-invariant loss, and has good generalization and applicability.
[0047] However, the inventors also found that the depth information obtained by the relative depth estimation algorithm does not have scale information and cannot be directly used for the flight control of the UAV. Therefore, it is necessary to post-process the model of the relative depth estimation algorithm so that the output results of the relative depth estimation model can be applied to the control algorithm of the micro-nano UAV.
[0048] In addition, the current relative depth estimation model has a large amount of computation and parameters, which makes it impossible to achieve high real-time reasoning on lightweight embedded processors on micro-nano drones.
[0049] Based on this, according to one aspect of the present application, the present application provides a relative depth estimation method for drone obstacle avoidance control. Figure 1 A flow chart of a relative depth estimation method according to an embodiment of the present application is shown. Figure 1 As shown, the relative depth estimation method includes steps S100-S600.
[0050] As an embodiment, the relative depth estimation method provided in the present application can be used for obstacle avoidance control of drones, but can also be used in the technical field of obstacle avoidance control of other unmanned equipment (such as unmanned vehicles or unmanned submarines, etc.), and the present application does not impose any restrictions on this.
[0051] Exemplarily, the relative depth estimation method may be executed by a relative depth estimation system having computing capabilities.
[0052] According to an example embodiment, in step S100 , the relative depth estimation system determines a test data set and a training data set of a preset relative depth estimation model.
[0053] For example, the relative depth estimation system responds to user instructions and infers the public dataset of any visual task based on a high-precision preset relative depth estimation model (such as depth-anything), and can obtain the corresponding relative depth estimation pseudo-labels as a training dataset.
[0054] For example, using pseudo-label data for model training has the characteristics of rich and easily accessible training data sets, which can improve the generalization of the model.
[0055] Public datasets refer to large datasets used in machine learning, deep learning, artificial intelligence, and other fields. For example, the public dataset may be an RGB (Red, Green, Blue) image.
[0056] As an embodiment, the public dataset is a diverse, rich and complex full-scene dataset, such as the coco dataset, the taskonomy dataset and the places365 dataset.
[0057] And, the relative depth estimation system determines the NYU Depth absolute depth dataset as the test dataset in response to the user instruction. Exemplarily, the NYU Depth absolute depth dataset is a commonly used dataset for depth estimation.
[0058] In step S200 , the relative depth estimation system trains an initial baseline relative depth estimation model based on a training data set to obtain a baseline relative depth estimation model.
[0059] For example, the relative depth estimation system responds to user instructions, selects and designs an initial baseline relative depth estimation model, and trains the initial baseline relative depth estimation model according to a training data set to obtain a baseline relative depth estimation model.
[0060] Figure 2 Another flow chart of the relative depth estimation method according to an embodiment of the present application is shown.
[0061] Alternatively, if Figure 2 As shown, step S200 may include steps S210 - S220 .
[0062] In step S210, the relative depth estimation system performs enhancement processing on the training data set to obtain an enhanced training data set.
[0063] For example, the relative depth estimation system responds to user instructions and enhances the training data set using enhancement methods such as horizontal flipping, hue saturation transformation, brightness and darkness contrast transformation, Gaussian blur, and random data cropping.
[0064] In step S220, the relative depth estimation system takes the first preset image in the enhanced training data set as input and sends it to the initial baseline relative depth estimation model for training until the preset relative depth estimation loss function reaches a convergence state to obtain a baseline relative depth estimation model.
[0065] For example, the first preset image is a selected RGB image in the enhanced training data set.
[0066] The relative depth estimation system responds to user instructions and uses a preset relative depth estimation loss function to train the designed initial baseline relative depth estimation model to obtain a trained baseline relative depth estimation model.
[0067] Optionally, the preset relative depth estimation loss function is:
[0068]
[0069] in, is the preset relative depth estimation loss function, N l is the size of the training dataset, is the scale-shift invariant loss function, d is the normalized relative depth truth value, d * is the predicted value of the relative depth after linear transformation of scale and displacement, α is the weight of gradient loss, is the gradient loss function;
[0070] The scale-shift invariant loss function is:
[0071]
[0072] Wherein, M is the number of pixels in the first preset image, and ρ is a specific type of scale-shift invariant loss function.
[0073] The gradient loss function is:
[0074]
[0075] Among them, K is the scale of the feature map, is the difference between the relative depth prediction maps at scale k.
[0076] Figure 3 Another flow chart showing the relative depth estimation method according to an embodiment of the present application; Figure 4 A schematic diagram showing a relative depth prediction map generation process according to an embodiment of the present application. Figure 5 A schematic diagram showing a relative depth prediction map according to an embodiment of the present application.
[0077] Alternatively, if Figure 3 As shown, step S220 may also include steps S221-225.
[0078] In step S221, the relative depth estimation system determines an encoder of an initial baseline relative depth estimation model based on the StarNet network structure, and determines a lightweight decoder.
[0079] For example, the initial baseline relative depth estimation model adopts a U-shaped structure of an encoder and a decoder, and the encoder is an improved StarNet_S1 model pre-trained on the selected ImageNet (ImageNet Large Scale Visual Recognition Challenge, a visual object recognition database). In this way, the application uses the StarNet network structure as an encoder to speed up the reasoning speed of the model while improving accuracy. The lightweight decoder uses a 3x3 convolutional neural network and a ReLU6 (Rectified LinearUnit, nonlinear activation function) combination module.
[0080] The relative depth estimation system performs inference on the RGB image based on the encoder and lightweight decoder to obtain a dense relative depth prediction map.
[0081] In step S222, the relative depth estimation system inputs the first preset image into the encoder to obtain a downsampled feature map of a first preset multiple.
[0082] Exemplarily, the first preset multiple may be 2, 4, 8, 16, 32, etc.
[0083] For example Figure 4 As shown, the relative depth estimation system responds to user instructions and inputs the RGB image into the improved StarNet_S1 encoder to obtain downsampled feature maps of multiples of 4, 8, 16, and 32, respectively.
[0084] In step S223, the relative depth estimation system decodes the downsampled feature map of the first preset multiple and adds the encoder features corresponding to the size of the downsampled feature map through a lightweight decoder to obtain a processed downsampled feature map.
[0085] In step S224, the relative depth estimation system upsamples the processed downsampled feature map to obtain a downsampled fused feature map of a second preset multiple.
[0086] Exemplarily, the second preset multiple corresponds to the size of the processed down-sampled feature map. For example, the second preset multiple can be custom set to 2 times.
[0087] For example Figure 4 As shown, the relative depth estimation system responds to user instructions and decodes the downsampled feature maps of multiples of 4, 8, 16 and 32 through a lightweight decoder, adds them to the encoder features of the corresponding feature map size, and then performs upsampling to obtain a 2x downsampled fusion feature map.
[0088] In step S225, the relative depth estimation system refines the downsampled fused feature map to obtain a relative depth prediction map.
[0089] For example Figure 4 As shown in the figure, the relative depth estimation system responds to the user's instructions and refines the 2x downsampled fusion feature map to obtain a dense relative depth prediction map with the same resolution as the RGB image (example figure is shown in Figure 5 shown).
[0090] As an embodiment, the encoder uses transfer learning and selects the improved StarNet_S1 model pre-trained on ImageNet. For example, the depth-separable 7x7 convolution structure in StarNet is changed to a 3x3 partial convolution, that is, the redundancy in the feature map is utilized and ordinary convolution is applied only to a part of the input channels without affecting the remaining channels. Such a setting can improve the operation speed of the model while ensuring the accuracy of the model, so as to achieve high real-time performance.
[0091] The decoder uses a combination of 3x3 convolution and ReLU6 modules. The 32-fold downsampled feature map extracted by the encoder is passed through the decoder to obtain the decoded downsampled feature map, and then the decoded downsampled feature map is upsampled to a 16-fold downsampled decoding feature map. After that, the obtained 16-fold downsampled decoding feature map is added to the encoder feature map of the corresponding feature map size, and then upsampled to an 8-fold downsampled feature map.
[0092] Similarly, the 8x downsampled feature map and the 4x downsampled feature map are decoded and added in the same way to obtain a 2x downsampled fused feature map. The 2x downsampled fused feature map is then refined using a refinement module to obtain a dense relative depth prediction map with the same resolution as the RGB image. The refinement module can be composed of a 3x3 convolution and ReLU6 module and a 1x1 convolution and ReLU6 module hybrid structure and a 2x upsampling operation.
[0093] In step S300 , the relative depth estimation system compresses the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model.
[0094] For example, the relative depth estimation system responds to user instructions and further compresses the baseline relative depth estimation model. The encoder is an improved StarNet_S150 model pre-trained on the selected ImageNet. The lightweight decoder uses a 3x3 convolutional neural network and ReLU6 combination module. The number of channels is adjusted according to the feature map corresponding to the encoder, so that the compressed initial lightweight relative depth estimation model can be obtained.
[0095] In step S400 , the relative depth estimation system performs model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model.
[0096] For example, the relative depth estimation system responds to user instructions and adopts isomorphic distillation technology. It uses the trained baseline relative depth estimation model as the teacher model and the initial lightweight relative depth estimation model as the student model, and distills the lightweight relative depth estimation model according to the preset distillation loss function to obtain a distilled lightweight relative depth estimation model with higher lightweight model accuracy.
[0097] Optionally, the relative depth estimation system performs model distillation on the initial lightweight relative depth estimation model based on a preset distillation loss function to obtain a lightweight relative depth estimation model.
[0098] The default distillation loss function is:
[0099]
[0100] Among them, the baseline relative depth estimation model is the teacher model, the initial lightweight relative depth estimation model is the student model, and R s is the relative depth predicted by the student model, R t is the relative depth predicted by the teacher model, y is the true relative depth in the training dataset, For R t The value after linear transformation of scale and displacement, m is the loss interval where the student model is better than the teacher model, and v is the supervision weight of the teacher model.
[0101] Exemplarily, the value of m may be 5, and the value of v may be 1.0.
[0102] Exemplarily, the present application can further improve the accuracy of the model by selecting a distillation loss function suitable for depth estimation.
[0103] In step S500 , the relative depth estimation system performs model testing on the lightweight relative depth estimation model according to the test data set.
[0104] For example, the relative depth estimation system responds to user instructions and tests the lightweight relative depth estimation model based on a test data set to evaluate the algorithm performance of the relative depth estimation method.
[0105] Figure 6 Another flow chart of the relative depth estimation method according to an embodiment of the present application is shown.
[0106] Alternatively, if Figure 6 As shown, step S500 may also include S510-S540.
[0107] In step S510 , the relative depth estimation system inputs a second preset image in the test data set into the lightweight relative depth estimation model to obtain a dense relative depth prediction map.
[0108] For example, the second preset image is a selected RGB image in the test data set. The relative depth estimation system responds to the user instruction, takes the RGB image as input, and obtains a dense relative depth prediction map through the trained lightweight relative depth estimation model.
[0109] In step S520, the relative depth estimation system determines the scale and displacement of the relative depth prediction map according to the relative depth prediction map and the true depth value of the second preset image.
[0110] In step S530, the relative depth estimation system performs a linear transformation of the scale and displacement of the relative depth prediction map according to the scale and displacement to obtain an absolute depth prediction map.
[0111] For example, the relative depth estimation system responds to user instructions and performs least squares calculation on the relative depth prediction map and the true depth value of the RGB image to obtain the corresponding scale and displacement. The relative depth prediction map is then linearly transformed in scale and displacement to convert the relative depth prediction map into an absolute depth prediction map.
[0112] In step S540, the relative depth estimation system evaluates whether the lightweight relative depth estimation model meets preset requirements based on the absolute depth prediction image and the true depth value of the second preset image.
[0113] For example, the relative depth estimation system responds to user instructions and calculates the confidence, root mean square error, and relative error between the absolute depth prediction map and the true depth value of the RGB image. The algorithm performance of the lightweight relative depth estimation model can be evaluated based on the results of the relative error calculation.
[0114] In step S600, the relative depth estimation system performs model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
[0115] For example, the relative depth estimation system responds to user instructions, divides the relative depth prediction map into several areas, and judges whether each area is a passable area based on the relative depth value in each area, so as to realize obstacle avoidance control of the UAV during flight.
[0116] Figure 7 Another flow chart of the relative depth estimation method according to an embodiment of the present application is shown. Figure 8 A schematic diagram showing a traffic area according to an embodiment of the present application.
[0117] Alternatively, if Figure 7 As shown, step S600 may also include S610-S640.
[0118] In step S610 , the relative depth estimation system divides the relative depth prediction map into rectangular areas of a first number in the horizontal direction and a second number in the vertical direction.
[0119] In step S620 , the relative depth estimation system traverses the relative depth values of all pixels in all rectangular areas.
[0120] In step S630, the relative depth estimation system determines the pixel ratio of pixels having relative depth values less than a first threshold in each rectangular region among all pixels in the current rectangular region.
[0121] In step S640, a rectangular area whose pixel ratio is greater than a second threshold is marked as an impassable area, and a rectangular area whose pixel ratio is less than or equal to the second threshold is marked as a passable area.
[0122] For example, the relative depth estimation system responds to the user instruction and divides the relative depth prediction map into N horizontal x , in the vertical direction, the number is N y The relative depth estimation system traverses the relative depth values of all pixels in each rectangular area, and calculates the relative depth value in each area that is less than the first preset threshold Threshold 1 The pixel ratio of all pixels in the current rectangular area 1 .
[0123] like Figure 8 As shown, in the pixel ratio Ratio 1 Greater than the second threshold Threshold 2 In the case of , the relative depth estimation system marks the current rectangular area as an impassable area; in the case of pixel ratio Ratio 1 Less than or equal to the second threshold Threshold 2 In this case, the relative depth estimation system marks the current rectangular area as a passable area.
[0124] Through the above embodiments, the present application obtains a baseline relative depth estimation model by training the initial baseline relative depth estimation model based on the training data set, and obtains a lightweight relative depth estimation model by performing model compression and model distillation on the baseline relative depth estimation model. In addition, the lightweight relative depth estimation model is tested and post-processed by the test data set, and the passable area can be judged only by using the relative depth prediction map, and the algorithm control of the drone flight obstacle avoidance control can be realized without distance information.
[0125] The lightweight relative depth estimation model and model post-processing algorithm provided in this application have the characteristics of reducing the amount of calculation, easy training, fast calculation speed, high precision and high generalization, and can achieve high real-time relative depth estimation on the system of low-power micro-nano UAV.
[0126] For example, according to an experimental embodiment of the present application, the computational complexity of the relative depth estimation method provided by the present application can be 0.609GFLOPs, the parameter quantity is 1.49MParams, and the inference time is 75ms. The confidence of the test result on nyu is 90.6%, the root mean square error is 0.399m, and the relative error ratio is 0.094.
[0127] According to another aspect of the present application, the present application provides a relative depth estimation system for unmanned aerial vehicle obstacle avoidance control. Fig. 9 FIG. 2 is a schematic diagram showing the structure of a relative depth estimation system according to an embodiment of the present application. Fig. 9 As shown, the relative depth estimation system 1 includes a data construction module 10 , a model training module 20 , a model compression module 30 , a model testing module 40 and a model processing module 50 .
[0128] According to an example embodiment, the data construction module 10 determines a test data set and a training data set of a preset relative depth estimation model.
[0129] For example, the data construction module 10 responds to user instructions and infers a public data set of any visual task based on a high-precision preset relative depth estimation model (such as depth-anything), and can obtain corresponding relative depth estimation pseudo-labels as a training data set.
[0130] For example, using pseudo-label data for model training has the characteristics of rich and easily accessible training data sets, which can improve the generalization of the model.
[0131] Public datasets refer to large datasets used in machine learning, deep learning, artificial intelligence, and other fields. For example, the public dataset may be an RGB (Red, Green, Blue) image.
[0132] As an embodiment, the public dataset is a diverse, rich and complex full-scene dataset, such as the coco dataset, the taskonomy dataset and the places365 dataset.
[0133] And, in response to the user instruction, the data construction module 10 determines the NYU Depth absolute depth dataset as the test dataset. Exemplarily, the NYU Depth absolute depth dataset is a commonly used dataset for depth estimation.
[0134] According to an example embodiment, the model training module 20 trains an initial baseline relative depth estimation model based on a training data set to obtain a baseline relative depth estimation model.
[0135] For example, the model training module 20 responds to user instructions, selects and designs an initial baseline relative depth estimation model, and performs model training on the initial baseline relative depth estimation model according to a training data set to obtain a baseline relative depth estimation model.
[0136] Optionally, the model training module 20 performs enhancement processing on the training data set to obtain an enhanced training data set.
[0137] For example, the model training module 20 responds to user instructions and enhances the training data set using enhancement methods such as horizontal flipping, hue saturation transformation, brightness contrast transformation, Gaussian blur and random data cropping.
[0138] Optionally, the model training module 20 takes the first preset image in the enhanced training data set as input and sends it to the initial baseline relative depth estimation model for training until the preset relative depth estimation loss function reaches a convergence state to obtain a baseline relative depth estimation model.
[0139] For example, the first preset image is a selected RGB image in the enhanced training data set.
[0140] The model training module 20 responds to user instructions and uses a preset relative depth estimation loss function to train the designed initial baseline relative depth estimation model to obtain a trained baseline relative depth estimation model.
[0141] Optionally, the preset relative depth estimation loss function is:
[0142]
[0143] in, is the preset relative depth estimation loss function, N l is the size of the training dataset, is the scale-shift invariant loss function, d is the normalized relative depth truth value, d * is the predicted value of the relative depth after linear transformation of scale and displacement, α is the weight of gradient loss, is the gradient loss function;
[0144] The scale-shift invariant loss function is:
[0145]
[0146] Wherein, M is the number of pixels in the first preset image, and ρ is a specific type of scale-shift invariant loss function.
[0147] The gradient loss function is:
[0148]
[0149] Among them, K is the scale of the feature map, is the difference between the relative depth prediction maps at scale k.
[0150] Optionally, the model training module 20 determines an encoder of an initial baseline relative depth estimation model based on the StarNet network structure, and determines a lightweight decoder.
[0151] For example, the initial baseline relative depth estimation model adopts a U-shaped structure of an encoder and a decoder, and the encoder is an improved StarNet_S1 model pre-trained on the selected ImageNet (ImageNet Large Scale Visual Recognition Challenge, a visual object recognition database). In this way, the application uses the StarNet network structure as an encoder to speed up the reasoning speed of the model while improving accuracy. The lightweight decoder uses a 3x3 convolutional neural network and a ReLU6 (Rectified LinearUnit, nonlinear activation function) combination module.
[0152] The model training module 20 performs reasoning on the RGB image based on the encoder and the lightweight decoder to obtain a dense relative depth prediction map.
[0153] Optionally, the model training module 20 inputs a first preset image into an encoder to obtain a downsampled feature map of a first preset multiple.
[0154] Exemplarily, the first preset multiple may be 2, 4, 8, 16, 32, etc.
[0155] The model training module 20 responds to user instructions and inputs the RGB image into the improved StarNet_S1 encoder to obtain downsampled feature maps of multiples of 4, 8, 16 and 32 respectively.
[0156] Optionally, the model training module 20 decodes the downsampled feature map of the first preset multiple and adds the encoder features corresponding to the size of the downsampled feature map through a lightweight decoder to obtain a processed downsampled feature map.
[0157] Optionally, the model training module 20 upsamples the processed downsampled feature map to obtain a downsampled fused feature map of a second preset multiple.
[0158] Exemplarily, the second preset multiple corresponds to the size of the processed down-sampled feature map. For example, the second preset multiple can be custom set to 2 times.
[0159] For example Figure 4As shown, the model training module 20 responds to user instructions, and decodes the downsampled feature maps of multiples of 4, 8, 16 and 32 through a lightweight decoder, adds the encoder features of the corresponding feature map size, and then performs upsampling, so as to obtain a 2x downsampled fusion feature map.
[0160] Optionally, the model training module 20 refines the downsampled fusion feature map to obtain a relative depth prediction map.
[0161] For example Figure 4 As shown, the model training module 20 responds to the user's instruction and refines the 2x downsampled fusion feature map to obtain a dense relative depth prediction map with the same resolution as the RGB image (example figure is shown in Figure 5 shown).
[0162] According to an example embodiment, the model compression module 30 performs model compression on the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model.
[0163] For example, the model compression module 30 responds to the user instruction to further compress the baseline relative depth estimation model. The encoder is an improved StarNet_S150 model pre-trained on the selected ImageNet. The lightweight decoder uses a 3x3 convolutional neural network and a ReLU6 combination module. The number of channels is adjusted according to the feature map corresponding to the encoder, so that the compressed initial lightweight relative depth estimation model can be obtained.
[0164] According to an example embodiment, the model compression module 30 performs model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model.
[0165] For example, the model compression module 30 responds to user instructions, adopts isomorphic distillation technology, uses the trained baseline relative depth estimation model as the teacher model, and the initial lightweight relative depth estimation model as the student model, and distills the lightweight relative depth estimation model according to a preset distillation loss function to obtain a distilled lightweight relative depth estimation model with higher lightweight model accuracy.
[0166] Optionally, the model compression module 30 performs model distillation on the initial lightweight relative depth estimation model based on a preset distillation loss function to obtain a lightweight relative depth estimation model.
[0167] The default distillation loss function is:
[0168]
[0169] Among them, the baseline relative depth estimation model is the teacher model, the initial lightweight relative depth estimation model is the student model, and R sis the relative depth predicted by the student model, R t is the relative depth predicted by the teacher model, y is the true relative depth in the training dataset, For R t The value after linear transformation of scale and displacement, m is the loss interval where the student model is better than the teacher model, and v is the supervision weight of the teacher model.
[0170] Exemplarily, the value of m may be 5, and the value of v may be 1.0.
[0171] Exemplarily, the present application can further improve the accuracy of the model by selecting a distillation loss function suitable for depth estimation.
[0172] According to an example embodiment, the model testing module 40 performs model testing on the lightweight relative depth estimation model according to the test data set.
[0173] For example, the model testing module 40 responds to a user instruction and tests the lightweight relative depth estimation model based on a test data set to evaluate the algorithm performance of the relative depth estimation method.
[0174] Optionally, the model testing module 40 inputs a second preset image in the test data set into the lightweight relative depth estimation model to obtain a dense relative depth prediction map.
[0175] For example, the second preset image is a selected RGB image in the test data set. The model testing module 40 responds to the user instruction, takes the RGB image as input, and obtains a dense relative depth prediction map through the trained lightweight relative depth estimation model.
[0176] Optionally, the model testing module 40 determines the scale and displacement of the relative depth prediction image according to the relative depth prediction image and the true depth value of the second preset image.
[0177] Optionally, the model testing module 40 performs a linear transformation of the scale and displacement of the relative depth prediction map according to the scale and displacement to obtain an absolute depth prediction map.
[0178] For example, the model testing module 40 responds to the user's instruction and performs least squares calculation on the relative depth prediction map and the true depth value of the RGB image to obtain the corresponding scale and displacement. Then, the relative depth prediction map is linearly transformed in scale and displacement to convert the relative depth prediction map into an absolute depth prediction map.
[0179] Optionally, the model testing module 40 evaluates whether the lightweight relative depth estimation model meets preset requirements based on the absolute depth prediction map and the true depth value of the second preset image.
[0180] For example, the model testing module 40 responds to user instructions and calculates the confidence, root mean square error and relative error of the absolute depth prediction map and the true depth value of the RGB image. The algorithm performance of the lightweight relative depth estimation model can be evaluated based on the result of the relative error calculation.
[0181] According to an example embodiment, the model processing module 50 performs model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
[0182] For example, the model processing module 50 responds to user instructions, divides the relative depth prediction map into several areas, and judges whether each area is a passable area based on the relative depth value in each area, so as to realize obstacle avoidance control of the drone during flight.
[0183] Optionally, the model processing module 50 divides the relative depth prediction map into rectangular areas of a first number in the horizontal direction and a second number in the vertical direction.
[0184] Optionally, the model processing module 50 traverses the relative depth values of all pixels in all rectangular areas.
[0185] Optionally, the model processing module 50 determines the pixel ratio of pixels having relative depth values less than a first threshold in each rectangular region among all pixels in the current rectangular region.
[0186] Optionally, the model processing module 50 marks a rectangular area whose pixel ratio is greater than a second threshold as an impassable area, and marks a rectangular area whose pixel ratio is less than or equal to the second threshold as a passable area.
[0187] For example, the model processing module 50 responds to the user instruction and divides the relative depth prediction map into N horizontal x , in the vertical direction, the number is N y The model processing module 50 traverses the relative depth values of all pixels in each rectangular area, and calculates the relative depth value of each area that is less than the first preset threshold Threshold 1 The pixel ratio of all pixels in the current rectangular area 1 .
[0188] like Figure 8 As shown, in the pixel ratio Ratio 1 Greater than the second threshold Threshold 2 In the case of , the model processing module 50 marks the current rectangular area as an impassable area; in the case of pixel ratio Ratio 1 Less than or equal to the second threshold Threshold2 In this case, the model processing module 50 marks the current rectangular area as a passable area.
[0189] Through the above embodiments, the present application obtains a baseline relative depth estimation model by training the initial baseline relative depth estimation model based on the training data set, and obtains a lightweight relative depth estimation model by performing model compression and model distillation on the baseline relative depth estimation model. In addition, the lightweight relative depth estimation model is tested and post-processed by the test data set, and the passable area can be judged only by using the relative depth prediction map, and the algorithm control of the drone flight obstacle avoidance control can be realized without distance information.
[0190] The lightweight relative depth estimation model and model post-processing algorithm provided in this application have the characteristics of reducing the amount of calculation, easy training, fast calculation speed, high precision and high generalization, and can achieve high real-time relative depth estimation on the system of low-power micro-nano UAV.
[0191] For example, according to an experimental embodiment of the present application, the computational complexity of the relative depth estimation method provided by the present application can be 0.609GFLOPs, the parameter quantity is 1.49MParams, and the inference time is 75ms. The confidence of the test result on nyu is 90.6%, the root mean square error is 0.399m, and the relative error ratio is 0.094.
[0192] According to another aspect of the present application, an electronic device is also provided. The electronic device includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the relative depth estimation method described above.
[0193] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the relative depth estimation method described above can be implemented.
[0194] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A relative depth estimation method for unmanned aerial vehicle obstacle avoidance control, characterized in that: include: Determine a test data set and a training data set of a preset relative depth estimation model; Training an initial baseline relative depth estimation model based on the training data set to obtain a baseline relative depth estimation model; Performing model compression on the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model; Performing model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model; Performing model testing on the lightweight relative depth estimation model according to the test data set; Model post-processing is performed on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
2. The relative depth estimation method according to claim 1, characterized in that: The training of the initial baseline relative depth estimation model based on the training data set to obtain the baseline relative depth estimation model includes: Performing enhancement processing on the training data set to obtain an enhanced training data set; The first preset image in the enhanced training data set is used as input and sent to the initial baseline relative depth estimation model for training until the preset relative depth estimation loss function reaches a convergence state to obtain the baseline relative depth estimation model.
3. The relative depth estimation method according to claim 2, characterized in that: The preset relative depth estimation loss function is: in, is the preset relative depth estimation loss function, N l is the size of the training dataset, is the scale-shift invariant loss function, d is the normalized relative depth truth value, d * is the predicted value of the relative depth after linear transformation of scale and displacement, α is the weight of gradient loss, is the gradient loss function; The scale-shift invariant loss function is: Wherein, M is the number of pixels in the first preset image, and ρ is the specific type of the scale-shift invariant loss function; The gradient loss function is: Among them, K is the scale of the feature map, is the difference between the relative depth prediction maps at scale k.
4. The relative depth estimation method according to claim 2, characterized in that: The taking the first preset image in the enhanced training data set as input and sending it to the initial baseline relative depth estimation model for training comprises: Determine an encoder of the initial baseline relative depth estimation model based on the StarNet network structure, and determine a lightweight decoder; Inputting the first preset image into the encoder to obtain a downsampled feature map of a first preset multiple; The lightweight decoder performs decoding processing on the downsampled feature map of the first preset multiple and adds encoder features corresponding to the size of the downsampled feature map to obtain a processed downsampled feature map; Upsampling the processed downsampled feature map to obtain a downsampled fusion feature map of a second preset multiple; The downsampled fusion feature map is refined to obtain a relative depth prediction map.
5. The relative depth estimation method according to claim 3, characterized in that: The performing model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model includes: Performing model distillation on the initial lightweight relative depth estimation model based on a preset distillation loss function to obtain the lightweight relative depth estimation model; The preset distillation loss function is: The baseline relative depth estimation model is the teacher model, the initial lightweight relative depth estimation model is the student model, and R s is the relative depth predicted by the student model, R t is the relative depth predicted by the teacher model, y is the true value of the relative depth in the training dataset, For R t The value after linear transformation of scale and displacement, m is the loss interval where the student model is better than the teacher model, and v is the supervision weight of the teacher model.
6. The relative depth estimation method according to claim 1, characterized in that: The performing model testing on the lightweight relative depth estimation model according to the test data set comprises: Inputting a second preset image in the test data set into the lightweight relative depth estimation model to obtain a dense relative depth prediction map; Determining a scale and a displacement of the relative depth prediction map according to the relative depth prediction map and a true depth value of the second preset image; Performing a linear transformation of the scale and displacement of the relative depth prediction map according to the scale and displacement to obtain an absolute depth prediction map; The lightweight relative depth estimation model is evaluated based on the absolute depth prediction map and the true depth value of the second preset image to see whether it meets the preset requirements.
7. The relative depth estimation method according to claim 6, characterized in that: The performing model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area includes: Dividing the relative depth prediction map into a first number of rectangular areas in the horizontal direction and a second number in the vertical direction; Traverse the relative depth values of all pixels in all rectangular areas; Determine the pixel ratio of pixels having a relative depth value less than a first threshold in each of the rectangular regions among all pixels in the current rectangular region; The rectangular area whose pixel ratio is greater than a second threshold is marked as an impassable area, and the rectangular area whose pixel ratio is less than or equal to the second threshold is marked as a passable area.
8. A relative depth estimation system for obstacle avoidance control of unmanned aerial vehicles, characterized in that: include: A data construction module, determining a test data set and a training data set of a preset relative depth estimation model; A model training module, which trains an initial baseline relative depth estimation model based on the training data set to obtain a baseline relative depth estimation model; A model compression module, which performs model compression on the baseline relative depth estimation model to obtain an initial lightweight relative depth estimation model, and performs model distillation on the initial lightweight relative depth estimation model to obtain a distilled lightweight relative depth estimation model; A model testing module, performing a model test on the lightweight relative depth estimation model according to the test data set; The model processing module performs model post-processing on the lightweight relative depth estimation model that passes the model test to determine whether a preset divided area of the relative depth prediction map obtained according to the lightweight relative depth estimation model is a passable area.
9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the relative depth estimation method as described in any one of claims 1-7.
10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the relative depth estimation method according to any one of claims 1 to 7 is implemented.